freedak/OpenMontage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial release — OpenMontage: the first open-source agentic video production system

Browse Source 
11 production pipelines, 47 tools, 124 agent skills.
Supports cloud APIs (fal.ai, OpenAI, ElevenLabs, Suno, HeyGen, Runway) and
free local providers (diffusers, Piper TTS, WAN 2.1, Hunyuan, CogVideo).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
calesthio
2026-03-29 08:25:17 -07:00
commit a3e735cc7a
1147 changed files with 240221 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/acestep/SKILL.md
+264
View File
@@ -0,0 +1,264 @@
---
name: acestep
description: AI music generation with ACE-Step 1.5 — background music, vocal tracks, covers, stem extraction for video production. Use when generating music, soundtracks, jingles, or working with audio stems. Triggers include background music, soundtrack, jingle, music generation, stem extraction, cover, style transfer, or musical composition tasks.
---
# ACE-Step 1.5 Music Generation
Open-source music generation (MIT license) via `tools/music_gen.py`. Runs on RunPod serverless.
Requires `RUNPOD_API_KEY` and `RUNPOD_ACESTEP_ENDPOINT_ID` in `.env` (run `--setup` to create endpoint).
## Quick Reference
```bash
# Basic generation
python tools/music_gen.py --prompt "Upbeat tech corporate" --duration 60 --output bg.mp3
# With musical control
python tools/music_gen.py --prompt "Calm ambient piano" --duration 30 --bpm 72 --key "D Major" --output ambient.mp3
# Scene presets (video production)
python tools/music_gen.py --preset corporate-bg --duration 60 --output bg.mp3
python tools/music_gen.py --preset tension --duration 20 --output problem.mp3
python tools/music_gen.py --preset cta --brand digital-samba --duration 15 --output cta.mp3
# Vocals with lyrics
python tools/music_gen.py --prompt "Indie pop jingle" --lyrics "[verse]\nBuild it better\nShip it faster" --duration 30 --output jingle.mp3
# Cover / style transfer
python tools/music_gen.py --cover --reference theme.mp3 --prompt "Jazz piano version" --duration 60 --output jazz_cover.mp3
# Stem extraction
python tools/music_gen.py --extract vocals --input mixed.mp3 --output vocals.mp3
# List presets
python tools/music_gen.py --list-presets
```
## Creating a Song (Step by Step)
### 1. Instrumental background track (simplest)
```bash
python tools/music_gen.py --prompt "Upbeat indie rock, driving drums, jangly guitar" --duration 60 --bpm 120 --key "G Major" --output track.mp3
```
### 2. Song with vocals and lyrics
Write lyrics in a temp file or pass inline. Use structure tags to control song sections.
```bash
# Write lyrics to a file first (recommended for longer songs)
cat > /tmp/lyrics.txt << 'LYRICS'
[Verse 1]
Walking through the morning light
Coffee in my hand feels right
Another day to build and dream
Nothing's ever what it seems
[Chorus - anthemic]
WE KEEP MOVING FORWARD
Through the noise and doubt
We keep moving forward
That's what it's about
[Verse 2]
Screens are glowing late at night
Shipping code until it's right
The deadline's close but so are we
Almost there, just wait and see
[Chorus - bigger]
WE KEEP MOVING FORWARD
Through the noise and doubt
We keep moving forward
That's what it's about
[Outro - fade]
(Moving forward...)
LYRICS
# Generate the song
python tools/music_gen.py \
--prompt "Upbeat indie rock anthem, male vocal, driving drums, electric guitar, studio polish" \
--lyrics "$(cat /tmp/lyrics.txt)" \
--duration 60 \
--bpm 128 \
--key "G Major" \
--output my_song.mp3
```
### 3. Using a preset for video background
```bash
python tools/music_gen.py --preset tension --duration 20 --output problem_scene.mp3
```
### Key tips for good results
- **Caption = overall style** (genre, instruments, mood, production quality)
- **Lyrics = temporal structure** (verse/chorus flow, vocal delivery)
- **UPPERCASE in lyrics** = high vocal intensity
- **Parentheses** = background vocals: "We rise (together)"
- **Keep 6-10 syllables per line** for natural rhythm
- **Don't describe the melody in the caption** — describe the *sound* and *feeling*
- **Use `--seed`** to lock randomness when iterating on prompt/lyrics
## Scene Presets
| Preset | BPM | Key | Use Case |
|--------|-----|-----|----------|
| `corporate-bg` | 110 | C Major | Professional background, presentations |
| `upbeat-tech` | 128 | G Major | Product launches, tech demos |
| `ambient` | 72 | D Major | Overview slides, reflective content |
| `dramatic` | 90 | D Minor | Reveals, announcements |
| `tension` | 85 | A Minor | Problem statements, challenges |
| `hopeful` | 120 | C Major | Solution reveals, resolutions |
| `cta` | 135 | E Major | Call to action, closing energy |
| `lofi` | 85 | F Major | Screen recordings, coding demos |
## Task Types
### text2music (default)
Generate music from text prompt + optional lyrics.
### cover
Style transfer from reference audio. Control blend with `--cover-strength` (0.0-1.0):
- **0.2** — Loose style inspiration (more creative freedom)
- **0.5** — Balanced style transfer
- **0.7** — Close to original structure (default)
- **1.0** — Maximum fidelity to source
### extract
Stem separation — isolate individual tracks from mixed audio.
Tracks: `vocals`, `drums`, `bass`, `guitar`, `piano`, `keyboard`, `strings`, `brass`, `woodwinds`, `other`
### repaint (future)
Regenerate a specific time segment within existing audio while preserving the rest.
### lego (future, requires base model)
Generate individual instrument tracks within an existing audio context.
### complete (future, requires base model)
Extend partial compositions by adding specified instruments.
## Prompt Engineering
### Caption Writing — Layer Dimensions
Write captions by layering multiple descriptive dimensions rather than single-word descriptions.
**Dimensions to include:**
- **Genre/Style**: pop, rock, jazz, electronic, lo-fi, synthwave, orchestral
- **Emotion/Mood**: melancholic, euphoric, dreamy, nostalgic, intimate, tense
- **Instruments**: acoustic guitar, synth pads, 808 drums, strings, brass, piano
- **Timbre**: warm, crisp, airy, punchy, lush, polished, raw
- **Era**: "80s synth-pop", "modern indie", "classical romantic"
- **Production**: lo-fi, studio-polished, live recording, cinematic
- **Vocal**: breathy, powerful, falsetto, raspy, spoken word (or "instrumental")
**Good**: "Slow melancholic piano ballad with intimate female vocal, warm strings building to powerful chorus, studio-polished production"
**Bad**: "Sad song"
### Key Principles
1. **Specificity over vagueness** — describe instruments, mood, production style
2. **Avoid contradictions** — don't request "classical strings" and "hardcore metal" simultaneously
3. **Repetition reinforces priority** — repeat important elements for emphasis
4. **Sparse captions = more creative freedom** — detailed captions constrain the model
5. **Use metadata params for BPM/key** — don't write "120 BPM" in the caption, use `--bpm 120`
### Lyrics Formatting
**Structure tags** (use in lyrics, not caption):
```
[Intro]
[Verse]
[Chorus]
[Bridge]
[Outro]
[Instrumental]
[Guitar Solo]
[Build]
[Drop]
[Breakdown]
```
**Vocal control** (prefix lines or sections):
```
[raspy vocal]
[whispered]
[falsetto]
[powerful belting]
[harmonies]
[ad-lib]
```
**Energy indicators:**
- UPPERCASE = high intensity ("WE RISE ABOVE")
- Parentheses = background vocals ("We rise (together)")
- Keep 6-10 syllables per line within sections for natural rhythm
**Example — Tech Product Jingle:**
```
[Verse]
Build it better, ship it faster
Every feature tells a story
[Chorus - anthemic]
THIS IS YOUR PLATFORM
Your vision, your stage
Digital Samba, every page
[Outro - fade]
(Build it better...)
```
## Video Production Integration
### Music for Scene Types
| Scene | Preset | Duration | Notes |
|-------|--------|----------|-------|
| Title | `dramatic` or `ambient` | 3-5s | Short, mood-setting |
| Problem | `tension` | 10-15s | Dark, unsettling |
| Solution | `hopeful` | 10-15s | Relief, optimism |
| Demo | `lofi` or `corporate-bg` | 30-120s | Non-distracting, matches demo length |
| Stats | `upbeat-tech` | 8-12s | Building credibility |
| CTA | `cta` | 5-10s | Maximum energy, punchy |
| Credits | `ambient` | 5-10s | Gentle fade-out |
### Timing Workflow
1. Plan scene durations first (from voiceover script)
2. Generate music to match: `--duration <scene_seconds>`
3. Music duration is precise (within 0.1s of requested)
4. For background music spanning multiple scenes: generate one long track
### Combining with Voiceover
Background music should be mixed at 10-20% volume in Remotion:
```tsx
<Audio src={staticFile('voiceover.mp3')} volume={1} />
<Audio src={staticFile('bg-music.mp3')} volume={0.15} />
```
For music under narration: use instrumental presets (`corporate-bg`, `ambient`, `lofi`).
For music-forward scenes (title, CTA): can use higher volume or vocal tracks.
### Brand Consistency
Use `--brand <name>` to load hints from `brands/<name>/brand.json`.
Use `--cover --reference brand_theme.mp3` to create variations of a brand's sonic identity.
For consistent sound across a project: fix the seed (`--seed 42`) and vary only duration/prompt.
## Technical Details
- **Output**: 48kHz MP3/WAV/FLAC
- **Duration range**: 10-600 seconds
- **BPM range**: 30-300
- **Inference**: ~2-3s on GPU (turbo, 8 steps), ~40-60s on Mac MPS
- **Turbo model**: 8 steps, no CFG needed, fast and good quality
- **Shift parameter**: 3.0 recommended for turbo (improves quality)
### When NOT to use ACE-Step
- **Voice cloning** — use Qwen3-TTS or ElevenLabs instead
- **Sound effects** — use ElevenLabs SFX (`tools/sfx.py`)
- **Speech/narration** — use voiceover tools, not music gen
- **Stem extraction from video** — extract audio first with FFmpeg, then use `--extract`
.claude/skills/agents/SKILL.md
+302
View File
@@ -0,0 +1,302 @@
---
name: agents
description: Build voice AI agents with ElevenLabs. Use when creating voice assistants, customer service bots, interactive voice characters, or any real-time voice conversation experience.
license: MIT
compatibility: Requires internet access and an ElevenLabs API key (ELEVENLABS_API_KEY).
metadata: {"openclaw": {"requires": {"env": ["ELEVENLABS_API_KEY"]}, "primaryEnv": "ELEVENLABS_API_KEY"}}
---
# ElevenLabs Agents Platform
Build voice AI agents with natural conversations, multiple LLM providers, custom tools, and easy web embedding.
> **Setup:** See [Installation Guide](references/installation.md) for CLI and SDK setup.
## Quick Start with CLI
The ElevenLabs CLI is the recommended way to create and manage agents:
```bash
# Install CLI and authenticate
npm install -g @elevenlabs/cli
elevenlabs auth login
# Initialize project and create an agent
elevenlabs agents init
elevenlabs agents add "My Assistant" --template complete
# Push to ElevenLabs platform
elevenlabs agents push
```
**Available templates:** `complete`, `minimal`, `voice-only`, `text-only`, `customer-service`, `assistant`
### Python
```python
from elevenlabs import ElevenLabs
client = ElevenLabs()
agent = client.conversational_ai.agents.create(
name="My Assistant",
enable_versioning=True,
conversation_config={
"agent": {
"first_message": "Hello! How can I help?",
"language": "en",
"prompt": {
"prompt": "You are a helpful assistant. Be concise and friendly.",
"llm": "gemini-2.0-flash",
"temperature": 0.7
}
},
"tts": {"voice_id": "JBFqnCBsd6RMkjVDRZzb"}
}
)
```
### JavaScript
```javascript
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
const client = new ElevenLabsClient();
const agent = await client.conversationalAi.agents.create({
name: "My Assistant",
enableVersioning: true,
conversationConfig: {
agent: {
firstMessage: "Hello! How can I help?",
language: "en",
prompt: {
prompt: "You are a helpful assistant.",
llm: "gemini-2.0-flash",
temperature: 0.7
}
},
tts: { voiceId: "JBFqnCBsd6RMkjVDRZzb" }
}
});
```
### cURL
```bash
curl -X POST "https://api.elevenlabs.io/v1/convai/agents/create?enable_versioning=true" \
-H "xi-api-key: $ELEVENLABS_API_KEY" -H "Content-Type: application/json" \
-d '{"name": "My Assistant", "conversation_config": {"agent": {"first_message": "Hello!", "language": "en", "prompt": {"prompt": "You are helpful.", "llm": "gemini-2.0-flash"}}, "tts": {"voice_id": "JBFqnCBsd6RMkjVDRZzb"}}}'
```
## Starting Conversations
**Server-side (Python):** Get signed URL for client connection:
```python
signed_url = client.conversational_ai.conversations.get_signed_url(
agent_id="your-agent-id",
environment="staging",
)
```
**Client-side (JavaScript):**
```javascript
import { Conversation } from "@elevenlabs/client";
const conversation = await Conversation.startSession({
agentId: "your-agent-id",
environment: "staging",
onMessage: (msg) => console.log("Agent:", msg.message),
onUserTranscript: (t) => console.log("User:", t.message),
onError: (e) => console.error(e)
});
```
**React Hook:**
```typescript
import { useConversation } from "@elevenlabs/react";
const conversation = useConversation({ onMessage: (msg) => console.log(msg) });
// Get a signed URL for the target environment from your backend, then:
await conversation.startSession({ signedUrl: token });
```
## Configuration
| Provider | Models |
|----------|--------|
| OpenAI | `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.1-nano`, `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo` |
| Anthropic | `claude-sonnet-4-6`, `claude-sonnet-4-5`, `claude-sonnet-4`, `claude-haiku-4-5`, `claude-3-7-sonnet`, `claude-3-5-sonnet`, `claude-3-haiku` |
| Google | `gemini-3.1-flash-lite-preview`, `gemini-3-pro-preview`, `gemini-3-flash-preview`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`, `gemini-2.0-flash`, `gemini-2.0-flash-lite` |
| ElevenLabs | `glm-45-air-fp8`, `qwen3-30b-a3b`, `gpt-oss-120b` |
| Custom | `custom-llm` (bring your own endpoint) |
Use `GET /v1/convai/llm/list` to inspect the current model catalog, including deprecation state, token/context limits, and capability flags such as image-input support.
**Popular voices:** `JBFqnCBsd6RMkjVDRZzb` (George), `EXAVITQu4vr4xnSDxMaL` (Sarah), `onwK4e9ZLuTAKqWW03F9` (Daniel), `XB0fDUnXU5powFXDhCwa` (Charlotte)
**Turn eagerness:** `patient` (waits longer for user to finish), `normal`, or `eager` (responds quickly)
See [Agent Configuration](references/agent-configuration.md) for all options.
## Tools
Extend agents with webhook, client, or built-in system tools. Tools are defined inside `conversation_config.agent.prompt`:
Workspace environment variables can resolve per-environment server tool URLs, headers, and auth connections, and runtime system variables such as `{{system__conversation_history}}` can pass full conversation context into tool calls when needed.
```python
"prompt": {
"prompt": "You are a helpful assistant that can check the weather.",
"llm": "gemini-2.0-flash",
"tools": [
# Webhook: server-side API call
{"type": "webhook", "name": "get_weather", "description": "Get weather",
"api_schema": {"url": "https://api.example.com/weather", "method": "POST",
"request_body_schema": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}},
# Client: runs in the browser
{"type": "client", "name": "show_product", "description": "Display a product",
"parameters": {"type": "object", "properties": {"productId": {"type": "string"}}, "required": ["productId"]}}
],
"built_in_tools": {
"end_call": {},
"transfer_to_number": {"transfers": [{"transfer_destination": {"type": "phone", "phone_number": "+1234567890"}, "condition": "User asks for human support"}]}
}
}
```
**Client tools** run in browser:
```javascript
clientTools: {
show_product: async ({ productId }) => {
document.getElementById("product").src = `/products/${productId}`;
return { success: true };
}
}
```
See [Client Tools Reference](references/client-tools.md) for complete documentation.
## Widget Embedding
```html
<elevenlabs-convai agent-id="your-agent-id"></elevenlabs-convai>
<script src="https://unpkg.com/@elevenlabs/convai-widget-embed" async type="text/javascript"></script>
```
Customize with attributes: `avatar-image-url`, `action-text`, `start-call-text`, `end-call-text`.
See [Widget Embedding Reference](references/widget-embedding.md) for all options.
## Outbound Calls
Make outbound phone calls using your agent via Twilio integration:
### Python
```python
response = client.conversational_ai.twilio.outbound_call(
agent_id="your-agent-id",
agent_phone_number_id="your-phone-number-id",
to_number="+1234567890",
call_recording_enabled=True
)
print(f"Call initiated: {response.conversation_id}")
```
### JavaScript
```javascript
const response = await client.conversationalAi.twilio.outboundCall({
agentId: "your-agent-id",
agentPhoneNumberId: "your-phone-number-id",
toNumber: "+1234567890",
callRecordingEnabled: true,
});
```
### cURL
```bash
curl -X POST "https://api.elevenlabs.io/v1/convai/twilio/outbound-call" \
-H "xi-api-key: $ELEVENLABS_API_KEY" -H "Content-Type: application/json" \
-d '{"agent_id": "your-agent-id", "agent_phone_number_id": "your-phone-number-id", "to_number": "+1234567890", "call_recording_enabled": true}'
```
See [Outbound Calls Reference](references/outbound-calls.md) for configuration overrides and dynamic variables.
## Managing Agents
### Using CLI (Recommended)
```bash
# List agents and check status
elevenlabs agents list
elevenlabs agents status
# Import agents from platform to local config
elevenlabs agents pull # Import all agents
elevenlabs agents pull --agent <agent-id> # Import specific agent
# Push local changes to platform
elevenlabs agents push # Upload configurations
elevenlabs agents push --dry-run # Preview changes first
# Add tools
elevenlabs tools add-webhook "Weather API"
elevenlabs tools add-client "UI Tool"
```
### Project Structure
The CLI creates a project structure for managing agents:
```
your_project/
├── agents.json # Agent definitions
├── tools.json # Tool configurations
├── tests.json # Test configurations
├── agent_configs/ # Individual agent configs
├── tool_configs/ # Individual tool configs
└── test_configs/ # Individual test configs
```
### SDK Examples
```python
# List
agents = client.conversational_ai.agents.list()
# Get
agent = client.conversational_ai.agents.get(agent_id="your-agent-id")
# Update (partial - only include fields to change)
client.conversational_ai.agents.update(agent_id="your-agent-id", name="New Name")
client.conversational_ai.agents.update(agent_id="your-agent-id",
conversation_config={
"agent": {"prompt": {"prompt": "New instructions", "llm": "claude-sonnet-4"}}
})
# Delete
client.conversational_ai.agents.delete(agent_id="your-agent-id")
```
See [Agent Configuration](references/agent-configuration.md) for all configuration options and SDK examples.
## Error Handling
```python
try:
agent = client.conversational_ai.agents.create(...)
except Exception as e:
print(f"API error: {e}")
```
Common errors: **401** (invalid key), **404** (not found), **422** (invalid config), **429** (rate limit)
## References
- [Installation Guide](references/installation.md) - SDK setup and migration
- [Agent Configuration](references/agent-configuration.md) - All config options and CRUD examples
- [Client Tools](references/client-tools.md) - Webhook, client, and system tools
- [Widget Embedding](references/widget-embedding.md) - Website integration
- [Outbound Calls](references/outbound-calls.md) - Twilio phone call integration
.claude/skills/agents/references/agent-configuration.md
+606
View File
@@ -0,0 +1,606 @@
# Agent Configuration
Complete reference for configuring conversational AI agents.
## Configuration Structure
```python
agent = client.conversational_ai.agents.create(
name="My Agent",
conversation_config={
"agent": {
"first_message": "Hello!",
"language": "en",
"prompt": { # LLM, system prompt, tools, and knowledge base
"prompt": "You are helpful.",
"llm": "gemini-2.0-flash",
"tools": [...],
"built_in_tools": {...}
}
},
"tts": {...}, # Voice and TTS model settings
"asr": {...}, # Speech recognition settings
"turn": {...}, # Turn-taking behavior
"conversation": {...}, # Duration, events, monitoring
"vad": {...}, # Voice activity detection config
"language_presets": {...} # Language-specific overrides
},
platform_settings={...} # Auth, call limits
)
```
## conversation_config
Controls the real-time conversation behavior.
### agent
```python
conversation_config={
"agent": {
"first_message": "Hello! How can I help you today?",
"language": "en",
"disable_first_message_interruptions": False,
"prompt": {
"prompt": "You are a helpful assistant.",
"llm": "gemini-2.0-flash",
"temperature": 0.7
}
}
}
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `first_message` | string | `""` | What the agent says when conversation starts |
| `language` | string | `"en"` | ISO 639-1 language code (en, es, fr, etc.) |
| `disable_first_message_interruptions` | bool | `false` | Prevent user from interrupting the first message |
| `hinglish_mode` | bool | `false` | When enabled and language is Hindi, agent responds in Hinglish |
| `dynamic_variables` | object | - | Config with `dynamic_variable_placeholders` containing key-value pairs |
| `prompt` | object | - | LLM configuration (see prompt section below) |
### tts (Text-to-Speech)
```python
conversation_config={
"tts": {
"voice_id": "JBFqnCBsd6RMkjVDRZzb",
"model_id": "eleven_flash_v2_5",
"stability": 0.5,
"similarity_boost": 0.8,
"speed": 1.0,
"optimize_streaming_latency": 3,
"expressive_mode": True
}
}
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `voice_id` | string | `"cjVigY5qzO86Huf0OWal"` | Voice to use |
| `model_id` | string | - | TTS model (see below) |
| `stability` | float | `0.5` | 0-1, lower = more expressive |
| `similarity_boost` | float | `0.8` | 0-1, higher = closer to original voice |
| `speed` | float | `1.0` | 0.7-1.2, speech speed multiplier |
| `optimize_streaming_latency` | int | - | 0-4, higher = faster but lower quality |
| `expressive_mode` | bool | `true` | Enable expressive voice generation |
| `agent_output_audio_format` | string | - | Output audio codec format |
| `pronunciation_dictionary_locators` | array | - | Pronunciation overrides |
**Available TTS models for agents:**
| Model ID | Languages | Latency |
|----------|-----------|---------|
| `eleven_flash_v2_5` | 32 | ~75ms (recommended) |
| `eleven_flash_v2` | English | ~75ms |
| `eleven_turbo_v2_5` | 32 | ~250-300ms |
| `eleven_turbo_v2` | English | ~250-300ms |
| `eleven_multilingual_v2` | 29 | Standard |
| `eleven_v3_conversational` | 70+ | Standard |
### asr (Automatic Speech Recognition)
```python
conversation_config={
"asr": {
"quality": "high",
"keywords": ["ElevenLabs", "TechCorp"],
"user_input_audio_format": "pcm_16000"
}
}
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `quality` | string | `"high"` | Transcription quality level |
| `provider` | string | `"elevenlabs"` | ASR provider (`elevenlabs` or `scribe_realtime`) |
| `keywords` | array | - | Words to boost recognition accuracy |
| `user_input_audio_format` | string | - | Input audio format (e.g., `pcm_16000`, `ulaw_8000`) |
### turn (Turn-Taking)
```python
conversation_config={
"turn": {
"turn_timeout": 7,
"turn_eagerness": "normal",
"silence_end_call_timeout": -1
}
}
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `turn_timeout` | number | `7` | Seconds to wait before re-engaging the user |
| `turn_eagerness` | string | `"normal"` | How quickly agent responds: `patient`, `normal`, or `eager` |
| `silence_end_call_timeout` | number | `-1` | Seconds of silence before ending call (-1 = disabled) |
| `initial_wait_time` | number | - | Seconds to wait for user to start speaking |
| `spelling_patience` | string | `"auto"` | Entity detection patience: `auto` or `off` |
| `speculative_turn` | bool | `false` | Enable speculative turn detection |
| `soft_timeout_config` | object | - | Configures a message if user is silent (see below) |
**soft_timeout_config:**
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `timeout_seconds` | number | `-1` | Seconds before soft timeout (-1 = disabled) |
| `message` | string | `"Hhmmmm...yeah."` | What agent says on timeout |
| `use_llm_generated_message` | bool | `false` | Let LLM generate the timeout message |
## prompt (nested in conversation_config.agent)
Configures the LLM behavior. This object lives at `conversation_config.agent.prompt`:
```python
conversation_config={
"agent": {
"prompt": {
"prompt": "You are a helpful customer service agent...",
"llm": "gemini-2.0-flash",
"temperature": 0.7,
"max_tokens": 500,
"tools": [...],
"built_in_tools": {...},
"knowledge_base": [...]
}
}
}
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `prompt` | string | `""` | System prompt defining agent behavior |
| `llm` | string | - | Model ID (see LLM providers below) |
| `temperature` | float | `0` | 0-1, higher = more creative |
| `max_tokens` | int | `-1` | Max tokens for LLM response (-1 = unlimited) |
| `reasoning_effort` | string | - | Reasoning depth: `none`, `minimal`, `low`, `medium`, `high` (model-dependent) |
| `thinking_budget` | int | - | Max thinking tokens for reasoning models |
| `tools` | array | - | Webhook and client tool definitions |
| `built_in_tools` | object | - | System tools (end_call, transfer, etc.) |
| `tool_ids` | array | - | References to pre-configured tools |
| `knowledge_base` | array | - | Documents for RAG |
| `custom_llm` | object | - | Custom LLM endpoint config |
| `timezone` | string | - | IANA timezone (e.g., `America/New_York`) |
| `backup_llm_config` | object | - | Fallback LLM configuration |
| `cascade_timeout_seconds` | number | `8` | Seconds before cascading to backup LLM (2-15) |
| `mcp_server_ids` | array | - | MCP server IDs to connect |
| `native_mcp_server_ids` | array | - | Native MCP server IDs |
| `ignore_default_personality` | bool | - | Skip default personality instructions |
Workspace environment variables let one agent configuration span multiple deployments. Use
`{{system_env__label}}` in server tool and MCP server URLs, `{ "env_var_label": "orders_api_key" }`
for secret-backed tool headers, and `{ "env_var_label": "orders_oauth" }` in `auth_connection`
to resolve per-environment auth connections at runtime.
### LLM Providers
| Provider | Model IDs |
|----------|-----------|
| OpenAI | `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.1-nano`, `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo` |
| Anthropic | `claude-sonnet-4-6`, `claude-sonnet-4-5`, `claude-sonnet-4`, `claude-haiku-4-5`, `claude-3-7-sonnet`, `claude-3-5-sonnet`, `claude-3-haiku` |
| Google | `gemini-3.1-flash-lite-preview`, `gemini-3-pro-preview`, `gemini-3-flash-preview`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`, `gemini-2.0-flash`, `gemini-2.0-flash-lite` |
| ElevenLabs | `glm-45-air-fp8`, `qwen3-30b-a3b`, `gpt-oss-120b` (hosted, ultra-low latency) |
| Custom | `custom-llm` (requires custom_llm config) |
Use `GET /v1/convai/llm/list` to inspect the current model catalog, including deprecation state, token/context limits, and capability flags such as image-input support.
### Custom LLM
The `custom_llm` field is nested inside `conversation_config.agent.prompt`:
```python
conversation_config={
"agent": {
"prompt": {
"prompt": "You are helpful.",
"llm": "custom-llm",
"custom_llm": {
"url": "https://your-llm-endpoint.com/v1/chat/completions",
"model_id": "your-model-id",
"api_key": {"secret_id": "your-secret-id"},
"api_type": "chat_completions" # or "responses"
}
}
}
}
```
## platform_settings
Platform-level configuration for security, limits, summaries, and widget behavior.
```python
platform_settings={
"summary_language": "en",
"widget": {
"show_agent_status": True,
"show_conversation_id": True
},
"auth": {
"enable_auth": True,
"allowlist": [{"hostname": "example.com"}]
},
"call_limits": {
"agent_concurrency_limit": 10,
"daily_limit": 100
}
}
```
### Top-Level Fields
| Field | Type | Description |
|-------|------|-------------|
| `summary_language` | string | Language for conversation analysis outputs such as summaries, titles, evaluation rationales, and data collection rationales. If omitted, ElevenLabs infers it from the conversation. |
| `widget` | object | Hosted widget and shareable page configuration. See the widget table below for selected options. |
| `auth` | object | Authentication and origin restrictions for agent access |
| `call_limits` | object | Concurrency and daily usage limits |
| `guardrails` | object | Built-in safety and policy controls for agent interactions |
| `privacy` | object | Recording, retention, and conversation history redaction settings |
### auth
| Field | Type | Description |
|-------|------|-------------|
| `enable_auth` | bool | Require signed URLs/tokens for connections |
| `allowlist` | array | Allowed origins for CORS |
| `shareable_token` | string | Public conversation token |
### call_limits
| Field | Type | Description |
|-------|------|-------------|
| `agent_concurrency_limit` | int | Max simultaneous conversations (default: -1, unlimited) |
| `daily_limit` | int | Max conversations per day (default: 100000) |
| `bursting_enabled` | bool | Allow exceeding limits at 2x cost (default: true) |
### guardrails
Use `platform_settings.guardrails` to configure built-in safety controls for user input and agent behavior. The fields below cover the current schema additions that are most relevant in agent configs.
| Field | Type | Description |
|-------|------|-------------|
| `version` | string | Guardrail config version. Use `"1"` for the current schema. |
| `focus` | object | Keeps the agent on-topic and aligned with the configured task. |
| `prompt_injection` | object | Detects prompt injection and instruction override attempts. |
| `custom` | object | Configures user-defined response validation guardrails. |
| `content` | object | Configures category-specific content moderation guardrails. |
**focus / prompt_injection:**
| Field | Type | Description |
|-------|------|-------------|
| `is_enabled` | bool | Enables the guardrail. |
**content:**
| Field | Type | Description |
|-------|------|-------------|
| `execution_mode` | string | Guardrail execution mode: `streaming` or `blocking`. |
| `config` | object | Category threshold settings for content moderation. |
**content.config:**
| Field | Type | Description |
|-------|------|-------------|
| `sexual` | object | Threshold settings for sexual content. |
| `violence` | object | Threshold settings for violent content. |
| `harassment` | object | Threshold settings for harassment. |
| `self_harm` | object | Threshold settings for self-harm content. |
| `profanity` | object | Threshold settings for profanity. |
| `religion_or_politics` | object | Threshold settings for religion or politics content. |
| `medical_and_legal_information` | object | Threshold settings for medical or legal information. |
**content.config.\<category\>:**
| Field | Type | Description |
|-------|------|-------------|
| `is_enabled` | bool | Enables moderation for the category. |
| `threshold` | number or string | Category threshold as a numeric score or one of `low`, `medium`, or `high`. |
Blocking content guardrails and custom guardrails support a `trigger_action` that either ends
the session immediately or retries the response. Retry removes the blocked reply, injects your
feedback as a system message, and re-generates up to 3 times before the platform falls back to
ending the session. Feedback templates can use `{{trigger_reason}}` and `{{agent_message}}`.
### privacy
Use `platform_settings.privacy` to control recording, retention, and redaction behavior. The redaction-specific field is:
| Field | Type | Description |
|-------|------|-------------|
| `conversation_history_redaction` | object | Redacts configured entity types from stored transcripts, audio, and analysis. |
**conversation_history_redaction:**
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | bool | `false` | Whether conversation history redaction is enabled |
| `entities` | array | - | Entity types to redact. Use parent types such as `name` or specific values such as `name.name_given`, `email_address`, `contact_number`, `dob`, and `age`. |
### widget
Use `platform_settings.widget` to configure the hosted widget and shareable page defaults. For client-side embed attributes, see the widget embedding reference.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `dismissible` | bool | `false` | Whether the widget can be dismissed by the user |
| `show_agent_status` | bool | `false` | Whether to show working, done, or error status while tools are running |
| `show_conversation_id` | bool | `true` | Whether to show the conversation ID after disconnection |
| `strip_audio_tags` | bool | `true` | Whether to strip audio markup from messages |
| `syntax_highlight_theme` | string | auto | Code block syntax highlighting theme (`light` or `dark`); omit it to let the widget auto-detect |
### conversation (inside conversation_config)
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `max_duration_seconds` | int | `600` | Max conversation duration |
| `text_only` | bool | `false` | Text-only mode (avoids audio pricing) |
| `monitoring_enabled` | bool | `false` | Enable real-time WebSocket monitoring |
## Additional Top-Level Fields
| Field | Type | Description |
|-------|------|-------------|
| `tags` | array | Classification labels for filtering (e.g., `["production"]`, `["test"]`) |
| `workflow` | object | Conversation flow definition and tool interaction sequences |
## Knowledge Base / RAG
Knowledge base is configured inside `conversation_config.agent.prompt`:
```python
agent = client.conversational_ai.agents.create(
name="Support Agent",
conversation_config={
"agent": {
"prompt": {
"prompt": "You are a support agent. Use the knowledge base to answer questions.",
"llm": "gemini-2.0-flash",
"knowledge_base": [
{"type": "file", "id": "doc-id", "name": "Product Guide", "usage_mode": "auto"}
],
"rag": {
"enabled": True,
"embedding_model": "qwen3_embedding_4b",
"max_documents_length": 50000,
"max_retrieved_rag_chunks_count": 20
}
}
},
"tts": {"voice_id": "JBFqnCBsd6RMkjVDRZzb"}
}
)
```
`rag.embedding_model` supports `e5_mistral_7b_instruct`, `multilingual_e5_large_instruct`, and `qwen3_embedding_4b`.
## CRUD Operations
### Using CLI (Recommended)
```bash
# Initialize project
elevenlabs agents init
# Create agent from template
elevenlabs agents add "My Agent" --template complete
elevenlabs agents add "Support Bot" --template customer-service
# List agents
elevenlabs agents list
# Check status
elevenlabs agents status
# Push local changes to platform
elevenlabs agents push
elevenlabs agents push --dry-run # Preview changes first
# Import agents from platform
elevenlabs agents pull # Import all
elevenlabs agents pull --agent <agent-id> # Import specific agent
elevenlabs agents pull --update # Override local configs
# View available templates
elevenlabs agents templates list
elevenlabs agents templates show <template-name>
# Add tools
elevenlabs tools add-webhook "API Tool"
elevenlabs tools add-client "UI Tool"
# Generate widget code
elevenlabs agents widget <agent-id>
```
### SDK: List Agents
```python
agents = client.conversational_ai.agents.list()
for agent in agents.agents:
print(f"{agent.name}: {agent.agent_id}")
```
```javascript
const agents = await client.conversationalAi.agents.list();
```
```bash
curl -X GET "https://api.elevenlabs.io/v1/convai/agents" -H "xi-api-key: $ELEVENLABS_API_KEY"
```
### SDK: Get Agent
```python
agent = client.conversational_ai.agents.get(agent_id="your-agent-id")
```
```javascript
const agent = await client.conversationalAi.agents.get("your-agent-id");
```
```bash
curl -X GET "https://api.elevenlabs.io/v1/convai/agents/your-agent-id" -H "xi-api-key: $ELEVENLABS_API_KEY"
```
### SDK: Update Agent
Only include fields you want to change. All other settings remain unchanged.
**Python:**
```python
# Update name
client.conversational_ai.agents.update(agent_id="id", name="New Name")
# Update TTS voice
client.conversational_ai.agents.update(agent_id="id", conversation_config={
"tts": {"voice_id": "EXAVITQu4vr4xnSDxMaL", "model_id": "eleven_flash_v2_5"}
})
# Update prompt/LLM (nested in agent)
client.conversational_ai.agents.update(agent_id="id", conversation_config={
"agent": {"prompt": {"prompt": "New instructions.", "llm": "claude-sonnet-4", "temperature": 0.8}}
})
# Update first message
client.conversational_ai.agents.update(agent_id="id", conversation_config={
"agent": {"first_message": "Welcome back!"}
})
# Update platform settings
client.conversational_ai.agents.update(agent_id="id", platform_settings={
"auth": {"enable_auth": True, "allowlist": [{"hostname": "myapp.com"}]}
})
```
**JavaScript:**
```javascript
await client.conversationalAi.agents.update("id", { name: "New Name" });
await client.conversationalAi.agents.update("id", {
conversationConfig: { tts: { voiceId: "EXAVITQu4vr4xnSDxMaL" } }
});
await client.conversationalAi.agents.update("id", {
conversationConfig: { agent: { prompt: { prompt: "New instructions.", llm: "claude-sonnet-4" } } }
});
```
**cURL:**
```bash
curl -X PATCH "https://api.elevenlabs.io/v1/convai/agents/your-agent-id" \
-H "xi-api-key: $ELEVENLABS_API_KEY" -H "Content-Type: application/json" \
-d '{"name": "New Name"}'
```
#### Updatable Fields
| Section | Fields |
|---------|--------|
| Root | `name`, `tags` |
| `conversation_config.agent` | `first_message`, `language`, `disable_first_message_interruptions`, `dynamic_variables` |
| `conversation_config.agent.prompt` | `prompt`, `llm`, `temperature`, `max_tokens`, `reasoning_effort`, `tools`, `built_in_tools`, `knowledge_base`, `custom_llm`, `timezone` |
| `conversation_config.tts` | `voice_id`, `model_id`, `stability`, `similarity_boost`, `speed`, `optimize_streaming_latency`, `expressive_mode` |
| `conversation_config.asr` | `quality`, `provider`, `keywords`, `user_input_audio_format` |
| `conversation_config.turn` | `turn_timeout`, `turn_eagerness`, `silence_end_call_timeout`, `soft_timeout_config` |
| `conversation_config.conversation` | `max_duration_seconds`, `text_only`, `monitoring_enabled` |
| `platform_settings` | `summary_language`, `guardrails`, `privacy` |
| `platform_settings.widget` | `dismissible`, `show_agent_status`, `show_conversation_id`, `strip_audio_tags`, `syntax_highlight_theme` |
| `platform_settings.auth` | `enable_auth`, `allowlist` |
| `platform_settings.call_limits` | `agent_concurrency_limit`, `daily_limit`, `bursting_enabled` |
### SDK: Delete Agent
```python
client.conversational_ai.agents.delete(agent_id="your-agent-id")
```
```javascript
await client.conversationalAi.agents.delete("your-agent-id");
```
```bash
curl -X DELETE "https://api.elevenlabs.io/v1/convai/agents/your-agent-id" -H "xi-api-key: $ELEVENLABS_API_KEY"
```
## CI/CD Integration
Use the CLI in your deployment pipeline:
```bash
# Set API key as environment variable
export ELEVENLABS_API_KEY="your-api-key"
# Push changes (non-interactive)
elevenlabs agents push
```
## Example Configurations
### Customer Support Agent
```python
agent = client.conversational_ai.agents.create(
name="Support Agent",
conversation_config={
"agent": {
"first_message": "Hi! Thanks for calling TechCorp support.",
"language": "en",
"prompt": {
"prompt": "You are a customer support agent. Be helpful, professional, concise.",
"llm": "gemini-2.0-flash",
"temperature": 0.5,
"built_in_tools": {
"end_call": {},
"transfer_to_number": {
"transfers": [{"transfer_destination": {"type": "phone", "phone_number": "+1234567890"}, "condition": "User asks for human support"}]
}
}
}
},
"tts": {"voice_id": "XB0fDUnXU5powFXDhCwa", "model_id": "eleven_flash_v2_5"},
"turn": {"turn_eagerness": "normal", "turn_timeout": 7},
"conversation": {"max_duration_seconds": 900}
}
)
```
### Low-Latency Assistant
```python
agent = client.conversational_ai.agents.create(
name="Quick Assistant",
conversation_config={
"agent": {
"first_message": "Hey! What do you need?",
"prompt": {
"prompt": "Fast, efficient assistant. Brief answers.",
"llm": "gemini-2.0-flash",
"temperature": 0.3,
"max_tokens": 100
}
},
"tts": {"voice_id": "JBFqnCBsd6RMkjVDRZzb", "model_id": "eleven_flash_v2_5", "optimize_streaming_latency": 4},
"turn": {"turn_eagerness": "eager", "turn_timeout": 3}
}
)
```
.claude/skills/agents/references/client-tools.md
+545
View File
@@ -0,0 +1,545 @@
# Client Tools
Extend your agent with custom capabilities. Tools let the agent take actions beyond just talking.
## Tool Types
| Type | Execution | Use Case |
|------|-----------|----------|
| **Webhook** | Server-side via HTTP | Database queries, API calls, secure operations |
| **Client** | Browser-side JavaScript | UI updates, local storage, navigation |
| **System** | Built-in ElevenLabs | End call, transfer, standard actions |
## Where Tools Live
Tools are defined inside `conversation_config.agent.prompt`. Webhook and client tools go in the `tools` array. System tools go in `built_in_tools`:
```python
conversation_config={
"agent": {
"prompt": {
"prompt": "You are helpful.",
"llm": "gemini-2.0-flash",
"tools": [...], # Webhook and client tools
"built_in_tools": {...} # System tools (end_call, transfer, etc.)
}
}
}
```
## Webhook Tools
Execute server-side logic when the agent needs external data or actions.
### Basic Webhook
```python
agent = client.conversational_ai.agents.create(
name="Weather Assistant",
conversation_config={
"agent": {
"prompt": {
"prompt": "You are a helpful assistant that can check the weather.",
"llm": "gemini-2.0-flash",
"tools": [{
"type": "webhook",
"name": "get_weather",
"description": "Get current weather for a city. Use when user asks about weather.",
"api_schema": {
"url": "https://api.example.com/weather",
"method": "POST",
"request_headers": {
"Authorization": "Bearer {{API_KEY}}"
},
"request_body_schema": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "City name, e.g., 'San Francisco'"
},
"units": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature units"
}
},
"required": ["city"]
}
}
}]
}
},
"tts": {"voice_id": "JBFqnCBsd6RMkjVDRZzb"}
}
)
```
### Webhook Request Format
When the agent calls a webhook tool, ElevenLabs sends:
```json
{
"tool_call_id": "call_abc123",
"tool_name": "get_weather",
"parameters": {
"city": "San Francisco",
"units": "fahrenheit"
},
"conversation_id": "conv_xyz789"
}
```
### Webhook Response Format
Your server should respond with:
```json
{
"result": "The weather in San Francisco is 68°F and sunny."
}
```
Or for structured data:
```json
{
"result": {
"temperature": 68,
"condition": "sunny",
"humidity": 45
}
}
```
### Webhook with Authentication
```python
# Inside conversation_config.agent.prompt.tools:
{
"type": "webhook",
"name": "lookup_order",
"description": "Look up order status by order ID",
"response_timeout_secs": 10,
"api_schema": {
"url": "https://api.mystore.com/orders/lookup",
"method": "POST",
"request_headers": {
"Authorization": "Bearer {{ORDER_API_KEY}}",
"X-Store-ID": "store_123"
},
"request_body_schema": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"description": "Order ID (e.g., ORD-12345)"
}
},
"required": ["order_id"]
}
}
}
```
Use workspace environment variables to keep a single server tool configuration working across
staging and production. `{{system_env__label}}` works in server tool URLs, secret environment
variables can populate `request_headers`, and auth-connection environment variables can populate
`api_schema.auth_connection`. The same environment-variable resolution model also applies to MCP
server connections.
```json
{
"api_schema": {
"url": "https://{{system_env__api_host}}.example.com/orders",
"method": "GET",
"request_headers": {
"X-Api-Key": { "env_var_label": "orders_api_key" }
},
"auth_connection": { "env_var_label": "orders_oauth" }
}
}
```
Workspace auth connections support OAuth2 client credentials, OAuth2 JWT, private key JWT,
basic auth, bearer auth, and custom header auth.
System dynamic variables are also available in tool parameters and headers. Use
`{{system__conversation_history}}` when a webhook or sub-agent needs the full conversation
context as a lazily evaluated JSON history object with user, agent, and tool entries.
### Webhook Tool Options
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `response_timeout_secs` | int | `20` | Timeout in seconds (5-120) |
| `disable_interruptions` | bool | `false` | Prevent user interruptions during tool execution |
| `execution_mode` | string | `"immediate"` | `immediate`, `post_tool_speech`, or `async` |
| `tool_call_sound` | string | - | Sound during execution: `typing`, `elevator1`-`elevator4` |
| `force_pre_tool_speech` | bool | `false` | Force agent to speak before executing tool |
| `tool_error_handling_mode` | string | `"auto"` | `auto`, `summarized`, `passthrough`, or `hide` |
**Note:** The default `api_schema.method` is `GET`. Always set `"method": "POST"` explicitly for webhook tools that send request bodies.
### Server Implementation (Node.js)
```javascript
app.post("/webhook/get_weather", async (req, res) => {
const { parameters, conversation_id } = req.body;
const { city, units = "fahrenheit" } = parameters;
// Fetch weather from your data source
const weather = await weatherService.get(city, units);
res.json({
result: `It's ${weather.temp}°${units === "celsius" ? "C" : "F"} and ${weather.condition} in ${city}.`,
});
});
```
### Server Implementation (Python)
```python
@app.post("/webhook/get_weather")
async def get_weather(request: Request):
data = await request.json()
city = data["parameters"]["city"]
units = data["parameters"].get("units", "fahrenheit")
# Fetch weather from your data source
weather = weather_service.get(city, units)
return {
"result": f"It's {weather['temp']}°{'C' if units == 'celsius' else 'F'} and {weather['condition']} in {city}."
}
```
## Client Tools
Execute JavaScript in the user's browser. Useful for UI updates, navigation, or accessing browser APIs.
### Defining Client Tools
Client tools are registered when starting a conversation:
```javascript
import { Conversation } from "@elevenlabs/client";
const conversation = await Conversation.startSession({
agentId: "your-agent-id",
clientTools: {
show_product: async ({ productId }) => {
// Update UI to show product
const modal = document.getElementById("product-modal");
modal.innerHTML = await fetchProductCard(productId);
modal.showModal();
return { success: true, message: "Showing product" };
},
navigate_to: async ({ page }) => {
// Navigate to a page
window.location.href = `/${page}`;
return { success: true };
},
save_preference: async ({ key, value }) => {
// Store in localStorage
localStorage.setItem(key, value);
return { saved: true };
},
},
});
```
### Registering Client Tools with Agent
Tell the agent about available client tools in `conversation_config.agent.prompt.tools`:
```python
agent = client.conversational_ai.agents.create(
name="Shopping Assistant",
conversation_config={
"agent": {
"prompt": {
"prompt": """You are a shopping assistant.
When users want to see a product, use show_product.
When users want to go somewhere, use navigate_to.""",
"llm": "gemini-2.0-flash",
"tools": [
{
"type": "client",
"name": "show_product",
"description": "Display a product card to the user",
"parameters": {
"type": "object",
"properties": {
"productId": {
"type": "string",
"description": "Product ID to display"
}
},
"required": ["productId"]
}
},
{
"type": "client",
"name": "navigate_to",
"description": "Navigate user to a different page",
"parameters": {
"type": "object",
"properties": {
"page": {
"type": "string",
"enum": ["cart", "checkout", "account", "home"],
"description": "Page to navigate to"
}
},
"required": ["page"]
}
}
]
}
},
"tts": {"voice_id": "JBFqnCBsd6RMkjVDRZzb"}
}
)
```
### Client Tool Options
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `expects_response` | bool | `false` | Whether the tool returns data to the agent |
### Client Tool Return Values
Return data that the agent can use in conversation:
```javascript
clientTools: {
check_cart: async () => {
const cart = JSON.parse(localStorage.getItem("cart") || "[]");
return {
itemCount: cart.length,
total: cart.reduce((sum, item) => sum + item.price, 0),
items: cart.map((item) => item.name),
};
};
}
```
The agent receives this data and can say: "You have 3 items in your cart totaling $45.99."
## System Tools (built_in_tools)
Built-in tools provided by ElevenLabs. These are configured in `conversation_config.agent.prompt.built_in_tools` (not in the `tools` array):
```python
"built_in_tools": {
"end_call": {},
"transfer_to_number": {...},
"transfer_to_agent": {...},
"language_detection": {},
"skip_turn": {},
"voicemail_detection": {...},
"play_keypad_touch_tone": {}
}
```
Current API schemas also expose `agent_prompt_change`, `memory_entry_create`, `memory_entry_delete`, `memory_entry_search`, and `memory_entry_update` in `built_in_tools`.
### end_call
Ends the current conversation:
```python
"built_in_tools": {
"end_call": {}
}
```
The agent can say "Goodbye!" and then end the call programmatically.
### transfer_to_number
Transfer to a phone number (requires telephony integration):
```python
"built_in_tools": {
"transfer_to_number": {
"transfers": [{
"transfer_destination": {"type": "phone", "phone_number": "+1234567890"},
"condition": "User asks to speak with a human agent"
}]
}
}
```
### transfer_to_agent
Transfer to another ElevenLabs agent:
```python
"built_in_tools": {
"transfer_to_agent": {
"transfers": [{
"agent_id": "other-agent-id",
"condition": "User asks about sales"
}]
}
}
```
## Best Practices
### Tool Descriptions
Write clear descriptions so the LLM knows when to use tools:
```python
# Good - specific and actionable
"description": "Look up order status. Use when customer asks about their order, delivery, or shipping."
# Bad - vague
"description": "Order tool"
```
### Parameter Descriptions
Help the LLM extract correct values:
```python
"parameters": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"description": "Order ID in format ORD-XXXXX (e.g., ORD-12345)"
},
"email": {
"type": "string",
"description": "Customer email address for verification"
}
}
}
```
### Error Handling
Configure how tool errors are shared with the agent using `tool_error_handling_mode`:
| Mode | Behavior |
|------|----------|
| `auto` | ElevenLabs automatically decides how to handle errors |
| `summarized` | Errors are summarized before being sent to the agent |
| `passthrough` | Full error details are passed to the agent |
| `hide` | Errors are hidden from the agent |
Return helpful error messages:
```javascript
// Server webhook
app.post("/webhook/lookup_order", async (req, res) => {
const { order_id } = req.body.parameters;
const order = await db.orders.find(order_id);
if (!order) {
return res.json({
result: {
error: true,
message: `Order ${order_id} not found. Please verify the order ID.`,
},
});
}
res.json({ result: order });
});
```
### Timeouts
Set reasonable timeouts for webhooks using `response_timeout_secs` (5-120 seconds, default 20):
```python
{
"type": "webhook",
"name": "slow_operation",
"description": "Run a slow operation",
"response_timeout_secs": 30,
"api_schema": {
"url": "https://api.example.com/slow-operation",
"method": "POST"
}
}
```
## Complete Example
```python
agent = client.conversational_ai.agents.create(
name="E-commerce Assistant",
conversation_config={
"agent": {
"first_message": "Hi! How can I help you today?",
"language": "en",
"prompt": {
"prompt": """You are an e-commerce support assistant.
Available actions:
- lookup_order: Check order status
- show_product: Display products to customer
- end_call: End conversation politely
- transfer_to_number: Transfer to human support
Always verify order ID before lookup. Offer transfer for complex issues.""",
"llm": "gemini-2.0-flash",
"tools": [
# Webhook: Server-side order lookup
{
"type": "webhook",
"name": "lookup_order",
"description": "Look up order status by order ID or email",
"api_schema": {
"url": "https://api.mystore.com/orders/lookup",
"method": "POST",
"request_headers": {"Authorization": "Bearer {{API_KEY}}"},
"request_body_schema": {
"type": "object",
"properties": {
"order_id": {"type": "string"},
"email": {"type": "string"}
}
}
}
},
# Client: Browser-side product display
{
"type": "client",
"name": "show_product",
"description": "Display product details to the customer",
"parameters": {
"type": "object",
"properties": {
"product_id": {"type": "string"}
},
"required": ["product_id"]
}
}
],
"built_in_tools": {
"end_call": {},
"transfer_to_number": {
"transfers": [{
"transfer_destination": {"type": "phone", "phone_number": "+1234567890"},
"condition": "User asks for human support"
}]
}
}
}
},
"tts": {"voice_id": "JBFqnCBsd6RMkjVDRZzb", "model_id": "eleven_flash_v2_5"}
}
)
```
.claude/skills/agents/references/installation.md
+131
View File
@@ -0,0 +1,131 @@
# Installation
## CLI (Recommended)
The ElevenLabs CLI is the recommended way to create and manage agents:
```bash
npm install -g @elevenlabs/cli
# or
pnpm add -g @elevenlabs/cli
# or
yarn global add @elevenlabs/cli
```
Requires Node.js 16.0.0 or higher.
### Authentication
```bash
elevenlabs auth login # Authenticate with API key
elevenlabs auth whoami # Verify current login status
elevenlabs auth logout # Remove stored credentials
```
API keys are securely stored in `~/.agents/api_keys.json`.
### Quick Start
```bash
# Initialize a new project
elevenlabs agents init
# Create an agent from template
elevenlabs agents add "My Assistant" --template complete
# Push to ElevenLabs platform
elevenlabs agents push
```
## JavaScript / TypeScript SDK
For programmatic access and client-side integration:
```bash
npm install @elevenlabs/elevenlabs-js
```
> **Important:** Always use `@elevenlabs/elevenlabs-js`. The old `elevenlabs` npm package (v1.x) is deprecated and should not be used.
```javascript
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
// Option 1: Environment variable (recommended)
// Set ELEVENLABS_API_KEY in your environment
const client = new ElevenLabsClient();
// Option 2: Pass directly
const client = new ElevenLabsClient({ apiKey: "your-api-key" });
```
### Migrating from deprecated packages
If you have old packages installed, remove them:
```bash
# Remove deprecated packages
npm uninstall elevenlabs
# Install the current packages
npm install @elevenlabs/elevenlabs-js
# For client-side/browser usage, also install:
npm install @elevenlabs/client # Browser client
npm install @elevenlabs/react # React hooks
```
**Import changes:**
```javascript
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
import { Conversation } from "@elevenlabs/client";
import { useConversation } from "@elevenlabs/react";
```
## Python
```bash
pip install elevenlabs
```
```python
from elevenlabs import ElevenLabs
# Option 1: Environment variable (recommended)
# Set ELEVENLABS_API_KEY in your environment
client = ElevenLabs()
# Option 2: Pass directly
client = ElevenLabs(api_key="your-api-key")
```
## cURL / REST API
Set your API key as an environment variable:
```bash
export ELEVENLABS_API_KEY="your-api-key"
```
Include in requests via the `xi-api-key` header:
```bash
curl -X POST "https://api.elevenlabs.io/v1/convai/agents/create" \
-H "xi-api-key: $ELEVENLABS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "My Agent", "conversation_config": {"agent": {"prompt": {"prompt": "You are helpful.", "llm": "gemini-2.0-flash"}}, "tts": {"voice_id": "JBFqnCBsd6RMkjVDRZzb"}}}'
```
## Getting an API Key
1. Sign up at [elevenlabs.io](https://elevenlabs.io)
2. Go to [API Keys](https://elevenlabs.io/app/settings/api-keys)
3. Click **Create API Key**
4. Copy and store securely
Or use the `setup-api-key` skill for guided setup.
## Environment Variables
| Variable | Description |
|----------|-------------|
| `ELEVENLABS_API_KEY` | Your ElevenLabs API key (required) |
.claude/skills/agents/references/outbound-calls.md
+170
View File
@@ -0,0 +1,170 @@
# Outbound Calls
Make outbound phone calls using your ElevenLabs agent via Twilio integration.
## Prerequisites
1. A configured ElevenLabs agent
2. A Twilio phone number linked to your agent (obtain `agent_phone_number_id` from ElevenLabs dashboard)
3. Your ElevenLabs API key
## Basic Usage
See the [main agents skill](../SKILL.md#outbound-calls) for basic Python, JavaScript, and cURL examples.
## Request Parameters
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `agent_id` | string | Yes | The ID of your ElevenLabs agent |
| `agent_phone_number_id` | string | Yes | The ID of the Twilio phone number linked to your agent |
| `to_number` | string | Yes | The destination phone number (E.164 format) |
| `conversation_initiation_client_data` | object | No | Override conversation settings for this call |
| `call_recording_enabled` | boolean | No | Whether to let Twilio record the call |
| `telephony_call_config` | object | No | Telephony call settings like ringing timeout |
## Response
```json
{
"success": true,
"message": "Call initiated successfully",
"conversation_id": "conv_abc123",
"callSid": "CA1234567890abcdef"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | Whether the call was initiated successfully |
| `message` | string | Status message |
| `conversation_id` | string | ElevenLabs conversation ID for tracking |
| `callSid` | string | Twilio Call SID for reference |
## Customizing the Call
Override agent settings for a specific call using `conversation_initiation_client_data`:
### Python
```python
response = client.conversational_ai.twilio.outbound_call(
agent_id="your-agent-id",
agent_phone_number_id="your-phone-number-id",
to_number="+1234567890",
call_recording_enabled=True,
conversation_initiation_client_data={
"conversation_config_override": {
"agent": {
"first_message": "Hello! This is a reminder about your appointment tomorrow.",
"language": "en"
},
"tts": {
"voice_id": "JBFqnCBsd6RMkjVDRZzb"
}
},
"dynamic_variables": {
"customer_name": "John",
"appointment_time": "2:00 PM"
}
}
)
```
### JavaScript
```javascript
const response = await client.conversationalAi.twilio.outboundCall({
agentId: "your-agent-id",
agentPhoneNumberId: "your-phone-number-id",
toNumber: "+1234567890",
callRecordingEnabled: true,
conversationInitiationClientData: {
conversationConfigOverride: {
agent: {
firstMessage: "Hello! This is a reminder about your appointment tomorrow.",
language: "en",
},
tts: {
voiceId: "JBFqnCBsd6RMkjVDRZzb",
},
},
dynamicVariables: {
customer_name: "John",
appointment_time: "2:00 PM",
},
},
});
```
## Configuration Overrides
### Agent Settings
| Option | Type | Description |
|--------|------|-------------|
| `first_message` | string | Custom greeting for this call |
| `language` | string | Language code (e.g., "en", "es", "fr") |
| `prompt` | object | Override agent prompt and LLM settings |
### TTS Settings
| Option | Type | Description |
|--------|------|-------------|
| `voice_id` | string | Voice ID to use for this call |
| `stability` | number | Voice stability (0.0-1.0) |
| `similarity_boost` | number | Voice similarity boost (0.0-1.0) |
| `speed` | number | Speech speed multiplier |
### Telephony Call Configuration
| Option | Type | Description |
|--------|------|-------------|
| `ringing_timeout_secs` | integer | How long to ring the recipient before giving up (default: `60`) |
### Dynamic Variables
Pass custom data to your agent's prompt using `dynamic_variables`. Reference them in your agent's prompt with `{{variable_name}}` syntax.
When assigning dynamic variables, you can use the `sanitize` option to remove sensitive values from tool responses before they are sent to the LLM and transcript, while still allowing variable assignment:
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `sanitize` | boolean | `false` | If true, the assignment's value is removed from tool responses before sending to LLM/transcript but still processed for variable assignment |
## Complete Example
```python
from elevenlabs import ElevenLabs
client = ElevenLabs()
# Make personalized outbound calls
customers = [
{"name": "Alice", "phone": "+1234567890", "balance": "$150.00"},
{"name": "Bob", "phone": "+0987654321", "balance": "$75.50"},
]
for customer in customers:
try:
response = client.conversational_ai.twilio.outbound_call(
agent_id="payment-reminder-agent",
agent_phone_number_id="your-phone-number-id",
to_number=customer["phone"],
call_recording_enabled=True,
conversation_initiation_client_data={
"conversation_config_override": {
"agent": {
"first_message": f"Hello {customer['name']}, this is a friendly reminder about your account."
}
},
"dynamic_variables": {
"customer_name": customer["name"],
"balance": customer["balance"]
}
}
)
print(f"Called {customer['name']}: {response.conversation_id}")
except Exception as e:
print(f"Failed to call {customer['name']}: {e}")
```
.claude/skills/agents/references/widget-embedding.md
+365
View File
@@ -0,0 +1,365 @@
# Widget Embedding
Add a voice AI agent to any website with the ElevenLabs conversation widget.
## Basic Embed
```html
<elevenlabs-convai agent-id="your-agent-id"></elevenlabs-convai>
<script src="https://unpkg.com/@elevenlabs/convai-widget-embed" async type="text/javascript"></script>
```
This creates a floating button that users can click to start a voice conversation.
> **Note:** Widgets currently require public agents with authentication disabled. For authenticated flows, use the SDKs.
## Widget Attributes
### Required
| Attribute | Description |
|-----------|-------------|
| `agent-id` | Your ElevenLabs agent ID |
| `signed-url` | Alternative to `agent-id` when using signed URLs |
### Appearance
| Attribute | Description | Default |
|-----------|-------------|---------|
| `avatar-image-url` | URL for agent avatar image | ElevenLabs logo |
| `avatar-orb-color-1` | Primary orb gradient color | `#2792dc` |
| `avatar-orb-color-2` | Secondary orb gradient color | `#9ce6e6` |
### Text Labels
| Attribute | Description | Default |
|-----------|-------------|---------|
| `action-text` | Tooltip when hovering | "Talk to AI" |
| `start-call-text` | Button to start call | "Start call" |
| `end-call-text` | Button to end call | "End call" |
| `expand-text` | Expand chat button | "Open" |
| `collapse-text` | Collapse chat button | "Close" |
| `listening-text` | Listening state label | "Listening..." |
| `speaking-text` | Speaking state label | "Assistant speaking" |
### Behavior
| Attribute | Description | Default |
|-----------|-------------|---------|
| `variant` | Widget style: `compact` or `expanded` | `compact` |
| `server-location` | Server region (`us`, `eu-residency`, `in-residency`, `global`) | `us` |
| `dismissible` | Allow the user to minimize the widget | `false` |
| `disable-banner` | Hide "Powered by ElevenLabs" | `false` |
## Examples
### Custom Avatar
```html
<elevenlabs-convai
agent-id="your-agent-id"
avatar-image-url="https://example.com/your-avatar.png"
></elevenlabs-convai>
```
### Custom Colors
```html
<elevenlabs-convai
agent-id="your-agent-id"
avatar-orb-color-1="#ff6b6b"
avatar-orb-color-2="#ffd93d"
></elevenlabs-convai>
```
### Custom Text
```html
<elevenlabs-convai
agent-id="your-agent-id"
action-text="Chat with our AI assistant"
start-call-text="Begin conversation"
end-call-text="Hang up"
></elevenlabs-convai>
```
### Expanded Variant
```html
<elevenlabs-convai
agent-id="your-agent-id"
variant="expanded"
></elevenlabs-convai>
```
### Full Customization
```html
<elevenlabs-convai
agent-id="your-agent-id"
avatar-image-url="https://example.com/support-agent.png"
avatar-orb-color-1="#4f46e5"
avatar-orb-color-2="#818cf8"
action-text="Talk to Support"
start-call-text="Start voice chat"
end-call-text="End conversation"
expand-text="Open assistant"
collapse-text="Minimize"
></elevenlabs-convai>
```
## CSS Customization
The widget uses Shadow DOM but exposes CSS custom properties:
```css
elevenlabs-convai {
--elevenlabs-convai-widget-width: 400px;
--elevenlabs-convai-widget-height: 600px;
}
```
### Positioning
By default, the widget appears in the bottom-right corner. Override with CSS:
```css
elevenlabs-convai {
position: fixed;
bottom: 20px;
right: 20px;
/* Or position differently */
left: 20px;
right: auto;
}
```
### Z-Index
```css
elevenlabs-convai {
z-index: 9999;
}
```
## JavaScript Control
Access the widget element to control it programmatically:
```html
<elevenlabs-convai id="my-widget" agent-id="your-agent-id"></elevenlabs-convai>
<script>
const widget = document.getElementById("my-widget");
// Start a conversation
widget.startConversation();
// End the conversation
widget.endConversation();
// Listen for events
widget.addEventListener("conversationStarted", () => {
console.log("Conversation started");
});
widget.addEventListener("conversationEnded", () => {
console.log("Conversation ended");
});
</script>
```
### Custom Trigger Button
Hide the default widget and use your own button:
```html
<style>
elevenlabs-convai {
display: none;
}
</style>
<button onclick="document.getElementById('widget').startConversation()">
Talk to AI
</button>
<elevenlabs-convai id="widget" agent-id="your-agent-id"></elevenlabs-convai>
```
## Authentication
For agents with authentication enabled, pass a signed URL:
```html
<elevenlabs-convai id="widget" agent-id="your-agent-id"></elevenlabs-convai>
<script>
async function startAuthenticatedConversation() {
// Get signed URL from your backend
const response = await fetch("/api/get-signed-url");
const { signedUrl } = await response.json();
const widget = document.getElementById("widget");
widget.setAttribute("signed-url", signedUrl);
widget.startConversation();
}
</script>
```
Your backend:
```python
@app.get("/api/get-signed-url")
def get_signed_url():
signed_url = client.conversational_ai.conversations.get_signed_url(
agent_id="your-agent-id"
)
return {"signedUrl": signed_url.signed_url}
```
## Mobile Considerations
### Responsive Positioning
```css
/* Desktop: bottom-right */
elevenlabs-convai {
position: fixed;
bottom: 20px;
right: 20px;
}
/* Mobile: full-width bottom */
@media (max-width: 768px) {
elevenlabs-convai {
bottom: 0;
right: 0;
left: 0;
--elevenlabs-convai-widget-width: 100%;
}
}
```
### Touch-Friendly
The widget is touch-optimized by default. For better mobile UX:
```css
@media (max-width: 768px) {
elevenlabs-convai {
/* Larger touch target */
transform: scale(1.1);
transform-origin: bottom right;
}
}
```
## Multiple Widgets
You can have multiple widgets for different agents:
```html
<elevenlabs-convai
agent-id="support-agent-id"
action-text="Support"
style="right: 20px"
></elevenlabs-convai>
<elevenlabs-convai
agent-id="sales-agent-id"
action-text="Sales"
style="right: 100px"
></elevenlabs-convai>
```
## Framework Integration
### React
```jsx
function App() {
useEffect(() => {
// Load widget script
const script = document.createElement("script");
script.src = "https://unpkg.com/@elevenlabs/convai-widget-embed";
script.async = true;
document.body.appendChild(script);
return () => document.body.removeChild(script);
}, []);
return (
<div>
<elevenlabs-convai agent-id="your-agent-id"></elevenlabs-convai>
</div>
);
}
```
### Vue
```vue
<template>
<div>
<elevenlabs-convai agent-id="your-agent-id"></elevenlabs-convai>
</div>
</template>
<script setup>
import { onMounted } from "vue";
onMounted(() => {
const script = document.createElement("script");
script.src = "https://unpkg.com/@elevenlabs/convai-widget-embed";
script.async = true;
document.body.appendChild(script);
});
</script>
```
### Next.js
```jsx
import Script from "next/script";
export default function Page() {
return (
<>
<Script
src="https://unpkg.com/@elevenlabs/convai-widget-embed"
strategy="lazyOnload"
/>
<elevenlabs-convai agent-id="your-agent-id"></elevenlabs-convai>
</>
);
}
```
## Troubleshooting
### Widget Not Appearing
1. Check that the agent ID is correct
2. Verify the script is loaded (check Network tab)
3. Check for JavaScript errors in console
4. Ensure no CSS is hiding the widget
### Audio Issues
1. Ensure HTTPS (microphone requires secure context)
2. Check browser permissions for microphone
3. Test in a supported browser (Chrome, Firefox, Safari, Edge)
### CORS Errors
If using authentication, ensure your domain is in the agent's allowlist:
```python
platform_settings={
"auth": {
"enable_auth": True,
"allowlist": ["https://yourdomain.com"]
}
}
```
.claude/skills/ai-video-gen/SKILL.md
+308
View File
@@ -0,0 +1,308 @@
---
name: ai-video-gen
description: |
Generate AI videos from text prompts using the HeyGen API. Use when: (1) Generating videos from text descriptions, (2) Creating AI-generated video clips for content production, (3) Image-to-video generation with a reference image, (4) Choosing between video generation providers (VEO, Kling, Sora, Runway, Seedance), (5) Working with HeyGen's /v1/workflows/executions endpoint for video generation.
allowed-tools: mcp__heygen__*
metadata:
openclaw:
requires:
env:
- HEYGEN_API_KEY
primaryEnv: HEYGEN_API_KEY
---
# Video Generation (HeyGen API)
Generate AI videos from text prompts. Supports multiple providers (VEO 3.1, Kling, Sora, Runway, Seedance), configurable aspect ratios, and optional reference images for image-to-video generation.
## Authentication
All requests require the `X-Api-Key` header. Set the `HEYGEN_API_KEY` environment variable.
```bash
curl -X POST "https://api.heygen.com/v1/workflows/executions" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{"workflow_type": "GenerateVideoNode", "input": {"prompt": "A drone shot flying over a coastal city at sunset"}}'
```
## Default Workflow
1. Call `POST /v1/workflows/executions` with `workflow_type: "GenerateVideoNode"` and your prompt
2. Receive a `execution_id` in the response
3. Poll `GET /v1/workflows/executions/{id}` every 10 seconds until status is `completed`
4. Use the returned `video_url` from the output
## Execute Video Generation
### Endpoint
`POST https://api.heygen.com/v1/workflows/executions`
### Request Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `workflow_type` | string | Y | Must be `"GenerateVideoNode"` |
| `input.prompt` | string | Y | Text description of the video to generate |
| `input.provider` | string | | Video generation provider (default: `"veo_3_1"`). See Providers below. |
| `input.aspect_ratio` | string | | Aspect ratio (default: `"16:9"`). Common values: `"16:9"`, `"9:16"`, `"1:1"` |
| `input.reference_image_url` | string | | Reference image URL for image-to-video generation |
| `input.tail_image_url` | string | | Tail image URL for last-frame guidance |
| `input.config` | object | | Provider-specific configuration overrides |
### Providers
| Provider | Value | Description |
|----------|-------|-------------|
| VEO 3.1 | `"veo_3_1"` | Google VEO 3.1 (default, highest quality) |
| VEO 3.1 Fast | `"veo_3_1_fast"` | Faster VEO 3.1 variant |
| VEO 3 | `"veo3"` | Google VEO 3 |
| VEO 3 Fast | `"veo3_fast"` | Faster VEO 3 variant |
| VEO 2 | `"veo2"` | Google VEO 2 |
| Kling Pro | `"kling_pro"` | Kling Pro model |
| Kling V2 | `"kling_v2"` | Kling V2 model |
| Sora V2 | `"sora_v2"` | OpenAI Sora V2 |
| Sora V2 Pro | `"sora_v2_pro"` | OpenAI Sora V2 Pro |
| Runway Gen-4 | `"runway_gen4"` | Runway Gen-4 |
| Seedance Lite | `"seedance_lite"` | Seedance Lite |
| Seedance Pro | `"seedance_pro"` | Seedance Pro |
| LTX Distilled | `"ltx_distilled"` | LTX Distilled (fastest) |
### curl
```bash
curl -X POST "https://api.heygen.com/v1/workflows/executions" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"workflow_type": "GenerateVideoNode",
"input": {
"prompt": "A drone shot flying over a coastal city at golden hour, cinematic lighting",
"provider": "veo_3_1",
"aspect_ratio": "16:9"
}
}'
```
### TypeScript
```typescript
interface GenerateVideoInput {
prompt: string;
provider?: string;
aspect_ratio?: string;
reference_image_url?: string;
tail_image_url?: string;
config?: Record<string, any>;
}
interface ExecuteResponse {
data: {
execution_id: string;
status: "submitted";
};
}
async function generateVideo(input: GenerateVideoInput): Promise<string> {
const response = await fetch("https://api.heygen.com/v1/workflows/executions", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
workflow_type: "GenerateVideoNode",
input,
}),
});
const json: ExecuteResponse = await response.json();
return json.data.execution_id;
}
```
### Python
```python
import requests
import os
def generate_video(
prompt: str,
provider: str = "veo_3_1",
aspect_ratio: str = "16:9",
reference_image_url: str | None = None,
tail_image_url: str | None = None,
) -> str:
payload = {
"workflow_type": "GenerateVideoNode",
"input": {
"prompt": prompt,
"provider": provider,
"aspect_ratio": aspect_ratio,
},
}
if reference_image_url:
payload["input"]["reference_image_url"] = reference_image_url
if tail_image_url:
payload["input"]["tail_image_url"] = tail_image_url
response = requests.post(
"https://api.heygen.com/v1/workflows/executions",
headers={
"X-Api-Key": os.environ["HEYGEN_API_KEY"],
"Content-Type": "application/json",
},
json=payload,
)
data = response.json()
return data["data"]["execution_id"]
```
### Response Format
```json
{
"data": {
"execution_id": "node-gw-v1d2e3o4",
"status": "submitted"
}
}
```
## Check Status
### Endpoint
`GET https://api.heygen.com/v1/workflows/executions/{execution_id}`
### curl
```bash
curl -X GET "https://api.heygen.com/v1/workflows/executions/node-gw-v1d2e3o4" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### Response Format (Completed)
```json
{
"data": {
"execution_id": "node-gw-v1d2e3o4",
"status": "completed",
"output": {
"video": {
"video_url": "https://resource.heygen.ai/generated/video.mp4",
"video_id": "abc123"
},
"asset_id": "asset-xyz789"
}
}
}
```
## Polling for Completion
```typescript
async function generateVideoAndWait(
input: GenerateVideoInput,
maxWaitMs = 600000,
pollIntervalMs = 10000
): Promise<{ video_url: string; video_id: string; asset_id: string }> {
const executionId = await generateVideo(input);
console.log(`Submitted video generation: ${executionId}`);
const startTime = Date.now();
while (Date.now() - startTime < maxWaitMs) {
const response = await fetch(
`https://api.heygen.com/v1/workflows/executions/${executionId}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
switch (data.status) {
case "completed":
return {
video_url: data.output.video.video_url,
video_id: data.output.video.video_id,
asset_id: data.output.asset_id,
};
case "failed":
throw new Error(data.error?.message || "Video generation failed");
case "not_found":
throw new Error("Workflow not found");
default:
await new Promise((r) => setTimeout(r, pollIntervalMs));
}
}
throw new Error("Video generation timed out");
}
```
## Usage Examples
### Simple Text-to-Video
```bash
curl -X POST "https://api.heygen.com/v1/workflows/executions" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"workflow_type": "GenerateVideoNode",
"input": {
"prompt": "A person walking through a sunlit park, shallow depth of field"
}
}'
```
### Image-to-Video
```json
{
"workflow_type": "GenerateVideoNode",
"input": {
"prompt": "Animate this product photo with a slow zoom and soft particle effects",
"reference_image_url": "https://example.com/product-photo.png",
"provider": "kling_pro"
}
}
```
### Vertical Format for Social Media
```json
{
"workflow_type": "GenerateVideoNode",
"input": {
"prompt": "A trendy coffee shop interior, camera slowly panning across the counter",
"aspect_ratio": "9:16",
"provider": "veo_3_1"
}
}
```
### Fast Generation with LTX
```json
{
"workflow_type": "GenerateVideoNode",
"input": {
"prompt": "Abstract colorful shapes morphing and flowing",
"provider": "ltx_distilled"
}
}
```
## Best Practices
1. **Be descriptive in prompts** — include camera movement, lighting, style, and mood details
2. **Default to VEO 3.1** for highest quality; use `ltx_distilled` or `veo3_fast` when speed matters
3. **Use reference images** for image-to-video generation — great for animating product photos or still images
4. **Video generation is the slowest workflow** — allow up to 5 minutes, poll every 10 seconds
5. **Aspect ratio matters** — use `9:16` for social media stories/reels, `16:9` for landscape, `1:1` for square
6. **Output includes `asset_id`** — use this to reference the generated video in other HeyGen workflows
7. **Output URLs are temporary** — download or save generated videos promptly
.claude/skills/avatar-video/SKILL.md
+117
View File
@@ -0,0 +1,117 @@
---
name: avatar-video
description: |
Create AI avatar videos with precise control over avatars, voices, scripts, scenes, and backgrounds using HeyGen's v2 API. Use when: (1) Choosing a specific avatar and voice for a video, (2) Writing exact scripts for an avatar to speak, (3) Building multi-scene videos with different backgrounds per scene, (4) Creating transparent WebM videos for compositing, (5) Using talking photos as video presenters, (6) Integrating HeyGen avatars with Remotion, (7) Batch video generation with exact specs, (8) Brand-consistent production videos with precise control.
homepage: https://docs.heygen.com/reference/create-a-video
allowed-tools: mcp__heygen__*
metadata:
openclaw:
requires:
env:
- HEYGEN_API_KEY
primaryEnv: HEYGEN_API_KEY
---
# Avatar Video
Create AI avatar videos with full control over avatars, voices, scripts, scenes, and backgrounds. Build single or multi-scene videos with exact configuration using HeyGen's `/v2/video/generate` API.
## Authentication
All requests require the `X-Api-Key` header. Set the `HEYGEN_API_KEY` environment variable.
```bash
curl -X GET "https://api.heygen.com/v2/avatars" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
## Tool Selection
If HeyGen MCP tools are available (`mcp__heygen__*`), **prefer them** over direct HTTP API calls — they handle authentication and request formatting automatically.
| Task | MCP Tool | Fallback (Direct API) |
|------|----------|----------------------|
| Check video status / get URL | `mcp__heygen__get_video` | `GET /v2/videos/{video_id}` |
| List account videos | `mcp__heygen__list_videos` | `GET /v2/videos` |
| Delete a video | `mcp__heygen__delete_video` | `DELETE /v2/videos/{video_id}` |
Video generation (`POST /v2/video/generate`) and avatar/voice listing are done via direct API calls — see reference files below.
## Default Workflow
1. **List avatars** — `GET /v2/avatars` → pick an avatar, preview it, note `avatar_id` and `default_voice_id`. See [avatars.md](references/avatars.md)
2. **List voices** (if needed) — `GET /v2/voices` → pick a voice matching the avatar's gender/language. See [voices.md](references/voices.md)
3. **Write the script** — Structure scenes with one concept each. See [scripts.md](references/scripts.md)
4. **Generate the video** — `POST /v2/video/generate` with avatar, voice, script, and background per scene. See [video-generation.md](references/video-generation.md)
5. **Poll for completion** — `GET /v2/videos/{video_id}` until status is `completed`. See [video-status.md](references/video-status.md)
## Quick Reference
| Task | Read |
|------|------|
| List and preview avatars | [avatars.md](references/avatars.md) |
| List and select voices | [voices.md](references/voices.md) |
| Write and structure scripts | [scripts.md](references/scripts.md) |
| Generate video (single or multi-scene) | [video-generation.md](references/video-generation.md) |
| Add custom backgrounds | [backgrounds.md](references/backgrounds.md) |
| Add captions / subtitles | [captions.md](references/captions.md) |
| Add text overlays | [text-overlays.md](references/text-overlays.md) |
| Create transparent WebM video | [video-generation.md](references/video-generation.md) (WebM section) |
| Use templates | [templates.md](references/templates.md) |
| Create avatar from photo | [photo-avatars.md](references/photo-avatars.md) |
| Check video status / download | [video-status.md](references/video-status.md) |
| Upload assets (images, audio) | [assets.md](references/assets.md) |
| Use with Remotion | [remotion-integration.md](references/remotion-integration.md) |
| Set up webhooks | [webhooks.md](references/webhooks.md) |
## When to Use This Skill vs Create Video
This skill is for **precise control** — you choose the avatar, write the exact script, configure each scene.
If the user just wants to **describe a video idea** and let AI handle the rest (script, avatar, visuals), use the **create-video** skill instead.
| User Says | Create Video Skill | This Skill |
|-----------|:------------------:|:----------:|
| "Make me a video about X" | ✓ | |
| "Create a product demo" | ✓ | |
| "I want avatar Y to say exactly Z" | | ✓ |
| "Multi-scene video with different backgrounds" | | ✓ |
| "Transparent WebM for compositing" | | ✓ |
| "Use this specific voice for my script" | | ✓ |
| "Batch generate videos with exact specs" | | ✓ |
## Reference Files
### Core Video Creation
- [references/avatars.md](references/avatars.md) - Listing avatars, styles, avatar_id selection
- [references/voices.md](references/voices.md) - Listing voices, locales, speed/pitch
- [references/scripts.md](references/scripts.md) - Writing scripts, pauses, pacing
- [references/video-generation.md](references/video-generation.md) - POST /v2/video/generate and multi-scene videos
### Video Customization
- [references/backgrounds.md](references/backgrounds.md) - Solid colors, images, video backgrounds
- [references/text-overlays.md](references/text-overlays.md) - Adding text with fonts and positioning
- [references/captions.md](references/captions.md) - Auto-generated captions and subtitles
### Advanced Features
- [references/templates.md](references/templates.md) - Template listing and variable replacement
- [references/photo-avatars.md](references/photo-avatars.md) - Creating avatars from photos
- [references/webhooks.md](references/webhooks.md) - Webhook endpoints and events
### Integration
- [references/remotion-integration.md](references/remotion-integration.md) - Using HeyGen in Remotion compositions
### Foundation
- [references/video-status.md](references/video-status.md) - Polling patterns and download URLs
- [references/assets.md](references/assets.md) - Uploading images, videos, audio
- [references/dimensions.md](references/dimensions.md) - Resolution and aspect ratios
- [references/quota.md](references/quota.md) - Credit system and usage limits
## Best Practices
1. **Preview avatars before generating** — Download `preview_image_url` so the user can see the avatar before committing
2. **Use avatar's default voice** — Most avatars have a `default_voice_id` pre-matched for natural results
3. **Fallback: match gender manually** — If no default voice, ensure avatar and voice genders match
4. **Use test mode for development** — Set `test: true` to avoid consuming credits (output will be watermarked)
5. **Set generous timeouts** — Video generation often takes 5-15 minutes, sometimes longer
6. **Validate inputs** — Check avatar and voice IDs exist before generating
.claude/skills/avatar-video/references/assets.md
+339
View File
@@ -0,0 +1,339 @@
---
name: assets
description: Uploading images, videos, and audio for use in HeyGen video generation
---
# Asset Upload and Management
HeyGen allows you to upload custom assets (images, videos, audio) for use in video generation, such as backgrounds, talking photo sources, and custom audio.
## Upload Flow
Asset uploads are a single-step process: POST the raw file binary directly to the upload endpoint. The Content-Type header must match the file's MIME type.
## Uploading an Asset
**Endpoint:** `POST https://upload.heygen.com/v1/asset`
### Request
| Header | Required | Description |
|--------|:--------:|-------------|
| `X-Api-Key` | ✓ | Your HeyGen API key |
| `Content-Type` | ✓ | MIME type of the file (e.g. `image/jpeg`) |
The request body is the raw binary file data. No JSON or form fields are needed.
### Response
| Field | Type | Description |
|-------|------|-------------|
| `code` | number | Status code (`100` = success) |
| `data.id` | string | Unique asset ID for use in video generation |
| `data.name` | string | Asset name |
| `data.file_type` | string | `image`, `video`, or `audio` |
| `data.url` | string | Accessible URL for the uploaded file |
| `data.image_key` | string \| null | Key for creating uploaded photo avatars (images only) |
| `data.folder_id` | string | Folder ID (empty if not in a folder) |
| `data.meta` | string \| null | Asset metadata |
| `data.created_ts` | number | Unix timestamp of creation |
### curl
```bash
curl -X POST "https://upload.heygen.com/v1/asset" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: image/jpeg" \
--data-binary '@./background.jpg'
```
### TypeScript
```typescript
import fs from "fs";
import path from "path";
interface AssetUploadResponse {
code: number;
data: {
id: string;
name: string;
file_type: string;
url: string;
image_key: string | null;
folder_id: string;
meta: string | null;
created_ts: number;
};
msg: string | null;
message: string | null;
}
async function uploadAsset(filePath: string, contentType: string): Promise<AssetUploadResponse["data"]> {
const resolvedPath = path.resolve(filePath);
const fileBuffer = fs.readFileSync(resolvedPath);
const response = await fetch("https://upload.heygen.com/v1/asset", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": contentType,
},
body: fileBuffer,
});
const json: AssetUploadResponse = await response.json();
if (json.code !== 100) {
throw new Error(json.message ?? "Upload failed");
}
return json.data;
}
// Usage
const asset = await uploadAsset("./background.jpg", "image/jpeg");
console.log(`Uploaded asset: ${asset.id}`);
console.log(`Asset URL: ${asset.url}`);
```
### TypeScript (with streams for large files)
```typescript
import fs from "fs";
import path from "path";
import { stat } from "fs/promises";
async function uploadLargeAsset(filePath: string, contentType: string): Promise<AssetUploadResponse["data"]> {
const resolvedPath = path.resolve(filePath);
const fileStats = await stat(resolvedPath);
const fileStream = fs.createReadStream(resolvedPath);
const response = await fetch("https://upload.heygen.com/v1/asset", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": contentType,
"Content-Length": fileStats.size.toString(),
},
body: fileStream as any,
// @ts-ignore - duplex is needed for streaming
duplex: "half",
});
const json: AssetUploadResponse = await response.json();
if (json.code !== 100) {
throw new Error(json.message ?? "Upload failed");
}
return json.data;
}
```
### Python
```python
import requests
import os
def upload_asset(file_path: str, content_type: str) -> dict:
with open(file_path, "rb") as f:
response = requests.post(
"https://upload.heygen.com/v1/asset",
headers={
"X-Api-Key": os.environ["HEYGEN_API_KEY"],
"Content-Type": content_type
},
data=f
)
data = response.json()
if data.get("code") != 100:
raise Exception(data.get("message", "Upload failed"))
return data["data"]
# Usage
asset = upload_asset("./background.jpg", "image/jpeg")
print(f"Uploaded asset: {asset['id']}")
print(f"Asset URL: {asset['url']}")
```
## Supported Content Types
| Type | Content-Type | Use Case |
|------|--------------|----------|
| JPEG | `image/jpeg` | Backgrounds, talking photos |
| PNG | `image/png` | Backgrounds, overlays |
| MP4 | `video/mp4` | Video backgrounds |
| WebM | `video/webm` | Video backgrounds |
| MP3 | `audio/mpeg` | Custom audio input |
| WAV | `audio/wav` | Custom audio input |
## Uploading from URL
If your asset is already hosted online:
```typescript
async function uploadFromUrl(sourceUrl: string, contentType: string): Promise<AssetUploadResponse["data"]> {
// 1. Validate and download the file
const url = new URL(sourceUrl);
if (url.protocol !== "https:") {
throw new Error("Only HTTPS URLs are supported");
}
const sourceResponse = await fetch(sourceUrl);
const buffer = Buffer.from(await sourceResponse.arrayBuffer());
// 2. Upload directly to HeyGen
const response = await fetch("https://upload.heygen.com/v1/asset", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": contentType,
},
body: buffer,
});
const json: AssetUploadResponse = await response.json();
if (json.code !== 100) {
throw new Error(json.message ?? "Upload failed");
}
return json.data;
}
```
## Using Uploaded Assets
### As Background Image
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello, this is a video with a custom background!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: asset.url, // Use the URL from the upload response
},
},
],
};
```
### As Talking Photo Source
```typescript
const talkingPhotoConfig = {
video_inputs: [
{
character: {
type: "talking_photo",
talking_photo_id: asset.id, // Use the ID from the upload response
},
voice: {
type: "text",
input_text: "Hello from my talking photo!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
};
```
### As Audio Input
```typescript
const audioConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "audio",
audio_url: asset.url, // Use the URL from the upload response
},
},
],
};
```
## Complete Upload Workflow
```typescript
async function createVideoWithCustomBackground(
backgroundPath: string,
script: string
): Promise<string> {
// 1. Upload background
console.log("Uploading background...");
const background = await uploadAsset(backgroundPath, "image/jpeg");
// 2. Create video config
const config = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: background.url,
},
},
],
dimension: { width: 1920, height: 1080 },
};
// 3. Generate video
console.log("Generating video...");
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
});
const { data } = await response.json();
return data.video_id;
}
```
## Asset Limitations
- **File size**: 10MB maximum
- **Image dimensions**: Recommended to match video dimensions
- **Audio duration**: Should match expected video length
- **Retention**: Assets may be deleted after a period of inactivity
## Best Practices
1. **Optimize images** - Resize to match video dimensions before uploading
2. **Use appropriate formats** - JPEG for photos, PNG for graphics with transparency
3. **Validate before upload** - Check file type and size locally first
4. **Handle upload errors** - Implement retry logic for failed uploads
5. **Cache asset IDs** - Reuse assets across multiple video generations
.claude/skills/avatar-video/references/avatars.md
+586
View File
@@ -0,0 +1,586 @@
---
name: avatars
description: Listing avatars, avatar styles, and avatar_id selection for HeyGen
---
# HeyGen Avatars
Avatars are the AI-generated presenters in HeyGen videos. You can use public avatars provided by HeyGen or create custom avatars.
## Previewing Avatars Before Generation
Always preview avatars before generating a video to ensure they match user preferences. Each avatar has preview URLs that can be opened directly in the browser - no downloading required.
### List Avatars and Show Previews
```typescript
async function listAndPreviewAvatars(openInBrowser = true): Promise<void> {
const response = await fetch("https://api.heygen.com/v2/avatars", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const { data } = await response.json();
for (const avatar of data.avatars.slice(0, 5)) {
console.log(`\n${avatar.avatar_name} (${avatar.gender})`);
console.log(` ID: ${avatar.avatar_id}`);
console.log(` Preview: ${avatar.preview_image_url}`);
}
// Preview URLs can be opened directly in any browser
for (const avatar of data.avatars.slice(0, 3)) {
console.log(`Open in browser: ${avatar.preview_image_url}`);
}
}
```
### Workflow: Preview Before Generate
1. **List available avatars** - get names, genders, and preview URLs
2. **Show preview URLs to user** - share `preview_image_url` for visual check
3. **User selects** preferred avatar by name or ID
4. **Get avatar details** for `default_voice_id`
5. **Generate video** with selected avatar
### Preview Fields in API Response
| Field | Description |
|-------|-------------|
| `preview_image_url` | Static image of the avatar (JPG) - publicly accessible URL |
| `preview_video_url` | Short video clip showing avatar animation |
Both URLs are publicly accessible - no authentication needed to view.
## Listing Available Avatars
### curl
```bash
curl -X GET "https://api.heygen.com/v2/avatars" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface Avatar {
avatar_id: string;
avatar_name: string;
gender: "male" | "female";
preview_image_url: string;
preview_video_url: string;
}
interface AvatarsResponse {
error: null | string;
data: {
avatars: Avatar[];
talking_photos: TalkingPhoto[];
};
}
async function listAvatars(): Promise<Avatar[]> {
const response = await fetch("https://api.heygen.com/v2/avatars", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const json: AvatarsResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.avatars;
}
```
### Python
```python
import requests
import os
def list_avatars() -> list:
response = requests.get(
"https://api.heygen.com/v2/avatars",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]["avatars"]
```
## Response Format
```json
{
"error": null,
"data": {
"avatars": [
{
"avatar_id": "josh_lite3_20230714",
"avatar_name": "Josh",
"gender": "male",
"preview_image_url": "https://files.heygen.ai/...",
"preview_video_url": "https://files.heygen.ai/..."
},
{
"avatar_id": "angela_expressive_20231010",
"avatar_name": "Angela",
"gender": "female",
"preview_image_url": "https://files.heygen.ai/...",
"preview_video_url": "https://files.heygen.ai/..."
}
],
"talking_photos": []
}
}
```
## Avatar Types
### Public Avatars
HeyGen provides a library of public avatars that anyone can use:
```typescript
// List only public avatars
const avatars = await listAvatars();
const publicAvatars = avatars.filter((a) => !a.avatar_id.startsWith("custom_"));
```
### Private/Custom Avatars
Custom avatars created from your own training footage:
```typescript
const customAvatars = avatars.filter((a) => a.avatar_id.startsWith("custom_"));
```
## Avatar Styles
Avatars support different rendering styles:
| Style | Description |
|-------|-------------|
| `normal` | Full body shot, standard framing |
| `closeUp` | Close-up on face, more expressive |
| `circle` | Avatar in circular frame (talking head) |
| `voice_only` | Audio only, no video rendering |
### When to Use Each Style
| Use Case | Recommended Style |
|----------|-------------------|
| Full-screen presenter video | `normal` |
| Personal/intimate content | `closeUp` |
| Picture-in-picture overlay | `circle` |
| Small corner widget | `circle` |
| Podcast/audio content | `voice_only` |
| Motion graphics with avatar overlay | `normal` or `closeUp` + transparent bg |
### Using Avatar Styles
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal", // "normal" | "closeUp" | "circle" | "voice_only"
},
voice: {
type: "text",
input_text: "Hello, world!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
};
```
### Circle Style for Talking Heads
Circle style is ideal for overlay compositions:
```typescript
// Circle avatar for picture-in-picture
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "circle",
},
voice: { ... },
background: {
type: "color",
value: "#00FF00", // Green for chroma key, or use webm endpoint
},
}
```
## Searching and Filtering Avatars
### By Gender
```typescript
function filterByGender(avatars: Avatar[], gender: "male" | "female"): Avatar[] {
return avatars.filter((a) => a.gender === gender);
}
const maleAvatars = filterByGender(avatars, "male");
const femaleAvatars = filterByGender(avatars, "female");
```
### By Name
```typescript
function searchByName(avatars: Avatar[], query: string): Avatar[] {
const lowerQuery = query.toLowerCase();
return avatars.filter((a) =>
a.avatar_name.toLowerCase().includes(lowerQuery)
);
}
const results = searchByName(avatars, "josh");
```
## Avatar Groups
Avatars are organized into groups for better management.
### List Avatar Groups
```bash
curl -X GET "https://api.heygen.com/v2/avatar_group.list?include_public=true" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
#### Query Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `include_public` | bool | false | Include public avatars in results |
#### TypeScript
```typescript
interface AvatarGroupItem {
id: string;
name: string;
created_at: number;
num_looks: number;
preview_image: string;
group_type: string;
train_status: string;
default_voice_id: string | null;
}
interface AvatarGroupListResponse {
error: null | string;
data: {
avatar_group_list: AvatarGroupItem[];
};
}
async function listAvatarGroups(
includePublic = true
): Promise<AvatarGroupListResponse["data"]> {
const params = new URLSearchParams({
include_public: includePublic.toString(),
});
const response = await fetch(
`https://api.heygen.com/v2/avatar_group.list?${params}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json: AvatarGroupListResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data;
}
```
### Get Avatars in a Group
```bash
curl -X GET "https://api.heygen.com/v2/avatar_group/{group_id}/avatars" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
## Using Avatars in Video Generation
### Basic Avatar Usage
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Welcome to our product demo!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
dimension: { width: 1920, height: 1080 },
};
```
### Multiple Scenes with Different Avatars
```typescript
const multiSceneConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hi, I'm Josh. Let me introduce my colleague.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
{
character: {
type: "avatar",
avatar_id: "angela_expressive_20231010",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello! I'm Angela. Nice to meet you!",
voice_id: "2d5b0e6a8c3f47d9a1b2c3d4e5f60718",
},
},
],
};
```
## Using Avatar's Default Voice
Many avatars have a `default_voice_id` that's pre-matched for natural results. **This is the recommended approach** rather than manually selecting voices.
### Recommended Flow
```
1. GET /v2/avatars → Get list of avatar_ids
2. GET /v2/avatar/{id}/details → Get default_voice_id for chosen avatar
3. POST /v2/video/generate → Use avatar_id + default_voice_id
```
### Get Avatar Details (v2 API)
Given an `avatar_id`, fetch its details including the default voice:
```bash
curl -X GET "https://api.heygen.com/v2/avatar/{avatar_id}/details" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
#### Response Format
```json
{
"error": null,
"data": {
"type": "avatar",
"id": "josh_lite3_20230714",
"name": "Josh",
"gender": "male",
"preview_image_url": "https://files.heygen.ai/...",
"preview_video_url": "https://files.heygen.ai/...",
"premium": false,
"is_public": true,
"default_voice_id": "1bd001e7e50f421d891986aad5158bc8",
"tags": ["AVATAR_IV"]
}
}
```
#### TypeScript
```typescript
interface AvatarDetails {
type: "avatar";
id: string;
name: string;
gender: "male" | "female";
preview_image_url: string;
preview_video_url: string;
premium: boolean;
is_public: boolean;
default_voice_id: string | null;
tags: string[];
}
async function getAvatarDetails(avatarId: string): Promise<AvatarDetails> {
const response = await fetch(
`https://api.heygen.com/v2/avatar/${avatarId}/details`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data;
}
// Usage: Get default voice for a known avatar
const details = await getAvatarDetails("josh_lite3_20230714");
if (details.default_voice_id) {
console.log(`Using ${details.name} with default voice: ${details.default_voice_id}`);
} else {
console.log(`${details.name} has no default voice, select manually`);
}
```
#### Complete Example: Generate Video with Any Avatar's Default Voice
```typescript
async function generateWithAvatarDefaultVoice(
avatarId: string,
script: string
): Promise<string> {
// 1. Get avatar details to find default voice
const avatar = await getAvatarDetails(avatarId);
if (!avatar.default_voice_id) {
throw new Error(`Avatar ${avatar.name} has no default voice`);
}
// 2. Generate video with the avatar's default voice
const videoId = await generateVideo({
video_inputs: [{
character: {
type: "avatar",
avatar_id: avatar.id,
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: avatar.default_voice_id,
},
}],
dimension: { width: 1920, height: 1080 },
});
return videoId;
}
```
### Why Use Default Voice?
1. **Guaranteed gender match** - Avatar and voice are pre-paired
2. **Natural lip sync** - Default voices are optimized for the avatar
3. **Simpler code** - No need to fetch and match voices separately
4. **Better quality** - HeyGen has tested this combination
## Selecting the Right Avatar
### Avatar Categories
HeyGen avatars fall into distinct categories. Match the category to your use case:
| Category | Examples | Best For |
|----------|----------|----------|
| **Business/Professional** | Josh, Angela, Wayne | Corporate videos, product demos, training |
| **Casual/Friendly** | Lily, various lifestyle avatars | Social media, informal content |
| **Themed/Seasonal** | Holiday-themed, costume avatars | Specific campaigns, seasonal content |
| **Expressive** | Avatars with "expressive" in name | Engaging storytelling, dynamic content |
### Selection Guidelines
**For business/professional content:**
- Choose avatars with neutral attire (business casual or formal)
- Avoid themed or seasonal avatars (holiday costumes, casual clothing)
- Preview the avatar to verify professional appearance
- Consider your audience demographics when selecting gender and appearance
**For casual/social content:**
- More flexibility in avatar choice
- Themed avatars can work for specific campaigns
- Match avatar energy to content tone
### Common Mistakes to Avoid
1. **Using themed avatars for business content** - A holiday-themed avatar looks unprofessional in a product demo
2. **Not previewing before generation** - Always check the preview URL to verify appearance
3. **Ignoring avatar style** - A `circle` style avatar may not work for full-screen presentations
4. **Mismatched voice gender** - Always use the avatar's `default_voice_id` or match genders manually
### Selection Checklist
Before generating a video:
- [ ] Previewed avatar image/video in browser
- [ ] Avatar appearance matches content tone (professional vs casual)
- [ ] Avatar style (`normal`, `closeUp`, `circle`) fits the video format
- [ ] Voice gender matches avatar gender
- [ ] Using `default_voice_id` when available
## Helper Functions
### Get Avatar by ID
```typescript
async function getAvatarById(avatarId: string): Promise<Avatar | null> {
const avatars = await listAvatars();
return avatars.find((a) => a.avatar_id === avatarId) || null;
}
```
### Validate Avatar ID
```typescript
async function isValidAvatarId(avatarId: string): Promise<boolean> {
const avatar = await getAvatarById(avatarId);
return avatar !== null;
}
```
### Get Random Avatar
```typescript
async function getRandomAvatar(gender?: "male" | "female"): Promise<Avatar> {
let avatars = await listAvatars();
if (gender) {
avatars = avatars.filter((a) => a.gender === gender);
}
const randomIndex = Math.floor(Math.random() * avatars.length);
return avatars[randomIndex];
}
```
## Common Avatar IDs
Some commonly used public avatar IDs (availability may vary):
| Avatar ID | Name | Gender |
|-----------|------|--------|
| `josh_lite3_20230714` | Josh | Male |
| `angela_expressive_20231010` | Angela | Female |
| `wayne_20240422` | Wayne | Male |
| `lily_20230614` | Lily | Female |
Always verify avatar availability by calling the list endpoint before using.
.claude/skills/avatar-video/references/backgrounds.md
+295
View File
@@ -0,0 +1,295 @@
---
name: backgrounds
description: Solid colors, images, and video backgrounds for HeyGen videos
---
# Video Backgrounds
HeyGen supports various background types to customize the appearance of your avatar videos.
## Background Types
| Type | Description |
|------|-------------|
| `color` | Solid color background |
| `image` | Static image background |
| `video` | Looping video background |
## Color Backgrounds
The simplest option - use a solid color:
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello with a colored background!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#FFFFFF", // White background
},
},
],
};
```
### Common Color Values
| Color | Hex Value | Use Case |
|-------|-----------|----------|
| White | `#FFFFFF` | Clean, professional |
| Black | `#000000` | Dramatic, cinematic |
| Blue | `#0066CC` | Corporate, trustworthy |
| Green | `#00FF00` | Chroma key (for compositing) |
| Gray | `#808080` | Neutral, modern |
### Using Transparent/Green Screen
For compositing in post-production:
```typescript
background: {
type: "color",
value: "#00FF00", // Green screen
}
```
## Image Backgrounds
Use a static image as background:
### From URL
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Check out this custom background!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: "https://example.com/my-background.jpg",
},
},
],
};
```
### From Uploaded Asset
First upload your image, then use the asset URL:
```typescript
// 1. Upload the image
const assetId = await uploadFile("./background.jpg", "image/jpeg");
// 2. Use in video config
const videoConfig = {
video_inputs: [
{
character: {...},
voice: {...},
background: {
type: "image",
url: `https://files.heygen.ai/asset/${assetId}`,
},
},
],
};
```
### Image Requirements
- **Formats**: JPEG, PNG
- **Recommended size**: Match video dimensions (e.g., 1920x1080 for 1080p)
- **Aspect ratio**: Should match video aspect ratio
- **File size**: Under 10MB recommended
## Video Backgrounds
Use a looping video as background:
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Dynamic video background!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "video",
url: "https://example.com/background-loop.mp4",
},
},
],
};
```
### Video Requirements
- **Format**: MP4 (H.264 codec recommended)
- **Looping**: Video will loop if shorter than avatar content
- **Audio**: Background video audio is typically muted
- **File size**: Under 100MB recommended
## Different Backgrounds Per Scene
Use different backgrounds for each scene:
```typescript
const multiBackgroundConfig = {
video_inputs: [
// Scene 1: Office background
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Let me start with an introduction.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: "https://example.com/office-bg.jpg",
},
},
// Scene 2: Product showcase
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "closeUp",
},
voice: {
type: "text",
input_text: "Now let me show you our product.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: "https://example.com/product-bg.jpg",
},
},
// Scene 3: Call to action
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Get started today!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#1a1a2e",
},
},
],
};
```
## Background Helper Functions
### TypeScript
```typescript
type BackgroundType = "color" | "image" | "video";
interface Background {
type: BackgroundType;
value?: string;
url?: string;
}
function createColorBackground(hexColor: string): Background {
return { type: "color", value: hexColor };
}
function createImageBackground(imageUrl: string): Background {
return { type: "image", url: imageUrl };
}
function createVideoBackground(videoUrl: string): Background {
return { type: "video", url: videoUrl };
}
// Preset backgrounds
const backgrounds = {
white: createColorBackground("#FFFFFF"),
black: createColorBackground("#000000"),
greenScreen: createColorBackground("#00FF00"),
corporate: createColorBackground("#0066CC"),
};
```
## Best Practices
1. **Match dimensions** - Background should match video dimensions
2. **Consider avatar position** - Leave space where avatar will appear
3. **Use contrasting colors** - Ensure avatar is visible against background
4. **Optimize file sizes** - Compress images/videos for faster processing
5. **Test with green screen** - For professional post-production workflows
6. **Keep backgrounds simple** - Avoid distracting elements behind the avatar
## Common Issues
### Background Not Showing
```typescript
// Wrong: missing url/value
background: {
type: "image"
}
// Correct
background: {
type: "image",
url: "https://example.com/bg.jpg"
}
```
### Aspect Ratio Mismatch
If your background doesn't match the video dimensions, it may be cropped or stretched. Always match your background aspect ratio to your video dimensions:
```typescript
// For 1920x1080 video
// Use 1920x1080 background image
// For 1080x1920 portrait video
// Use 1080x1920 background image
```
### Video Background Audio
Background video audio is typically muted to avoid conflicting with the avatar's voice. If you need background music, add it as a separate audio track in post-production.
.claude/skills/avatar-video/references/captions.md
+277
View File
@@ -0,0 +1,277 @@
---
name: captions
description: Auto-generated captions and subtitle options for HeyGen videos
---
# Video Captions
HeyGen can automatically generate captions (subtitles) for your videos, improving accessibility and engagement.
## Enabling Captions
Captions can be enabled when generating a video:
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello! This video will have automatic captions.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
// Caption settings (availability varies by plan)
caption: true,
};
```
## Caption Configuration Options
```typescript
interface CaptionConfig {
// Enable/disable captions
enabled: boolean;
// Caption style
style?: {
font_family?: string;
font_size?: number;
font_color?: string;
background_color?: string;
position?: "top" | "bottom";
};
// Language for caption generation
language?: string;
}
```
## Caption Styles
### Basic Captions
```typescript
const config = {
video_inputs: [...],
caption: true, // Enable with default styling
};
```
### Styled Captions
```typescript
const config = {
video_inputs: [...],
caption: {
enabled: true,
style: {
font_family: "Arial",
font_size: 32,
font_color: "#FFFFFF",
background_color: "rgba(0, 0, 0, 0.7)",
position: "bottom",
},
},
};
```
## Multi-Language Captions
For videos in different languages, captions are generated based on the voice language:
```typescript
// Spanish video with Spanish captions
const spanishConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "¡Hola! Este video tendrá subtítulos en español.",
voice_id: "spanish_voice_id",
},
},
],
caption: true,
};
```
## Working with SRT Files
### SRT File Format
Standard SRT format:
```srt
1
00:00:00,000 --> 00:00:03,000
Hello! This video will have
2
00:00:03,000 --> 00:00:06,000
automatic captions generated.
3
00:00:06,000 --> 00:00:09,000
They sync with the audio.
```
### Using Custom SRT
For video translation, you can provide your own SRT:
```typescript
const translationConfig = {
input_video_id: "original_video_id",
output_languages: ["es-ES", "fr-FR"],
srt_key: "path/to/custom.srt", // Custom SRT file
srt_role: "input", // "input" or "output"
};
```
## Caption Positioning
### Bottom (Default)
Standard position for most videos:
```typescript
caption: {
enabled: true,
style: {
position: "bottom"
}
}
```
### Top
For videos where bottom space is occupied:
```typescript
caption: {
enabled: true,
style: {
position: "top"
}
}
```
## Accessibility Best Practices
1. **Always enable captions** - Improves accessibility for deaf/hard-of-hearing viewers
2. **Use high contrast** - White text on dark background or vice versa
3. **Readable font size** - At least 24px for standard video, larger for mobile
4. **Don't cover important content** - Position captions away from key visual elements
5. **Sync timing** - Ensure captions match audio timing accurately
## Caption Helper Functions
```typescript
interface CaptionStyle {
font_family: string;
font_size: number;
font_color: string;
background_color: string;
position: "top" | "bottom";
}
const captionPresets: Record<string, CaptionStyle> = {
default: {
font_family: "Arial",
font_size: 32,
font_color: "#FFFFFF",
background_color: "rgba(0, 0, 0, 0.7)",
position: "bottom",
},
minimal: {
font_family: "Arial",
font_size: 28,
font_color: "#FFFFFF",
background_color: "transparent",
position: "bottom",
},
bold: {
font_family: "Arial",
font_size: 36,
font_color: "#FFFFFF",
background_color: "rgba(0, 0, 0, 0.9)",
position: "bottom",
},
branded: {
font_family: "Roboto",
font_size: 30,
font_color: "#00D1FF",
background_color: "rgba(26, 26, 46, 0.9)",
position: "bottom",
},
};
function createCaptionConfig(preset: keyof typeof captionPresets) {
return {
enabled: true,
style: captionPresets[preset],
};
}
```
## Social Media Caption Considerations
### TikTok / Instagram Reels
- Position captions in center or upper portion
- Avoid bottom 20% (covered by UI elements)
- Use larger font sizes for mobile viewing
```typescript
const socialCaptions = {
enabled: true,
style: {
font_size: 42,
position: "top", // Avoid bottom UI elements
},
};
```
### YouTube
- Standard bottom captions work well
- YouTube also supports closed captions upload
### LinkedIn
- Captions highly recommended (many watch without sound)
- Professional styling preferred
## Limitations
- Caption styles may be limited depending on your subscription tier
- Some advanced caption features may require the web interface
- Multi-speaker caption detection may have limited availability
- Caption accuracy depends on audio quality and speech clarity
## Integration with Video Translation
When using video translation, captions are automatically handled:
```typescript
// Video translation includes caption generation
const translationConfig = {
input_video_id: "original_video_id",
output_languages: ["es-ES"],
// Captions generated in target language
};
```
See the **video-translate** skill for more details on video translation.
.claude/skills/avatar-video/references/dimensions.md
+314
View File
@@ -0,0 +1,314 @@
---
name: dimensions
description: Resolution options (720p/1080p) and aspect ratios for HeyGen videos
---
# Video Dimensions and Resolution
HeyGen supports various video dimensions and aspect ratios to fit different platforms and use cases.
## Standard Resolutions
### Landscape (16:9)
| Resolution | Width | Height | Use Case |
|------------|-------|--------|----------|
| 720p | 1280 | 720 | Standard quality, faster processing |
| 1080p | 1920 | 1080 | High quality, most common |
### Portrait (9:16)
| Resolution | Width | Height | Use Case |
|------------|-------|--------|----------|
| 720p | 720 | 1280 | Mobile-first content |
| 1080p | 1080 | 1920 | High quality vertical |
### Square (1:1)
| Resolution | Width | Height | Use Case |
|------------|-------|--------|----------|
| 720p | 720 | 720 | Social media posts |
| 1080p | 1080 | 1080 | High quality square |
## Setting Dimensions
### TypeScript
```typescript
// Landscape 1080p
const landscapeConfig = {
video_inputs: [...],
dimension: {
width: 1920,
height: 1080
}
};
// Portrait 1080p
const portraitConfig = {
video_inputs: [...],
dimension: {
width: 1080,
height: 1920
}
};
// Square 1080p
const squareConfig = {
video_inputs: [...],
dimension: {
width: 1080,
height: 1080
}
};
```
### curl
```bash
# Landscape 1080p
curl -X POST "https://api.heygen.com/v2/video/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"video_inputs": [...],
"dimension": {
"width": 1920,
"height": 1080
}
}'
```
## Dimension Helper Functions
```typescript
type AspectRatio = "16:9" | "9:16" | "1:1" | "4:3" | "4:5";
type Quality = "720p" | "1080p";
interface Dimensions {
width: number;
height: number;
}
function getDimensions(aspectRatio: AspectRatio, quality: Quality): Dimensions {
const configs: Record<AspectRatio, Record<Quality, Dimensions>> = {
"16:9": {
"720p": { width: 1280, height: 720 },
"1080p": { width: 1920, height: 1080 },
},
"9:16": {
"720p": { width: 720, height: 1280 },
"1080p": { width: 1080, height: 1920 },
},
"1:1": {
"720p": { width: 720, height: 720 },
"1080p": { width: 1080, height: 1080 },
},
"4:3": {
"720p": { width: 960, height: 720 },
"1080p": { width: 1440, height: 1080 },
},
"4:5": {
"720p": { width: 576, height: 720 },
"1080p": { width: 864, height: 1080 },
},
};
return configs[aspectRatio][quality];
}
// Usage
const youTubeDimensions = getDimensions("16:9", "1080p");
const tikTokDimensions = getDimensions("9:16", "1080p");
const instagramDimensions = getDimensions("1:1", "1080p");
```
## Platform-Specific Recommendations
### YouTube
```typescript
const youtubeConfig = {
video_inputs: [...],
dimension: { width: 1920, height: 1080 }, // 16:9 landscape
};
```
### TikTok / Instagram Reels / YouTube Shorts
```typescript
const shortFormConfig = {
video_inputs: [...],
dimension: { width: 1080, height: 1920 }, // 9:16 portrait
};
```
### Instagram Feed Post
```typescript
const instagramFeedConfig = {
video_inputs: [...],
dimension: { width: 1080, height: 1080 }, // 1:1 square
};
```
### LinkedIn
```typescript
const linkedinConfig = {
video_inputs: [...],
dimension: { width: 1920, height: 1080 }, // 16:9 landscape preferred
};
```
### Twitter/X
```typescript
const twitterConfig = {
video_inputs: [...],
dimension: { width: 1280, height: 720 }, // 16:9, 720p is common
};
```
## Avatar IV Dimensions
For Avatar IV (photo-based avatars), dimensions are set via orientation:
```typescript
type VideoOrientation = "portrait" | "landscape" | "square";
function getAvatarIVDimensions(orientation: VideoOrientation): Dimensions {
switch (orientation) {
case "portrait":
return { width: 720, height: 1280 };
case "landscape":
return { width: 1280, height: 720 };
case "square":
return { width: 720, height: 720 };
}
}
```
## Custom Dimensions
HeyGen supports custom dimensions within limits:
```typescript
const customConfig = {
video_inputs: [...],
dimension: {
width: 1600,
height: 900 // Custom 16:9 at non-standard resolution
}
};
```
### Dimension Constraints
- **Minimum**: 128px on any side
- **Maximum**: 4096px on any side
- **Must be even numbers**: Both width and height must be divisible by 2
```typescript
function validateDimensions(width: number, height: number): boolean {
if (width < 128 || height < 128) {
throw new Error("Dimensions must be at least 128px");
}
if (width > 4096 || height > 4096) {
throw new Error("Dimensions cannot exceed 4096px");
}
if (width % 2 !== 0 || height % 2 !== 0) {
throw new Error("Dimensions must be even numbers");
}
return true;
}
```
## Resolution vs. Credit Cost
Higher resolutions may consume more credits:
| Resolution | Relative Cost |
|------------|---------------|
| 720p | Base rate |
| 1080p | ~1.5x base rate |
Consider using 720p for drafts and testing, then 1080p for final output.
## Background Considerations
Match background image/video dimensions to your video dimensions:
```typescript
// For 1080p landscape video
const config = {
video_inputs: [
{
character: {...},
voice: {...},
background: {
type: "image",
url: "https://example.com/1920x1080-background.jpg" // Match video dimensions
}
}
],
dimension: { width: 1920, height: 1080 }
};
```
## Creating a Video Config Factory
```typescript
interface VideoConfigOptions {
script: string;
avatarId: string;
voiceId: string;
platform: "youtube" | "tiktok" | "instagram_feed" | "instagram_story" | "linkedin";
quality?: "720p" | "1080p";
}
function createVideoConfig(options: VideoConfigOptions) {
const platformDimensions: Record<string, Dimensions> = {
youtube: { width: 1920, height: 1080 },
tiktok: { width: 1080, height: 1920 },
instagram_feed: { width: 1080, height: 1080 },
instagram_story: { width: 1080, height: 1920 },
linkedin: { width: 1920, height: 1080 },
};
const dimension = platformDimensions[options.platform];
// Scale down for 720p if requested
if (options.quality === "720p") {
dimension.width = Math.round((dimension.width * 720) / 1080);
dimension.height = Math.round((dimension.height * 720) / 1080);
}
return {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: options.avatarId,
avatar_style: "normal",
},
voice: {
type: "text",
input_text: options.script,
voice_id: options.voiceId,
},
},
],
dimension,
};
}
// Usage
const tiktokVideo = createVideoConfig({
script: "Hey everyone! Check this out!",
avatarId: "josh_lite3_20230714",
voiceId: "1bd001e7e50f421d891986aad5158bc8",
platform: "tiktok",
quality: "1080p",
});
```
.claude/skills/avatar-video/references/photo-avatars.md
+853
View File
@@ -0,0 +1,853 @@
---
name: photo-avatars
description: Creating avatars from photos (talking photos) for HeyGen
---
# Photo Avatars (Talking Photos)
Photo avatars allow you to animate a static photo and make it speak. This is useful for creating personalized video content from portraits, headshots, or any suitable image.
## Creating a Photo Avatar from an Uploaded Image
The workflow is: **Upload Image → Create Avatar Group → Use in Video**
### Step 1: Upload the Image
Upload a portrait photo using the asset upload endpoint. The response includes an `image_key` which you'll use in the next step.
```bash
curl -X POST "https://upload.heygen.com/v1/asset" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: image/jpeg" \
--data-binary '@./portrait.jpg'
```
Response:
```json
{
"code": 100,
"data": {
"id": "741299e941764988b432ed3a6757878f",
"name": "741299e941764988b432ed3a6757878f",
"file_type": "image",
"url": "https://resource2.heygen.ai/image/.../original.jpg",
"image_key": "image/741299e941764988b432ed3a6757878f/original.jpg"
}
}
```
> **Important:** Save the `image_key` field (not the `id`). The `image_key` is the S3 path used to create the photo avatar.
See [assets.md](assets.md) for full upload details.
### Step 2: Create Photo Avatar Group
Use the `image_key` from the upload response to create a photo avatar group. This processes the image and creates a usable photo avatar.
**Endpoint:** `POST https://api.heygen.com/v2/photo_avatar/avatar_group/create`
```bash
curl -X POST "https://api.heygen.com/v2/photo_avatar/avatar_group/create" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"image_key": "image/741299e941764988b432ed3a6757878f/original.jpg",
"name": "My Photo Avatar"
}'
```
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `image_key` | string | ✓ | S3 image key from upload response |
| `name` | string | ✓ | Display name for the avatar |
| `generation_id` | string | | If using AI-generated photo (see below) |
Response:
```json
{
"error": null,
"data": {
"id": "045c260bc0364727b2cbe50442c3a5bf",
"image_url": "https://files2.heygen.ai/...",
"created_at": 1771798135.777256,
"name": "My Photo Avatar",
"status": "pending",
"group_id": "045c260bc0364727b2cbe50442c3a5bf",
"is_motion": false,
"business_type": "uploaded"
}
}
```
The `id` (same as `group_id`) is your `talking_photo_id` for video generation.
### Step 3: Wait for Processing
The photo avatar starts with `status: "pending"` and transitions to `"completed"` within seconds. Poll the status endpoint:
**Endpoint:** `GET https://api.heygen.com/v2/photo_avatar/{id}`
```bash
curl "https://api.heygen.com/v2/photo_avatar/045c260bc0364727b2cbe50442c3a5bf" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
Wait until `status` is `"completed"` before using in video generation.
### Step 4: Use in Video Generation
Use the photo avatar `id` as `talking_photo_id`:
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "talking_photo",
talking_photo_id: "045c260bc0364727b2cbe50442c3a5bf",
},
voice: {
type: "text",
input_text: "Hello! This is my photo avatar speaking.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
dimension: { width: 1920, height: 1080 },
};
```
## TypeScript: Complete Workflow
```typescript
import fs from "fs";
import path from "path";
interface AssetUploadResponse {
code: number;
data: {
id: string;
image_key: string;
url: string;
};
}
interface PhotoAvatarResponse {
error: string | null;
data: {
id: string;
group_id: string;
image_url: string;
name: string;
status: string;
is_motion: boolean;
business_type: string;
};
}
async function createPhotoAvatar(
imagePath: string,
name: string
): Promise<string> {
// 1. Upload image
const resolvedPath = path.resolve(imagePath);
const fileBuffer = fs.readFileSync(resolvedPath);
const uploadResponse = await fetch("https://upload.heygen.com/v1/asset", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "image/jpeg",
},
body: fileBuffer,
});
const uploadJson: AssetUploadResponse = await uploadResponse.json();
if (uploadJson.code !== 100) {
throw new Error("Upload failed");
}
const imageKey = uploadJson.data.image_key;
// 2. Create avatar group
const createResponse = await fetch(
"https://api.heygen.com/v2/photo_avatar/avatar_group/create",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({ image_key: imageKey, name }),
}
);
const createJson: PhotoAvatarResponse = await createResponse.json();
if (createJson.error) {
throw new Error(createJson.error);
}
const photoAvatarId = createJson.data.id;
// 3. Wait for processing
await waitForPhotoAvatar(photoAvatarId);
return photoAvatarId;
}
async function waitForPhotoAvatar(id: string): Promise<void> {
for (let i = 0; i < 30; i++) {
const response = await fetch(
`https://api.heygen.com/v2/photo_avatar/${id}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json: PhotoAvatarResponse = await response.json();
if (json.data.status === "completed") return;
if (json.data.status === "failed") {
throw new Error("Photo avatar processing failed");
}
await new Promise((r) => setTimeout(r, 2000));
}
throw new Error("Photo avatar processing timed out");
}
async function createVideoFromPhoto(
photoPath: string,
script: string,
voiceId: string
): Promise<string> {
// 1. Create photo avatar
const talkingPhotoId = await createPhotoAvatar(photoPath, "Video Avatar");
// 2. Generate video
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
video_inputs: [
{
character: {
type: "talking_photo",
talking_photo_id: talkingPhotoId,
},
voice: {
type: "text",
input_text: script,
voice_id: voiceId,
},
},
],
dimension: { width: 1920, height: 1080 },
}),
});
const { data } = await response.json();
return data.video_id;
}
```
## Python: Complete Workflow
```python
import requests
import os
import time
def create_photo_avatar(image_path: str, name: str) -> str:
api_key = os.environ["HEYGEN_API_KEY"]
# 1. Upload image
with open(image_path, "rb") as f:
upload_resp = requests.post(
"https://upload.heygen.com/v1/asset",
headers={
"X-Api-Key": api_key,
"Content-Type": "image/jpeg",
},
data=f,
)
upload_data = upload_resp.json()
if upload_data.get("code") != 100:
raise Exception("Upload failed")
image_key = upload_data["data"]["image_key"]
# 2. Create avatar group
create_resp = requests.post(
"https://api.heygen.com/v2/photo_avatar/avatar_group/create",
headers={
"X-Api-Key": api_key,
"Content-Type": "application/json",
},
json={"image_key": image_key, "name": name},
)
create_data = create_resp.json()
if create_data.get("error"):
raise Exception(create_data["error"])
photo_avatar_id = create_data["data"]["id"]
# 3. Wait for processing
for _ in range(30):
status_resp = requests.get(
f"https://api.heygen.com/v2/photo_avatar/{photo_avatar_id}",
headers={"X-Api-Key": api_key},
)
status = status_resp.json()["data"]["status"]
if status == "completed":
return photo_avatar_id
if status == "failed":
raise Exception("Photo avatar processing failed")
time.sleep(2)
raise Exception("Photo avatar processing timed out")
```
## Listing Existing Talking Photos
Retrieve all talking photos in your account:
**Endpoint:** `GET https://api.heygen.com/v1/talking_photo.list`
```bash
curl "https://api.heygen.com/v1/talking_photo.list" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
Response:
```json
{
"code": 100,
"data": [
{
"id": "ef0ed70f72c6497793e5e36e434d2aea",
"image_url": "https://files2.heygen.ai/talking_photo/.../image.WEBP",
"circle_image": ""
}
]
}
```
Each `id` can be used as `talking_photo_id` in video generation.
## Adding Photos to an Existing Group
Add additional photo looks to an existing avatar group:
**Endpoint:** `POST https://api.heygen.com/v2/photo_avatar/avatar_group/add`
```typescript
async function addPhotosToGroup(
groupId: string,
imageKeys: string[],
name: string
): Promise<void> {
const response = await fetch(
"https://api.heygen.com/v2/photo_avatar/avatar_group/add",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
group_id: groupId,
image_keys: imageKeys,
name,
}),
}
);
const json = await response.json();
if (json.error) {
throw new Error(json.error);
}
}
```
## Training a Photo Avatar Group
Train the avatar group for improved animation quality:
**Endpoint:** `POST https://api.heygen.com/v2/photo_avatar/train`
```bash
curl -X POST "https://api.heygen.com/v2/photo_avatar/train" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{"group_id": "045c260bc0364727b2cbe50442c3a5bf"}'
```
Check training status:
**Endpoint:** `GET https://api.heygen.com/v2/photo_avatar/train/status/{group_id}`
## Avatar IV Video Generation
Avatar IV is HeyGen's latest photo avatar technology with improved quality and natural motion. It generates a video directly from an uploaded image, bypassing the avatar group creation step.
**Endpoint:** `POST https://api.heygen.com/v2/video/av4/generate`
```bash
curl -X POST "https://api.heygen.com/v2/video/av4/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"image_key": "image/741299e941764988b432ed3a6757878f/original.jpg",
"script": "Hello! This is Avatar IV with enhanced quality.",
"voice_id": "1bd001e7e50f421d891986aad5158bc8",
"video_orientation": "landscape",
"video_title": "My Avatar IV Video"
}'
```
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `image_key` | string | ✓ | S3 image key from asset upload |
| `script` | string | ✓ | Text for the avatar to speak |
| `voice_id` | string | ✓ | Voice to use |
| `video_orientation` | string | | `"portrait"`, `"landscape"`, or `"square"` |
| `video_title` | string | | Title for the video |
| `fit` | string | | `"cover"` or `"contain"` |
| `custom_motion_prompt` | string | | Motion/expression description |
| `enhance_custom_motion_prompt` | boolean | | Enhance the motion prompt with AI |
### TypeScript
```typescript
interface AvatarIVRequest {
image_key: string;
script: string;
voice_id: string;
video_orientation?: "portrait" | "landscape" | "square";
video_title?: string;
fit?: "cover" | "contain";
custom_motion_prompt?: string;
enhance_custom_motion_prompt?: boolean;
}
interface AvatarIVResponse {
error: null | string;
data: {
video_id: string;
};
}
async function generateAvatarIVVideo(
config: AvatarIVRequest
): Promise<string> {
const response = await fetch(
"https://api.heygen.com/v2/video/av4/generate",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
}
);
const json: AvatarIVResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.video_id;
}
```
### Avatar IV Options
| Orientation | Dimensions | Use Case |
|-------------|------------|----------|
| `portrait` | 720x1280 | TikTok, Stories |
| `landscape` | 1280x720 | YouTube, Web |
| `square` | 720x720 | Instagram Feed |
| Fit | Description |
|-----|-------------|
| `cover` | Fill the frame, may crop edges |
| `contain` | Fit entire image, may show background |
### Custom Motion Prompts
```typescript
const videoId = await generateAvatarIVVideo({
image_key: "image/.../original.jpg",
script: "Let me tell you about our product.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
custom_motion_prompt: "nodding head and smiling",
enhance_custom_motion_prompt: true,
});
```
## Generating AI Photo Avatars
Generate synthetic photo avatars from text descriptions instead of uploading a photo.
**Endpoint:** `POST https://api.heygen.com/v2/photo_avatar/photo/generate`
> **IMPORTANT: All 8 fields are REQUIRED.** The API will reject requests missing any field.
> When a user asks to "generate an AI avatar of a professional man", you need to ask for or select values for ALL fields below.
### Required Fields (ALL must be provided)
| Field | Type | Allowed Values |
|-------|------|----------------|
| `name` | string | Name for the generated avatar |
| `age` | enum | `"Young Adult"`, `"Early Middle Age"`, `"Late Middle Age"`, `"Senior"`, `"Unspecified"` |
| `gender` | enum | `"Woman"`, `"Man"`, `"Unspecified"` |
| `ethnicity` | enum | `"White"`, `"Black"`, `"Asian American"`, `"East Asian"`, `"South East Asian"`, `"South Asian"`, `"Middle Eastern"`, `"Pacific"`, `"Hispanic"`, `"Unspecified"` |
| `orientation` | enum | `"square"`, `"horizontal"`, `"vertical"` |
| `pose` | enum | `"half_body"`, `"close_up"`, `"full_body"` |
| `style` | enum | `"Realistic"`, `"Pixar"`, `"Cinematic"`, `"Vintage"`, `"Noir"`, `"Cyberpunk"`, `"Unspecified"` |
| `appearance` | string | Text prompt describing appearance (clothing, mood, lighting, etc). Max 1000 chars |
### curl Example
```bash
curl -X POST "https://api.heygen.com/v2/photo_avatar/photo/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Sarah Product Demo",
"age": "Young Adult",
"gender": "Woman",
"ethnicity": "White",
"orientation": "horizontal",
"pose": "half_body",
"style": "Realistic",
"appearance": "Professional woman with a friendly smile, wearing a navy blue blazer over a white blouse, soft studio lighting, clean neutral background"
}'
```
Response:
```json
{
"error": null,
"data": {
"generation_id": "6a7f7f2795de4599bec7cf1e06babe30"
}
}
```
### Check Generation Status
**Endpoint:** `GET https://api.heygen.com/v2/photo_avatar/generation/{generation_id}`
The response includes multiple generated images to choose from:
```json
{
"error": null,
"data": {
"id": "6a7f7f2795de4599bec7cf1e06babe30",
"status": "success",
"image_url_list": [
"https://resource2.heygen.ai/photo_generation/.../image1.jpg",
"https://resource2.heygen.ai/photo_generation/.../image2.jpg",
"https://resource2.heygen.ai/photo_generation/.../image3.jpg",
"https://resource2.heygen.ai/photo_generation/.../image4.jpg"
],
"image_key_list": [
"photo_generation/.../image1.jpg",
"photo_generation/.../image2.jpg",
"photo_generation/.../image3.jpg",
"photo_generation/.../image4.jpg"
]
}
}
```
### TypeScript
```typescript
interface GeneratePhotoAvatarRequest {
name: string;
age: "Young Adult" | "Early Middle Age" | "Late Middle Age" | "Senior" | "Unspecified";
gender: "Woman" | "Man" | "Unspecified";
ethnicity: "White" | "Black" | "Asian American" | "East Asian" | "South East Asian" | "South Asian" | "Middle Eastern" | "Pacific" | "Hispanic" | "Unspecified";
orientation: "square" | "horizontal" | "vertical";
pose: "half_body" | "close_up" | "full_body";
style: "Realistic" | "Pixar" | "Cinematic" | "Vintage" | "Noir" | "Cyberpunk" | "Unspecified";
appearance: string;
}
interface GeneratePhotoAvatarResponse {
error: string | null;
data: {
generation_id: string;
};
}
interface PhotoGenerationStatus {
error: string | null;
data: {
id: string;
status: "pending" | "processing" | "success" | "failed";
msg: string | null;
image_url_list?: string[];
image_key_list?: string[];
};
}
async function generatePhotoAvatar(
config: GeneratePhotoAvatarRequest
): Promise<string> {
const response = await fetch(
"https://api.heygen.com/v2/photo_avatar/photo/generate",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
}
);
const json: GeneratePhotoAvatarResponse = await response.json();
if (json.error) {
throw new Error(`Photo avatar generation failed: ${json.error}`);
}
return json.data.generation_id;
}
async function waitForPhotoGeneration(
generationId: string
): Promise<string[]> {
for (let i = 0; i < 60; i++) {
const response = await fetch(
`https://api.heygen.com/v2/photo_avatar/generation/${generationId}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json: PhotoGenerationStatus = await response.json();
if (json.error) throw new Error(json.error);
if (json.data.status === "success") {
return json.data.image_key_list!;
}
if (json.data.status === "failed") {
throw new Error(json.data.msg ?? "Photo generation failed");
}
await new Promise((r) => setTimeout(r, 5000));
}
throw new Error("Photo generation timed out");
}
```
### AI Photo → Avatar Group → Video
Use a generated AI photo to create an avatar group, then generate a video:
```typescript
// 1. Generate AI photo
const generationId = await generatePhotoAvatar({
name: "Product Demo Host",
age: "Young Adult",
gender: "Woman",
ethnicity: "Unspecified",
orientation: "horizontal",
pose: "half_body",
style: "Realistic",
appearance: "Professional woman, navy blazer, friendly smile, soft lighting",
});
// 2. Wait for generation and pick first result
const imageKeys = await waitForPhotoGeneration(generationId);
const selectedImageKey = imageKeys[0];
// 3. Create avatar group from the AI photo
const createResponse = await fetch(
"https://api.heygen.com/v2/photo_avatar/avatar_group/create",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
image_key: selectedImageKey,
name: "Product Demo Host",
generation_id: generationId,
}),
}
);
const { data } = await createResponse.json();
const talkingPhotoId = data.id;
// 4. Generate video (after status is "completed")
const videoId = await generateVideo({
video_inputs: [{
character: {
type: "talking_photo",
talking_photo_id: talkingPhotoId,
},
voice: {
type: "text",
input_text: "Welcome to our product demo!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
}],
dimension: { width: 1920, height: 1080 },
});
```
### Pre-Generation Checklist
Before calling the AI generation API, ensure you have values for ALL fields:
| # | Field | Question to Ask / Default |
|---|-------|---------------------------|
| 1 | `name` | What should we call this avatar? |
| 2 | `age` | Young Adult / Early Middle Age / Late Middle Age / Senior? |
| 3 | `gender` | Woman / Man? |
| 4 | `ethnicity` | Which ethnicity? (see enum values above) |
| 5 | `orientation` | horizontal (landscape) / vertical (portrait) / square? |
| 6 | `pose` | half_body (recommended) / close_up / full_body? |
| 7 | `style` | Realistic (recommended) / Cinematic / other? |
| 8 | `appearance` | Describe clothing, expression, lighting, background |
**If the user only provides a vague request** like "create a professional looking man", ask them to specify the missing fields OR make reasonable defaults (e.g., "Early Middle Age", "Realistic" style, "half_body" pose, "horizontal" orientation).
### Appearance Prompt Tips
The `appearance` field is a text prompt - be descriptive:
**Good prompts:**
- "Professional woman with shoulder-length brown hair, wearing a light blue button-down shirt, warm friendly smile, soft studio lighting, clean white background"
- "Young man with short black hair, casual tech startup style, wearing a dark hoodie, confident expression, modern office background with plants"
**Avoid:**
- Vague descriptions: "a nice person"
- Conflicting attributes
- Requesting specific real people
## Managing Photo Avatars
### Get Photo Avatar Details
**Endpoint:** `GET https://api.heygen.com/v2/photo_avatar/{id}`
```typescript
async function getPhotoAvatar(id: string): Promise<PhotoAvatarResponse> {
const response = await fetch(
`https://api.heygen.com/v2/photo_avatar/${id}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
return response.json();
}
```
### Delete Photo Avatar
**Endpoint:** `DELETE https://api.heygen.com/v2/photo_avatar/{id}`
```typescript
async function deletePhotoAvatar(id: string): Promise<void> {
const response = await fetch(
`https://api.heygen.com/v2/photo_avatar/${id}`,
{
method: "DELETE",
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
}
);
if (!response.ok) {
throw new Error("Failed to delete photo avatar");
}
}
```
### Delete Photo Avatar Group
**Endpoint:** `DELETE https://api.heygen.com/v2/photo_avatar_group/{group_id}`
```typescript
async function deletePhotoAvatarGroup(groupId: string): Promise<void> {
const response = await fetch(
`https://api.heygen.com/v2/photo_avatar_group/${groupId}`,
{
method: "DELETE",
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
}
);
if (!response.ok) {
throw new Error("Failed to delete photo avatar group");
}
}
```
## API Reference
| Endpoint | Method | Description |
|----------|--------|-------------|
| `upload.heygen.com/v1/asset` | POST | Upload image (returns `image_key`) |
| `/v2/photo_avatar/avatar_group/create` | POST | Create photo avatar from `image_key` |
| `/v2/photo_avatar/avatar_group/add` | POST | Add photos to existing group |
| `/v2/photo_avatar/train` | POST | Train avatar group |
| `/v2/photo_avatar/train/status/{group_id}` | GET | Check training status |
| `/v2/photo_avatar/{id}` | GET | Get photo avatar details/status |
| `/v2/photo_avatar/{id}` | DELETE | Delete photo avatar |
| `/v2/photo_avatar_group/{id}` | DELETE | Delete avatar group |
| `/v2/photo_avatar/photo/generate` | POST | Generate AI photo from text |
| `/v2/photo_avatar/generation/{id}` | GET | Check AI generation status |
| `/v2/video/av4/generate` | POST | Avatar IV video from `image_key` |
| `/v1/talking_photo.list` | GET | List all existing talking photos |
| `/v2/video/generate` | POST | Generate video with `talking_photo_id` |
## Photo Requirements
### Technical Requirements
| Aspect | Requirement |
|--------|-------------|
| Format | JPEG, PNG |
| Resolution | Minimum 512x512px |
| File size | Under 10MB |
| Face visibility | Clear, front-facing |
### Quality Guidelines
1. **Lighting** - Even, natural lighting on face
2. **Expression** - Neutral or slight smile
3. **Background** - Simple, uncluttered
4. **Face position** - Centered, not cut off
5. **Clarity** - Sharp, in focus
6. **Angle** - Straight-on or slight angle
## Best Practices
1. **Use high-quality photos** - Better input = better output
2. **Front-facing portraits** - Work best for animation
3. **Neutral expressions** - Allow for more natural animation
4. **Use Avatar IV for best quality** - Latest generation technology
5. **Train avatar groups** - Improves animation quality
6. **Reuse photo avatar IDs** - Once created, use the same `talking_photo_id` across multiple videos
## Limitations
- Photo quality significantly affects output
- Side-profile photos have limited support
- Full-body photos may not animate properly
- Some expressions may look unnatural
- Processing time varies by complexity
.claude/skills/avatar-video/references/quota.md
+195
View File
@@ -0,0 +1,195 @@
---
name: quota
description: Credit system, usage limits, and checking remaining quota for HeyGen
---
# HeyGen Quota and Credits
HeyGen uses a credit-based system for video generation. Understanding quota management helps prevent failed video generation requests.
## Checking Remaining Quota
### curl
```bash
curl -X GET "https://api.heygen.com/v2/user/remaining_quota" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface QuotaResponse {
error: null | string;
data: {
remaining_quota: number;
used_quota: number;
};
}
const response = await fetch("https://api.heygen.com/v2/user/remaining_quota", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const { data }: QuotaResponse = await response.json();
console.log(`Remaining credits: ${data.remaining_quota}`);
```
### Python
```python
import requests
import os
response = requests.get(
"https://api.heygen.com/v2/user/remaining_quota",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()["data"]
print(f"Remaining credits: {data['remaining_quota']}")
```
## Response Format
```json
{
"error": null,
"data": {
"remaining_quota": 450,
"used_quota": 50
}
}
```
## Credit Consumption
Different operations consume different amounts of credits:
| Operation | Credit Cost | Notes |
|-----------|-------------|-------|
| Standard video (1 min) | ~1 credit per minute | Varies by resolution |
| 720p video | Base rate | Standard quality |
| 1080p video | ~1.5x base rate | Higher quality |
| Video translation | Varies | Depends on video length |
| Streaming avatar | Per session | Real-time usage |
## Pre-Generation Quota Check
Always verify sufficient quota before generating videos:
```typescript
async function generateVideoWithQuotaCheck(videoConfig: VideoConfig) {
// Check quota first
const quotaResponse = await fetch(
"https://api.heygen.com/v2/user/remaining_quota",
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data: quota } = await quotaResponse.json();
// Estimate required credits (rough estimate: 1 credit per minute)
const estimatedMinutes = videoConfig.estimatedDuration / 60;
const requiredCredits = Math.ceil(estimatedMinutes);
if (quota.remaining_quota < requiredCredits) {
throw new Error(
`Insufficient credits. Need ${requiredCredits}, have ${quota.remaining_quota}`
);
}
// Proceed with video generation
return generateVideo(videoConfig);
}
```
## Quota Management Best Practices
### 1. Monitor Usage Regularly
```typescript
async function logQuotaUsage() {
const response = await fetch(
"https://api.heygen.com/v2/user/remaining_quota",
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
console.log({
remaining: data.remaining_quota,
used: data.used_quota,
percentUsed: (
(data.used_quota / (data.remaining_quota + data.used_quota)) *
100
).toFixed(1),
});
}
```
### 2. Set Up Alerts
```typescript
const QUOTA_WARNING_THRESHOLD = 50;
async function checkQuotaWithAlert() {
const response = await fetch(
"https://api.heygen.com/v2/user/remaining_quota",
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
if (data.remaining_quota < QUOTA_WARNING_THRESHOLD) {
// Send alert (email, Slack, etc.)
await sendAlert(`Low HeyGen quota: ${data.remaining_quota} credits remaining`);
}
return data;
}
```
### 3. Use Test Mode for Development
When available, use test mode to avoid consuming credits during development:
```typescript
const videoConfig = {
test: true, // Use test mode during development
video_inputs: [...],
};
// Test videos may have watermarks but don't consume credits
```
## Subscription Tiers
Different subscription tiers have different quota allocations and features:
| Tier | Features |
|------|----------|
| Free | Limited credits, basic features |
| Creator | More credits, standard avatars |
| Team | Higher limits, team collaboration |
| Enterprise | Custom limits, API access, priority support |
API access typically requires Enterprise tier or higher.
## Error Handling for Quota Issues
```typescript
async function handleQuotaError(error: any) {
if (error.message.includes("quota") || error.message.includes("credit")) {
console.error("Quota exceeded. Consider:");
console.error("1. Upgrading your subscription");
console.error("2. Waiting for quota reset");
console.error("3. Purchasing additional credits");
// Check current quota
const quota = await getQuota();
console.error(`Current remaining: ${quota.remaining_quota}`);
}
throw error;
}
```
.claude/skills/avatar-video/references/remotion-integration.md
+705
View File
@@ -0,0 +1,705 @@
---
name: remotion-integration
description: Using HeyGen avatar videos in Remotion compositions
---
# HeyGen + Remotion Integration
This guide covers workflows for generating HeyGen avatar videos and using them in Remotion compositions.
## Quick Start
```typescript
// 1. Get avatar with default voice
const avatar = await getAvatarDetails(avatarId);
// 2. Generate video (MP4 with background - most common)
const videoId = await generateVideo({
video_inputs: [{
character: { type: "avatar", avatar_id: avatar.id, avatar_style: "normal" },
voice: { type: "text", input_text: script, voice_id: avatar.default_voice_id },
background: { type: "color", value: "#1a1a2e" },
}],
dimension: { width: 1920, height: 1080 },
});
// 3. Poll for completion (10-15+ min)
// 4. Use in Remotion with motion graphics overlaid on top
```
## Overview
A typical workflow:
1. Generate avatar video with HeyGen
2. Wait for completion and get video URL
3. Download or use URL directly in Remotion
4. Compose with other elements (backgrounds, overlays, animations)
## Choosing the Right Output Format
| Your Composition | Recommended | Why |
|------------------|-------------|-----|
| Avatar as presenter with overlays | MP4 + background | Simpler, overlays go on top |
| Loom-style (avatar over screen recording) | WebM + `closeUp`, mask in Remotion | Need transparency, apply circle mask in CSS |
| Avatar overlaid ON other video/content | WebM (transparent) | Need to see through to content behind |
| Full-screen avatar | MP4 + background | Standard approach |
**Use MP4 with background for most cases.** Use WebM when you need to see content *behind* the avatar.
**Note:** WebM only supports `normal` and `closeUp` styles. For circular framing, use CSS `border-radius: 50%` in Remotion.
## Recommended: Parallel Development Workflow
HeyGen video generation takes **10-15+ minutes**. Don't wait - work in parallel:
1. **Start HeyGen generation** - save `video_id` to a file, exit immediately
2. **Build Remotion composition** - use a placeholder or the avatar's `preview_video_url` (a short loop)
3. **Check HeyGen status** periodically or when done building
4. **Swap placeholder** for real video URL once ready
**Estimate duration from script**: ~150 words/minute speech rate, so `wordCount / 150 * 60 * fps` gives approximate frames.
**Composition tip**: Design components to work with or without the avatar video, so motion graphics can be tested independently.
## Dimension Alignment
**Critical**: Match HeyGen output dimensions to your Remotion composition.
### Common Dimension Presets
```typescript
// Shared dimension constants for both HeyGen and Remotion
const DIMENSIONS = {
landscape_1080p: { width: 1920, height: 1080 },
landscape_720p: { width: 1280, height: 720 },
portrait_1080p: { width: 1080, height: 1920 },
portrait_720p: { width: 720, height: 1280 },
square_1080p: { width: 1080, height: 1080 },
square_720p: { width: 720, height: 720 },
} as const;
type DimensionPreset = keyof typeof DIMENSIONS;
```
### HeyGen Video Generation
```typescript
// Generate HeyGen video with specific dimensions
async function generateHeyGenVideo(
script: string,
avatarId: string,
voiceId: string,
preset: DimensionPreset
): Promise<string> {
const dimension = DIMENSIONS[preset];
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
video_inputs: [
{
character: {
type: "avatar",
avatar_id: avatarId,
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: voiceId,
},
background: {
type: "color",
value: "#00FF00", // Green screen for compositing
},
},
],
dimension,
}),
});
const { data } = await response.json();
return data.video_id;
}
```
### Remotion Composition Setup
```tsx
// remotion/src/Root.tsx
import { Composition } from "remotion";
import { AvatarComposition } from "./AvatarComposition";
const DIMENSIONS = {
landscape_1080p: { width: 1920, height: 1080 },
// ... same as above
};
export const RemotionRoot: React.FC = () => {
return (
<>
<Composition
id="AvatarVideo"
component={AvatarComposition}
durationInFrames={300} // Will be set dynamically
fps={30}
width={DIMENSIONS.landscape_1080p.width}
height={DIMENSIONS.landscape_1080p.height}
defaultProps={{
avatarVideoUrl: "",
}}
/>
</>
);
};
```
## Generating Avatar Video for Remotion
### Standard: MP4 with Background
Most Remotion compositions work best with MP4 + background. Overlays and motion graphics go on top:
```typescript
async function generateAvatarForRemotion(
script: string,
avatarId: string,
voiceId: string,
options: {
style?: "normal" | "closeUp" | "circle";
backgroundColor?: string;
} = {}
): Promise<string> {
const { style = "normal", backgroundColor = "#1a1a2e" } = options;
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
video_inputs: [{
character: {
type: "avatar",
avatar_id: avatarId,
avatar_style: style,
},
voice: {
type: "text",
input_text: script,
voice_id: voiceId,
},
background: {
type: "color",
value: backgroundColor,
},
}],
dimension: { width: 1920, height: 1080 },
}),
});
const { data } = await response.json();
return data.video_id;
}
```
### Transparent Background (WebM)
Only use when you need to see content *behind* the avatar (e.g., avatar overlaid on screen recording):
```typescript
// Use /v1/video.webm endpoint for transparent background
// Note: Different structure than /v2/video/generate
const response = await fetch("https://api.heygen.com/v1/video.webm", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
avatar_pose_id: avatarPoseId, // Required: avatar pose ID
avatar_style: "normal", // Required: "normal" or "closeUp" only
input_text: script, // Required (with voice_id)
voice_id: voiceId, // Required (with input_text)
dimension: { width: 1920, height: 1080 },
}),
});
```
## Using HeyGen Video in Remotion
### Important: Use OffthreadVideo for Frame-Accurate Rendering
**Always use `OffthreadVideo` instead of `Video`** for HeyGen avatar videos. The basic `Video` component uses the browser's video decoder which isn't frame-accurate, causing jitter during rendering. `OffthreadVideo` extracts frames via FFmpeg for smooth, accurate playback.
`OffthreadVideo` is included in the core `remotion` package - no additional install needed.
### Basic Usage
```tsx
// remotion/src/AvatarComposition.tsx
import { OffthreadVideo, useVideoConfig } from "remotion";
interface AvatarCompositionProps {
avatarVideoUrl: string;
}
export const AvatarComposition: React.FC<AvatarCompositionProps> = ({
avatarVideoUrl,
}) => {
return (
<div style={{ flex: 1, backgroundColor: "#1a1a2e" }}>
<OffthreadVideo
src={avatarVideoUrl}
style={{
width: "100%",
height: "100%",
objectFit: "contain",
}}
/>
</div>
);
};
```
### WebM with Transparent Background (Recommended)
Using WebM from `/v1/video.webm` - no chroma keying needed:
```tsx
import { OffthreadVideo, AbsoluteFill, Sequence } from "remotion";
export const AvatarWithMotionGraphics: React.FC<{
avatarWebmUrl: string
}> = ({ avatarWebmUrl }) => {
return (
<AbsoluteFill>
{/* Layer 1: Your background/content */}
<AbsoluteFill style={{ backgroundColor: "#1a1a2e" }}>
<YourMotionGraphics />
</AbsoluteFill>
{/* Layer 2: Avatar with transparent background - use OffthreadVideo for frame-accurate rendering */}
<OffthreadVideo
src={avatarWebmUrl}
transparent
style={{
position: "absolute",
bottom: 0,
right: 0,
width: "50%",
height: "auto",
}}
/>
{/* Layer 3: Overlays on top of avatar */}
<Sequence from={30}>
<AnimatedTitle text="Welcome!" />
</Sequence>
</AbsoluteFill>
);
};
```
### Loom-Style: Circle Avatar Over Screen Recording
Use `closeUp` style + WebM, then apply circular mask in Remotion:
```tsx
import { OffthreadVideo, AbsoluteFill } from "remotion";
export const LoomStyleComposition: React.FC<{
screenRecordingUrl: string;
avatarWebmUrl: string; // Generated with avatar_style: "closeUp" via /v1/video.webm
}> = ({ screenRecordingUrl, avatarWebmUrl }) => {
return (
<AbsoluteFill>
{/* Screen recording fills the frame */}
<OffthreadVideo src={screenRecordingUrl} style={{ width: "100%", height: "100%" }} />
{/* Avatar with circular mask - transparent bg shows screen behind */}
<OffthreadVideo
src={avatarWebmUrl}
transparent
style={{
position: "absolute",
bottom: 40,
left: 40,
width: 180,
height: 180,
borderRadius: "50%", // Circular mask applied in CSS
overflow: "hidden",
objectFit: "cover",
}}
/>
</AbsoluteFill>
);
};
```
**Note:** WebM doesn't support `circle` style - use `normal` or `closeUp` and apply circular masking via CSS.
### Legacy: Green Screen with Chroma Key
If using MP4 with green background (not recommended - use WebM instead):
```tsx
// Note: True chroma key requires WebGL or post-processing
// WebM transparent background is much simpler
<OffthreadVideo
src={avatarVideoUrl}
style={{
mixBlendMode: "multiply", // Basic compositing only
}}
/>
```
### Layered Composition
```tsx
import { OffthreadVideo, Sequence, useVideoConfig, Img } from "remotion";
interface LayeredAvatarProps {
avatarVideoUrl: string;
backgroundUrl: string;
logoUrl: string;
title: string;
}
export const LayeredAvatarComposition: React.FC<LayeredAvatarProps> = ({
avatarVideoUrl,
backgroundUrl,
logoUrl,
title,
}) => {
const { fps } = useVideoConfig();
return (
<div style={{ position: "relative", width: "100%", height: "100%" }}>
{/* Layer 1: Background */}
<Img
src={backgroundUrl}
style={{
position: "absolute",
width: "100%",
height: "100%",
objectFit: "cover",
}}
/>
{/* Layer 2: Avatar video - use OffthreadVideo to prevent jitter */}
<OffthreadVideo
src={avatarVideoUrl}
style={{
position: "absolute",
bottom: 0,
right: 0,
width: "40%",
height: "auto",
}}
/>
{/* Layer 3: Title (appears after 1 second) */}
<Sequence from={fps}>
<div
style={{
position: "absolute",
top: 50,
left: 50,
color: "white",
fontSize: 48,
fontWeight: "bold",
}}
>
{title}
</div>
</Sequence>
{/* Layer 4: Logo */}
<Img
src={logoUrl}
style={{
position: "absolute",
top: 20,
right: 20,
width: 100,
height: "auto",
}}
/>
</div>
);
};
```
## Complete Workflow
### Generate and Compose
```typescript
import { bundle } from "@remotion/bundler";
import { renderMedia, selectComposition } from "@remotion/renderer";
async function generateAvatarVideoForRemotion(
script: string,
outputPath: string
) {
// 1. Generate HeyGen video
console.log("Generating HeyGen avatar video...");
const videoId = await generateHeyGenVideo(
script,
"josh_lite3_20230714",
"1bd001e7e50f421d891986aad5158bc8",
"landscape_1080p"
);
// 2. Wait for completion
console.log("Waiting for HeyGen video...");
const avatarVideoUrl = await waitForVideo(videoId);
console.log(`HeyGen video ready: ${avatarVideoUrl}`);
// 3. Get video duration for Remotion
const avatarDuration = await getVideoDuration(avatarVideoUrl);
const durationInFrames = Math.ceil(avatarDuration * 30); // 30 fps
// 4. Bundle Remotion project
console.log("Bundling Remotion project...");
const bundleLocation = await bundle({
entryPoint: "./remotion/src/index.ts",
});
// 5. Select composition
const composition = await selectComposition({
serveUrl: bundleLocation,
id: "AvatarVideo",
inputProps: {
avatarVideoUrl,
},
});
// 6. Render final video
console.log("Rendering final composition...");
await renderMedia({
composition: {
...composition,
durationInFrames,
},
serveUrl: bundleLocation,
codec: "h264",
outputLocation: outputPath,
inputProps: {
avatarVideoUrl,
},
});
console.log(`Final video rendered: ${outputPath}`);
return outputPath;
}
```
### Dynamic Duration with calculateMetadata
```tsx
// remotion/src/AvatarComposition.tsx
import { CalculateMetadataFunction } from "remotion";
export const calculateAvatarMetadata: CalculateMetadataFunction<
AvatarCompositionProps
> = async ({ props }) => {
// Fetch video duration from HeyGen video
const duration = await getVideoDurationInSeconds(props.avatarVideoUrl);
return {
durationInFrames: Math.ceil(duration * 30),
fps: 30,
width: 1920,
height: 1080,
};
};
// In Root.tsx
<Composition
id="AvatarVideo"
component={AvatarComposition}
calculateMetadata={calculateAvatarMetadata}
defaultProps={{
avatarVideoUrl: "",
}}
/>
```
## Best Practices
### 1. Use Green Screen for Flexibility
Generate HeyGen videos with green screen background when you want to composite:
```typescript
background: {
type: "color",
value: "#00FF00", // Pure green for chroma key
}
```
### 2. Match Frame Rates
HeyGen default is 25 fps. Consider this when setting Remotion fps:
```typescript
// Option 1: Match HeyGen's 25 fps
fps: 25
// Option 2: Use 30 fps with playback rate adjustment
<OffthreadVideo
src={avatarVideoUrl}
playbackRate={25/30} // Slow down slightly to match
/>
```
### 3. URL vs Download: When to Use Each
**Use URL directly** when:
- Previewing in Remotion Studio (`npm run dev`)
- URL won't expire before render completes
- You want faster iteration during development
```tsx
// Direct URL usage - simpler, faster for dev
<OffthreadVideo src={avatarVideoUrl} />
```
**Download first** when:
- URL has expiration (HeyGen URLs expire after ~24 hours)
- Rendering will happen later or repeatedly
- Network reliability is a concern
- You need offline rendering
```typescript
// Download with retry for reliability
async function downloadVideoWithRetry(
url: string,
outputPath: string,
maxRetries = 5
): Promise<string> {
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await fetch(url);
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const buffer = await response.arrayBuffer();
await fs.promises.writeFile(outputPath, Buffer.from(buffer));
return outputPath;
} catch (error) {
const delay = 2000 * Math.pow(2, attempt);
console.log(`Retry ${attempt + 1}/${maxRetries} in ${delay}ms...`);
await new Promise((r) => setTimeout(r, delay));
}
}
throw new Error("Download failed after retries");
}
// Use local file in Remotion
const localPath = await downloadVideoWithRetry(avatarVideoUrl, "./public/avatar.mp4");
```
**Hybrid approach** (recommended for production):
```typescript
// Save both URL and local path in metadata
const metadata = {
videoUrl: result.video_url, // For quick preview
localPath: "./public/avatar.mp4", // For reliable rendering
expiresAt: Date.now() + 24 * 60 * 60 * 1000, // URL expiration
};
// In Remotion component, prefer local if available
const videoSrc = fs.existsSync(localPath) ? staticFile("avatar.mp4") : avatarVideoUrl;
```
### 4. Handle Avatar Positioning
Common avatar positions in compositions:
```typescript
const AVATAR_POSITIONS = {
fullscreen: { width: "100%", height: "100%", position: "center" },
bottomRight: { width: "40%", bottom: 0, right: 0 },
bottomLeft: { width: "40%", bottom: 0, left: 0 },
pictureInPicture: { width: "25%", bottom: 20, right: 20 },
leftThird: { width: "33%", left: 0, height: "100%" },
};
```
## Output Formats
### HeyGen Output
- Format: MP4 (H.264)
- Audio: AAC
- Resolution: As specified in request
### Remotion Output
- Codec: H.264 (default), VP8, VP9, ProRes
- Match or exceed HeyGen quality settings
```typescript
await renderMedia({
codec: "h264",
crf: 18, // High quality
// ...
});
```
## Troubleshooting
### Video Not Playing in Remotion
1. Check URL accessibility (CORS issues)
2. Verify video format compatibility
3. Try downloading locally first
### Dimension Mismatch
Ensure both HeyGen and Remotion use identical dimensions:
```typescript
// Shared config
const VIDEO_CONFIG = {
width: 1920,
height: 1080,
fps: 30,
};
// HeyGen
dimension: { width: VIDEO_CONFIG.width, height: VIDEO_CONFIG.height }
// Remotion
<Composition width={VIDEO_CONFIG.width} height={VIDEO_CONFIG.height} />
```
### Video Jitter During Rendering
If avatar video appears jittery or stuttery in rendered output:
1. **Use `OffthreadVideo` instead of `Video`** - The basic `Video` component uses the browser's video decoder which isn't frame-accurate
2. Update imports (no additional install needed - it's in core `remotion`):
```tsx
// Before (causes jitter)
import { Video } from "remotion";
// After (frame-accurate)
import { OffthreadVideo } from "remotion";
```
3. For WebM with transparency, add the `transparent` prop:
```tsx
<OffthreadVideo src={avatarWebmUrl} transparent />
```
### Audio Sync Issues
If avatar audio drifts:
- Verify source video frame rate
- Check for encoding issues
- Consider re-encoding with consistent settings
.claude/skills/avatar-video/references/scripts.md
+322
View File
@@ -0,0 +1,322 @@
---
name: scripts
description: Writing effective scripts for HeyGen AI avatar videos
---
# Writing Scripts for HeyGen Videos
Scripts for AI avatar videos have different requirements than scripts for human presenters. This guide covers best practices for writing scripts that sound natural and render well.
## Script Basics
### Speech Rate and Duration
Typical speech is approximately **150 words per minute** at normal speed (1.0x). Use this as a rough estimate for planning script length.
| Script Length | Approximate Duration |
|---------------|---------------------|
| 75 words | 30 seconds |
| 150 words | 1 minute |
| 300 words | 2 minutes |
| 450 words | 3 minutes |
| 750 words | 5 minutes |
```typescript
// Estimate video duration from script
function estimateDuration(script: string, speed: number = 1.0): number {
const words = script.split(/\s+/).filter(w => w.length > 0).length;
const wordsPerMinute = 150 * speed;
return words / wordsPerMinute * 60; // seconds
}
// Estimate frames for Remotion
function estimateFrames(script: string, fps: number = 30, speed: number = 1.0): number {
const durationSeconds = estimateDuration(script, speed);
return Math.ceil(durationSeconds * fps);
}
```
### Sentence Structure
**Keep sentences short.** AI voices handle shorter sentences more naturally.
| Guideline | Example |
|-----------|---------|
| **Good**: 10-20 words per sentence | "Our platform helps teams collaborate. It syncs in real-time across all devices." |
| **Avoid**: 30+ word run-on sentences | "Our platform helps teams collaborate more effectively by providing real-time synchronization across all devices while also offering offline support and automatic conflict resolution." |
### Punctuation Affects Delivery
| Punctuation | Effect |
|-------------|--------|
| Period `.` | Full stop, natural pause |
| Comma `,` | Brief pause |
| Question mark `?` | Rising intonation |
| Exclamation `!` | Emphasis (use sparingly) |
| Ellipsis `...` | Trailing off, slight pause |
## Adding Pauses with Break Tags
Use SSML-style `<break>` tags for precise pause control:
```
<break time="Xs"/>
```
Where `X` is seconds (e.g., `0.5s`, `1s`, `1.5s`, `2s`).
### Formatting Rules
| Rule | Correct | Incorrect |
|------|---------|-----------|
| Space before tag | `word <break time="1s"/>` | `word<break time="1s"/>` |
| Space after tag | `<break time="1s"/> word` | `<break time="1s"/>word` |
| Use seconds with "s" | `<break time="1.5s"/>` | `<break time="1500ms"/>` |
| Self-closing tag | `<break time="1s"/>` | `<break time="1s"></break>` |
### When to Use Pauses
| Situation | Recommended Pause | Example |
|-----------|-------------------|---------|
| After greeting | 0.5-1s | `Hello! <break time="0.5s"/> Welcome to...` |
| Between sections | 1-1.5s | `...that's feature one. <break time="1.5s"/> Now let's look at...` |
| Before key point | 0.5s | `The most important thing is <break time="0.5s"/> consistency.` |
| For dramatic effect | 1.5-2s | `And the winner is... <break time="2s"/> you!` |
| After question | 1s | `Sound good? <break time="1s"/> Let's get started.` |
| List items | 0.5s | `First, speed. <break time="0.5s"/> Second, reliability.` |
### Pause Duration Guide
| Duration | Feel | Use For |
|----------|------|---------|
| 0.3-0.5s | Brief breath | Between clauses, light emphasis |
| 0.5-1s | Natural pause | Sentence breaks, transitions |
| 1-1.5s | Deliberate pause | Section changes, setup for key points |
| 1.5-2s | Dramatic | Reveals, important announcements |
| 2s+ | Long pause | Use sparingly, can feel unnatural |
### Examples
```typescript
// Section transitions
const script = `
Welcome to our product overview. <break time="1s"/>
Today I'll cover three key features. <break time="0.5s"/>
First, let's look at the dashboard. <break time="1.5s"/>
As you can see, it's designed for simplicity. <break time="0.5s"/>
Every action is just one click away.
`;
// Building suspense
const announcement = `
We've been working on something special. <break time="1s"/>
After months of development... <break time="1.5s"/>
I'm excited to announce <break time="0.5s"/> our new AI assistant.
`;
// List with rhythm
const features = `
Our platform offers three core benefits. <break time="0.5s"/>
Speed. <break time="0.5s"/>
Reliability. <break time="0.5s"/>
And simplicity. <break time="1s"/>
Let me show you each one.
`;
```
### Consecutive Breaks
Multiple consecutive breaks are combined:
```typescript
// These two breaks:
"Hello <break time=\"1s\"/> <break time=\"0.5s\"/> world"
// Are treated as a single 1.5s pause
```
## Script Structure Templates
### Product Demo (60 seconds, ~150 words)
```typescript
const productDemo = `
Hi, I'm [Name], and I'm excited to show you [Product]. <break time="1s"/>
[Product] helps you [main benefit] in just [timeframe]. <break time="0.5s"/>
Here's how it works. <break time="1s"/>
First, [step 1]. <break time="0.5s"/>
Then, [step 2]. <break time="0.5s"/>
And finally, [step 3]. <break time="1s"/>
What used to take [old time] now takes [new time]. <break time="0.5s"/>
Ready to get started? <break time="0.5s"/>
Visit [website] today.
`;
```
### Tutorial Introduction (90 seconds, ~225 words)
```typescript
const tutorial = `
Welcome to this tutorial on [topic]. <break time="0.5s"/>
I'm [Name], and I'll guide you through everything you need to know. <break time="1s"/>
By the end of this video, you'll be able to [outcome 1], [outcome 2], and [outcome 3]. <break time="1s"/>
Let's start with the basics. <break time="1.5s"/>
[Section 1 content - 2-3 sentences] <break time="1s"/>
Now that you understand [concept], let's move on to [next topic]. <break time="1.5s"/>
[Section 2 content - 2-3 sentences] <break time="1s"/>
And finally, let's cover [last topic]. <break time="1.5s"/>
[Section 3 content - 2-3 sentences] <break time="1s"/>
That's everything you need to get started. <break time="0.5s"/>
If you have questions, leave a comment below. <break time="0.5s"/>
Thanks for watching!
`;
```
### Announcement (30 seconds, ~75 words)
```typescript
const announcement = `
Big news! <break time="0.5s"/>
We're thrilled to announce [announcement]. <break time="1s"/>
This means [benefit 1] and [benefit 2] for all our users. <break time="0.5s"/>
Starting [date], you'll be able to [new capability]. <break time="1s"/>
Head to [location] to learn more. <break time="0.5s"/>
We can't wait to hear what you think!
`;
```
## Writing Tips for AI Voices
### Do
- **Write conversationally** - Read it aloud to check flow
- **Use contractions** - "We're" not "We are", "It's" not "It is"
- **Break up long sentences** - Split at natural pause points
- **Spell out abbreviations** - "API" may sound like "a pee eye"
- **Add pauses for emphasis** - Guide the listener's attention
- **End sections clearly** - Don't trail off mid-thought
### Avoid
- **Jargon without context** - Explain technical terms
- **Long parentheticals** - Move to separate sentences
- **Ambiguous pronunciations** - "read" (present) vs "read" (past)
- **Excessive exclamation marks** - One per script is usually enough
- **Run-on sentences** - Break into digestible chunks
- **Dense information** - Space out facts with pauses
### Pronunciation Hints
For words that might be mispronounced, spell phonetically or add hints:
```typescript
// Technical terms
const script1 = "Our API (A-P-I) handles authentication...";
// Ambiguous words
const script2 = "I read (red) the documentation yesterday...";
// Brand names
const script3 = "Welcome to HeyGen (hey-jen)...";
```
## Multi-Scene Scripts
When splitting scripts across scenes (for different backgrounds or avatars):
```typescript
const multiSceneVideo = {
video_inputs: [
{
// Scene 1: Introduction
character: { type: "avatar", avatar_id: "josh_lite3_20230714", avatar_style: "normal" },
voice: {
type: "text",
input_text: "Welcome to our quarterly update. <break time=\"1s\"/> I'm Josh, and I'll walk you through the highlights.",
voice_id: "voice_id_here",
},
background: { type: "color", value: "#1a1a2e" },
},
{
// Scene 2: Main content (different background)
character: { type: "avatar", avatar_id: "josh_lite3_20230714", avatar_style: "normal" },
voice: {
type: "text",
input_text: "Let's start with revenue. <break time=\"0.5s\"/> We grew 25 percent quarter over quarter. <break time=\"1s\"/> Here's what drove that growth.",
voice_id: "voice_id_here",
},
background: { type: "image", url: "https://..." },
},
// ... more scenes
],
};
```
### Scene Transition Tips
- End each scene with a complete thought
- Start new scenes with brief context
- Maintain consistent tone across scenes
- Use pauses at scene starts to let visuals register
## Testing Your Script
Before generating the full video:
1. **Read aloud** - Time yourself, check for awkward phrasing
2. **Count words** - Verify expected duration
3. **Check break tags** - Ensure proper spacing and syntax
4. **Preview with short clip** - Generate a 10-second test if unsure about pronunciation
```typescript
// Test a small portion first
const testScript = script.split('.').slice(0, 2).join('.') + '.';
const testVideoId = await generateVideo({
video_inputs: [{
character: { type: "avatar", avatar_id: avatarId, avatar_style: "normal" },
voice: { type: "text", input_text: testScript, voice_id: voiceId },
}],
dimension: { width: 1280, height: 720 }, // Lower res for test
});
```
## Voice Speed Adjustment
Adjust delivery speed in the voice configuration:
```typescript
voice: {
type: "text",
input_text: script,
voice_id: "voice_id",
speed: 1.1, // Slightly faster (range: 0.5 - 2.0)
}
```
| Speed | Effect | Use Case |
|-------|--------|----------|
| 0.8-0.9 | Slower, deliberate | Complex topics, older audiences |
| 1.0 | Normal | General use |
| 1.1-1.2 | Slightly faster | Energetic content, younger audiences |
| 1.3+ | Fast | Use sparingly, may reduce clarity |
See [voices.md](voices.md) for full voice configuration options.
.claude/skills/avatar-video/references/templates.md
+433
View File
@@ -0,0 +1,433 @@
---
name: templates
description: Template listing and variable replacement for HeyGen videos
---
# Video Templates
HeyGen templates allow you to create reusable video structures with variable placeholders, enabling personalized video generation at scale.
## Listing Templates
### curl
```bash
curl -X GET "https://api.heygen.com/v2/templates" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface Template {
template_id: string;
name: string;
thumbnail_url: string;
variables: TemplateVariable[];
}
interface TemplateVariable {
name: string;
type: "text" | "image" | "audio";
properties?: {
max_length?: number;
default_value?: string;
};
}
interface TemplatesResponse {
error: null | string;
data: {
templates: Template[];
};
}
async function listTemplates(): Promise<Template[]> {
const response = await fetch("https://api.heygen.com/v2/templates", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const json: TemplatesResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.templates;
}
```
### Python
```python
import requests
import os
def list_templates() -> list:
response = requests.get(
"https://api.heygen.com/v2/templates",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]["templates"]
```
## Response Format
```json
{
"error": null,
"data": {
"templates": [
{
"template_id": "template_abc123",
"name": "Product Announcement",
"thumbnail_url": "https://files.heygen.ai/...",
"variables": [
{
"name": "product_name",
"type": "text",
"properties": {
"max_length": 50
}
},
{
"name": "presenter_script",
"type": "text",
"properties": {
"max_length": 500
}
},
{
"name": "product_image",
"type": "image"
}
]
}
]
}
}
```
## Getting Template Details
### curl
```bash
curl -X GET "https://api.heygen.com/v2/template/{template_id}" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
async function getTemplate(templateId: string): Promise<Template> {
const response = await fetch(
`https://api.heygen.com/v2/template/${templateId}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data;
}
```
## Generating Video from Template
### Request Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `variables` | object | ✓ | Key-value pairs matching template variables |
| `test` | boolean | | Test mode (watermarked, no credits) |
| `title` | string | | Video name for organization |
| `callback_id` | string | | Custom ID for webhook tracking |
| `callback_url` | string | | URL for completion notification |
**Note:** The `variables` object keys must match the template's defined variable names. Check template details to see which variables are defined.
### curl
```bash
curl -X POST "https://api.heygen.com/v2/template/{template_id}/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"test": false,
"variables": {
"product_name": "SuperWidget Pro",
"presenter_script": "Introducing our latest innovation!",
"product_image": "https://example.com/product.jpg"
}
}'
```
### TypeScript
```typescript
interface TemplateGenerateRequest {
variables: Record<string, string>; // Required
test?: boolean;
title?: string;
callback_id?: string;
callback_url?: string;
}
interface TemplateGenerateResponse {
error: null | string;
data: {
video_id: string;
};
}
async function generateFromTemplate(
templateId: string,
variables: Record<string, string>,
test: boolean = false
): Promise<string> {
const response = await fetch(
`https://api.heygen.com/v2/template/${templateId}/generate`,
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({ test, variables }),
}
);
const json: TemplateGenerateResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.video_id;
}
```
### Python
```python
def generate_from_template(template_id: str, variables: dict, test: bool = False) -> str:
response = requests.post(
f"https://api.heygen.com/v2/template/{template_id}/generate",
headers={
"X-Api-Key": os.environ["HEYGEN_API_KEY"],
"Content-Type": "application/json"
},
json={
"test": test,
"variables": variables
}
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]["video_id"]
```
## Variable Types
### Text Variables
For dynamic text content:
```typescript
const variables = {
customer_name: "John Smith",
product_name: "SuperWidget Pro",
price: "$99.99",
cta_text: "Order Now!",
};
```
### Image Variables
For dynamic images (backgrounds, product shots):
```typescript
const variables = {
product_image: "https://example.com/product.jpg",
logo: "https://example.com/logo.png",
background: "https://example.com/bg.jpg",
};
```
### Audio Variables
For custom audio content:
```typescript
const variables = {
background_music: "https://example.com/music.mp3",
custom_voiceover: "https://example.com/voiceover.mp3",
};
```
## Batch Video Generation
Generate multiple personalized videos from a template:
```typescript
interface PersonalizationData {
name: string;
email: string;
company: string;
customMessage: string;
}
async function batchGenerateVideos(
templateId: string,
recipients: PersonalizationData[]
): Promise<string[]> {
const videoIds: string[] = [];
for (const recipient of recipients) {
const variables = {
recipient_name: recipient.name,
company_name: recipient.company,
personalized_message: recipient.customMessage,
};
const videoId = await generateFromTemplate(templateId, variables);
videoIds.push(videoId);
// Rate limiting: add delay between requests
await new Promise((r) => setTimeout(r, 1000));
}
return videoIds;
}
// Usage
const recipients = [
{
name: "John Smith",
email: "john@example.com",
company: "Acme Inc",
customMessage: "Thanks for your interest in our product!",
},
{
name: "Jane Doe",
email: "jane@example.com",
company: "Tech Corp",
customMessage: "We'd love to show you a demo!",
},
];
const videoIds = await batchGenerateVideos("template_abc123", recipients);
```
## Template Validation
Validate variables before generating:
```typescript
function validateTemplateVariables(
template: Template,
variables: Record<string, string>
): { valid: boolean; errors: string[] } {
const errors: string[] = [];
for (const templateVar of template.variables) {
const value = variables[templateVar.name];
// Check if required variable is provided
if (!value) {
errors.push(`Missing required variable: ${templateVar.name}`);
continue;
}
// Check text length limits
if (templateVar.type === "text" && templateVar.properties?.max_length) {
if (value.length > templateVar.properties.max_length) {
errors.push(
`Variable "${templateVar.name}" exceeds max length of ${templateVar.properties.max_length}`
);
}
}
// Validate image URLs
if (templateVar.type === "image") {
try {
new URL(value);
} catch {
errors.push(`Variable "${templateVar.name}" is not a valid URL`);
}
}
}
return {
valid: errors.length === 0,
errors,
};
}
```
## Complete Template Workflow
```typescript
async function createPersonalizedVideo(
templateId: string,
personalization: Record<string, string>
): Promise<string> {
// 1. Get template details
const template = await getTemplate(templateId);
console.log(`Using template: ${template.name}`);
// 2. Validate variables
const validation = validateTemplateVariables(template, personalization);
if (!validation.valid) {
throw new Error(`Validation errors: ${validation.errors.join(", ")}`);
}
// 3. Generate video
console.log("Generating video...");
const videoId = await generateFromTemplate(templateId, personalization);
console.log(`Video ID: ${videoId}`);
// 4. Wait for completion
const videoUrl = await waitForVideo(videoId);
console.log(`Video ready: ${videoUrl}`);
return videoUrl;
}
// Usage
const videoUrl = await createPersonalizedVideo("template_abc123", {
customer_name: "John Smith",
product_name: "SuperWidget Pro",
offer_details: "Get 20% off your first order!",
});
```
## Best Practices
1. **Design for flexibility** - Create templates with generic placeholders
2. **Set reasonable limits** - Define max lengths for text variables
3. **Validate inputs** - Check variable values before generating
4. **Use test mode** - Test with `test: true` to verify before production
5. **Implement rate limiting** - Add delays for batch generation
6. **Cache template data** - Reduce API calls by caching template details
7. **Error handling** - Gracefully handle generation failures
## Use Cases
- **Sales outreach** - Personalized prospect videos
- **Customer onboarding** - Welcome videos with customer name
- **Product updates** - Announcements with dynamic content
- **Training** - Customized training modules
- **Marketing campaigns** - Targeted promotional videos
.claude/skills/avatar-video/references/text-overlays.md
+316
View File
@@ -0,0 +1,316 @@
---
name: text-overlays
description: Adding text overlays with fonts and positioning to HeyGen videos
---
# Text Overlays
Add text overlays to your HeyGen videos for titles, captions, lower thirds, and other on-screen text elements.
## Basic Text Overlay
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Welcome to our presentation!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#1a1a2e",
},
},
],
// Text overlay configuration (if supported in your API tier)
// Note: Availability varies by plan
};
```
## Text Overlay Configuration
Text overlays typically support these properties:
```typescript
interface TextOverlay {
text: string;
x: number; // X position (pixels or percentage)
y: number; // Y position (pixels or percentage)
width?: number; // Text box width
height?: number; // Text box height
font_family?: string;
font_size?: number;
font_color?: string;
background_color?: string;
text_align?: "left" | "center" | "right";
duration?: {
start: number; // Start time in seconds
end: number; // End time in seconds
};
}
```
## Positioning Text
### Coordinate System
- **Origin**: Top-left corner (0, 0)
- **X-axis**: Increases to the right
- **Y-axis**: Increases downward
- **Units**: Typically pixels or percentage of video dimensions
### Common Positions
For a 1920x1080 video:
| Position | X | Y | Description |
|----------|---|---|-------------|
| Top-left | 50 | 50 | Upper left corner |
| Top-center | 960 | 50 | Top center |
| Top-right | 1870 | 50 | Upper right corner |
| Center | 960 | 540 | Dead center |
| Bottom-left | 50 | 1030 | Lower third left |
| Bottom-center | 960 | 1030 | Lower third center |
### Position Helper Function
```typescript
interface Position {
x: number;
y: number;
}
function getTextPosition(
location: "top-left" | "top-center" | "top-right" | "center" | "bottom-left" | "bottom-center" | "bottom-right",
videoWidth: number,
videoHeight: number,
padding: number = 50
): Position {
const positions: Record<string, Position> = {
"top-left": { x: padding, y: padding },
"top-center": { x: videoWidth / 2, y: padding },
"top-right": { x: videoWidth - padding, y: padding },
"center": { x: videoWidth / 2, y: videoHeight / 2 },
"bottom-left": { x: padding, y: videoHeight - padding },
"bottom-center": { x: videoWidth / 2, y: videoHeight - padding },
"bottom-right": { x: videoWidth - padding, y: videoHeight - padding },
};
return positions[location];
}
```
## Font Styling
### Available Font Properties
```typescript
const textStyle = {
font_family: "Arial",
font_size: 48,
font_color: "#FFFFFF",
font_weight: "bold",
background_color: "rgba(0, 0, 0, 0.5)",
text_align: "center",
};
```
### Common Font Families
| Font | Style | Use Case |
|------|-------|----------|
| Arial | Sans-serif | Clean, universal |
| Helvetica | Sans-serif | Modern, professional |
| Times New Roman | Serif | Traditional, formal |
| Georgia | Serif | Elegant, readable |
| Roboto | Sans-serif | Modern, digital |
| Open Sans | Sans-serif | Friendly, accessible |
## Common Text Overlay Patterns
### Title Card
```typescript
const titleOverlay = {
text: "Product Demo",
x: 960,
y: 540,
font_family: "Arial",
font_size: 72,
font_color: "#FFFFFF",
text_align: "center",
duration: {
start: 0,
end: 3,
},
};
```
### Lower Third (Name/Title)
```typescript
const lowerThirdOverlay = {
text: "John Smith\nCEO, Company Inc.",
x: 100,
y: 900,
font_family: "Arial",
font_size: 36,
font_color: "#FFFFFF",
background_color: "rgba(0, 102, 204, 0.9)",
text_align: "left",
duration: {
start: 2,
end: 8,
},
};
```
### Call to Action
```typescript
const ctaOverlay = {
text: "Visit example.com",
x: 960,
y: 1000,
font_family: "Arial",
font_size: 42,
font_color: "#FFD700",
text_align: "center",
duration: {
start: 25,
end: 30,
},
};
```
## Creating Text Overlay Templates
```typescript
interface TextOverlayTemplate {
name: string;
style: Partial<TextOverlay>;
}
const templates: TextOverlayTemplate[] = [
{
name: "title",
style: {
font_family: "Arial",
font_size: 72,
font_color: "#FFFFFF",
text_align: "center",
},
},
{
name: "subtitle",
style: {
font_family: "Arial",
font_size: 42,
font_color: "#CCCCCC",
text_align: "center",
},
},
{
name: "lower-third",
style: {
font_family: "Arial",
font_size: 36,
font_color: "#FFFFFF",
background_color: "rgba(0, 0, 0, 0.7)",
text_align: "left",
},
},
{
name: "caption",
style: {
font_family: "Arial",
font_size: 32,
font_color: "#FFFFFF",
background_color: "rgba(0, 0, 0, 0.5)",
text_align: "center",
},
},
];
function createTextOverlay(
text: string,
templateName: string,
position: Position,
duration?: { start: number; end: number }
): TextOverlay {
const template = templates.find((t) => t.name === templateName);
if (!template) {
throw new Error(`Template "${templateName}" not found`);
}
return {
text,
x: position.x,
y: position.y,
...template.style,
duration,
};
}
```
## Timing Text Overlays
Coordinate text appearance with your script:
```typescript
// Script with timing markers
const script = `
Hello and welcome. [0:00 - 0:03]
Let me show you our features. [0:03 - 0:08]
First, we have analytics. [0:08 - 0:15]
Get started today! [0:15 - 0:20]
`;
// Matching text overlays
const overlays = [
{
text: "Welcome",
duration: { start: 0, end: 3 },
...titleStyle,
},
{
text: "Feature Overview",
duration: { start: 3, end: 8 },
...subtitleStyle,
},
{
text: "Analytics Dashboard",
duration: { start: 8, end: 15 },
...lowerThirdStyle,
},
{
text: "www.example.com",
duration: { start: 15, end: 20 },
...ctaStyle,
},
];
```
## Best Practices
1. **Readability** - Use sufficient contrast between text and background
2. **Size** - Ensure text is large enough to read on mobile devices
3. **Duration** - Give viewers enough time to read (rule of thumb: 3 seconds minimum)
4. **Positioning** - Don't overlap with the avatar's face
5. **Consistency** - Use consistent fonts and styles throughout
6. **Accessibility** - Consider color-blind friendly palettes
## Limitations
- Text overlay support varies by subscription tier
- Some advanced styling options may not be available via API
- Complex animations may require post-production tools
- For auto-generated captions, see [captions.md](captions.md)
.claude/skills/avatar-video/references/video-generation.md
+770
View File
@@ -0,0 +1,770 @@
---
name: video-generation
description: POST /v2/video/generate workflow and multi-scene videos for HeyGen
---
# Video Generation
## Table of Contents
- [Video Output Formats](#video-output-formats)
- [Basic Video Generation](#basic-video-generation)
- [Request Fields](#request-fields)
- [Video Configuration Options](#video-configuration-options)
- [Multi-Scene Videos](#multi-scene-videos)
- [Using Different Character Types](#using-different-character-types)
- [Voice Input Types](#voice-input-types)
- [Complete Workflow Example](#complete-workflow-example)
- [Error Handling](#error-handling)
- [Script Length Limits](#script-length-limits)
- [Adding Pauses to Scripts](#adding-pauses-to-scripts)
- [Test Mode](#test-mode)
- [Production-Ready Workflow](#production-ready-workflow)
- [Transparent Background Videos (WebM)](#transparent-background-videos-webm)
- [Best Practices](#best-practices)
---
The `/v2/video/generate` endpoint is the primary way to create AI avatar videos with HeyGen.
## Video Output Formats
| Endpoint | Format | Use Case |
|----------|--------|----------|
| `/v2/video/generate` | MP4 | **Standard** - videos with background (most common) |
| `/v1/video.webm` | WebM | Transparent background - only when needed |
Use MP4 with background for most cases. WebM is only needed when you want to see content *behind* the avatar (e.g., overlaying avatar on a screen recording).
## Basic Video Generation
### curl
```bash
curl -X POST "https://api.heygen.com/v2/video/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"video_inputs": [
{
"character": {
"type": "avatar",
"avatar_id": "josh_lite3_20230714",
"avatar_style": "normal"
},
"voice": {
"type": "text",
"input_text": "Hello! Welcome to HeyGen.",
"voice_id": "1bd001e7e50f421d891986aad5158bc8"
}
}
],
"dimension": {
"width": 1920,
"height": 1080
}
}'
```
## Request Fields
### Top-Level Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `video_inputs` | array | ✓ | Array of 1-50 video input objects |
| `dimension` | object | | Video dimensions `{width, height}` |
| `title` | string | | Video name for organization |
| `test` | boolean | | Test mode (watermarked, no credits) |
| `caption` | boolean | | Enable auto-captions |
| `callback_id` | string | | Custom ID for webhook tracking |
| `callback_url` | string | | URL for completion notification |
| `folder_id` | string | | Storage folder ID |
### video_inputs[].character Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `type` | string | ✓ | `"avatar"` or `"talking_photo"` |
| `avatar_id` | string | ✓* | Avatar ID (*required when type is "avatar") |
| `talking_photo_id` | string | ✓* | Photo ID (*required when type is "talking_photo") |
| `avatar_style` | string | | `"normal"`, `"closeUp"`, or `"circle"` |
| `scale` | number | | Avatar scale factor |
| `offset` | object | | Position offset `{x, y}` |
### video_inputs[].voice Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `type` | string | ✓ | `"text"`, `"audio"`, or `"silence"` |
| `voice_id` | string | ✓* | Voice ID (*required when type is "text") |
| `input_text` | string | ✓* | Script text (*required when type is "text") |
| `audio_url` | string | ✓* | Audio URL (*required when type is "audio") |
| `duration` | number | ✓* | Duration in seconds (*required when type is "silence") |
| `speed` | number | | Speech speed 0.5-2.0 (default 1.0) |
| `pitch` | number | | Voice pitch -20 to 20 (default 0) |
### video_inputs[].background Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `type` | string | | `"color"`, `"image"`, or `"video"` |
| `value` | string | | Hex color (when type is "color") |
| `url` | string | | Image/video URL (when type is "image"/"video") |
| `fit` | string | | `"cover"` or `"contain"` |
### TypeScript
```typescript
// Required fields have no '?' - optional fields have '?'
interface VideoInput {
character: {
type: "avatar" | "talking_photo"; // Required
avatar_id?: string; // Required when type="avatar"
talking_photo_id?: string; // Required when type="talking_photo"
avatar_style?: "normal" | "closeUp" | "circle";
scale?: number;
offset?: { x: number; y: number };
};
voice: {
type: "text" | "audio" | "silence"; // Required
input_text?: string; // Required when type="text"
voice_id?: string; // Required when type="text"
audio_url?: string; // Required when type="audio"
duration?: number; // Required when type="silence"
speed?: number;
pitch?: number;
};
background?: {
type?: "color" | "image" | "video";
value?: string;
url?: string;
fit?: "cover" | "contain";
};
}
interface VideoGenerateRequest {
video_inputs: VideoInput[]; // Required
dimension?: { width: number; height: number };
test?: boolean;
title?: string;
caption?: boolean;
callback_id?: string;
callback_url?: string;
folder_id?: string;
}
interface VideoGenerateResponse {
error: null | string;
data: {
video_id: string;
};
}
async function generateVideo(config: VideoGenerateRequest): Promise<string> {
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
});
const json: VideoGenerateResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.video_id;
}
```
### Python
```python
import requests
import os
def generate_video(config: dict) -> str:
response = requests.post(
"https://api.heygen.com/v2/video/generate",
headers={
"X-Api-Key": os.environ["HEYGEN_API_KEY"],
"Content-Type": "application/json"
},
json=config
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]["video_id"]
```
## Video Configuration Options
### Full Configuration Example
```typescript
const fullConfig: VideoGenerateRequest = {
// Test mode (no credits consumed, watermarked output)
test: false,
// Video title (for organization)
title: "Product Demo Video",
// Video dimensions
dimension: {
width: 1920,
height: 1080,
},
// Video scenes/inputs
video_inputs: [
{
// Avatar configuration
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
// Voice configuration
voice: {
type: "text",
input_text: "Welcome to our product demonstration!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
speed: 1.0,
pitch: 0,
},
// Background configuration
background: {
type: "color",
value: "#FFFFFF",
},
},
],
};
```
## Multi-Scene Videos
Create videos with multiple scenes:
```typescript
const multiSceneConfig = {
video_inputs: [
// Scene 1: Introduction
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello! Today I'll show you three key features.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#1a1a2e",
},
},
// Scene 2: Feature 1
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "closeUp",
},
voice: {
type: "text",
input_text: "First, let's look at our dashboard.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: "https://example.com/dashboard-bg.jpg",
},
},
// Scene 3: Conclusion
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Thanks for watching! Try it today.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#1a1a2e",
},
},
],
dimension: { width: 1920, height: 1080 },
};
```
## Using Different Character Types
### Avatar
```typescript
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal"
}
}
```
### Talking Photo
```typescript
{
character: {
type: "talking_photo",
talking_photo_id: "your_talking_photo_id"
}
}
```
## Voice Input Types
### Text-to-Speech
```typescript
{
voice: {
type: "text",
input_text: "Your script here",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
speed: 1.0, // 0.5 - 2.0
pitch: 0 // -20 to 20
}
}
```
### Custom Audio
```typescript
{
voice: {
type: "audio",
audio_url: "https://example.com/your-audio.mp3"
}
}
```
## Complete Workflow Example
```typescript
async function createVideo(script: string, avatarId: string, voiceId: string) {
// 1. Generate video
console.log("Starting video generation...");
const videoId = await generateVideo({
video_inputs: [
{
character: {
type: "avatar",
avatar_id: avatarId,
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: voiceId,
},
background: {
type: "color",
value: "#FFFFFF",
},
},
],
dimension: { width: 1920, height: 1080 },
});
console.log(`Video ID: ${videoId}`);
// 2. Poll for completion
console.log("Waiting for video completion...");
const videoUrl = await waitForVideo(videoId);
console.log(`Video ready: ${videoUrl}`);
return videoUrl;
}
// Helper function for polling
async function waitForVideo(videoId: string): Promise<string> {
const maxAttempts = 60;
const pollInterval = 10000; // 10 seconds
for (let i = 0; i < maxAttempts; i++) {
const response = await fetch(
`https://api.heygen.com/v2/videos/${videoId}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
if (data.status === "completed") {
return data.video_url;
} else if (data.status === "failed") {
throw new Error(data.failure_message || "Video generation failed");
}
await new Promise((r) => setTimeout(r, pollInterval));
}
throw new Error("Video generation timed out");
}
```
## Error Handling
```typescript
async function generateVideoSafe(config: VideoGenerateRequest) {
try {
const videoId = await generateVideo(config);
return { success: true, videoId };
} catch (error) {
// Common errors
if (error.message.includes("quota")) {
console.error("Insufficient credits");
} else if (error.message.includes("avatar")) {
console.error("Invalid avatar ID");
} else if (error.message.includes("voice")) {
console.error("Invalid voice ID");
} else if (error.message.includes("script")) {
console.error("Script too long or invalid");
}
return { success: false, error: error.message };
}
}
```
## Script Length Limits
| Tier | Max Characters |
|------|----------------|
| Free | ~500 |
| Creator | ~1,500 |
| Team | ~3,000 |
| Enterprise | ~5,000+ |
## Adding Pauses to Scripts
Use `<break>` tags to add pauses in your script:
```typescript
const script = "Welcome to our demo. <break time=\"1s\"/> Let me show you the features.";
```
**Format:** `<break time="Xs"/>` where X is seconds (e.g., `1s`, `1.5s`, `0.5s`)
**Important:** Break tags must have spaces before and after them.
See [voices.md](voices.md) for detailed break tag documentation.
## Test Mode
Use test mode during development:
```typescript
const config = {
test: true, // Watermarked output, no credits consumed
video_inputs: [...],
};
```
## Production-Ready Workflow
Complete example using avatar's default voice (recommended), proper timeouts, and retry logic:
```typescript
interface VideoGenerationResult {
videoId: string;
videoUrl: string;
duration: number;
avatarId: string;
voiceId: string;
avatarName: string;
}
async function generateAvatarVideo(
script: string,
options: {
avatarId?: string; // Specific avatar, or will pick first available
width?: number;
height?: number;
} = {}
): Promise<VideoGenerationResult> {
const { width = 1920, height = 1080 } = options;
let { avatarId } = options;
// 1. List avatars if no specific one provided
if (!avatarId) {
console.log("Listing available avatars...");
const listResponse = await fetch("https://api.heygen.com/v2/avatars", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const listData = await listResponse.json();
if (!listData.data?.avatars?.length) {
throw new Error("No avatars available");
}
avatarId = listData.data.avatars[0].avatar_id;
}
// 2. Get avatar details including default_voice_id
console.log(`Getting details for avatar: ${avatarId}`);
const detailsResponse = await fetch(
`https://api.heygen.com/v2/avatar/${avatarId}/details`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data: avatar } = await detailsResponse.json();
if (!avatar.default_voice_id) {
throw new Error(`Avatar ${avatar.name} has no default voice - select voice manually`);
}
console.log(`Using avatar: ${avatar.name} with default voice: ${avatar.default_voice_id}`);
// 3. Generate video using avatar's default voice
const videoId = await generateVideo({
video_inputs: [{
character: {
type: "avatar",
avatar_id: avatar.id, // from details response
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: avatar.default_voice_id, // pre-matched default voice
speed: 1.0,
},
background: {
type: "color",
value: "#1a1a2e",
},
}],
dimension: { width, height },
});
console.log(`Video ID: ${videoId}`);
// 3. Wait for completion (20 minute timeout - generation can take 15+ min)
console.log("Waiting for video generation (typically 5-15 minutes, can be longer)...");
const result = await waitForVideo(
videoId,
process.env.HEYGEN_API_KEY!,
(status, elapsed) => {
console.log(` [${Math.round(elapsed / 1000)}s] ${status}`);
},
1200000 // 20 minute timeout for safety
);
return {
videoId,
videoUrl: result.video_url!,
duration: result.duration!,
avatarId: avatar.id,
voiceId: avatar.default_voice_id,
avatarName: avatar.name,
};
}
// Usage - let it pick an avatar automatically
const result = await generateAvatarVideo(
"Hello! Welcome to our product demonstration."
);
console.log(`Video ready: ${result.videoUrl}`);
// Or specify a known avatar_id
const result2 = await generateAvatarVideo(
"Hello! Welcome to our product demonstration.",
{ avatarId: "josh_lite3_20230714" }
);
```
## Transparent Background Videos (WebM)
Use WebM **only when you need transparency** - i.e., when the avatar should be overlaid on other video content and you need to see through to what's behind.
**Don't need WebM for:**
- Avatar with motion graphics/text overlaid ON TOP of avatar
- Picture-in-picture with solid background
- Standard presenter videos
**Do need WebM for:**
- Avatar overlaid on screen recording
- Avatar floating over video background
- True alpha-channel compositing
### WebM Request Fields
**Note:** The WebM endpoint (`/v1/video.webm`) uses a different structure than `/v2/video/generate`.
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `avatar_pose_id` | string | ✓ | Avatar pose ID (from avatar details) |
| `avatar_style` | string | ✓ | `"normal"` or `"closeUp"` only (no circle) |
| `input_text` | string | ✓* | Script text (*required if not using input_audio) |
| `voice_id` | string | ✓* | Voice ID (*required with input_text) |
| `input_audio` | string | ✓* | Audio URL (*required if not using input_text) |
| `dimension` | object | | `{width, height}` (default: 1280x720) |
**Either** (`input_text` + `voice_id`) **OR** `input_audio` must be provided, but not both.
### curl
```bash
curl -X POST "https://api.heygen.com/v1/video.webm" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"avatar_pose_id": "josh_lite3_20230714",
"avatar_style": "normal",
"input_text": "Hello! This video has a transparent background.",
"voice_id": "1bd001e7e50f421d891986aad5158bc8",
"dimension": {
"width": 1920,
"height": 1080
}
}'
```
### TypeScript
```typescript
interface WebMVideoRequest {
avatar_pose_id: string; // Required
avatar_style: "normal" | "closeUp"; // Required (no circle support)
input_text?: string; // Required if not using input_audio
voice_id?: string; // Required with input_text
input_audio?: string; // Required if not using input_text
dimension?: { width: number; height: number };
}
async function generateTransparentVideo(
script: string,
avatarPoseId: string,
voiceId: string
): Promise<string> {
const response = await fetch("https://api.heygen.com/v1/video.webm", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
avatar_pose_id: avatarPoseId, // Required
avatar_style: "normal", // Required: "normal" or "closeUp"
input_text: script, // Required (with voice_id)
voice_id: voiceId, // Required (with input_text)
dimension: { width: 1920, height: 1080 },
}),
});
const { data } = await response.json();
return data.video_id;
}
```
### When to Use WebM vs MP4
| Scenario | Format | Why |
|----------|--------|-----|
| Avatar with overlays on top | **MP4** | Overlays go on top, don't need transparency |
| Standard presenter | **MP4** | Simpler, more compatible |
| Loom-style (avatar over screen recording) | **WebM** + `normal`/`closeUp` | Need transparency, crop to circle in post |
| Avatar floating over video content | **WebM** | Need to see content behind avatar |
**Note:** WebM only supports `normal` and `closeUp` styles. Circle style is not supported for WebM - apply circular masking in your video editor/Remotion instead.
### WebM Example: Loom-Style (Avatar Over Screen Recording)
Generate with `normal` or `closeUp` style (circle not supported for WebM):
```typescript
// Generate avatar with transparent background
const videoId = await fetch("https://api.heygen.com/v1/video.webm", {
method: "POST",
headers: { "X-Api-Key": apiKey, "Content-Type": "application/json" },
body: JSON.stringify({
avatar_pose_id: avatarPoseId, // Required
avatar_style: "closeUp", // Required: "normal" or "closeUp" only
input_text: script, // Required (with voice_id)
voice_id: voiceId, // Required (with input_text)
dimension: { width: 1920, height: 1080 },
}),
}).then(r => r.json()).then(d => d.data.video_id);
```
Apply circular masking in Remotion:
```tsx
import { Video, AbsoluteFill } from "remotion";
export const LoomStyleVideo: React.FC<{
screenRecordingUrl: string;
avatarWebmUrl: string;
}> = ({ screenRecordingUrl, avatarWebmUrl }) => {
return (
<AbsoluteFill>
{/* Screen recording as base layer */}
<Video src={screenRecordingUrl} style={{ width: "100%", height: "100%" }} />
{/* Avatar with circular mask applied in CSS */}
<Video
src={avatarWebmUrl}
style={{
position: "absolute",
bottom: 20,
left: 20,
width: 150,
height: 150,
borderRadius: "50%", // Circular mask
overflow: "hidden",
objectFit: "cover",
}}
/>
</AbsoluteFill>
);
};
```
### Note on Status Polling
WebM videos use the same status endpoint as MP4:
```typescript
// Same polling as regular videos
const status = await getVideoStatus(videoId);
// status.video_url will be a .webm file
```
## Best Practices
1. **Preview avatars before generating** - Download `preview_image_url` so user can see what the avatar looks like before committing to a video (see [avatars.md](avatars.md))
2. **Use avatar's default voice** - Most avatars have a `default_voice_id` that's pre-matched for natural results (see [avatars.md](avatars.md))
2. **Fallback: match gender manually** - If no default voice, ensure avatar and voice genders match (see [voices.md](voices.md))
3. **Validate inputs** - Check avatar and voice IDs before generating
4. **Use test mode** - Test configurations without consuming credits
5. **Set generous timeouts** - Use 15-20 minutes; generation often takes 10-15 min, sometimes longer
6. **Consider async patterns** - For long videos, save video_id and check status later (see [video-status.md](video-status.md))
7. **Handle errors gracefully** - Implement proper error handling
8. **Monitor progress** - Implement polling with progress feedback
9. **Optimize scripts** - Keep scripts concise and natural
10. **Consider dimensions** - Match dimensions to your use case (see [dimensions.md](dimensions.md))
.claude/skills/avatar-video/references/video-status.md
+502
View File
@@ -0,0 +1,502 @@
---
name: video-status
description: Polling patterns, status types, and retrieving download URLs for HeyGen videos
---
# Video Status and Polling
After generating a video, you need to poll for status until the video is complete. HeyGen processes videos asynchronously.
## MCP Tool (Preferred)
If the HeyGen MCP server is connected, use `mcp__heygen__get_video` with the `videoId` parameter. It returns status, video_url, thumbnail_url, duration, title, gif_url, captioned_video_url, and other metadata in a single call.
## Checking Video Status (Direct API)
### curl
```bash
curl -X GET "https://api.heygen.com/v2/videos/YOUR_VIDEO_ID" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface VideoStatusResponse {
error: null | string;
data: {
id: string;
status: "pending" | "processing" | "completed" | "failed";
video_url?: string;
thumbnail_url?: string;
duration?: number;
title?: string;
created_at?: string;
completed_at?: string;
gif_url?: string;
captioned_video_url?: string;
subtitle_url?: string;
folder_id?: string;
output_language?: string;
failure_code?: string;
failure_message?: string;
};
}
async function getVideoStatus(videoId: string): Promise<VideoStatusResponse["data"]> {
const response = await fetch(
`https://api.heygen.com/v2/videos/${videoId}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json: VideoStatusResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data;
}
```
### Python
```python
import requests
import os
def get_video_status(video_id: str) -> dict:
response = requests.get(
f"https://api.heygen.com/v2/videos/{video_id}",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]
```
## Video Status Types
| Status | Description |
|--------|-------------|
| `pending` | Video is queued for processing |
| `processing` | Video is being generated |
| `completed` | Video is ready for download |
| `failed` | Video generation failed |
## Expected Generation Times
Video generation typically takes **5-15 minutes**, but can exceed 20 minutes during peak load or for longer scripts.
| Factor | Impact |
|--------|--------|
| Script length | Longer scripts = significantly longer processing |
| Resolution | 1080p takes longer than 720p |
| Avatar complexity | Some avatars render faster |
| Queue load | Peak hours may cause 15-20+ minute waits |
| Multiple scenes | Each scene adds processing time |
**Recommendations**:
- Set timeout to **15-20 minutes** (900,000-1,200,000 ms) for safety
- For scripts > 2 minutes of speech, expect 15+ minutes
- Consider async patterns (save video_id, check later) for long videos
## Response Format
### Completed Video
```json
{
"error": null,
"data": {
"id": "abc123",
"status": "completed",
"video_url": "https://files.heygen.ai/video/abc123.mp4",
"thumbnail_url": "https://files.heygen.ai/thumbnail/abc123.jpg",
"duration": 45.2,
"title": "My Video",
"created_at": "2024-01-15T10:30:00Z",
"completed_at": "2024-01-15T10:38:00Z",
"gif_url": "https://files.heygen.ai/gif/abc123.gif",
"captioned_video_url": null,
"subtitle_url": null,
"folder_id": null,
"output_language": "en"
}
}
```
### Failed Video
```json
{
"error": null,
"data": {
"id": "abc123",
"status": "failed",
"failure_code": "script_too_long",
"failure_message": "Script too long for selected avatar"
}
}
```
## Polling Implementation
### Basic Polling
```typescript
async function waitForVideo(
videoId: string,
maxWaitMs = 600000, // 10 minutes
pollIntervalMs = 5000 // 5 seconds
): Promise<string> {
const startTime = Date.now();
while (Date.now() - startTime < maxWaitMs) {
const status = await getVideoStatus(videoId);
switch (status.status) {
case "completed":
return status.video_url!;
case "failed":
throw new Error(status.failure_message || "Video generation failed");
case "pending":
case "processing":
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
break;
}
}
throw new Error("Video generation timed out");
}
```
### Polling with Progress Callback
```typescript
type ProgressCallback = (status: string, elapsed: number) => void;
async function waitForVideoWithProgress(
videoId: string,
onProgress?: ProgressCallback,
maxWaitMs = 600000,
pollIntervalMs = 5000
): Promise<string> {
const startTime = Date.now();
while (Date.now() - startTime < maxWaitMs) {
const elapsed = Date.now() - startTime;
const status = await getVideoStatus(videoId);
onProgress?.(status.status, elapsed);
switch (status.status) {
case "completed":
return status.video_url!;
case "failed":
throw new Error(status.failure_message || "Video generation failed");
default:
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
}
}
throw new Error("Video generation timed out");
}
// Usage
const videoUrl = await waitForVideoWithProgress(
videoId,
(status, elapsed) => {
console.log(`Status: ${status}, Elapsed: ${Math.round(elapsed / 1000)}s`);
}
);
```
### Python Polling
```python
import time
from typing import Optional, Callable
def wait_for_video(
video_id: str,
max_wait_seconds: int = 600,
poll_interval: int = 5,
on_progress: Optional[Callable[[str, int], None]] = None
) -> str:
start_time = time.time()
while time.time() - start_time < max_wait_seconds:
elapsed = int(time.time() - start_time)
status_data = get_video_status(video_id)
status = status_data["status"]
if on_progress:
on_progress(status, elapsed)
if status == "completed":
return status_data["video_url"]
elif status == "failed":
raise Exception(status_data.get("failure_message", "Video generation failed"))
time.sleep(poll_interval)
raise Exception("Video generation timed out")
# Usage
def progress_callback(status: str, elapsed: int):
print(f"Status: {status}, Elapsed: {elapsed}s")
video_url = wait_for_video(video_id, on_progress=progress_callback)
```
## Downloading the Video
Once the video is complete, download it. **Important**: The video URL may not be immediately available after status shows "completed". Use retry logic with backoff.
### TypeScript (with retry)
```typescript
import fs from "fs";
import path from "path";
async function downloadVideoWithRetry(
videoUrl: string,
outputPath = "./output/video.mp4",
maxRetries = 5,
initialDelayMs = 2000
): Promise<void> {
let lastError: Error | null = null;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await fetch(videoUrl);
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
const arrayBuffer = await response.arrayBuffer();
fs.writeFileSync(path.resolve(outputPath), Buffer.from(arrayBuffer));
console.log(`Video downloaded to ${outputPath}`);
return;
} catch (error) {
lastError = error as Error;
const delay = initialDelayMs * Math.pow(2, attempt); // Exponential backoff
console.log(`Download attempt ${attempt + 1} failed, retrying in ${delay}ms...`);
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
throw new Error(`Failed to download after ${maxRetries} attempts: ${lastError?.message}`);
}
```
### Python (with retry)
```python
import requests
import time
def download_video_with_retry(
video_url: str,
output_path: str,
max_retries: int = 5,
initial_delay: float = 2.0
) -> None:
last_error = None
for attempt in range(max_retries):
try:
response = requests.get(video_url, stream=True, timeout=60)
response.raise_for_status()
with open(output_path, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
print(f"Video downloaded to {output_path}")
return
except Exception as e:
last_error = e
delay = initial_delay * (2 ** attempt) # Exponential backoff
print(f"Download attempt {attempt + 1} failed, retrying in {delay}s...")
time.sleep(delay)
raise Exception(f"Failed to download after {max_retries} attempts: {last_error}")
```
### Simple Download (no retry)
For quick scripts where you'll retry manually:
```typescript
async function downloadVideo(videoUrl: string, outputPath = "./output/video.mp4") {
const response = await fetch(videoUrl);
if (!response.ok) {
throw new Error(`Failed to download: ${response.status}`);
}
const arrayBuffer = await response.arrayBuffer();
fs.writeFileSync(path.resolve(outputPath), Buffer.from(arrayBuffer));
}
```
## Complete Workflow Example
```typescript
async function generateAndDownloadVideo(config: VideoConfig): Promise<string> {
// 1. Generate video
const generateResponse = await fetch(
"https://api.heygen.com/v2/video/generate",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
}
);
const { data: generateData } = await generateResponse.json();
const videoId = generateData.video_id;
console.log(`Video ID: ${videoId}`);
// 2. Poll for completion
const videoUrl = await waitForVideoWithProgress(
videoId,
(status, elapsed) => {
console.log(`[${Math.round(elapsed / 1000)}s] Status: ${status}`);
}
);
// 3. Download
const outputPath = `./output/${videoId}.mp4`;
await downloadVideo(videoUrl, outputPath);
return outputPath;
}
```
## Resumable Status Checking
For long-running generations, save the video_id and check status later rather than keeping a process waiting.
### Save State After Generation
```typescript
interface PendingVideo {
videoId: string;
createdAt: string;
script: string;
avatarId: string;
voiceId: string;
}
async function startVideoGeneration(config: VideoGenerateRequest): Promise<PendingVideo> {
const videoId = await generateVideo(config);
const pending: PendingVideo = {
videoId,
createdAt: new Date().toISOString(),
script: config.video_inputs[0].voice.input_text!,
avatarId: config.video_inputs[0].character.avatar_id!,
voiceId: config.video_inputs[0].voice.voice_id!,
};
// Save to file for later retrieval
fs.writeFileSync("pending-video.json", JSON.stringify(pending, null, 2));
console.log(`Video generation started. ID: ${videoId}`);
console.log("Check status later with: checkVideoStatus()");
return pending;
}
```
### Check Status Later
```typescript
async function checkVideoStatus(): Promise<void> {
if (!fs.existsSync("pending-video.json")) {
console.log("No pending video found");
return;
}
const pending: PendingVideo = JSON.parse(
fs.readFileSync("pending-video.json", "utf-8")
);
const elapsed = Date.now() - new Date(pending.createdAt).getTime();
console.log(`Checking video ${pending.videoId} (started ${Math.round(elapsed / 60000)} min ago)...`);
const status = await getVideoStatus(pending.videoId);
switch (status.status) {
case "completed":
console.log(`Video ready: ${status.video_url}`);
console.log(`Duration: ${status.duration}s`);
// Clean up pending file
fs.unlinkSync("pending-video.json");
// Save result
fs.writeFileSync("video-result.json", JSON.stringify({
...pending,
videoUrl: status.video_url,
thumbnailUrl: status.thumbnail_url,
duration: status.duration,
title: status.title,
createdAt: status.created_at,
completedAt: status.completed_at,
}, null, 2));
break;
case "failed":
console.error(`Video failed: ${status.failure_message}`);
fs.unlinkSync("pending-video.json");
break;
default:
console.log(`Status: ${status.status} - check again in a few minutes`);
}
}
```
### CLI-Friendly Pattern
```typescript
// generate-video.ts - Start generation and exit
async function main() {
const pending = await startVideoGeneration(config);
console.log(`\nVideo ID saved. Run 'npx tsx check-status.ts' to check progress.`);
process.exit(0); // Exit immediately, don't wait
}
// check-status.ts - Check and optionally wait
async function main() {
const args = process.argv.slice(2);
const shouldWait = args.includes("--wait");
if (shouldWait) {
// Poll until complete (with 20 min timeout)
const result = await waitForVideo(pending.videoId, apiKey, onProgress, 1200000);
console.log(`Done: ${result.video_url}`);
} else {
// Just check once and report
await checkVideoStatus();
}
}
```
## Alternative: Using Webhooks
Instead of polling, you can use webhooks to receive notifications when videos complete. See [webhooks.md](webhooks.md) for details. Webhooks are ideal for production systems where you don't want to maintain polling connections.
## Best Practices
1. **Use exponential backoff** - Increase poll intervals for long-running jobs
2. **Set reasonable timeouts** - Most videos complete within 10 minutes
3. **Handle failures gracefully** - Check error messages for actionable feedback
4. **Consider webhooks** - For production systems, webhooks are more efficient than polling
5. **Cache video URLs** - Downloaded video URLs are valid for a limited time
.claude/skills/avatar-video/references/voices.md
+505
View File
@@ -0,0 +1,505 @@
---
name: voices
description: Listing voices, locales, speed/pitch configuration for HeyGen
---
# HeyGen Voices
HeyGen provides a wide variety of AI voices for different languages, accents, and styles. Voices convert your text script into natural-sounding speech.
## Listing Available Voices
### curl
```bash
curl -X GET "https://api.heygen.com/v2/voices" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface Voice {
voice_id: string;
name: string;
language: string;
gender: "male" | "female";
preview_audio: string;
support_pause: boolean;
emotion_support: boolean;
}
interface VoicesResponse {
error: null | string;
data: {
voices: Voice[];
};
}
async function listVoices(): Promise<Voice[]> {
const response = await fetch("https://api.heygen.com/v2/voices", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const json: VoicesResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.voices;
}
```
### Python
```python
import requests
import os
def list_voices() -> list:
response = requests.get(
"https://api.heygen.com/v2/voices",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]["voices"]
```
## Response Format
```json
{
"error": null,
"data": {
"voices": [
{
"voice_id": "1bd001e7e50f421d891986aad5158bc8",
"name": "Sara",
"language": "English",
"gender": "female",
"preview_audio": "https://files.heygen.ai/...",
"support_pause": true,
"emotion_support": true
},
{
"voice_id": "de8b5d78f2e0485f88d1e9f5c8e7f9a6",
"name": "Paul",
"language": "English",
"gender": "male",
"preview_audio": "https://files.heygen.ai/...",
"support_pause": true,
"emotion_support": false
}
]
}
}
```
## Supported Languages
HeyGen supports many languages including:
| Language | Code | Notes |
|----------|------|-------|
| English (US) | en-US | Multiple voice options |
| English (UK) | en-GB | British accent |
| Spanish | es-ES | Spain Spanish |
| Spanish (Latin) | es-MX | Mexican Spanish |
| French | fr-FR | France French |
| German | de-DE | Standard German |
| Portuguese | pt-BR | Brazilian Portuguese |
| Chinese (Mandarin) | zh-CN | Simplified Chinese |
| Japanese | ja-JP | Standard Japanese |
| Korean | ko-KR | Standard Korean |
| Italian | it-IT | Standard Italian |
| Dutch | nl-NL | Standard Dutch |
| Polish | pl-PL | Standard Polish |
| Arabic | ar-SA | Saudi Arabic |
## Using Voices in Video Generation
### Basic Voice Usage
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello! Welcome to our presentation.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
};
```
### Voice with Speed Adjustment
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "This is spoken at a faster pace.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
speed: 1.2, // 1.0 is normal, range: 0.5 - 2.0
},
},
],
};
```
### Voice with Pitch Adjustment
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "This has a higher pitch.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
pitch: 10, // Range: -20 to 20
},
},
],
};
```
## Adding Pauses with Break Tags
HeyGen supports SSML-style `<break>` tags to add pauses in scripts.
### Break Tag Format
```
<break time="Xs"/>
```
Where `X` is the duration in seconds (e.g., `1s`, `1.5s`, `0.5s`).
### Requirements
| Rule | Example |
|------|---------|
| Use seconds with "s" suffix | `<break time="1.5s"/>` ✓ |
| Must have space before tag | `word <break time="1s"/>` ✓ |
| Must have space after tag | `<break time="1s"/> word` ✓ |
| Self-closing tag | `<break time="1s"/>` ✓ |
**Incorrect:** `word<break time="1s"/>word` (no spaces)
**Correct:** `word <break time="1s"/> word`
### Examples
```typescript
// Single pause
const script1 = "Hello and welcome. <break time=\"1s\"/> Let me introduce our product.";
// Multiple pauses
const script2 = "First point. <break time=\"1.5s\"/> Second point. <break time=\"1s\"/> Third point.";
// Pause at start (dramatic opening)
const script3 = "<break time=\"0.5s\"/> Welcome to our presentation.";
// Longer pause for emphasis
const script4 = "And the winner is... <break time=\"2s\"/> You!";
```
### Full Example
```typescript
const scriptWithPauses = `
Welcome to our product demo. <break time="1s"/>
Today I'll show you three key features. <break time="0.5s"/>
First, let's look at the dashboard. <break time="1.5s"/>
As you can see, it's incredibly intuitive.
`;
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: scriptWithPauses,
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
};
```
### Consecutive Breaks
Multiple consecutive break tags are automatically combined:
```typescript
// These two breaks:
"Hello <break time=\"1s\"/> <break time=\"0.5s\"/> world"
// Are treated as a single 1.5s pause
```
### Best Practices
1. **Use for emphasis** - Add pauses before important points
2. **Keep pauses reasonable** - 0.5s to 2s is typical; longer feels unnatural
3. **Match natural speech** - Add pauses where a human would breathe or pause
4. **Test the output** - Listen to generated audio to verify timing feels right
## Using Custom Audio Instead of TTS
Instead of text-to-speech, you can provide your own audio:
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "audio",
audio_url: "https://example.com/my-audio.mp3",
},
},
],
};
```
## Filtering Voices
### By Language
```typescript
function filterByLanguage(voices: Voice[], language: string): Voice[] {
return voices.filter((v) =>
v.language.toLowerCase().includes(language.toLowerCase())
);
}
const englishVoices = filterByLanguage(voices, "english");
const spanishVoices = filterByLanguage(voices, "spanish");
```
### By Gender
```typescript
function filterByGender(voices: Voice[], gender: "male" | "female"): Voice[] {
return voices.filter((v) => v.gender === gender);
}
const femaleVoices = filterByGender(voices, "female");
```
### By Features
```typescript
function filterByFeatures(
voices: Voice[],
options: { supportPause?: boolean; emotionSupport?: boolean }
): Voice[] {
return voices.filter((v) => {
if (options.supportPause !== undefined && v.support_pause !== options.supportPause) {
return false;
}
if (options.emotionSupport !== undefined && v.emotion_support !== options.emotionSupport) {
return false;
}
return true;
});
}
const expressiveVoices = filterByFeatures(voices, { emotionSupport: true });
```
## Voice Selection Helper
```typescript
interface VoiceSelectionCriteria {
language?: string;
gender?: "male" | "female";
supportPause?: boolean;
emotionSupport?: boolean;
}
async function findVoice(criteria: VoiceSelectionCriteria): Promise<Voice | null> {
const voices = await listVoices();
const filtered = voices.filter((v) => {
if (criteria.language && !v.language.toLowerCase().includes(criteria.language.toLowerCase())) {
return false;
}
if (criteria.gender && v.gender !== criteria.gender) {
return false;
}
if (criteria.supportPause !== undefined && v.support_pause !== criteria.supportPause) {
return false;
}
if (criteria.emotionSupport !== undefined && v.emotion_support !== criteria.emotionSupport) {
return false;
}
return true;
});
return filtered[0] || null;
}
// Usage
const voice = await findVoice({
language: "english",
gender: "female",
emotionSupport: true,
});
```
## Multi-Language Videos
Create videos with different languages per scene:
```typescript
const multiLanguageConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello! Welcome to our global product launch.",
voice_id: "english_voice_id",
},
},
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hola! Bienvenidos al lanzamiento global de nuestro producto.",
voice_id: "spanish_voice_id",
},
},
],
};
```
## Matching Voice to Avatar
### Recommended: Use Avatar's Default Voice
Many avatars have a `default_voice_id` that's pre-matched. **This is the best approach.**
```typescript
// Using v2 API to get avatar with default voice
const response = await fetch(
"https://api.heygen.com/v2/avatar_group.list?include_public=true",
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
// Find avatar with a default voice
const avatar = data.avatar_group_list.find((a: any) => a.default_voice_id);
if (avatar) {
const videoConfig = {
video_inputs: [{
character: { type: "avatar", avatar_id: avatar.id },
voice: {
type: "text",
input_text: script,
voice_id: avatar.default_voice_id, // Pre-matched voice
},
}],
};
}
```
See [avatars.md](avatars.md) for complete examples.
### Fallback: Match Gender Manually
If avatar has no default voice, match genders manually:
```typescript
interface AvatarVoicePair {
avatarId: string;
voiceId: string;
gender: "male" | "female";
}
async function findMatchingAvatarAndVoice(
preferredGender?: "male" | "female"
): Promise<AvatarVoicePair> {
const [avatars, voices] = await Promise.all([
listAvatars(),
listVoices(),
]);
// Default to male if no preference
const gender = preferredGender || "male";
// Find avatar with matching gender
const avatar = avatars.find((a) => a.gender === gender);
if (!avatar) {
throw new Error(`No ${gender} avatar available`);
}
// Find voice with matching gender AND language
const voice = voices.find(
(v) => v.gender === gender && v.language.toLowerCase().includes("english")
);
if (!voice) {
throw new Error(`No ${gender} English voice available`);
}
return {
avatarId: avatar.avatar_id,
voiceId: voice.voice_id,
gender,
};
}
```
## Best Practices
1. **Match voice gender to avatar** - Always pair male voices with male avatars, female with female
2. **Match voice to content** - Use professional voices for business content
3. **Test voice previews** - Listen to preview audio before selecting
4. **Consider locale** - Match voice accent to target audience
5. **Use natural pacing** - Adjust speed for clarity, typically 0.9-1.1x
6. **Add pauses** - Use SSML breaks for more natural speech flow
7. **Validate availability** - Always verify voice_id exists before using
.claude/skills/avatar-video/references/webhooks.md
+393
View File
@@ -0,0 +1,393 @@
---
name: webhooks
description: Registering webhook endpoints and event types for HeyGen
---
# Webhooks
Webhooks allow HeyGen to notify your application when events occur, such as video completion. This is more efficient than polling for status updates.
## Overview
Instead of repeatedly checking video status, webhooks push notifications to your server when:
- Video generation completes
- Video generation fails
- Translation completes
- Avatar training completes
- Other async operations finish
## Setting Up a Webhook Endpoint
Your webhook endpoint should:
1. Accept POST requests
2. Return 200 status quickly
3. Handle events asynchronously
### Express.js Example
```typescript
import express from "express";
import crypto from "crypto";
const app = express();
app.use(express.json());
// Webhook endpoint
app.post("/webhook/heygen", async (req, res) => {
// Acknowledge receipt immediately
res.status(200).send("OK");
// Process event asynchronously
processWebhookEvent(req.body).catch(console.error);
});
async function processWebhookEvent(event: HeyGenWebhookEvent) {
console.log(`Received event: ${event.event_type}`);
switch (event.event_type) {
case "avatar_video.success":
await handleVideoSuccess(event);
break;
case "avatar_video.fail":
await handleVideoFailure(event);
break;
case "video_translate.success":
await handleTranslationSuccess(event);
break;
default:
console.log(`Unknown event type: ${event.event_type}`);
}
}
app.listen(3000, () => {
console.log("Webhook server running on port 3000");
});
```
### Python Flask Example
```python
from flask import Flask, request, jsonify
import threading
app = Flask(__name__)
@app.route("/webhook/heygen", methods=["POST"])
def heygen_webhook():
event = request.json
# Acknowledge immediately
response = jsonify({"status": "received"})
# Process asynchronously
thread = threading.Thread(
target=process_webhook_event,
args=(event,)
)
thread.start()
return response, 200
def process_webhook_event(event):
event_type = event.get("event_type")
print(f"Received event: {event_type}")
if event_type == "avatar_video.success":
handle_video_success(event)
elif event_type == "avatar_video.fail":
handle_video_failure(event)
elif event_type == "video_translate.success":
handle_translation_success(event)
if __name__ == "__main__":
app.run(port=3000)
```
## Webhook Event Types
| Event Type | Description |
|------------|-------------|
| `avatar_video.success` | Video generation completed |
| `avatar_video.fail` | Video generation failed |
| `video_translate.success` | Translation completed |
| `video_translate.fail` | Translation failed |
| `instant_avatar.success` | Instant avatar created |
| `instant_avatar.fail` | Instant avatar creation failed |
## Event Payload Structure
### Video Success Event
```typescript
interface VideoSuccessEvent {
event_type: "avatar_video.success";
event_data: {
video_id: string;
video_url: string;
thumbnail_url: string;
duration: number;
callback_id?: string;
};
}
```
```json
{
"event_type": "avatar_video.success",
"event_data": {
"video_id": "abc123",
"video_url": "https://files.heygen.ai/video/abc123.mp4",
"thumbnail_url": "https://files.heygen.ai/thumbnail/abc123.jpg",
"duration": 45.2,
"callback_id": "your_custom_id"
}
}
```
### Video Failure Event
```typescript
interface VideoFailureEvent {
event_type: "avatar_video.fail";
event_data: {
video_id: string;
error: string;
callback_id?: string;
};
}
```
```json
{
"event_type": "avatar_video.fail",
"event_data": {
"video_id": "abc123",
"error": "Script too long for selected avatar",
"callback_id": "your_custom_id"
}
}
```
## Registering a Webhook URL
Configure your webhook URL through the HeyGen dashboard or API:
### Request Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `url` | string | ✓ | Your webhook endpoint URL |
| `events` | array | ✓ | Event types to subscribe to |
| `secret` | string | | Shared secret for signature verification |
### Via API
```bash
curl -X POST "https://api.heygen.com/v1/webhook/endpoint.add" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://your-domain.com/webhook/heygen",
"events": ["avatar_video.success", "avatar_video.fail"]
}'
```
### TypeScript
```typescript
interface WebhookConfig {
url: string; // Required
events: string[]; // Required
secret?: string;
}
async function registerWebhook(config: WebhookConfig): Promise<void> {
const response = await fetch("https://api.heygen.com/v1/webhook/endpoint.add", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
});
const json = await response.json();
if (json.error) {
throw new Error(json.error);
}
}
```
## Using Callback IDs
Track which video triggered a webhook with callback IDs:
### Include Callback ID in Video Generation
```typescript
const videoConfig = {
video_inputs: [...],
callback_id: "order_12345", // Your custom identifier
};
```
### Handle in Webhook
```typescript
async function handleVideoSuccess(event: VideoSuccessEvent) {
const { video_id, video_url, callback_id } = event.event_data;
if (callback_id) {
// Look up your original request
const order = await getOrderByCallbackId(callback_id);
await updateOrderWithVideo(order.id, video_url);
}
}
```
## Webhook Security
### Verify Webhook Signatures
If HeyGen provides signature verification:
```typescript
import crypto from "crypto";
function verifyWebhookSignature(
payload: string,
signature: string,
secret: string
): boolean {
const expectedSignature = crypto
.createHmac("sha256", secret)
.update(payload)
.digest("hex");
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expectedSignature)
);
}
// In your webhook handler
app.post("/webhook/heygen", (req, res) => {
const signature = req.headers["x-heygen-signature"] as string;
const payload = JSON.stringify(req.body);
if (!verifyWebhookSignature(payload, signature, WEBHOOK_SECRET)) {
return res.status(401).send("Invalid signature");
}
// Process event...
});
```
### Validate Event Origin
```typescript
function isValidHeygenEvent(event: any): boolean {
// Check required fields
if (!event.event_type || !event.event_data) {
return false;
}
// Check event type is known
const validEventTypes = [
"avatar_video.success",
"avatar_video.fail",
"video_translate.success",
"video_translate.fail",
];
return validEventTypes.includes(event.event_type);
}
```
## Handling Webhook Failures
Implement retry logic and error handling:
```typescript
async function processWebhookEvent(event: HeyGenWebhookEvent) {
const maxRetries = 3;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await handleEvent(event);
return;
} catch (error) {
console.error(`Attempt ${attempt} failed:`, error);
if (attempt < maxRetries) {
// Exponential backoff
await new Promise((r) => setTimeout(r, Math.pow(2, attempt) * 1000));
}
}
}
// Store failed event for manual review
await storeFailedEvent(event);
}
```
## Webhook vs Polling Comparison
| Aspect | Webhook | Polling |
|--------|---------|---------|
| Latency | Immediate | Depends on interval |
| Efficiency | High (push) | Low (repeated requests) |
| Complexity | Requires endpoint | Simpler to implement |
| Reliability | Needs retry handling | Guaranteed delivery |
| Cost | Lower API usage | Higher API usage |
## Testing Webhooks
### Local Development with ngrok
```bash
# Start ngrok tunnel
ngrok http 3000
# Use ngrok URL as webhook endpoint
# https://abc123.ngrok.io/webhook/heygen
```
### Webhook Testing Tool
```typescript
// Test webhook locally
async function simulateWebhook(event: HeyGenWebhookEvent) {
const response = await fetch("http://localhost:3000/webhook/heygen", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(event),
});
console.log(`Response: ${response.status}`);
}
// Simulate success event
await simulateWebhook({
event_type: "avatar_video.success",
event_data: {
video_id: "test_123",
video_url: "https://example.com/test.mp4",
thumbnail_url: "https://example.com/test.jpg",
duration: 30,
callback_id: "test_callback",
},
});
```
## Best Practices
1. **Respond quickly** - Return 200 within 5 seconds, process async
2. **Handle duplicates** - Same event may be sent multiple times
3. **Implement retries** - Handle temporary processing failures
4. **Log everything** - Store webhook payloads for debugging
5. **Use callback IDs** - Track requests through the system
6. **Secure endpoints** - Verify signatures, use HTTPS
7. **Monitor health** - Track webhook success rates
8. **Queue processing** - Use job queues for heavy processing
.claude/skills/beautiful-mermaid/SKILL.md
+171
View File
@@ -0,0 +1,171 @@
---
name: beautiful-mermaid
description: Render Mermaid diagrams as SVG and PNG using the Beautiful Mermaid library. Use when the user asks to render a Mermaid diagram.
---
# Beautiful Mermaid Diagram Rendering
Render Mermaid diagrams as SVG and PNG images using the Beautiful Mermaid library.
## Dependencies
This skill requires the `agent-browser` skill for PNG rendering. Load it before proceeding with PNG capture.
## Supported Diagram Types
- **Flowchart** - Process flows, decision trees, CI/CD pipelines
- **Sequence** - API calls, OAuth flows, database transactions
- **State** - State machines, connection lifecycles
- **Class** - UML class diagrams, design patterns
- **Entity-Relationship** - Database schemas, data models
## Available Themes
Default, Dracula, Solarized, Zinc Dark, Tokyo Night, Tokyo Night Storm, Tokyo Night Light, Catppuccin Latte, Nord, Nord Light, GitHub Dark, GitHub Light, One Dark.
If no theme is specified, use `default`.
## Common Syntax Patterns
### Flowchart Edge Labels
Use pipe syntax for edge labels:
```mermaid
A -->|label| B
A ---|label| B
```
Avoid space-dash syntax which can cause incomplete renders:
```mermaid
A -- label --> B # May cause issues
```
### Node Labels with Special Characters
Wrap labels containing special characters in quotes:
```mermaid
A["Label with (parens)"]
B["Label with / slash"]
```
## Workflow
### Step 1: Generate or Validate Mermaid Code
If the user provides a description rather than code, generate valid Mermaid syntax. Consult `references/mermaid-syntax.md` for full syntax details.
### Step 2: Render SVG
Run the rendering script to produce an SVG file:
```bash
bun run scripts/render.ts --code "graph TD; A-->B" --output diagram --theme default
```
Or from a file:
```bash
bun run scripts/render.ts --input diagram.mmd --output diagram --theme tokyo-night
```
Alternative runtimes:
```bash
npx tsx scripts/render.ts --code "..." --output diagram
deno run --allow-read --allow-write --allow-net scripts/render.ts --code "..." --output diagram
```
This produces `<output>.svg` in the current working directory.
### Step 3: Create HTML Wrapper
Run the HTML wrapper script to prepare for screenshot:
```bash
bun run scripts/create-html.ts --svg diagram.svg --output diagram.html
```
This creates a minimal HTML file that displays the SVG with proper padding and background.
### Step 4: Capture High-Resolution PNG with agent-browser
Use the agent-browser CLI to capture a high-quality screenshot. Refer to the `agent-browser` skill for full CLI documentation.
```bash
# Set 4K viewport for high-resolution capture
agent-browser set viewport 3840 2160
# Open the HTML wrapper
agent-browser open "file://$(pwd)/diagram.html"
# Wait for render to complete
agent-browser wait 1000
# Capture full-page screenshot
agent-browser screenshot --full diagram.png
# Close browser
agent-browser close
```
For even higher resolution on complex diagrams, increase the viewport further or use the `--padding` option when creating the HTML wrapper to give the diagram more space.
### Step 5: Clean Up Intermediary Files
After rendering, remove all intermediary files. Only the final `.svg` and `.png` should remain.
Files to clean up:
- The HTML wrapper file (e.g., `diagram.html`)
- Any temporary `.mmd` files created to hold diagram code
- Any other files created during the rendering process
```bash
rm diagram.html
```
If a temporary `.mmd` file was created, remove it as well.
## Output
Both outputs are always produced:
- **SVG**: Vector format, infinitely scalable, small file size
- **PNG**: High-resolution raster, captured at 4K (3840×2160) viewport with minimum 1200px diagram width
Files are saved to the current working directory unless the user explicitly specifies a different path.
## Theme Selection Guide
| Theme | Background | Best For |
|-------|------------|----------|
| default | Light grey | General use |
| dracula | Dark purple | Dark mode preference |
| tokyo-night | Dark blue | Modern dark aesthetic |
| tokyo-night-storm | Darker blue | Higher contrast |
| nord | Dark arctic | Muted, calm visuals |
| nord-light | Light arctic | Light mode with soft tones |
| github-dark | GitHub dark | Matches GitHub UI |
| github-light | GitHub light | Matches GitHub UI |
| catppuccin-latte | Warm light | Soft pastel aesthetic |
| solarized | Tan/cream | Solarized colour scheme |
| one-dark | Atom dark | Atom editor aesthetic |
| zinc-dark | Neutral dark | Minimal, no colour bias |
## Troubleshooting
### Theme not applied
Check the render script output for the `bg` and `fg` values, or inspect the SVG's opening tag for `--bg` and `--fg` CSS custom properties.
### Diagram appears cut off or incomplete
- Check edge label syntax — use `-->|label|` pipe notation, not `-- label -->`
- Verify all node IDs are unique
- Check for unclosed brackets in node labels
### Render produces empty or malformed SVG
- Validate Mermaid syntax at https://mermaid.live before rendering
- Check for special characters that need escaping (wrap in quotes)
- Ensure flowchart direction is specified (`graph TD`, `graph LR`, etc.)
.claude/skills/beautiful-mermaid/references/mermaid-syntax.md
+235
View File
@@ -0,0 +1,235 @@
# Mermaid Syntax Reference
Quick reference for generating valid Mermaid diagram code.
## Flowchart
```mermaid
graph TD
A[Start] --> B{Decision}
B -->|Yes| C[Action 1]
B -->|No| D[Action 2]
C --> E[End]
D --> E
```
### Direction
- `TD` / `TB` - Top to bottom
- `BT` - Bottom to top
- `LR` - Left to right
- `RL` - Right to left
### Node Shapes
- `A[Text]` - Rectangle
- `A(Text)` - Rounded rectangle
- `A([Text])` - Stadium/pill
- `A[[Text]]` - Subroutine
- `A[(Text)]` - Cylinder (database)
- `A((Text))` - Circle
- `A>Text]` - Asymmetric
- `A{Text}` - Diamond (decision)
- `A{{Text}}` - Hexagon
- `A[/Text/]` - Parallelogram
- `A[\Text\]` - Parallelogram alt
- `A[/Text\]` - Trapezoid
- `A[\Text/]` - Trapezoid alt
### Edge Styles
- `A --> B` - Arrow
- `A --- B` - Line
- `A -.-> B` - Dotted arrow
- `A ==> B` - Thick arrow
- `A -->|text| B` - Arrow with label (preferred)
- `A ---|text| B` - Line with label (preferred)
**Important**: Always use pipe syntax `-->|label|` for edge labels. The space-dash syntax `-- label -->` can cause incomplete renders.
### Subgraphs
```mermaid
graph TD
subgraph Group1 [Label]
A --> B
end
subgraph Group2
C --> D
end
B --> C
```
## Sequence Diagram
```mermaid
sequenceDiagram
participant A as Alice
participant B as Bob
A->>B: Hello
B-->>A: Hi there
A->>+B: Start process
B-->>-A: Done
```
### Arrow Types
- `->>` - Solid arrow
- `-->>` - Dashed arrow
- `-x` - Solid with x
- `--x` - Dashed with x
- `-)` - Solid open arrow
- `--)` - Dashed open arrow
### Activations
- `+` after arrow activates participant
- `-` after arrow deactivates participant
### Notes and Boxes
```mermaid
sequenceDiagram
Note over A,B: Shared note
Note right of A: Side note
rect rgb(200, 220, 255)
A->>B: In a box
end
```
### Loops and Conditionals
```mermaid
sequenceDiagram
loop Every minute
A->>B: Ping
end
alt Success
B-->>A: Pong
else Failure
B-->>A: Error
end
opt Optional
A->>B: Extra step
end
```
## State Diagram
```mermaid
stateDiagram-v2
[*] --> Idle
Idle --> Processing : start
Processing --> Done : complete
Processing --> Error : fail
Error --> Idle : reset
Done --> [*]
```
### Composite States
```mermaid
stateDiagram-v2
state Active {
[*] --> Running
Running --> Paused : pause
Paused --> Running : resume
}
Idle --> Active : activate
Active --> Idle : deactivate
```
### Notes
```mermaid
stateDiagram-v2
State1 : Description here
note right of State1
Additional info
end note
```
## Class Diagram
```mermaid
classDiagram
class Animal {
+String name
+int age
+makeSound() void
}
class Dog {
+bark() void
}
Animal <|-- Dog : extends
```
### Relationships
- `<|--` - Inheritance
- `*--` - Composition
- `o--` - Aggregation
- `-->` - Association
- `--` - Link (solid)
- `..>` - Dependency
- `..|>` - Realisation
- `..` - Link (dashed)
### Cardinality
```mermaid
classDiagram
Customer "1" --> "*" Order
Order "1" --> "1..*" LineItem
```
### Visibility
- `+` Public
- `-` Private
- `#` Protected
- `~` Package/Internal
## Entity-Relationship Diagram
```mermaid
erDiagram
CUSTOMER ||--o{ ORDER : places
ORDER ||--|{ LINE-ITEM : contains
PRODUCT }|..|{ LINE-ITEM : "ordered in"
```
### Relationship Types
- `||` - Exactly one
- `|{` - One or more
- `o{` - Zero or more
- `o|` - Zero or one
### Identifying vs Non-identifying
- `--` - Identifying (solid)
- `..` - Non-identifying (dashed)
### Attributes
```mermaid
erDiagram
CUSTOMER {
string id PK
string name
string email UK
}
ORDER {
int id PK
string customer_id FK
date created_at
}
```
## Styling
### CSS Classes
```mermaid
graph TD
A:::highlight --> B
classDef highlight fill:#f96,stroke:#333
```
### Inline Styles
```mermaid
graph TD
A --> B
style A fill:#bbf,stroke:#333
```
## Tips
1. **Escape special characters**: Use quotes for labels with special chars: `A["Label with (parens)"]`
2. **Multi-line labels**: Use `<br/>` for line breaks
3. **Comments**: Use `%%` for comments that won't render
4. **IDs vs Labels**: Node IDs should be simple, labels can be complex: `node1["Complex Label Here"]`
.claude/skills/beautiful-mermaid/scripts/create-html.ts
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env -S npx tsx
/**
* Create an HTML wrapper for an SVG to enable high-quality PNG capture
*
* Usage:
* bun run create-html.ts --svg diagram.svg --output diagram.html
* bun run create-html.ts --svg diagram.svg --output diagram.html --padding 40
*
* Runtimes:
* bun run create-html.ts ...
* npx tsx create-html.ts ...
* deno run --allow-read --allow-write create-html.ts ...
*/
import { readFileSync, writeFileSync, existsSync } from "node:fs";
import { resolve, basename } from "node:path";
interface Args {
svg: string;
output: string;
padding: number;
background?: string;
}
function parseArgs(): Args {
const args = process.argv.slice(2);
const result: Partial<Args> = { padding: 40 };
for (let i = 0; i < args.length; i++) {
const arg = args[i];
const next = args[i + 1];
switch (arg) {
case "--svg":
case "-s":
result.svg = next;
i++;
break;
case "--output":
case "-o":
result.output = next;
i++;
break;
case "--padding":
case "-p":
result.padding = parseInt(next, 10) || 40;
i++;
break;
case "--background":
case "-b":
result.background = next;
i++;
break;
case "--help":
case "-h":
printHelp();
process.exit(0);
}
}
if (!result.svg) {
console.error("Error: --svg is required");
printHelp();
process.exit(1);
}
if (!result.output) {
console.error("Error: --output is required");
printHelp();
process.exit(1);
}
return result as Args;
}
function printHelp(): void {
console.log(`
SVG to HTML Wrapper
Creates a minimal HTML file for screenshot capture of SVG diagrams.
Usage:
create-html.ts --svg <file.svg> --output <file.html> [options]
Options:
-s, --svg <file> Input SVG file
-o, --output <file> Output HTML file
-p, --padding <pixels> Padding around SVG (default: 40)
-b, --background <color> Background colour (auto-detected from SVG)
-h, --help Show this help
Examples:
create-html.ts --svg diagram.svg --output diagram.html
create-html.ts --svg diagram.svg --output diagram.html --padding 60
create-html.ts --svg diagram.svg --output diagram.html --background "#1a1b26"
`);
}
function extractBackgroundFromSvg(svgContent: string): string | null {
// Try to extract background from SVG style or rect
const bgMatch = svgContent.match(/background(?:-color)?:\s*([^;"\s]+)/i);
if (bgMatch) return bgMatch[1];
// Check for a background rect
const rectMatch = svgContent.match(
/<rect[^>]*fill="([^"]+)"[^>]*(?:width="100%"|height="100%")/i
);
if (rectMatch) return rectMatch[1];
// Check style attribute on svg element
const svgStyleMatch = svgContent.match(
/<svg[^>]*style="[^"]*background(?:-color)?:\s*([^;"\s]+)/i
);
if (svgStyleMatch) return svgStyleMatch[1];
return null;
}
function main(): void {
const args = parseArgs();
const svgPath = resolve(args.svg);
if (!existsSync(svgPath)) {
console.error(`SVG file not found: ${svgPath}`);
process.exit(1);
}
const svgContent = readFileSync(svgPath, "utf-8");
// Determine background colour
const background =
args.background ?? extractBackgroundFromSvg(svgContent) ?? "#ffffff";
// Create HTML wrapper optimised for high-resolution screenshot
// SVG renders at natural size with generous padding, no constraints
const html = `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>${basename(args.svg, ".svg")}</title>
<style>
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
html, body {
background: ${background};
}
.container {
padding: ${args.padding}px;
display: inline-block;
background: ${background};
}
.container svg {
display: block;
min-width: 1200px;
height: auto;
}
</style>
</head>
<body>
<div class="container">
${svgContent}
</div>
</body>
</html>`;
const outputPath = resolve(args.output);
writeFileSync(outputPath, html, "utf-8");
console.log(`HTML wrapper written to: ${outputPath}`);
console.log(`Background colour: ${background}`);
}
main();
.claude/skills/beautiful-mermaid/scripts/render.ts
+221
View File
@@ -0,0 +1,221 @@
#!/usr/bin/env -S npx tsx
/**
* Render a Mermaid diagram to SVG using Beautiful Mermaid
*
* Usage:
* bun run render.ts --input diagram.mmd --output diagram --theme tokyo-night
* bun run render.ts --code "graph TD; A-->B" --output diagram
*
* Runtimes:
* bun run render.ts ...
* npx tsx render.ts ...
* deno run --allow-read --allow-write --allow-net render.ts ...
*
* Output:
* Produces <output>.svg
*/
import { readFileSync, writeFileSync, existsSync } from "node:fs";
import { resolve } from "node:path";
const THEMES = [
"default",
"dracula",
"solarized",
"zinc-dark",
"tokyo-night",
"tokyo-night-storm",
"tokyo-night-light",
"catppuccin-latte",
"nord",
"nord-light",
"github-dark",
"github-light",
"one-dark",
] as const;
type Theme = (typeof THEMES)[number];
interface Args {
input?: string;
code?: string;
output: string;
theme: Theme;
}
function parseArgs(): Args {
const args = process.argv.slice(2);
const result: Partial<Args> = { theme: "default" };
for (let i = 0; i < args.length; i++) {
const arg = args[i];
const next = args[i + 1];
switch (arg) {
case "--input":
case "-i":
result.input = next;
i++;
break;
case "--code":
case "-c":
result.code = next;
i++;
break;
case "--output":
case "-o":
result.output = next;
i++;
break;
case "--theme":
case "-t":
if (next && THEMES.includes(next as Theme)) {
result.theme = next as Theme;
} else {
console.error(`Invalid theme: ${next}`);
console.error(`Available themes: ${THEMES.join(", ")}`);
process.exit(1);
}
i++;
break;
case "--help":
case "-h":
printHelp();
process.exit(0);
}
}
if (!result.input && !result.code) {
console.error("Error: Either --input or --code is required");
printHelp();
process.exit(1);
}
if (!result.output) {
console.error("Error: --output is required");
printHelp();
process.exit(1);
}
return result as Args;
}
function printHelp(): void {
console.log(`
Beautiful Mermaid Renderer
Renders Mermaid diagrams to SVG.
Usage:
render.ts --input <file.mmd> --output <basename> [--theme <theme>]
render.ts --code "<mermaid code>" --output <basename> [--theme <theme>]
Options:
-i, --input <file> Input Mermaid file (.mmd)
-c, --code <string> Mermaid code as string
-o, --output <name> Output base name (without extension)
-t, --theme <theme> Theme name (default: default)
-h, --help Show this help
Available themes:
${THEMES.join(", ")}
Output:
Produces <output>.svg
Examples:
render.ts -i diagram.mmd -o diagram -t tokyo-night
render.ts -c "graph TD; A-->B" -o simple
`);
}
function detectRuntime(): "bun" | "deno" | "node" {
if (typeof (globalThis as any).Bun !== "undefined") return "bun";
if (typeof (globalThis as any).Deno !== "undefined") return "deno";
return "node";
}
async function ensurePackage(name: string): Promise<any> {
const runtime = detectRuntime();
try {
if (runtime === "deno") {
return await import(`npm:${name}`);
}
return await import(name);
} catch {
console.error(`${name} not found. Installing...`);
const { execSync } = await import("node:child_process");
try {
if (runtime === "bun") {
execSync(`bun add ${name}`, { stdio: "inherit" });
} else if (runtime === "deno") {
return await import(`npm:${name}`);
} else {
execSync(`npm install ${name}`, { stdio: "inherit" });
}
return await import(name);
} catch (installError) {
console.error(`Failed to install ${name}:`, installError);
process.exit(1);
}
}
}
function getThemeConfig(themeName: Theme): { bg: string; fg: string } {
const themeConfigs: Record<Theme, { bg: string; fg: string }> = {
default: { bg: "#f5f5f5", fg: "#333333" },
dracula: { bg: "#282a36", fg: "#f8f8f2" },
solarized: { bg: "#fdf6e3", fg: "#657b83" },
"zinc-dark": { bg: "#18181b", fg: "#fafafa" },
"tokyo-night": { bg: "#1a1b26", fg: "#a9b1d6" },
"tokyo-night-storm": { bg: "#24283b", fg: "#a9b1d6" },
"tokyo-night-light": { bg: "#d5d6db", fg: "#343b58" },
"catppuccin-latte": { bg: "#eff1f5", fg: "#4c4f69" },
nord: { bg: "#2e3440", fg: "#eceff4" },
"nord-light": { bg: "#eceff4", fg: "#2e3440" },
"github-dark": { bg: "#0d1117", fg: "#c9d1d9" },
"github-light": { bg: "#ffffff", fg: "#24292f" },
"one-dark": { bg: "#282c34", fg: "#abb2bf" },
};
return themeConfigs[themeName];
}
async function main(): Promise<void> {
const args = parseArgs();
let mermaidCode: string;
if (args.input) {
const inputPath = resolve(args.input);
if (!existsSync(inputPath)) {
console.error(`Input file not found: ${inputPath}`);
process.exit(1);
}
mermaidCode = readFileSync(inputPath, "utf-8");
} else {
mermaidCode = args.code!;
}
console.log(`Rendering diagram with theme: ${args.theme}`);
const beautifulMermaid = await ensurePackage("beautiful-mermaid");
const renderMermaid = beautifulMermaid.renderMermaid;
const THEMES = beautifulMermaid.THEMES;
const themeConfig = THEMES?.[args.theme] ?? getThemeConfig(args.theme);
console.log(`Using theme: bg=${themeConfig.bg}, fg=${themeConfig.fg}`);
const svg = await renderMermaid(mermaidCode, themeConfig);
const svgPath = resolve(`${args.output}.svg`);
writeFileSync(svgPath, svg, "utf-8");
console.log(`SVG written to: ${svgPath}`);
}
main().catch((err) => {
console.error("Error:", err.message);
process.exit(1);
});
.claude/skills/bfl-api/SKILL.md
+264
View File
@@ -0,0 +1,264 @@
---
name: bfl-api
description: BFL FLUX API integration guide covering endpoints, async polling patterns, rate limiting, error handling, webhooks, and regional endpoints with Python and TypeScript code examples.
metadata:
author: Black Forest Labs
version: "1.0.0"
tags: flux, bfl, api, integration, webhooks, rate-limiting
---
# BFL API Integration Guide
Use this skill when integrating BFL FLUX APIs into applications for image generation, editing, and processing.
## First: Check API Key
**Before generating images, verify your API key is set:**
```bash
echo $BFL_API_KEY
```
If empty or you see "Not authenticated" errors, see [API Key Setup](#api-key-setup) below.
## Important: Image URLs Expire in 10 Minutes
Result URLs from the API are temporary. Download images immediately after generation completes - do not store or cache the URLs themselves.
## When to Use
- Setting up BFL API client
- Implementing async polling patterns
- Handling rate limits and errors
- Configuring webhooks for production
- Selecting regional endpoints
- Building production-ready integrations
## Quick Reference
### Base Endpoints
| Region | Endpoint | Use Case |
| ------ | ----------------------- | --------------------------- |
| Global | `https://api.bfl.ai` | Default, automatic failover |
| EU | `https://api.eu.bfl.ai` | GDPR compliance |
| US | `https://api.us.bfl.ai` | US data residency |
### Model Endpoints & Pricing
> **Credit pricing:** 1 credit = $0.01 USD. FLUX.2 uses megapixel-based pricing (cost scales with resolution).
#### FLUX.2 Models
| Model | Path | 1st MP | +MP | 1MP T2I | 1MP I2I | Best For |
| ----------------- | --------------------- | ------ | ---- | ------- | ------- | ---------------------------------- |
| FLUX.2 [klein] 4B | `/v1/flux-2-klein-4b` | 1.4c | 0.1c | $0.014 | $0.015 | Real-time, high volume |
| FLUX.2 [klein] 9B | `/v1/flux-2-klein-9b` | 1.5c | 0.2c | $0.015 | $0.017 | Balanced quality/speed |
| FLUX.2 [pro] | `/v1/flux-2-pro` | 3c | 1.5c | $0.03 | $0.045 | Production, fast turnaround |
| FLUX.2 [max] | `/v1/flux-2-max` | 7c | 3c | $0.07 | $0.10 | Maximum quality |
| FLUX.2 [flex] | `/v1/flux-2-flex` | 5c | 5c | $0.05 | $0.10 | Typography, adjustable controls |
| FLUX.2 [dev] | - | - | - | Free | Free | Local development (non-commercial) |
> **Pricing formula:** `(firstMP + (outputMP-1) * mpPrice) + (inputMP * mpPrice)` in cents
#### FLUX.1 Models
| Model | Path | Price/Image | Best For |
| -------------------- | ------------------------ | ----------- | ----------------------------- |
| FLUX.1 Kontext [pro] | `/v1/flux-kontext` | $0.04 | Image editing with context |
| FLUX.1 Kontext [max] | `/v1/flux-kontext-max` | $0.08 | Max quality editing |
| FLUX1.1 [pro] | `/v1/flux-pro-1.1` | $0.04 | Standard T2I, fast & reliable |
| FLUX1.1 [pro] Ultra | `/v1/flux-pro-1.1-ultra` | $0.06 | Ultra high-resolution |
| FLUX1.1 [pro] Raw | `/v1/flux-pro-1.1-raw` | $0.06 | Candid photography feel |
| FLUX.1 Fill [pro] | `/v1/flux-pro-1.0-fill` | $0.05 | Inpainting |
> **Tip:** All FLUX.2 models support image editing via the `input_image` parameter - no separate editing endpoint needed. Use [bfl.ai/pricing](https://bfl.ai/pricing) calculator for exact costs at different resolutions.
### Image Input for Editing
**Preferred: Use URLs directly** - simpler and more convenient than base64.
**Single image editing:**
```bash
curl -X POST "https://api.bfl.ai/v1/flux-2-pro" \
-H "x-key: $BFL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "Change the background to a sunset",
"input_image": "https://example.com/photo.jpg"
}'
```
**Multi-reference editing:**
```bash
curl -X POST "https://api.bfl.ai/v1/flux-2-pro" \
-H "x-key: $BFL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "The person from image 1 in the environment from image 2",
"input_image": "https://example.com/person.jpg",
"input_image_2": "https://example.com/background.jpg"
}'
```
The API fetches URLs automatically. Both URL and base64 work, but URLs are recommended when available.
### Multi-Reference I2I
FLUX.2 models support multiple input images for combining elements, style transfer, and character consistency:
| Model | Max References |
| --------------------- | -------------- |
| FLUX.2 [klein] | 4 images |
| FLUX.2 [pro/max/flex] | 8 images |
**Parameters:** `input_image`, `input_image_2`, `input_image_3`, ... `input_image_8`
**Prompt pattern:** Reference images by number in your prompt:
- "The subject from image 1 in the environment from image 2"
- "Apply the style of image 2 to the scene in image 1"
- "The person from image 1 wearing the outfit from image 2, in the pose from image 3"
> For detailed multi-reference patterns (character consistency, style transfer, pose guidance), see `flux-best-practices/rules/multi-reference-editing.md`
### Rate Limits
| Tier | Concurrent Requests |
| ------------------------- | ------------------- |
| Standard (most endpoints) | 24 |
### Polling vs Webhooks
| Approach | Use When |
| ------------ | ------------------------------------------------------------------------------------ |
| **Polling** | Scripts, CLI tools, local development, single requests, simple integrations |
| **Webhooks** | Production apps, high volume, server-to-server, when you need immediate notification |
**Start with polling** - it's simpler and works everywhere. Switch to webhooks when you need to scale or want event-driven architecture.
### Key Behaviors
- **Polling**: Response includes `polling_url` for async results
- **URL Expiration**: Result URLs expire after 10 minutes
- **Webhook Support**: Configure `webhook_url` for production workloads
## API Key Setup
**Required**: The `BFL_API_KEY` environment variable must be set before using the API.
### Quick Check
```bash
echo $BFL_API_KEY
```
### If Not Set
1. **Get a key**: Go to https://dashboard.bfl.ai/get-started → Click **"Create Key"** → Select organization
2. **Save to `.env`** (recommended for persistence):
```bash
echo 'BFL_API_KEY=bfl_your_key_here' >> .env
echo '.env' >> .gitignore # Don't commit secrets
```
See [references/api-key-setup.md](references/api-key-setup.md) for detailed setup instructions.
## Authentication
```bash
x-key: YOUR_API_KEY
```
## Basic Request Flow
```
1. POST request to model endpoint
└─> Response: { "polling_url": "..." }
2. GET polling_url (repeat until complete)
└─> Response: { "status": "Pending" | "Ready" | "Error", ... }
3. When Ready, download result URL
└─> URL expires in 10 minutes - download immediately
```
## Related
- **Prompting best practices** (T2I, I2I, typography, colors): see the **flux-best-practices** skill
- **Multi-reference patterns** (character consistency, style transfer, pose guidance): see `flux-best-practices/rules/multi-reference-editing.md`
## References
- [references/api-key-setup.md](references/api-key-setup.md) - **API key creation and configuration**
- [references/endpoints.md](references/endpoints.md) - Complete endpoint documentation
- [references/polling-patterns.md](references/polling-patterns.md) - Async polling implementation
- [references/rate-limiting.md](references/rate-limiting.md) - Rate limit handling strategies
- [references/error-handling.md](references/error-handling.md) - Error codes and recovery
- [references/webhook-integration.md](references/webhook-integration.md) - Webhook setup and security
### Code Examples
> **Note:** cURL examples are preferred by default as they work universally without requiring Python or Node.js. Use language-specific clients when building production applications.
- [references/code-examples/curl-examples.sh](references/code-examples/curl-examples.sh) - **cURL examples (recommended)**
- [references/code-examples/python-client.py](references/code-examples/python-client.py) - Python client
- [references/code-examples/typescript-client.ts](references/code-examples/typescript-client.ts) - TypeScript client
## Quick Start Example
### 1. Submit Generation Request
```bash
curl -s -X POST "https://api.bfl.ai/v1/flux-2-pro" \
-H "x-key: $BFL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"prompt": "A serene mountain landscape at sunset", "width": 1024, "height": 1024}'
```
Response:
```json
{ "id": "abc123", "polling_url": "https://api.bfl.ai/v1/get_result?id=abc123" }
```
### 2. Poll for Result
```bash
curl -s "POLLING_URL" -H "x-key: $BFL_API_KEY"
```
Response when ready:
```json
{ "status": "Ready", "result": { "sample": "https://...", "seed": 1234 } }
```
### 3. Download Image
```bash
curl -s -o output.png "IMAGE_URL"
```
> **Tip:** Result URLs expire in 10 minutes. Download immediately after status becomes `Ready`.
### 4. Multi-Reference Example
Combine elements from multiple images:
```bash
curl -s -X POST "https://api.bfl.ai/v1/flux-2-pro" \
-H "x-key: $BFL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "The cat from image 1 sitting in the cozy room from image 2",
"input_image": "https://example.com/cat.jpg",
"input_image_2": "https://example.com/room.jpg",
"width": 1024,
"height": 1024
}'
```
Reference images by number in your prompt. See [Multi-Reference I2I](#multi-reference-i2i) for limits and patterns.
.claude/skills/bfl-api/references/api-key-setup.md
+55
View File
@@ -0,0 +1,55 @@
---
name: api-key-setup
description: How to obtain and configure a BFL API key
---
# API Key Setup
> **Important:** Always verify your API key before attempting image generation. Missing or invalid keys result in "Not authenticated" errors.
## Quick Validation
Run this first to check if your key is configured and valid:
```bash
# Check if key is set
[ -z "$BFL_API_KEY" ] && echo "Error: BFL_API_KEY not set" || echo "OK: Key configured"
```
If not set, follow the steps below.
## Get a Key
1. Go to **https://dashboard.bfl.ai/get-started**
2. Click **"Create Key"**
3. Select organization (ask user if multiple options)
4. Copy the key (starts with `bfl_`)
## For Agents Making Direct API Calls
When `BFL_API_KEY` is not set in the current session:
1. **Check for existing `.env`**:
```bash
grep BFL_API_KEY .env 2>/dev/null
```
2. **If found, export it**:
```bash
export BFL_API_KEY=$(grep BFL_API_KEY .env | cut -d '=' -f2)
```
3. **If not found, ask the user** for their key:
> "I need a BFL API key to generate images. Please:
> 1. Go to https://dashboard.bfl.ai/get-started
> 2. Click 'Create Key' and copy it
> 3. Paste it here"
4. **Save and export**:
```bash
echo 'BFL_API_KEY=bfl_provided_key' >> .env
echo '.env' >> .gitignore
export BFL_API_KEY=bfl_provided_key
```
Now `$BFL_API_KEY` is available for direct curl/API calls in the session.
.claude/skills/bfl-api/references/code-examples/curl-examples.sh
+160
View File
@@ -0,0 +1,160 @@
#!/bin/bash
#
# BFL FLUX API - cURL Examples
#
# These examples demonstrate how to use the BFL FLUX API with cURL.
# Replace YOUR_API_KEY with your actual API key from https://dashboard.bfl.ai
#
API_KEY="${BFL_API_KEY:-YOUR_API_KEY}"
BASE_URL="https://api.bfl.ai"
# -----------------------------------------------------------------------------
# FIRST: Verify API Key is Set
# -----------------------------------------------------------------------------
# Always check this before making requests to avoid "Not authenticated" errors
if [ "$API_KEY" = "YOUR_API_KEY" ] || [ -z "$API_KEY" ]; then
echo "Error: BFL_API_KEY not set"
echo ""
echo "To fix:"
echo " 1. Get a key at https://dashboard.bfl.ai/get-started"
echo " 2. Run: export BFL_API_KEY=your_key_here"
exit 1
fi
echo "OK: API key configured"
# -----------------------------------------------------------------------------
# Example 1: Basic Image Generation with FLUX.2 Pro
# -----------------------------------------------------------------------------
echo "=== Submitting generation request ==="
RESPONSE=$(curl -s -X POST "${BASE_URL}/v1/flux-2-pro" \
-H "x-key: ${API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"prompt": "A serene mountain landscape at golden hour, dramatic lighting",
"width": 1024,
"height": 1024
}')
echo "Response: ${RESPONSE}"
# Extract polling URL
POLLING_URL=$(echo "${RESPONSE}" | grep -o '"polling_url":"[^"]*"' | cut -d'"' -f4)
echo "Polling URL: ${POLLING_URL}"
# -----------------------------------------------------------------------------
# Example 2: Poll for Result
# -----------------------------------------------------------------------------
echo ""
echo "=== Polling for result ==="
while true; do
RESULT=$(curl -s "${POLLING_URL}" -H "x-key: ${API_KEY}")
STATUS=$(echo "${RESULT}" | grep -o '"status":"[^"]*"' | cut -d'"' -f4)
echo "Status: ${STATUS}"
if [ "${STATUS}" = "Ready" ]; then
IMAGE_URL=$(echo "${RESULT}" | grep -o '"sample":"[^"]*"' | cut -d'"' -f4)
echo "Image URL: ${IMAGE_URL}"
break
elif [ "${STATUS}" = "Error" ]; then
echo "Generation failed!"
echo "${RESULT}"
exit 1
fi
sleep 2
done
# -----------------------------------------------------------------------------
# Example 3: Download the Image
# -----------------------------------------------------------------------------
echo ""
echo "=== Downloading image ==="
curl -s -o output.png "${IMAGE_URL}"
echo "Saved to output.png"
# =============================================================================
# ONE-LINER EXAMPLES (for quick reference)
# =============================================================================
# Submit request (returns polling_url):
# curl -s -X POST "https://api.bfl.ai/v1/flux-2-pro" \
# -H "x-key: YOUR_API_KEY" \
# -H "Content-Type: application/json" \
# -d '{"prompt": "A red apple", "width": 1024, "height": 1024}'
# Poll for result (replace POLLING_URL):
# curl -s "POLLING_URL" -H "x-key: YOUR_API_KEY"
# Download image (replace IMAGE_URL):
# curl -s -o output.png "IMAGE_URL"
# =============================================================================
# IMAGE-TO-IMAGE EDITING
# =============================================================================
# Preferred: Pass image URLs directly - simpler and more convenient than base64.
# The API fetches URLs automatically. Both URL and base64 work.
echo ""
echo "=== Image-to-Image Edit Example ==="
# Edit an image using its URL directly
I2I_RESPONSE=$(curl -s -X POST "${BASE_URL}/v1/flux-2-pro" \
-H "x-key: ${API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"prompt": "Change the background to a sunset beach",
"input_image": "https://example.com/photo.jpg"
}')
echo "I2I Response: ${I2I_RESPONSE}"
# Multi-reference example (combine elements from multiple images)
# curl -s -X POST "${BASE_URL}/v1/flux-2-max" \
# -H "x-key: ${API_KEY}" \
# -H "Content-Type: application/json" \
# -d '{
# "prompt": "Person from image 1 wearing outfit from image 2 in setting from image 3",
# "input_image": "https://example.com/person.jpg",
# "input_image_2": "https://example.com/outfit.jpg",
# "input_image_3": "https://example.com/location.jpg"
# }'
# =============================================================================
# MODEL ENDPOINT EXAMPLES
# =============================================================================
# FLUX.2 [klein] 4B - Fastest
# curl -s -X POST "https://api.bfl.ai/v1/flux-2-klein-4b" ...
# FLUX.2 [klein] 9B - Fast with better quality
# curl -s -X POST "https://api.bfl.ai/v1/flux-2-klein-9b" ...
# FLUX.2 [pro] - Production balanced
# curl -s -X POST "https://api.bfl.ai/v1/flux-2-pro" ...
# FLUX.2 [max] - Highest quality
# curl -s -X POST "https://api.bfl.ai/v1/flux-2-max" ...
# FLUX.2 [flex] - Best for typography
# curl -s -X POST "https://api.bfl.ai/v1/flux-2-flex" ...
# =============================================================================
# REGIONAL ENDPOINTS
# =============================================================================
# Global (default): https://api.bfl.ai
# EU (GDPR): https://api.eu.bfl.ai
# US: https://api.us.bfl.ai
.claude/skills/bfl-api/references/code-examples/python-client.py
+476
View File
@@ -0,0 +1,476 @@
"""
BFL FLUX API Python Client
A complete, production-ready Python client for the BFL FLUX API.
Includes rate limiting, retry logic, webhook support, and async operations.
Usage:
from bfl_client import BFLClient
client = BFLClient("your-api-key")
result = client.generate("flux-2-pro", "A beautiful sunset")
print(f"Image URL: {result['url']}")
"""
import os
import time
import hmac
import hashlib
import logging
from typing import Optional, Dict, Any, List, Callable
from dataclasses import dataclass
from threading import Semaphore, Lock
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# --- Exceptions ---
class BFLError(Exception):
"""Base exception for BFL API errors."""
def __init__(self, message: str, status_code: int = None, error_code: str = None):
self.message = message
self.status_code = status_code
self.error_code = error_code
super().__init__(message)
class AuthenticationError(BFLError):
"""API key or authentication issue."""
pass
class InsufficientCreditsError(BFLError):
"""Account needs more credits."""
pass
class RateLimitError(BFLError):
"""Too many concurrent requests."""
def __init__(self, message: str, retry_after: int = 5):
super().__init__(message, 429, "rate_limit_exceeded")
self.retry_after = retry_after
class ValidationError(BFLError):
"""Invalid request parameters."""
pass
class GenerationError(BFLError):
"""Generation failed."""
pass
# --- Data Classes ---
@dataclass
class GenerationResult:
"""Result of a successful generation."""
id: str
url: str
width: int
height: int
raw: Dict[str, Any]
# --- Client ---
class BFLClient:
"""
Production-ready BFL FLUX API client.
Features:
- Rate limiting with semaphore
- Automatic retries with exponential backoff
- Webhook support
- Batch processing
- Async operations
Example:
client = BFLClient("your-api-key")
result = client.generate("flux-2-pro", "A sunset over mountains")
client.download(result.url, "sunset.png")
"""
BASE_URLS = {
"global": "https://api.bfl.ai",
"eu": "https://api.eu.bfl.ai",
"us": "https://api.us.bfl.ai",
}
RATE_LIMITS = {
"default": 24,
"flux-kontext-max": 6,
}
def __init__(
self,
api_key: str,
region: str = "global",
max_concurrent: int = None,
timeout: int = 120,
):
"""
Initialize the BFL client.
Args:
api_key: Your BFL API key
region: API region ("global", "eu", "us")
max_concurrent: Max concurrent requests (default: 24)
timeout: Default polling timeout in seconds
"""
self.api_key = api_key
self.base_url = self.BASE_URLS.get(region, self.BASE_URLS["global"])
self.timeout = timeout
self.max_concurrent = max_concurrent or self.RATE_LIMITS["default"]
self.semaphore = Semaphore(self.max_concurrent)
self.headers = {
"x-key": api_key,
"Content-Type": "application/json",
}
def generate(
self,
model: str,
prompt: str,
width: int = 1024,
height: int = 1024,
seed: int = None,
safety_tolerance: int = 2,
output_format: str = "png",
webhook_url: str = None,
webhook_secret: str = None,
timeout: int = None,
**kwargs,
) -> GenerationResult:
"""
Generate an image from a text prompt.
Args:
model: Model to use (e.g., "flux-2-pro", "flux-2-max")
prompt: Text description of the image
width: Image width (multiple of 16)
height: Image height (multiple of 16)
seed: Random seed for reproducibility
safety_tolerance: 0 (strict) to 5 (permissive)
output_format: "png" or "jpeg"
webhook_url: URL for async notification
webhook_secret: Secret for webhook signature
timeout: Polling timeout override
**kwargs: Additional model-specific parameters
Returns:
GenerationResult with image URL and metadata
"""
# Validate dimensions
self._validate_dimensions(width, height)
# Build payload
payload = {
"prompt": prompt,
"width": width,
"height": height,
"safety_tolerance": safety_tolerance,
"output_format": output_format,
**kwargs,
}
if seed is not None:
payload["seed"] = seed
if webhook_url:
payload["webhook_url"] = webhook_url
if webhook_secret:
payload["webhook_secret"] = webhook_secret
# Rate-limited request
with self.semaphore:
return self._submit_and_poll(model, payload, timeout or self.timeout)
def generate_i2i(
self,
model: str,
prompt: str,
input_image: str,
additional_images: List[str] = None,
**kwargs,
) -> GenerationResult:
"""
Generate an image from another image (image-to-image).
Preferred: Pass image URLs directly - simpler and more convenient than base64.
The API fetches URLs automatically. Both URL and base64 work.
Args:
model: Model to use (e.g., "flux-2-pro", "flux-2-max")
prompt: Edit instructions
input_image: Image URL (preferred) or base64
additional_images: List of additional reference image URLs or base64
**kwargs: Additional parameters
Returns:
GenerationResult with edited image
Example:
result = client.generate_i2i(
"flux-2-pro",
"Change the background to a sunset",
"https://example.com/photo.jpg" # URL is simpler!
)
"""
payload = {
"prompt": prompt,
"input_image": input_image,
**kwargs,
}
# Add additional images
if additional_images:
for i, img in enumerate(additional_images[:7], start=2):
payload[f"input_image_{i}"] = img
with self.semaphore:
return self._submit_and_poll(model, payload, self.timeout)
def generate_batch(
self,
model: str,
prompts: List[str],
max_workers: int = None,
**kwargs,
) -> List[GenerationResult]:
"""
Generate multiple images concurrently.
Args:
model: Model to use
prompts: List of prompts
max_workers: Number of concurrent workers
**kwargs: Parameters applied to all generations
Returns:
List of GenerationResult objects
"""
max_workers = max_workers or min(len(prompts), self.max_concurrent)
results = []
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {
executor.submit(self.generate, model, prompt, **kwargs): prompt
for prompt in prompts
}
for future in as_completed(futures):
try:
result = future.result()
results.append(result)
except Exception as e:
logger.error(f"Generation failed: {e}")
results.append(None)
return results
def download(self, url: str, output_path: str) -> str:
"""
Download a generated image.
Args:
url: Image URL (expires in 10 minutes)
output_path: Local path to save the image
Returns:
Path to saved file
"""
response = requests.get(url, timeout=60)
response.raise_for_status()
with open(output_path, "wb") as f:
f.write(response.content)
return output_path
def _submit_and_poll(
self,
model: str,
payload: Dict[str, Any],
timeout: int,
) -> GenerationResult:
"""Submit request and poll for result."""
endpoint = f"{self.base_url}/v1/{model}"
# Submit with retry
response = self._request_with_retry(
"POST",
endpoint,
json=payload,
)
polling_url = response["polling_url"]
generation_id = response.get("id", polling_url.split("=")[-1])
# Poll for result
result = self._poll(polling_url, timeout)
return GenerationResult(
id=generation_id,
url=result["sample"],
width=result.get("width", payload.get("width")),
height=result.get("height", payload.get("height")),
raw=result,
)
def _poll(self, polling_url: str, timeout: int) -> Dict[str, Any]:
"""Poll until completion or timeout."""
start_time = time.time()
delay = 1.0
while time.time() - start_time < timeout:
response = self._request_with_retry("GET", polling_url)
status = response.get("status")
if status == "Ready":
return response.get("result", response)
elif status == "Error":
error = response.get("error", "Generation failed")
raise GenerationError(error)
# Exponential backoff (cap at 5 seconds)
time.sleep(delay)
delay = min(delay * 1.5, 5.0)
raise TimeoutError(f"Generation timed out after {timeout}s")
def _request_with_retry(
self,
method: str,
url: str,
max_retries: int = 3,
**kwargs,
) -> Dict[str, Any]:
"""Make HTTP request with retry logic."""
last_exception = None
for attempt in range(max_retries):
try:
response = requests.request(
method,
url,
headers=self.headers,
timeout=30,
**kwargs,
)
return self._handle_response(response)
except RateLimitError as e:
logger.warning(f"Rate limited, waiting {e.retry_after}s")
time.sleep(e.retry_after * (attempt + 1))
last_exception = e
except BFLError as e:
if e.status_code and e.status_code >= 500:
wait_time = 2 ** attempt
logger.warning(f"Server error, retrying in {wait_time}s")
time.sleep(wait_time)
last_exception = e
else:
raise
raise last_exception
def _handle_response(self, response: requests.Response) -> Dict[str, Any]:
"""Process API response and raise appropriate errors."""
if response.status_code == 200:
return response.json()
try:
error_data = response.json()
except:
error_data = {"message": response.text}
error_code = error_data.get("error", "unknown")
message = error_data.get("message", "Unknown error")
if response.status_code == 401:
raise AuthenticationError(message, 401, error_code)
elif response.status_code == 402:
raise InsufficientCreditsError(message, 402, error_code)
elif response.status_code == 429:
retry_after = int(response.headers.get("Retry-After", 5))
raise RateLimitError(message, retry_after)
elif response.status_code == 400:
raise ValidationError(message, 400, error_code)
else:
raise BFLError(message, response.status_code, error_code)
def _validate_dimensions(self, width: int, height: int):
"""Validate image dimensions."""
if width % 16 != 0:
raise ValidationError(f"Width {width} must be a multiple of 16")
if height % 16 != 0:
raise ValidationError(f"Height {height} must be a multiple of 16")
if width * height > 4_000_000:
raise ValidationError(f"Total pixels ({width}x{height}) exceeds 4MP limit")
if width < 64 or height < 64:
raise ValidationError("Minimum dimension is 64 pixels")
# --- Webhook Verification ---
def verify_webhook_signature(payload: bytes, signature: str, secret: str) -> bool:
"""
Verify a webhook signature from BFL.
Args:
payload: Raw request body
signature: X-BFL-Signature header value
secret: Your webhook secret
Returns:
True if signature is valid
"""
if not signature or not signature.startswith("sha256="):
return False
expected = hmac.new(
secret.encode("utf-8"),
payload,
hashlib.sha256,
).hexdigest()
provided = signature[7:] # Remove 'sha256=' prefix
return hmac.compare_digest(expected, provided)
# --- Example Usage ---
if __name__ == "__main__":
# Get API key from environment
api_key = os.environ.get("BFL_API_KEY")
if not api_key:
print("Set BFL_API_KEY environment variable")
exit(1)
# Create client
client = BFLClient(api_key)
# Generate a single image
print("Generating image...")
result = client.generate(
model="flux-2-pro",
prompt="A serene mountain landscape at golden hour, dramatic lighting",
width=1024,
height=1024,
)
print(f"Generated: {result.url}")
# Download the image
client.download(result.url, "output.png")
print("Saved to output.png")
.claude/skills/bfl-api/references/code-examples/typescript-client.ts
+480
View File
@@ -0,0 +1,480 @@
/**
* BFL FLUX API TypeScript Client
*
* A complete, production-ready TypeScript client for the BFL FLUX API.
* Includes rate limiting, retry logic, webhook support, and async operations.
*
* Usage:
* import { BFLClient } from './bfl-client';
*
* const client = new BFLClient('your-api-key');
* const result = await client.generate('flux-2-pro', 'A beautiful sunset');
* console.log(`Image URL: ${result.url}`);
*/
import * as crypto from "crypto";
// --- Types ---
export interface GenerationResult {
id: string;
url: string;
width: number;
height: number;
raw: Record<string, unknown>;
}
export interface GenerateOptions {
width?: number;
height?: number;
seed?: number;
safetyTolerance?: number;
outputFormat?: "png" | "jpeg";
webhookUrl?: string;
webhookSecret?: string;
timeout?: number;
steps?: number; // For flex model
guidance?: number; // For flex model
}
export interface I2IOptions extends GenerateOptions {
additionalImages?: string[];
}
export type Region = "global" | "eu" | "us";
// --- Errors ---
export class BFLError extends Error {
constructor(
message: string,
public statusCode?: number,
public errorCode?: string
) {
super(message);
this.name = "BFLError";
}
}
export class AuthenticationError extends BFLError {
constructor(message: string) {
super(message, 401, "authentication_error");
this.name = "AuthenticationError";
}
}
export class InsufficientCreditsError extends BFLError {
constructor(message: string) {
super(message, 402, "insufficient_credits");
this.name = "InsufficientCreditsError";
}
}
export class RateLimitError extends BFLError {
constructor(
message: string,
public retryAfter: number = 5
) {
super(message, 429, "rate_limit_exceeded");
this.name = "RateLimitError";
}
}
export class ValidationError extends BFLError {
constructor(message: string) {
super(message, 400, "validation_error");
this.name = "ValidationError";
}
}
export class GenerationError extends BFLError {
constructor(message: string) {
super(message, undefined, "generation_error");
this.name = "GenerationError";
}
}
// --- Rate Limiter ---
class Semaphore {
private permits: number;
private waiting: Array<() => void> = [];
constructor(permits: number) {
this.permits = permits;
}
async acquire(): Promise<void> {
if (this.permits > 0) {
this.permits--;
return;
}
return new Promise((resolve) => {
this.waiting.push(resolve);
});
}
release(): void {
if (this.waiting.length > 0) {
const next = this.waiting.shift();
next?.();
} else {
this.permits++;
}
}
}
// --- Client ---
export class BFLClient {
private static readonly BASE_URLS: Record<Region, string> = {
global: "https://api.bfl.ai",
eu: "https://api.eu.bfl.ai",
us: "https://api.us.bfl.ai",
};
private static readonly RATE_LIMITS: Record<string, number> = {
default: 24,
"flux-kontext-max": 6,
};
private readonly baseUrl: string;
private readonly headers: Record<string, string>;
private readonly timeout: number;
private readonly semaphore: Semaphore;
/**
* Create a new BFL client.
*
* @param apiKey - Your BFL API key
* @param region - API region ("global", "eu", "us")
* @param maxConcurrent - Max concurrent requests (default: 24)
* @param timeout - Default polling timeout in milliseconds
*/
constructor(
private readonly apiKey: string,
region: Region = "global",
maxConcurrent: number = 24,
timeout: number = 120000
) {
this.baseUrl = BFLClient.BASE_URLS[region];
this.timeout = timeout;
this.semaphore = new Semaphore(maxConcurrent);
this.headers = {
"x-key": apiKey,
"Content-Type": "application/json",
};
}
/**
* Generate an image from a text prompt.
*/
async generate(
model: string,
prompt: string,
options: GenerateOptions = {}
): Promise<GenerationResult> {
const {
width = 1024,
height = 1024,
seed,
safetyTolerance = 2,
outputFormat = "png",
webhookUrl,
webhookSecret,
timeout = this.timeout,
steps,
guidance,
} = options;
// Validate dimensions
this.validateDimensions(width, height);
// Build payload
const payload: Record<string, unknown> = {
prompt,
width,
height,
safety_tolerance: safetyTolerance,
output_format: outputFormat,
};
if (seed !== undefined) payload.seed = seed;
if (webhookUrl) payload.webhook_url = webhookUrl;
if (webhookSecret) payload.webhook_secret = webhookSecret;
if (steps !== undefined) payload.steps = steps;
if (guidance !== undefined) payload.guidance = guidance;
// Rate-limited request
await this.semaphore.acquire();
try {
return await this.submitAndPoll(model, payload, timeout);
} finally {
this.semaphore.release();
}
}
/**
* Generate an image from another image (image-to-image).
*
* Preferred: Pass image URLs directly - simpler and more convenient than base64.
* The API fetches URLs automatically. Both URL and base64 work.
*
* @param model - Model to use (e.g., "flux-2-pro", "flux-2-max")
* @param prompt - Edit instructions
* @param inputImage - Image URL (preferred) or base64
* @param options - Additional options including more reference image URLs or base64
*
* @example
* const result = await client.generateI2I(
* "flux-2-pro",
* "Change the background to a sunset",
* "https://example.com/photo.jpg" // URL is simpler!
* );
*/
async generateI2I(
model: string,
prompt: string,
inputImage: string,
options: I2IOptions = {}
): Promise<GenerationResult> {
const { additionalImages = [], ...rest } = options;
const payload: Record<string, unknown> = {
prompt,
input_image: inputImage,
};
// Add additional images
additionalImages.slice(0, 7).forEach((img, i) => {
payload[`input_image_${i + 2}`] = img;
});
await this.semaphore.acquire();
try {
return await this.submitAndPoll(model, payload, rest.timeout ?? this.timeout);
} finally {
this.semaphore.release();
}
}
/**
* Generate multiple images concurrently.
*/
async generateBatch(
model: string,
prompts: string[],
options: GenerateOptions = {}
): Promise<Array<GenerationResult | Error>> {
const tasks = prompts.map((prompt) =>
this.generate(model, prompt, options).catch((e) => e)
);
return Promise.all(tasks);
}
/**
* Download a generated image.
*/
async download(url: string): Promise<ArrayBuffer> {
const response = await fetch(url);
if (!response.ok) {
throw new BFLError(`Failed to download: ${response.status}`);
}
return response.arrayBuffer();
}
private async submitAndPoll(
model: string,
payload: Record<string, unknown>,
timeout: number
): Promise<GenerationResult> {
const endpoint = `${this.baseUrl}/v1/${model}`;
// Submit request
const submitResponse = await this.requestWithRetry("POST", endpoint, payload);
const pollingUrl = submitResponse.polling_url as string;
const generationId =
(submitResponse.id as string) ?? pollingUrl.split("=").pop() ?? "unknown";
// Poll for result
const result = await this.poll(pollingUrl, timeout);
return {
id: generationId,
url: result.sample as string,
width: (result.width as number) ?? (payload.width as number),
height: (result.height as number) ?? (payload.height as number),
raw: result,
};
}
private async poll(
pollingUrl: string,
timeout: number
): Promise<Record<string, unknown>> {
const startTime = Date.now();
let delay = 1000;
while (Date.now() - startTime < timeout) {
const response = await this.requestWithRetry("GET", pollingUrl);
const status = response.status as string;
if (status === "Ready") {
return (response.result as Record<string, unknown>) ?? response;
} else if (status === "Error") {
throw new GenerationError((response.error as string) ?? "Generation failed");
}
// Exponential backoff (cap at 5 seconds)
await this.sleep(delay);
delay = Math.min(delay * 1.5, 5000);
}
throw new Error(`Generation timed out after ${timeout}ms`);
}
private async requestWithRetry(
method: string,
url: string,
body?: Record<string, unknown>,
maxRetries: number = 3
): Promise<Record<string, unknown>> {
let lastError: Error | undefined;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await fetch(url, {
method,
headers: this.headers,
body: body ? JSON.stringify(body) : undefined,
});
return await this.handleResponse(response);
} catch (e) {
if (e instanceof RateLimitError) {
console.warn(`Rate limited, waiting ${e.retryAfter}s`);
await this.sleep(e.retryAfter * 1000 * (attempt + 1));
lastError = e;
} else if (e instanceof BFLError && e.statusCode && e.statusCode >= 500) {
const waitTime = Math.pow(2, attempt) * 1000;
console.warn(`Server error, retrying in ${waitTime}ms`);
await this.sleep(waitTime);
lastError = e;
} else {
throw e;
}
}
}
throw lastError ?? new Error("Max retries exceeded");
}
private async handleResponse(response: Response): Promise<Record<string, unknown>> {
if (response.ok) {
return response.json();
}
let errorData: Record<string, unknown>;
try {
errorData = await response.json();
} catch {
errorData = { message: await response.text() };
}
const errorCode = (errorData.error as string) ?? "unknown";
const message = (errorData.message as string) ?? "Unknown error";
switch (response.status) {
case 401:
throw new AuthenticationError(message);
case 402:
throw new InsufficientCreditsError(message);
case 429:
const retryAfter = parseInt(response.headers.get("Retry-After") ?? "5", 10);
throw new RateLimitError(message, retryAfter);
case 400:
throw new ValidationError(message);
default:
throw new BFLError(message, response.status, errorCode);
}
}
private validateDimensions(width: number, height: number): void {
if (width % 16 !== 0) {
throw new ValidationError(`Width ${width} must be a multiple of 16`);
}
if (height % 16 !== 0) {
throw new ValidationError(`Height ${height} must be a multiple of 16`);
}
if (width * height > 4_000_000) {
throw new ValidationError(`Total pixels (${width}x${height}) exceeds 4MP limit`);
}
if (width < 64 || height < 64) {
throw new ValidationError("Minimum dimension is 64 pixels");
}
}
private sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
}
// --- Webhook Verification ---
/**
* Verify a webhook signature from BFL.
*
* @param payload - Raw request body as string
* @param signature - X-BFL-Signature header value
* @param secret - Your webhook secret
* @returns True if signature is valid
*/
export function verifyWebhookSignature(
payload: string,
signature: string,
secret: string
): boolean {
if (!signature || !signature.startsWith("sha256=")) {
return false;
}
const expectedSignature = crypto
.createHmac("sha256", secret)
.update(payload)
.digest("hex");
const providedSignature = signature.slice(7); // Remove 'sha256=' prefix
return crypto.timingSafeEqual(
Buffer.from(expectedSignature),
Buffer.from(providedSignature)
);
}
// --- Example Usage ---
async function main() {
const apiKey = process.env.BFL_API_KEY;
if (!apiKey) {
console.error("Set BFL_API_KEY environment variable");
process.exit(1);
}
const client = new BFLClient(apiKey);
console.log("Generating image...");
const result = await client.generate("flux-2-pro", "A serene mountain landscape at golden hour", {
width: 1024,
height: 1024,
});
console.log(`Generated: ${result.url}`);
console.log(`Image ID: ${result.id}`);
}
// Run if executed directly
main().catch(console.error);
.claude/skills/bfl-api/references/endpoints.md
+286
View File
@@ -0,0 +1,286 @@
---
name: endpoints
description: Complete BFL API endpoint documentation
---
# BFL API Endpoints
Complete reference for all BFL FLUX API endpoints.
## Base URLs
| Region | Endpoint | Use Case |
| ------ | ----------------------- | ---------------------------------- |
| Global | `https://api.bfl.ai` | Default, automatic failover |
| EU | `https://api.eu.bfl.ai` | GDPR compliance, EU data residency |
| US | `https://api.us.bfl.ai` | US data residency |
**Recommendation:** Use the global endpoint (`api.bfl.ai`) unless you have specific regional requirements.
## Authentication
All requests require the `x-key` header with your API key:
```bash
x-key: YOUR_API_KEY
```
## FLUX.2 Text-to-Image and Image-to-Image Endpoints
### FLUX.2 [klein] 4B
```
POST /v1/flux-2-klein-4b
```
Fastest generation, 4B parameters.
### FLUX.2 [klein] 9B
```
POST /v1/flux-2-klein-9b
```
Fast generation with better quality, 9B parameters.
### FLUX.2 [max]
```
POST /v1/flux-2-max
```
Highest quality, supports grounding search.
### FLUX.2 [pro]
```
POST /v1/flux-2-pro
```
Production balanced quality and speed.
### FLUX.2 [flex]
```
POST /v1/flux-2-flex
```
Typography optimized, adjustable steps/guidance.
## FLUX.1 Endpoints
### FLUX1.1 [pro]
```
POST /v1/flux-pro-1.1
```
Text-to-image generation.
### FLUX.1 Kontext
```
POST /v1/flux-kontext
```
### FLUX.1 Kontext Max
```
POST /v1/flux-kontext-max
```
### FLUX.1 Fill
```
POST /v1/flux-fill
```
Inpainting and object removal - you can achieve inpainting and object removal with specific prompting style with FLUX.2 models for better performance.
## Common Request Parameters
### Text-to-Image (T2I)
| Parameter | Type | Required | Description |
| ------------------ | ------- | -------- | -------------------------------------------- |
| `prompt` | string | Yes | Text description (up to 32K tokens) |
| `width` | integer | No | Image width (multiple of 16, max 4MP total) |
| `height` | integer | No | Image height (multiple of 16, max 4MP total) |
| `seed` | integer | No | Random seed for reproducibility |
| `safety_tolerance` | integer | No | 0 (strict) to 5 (permissive), default 2 |
| `output_format` | string | No | "jpeg" or "png", default "jpeg" |
| `webhook_url` | string | No | URL for async notification |
| `webhook_secret` | string | No | Secret for webhook signature |
### Image-to-Image (I2I)
> **Important:** All FLUX.2 models (klein, pro, max, flex) support image-to-image editing via the `input_image` parameter. FLUX.2 is recommended over FLUX.1 Kontext for editing.
> **Preferred: Use URLs directly** - The API fetches URLs automatically, which is simpler and more convenient than downloading and encoding to base64. Both URL and base64 work, but URLs are recommended when available.
| Parameter | Type | Required | Description |
| --------------------------------- | ------- | -------- | -------------------------------------------------------------- |
| `prompt` | string | Yes | Edit instruction |
| `input_image` | string | Yes | **URL (preferred)** or base64 - API fetches URLs automatically |
| `input_image_2` - `input_image_8` | string | No | Additional reference URLs or base64 |
| `width` | integer | No | Output width |
| `height` | integer | No | Output height |
### FLUX.2 [flex] Specific
| Parameter | Type | Default | Description |
| ---------- | ------- | ------- | ----------------------- |
| `steps` | integer | 50 | Inference steps (1-50) |
| `guidance` | float | 4.5 | Guidance scale (1.5-10) |
## Resolution Constraints
- **Minimum:** 64x64 pixels
- **Maximum:** 4MP total (width x height)
- **Multiple of:** 16 (both dimensions)
### Common Resolutions
| Aspect Ratio | Resolution | Megapixels |
| --------------- | ---------- | ---------- |
| 1:1 (Square) | 1024x1024 | 1.05 MP |
| 16:9 (Wide) | 1920x1080 | 2.07 MP |
| 9:16 (Portrait) | 1080x1920 | 2.07 MP |
| 4:3 (Classic) | 1536x1152 | 1.77 MP |
| 2:1 (Panorama) | 2048x1024 | 2.10 MP |
## Example Requests
### Basic T2I Request
```bash
curl -X POST "https://api.bfl.ai/v1/flux-2-pro" \
-H "x-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "A serene mountain landscape at golden hour",
"width": 1024,
"height": 1024
}'
```
### Response
```json
{
"id": "gen_abc123xyz",
"polling_url": "https://api.bfl.ai/v1/get_result?id=gen_abc123xyz"
}
```
### T2I with All Options
```bash
curl -X POST "https://api.bfl.ai/v1/flux-2-max" \
-H "x-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "Professional headshot of a business executive",
"width": 1024,
"height": 1280,
"seed": 42,
"safety_tolerance": 2,
"output_format": "png",
"webhook_url": "https://your-server.com/webhook",
"webhook_secret": "your-secret-key"
}'
```
### I2I Request (FLUX.2 - Recommended)
Edit images using any FLUX.2 model by passing the source image URL directly:
```bash
curl -X POST "https://api.bfl.ai/v1/flux-2-klein-9b" \
-H "x-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "Change the floor color to light blue",
"input_image": "https://example.com/room-photo.jpg"
}'
```
For higher quality edits, use FLUX.2 [pro] or [max]:
```bash
curl -X POST "https://api.bfl.ai/v1/flux-2-pro" \
-H "x-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "Change the background to a beach sunset",
"input_image": "https://example.com/portrait.jpg"
}'
```
### Multi-Reference I2I (FLUX.2)
```bash
curl -X POST "https://api.bfl.ai/v1/flux-2-max" \
-H "x-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "Person from image 1 wearing outfit from image 2 in setting from image 3",
"input_image": "https://example.com/person.jpg",
"input_image_2": "https://example.com/outfit.jpg",
"input_image_3": "https://example.com/location.jpg"
}'
```
### FLUX.2 [flex] with Custom Steps
```bash
curl -X POST "https://api.bfl.ai/v1/flux-2-flex" \
-H "x-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "A poster with text \"SUMMER SALE\" in bold typography",
"steps": 50,
"guidance": 7.0
}'
```
## Polling Endpoint
### Get Result
```
GET /v1/get_result?id={generation_id}
```
### Response States
```json
// Pending
{ "status": "Pending" }
// Ready
{
"status": "Ready",
"result": {
"sample": "https://bfldeliveryprod.blob.core.windows.net/results/...",
"prompt": "...",
"seed": 1234567890
}
}
// Error
{
"status": "Error",
"error": "Error description"
}
```
## Error Responses
| Status Code | Meaning | Action |
| ----------- | ---------------- | ------------------ |
| 400 | Bad Request | Check parameters |
| 401 | Unauthorized | Verify API key |
| 402 | Payment Required | Add credits |
| 429 | Rate Limited | Implement backoff |
| 500 | Server Error | Retry with backoff |
.claude/skills/bfl-api/references/error-handling.md
+334
View File
@@ -0,0 +1,334 @@
---
name: error-handling
description: Error codes and recovery strategies for BFL API
---
# Error Handling
Comprehensive guide to handling errors from the BFL API.
## HTTP Status Codes
| Code | Meaning | Cause | Action |
|------|---------|-------|--------|
| 200 | OK | Request successful | Process response |
| 400 | Bad Request | Invalid parameters | Check request format |
| 401 | Unauthorized | Invalid/missing API key | Verify credentials |
| 402 | Payment Required | Insufficient credits | Add credits to account |
| 403 | Forbidden | Access denied | Check permissions |
| 404 | Not Found | Invalid endpoint | Verify URL |
| 429 | Too Many Requests | Rate limited | Implement backoff |
| 500 | Internal Server Error | Server issue | Retry with backoff |
| 502 | Bad Gateway | Network issue | Retry with backoff |
| 503 | Service Unavailable | Temporary outage | Retry with backoff |
## Error Response Format
```json
{
"error": "error_code",
"message": "Human-readable description",
"details": {
"field": "specific field info"
}
}
```
## Common Errors and Solutions
### Authentication Errors (401)
```json
{
"error": "invalid_api_key",
"message": "The provided API key is invalid or expired"
}
```
**Solution:**
```python
def verify_api_key(api_key):
if not api_key:
raise ValueError("API key is required")
if not api_key.startswith("bfl_"):
raise ValueError("Invalid API key format")
```
### Insufficient Credits (402)
```json
{
"error": "insufficient_credits",
"message": "Your account does not have enough credits"
}
```
**Solution:**
```python
def handle_payment_error(response):
if response.status_code == 402:
# Log and alert
logging.error("Insufficient credits - add funds")
# Optionally pause operations
raise InsufficientCreditsError("Add credits to continue")
```
### Rate Limiting (429)
```json
{
"error": "rate_limit_exceeded",
"message": "Too many concurrent requests",
"retry_after": 5
}
```
**Solution:**
```python
def handle_rate_limit(response):
if response.status_code == 429:
retry_after = int(response.headers.get('Retry-After', 5))
time.sleep(retry_after)
return True # Signal to retry
return False
```
### Validation Errors (400)
```json
{
"error": "validation_error",
"message": "Invalid request parameters",
"details": {
"width": "Must be a multiple of 16",
"prompt": "Cannot be empty"
}
}
```
**Solution:**
```python
def validate_request(prompt, width, height):
errors = []
if not prompt or not prompt.strip():
errors.append("Prompt cannot be empty")
if width % 16 != 0:
errors.append(f"Width {width} must be multiple of 16")
if height % 16 != 0:
errors.append(f"Height {height} must be multiple of 16")
if width * height > 4_000_000:
errors.append("Total pixels cannot exceed 4MP")
if errors:
raise ValidationError(errors)
```
### Generation Failures
Failures during polling:
```json
{
"status": "Error",
"error": "content_policy_violation",
"message": "The prompt violated content policy"
}
```
**Common failure reasons:**
- `content_policy_violation` - Prompt/image flagged by safety
- `generation_timeout` - Took too long to generate
- `internal_error` - Server-side issue
- `invalid_image` - Input image couldn't be processed
## Retry Strategy
```python
import time
import random
class RetryableError(Exception):
"""Errors that can be retried."""
pass
class NonRetryableError(Exception):
"""Errors that should not be retried."""
pass
def classify_error(status_code, error_code):
"""Determine if error is retryable."""
# Retryable
if status_code in [429, 500, 502, 503]:
return RetryableError
# Non-retryable
if status_code in [400, 401, 402, 403]:
return NonRetryableError
# Generation failures
if error_code in ['generation_timeout', 'internal_error']:
return RetryableError
if error_code in ['content_policy_violation', 'invalid_image']:
return NonRetryableError
return RetryableError # Default to retryable
def make_request_with_retry(func, max_retries=3):
"""Execute function with retry logic."""
last_exception = None
for attempt in range(max_retries):
try:
return func()
except RetryableError as e:
last_exception = e
wait_time = (2 ** attempt) + random.uniform(0, 1)
print(f"Attempt {attempt + 1} failed, retrying in {wait_time:.1f}s")
time.sleep(wait_time)
except NonRetryableError:
raise # Don't retry
raise last_exception
```
## Comprehensive Error Handler
```python
import logging
class BFLError(Exception):
"""Base exception for BFL API errors."""
def __init__(self, message, status_code=None, error_code=None):
self.message = message
self.status_code = status_code
self.error_code = error_code
super().__init__(message)
class AuthenticationError(BFLError):
"""API key or authentication issue."""
pass
class InsufficientCreditsError(BFLError):
"""Account needs more credits."""
pass
class RateLimitError(BFLError):
"""Too many concurrent requests."""
def __init__(self, message, retry_after=5):
super().__init__(message, 429, "rate_limit_exceeded")
self.retry_after = retry_after
class ValidationError(BFLError):
"""Invalid request parameters."""
pass
class GenerationError(BFLError):
"""Generation failed."""
pass
def handle_response(response):
"""Process API response and raise appropriate errors."""
if response.status_code == 200:
return response.json()
try:
error_data = response.json()
except:
error_data = {"message": response.text}
error_code = error_data.get("error", "unknown")
message = error_data.get("message", "Unknown error")
if response.status_code == 401:
raise AuthenticationError(message, 401, error_code)
if response.status_code == 402:
raise InsufficientCreditsError(message, 402, error_code)
if response.status_code == 429:
retry_after = int(response.headers.get('Retry-After', 5))
raise RateLimitError(message, retry_after)
if response.status_code == 400:
raise ValidationError(message, 400, error_code)
if response.status_code >= 500:
raise BFLError(f"Server error: {message}", response.status_code, error_code)
raise BFLError(message, response.status_code, error_code)
```
## Logging Best Practices
```python
import logging
import json
def setup_logging():
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
def log_request(endpoint, payload):
logging.info(f"Request: POST {endpoint}")
logging.debug(f"Payload: {json.dumps(payload, indent=2)}")
def log_error(error, context=None):
logging.error(f"Error: {error}")
if context:
logging.error(f"Context: {context}")
def log_generation_failure(status, error, prompt):
logging.warning(f"Generation failed: {error}")
logging.debug(f"Failed prompt: {prompt[:100]}...")
```
## Circuit Breaker Pattern
For production systems:
```python
import time
from threading import Lock
class CircuitBreaker:
def __init__(self, failure_threshold=5, reset_timeout=60):
self.failure_threshold = failure_threshold
self.reset_timeout = reset_timeout
self.failures = 0
self.last_failure_time = None
self.state = "closed" # closed, open, half-open
self.lock = Lock()
def record_success(self):
with self.lock:
self.failures = 0
self.state = "closed"
def record_failure(self):
with self.lock:
self.failures += 1
self.last_failure_time = time.time()
if self.failures >= self.failure_threshold:
self.state = "open"
def can_proceed(self):
with self.lock:
if self.state == "closed":
return True
if self.state == "open":
if time.time() - self.last_failure_time > self.reset_timeout:
self.state = "half-open"
return True
return False
# half-open: allow one request to test
return True
```
.claude/skills/bfl-api/references/polling-patterns.md
+240
View File
@@ -0,0 +1,240 @@
---
name: polling-patterns
description: Implementing async polling for BFL API responses
---
# Polling Patterns
BFL API uses asynchronous generation. All requests return a `polling_url` for status checking.
## Basic Flow
```
1. POST request to model endpoint
└─> Immediate response: { "polling_url": "..." }
2. GET polling_url (repeat until complete)
└─> { "status": "Pending" | "Ready" | "Error" }
3. When "Ready", download result sample URL
└─> URL expires in 10 minutes
```
## Response States
| Status | Description | Action |
|--------|-------------|--------|
| `Pending` | Request queued/processing | Continue polling |
| `Ready` | Generation finished | Download result |
| `Error` | Generation Error | Handle error |
## Polling Strategies
### Simple Fixed Interval
```python
import time
import requests
def poll_fixed_interval(polling_url, headers, interval=2, timeout=120):
"""Simple polling with fixed interval."""
start_time = time.time()
while time.time() - start_time < timeout:
response = requests.get(polling_url, headers=headers)
data = response.json()
if data["status"] == "Ready":
return data["result"]
elif data["status"] == "Error":
raise Exception(data.get("error", "Generation Error"))
time.sleep(interval)
raise TimeoutError("Polling timeout exceeded")
```
### Exponential Backoff (Recommended)
```python
import time
import random
import requests
def poll_with_backoff(polling_url, headers, max_attempts=30):
"""Polling with exponential backoff and jitter."""
base_delay = 0.5 # Start with 500ms
max_delay = 10 # Cap at 10 seconds
for attempt in range(max_attempts):
response = requests.get(polling_url, headers=headers)
data = response.json()
if data["status"] == "Ready":
return data["result"]
elif data["status"] == "Error":
raise Exception(data.get("error", "Generation Error"))
# Exponential backoff with jitter
delay = min(base_delay * (2 ** attempt), max_delay)
jitter = random.uniform(0, delay * 0.1) # 10% jitter
time.sleep(delay + jitter)
raise TimeoutError("Max polling attempts exceeded")
```
### Adaptive Polling
```python
import time
import requests
def poll_adaptive(polling_url, headers, timeout=120):
"""Adaptive polling that adjusts based on status."""
start_time = time.time()
delays = {
"Pending": 2.0, # Queue/processing
None: 1.5 # Unknown/default
}
while time.time() - start_time < timeout:
response = requests.get(polling_url, headers=headers)
data = response.json()
status = data.get("status")
if status == "Ready":
return data["result"]
elif status == "Error":
raise Exception(data.get("error", "Generation Error"))
delay = delays.get(status, delays[None])
time.sleep(delay)
raise TimeoutError("Polling timeout exceeded")
```
## Complete Example: Submit and Poll
```python
import time
import requests
class BFLClient:
def __init__(self, api_key, base_url="https://api.bfl.ai"):
self.base_url = base_url
self.headers = {
"x-key": api_key,
"Content-Type": "application/json"
}
def generate(self, model, prompt, **kwargs):
"""Submit generation request and poll for result."""
# Submit request
endpoint = f"{self.base_url}/v1/{model}"
payload = {"prompt": prompt, **kwargs}
response = requests.post(endpoint, headers=self.headers, json=payload)
response.raise_for_status()
polling_url = response.json()["polling_url"]
# Poll for result
return self._poll(polling_url)
def _poll(self, polling_url, timeout=120):
"""Poll until completion or timeout."""
start = time.time()
delay = 1.0
while time.time() - start < timeout:
response = requests.get(polling_url, headers=self.headers)
data = response.json()
if data["status"] == "Ready":
return data["result"]
elif data["status"] == "Error":
raise Exception(data.get("error"))
time.sleep(delay)
delay = min(delay * 1.5, 5.0) # Gradual backoff
raise TimeoutError("Generation timed out")
# Usage
client = BFLClient("your-api-key")
result = client.generate(
model="flux-2-pro",
prompt="A beautiful sunset over mountains"
)
print(f"Image URL: {result['sample']}")
```
## URL Expiration
**Critical:** Result URLs expire after 10 minutes. Always download immediately.
```python
def download_result(result_url, output_path):
"""Download result image before URL expires."""
response = requests.get(result_url)
response.raise_for_status()
with open(output_path, 'wb') as f:
f.write(response.content)
return output_path
```
## Batch Processing with Polling
```python
import asyncio
import aiohttp
async def generate_batch(client, prompts, model="flux-2-pro"):
"""Generate multiple images concurrently."""
async with aiohttp.ClientSession() as session:
# Submit all requests
tasks = []
for prompt in prompts:
task = submit_and_poll(session, client, model, prompt)
tasks.append(task)
# Wait for all to complete
results = await asyncio.gather(*tasks, return_exceptions=True)
return results
async def submit_and_poll(session, client, model, prompt):
"""Async submit and poll for single image."""
# Submit
async with session.post(
f"{client.base_url}/v1/{model}",
headers=client.headers,
json={"prompt": prompt}
) as response:
data = await response.json()
polling_url = data["polling_url"]
# Poll
while True:
async with session.get(polling_url, headers=client.headers) as response:
data = await response.json()
if data["status"] == "Ready":
return data["result"]
elif data["status"] == "Error":
raise Exception(data.get("error"))
await asyncio.sleep(2)
```
## Best Practices
1. **Always implement timeouts** - Never poll indefinitely
2. **Use exponential backoff** - Reduces server load, handles congestion
3. **Add jitter** - Prevents thundering herd when polling multiple requests
4. **Handle all status values** - Including unexpected ones
5. **Download immediately** - URLs expire in 10 minutes
6. **Log polling attempts** - Useful for debugging and monitoring
7. **Respect rate limits** - Implement proper backoff on 429 responses
.claude/skills/bfl-api/references/rate-limiting.md
+246
View File
@@ -0,0 +1,246 @@
---
name: rate-limiting
description: Understanding and handling BFL API rate limits
---
# Rate Limiting
BFL API enforces rate limits to ensure fair usage and system stability.
## Current Limits
| Endpoint Category | Concurrent Requests |
| ---------------------- | ------------------- |
| Standard (most models) | 24 |
**Concurrent requests** means in-flight requests (submitted but not yet completed).
## Rate Limit Headers
Check response headers for rate limit status:
```
X-RateLimit-Limit: 24
X-RateLimit-Remaining: 23
X-RateLimit-Reset: 1640000000
```
## HTTP 429 Response
When rate limited, you receive HTTP 429:
```json
{
"error": "rate_limit_exceeded",
"message": "Too many concurrent requests",
"retry_after": 5
}
```
## Handling Strategies
### 1. Client-Side Tracking
Track active requests to stay under limits:
```python
from threading import Lock, Semaphore
class RateLimitedClient:
def __init__(self, api_key, max_concurrent=24):
self.api_key = api_key
self.semaphore = Semaphore(max_concurrent)
def generate(self, model, prompt, **kwargs):
with self.semaphore: # Blocks if at limit
return self._make_request(model, prompt, **kwargs)
def _make_request(self, model, prompt, **kwargs):
# Submit request
response = requests.post(...)
polling_url = response.json()["polling_url"]
# Poll until complete (request still "active")
return self._poll(polling_url)
```
### 2. Retry with Exponential Backoff
```python
import time
def request_with_retry(endpoint, payload, headers, max_retries=5):
"""Make request with automatic retry on rate limit."""
for attempt in range(max_retries):
response = requests.post(endpoint, json=payload, headers=headers)
if response.status_code == 429:
retry_after = int(response.headers.get('Retry-After', 5))
wait_time = retry_after * (2 ** attempt) # Exponential backoff
print(f"Rate limited. Waiting {wait_time}s...")
time.sleep(wait_time)
continue
response.raise_for_status()
return response
raise Exception("Max retries exceeded due to rate limiting")
```
### 3. Queue-Based Architecture
For high-volume applications:
```python
from queue import Queue
from threading import Thread
import time
class RequestQueue:
def __init__(self, api_key, max_concurrent=24):
self.api_key = api_key
self.queue = Queue()
self.active = 0
self.max_concurrent = max_concurrent
self.lock = Lock()
# Start worker threads
for _ in range(max_concurrent):
worker = Thread(target=self._worker, daemon=True)
worker.start()
def submit(self, model, prompt, callback):
"""Submit request to queue."""
self.queue.put({
'model': model,
'prompt': prompt,
'callback': callback
})
def _worker(self):
"""Process queue items."""
while True:
item = self.queue.get()
try:
result = self._process(item)
item['callback'](result, None)
except Exception as e:
item['callback'](None, e)
finally:
self.queue.task_done()
def _process(self, item):
# Make request and poll
...
# Usage
queue = RequestQueue("your-api-key")
def handle_result(result, error):
if error:
print(f"Error: {error}")
else:
print(f"Generated: {result['sample']}")
queue.submit("flux-2-pro", "A sunset", handle_result)
```
### 4. Async with Semaphore
```python
import asyncio
import aiohttp
class AsyncRateLimitedClient:
def __init__(self, api_key, max_concurrent=24):
self.api_key = api_key
self.semaphore = asyncio.Semaphore(max_concurrent)
self.headers = {"x-key": api_key}
async def generate(self, model, prompt):
async with self.semaphore:
async with aiohttp.ClientSession() as session:
# Submit
async with session.post(
f"https://api.bfl.ai/v1/{model}",
headers=self.headers,
json={"prompt": prompt}
) as response:
data = await response.json()
polling_url = data["polling_url"]
# Poll until complete
while True:
async with session.get(
polling_url,
headers=self.headers
) as response:
data = await response.json()
if data["status"] == "Ready":
return data["result"]
elif data["status"] == "Error":
raise Exception(data.get("error"))
await asyncio.sleep(2)
# Usage
async def main():
client = AsyncRateLimitedClient("your-api-key")
# Generate 50 images with rate limiting
prompts = [f"Image {i}" for i in range(50)]
tasks = [client.generate("flux-2-pro", p) for p in prompts]
results = await asyncio.gather(*tasks)
asyncio.run(main())
```
## Monitoring Rate Limits
```python
class RateLimitMonitor:
def __init__(self):
self.requests_made = 0
self.rate_limit_hits = 0
self.lock = Lock()
def record_request(self, response):
with self.lock:
self.requests_made += 1
if response.status_code == 429:
self.rate_limit_hits += 1
def get_stats(self):
return {
"total_requests": self.requests_made,
"rate_limit_hits": self.rate_limit_hits,
"hit_rate": self.rate_limit_hits / max(self.requests_made, 1)
}
```
## Best Practices
1. **Track active requests** - Know how many are in-flight
2. **Implement client-side limits** - Stay under limits proactively
3. **Use semaphores** - Clean way to limit concurrency
4. **Queue for high volume** - Buffer requests when traffic spikes
5. **Monitor headers** - React to remaining quota
6. **Graceful degradation** - Queue or delay when near limits
7. **Different limits per endpoint** - Remember Kontext Max is 6, not 24
## Regional Distribution
For very high volume, consider distributing across regions:
```python
ENDPOINTS = [
"https://api.bfl.ai",
"https://api.eu.bfl.ai",
"https://api.us.bfl.ai"
]
def get_endpoint():
"""Round-robin or least-loaded selection."""
return random.choice(ENDPOINTS)
```
Note: Verify regional rate limits are independent before relying on this strategy.
.claude/skills/bfl-api/references/webhook-integration.md
+339
View File
@@ -0,0 +1,339 @@
---
name: webhook-integration
description: Setting up webhooks for production BFL API integration
---
# Webhook Integration
For production workloads, use webhooks instead of polling to receive generation results.
## Benefits Over Polling
- **Reduced API calls** - No repeated polling requests
- **Immediate notification** - Know exactly when generation completes
- **Better resource efficiency** - No wasted compute on polling
- **Scalable architecture** - Event-driven design
## Setup
### Request with Webhook
Include `webhook_url` and optionally `webhook_secret` in your request:
```bash
curl -X POST "https://api.bfl.ai/v1/flux-2-pro" \
-H "x-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "A beautiful sunset over mountains",
"webhook_url": "https://your-server.com/api/bfl-webhook",
"webhook_secret": "your-secret-key-here"
}'
```
### Webhook Payload
When generation completes, BFL sends a POST request to your webhook URL:
```json
{
"id": "gen_abc123xyz",
"status": "Ready",
"result": {
"sample": "https://bfldeliveryprod.blob.core.windows.net/results/...",
"prompt": "...",
"seed": 1234567890
},
"timestamp": "2025-01-15T10:30:00Z"
}
```
For failures:
```json
{
"id": "gen_abc123xyz",
"status": "Error",
"error": "content_policy_violation",
"message": "The prompt violated content policy",
"timestamp": "2025-01-15T10:30:00Z"
}
```
## Security
### Signature Verification
When `webhook_secret` is provided, BFL signs the payload with HMAC-SHA256:
```
X-BFL-Signature: sha256=<hex-encoded-signature>
```
### Verification Implementation
```python
import hmac
import hashlib
def verify_webhook_signature(payload, signature, secret):
"""Verify the webhook came from BFL."""
if not signature or not signature.startswith('sha256='):
return False
expected_signature = hmac.new(
secret.encode('utf-8'),
payload,
hashlib.sha256
).hexdigest()
provided_signature = signature[7:] # Remove 'sha256=' prefix
return hmac.compare_digest(expected_signature, provided_signature)
```
### Flask Handler with Verification
```python
from flask import Flask, request, jsonify
import hmac
import hashlib
import requests
app = Flask(__name__)
WEBHOOK_SECRET = "your-secret-key-here"
@app.route('/api/bfl-webhook', methods=['POST'])
def handle_webhook():
# Verify signature
signature = request.headers.get('X-BFL-Signature')
if not verify_webhook_signature(request.data, signature, WEBHOOK_SECRET):
return jsonify({'error': 'Invalid signature'}), 401
data = request.json
if data['status'] == 'Ready':
handle_completion(data)
elif data['status'] == 'Error':
handle_failure(data)
return jsonify({'status': 'received'}), 200
def handle_completion(data):
generation_id = data['id']
result_url = data['result']['sample']
# Download image immediately (URL expires in 10 min)
image_data = requests.get(result_url).content
# Store to your storage
store_image(generation_id, image_data)
# Update your database
update_generation_status(generation_id, 'completed')
# Notify your application/users
notify_completion(generation_id)
def handle_failure(data):
generation_id = data['id']
error = data.get('error', 'unknown')
# Log the failure
log_generation_failure(generation_id, error)
# Update your database
update_generation_status(generation_id, 'failed', error)
# Maybe retry or notify
handle_generation_error(generation_id, error)
```
### Express.js Handler
```javascript
const express = require('express');
const crypto = require('crypto');
const axios = require('axios');
const app = express();
app.use(express.raw({ type: 'application/json' }));
const WEBHOOK_SECRET = 'your-secret-key-here';
function verifySignature(payload, signature, secret) {
if (!signature || !signature.startsWith('sha256=')) {
return false;
}
const expectedSignature = crypto
.createHmac('sha256', secret)
.update(payload)
.digest('hex');
const providedSignature = signature.slice(7);
return crypto.timingSafeEqual(
Buffer.from(expectedSignature),
Buffer.from(providedSignature)
);
}
app.post('/api/bfl-webhook', async (req, res) => {
const signature = req.headers['x-bfl-signature'];
if (!verifySignature(req.body, signature, WEBHOOK_SECRET)) {
return res.status(401).json({ error: 'Invalid signature' });
}
const data = JSON.parse(req.body);
if (data.status === 'Ready') {
// Download image (URL expires in 10 min)
const imageResponse = await axios.get(data.result.sample, {
responseType: 'arraybuffer'
});
// Store the image
await storeImage(data.id, imageResponse.data);
}
res.json({ status: 'received' });
});
```
## Requirements
### HTTPS Required
Webhook URLs **must use HTTPS** in production. BFL will not send webhooks to HTTP endpoints.
### Response Requirements
- Respond with 2xx status code to acknowledge receipt
- Respond within 30 seconds
- Keep handler fast - offload heavy processing
### Retry Policy
BFL retries failed webhook deliveries:
| Attempt | Delay |
|---------|-------|
| 1st retry | 1 second |
| 2nd retry | 5 seconds |
| 3rd retry | 30 seconds |
After 3 failed attempts, the webhook is abandoned. Fall back to polling if critical.
## Idempotency
Handle duplicate webhook deliveries:
```python
from functools import lru_cache
import redis
redis_client = redis.Redis()
def is_duplicate_webhook(generation_id):
"""Check if we've already processed this webhook."""
key = f"webhook:processed:{generation_id}"
# Try to set with NX (only if not exists)
was_set = redis_client.set(key, "1", nx=True, ex=3600) # 1 hour TTL
return not was_set # If we couldn't set it, it's a duplicate
@app.route('/api/bfl-webhook', methods=['POST'])
def handle_webhook():
# ... signature verification ...
data = request.json
generation_id = data['id']
if is_duplicate_webhook(generation_id):
return jsonify({'status': 'already_processed'}), 200
# Process webhook...
```
## Hybrid Approach
Combine webhooks with polling fallback:
```python
class HybridClient:
def __init__(self, api_key, webhook_url, webhook_secret):
self.api_key = api_key
self.webhook_url = webhook_url
self.webhook_secret = webhook_secret
self.pending = {} # Track pending generations
def generate(self, prompt, timeout=300):
"""Generate with webhook, fall back to polling."""
response = self._submit(prompt)
generation_id = response['id']
polling_url = response['polling_url']
# Wait for webhook (with timeout)
result = self._wait_for_webhook(generation_id, timeout=timeout)
if result is None:
# Webhook didn't arrive, fall back to polling
result = self._poll(polling_url, timeout=60)
return result
def _submit(self, prompt):
return requests.post(
"https://api.bfl.ai/v1/flux-2-pro",
headers={"x-key": self.api_key},
json={
"prompt": prompt,
"webhook_url": self.webhook_url,
"webhook_secret": self.webhook_secret
}
).json()
def receive_webhook(self, data):
"""Called by webhook handler."""
generation_id = data['id']
if generation_id in self.pending:
self.pending[generation_id].set_result(data)
```
## Monitoring
Track webhook health:
```python
import time
class WebhookMetrics:
def __init__(self):
self.received = 0
self.processed = 0
self.failed = 0
self.avg_latency = 0
def record_webhook(self, generation_id, submit_time):
self.received += 1
latency = time.time() - submit_time
self.avg_latency = (self.avg_latency * (self.received - 1) + latency) / self.received
def record_success(self):
self.processed += 1
def record_failure(self):
self.failed += 1
def get_stats(self):
return {
"received": self.received,
"processed": self.processed,
"failed": self.failed,
"success_rate": self.processed / max(self.received, 1),
"avg_latency_seconds": self.avg_latency
}
```
.claude/skills/create-video/SKILL.md
+100
View File
@@ -0,0 +1,100 @@
---
name: create-video
description: |
Create videos from a text prompt using HeyGen's Video Agent. Use when: (1) Creating a video from a description or idea, (2) Generating explainer, demo, or marketing videos from a prompt, (3) Making a video without specifying exact avatars, voices, or scenes, (4) Quick video prototyping or drafts, (5) One-shot prompt-to-video generation, (6) User says "make me a video" or "create a video about X".
homepage: https://docs.heygen.com/reference/generate-video-agent
allowed-tools: mcp__heygen__*
metadata:
openclaw:
requires:
env:
- HEYGEN_API_KEY
primaryEnv: HEYGEN_API_KEY
---
# Create Video
Generate complete videos from a text prompt. Describe what you want and the AI handles script writing, avatar selection, visuals, voiceover, pacing, and captions automatically.
## Authentication
All requests require the `X-Api-Key` header. Set the `HEYGEN_API_KEY` environment variable.
```bash
curl -X POST "https://api.heygen.com/v1/video_agent/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{"prompt": "Create a 60-second product demo video."}'
```
## Tool Selection
If HeyGen MCP tools are available (`mcp__heygen__*`), **prefer them** over direct HTTP API calls — they handle authentication and request formatting automatically.
| Task | MCP Tool | Fallback (Direct API) |
|------|----------|----------------------|
| Generate video from prompt | `mcp__heygen__generate_video_agent` | `POST /v1/video_agent/generate` |
| Check video status / get URL | `mcp__heygen__get_video` | `GET /v2/videos/{video_id}` |
| List account videos | `mcp__heygen__list_videos` | `GET /v2/videos` |
| Delete a video | `mcp__heygen__delete_video` | `DELETE /v2/videos/{video_id}` |
If no HeyGen MCP tools are available, use direct HTTP API calls as documented in the reference files.
## Default Workflow
Always use [prompt-optimizer.md](references/prompt-optimizer.md) guidelines to structure prompts with scenes, timing, and visual styles.
**With MCP tools:**
1. Write an optimized prompt using [prompt-optimizer.md](references/prompt-optimizer.md) → [visual-styles.md](references/visual-styles.md)
2. Call `mcp__heygen__generate_video_agent` with prompt and config (duration_sec, orientation, avatar_id)
3. Call `mcp__heygen__get_video` with the returned video_id to poll status and get the download URL
**Without MCP tools (direct API):**
1. Write an optimized prompt using [prompt-optimizer.md](references/prompt-optimizer.md) → [visual-styles.md](references/visual-styles.md)
2. `POST /v1/video_agent/generate` — see [video-agent.md](references/video-agent.md)
3. `GET /v2/videos/<id>` — see [video-status.md](references/video-status.md)
## Quick Reference
| Task | MCP Tool | Read |
|------|----------|------|
| Generate video from prompt | `mcp__heygen__generate_video_agent` | [prompt-optimizer.md](references/prompt-optimizer.md) → [visual-styles.md](references/visual-styles.md) → [video-agent.md](references/video-agent.md) |
| Check video status / get download URL | `mcp__heygen__get_video` | [video-status.md](references/video-status.md) |
| Upload reference files for prompt | — | [assets.md](references/assets.md) |
## When to Use This Skill vs Avatar Video
This skill is for **prompt-based video creation** — describe what you want, and the AI handles the rest.
If the user needs **precise control** over specific avatars, exact scripts, per-scene voice/background configuration, or multi-scene composition, use the **avatar-video** skill instead.
| User Says | This Skill | Avatar Video Skill |
|-----------|:----------:|:------------------:|
| "Make me a video about X" | ✓ | |
| "Create a product demo" | ✓ | |
| "I want avatar Y to say exactly Z" | | ✓ |
| "Multi-scene video with different backgrounds" | | ✓ |
| "Transparent WebM for compositing" | | ✓ |
## Reference Files
### Core Workflow
- [references/prompt-optimizer.md](references/prompt-optimizer.md) - Writing effective prompts (core workflow + rules)
- [references/visual-styles.md](references/visual-styles.md) - 20 named visual styles with full specs
- [references/prompt-examples.md](references/prompt-examples.md) - Full production prompt example + ready-to-use templates
- [references/video-agent.md](references/video-agent.md) - Video Agent API endpoint details
### Foundation
- [references/video-status.md](references/video-status.md) - Polling patterns and download URLs
- [references/webhooks.md](references/webhooks.md) - Webhook endpoints and events
- [references/assets.md](references/assets.md) - Uploading images, videos, audio as references
- [references/dimensions.md](references/dimensions.md) - Resolution and aspect ratios
- [references/quota.md](references/quota.md) - Credit system and usage limits
## Best Practices
1. **Optimize your prompt** — The difference between mediocre and professional results depends entirely on prompt quality. Always use the prompt optimizer
2. **Specify duration** — Use `config.duration_sec` for predictable length
3. **Lock avatar if needed** — Use `config.avatar_id` for consistency across videos
4. **Upload reference files** — Help the agent understand your brand/product
5. **Iterate on prompts** — Refine based on results; Video Agent is great for quick iterations
.claude/skills/create-video/references/assets.md
+339
View File
@@ -0,0 +1,339 @@
---
name: assets
description: Uploading images, videos, and audio for use in HeyGen video generation
---
# Asset Upload and Management
HeyGen allows you to upload custom assets (images, videos, audio) for use in video generation, such as backgrounds, talking photo sources, and custom audio.
## Upload Flow
Asset uploads are a single-step process: POST the raw file binary directly to the upload endpoint. The Content-Type header must match the file's MIME type.
## Uploading an Asset
**Endpoint:** `POST https://upload.heygen.com/v1/asset`
### Request
| Header | Required | Description |
|--------|:--------:|-------------|
| `X-Api-Key` | ✓ | Your HeyGen API key |
| `Content-Type` | ✓ | MIME type of the file (e.g. `image/jpeg`) |
The request body is the raw binary file data. No JSON or form fields are needed.
### Response
| Field | Type | Description |
|-------|------|-------------|
| `code` | number | Status code (`100` = success) |
| `data.id` | string | Unique asset ID for use in video generation |
| `data.name` | string | Asset name |
| `data.file_type` | string | `image`, `video`, or `audio` |
| `data.url` | string | Accessible URL for the uploaded file |
| `data.image_key` | string \| null | Key for creating uploaded photo avatars (images only) |
| `data.folder_id` | string | Folder ID (empty if not in a folder) |
| `data.meta` | string \| null | Asset metadata |
| `data.created_ts` | number | Unix timestamp of creation |
### curl
```bash
curl -X POST "https://upload.heygen.com/v1/asset" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: image/jpeg" \
--data-binary '@./background.jpg'
```
### TypeScript
```typescript
import fs from "fs";
import path from "path";
interface AssetUploadResponse {
code: number;
data: {
id: string;
name: string;
file_type: string;
url: string;
image_key: string | null;
folder_id: string;
meta: string | null;
created_ts: number;
};
msg: string | null;
message: string | null;
}
async function uploadAsset(filePath: string, contentType: string): Promise<AssetUploadResponse["data"]> {
const resolvedPath = path.resolve(filePath);
const fileBuffer = fs.readFileSync(resolvedPath);
const response = await fetch("https://upload.heygen.com/v1/asset", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": contentType,
},
body: fileBuffer,
});
const json: AssetUploadResponse = await response.json();
if (json.code !== 100) {
throw new Error(json.message ?? "Upload failed");
}
return json.data;
}
// Usage
const asset = await uploadAsset("./background.jpg", "image/jpeg");
console.log(`Uploaded asset: ${asset.id}`);
console.log(`Asset URL: ${asset.url}`);
```
### TypeScript (with streams for large files)
```typescript
import fs from "fs";
import path from "path";
import { stat } from "fs/promises";
async function uploadLargeAsset(filePath: string, contentType: string): Promise<AssetUploadResponse["data"]> {
const resolvedPath = path.resolve(filePath);
const fileStats = await stat(resolvedPath);
const fileStream = fs.createReadStream(resolvedPath);
const response = await fetch("https://upload.heygen.com/v1/asset", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": contentType,
"Content-Length": fileStats.size.toString(),
},
body: fileStream as any,
// @ts-ignore - duplex is needed for streaming
duplex: "half",
});
const json: AssetUploadResponse = await response.json();
if (json.code !== 100) {
throw new Error(json.message ?? "Upload failed");
}
return json.data;
}
```
### Python
```python
import requests
import os
def upload_asset(file_path: str, content_type: str) -> dict:
with open(file_path, "rb") as f:
response = requests.post(
"https://upload.heygen.com/v1/asset",
headers={
"X-Api-Key": os.environ["HEYGEN_API_KEY"],
"Content-Type": content_type
},
data=f
)
data = response.json()
if data.get("code") != 100:
raise Exception(data.get("message", "Upload failed"))
return data["data"]
# Usage
asset = upload_asset("./background.jpg", "image/jpeg")
print(f"Uploaded asset: {asset['id']}")
print(f"Asset URL: {asset['url']}")
```
## Supported Content Types
| Type | Content-Type | Use Case |
|------|--------------|----------|
| JPEG | `image/jpeg` | Backgrounds, talking photos |
| PNG | `image/png` | Backgrounds, overlays |
| MP4 | `video/mp4` | Video backgrounds |
| WebM | `video/webm` | Video backgrounds |
| MP3 | `audio/mpeg` | Custom audio input |
| WAV | `audio/wav` | Custom audio input |
## Uploading from URL
If your asset is already hosted online:
```typescript
async function uploadFromUrl(sourceUrl: string, contentType: string): Promise<AssetUploadResponse["data"]> {
// 1. Validate and download the file
const url = new URL(sourceUrl);
if (url.protocol !== "https:") {
throw new Error("Only HTTPS URLs are supported");
}
const sourceResponse = await fetch(sourceUrl);
const buffer = Buffer.from(await sourceResponse.arrayBuffer());
// 2. Upload directly to HeyGen
const response = await fetch("https://upload.heygen.com/v1/asset", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": contentType,
},
body: buffer,
});
const json: AssetUploadResponse = await response.json();
if (json.code !== 100) {
throw new Error(json.message ?? "Upload failed");
}
return json.data;
}
```
## Using Uploaded Assets
### As Background Image
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello, this is a video with a custom background!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: asset.url, // Use the URL from the upload response
},
},
],
};
```
### As Talking Photo Source
```typescript
const talkingPhotoConfig = {
video_inputs: [
{
character: {
type: "talking_photo",
talking_photo_id: asset.id, // Use the ID from the upload response
},
voice: {
type: "text",
input_text: "Hello from my talking photo!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
};
```
### As Audio Input
```typescript
const audioConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "audio",
audio_url: asset.url, // Use the URL from the upload response
},
},
],
};
```
## Complete Upload Workflow
```typescript
async function createVideoWithCustomBackground(
backgroundPath: string,
script: string
): Promise<string> {
// 1. Upload background
console.log("Uploading background...");
const background = await uploadAsset(backgroundPath, "image/jpeg");
// 2. Create video config
const config = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: background.url,
},
},
],
dimension: { width: 1920, height: 1080 },
};
// 3. Generate video
console.log("Generating video...");
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
});
const { data } = await response.json();
return data.video_id;
}
```
## Asset Limitations
- **File size**: 10MB maximum
- **Image dimensions**: Recommended to match video dimensions
- **Audio duration**: Should match expected video length
- **Retention**: Assets may be deleted after a period of inactivity
## Best Practices
1. **Optimize images** - Resize to match video dimensions before uploading
2. **Use appropriate formats** - JPEG for photos, PNG for graphics with transparency
3. **Validate before upload** - Check file type and size locally first
4. **Handle upload errors** - Implement retry logic for failed uploads
5. **Cache asset IDs** - Reuse assets across multiple video generations
.claude/skills/create-video/references/dimensions.md
+314
View File
@@ -0,0 +1,314 @@
---
name: dimensions
description: Resolution options (720p/1080p) and aspect ratios for HeyGen videos
---
# Video Dimensions and Resolution
HeyGen supports various video dimensions and aspect ratios to fit different platforms and use cases.
## Standard Resolutions
### Landscape (16:9)
| Resolution | Width | Height | Use Case |
|------------|-------|--------|----------|
| 720p | 1280 | 720 | Standard quality, faster processing |
| 1080p | 1920 | 1080 | High quality, most common |
### Portrait (9:16)
| Resolution | Width | Height | Use Case |
|------------|-------|--------|----------|
| 720p | 720 | 1280 | Mobile-first content |
| 1080p | 1080 | 1920 | High quality vertical |
### Square (1:1)
| Resolution | Width | Height | Use Case |
|------------|-------|--------|----------|
| 720p | 720 | 720 | Social media posts |
| 1080p | 1080 | 1080 | High quality square |
## Setting Dimensions
### TypeScript
```typescript
// Landscape 1080p
const landscapeConfig = {
video_inputs: [...],
dimension: {
width: 1920,
height: 1080
}
};
// Portrait 1080p
const portraitConfig = {
video_inputs: [...],
dimension: {
width: 1080,
height: 1920
}
};
// Square 1080p
const squareConfig = {
video_inputs: [...],
dimension: {
width: 1080,
height: 1080
}
};
```
### curl
```bash
# Landscape 1080p
curl -X POST "https://api.heygen.com/v2/video/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"video_inputs": [...],
"dimension": {
"width": 1920,
"height": 1080
}
}'
```
## Dimension Helper Functions
```typescript
type AspectRatio = "16:9" | "9:16" | "1:1" | "4:3" | "4:5";
type Quality = "720p" | "1080p";
interface Dimensions {
width: number;
height: number;
}
function getDimensions(aspectRatio: AspectRatio, quality: Quality): Dimensions {
const configs: Record<AspectRatio, Record<Quality, Dimensions>> = {
"16:9": {
"720p": { width: 1280, height: 720 },
"1080p": { width: 1920, height: 1080 },
},
"9:16": {
"720p": { width: 720, height: 1280 },
"1080p": { width: 1080, height: 1920 },
},
"1:1": {
"720p": { width: 720, height: 720 },
"1080p": { width: 1080, height: 1080 },
},
"4:3": {
"720p": { width: 960, height: 720 },
"1080p": { width: 1440, height: 1080 },
},
"4:5": {
"720p": { width: 576, height: 720 },
"1080p": { width: 864, height: 1080 },
},
};
return configs[aspectRatio][quality];
}
// Usage
const youTubeDimensions = getDimensions("16:9", "1080p");
const tikTokDimensions = getDimensions("9:16", "1080p");
const instagramDimensions = getDimensions("1:1", "1080p");
```
## Platform-Specific Recommendations
### YouTube
```typescript
const youtubeConfig = {
video_inputs: [...],
dimension: { width: 1920, height: 1080 }, // 16:9 landscape
};
```
### TikTok / Instagram Reels / YouTube Shorts
```typescript
const shortFormConfig = {
video_inputs: [...],
dimension: { width: 1080, height: 1920 }, // 9:16 portrait
};
```
### Instagram Feed Post
```typescript
const instagramFeedConfig = {
video_inputs: [...],
dimension: { width: 1080, height: 1080 }, // 1:1 square
};
```
### LinkedIn
```typescript
const linkedinConfig = {
video_inputs: [...],
dimension: { width: 1920, height: 1080 }, // 16:9 landscape preferred
};
```
### Twitter/X
```typescript
const twitterConfig = {
video_inputs: [...],
dimension: { width: 1280, height: 720 }, // 16:9, 720p is common
};
```
## Avatar IV Dimensions
For Avatar IV (photo-based avatars), dimensions are set via orientation:
```typescript
type VideoOrientation = "portrait" | "landscape" | "square";
function getAvatarIVDimensions(orientation: VideoOrientation): Dimensions {
switch (orientation) {
case "portrait":
return { width: 720, height: 1280 };
case "landscape":
return { width: 1280, height: 720 };
case "square":
return { width: 720, height: 720 };
}
}
```
## Custom Dimensions
HeyGen supports custom dimensions within limits:
```typescript
const customConfig = {
video_inputs: [...],
dimension: {
width: 1600,
height: 900 // Custom 16:9 at non-standard resolution
}
};
```
### Dimension Constraints
- **Minimum**: 128px on any side
- **Maximum**: 4096px on any side
- **Must be even numbers**: Both width and height must be divisible by 2
```typescript
function validateDimensions(width: number, height: number): boolean {
if (width < 128 || height < 128) {
throw new Error("Dimensions must be at least 128px");
}
if (width > 4096 || height > 4096) {
throw new Error("Dimensions cannot exceed 4096px");
}
if (width % 2 !== 0 || height % 2 !== 0) {
throw new Error("Dimensions must be even numbers");
}
return true;
}
```
## Resolution vs. Credit Cost
Higher resolutions may consume more credits:
| Resolution | Relative Cost |
|------------|---------------|
| 720p | Base rate |
| 1080p | ~1.5x base rate |
Consider using 720p for drafts and testing, then 1080p for final output.
## Background Considerations
Match background image/video dimensions to your video dimensions:
```typescript
// For 1080p landscape video
const config = {
video_inputs: [
{
character: {...},
voice: {...},
background: {
type: "image",
url: "https://example.com/1920x1080-background.jpg" // Match video dimensions
}
}
],
dimension: { width: 1920, height: 1080 }
};
```
## Creating a Video Config Factory
```typescript
interface VideoConfigOptions {
script: string;
avatarId: string;
voiceId: string;
platform: "youtube" | "tiktok" | "instagram_feed" | "instagram_story" | "linkedin";
quality?: "720p" | "1080p";
}
function createVideoConfig(options: VideoConfigOptions) {
const platformDimensions: Record<string, Dimensions> = {
youtube: { width: 1920, height: 1080 },
tiktok: { width: 1080, height: 1920 },
instagram_feed: { width: 1080, height: 1080 },
instagram_story: { width: 1080, height: 1920 },
linkedin: { width: 1920, height: 1080 },
};
const dimension = platformDimensions[options.platform];
// Scale down for 720p if requested
if (options.quality === "720p") {
dimension.width = Math.round((dimension.width * 720) / 1080);
dimension.height = Math.round((dimension.height * 720) / 1080);
}
return {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: options.avatarId,
avatar_style: "normal",
},
voice: {
type: "text",
input_text: options.script,
voice_id: options.voiceId,
},
},
],
dimension,
};
}
// Usage
const tiktokVideo = createVideoConfig({
script: "Hey everyone! Check this out!",
avatarId: "josh_lite3_20230714",
voiceId: "1bd001e7e50f421d891986aad5158bc8",
platform: "tiktok",
quality: "1080p",
});
```
.claude/skills/create-video/references/prompt-examples.md
+206
View File
@@ -0,0 +1,206 @@
---
name: prompt-examples
description: Full production prompt examples and ready-to-use templates for Video Agent
---
# Video Agent Prompt Examples
## Full Example: Brief to Production Prompt
### Input Brief
```
Topic: Monthly company report for a SaaS startup
Key data: $141M ARR (up from $54M), 1.85M signups (+28%), 3M paid videos/month
Customer story: Creator built AI character, 2.5M followers, 20 min/video
Challenge: Organic traffic volatile, -16% last week
Duration: ~90 seconds
Tone: Confident CEO, data-backed
```
### Output Prompt
```
FORMAT: Bloomberg-style company report. 90 seconds. Fast-paced, data-dense.
Record-breaking month. Proud but analytical.
TONE: Confident, direct, data-backed. Highlights hit hard with numbers.
Customer stories are the emotional core. Challenges are honest — no spin.
AVATAR: Man in simple black crew-neck tee, standing in a modern glass-walled
office at golden hour. Behind him, a wall-mounted display shows the company logo
in soft blue glow. Monitor to his right shows a dashboard with upward-trending
charts. Desk beside him: laptop, half-empty flat white, scattered sticky notes.
Warm afternoon light through floor-to-ceiling windows, long shadows on polished
concrete. Minimal, focused startup HQ.
STYLE — SWISS PULSE (Müller-Brockmann): Grid-locked compositions. Black (#1a1a1a),
white, electric blue (#0066FF), warm amber (#FF9500) for records. Helvetica Bold
headlines, Regular labels. Numbers LARGE. Animated counters count up from 0.
Diagonal compositions on accent moments. Grid wipe transitions. No dissolves.
CRITICAL ON-SCREEN TEXT (display literally):
- "1.85M SIGNUPS — +28% MoM"
- "$2.12M NEW SUBSCRIPTION REVENUE"
- "$54M → $141M ARR"
- "2.5M FOLLOWERS" and "20 MIN / VIDEO"
- Quote: "Use technology to serve the message, not distract from it."
- "ORGANIC: 65% OF SUBS — VOLATILE"
MUSIC: Upbeat electronic with a driving beat. Tycho meets Bloomberg opening theme.
Builds through highlights, warms for customer story, softens for challenges, peaks
on close.
---
SCENE 1 — A-ROLL (8s)
[Avatar center-frame, energetic, leaning slightly forward]
VOICEOVER: "January was a record month. New highs across acquisition, revenue,
and product velocity. Here's the full picture."
Lower-third SLIDES in: "COMPANY NAME | JANUARY 2026" white on blue bar.
Grid wipe.
SCENE 2 — FULL SCREEN B-ROLL (12s)
[NO AVATAR — motion graphic only]
VOICEOVER: "One-point-eight-five million signups — twenty-eight percent month
over month. Two-point-one-two million in new subscription revenue. Both all-time
highs."
LAYER 1: Dark #1a1a1a background with thin grid lines pulsing at 8% opacity.
LAYER 2: "1.85M" SLAMS in from left, white Bold 140pt. "SIGNUPS" types on
in electric blue 32pt uppercase. "+28% MoM" appears in amber.
LAYER 3: Three stat cards CASCADE from top-right, staggered 0.3s:
"$2.12M New Revenue" — "$3.4M Business ARR" — "$3M Pro ARR."
Each number COUNTS UP from 0.
LAYER 4: Bottom ticker scrolls: "Non-brand search +36% • Brand impressions 9.2M
• Weekly subs +20.5%"
LAYER 5: Grid lines RIPPLE outward on "1.85M" slam. Diagonal amber bar behind
stat cards.
Hard cut.
SCENE 3 — FULL SCREEN B-ROLL (12s)
[NO AVATAR — motion graphic only]
VOICEOVER: "Zoom out. Twelve months ago — fifty-four million ARR. Today —
one hundred forty-one million. Nearly three X in a single year."
LAYER 1: Dark background, subtle grid scrolling upward.
LAYER 2: Animated line chart DRAWS ITSELF left to right. Y-axis: $50M to $150M.
Final point "$140.84M" glows amber and pulses.
LAYER 3: Milestone annotations float in at key data points.
LAYER 4: Second smaller chart below — "Paid Videos" 0.91M to 2.97M, same style.
LAYER 5: Thin grid lines converge toward final data point. Scan line sweeps.
Grid wipe.
SCENE 4 — A-ROLL (8s)
[Avatar center-frame, warm tone, genuine smile]
VOICEOVER: "But the numbers only tell half the story. The other half is the
people building on the platform."
Lower-third: "Customer Spotlight"
SCENE 5 — FULL SCREEN B-ROLL (12s)
[NO AVATAR — warm palette]
VOICEOVER: "An AI character built entirely on the platform. Twenty minutes
per video. Two-point-five million Instagram followers. The creator's principle:
use technology to serve the message, not distract from it."
LAYER 1: Dark background with warm amber grid lines at low opacity.
LAYER 2: "CHARACTER NAME" in large white, center-top, 80pt.
LAYER 3: Stats cascade from right: "2.5M Followers" COUNTS UP in amber —
"20 min/video" — "7x Faster." Each a glowing node.
LAYER 4: Quote card SLIDES UP: "Use technology to serve the message, not
distract from it." Types on word by word.
LAYER 5: Warm light bloom. Grid lines soften into curved arcs.
Grid wipe.
SCENE 6 — A-ROLL (10s)
[Avatar center-frame, serious/candid]
VOICEOVER: "Now the honest part. Organic drives sixty-five percent of
subscriptions and it's volatile. Non-brand traffic dropped sixteen percent
last week. We've rebuilt attribution and we're investing in SEO."
Lower-third: "Challenges"
SCENE 7 — A-ROLL (7s)
[Avatar center-frame, energy lifts, direct eye contact]
VOICEOVER: "Fifty-four million to one-forty-one in twelve months. Three million
paid videos a month. January set the bar — now we raise it."
End card: Logo centered, blue glow fade-in. Grid lines converge. Music peaks.
---
NARRATION STYLE: CEO energy — conviction backed by data. Fast on highlights.
Warm on customer stories. Candid on challenges. Close with forward momentum.
```
## Ready-to-Use Templates
### Tech News Briefing
```
FORMAT: 75-second high-energy tech briefing. Think: Bloomberg meets Vice.
AVATAR: [Presenter in tech-casual at a multi-monitor station.
Describe clothing, monitor content, desk items, lighting.]
STYLE — DECONSTRUCTED (Brody): Dark grey #1a1a1a, rust orange #D4501E.
Type at angles, overlapping. Gritty textures. Smash cut transitions.
CRITICAL ON-SCREEN TEXT:
- [List every stat, quote, handle that must appear]
SCENE 1 — A-ROLL (8s): Hook with energy. State what's happening.
SCENE 2 — B-ROLL (12s): First story with layered visuals (L1-L5).
SCENE 3 — A-ROLL + OVERLAY (10s): Second story, split frame.
SCENE 4 — B-ROLL (10s): Third story or dramatic data point.
SCENE 5 — A-ROLL (8s): Wrap-up and forward look.
```
### Product Comparison
```
FORMAT: 60-second comparison. [Product A] vs [Product B]. Data-driven.
AVATAR: [Presenter in review studio. Desk with both products visible.]
STYLE — DIGITAL GRID (Crouwel): Dark #0a0a0a, cyan #00D4FF and amber #FFB800.
Two-color coding: cyan = Product A, amber = Product B. Monospaced type.
CRITICAL ON-SCREEN TEXT:
- [Key stats for each product]
- [Pricing, features, differentiators]
Use SPLIT FRAME B-roll: Product A left, Product B right.
```
### Strategy Presentation
```
FORMAT: 90-second strategy briefing. Bloomberg meets board meeting.
AVATAR: [Executive in blazer over tee. Conference room with whiteboard frameworks.]
STYLE — SWISS PULSE (Müller-Brockmann): Black/white + blue #0066FF.
Grid-locked. Helvetica. Animated counters. Grid wipe transitions.
CRITICAL ON-SCREEN TEXT:
- [Framework labels, quadrant labels, key quotes]
Build frameworks visually: draw axes, plot positions, animate labels.
```
### Social Ad (30 seconds)
```
FORMAT: 30-second social ad. Maximum energy. Portrait 9:16.
AVATAR: [Creator-style presenter. Ring light, colorful background.]
STYLE — CARNIVAL SURGE (Lins): Hot pink, yellow, teal. Collage layering.
Text MASSIVE at angles. Confetti. Smash cuts.
Three scenes: Hook (8s) → Value prop (12s) → CTA (10s).
Text fills 50-80% of every frame. Numbers SLAM.
```
### Premium Report
```
FORMAT: 120-second investor-grade report. Understated authority.
AVATAR: [Tailored merino sweater. Architectural room, diffused natural light.]
STYLE — VELVET STANDARD (Vignelli): Black, white, gold #c9a84c.
Thin ALL CAPS, wide spacing. Generous negative space.
Slow cross-dissolves. Numbers fade in with weight.
```
.claude/skills/create-video/references/prompt-optimizer.md
+289
View File
@@ -0,0 +1,289 @@
---
name: prompt-optimizer
description: Write production-quality prompts for HeyGen Video Agent — from basic ideas to fully art-directed scene-by-scene scripts
---
# Video Agent Prompt Optimizer
Write effective prompts for the HeyGen Video Agent API. Based on patterns from 40+ produced videos.
**The core insight: Video Agent is an HTML interpreter.** It renders layouts, typography, and structured content natively. Describe B-roll as layered text motion graphics with action verbs ("slams in," "types on," "counts up") — not layout specs ("upper-left, 48pt").
## Reference Files
| File | Load when... |
|------|-------------|
| [visual-styles.md](visual-styles.md) | Choosing a visual style (20 styles with full specs) |
| [prompt-examples.md](prompt-examples.md) | Writing a prompt from scratch (full production example + templates) |
## Workflow: Brief to Prompt
1. **Pull data** — Research the topic: web search, APIs, internal docs. Gather real quotes, stats, handles
2. **Synthesize a thesis** — Not a list. A story. *"X is happening because Y — here's the proof."* Group into 3-5 themes with a narrative arc
3. **Choose a style** — Match mood first, content second. Ask: *"What should the viewer FEEL?"* See [visual-styles.md](visual-styles.md)
4. **Write the avatar** — Thematic wardrobe matching content's emotional context. Brand logos and content-specific props in the set (see Avatar Guide below)
5. **Extract critical text** — List every number, quote, handle, and label that must appear literally
6. **Break into scenes** — One concept per scene. Rotate scene types. Never 3+ of same type in a row. At least 2 pure B-roll scenes
7. **Write voiceover** — Spell out numbers in VO ("one-point-eight-five million"), use figures on screen ("1.85M"). Narration on EVERY scene including B-roll
8. **Layer each B-roll scene** — L1 background, L2 hero, L3 supporting, L4 info bar, L5 effects. Every element must MOVE
9. **Add music direction** — Reference artists, describe energy arc
10. **Add narration style** — How to deliver: fast/slow, where to pause, emotional register per section
## Prompt Anatomy
Every production-quality prompt follows this structure:
```
FORMAT: What kind of video, how long, what energy
TONE: Emotional register, references
AVATAR: Detailed physical + environment description (60-100 words)
STYLE: Named aesthetic with colors, typography, motion rules, transitions
CRITICAL ON-SCREEN TEXT: Exact strings that must appear
SCENE-BY-SCENE: Individual scene breakdowns with VO and layered visuals
MUSIC: Genre, reference artists, energy arc
NARRATION STYLE: How to deliver the voiceover
```
### FORMAT
```
FORMAT: 75-second high-energy tech daily briefing. Think: a creator who just got amazing news.
FORMAT: Bloomberg-style strategy briefing. 100-120 seconds. CEO-delivered.
```
### TONE
```
TONE: Confident, direct, data-backed. Highlights hit hard. Lowlights are honest — no spin.
TONE: Edgy, punk tech commentary. Vice News meets The Face magazine — raw, confrontational.
```
### CRITICAL ON-SCREEN TEXT
List every exact string that must appear on screen. Without this, the agent may summarize, round numbers, or rephrase quotes.
```
CRITICAL ON-SCREEN TEXT (display literally):
- "$141M ARR — All-Time High"
- "1.85M Signups — +28% MoM"
- Quote: "Use technology to serve the message, not distract from it." — Shalev Hani
- "@username" — exact social handle
```
### MUSIC & NARRATION
```
MUSIC: Driving electronic, heavy bass drops on key numbers. Run the Jewels meets
a tech keynote. Builds relentlessly, only softens for customer stories.
NARRATION STYLE: High energy throughout. Let numbers PUNCH — pause before big ones,
then deliver hard. Customer stories get warmth. The close should feel like a mic drop.
```
## Avatar Description Guide
**The avatar is NOT a fixed headshot** — design it for each video like a movie character. Think costume designer + set designer.
### Thematic Wardrobe Rule
The avatar's outfit and environment MUST match the content's emotional/cultural context:
| Content Type | Avatar Design | NOT This |
|---|---|---|
| Chinese New Year | Red qipao with gold embroidery, lantern-lit courtyard | "Reporter in a blazer" |
| Breaking tech news | Field reporter, windswept hair, earpiece, city skyline | "Anchor at a desk" |
| Sleep science | Oversized cream knit, cross-legged on bed, warm lamp | "Analyst in a lab" |
| Reddit community | Messy desk, Reddit alien on monitors, upvote arrows on wall | "Researcher in a studio" |
### What to Specify
| Element | Weak | Strong |
|---------|------|--------|
| Clothing | "Business casual" | "Black ribbed merino turtleneck, high collar framing jaw" |
| Environment | "An office" | "Glass-walled conference room. Whiteboard with hand-drawn tier pyramid" |
| Monitor content | "Computer screens" | "Monitor shows scrolling green terminal text and red security alerts" |
| Lighting | "Well lit" | "Cool blue monitor glow from left, warm amber desk lamp from right" |
### Template
```
AVATAR: [Clothing — fabric, color, fit, accessories, posture].
[Setting — specific props, brand logos, what's on the walls].
[Monitors/desk — content visible on screens, items on desk].
[Lighting — direction, color temperature]. [Mood of the space].
60-100 words. 3+ content-specific props. Brand elements visible.
```
## Scene Types
| Type | Format | When to Use |
|------|--------|-------------|
| **A-ROLL** | Avatar speaking to camera | Intros, key insights, CTAs, emotional beats |
| **FULL SCREEN B-ROLL** | No avatar — motion graphics only | Data visualization, information-dense content |
| **A-ROLL + OVERLAY** | Split frame: avatar + content | Presenting data while maintaining human connection |
**Rotation is mandatory.** Never 3+ of the same type in a row. Every prompt needs at least 2 pure B-roll scenes.
**Voiceover on EVERY scene.** Every B-roll scene MUST include a `VOICEOVER:` line. Silent B-roll = broken video.
### Scene Anatomy
**A-ROLL:**
```
SCENE 1 — A-ROLL (10s)
[Avatar center-frame, excited, hands gesturing]
VOICEOVER: "The exact script for this scene."
Lower-third: "TITLE TEXT" white on blue bar.
```
**B-ROLL with layers:**
```
SCENE 2 — FULL SCREEN B-ROLL (12s)
[NO AVATAR — motion graphic only]
VOICEOVER: "The exact script for this scene."
LAYER 1: Dark #1a1a1a background with subtle grid lines pulsing.
LAYER 2: "HEADLINE" SLAMS in from left in white Bold 100pt at -5 degrees.
LAYER 3: Three data cards CASCADE from right, staggered 0.3s.
LAYER 4: Bottom ticker SLIDES in: "supporting text scrolling continuously."
LAYER 5: Grid lines RIPPLE outward from impact point.
Hard cut.
```
**A-ROLL + OVERLAY:**
```
SCENE 3 — A-ROLL + OVERLAY (10s)
[SPLIT — Avatar LEFT 35%. Content RIGHT 65%. NO overlap.]
Avatar gestures toward content side.
VOICEOVER: "The exact script for this scene."
RIGHT SIDE: "HEADLINE" in cyan 60pt. Three stats COUNT UP below.
```
Alternate which side the avatar appears on between overlay scenes.
## The Visual Layer System
Break B-roll into 5 stacked layers. This is the most powerful technique for motion graphics scenes.
| Layer | Purpose | Examples |
|-------|---------|---------|
| **L1** | Background | Textured surface, grid, gradient, color field |
| **L2** | Hero content | Main headline/number that dominates the frame |
| **L3** | Supporting data | Cards, stats, bullet points, secondary information |
| **L4** | Information bar | Tickers, labels, source attributions, quotes |
| **L5** | Effects | Particles, glitches, grid animations, ambient motion |
Every B-roll: 4+ layers. Every overlay content side: 3+ layers. **Every element must MOVE.**
## Motion Vocabulary
### High Energy
| Verb | Example |
|------|---------|
| **SLAMS** | `"$95M" SLAMS in from left at -5 degrees` |
| **CRASHES** | `Title CRASHES in from right, screen-shake on impact` |
| **PUNCHES** | `Quote card PUNCHES up from bottom` |
| **STAMPS** | `Data blocks STAMP in staggered 0.4s` |
| **SHATTERS** | `Text SHATTERS after 1.5s, revealing number underneath` |
### Medium Energy
| Verb | Example |
|------|---------|
| **CASCADE** | `Three cards CASCADE from top, staggered 0.3s` |
| **SLIDES** | `Ticker SLIDES in from right — continuous scroll` |
| **DROPS** | `"TIER 1" DROPS in with white flash` |
| **FILLS** | `Progress bar FILLS 0 to 90% in orange` |
| **DRAWS** | `Chart line DRAWS itself left to right` |
### Low Energy
| Verb | Example |
|------|---------|
| **types on** | `Quote types on word by word in italic white` |
| **fades in** | `Logo fades in at center, held for 3 seconds` |
| **FLOATS** | `Bokeh orbs FLOAT across frame at different speeds` |
| **morphs** | `Number morphs from 17 to 18.9` |
| **COUNTS UP** | `"1.85M" COUNTS UP from 0 in amber 96pt` |
## Transition Types
| Transition | Energy | Styles It Fits |
|------------|--------|---------------|
| Smash cut | Aggressive | Deconstructed, Maximalist, Carnival Surge |
| White flash frame | Punchy | Deconstructed, Maximalist |
| Grid wipe | Systematic | Swiss Pulse, Digital Grid |
| Hard cut | Clean | Swiss Pulse, Shadow Cut |
| Liquid dissolve | Elegant | Data Drift, Dream State |
| Slow cross-dissolve | Refined | Velvet Standard |
| Pop cut / bounce | Fun | Play Mode, Carnival Surge |
| Snap cut | Urgent | Red Wire, Contact Sheet |
| Soft dissolve | Warm | Soft Signal, Warm Grain, Quiet Drama |
| Iris wipe | Nostalgic | Heritage Reel |
## Timing Guidelines
| Content Type | Duration |
|--------------|----------|
| Hook/Intro (A-roll) | 6-10 seconds |
| Data-heavy B-roll | 10-15 seconds (NEVER ≤5s — causes black frames) |
| A-roll + Overlay | 8-12 seconds |
| CTA / Close (A-roll) | 6-8 seconds |
**Common video lengths:** Social clip: 30-45s (5-7 scenes) | Briefing: 60-75s (7-9 scenes) | Deep dive: 90-120s (10-13 scenes)
**Speaking pace:** ~150 words/minute. Calculate: `words / 150 * 60 = seconds`
## What Doesn't Work
Patterns that consistently produce poor results:
**Layout language** — Screen coordinates cause empty/black B-roll:
```
❌ "UPPER-LEFT: headline in 48pt Helvetica"
❌ "CENTER-SCREEN: display at coordinates (400, 300)"
✅ "135K" SLAMS in from left, white Impact 120pt, fills 40% of frame.
```
**Named artists without specs** — "Ikko Tanaka style" means nothing to Video Agent. Translate to concrete rules:
```
❌ "Use an Ikko Tanaka style"
✅ "Flat color blocks, maximum 3 colors per frame, 60% negative space, typography as primary element"
```
**Style examples injected into prompts** — Full example scenes from a style library confuse the agent. Use the style's **rules**, not example scenes.
**Forced short B-roll (≤5 seconds)** — Too short for rendering. Every tested video with 5s B-roll had empty/black screens. Use 10-15s.
**Content as a list, not a story** — "Here are 5 tweets" produces flat videos. Always synthesize: *"X is happening because Y — here's the proof."*
## Production Insights
### Style Performance (from 40+ videos)
| Rank | Style | Strength |
|------|-------|----------|
| 1 | Deconstructed (Brody) | Most reliable across all topics |
| 2 | Swiss Pulse (Müller-Brockmann) | Best for data-heavy content |
| 3 | Digital Grid (Crouwel) | Strong for tech topics |
| 4 | Geometric Bold (Tanaka) | Elegant and versatile |
| 5 | Maximalist Type (Scher) | High energy, use sparingly |
### Duration by Approach
| Approach | Avg Duration | Quality |
|----------|-------------|---------|
| Natural storyboard + custom avatar | ~106s | Best |
| Natural storyboard, no custom avatar | ~69s | Good |
| Forced short scenes + custom avatar | ~71s | Mixed |
| Layout language prompts | ~48s | Poor |
## Quality Checklist
- [ ] Thesis-driven — story, not bullet points
- [ ] Style named with colors, typography, motion, transitions (see [visual-styles.md](visual-styles.md))
- [ ] Avatar has thematic wardrobe + branded environment (60-100 words)
- [ ] Critical text listed — every stat, quote, label
- [ ] Scenes rotate types — never 3+ same type. At least 2 B-roll scenes
- [ ] Every scene has VOICEOVER — including B-roll
- [ ] B-roll scenes have 4+ layers, every element has motion verbs
- [ ] B-roll scenes are 10-15 seconds (never ≤5s)
- [ ] Brand logos appear when discussing companies
- [ ] Every element moves — no static frames
.claude/skills/create-video/references/quota.md
+195
View File
@@ -0,0 +1,195 @@
---
name: quota
description: Credit system, usage limits, and checking remaining quota for HeyGen
---
# HeyGen Quota and Credits
HeyGen uses a credit-based system for video generation. Understanding quota management helps prevent failed video generation requests.
## Checking Remaining Quota
### curl
```bash
curl -X GET "https://api.heygen.com/v2/user/remaining_quota" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface QuotaResponse {
error: null | string;
data: {
remaining_quota: number;
used_quota: number;
};
}
const response = await fetch("https://api.heygen.com/v2/user/remaining_quota", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const { data }: QuotaResponse = await response.json();
console.log(`Remaining credits: ${data.remaining_quota}`);
```
### Python
```python
import requests
import os
response = requests.get(
"https://api.heygen.com/v2/user/remaining_quota",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()["data"]
print(f"Remaining credits: {data['remaining_quota']}")
```
## Response Format
```json
{
"error": null,
"data": {
"remaining_quota": 450,
"used_quota": 50
}
}
```
## Credit Consumption
Different operations consume different amounts of credits:
| Operation | Credit Cost | Notes |
|-----------|-------------|-------|
| Standard video (1 min) | ~1 credit per minute | Varies by resolution |
| 720p video | Base rate | Standard quality |
| 1080p video | ~1.5x base rate | Higher quality |
| Video translation | Varies | Depends on video length |
| Streaming avatar | Per session | Real-time usage |
## Pre-Generation Quota Check
Always verify sufficient quota before generating videos:
```typescript
async function generateVideoWithQuotaCheck(videoConfig: VideoConfig) {
// Check quota first
const quotaResponse = await fetch(
"https://api.heygen.com/v2/user/remaining_quota",
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data: quota } = await quotaResponse.json();
// Estimate required credits (rough estimate: 1 credit per minute)
const estimatedMinutes = videoConfig.estimatedDuration / 60;
const requiredCredits = Math.ceil(estimatedMinutes);
if (quota.remaining_quota < requiredCredits) {
throw new Error(
`Insufficient credits. Need ${requiredCredits}, have ${quota.remaining_quota}`
);
}
// Proceed with video generation
return generateVideo(videoConfig);
}
```
## Quota Management Best Practices
### 1. Monitor Usage Regularly
```typescript
async function logQuotaUsage() {
const response = await fetch(
"https://api.heygen.com/v2/user/remaining_quota",
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
console.log({
remaining: data.remaining_quota,
used: data.used_quota,
percentUsed: (
(data.used_quota / (data.remaining_quota + data.used_quota)) *
100
).toFixed(1),
});
}
```
### 2. Set Up Alerts
```typescript
const QUOTA_WARNING_THRESHOLD = 50;
async function checkQuotaWithAlert() {
const response = await fetch(
"https://api.heygen.com/v2/user/remaining_quota",
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
if (data.remaining_quota < QUOTA_WARNING_THRESHOLD) {
// Send alert (email, Slack, etc.)
await sendAlert(`Low HeyGen quota: ${data.remaining_quota} credits remaining`);
}
return data;
}
```
### 3. Use Test Mode for Development
When available, use test mode to avoid consuming credits during development:
```typescript
const videoConfig = {
test: true, // Use test mode during development
video_inputs: [...],
};
// Test videos may have watermarks but don't consume credits
```
## Subscription Tiers
Different subscription tiers have different quota allocations and features:
| Tier | Features |
|------|----------|
| Free | Limited credits, basic features |
| Creator | More credits, standard avatars |
| Team | Higher limits, team collaboration |
| Enterprise | Custom limits, API access, priority support |
API access typically requires Enterprise tier or higher.
## Error Handling for Quota Issues
```typescript
async function handleQuotaError(error: any) {
if (error.message.includes("quota") || error.message.includes("credit")) {
console.error("Quota exceeded. Consider:");
console.error("1. Upgrading your subscription");
console.error("2. Waiting for quota reset");
console.error("3. Purchasing additional credits");
// Check current quota
const quota = await getQuota();
console.error(`Current remaining: ${quota.remaining_quota}`);
}
throw error;
}
```
.claude/skills/create-video/references/video-agent.md
+347
View File
@@ -0,0 +1,347 @@
---
name: video-agent
description: One-shot prompt video generation with HeyGen Video Agent API
---
# Video Agent API
The Video Agent API generates complete videos from a single text prompt. Unlike the standard video generation API which requires detailed scene-by-scene configuration, Video Agent automatically handles script writing, avatar selection, visuals, voiceover, pacing, and captions.
## MCP Tool (Preferred)
If the HeyGen MCP server is connected, use `mcp__heygen__generate_video_agent` instead of direct API calls:
```
Tool: mcp__heygen__generate_video_agent
Parameters:
prompt: "<optimized prompt from prompt-optimizer.md>"
config:
duration_sec: 90 # optional, 5-300
avatar_id: "avatar_id" # optional, agent selects if omitted
orientation: "landscape" # optional, "landscape" or "portrait"
files: # optional
- asset_id: "uploaded_asset_id"
```
Then check status with `mcp__heygen__get_video` using the returned `video_id`.
The prompt quality is still the critical factor — always follow [prompt-optimizer.md](prompt-optimizer.md) regardless of whether you use MCP or direct API.
## When to Use Video Agent vs Standard API
| Use Case | Recommended API |
|----------|-----------------|
| Quick video from idea | Video Agent |
| Precise control over scenes, avatars, timing | Standard v2/video/generate |
| Automated content generation at scale | Video Agent |
| Specific avatar with exact script | Standard v2/video/generate |
| Prototype or draft video | Video Agent |
| Brand-consistent production video | Standard v2/video/generate |
## Before You Call This API
**Required step:** Optimize your prompt using [prompt-optimizer.md](prompt-optimizer.md) before generating a video. The difference between mediocre and professional results depends entirely on prompt quality.
Quick checklist:
1. Define visual style (colors, aesthetic) — see [visual-styles.md](visual-styles.md)
2. Structure scenes with specific scene types
3. Write VO script at ~150 words/minute
4. Specify media types for each scene (Motion Graphics, Stock, AI-generated)
## Direct API Endpoint
```
POST https://api.heygen.com/v1/video_agent/generate
```
## Request Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `prompt` | string | ✓ | Text prompt describing the video you want |
| `config` | object | | Configuration options (see below) |
| `files` | array | | Asset files to reference in generation |
| `callback_id` | string | | Custom ID for tracking. **Requires `callback_url` to also be set** — omit both if you don't need webhooks |
| `callback_url` | string | | Webhook URL for completion notification |
### Config Object
| Field | Type | Description |
|-------|------|-------------|
| `duration_sec` | integer | Approximate duration in seconds (5-300) |
| `avatar_id` | string | Specific avatar to use (optional - agent selects if not provided) |
| `orientation` | string | `"portrait"` or `"landscape"` |
### Files Array
| Field | Type | Description |
|-------|------|-------------|
| `asset_id` | string | Asset ID of uploaded file to reference |
## Response Format
```json
{
"error": null,
"data": {
"video_id": "abc123"
}
}
```
## curl Example
```bash
curl -X POST "https://api.heygen.com/v1/video_agent/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "Create a 60-second product demo video for a new AI-powered calendar app. The tone should be professional but friendly, targeting busy professionals. Highlight the smart scheduling feature and time zone handling."
}'
```
## TypeScript
```typescript
interface VideoAgentConfig {
duration_sec?: number; // 5-300 seconds
avatar_id?: string; // Optional: specific avatar
orientation?: "portrait" | "landscape";
}
interface VideoAgentFile {
asset_id: string;
}
interface VideoAgentRequest {
prompt: string; // Required
config?: VideoAgentConfig;
files?: VideoAgentFile[];
callback_id?: string; // Requires callback_url if set
callback_url?: string;
}
interface VideoAgentResponse {
error: string | null;
data: {
video_id: string;
};
}
async function generateWithVideoAgent(
prompt: string,
config?: VideoAgentConfig
): Promise<string> {
const request: VideoAgentRequest = { prompt };
if (config) {
request.config = config;
}
const response = await fetch(
"https://api.heygen.com/v1/video_agent/generate",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(request),
}
);
const json: VideoAgentResponse = await response.json();
if (json.error) {
throw new Error(`Video Agent failed: ${json.error}`);
}
return json.data.video_id;
}
```
## Python
```python
import requests
import os
from typing import Optional
def generate_with_video_agent(
prompt: str,
duration_sec: Optional[int] = None,
avatar_id: Optional[str] = None,
orientation: Optional[str] = None
) -> str:
request_body = {"prompt": prompt}
config = {}
if duration_sec:
config["duration_sec"] = duration_sec
if avatar_id:
config["avatar_id"] = avatar_id
if orientation:
config["orientation"] = orientation
if config:
request_body["config"] = config
response = requests.post(
"https://api.heygen.com/v1/video_agent/generate",
headers={
"X-Api-Key": os.environ["HEYGEN_API_KEY"],
"Content-Type": "application/json"
},
json=request_body
)
data = response.json()
if data.get("error"):
raise Exception(f"Video Agent failed: {data['error']}")
return data["data"]["video_id"]
```
## Examples
### Basic: Prompt Only
```typescript
const videoId = await generateWithVideoAgent(
"Create a 30-second welcome video for new employees at a tech startup. Keep it energetic and modern."
);
```
### With Duration and Orientation
```typescript
const videoId = await generateWithVideoAgent(
"Explain the benefits of cloud computing for small businesses. Use simple language and real-world examples.",
{
duration_sec: 90,
orientation: "landscape"
}
);
```
### With Specific Avatar
```typescript
const videoId = await generateWithVideoAgent(
"Present quarterly sales results. Professional tone, data-focused.",
{
duration_sec: 120,
avatar_id: "josh_lite3_20230714",
orientation: "landscape"
}
);
```
### With Reference Files
Upload assets first, then reference them:
```typescript
// 1. Upload reference materials (see assets.md)
const logoAssetId = await uploadFile("./company-logo.png", "image/png");
const productImageId = await uploadFile("./product-screenshot.png", "image/png");
// 2. Generate video with references
const response = await fetch(
"https://api.heygen.com/v1/video_agent/generate",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
prompt: "Create a product demo video showcasing our new dashboard feature. Use the uploaded screenshots as visual references.",
config: {
duration_sec: 60,
orientation: "landscape"
},
files: [
{ asset_id: logoAssetId },
{ asset_id: productImageId }
]
}),
}
);
```
## Writing Effective Prompts
See **[prompt-optimizer.md](prompt-optimizer.md)** for comprehensive prompt writing guidance.
The prompt optimizer covers:
- Prompt complexity levels (basic → scene-by-scene)
- Visual style taxonomy and color specification
- Media type selection (Motion Graphics vs Stock vs AI-generated)
- Scene structure and timing calculations
- Ready-to-use templates for common video types
## Checking Video Status
Video Agent returns a `video_id` - use the standard status endpoint to check progress:
```typescript
// Same polling as standard video generation
const videoUrl = await waitForVideo(videoId);
```
See [video-status.md](video-status.md) for polling implementation.
## Comparison: Video Agent vs Standard API
### Video Agent Request
```typescript
// Simple: describe what you want
const videoId = await generateWithVideoAgent(
"Create a 60-second tutorial on setting up two-factor authentication. Professional tone, step-by-step."
);
```
### Equivalent Standard API Request
```typescript
// Complex: specify every detail
const videoId = await generateVideo({
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Welcome to this tutorial on two-factor authentication...",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#1a1a2e",
},
},
// ... more scenes for each step
],
dimension: { width: 1920, height: 1080 },
});
```
## Limitations
- Less control over exact script wording
- Avatar selection may vary if not specified
- Scene composition is automated
- May not match precise brand guidelines
- Duration is approximate, not exact
## Best Practices
1. **Be specific in prompts** - More detail = better results
2. **Specify duration** - Use `config.duration_sec` for predictable length
3. **Lock avatar if needed** - Use `config.avatar_id` for consistency
4. **Upload reference files** - Help agent understand your brand/product
5. **Iterate on prompts** - Refine based on results
6. **Use for drafts** - Video Agent is great for quick iterations before final production
.claude/skills/create-video/references/video-status.md
+502
View File
@@ -0,0 +1,502 @@
---
name: video-status
description: Polling patterns, status types, and retrieving download URLs for HeyGen videos
---
# Video Status and Polling
After generating a video, you need to poll for status until the video is complete. HeyGen processes videos asynchronously.
## MCP Tool (Preferred)
If the HeyGen MCP server is connected, use `mcp__heygen__get_video` with the `videoId` parameter. It returns status, video_url, thumbnail_url, duration, title, gif_url, captioned_video_url, and other metadata in a single call.
## Checking Video Status (Direct API)
### curl
```bash
curl -X GET "https://api.heygen.com/v2/videos/YOUR_VIDEO_ID" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface VideoStatusResponse {
error: null | string;
data: {
id: string;
status: "pending" | "processing" | "completed" | "failed";
video_url?: string;
thumbnail_url?: string;
duration?: number;
title?: string;
created_at?: string;
completed_at?: string;
gif_url?: string;
captioned_video_url?: string;
subtitle_url?: string;
folder_id?: string;
output_language?: string;
failure_code?: string;
failure_message?: string;
};
}
async function getVideoStatus(videoId: string): Promise<VideoStatusResponse["data"]> {
const response = await fetch(
`https://api.heygen.com/v2/videos/${videoId}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json: VideoStatusResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data;
}
```
### Python
```python
import requests
import os
def get_video_status(video_id: str) -> dict:
response = requests.get(
f"https://api.heygen.com/v2/videos/{video_id}",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]
```
## Video Status Types
| Status | Description |
|--------|-------------|
| `pending` | Video is queued for processing |
| `processing` | Video is being generated |
| `completed` | Video is ready for download |
| `failed` | Video generation failed |
## Expected Generation Times
Video generation typically takes **5-15 minutes**, but can exceed 20 minutes during peak load or for longer scripts.
| Factor | Impact |
|--------|--------|
| Script length | Longer scripts = significantly longer processing |
| Resolution | 1080p takes longer than 720p |
| Avatar complexity | Some avatars render faster |
| Queue load | Peak hours may cause 15-20+ minute waits |
| Multiple scenes | Each scene adds processing time |
**Recommendations**:
- Set timeout to **15-20 minutes** (900,000-1,200,000 ms) for safety
- For scripts > 2 minutes of speech, expect 15+ minutes
- Consider async patterns (save video_id, check later) for long videos
## Response Format
### Completed Video
```json
{
"error": null,
"data": {
"id": "abc123",
"status": "completed",
"video_url": "https://files.heygen.ai/video/abc123.mp4",
"thumbnail_url": "https://files.heygen.ai/thumbnail/abc123.jpg",
"duration": 45.2,
"title": "My Video",
"created_at": "2024-01-15T10:30:00Z",
"completed_at": "2024-01-15T10:38:00Z",
"gif_url": "https://files.heygen.ai/gif/abc123.gif",
"captioned_video_url": null,
"subtitle_url": null,
"folder_id": null,
"output_language": "en"
}
}
```
### Failed Video
```json
{
"error": null,
"data": {
"id": "abc123",
"status": "failed",
"failure_code": "script_too_long",
"failure_message": "Script too long for selected avatar"
}
}
```
## Polling Implementation
### Basic Polling
```typescript
async function waitForVideo(
videoId: string,
maxWaitMs = 600000, // 10 minutes
pollIntervalMs = 5000 // 5 seconds
): Promise<string> {
const startTime = Date.now();
while (Date.now() - startTime < maxWaitMs) {
const status = await getVideoStatus(videoId);
switch (status.status) {
case "completed":
return status.video_url!;
case "failed":
throw new Error(status.failure_message || "Video generation failed");
case "pending":
case "processing":
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
break;
}
}
throw new Error("Video generation timed out");
}
```
### Polling with Progress Callback
```typescript
type ProgressCallback = (status: string, elapsed: number) => void;
async function waitForVideoWithProgress(
videoId: string,
onProgress?: ProgressCallback,
maxWaitMs = 600000,
pollIntervalMs = 5000
): Promise<string> {
const startTime = Date.now();
while (Date.now() - startTime < maxWaitMs) {
const elapsed = Date.now() - startTime;
const status = await getVideoStatus(videoId);
onProgress?.(status.status, elapsed);
switch (status.status) {
case "completed":
return status.video_url!;
case "failed":
throw new Error(status.failure_message || "Video generation failed");
default:
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
}
}
throw new Error("Video generation timed out");
}
// Usage
const videoUrl = await waitForVideoWithProgress(
videoId,
(status, elapsed) => {
console.log(`Status: ${status}, Elapsed: ${Math.round(elapsed / 1000)}s`);
}
);
```
### Python Polling
```python
import time
from typing import Optional, Callable
def wait_for_video(
video_id: str,
max_wait_seconds: int = 600,
poll_interval: int = 5,
on_progress: Optional[Callable[[str, int], None]] = None
) -> str:
start_time = time.time()
while time.time() - start_time < max_wait_seconds:
elapsed = int(time.time() - start_time)
status_data = get_video_status(video_id)
status = status_data["status"]
if on_progress:
on_progress(status, elapsed)
if status == "completed":
return status_data["video_url"]
elif status == "failed":
raise Exception(status_data.get("failure_message", "Video generation failed"))
time.sleep(poll_interval)
raise Exception("Video generation timed out")
# Usage
def progress_callback(status: str, elapsed: int):
print(f"Status: {status}, Elapsed: {elapsed}s")
video_url = wait_for_video(video_id, on_progress=progress_callback)
```
## Downloading the Video
Once the video is complete, download it. **Important**: The video URL may not be immediately available after status shows "completed". Use retry logic with backoff.
### TypeScript (with retry)
```typescript
import fs from "fs";
import path from "path";
async function downloadVideoWithRetry(
videoUrl: string,
outputPath = "./output/video.mp4",
maxRetries = 5,
initialDelayMs = 2000
): Promise<void> {
let lastError: Error | null = null;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await fetch(videoUrl);
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
const arrayBuffer = await response.arrayBuffer();
fs.writeFileSync(path.resolve(outputPath), Buffer.from(arrayBuffer));
console.log(`Video downloaded to ${outputPath}`);
return;
} catch (error) {
lastError = error as Error;
const delay = initialDelayMs * Math.pow(2, attempt); // Exponential backoff
console.log(`Download attempt ${attempt + 1} failed, retrying in ${delay}ms...`);
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
throw new Error(`Failed to download after ${maxRetries} attempts: ${lastError?.message}`);
}
```
### Python (with retry)
```python
import requests
import time
def download_video_with_retry(
video_url: str,
output_path: str,
max_retries: int = 5,
initial_delay: float = 2.0
) -> None:
last_error = None
for attempt in range(max_retries):
try:
response = requests.get(video_url, stream=True, timeout=60)
response.raise_for_status()
with open(output_path, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
print(f"Video downloaded to {output_path}")
return
except Exception as e:
last_error = e
delay = initial_delay * (2 ** attempt) # Exponential backoff
print(f"Download attempt {attempt + 1} failed, retrying in {delay}s...")
time.sleep(delay)
raise Exception(f"Failed to download after {max_retries} attempts: {last_error}")
```
### Simple Download (no retry)
For quick scripts where you'll retry manually:
```typescript
async function downloadVideo(videoUrl: string, outputPath = "./output/video.mp4") {
const response = await fetch(videoUrl);
if (!response.ok) {
throw new Error(`Failed to download: ${response.status}`);
}
const arrayBuffer = await response.arrayBuffer();
fs.writeFileSync(path.resolve(outputPath), Buffer.from(arrayBuffer));
}
```
## Complete Workflow Example
```typescript
async function generateAndDownloadVideo(config: VideoConfig): Promise<string> {
// 1. Generate video
const generateResponse = await fetch(
"https://api.heygen.com/v2/video/generate",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
}
);
const { data: generateData } = await generateResponse.json();
const videoId = generateData.video_id;
console.log(`Video ID: ${videoId}`);
// 2. Poll for completion
const videoUrl = await waitForVideoWithProgress(
videoId,
(status, elapsed) => {
console.log(`[${Math.round(elapsed / 1000)}s] Status: ${status}`);
}
);
// 3. Download
const outputPath = `./output/${videoId}.mp4`;
await downloadVideo(videoUrl, outputPath);
return outputPath;
}
```
## Resumable Status Checking
For long-running generations, save the video_id and check status later rather than keeping a process waiting.
### Save State After Generation
```typescript
interface PendingVideo {
videoId: string;
createdAt: string;
script: string;
avatarId: string;
voiceId: string;
}
async function startVideoGeneration(config: VideoGenerateRequest): Promise<PendingVideo> {
const videoId = await generateVideo(config);
const pending: PendingVideo = {
videoId,
createdAt: new Date().toISOString(),
script: config.video_inputs[0].voice.input_text!,
avatarId: config.video_inputs[0].character.avatar_id!,
voiceId: config.video_inputs[0].voice.voice_id!,
};
// Save to file for later retrieval
fs.writeFileSync("pending-video.json", JSON.stringify(pending, null, 2));
console.log(`Video generation started. ID: ${videoId}`);
console.log("Check status later with: checkVideoStatus()");
return pending;
}
```
### Check Status Later
```typescript
async function checkVideoStatus(): Promise<void> {
if (!fs.existsSync("pending-video.json")) {
console.log("No pending video found");
return;
}
const pending: PendingVideo = JSON.parse(
fs.readFileSync("pending-video.json", "utf-8")
);
const elapsed = Date.now() - new Date(pending.createdAt).getTime();
console.log(`Checking video ${pending.videoId} (started ${Math.round(elapsed / 60000)} min ago)...`);
const status = await getVideoStatus(pending.videoId);
switch (status.status) {
case "completed":
console.log(`Video ready: ${status.video_url}`);
console.log(`Duration: ${status.duration}s`);
// Clean up pending file
fs.unlinkSync("pending-video.json");
// Save result
fs.writeFileSync("video-result.json", JSON.stringify({
...pending,
videoUrl: status.video_url,
thumbnailUrl: status.thumbnail_url,
duration: status.duration,
title: status.title,
createdAt: status.created_at,
completedAt: status.completed_at,
}, null, 2));
break;
case "failed":
console.error(`Video failed: ${status.failure_message}`);
fs.unlinkSync("pending-video.json");
break;
default:
console.log(`Status: ${status.status} - check again in a few minutes`);
}
}
```
### CLI-Friendly Pattern
```typescript
// generate-video.ts - Start generation and exit
async function main() {
const pending = await startVideoGeneration(config);
console.log(`\nVideo ID saved. Run 'npx tsx check-status.ts' to check progress.`);
process.exit(0); // Exit immediately, don't wait
}
// check-status.ts - Check and optionally wait
async function main() {
const args = process.argv.slice(2);
const shouldWait = args.includes("--wait");
if (shouldWait) {
// Poll until complete (with 20 min timeout)
const result = await waitForVideo(pending.videoId, apiKey, onProgress, 1200000);
console.log(`Done: ${result.video_url}`);
} else {
// Just check once and report
await checkVideoStatus();
}
}
```
## Alternative: Using Webhooks
Instead of polling, you can use webhooks to receive notifications when videos complete. See [webhooks.md](webhooks.md) for details. Webhooks are ideal for production systems where you don't want to maintain polling connections.
## Best Practices
1. **Use exponential backoff** - Increase poll intervals for long-running jobs
2. **Set reasonable timeouts** - Most videos complete within 10 minutes
3. **Handle failures gracefully** - Check error messages for actionable feedback
4. **Consider webhooks** - For production systems, webhooks are more efficient than polling
5. **Cache video URLs** - Downloaded video URLs are valid for a limited time
.claude/skills/create-video/references/visual-styles.md
+367
View File
@@ -0,0 +1,367 @@
---
name: visual-styles
description: 20 named visual styles for Video Agent prompts — each with colors, typography, motion, and transitions
---
# Visual Style Library — 20 Styles
Named visual styles for Video Agent prompts. Each is inspired by a real graphic designer. Ordered by mood intensity.
**Picking a style:** Match mood first, content second. Ask: *"What should the viewer FEEL?"*
**Using a style:** Copy the style block into your prompt's STYLE section. Use the visual language rules — don't inject the example B-roll scenes (they confuse the agent).
**Custom styles:** These are examples. Create your own by combining elements, referencing other designers, art movements, or cultural aesthetics. The pattern: **named style + designer reference + color palette + typography + motion rules + transitions.**
## Quick Reference
| # | Style | Artist | Mood | Best For |
|---|---|---|---|---|
| 1 | Soft Signal | Sagmeister | Intimate, warm | Personal stories, wellness |
| 2 | Warm Grain | Eksell | Organic, friendly | Environmental, sustainability |
| 3 | Quiet Drama | Ray | Humanist, contemplative | Profiles, biographical |
| 4 | Heritage Reel | Cassandre | Nostalgic, vintage | History, retrospectives |
| 5 | Silk Route | Abedini | Flowing, mysterious | Global affairs, cross-cultural |
| 6 | Swiss Pulse | Müller-Brockmann | Clinical, precise | Data-heavy, analytical |
| 7 | Geometric Bold | Tanaka | Minimal, elegant | Lifestyle, visual essays |
| 8 | Velvet Standard | Vignelli | Premium, timeless | Luxury, investor updates |
| 9 | Digital Grid | Crouwel | Systematic, technical | Infrastructure, engineering |
| 10 | Contact Sheet | Brodovitch | Editorial, investigative | Journalism, deep dives |
| 11 | Folk Frequency | Terrazas | Cultural, vivid | Festivals, food, heritage |
| 12 | Earth Pulse | Ghariokwu | Grounded, communal | Community, grassroots |
| 13 | Dream State | Tomaszewski | Surreal, poetic | Op-eds, philosophy |
| 14 | Play Mode | Ahn Sang-soo | Playful, irreverent | Entertainment, pop culture |
| 15 | Carnival Surge | Lins | Euphoric, celebratory | Milestones, hype |
| 16 | Shadow Cut | Hillmann | Dark, cinematic | Exposés, investigations |
| 17 | Deconstructed | Brody | Industrial, raw | Tech news, punk energy |
| 18 | Maximalist Type | Scher | Loud, kinetic | Big announcements, launches |
| 19 | Data Drift | Anadol | Futuristic, immersive | AI/tech, innovation |
| 20 | Red Wire | Tartakover | Urgent, immediate | Breaking news, crisis |
## Mood-to-Style Guide
| Content feels... | Use... |
|---|---|
| Personal, intimate | Soft Signal, Quiet Drama |
| Natural, earthy | Warm Grain, Earth Pulse |
| Nostalgic, historical | Heritage Reel |
| Data-driven, analytical | Swiss Pulse, Digital Grid |
| Elegant, premium | Velvet Standard, Geometric Bold |
| Cultural, global | Silk Route, Folk Frequency |
| Investigative, serious | Contact Sheet, Shadow Cut |
| Fun, lighthearted | Play Mode, Carnival Surge |
| Philosophical, abstract | Dream State |
| Punk, grassroots, raw | Deconstructed |
| Hype, loud, high-energy | Maximalist Type |
| Tech-forward, futuristic | Data Drift |
| Breaking, urgent | Red Wire |
---
## 1. Soft Signal — Stefan Sagmeister
**Mood:** Intimate, warm | **Best for:** Personal stories, wellness, reflections
- Warm amber and cream with dusty rose, sage green, honey gold accents
- Handwritten-style text overlays — personal, lowercase, delicate
- Close-up framing: hands, faces, textures. Macro lens feel
- Slow drifts and floats, never snaps. Soft dissolves, warm light leaks
```
STYLE — SOFT SIGNAL (Sagmeister): Warm amber/cream, dusty rose, sage green.
Handwritten-style text. Close-up framing. Slow drifts and floats.
Soft dissolves with warm light leaks.
```
## 2. Warm Grain — Olle Eksell
**Mood:** Organic, friendly | **Best for:** Environmental, sustainability, community
- Earth tones: ochre, forest green, terracotta, cream, soft brown
- Rounded sans-serif type. Organic rounded compositions — nothing angular
- 16mm film grain, slightly desaturated. Natural textures: wood, linen, stone
- Gentle wipes, soft cuts, unhurried
```
STYLE — WARM GRAIN (Eksell): Earth tones — ochre, forest green, terracotta, cream.
Organic rounded compositions. 16mm film grain. Rounded sans-serif.
Gentle wipes and soft cuts.
```
## 3. Quiet Drama — Satyajit Ray
**Mood:** Humanist, contemplative | **Best for:** Profiles, biographical, cultural
- Muted warm: sepia, deep brown, soft gold, off-white, charcoal
- Clean serif type, positioned with care. Portrait framing
- Strong single-source contrast: window light, single lamp
- Deliberate pacing, longer holds. Slow fades to black
```
STYLE — QUIET DRAMA (Ray): Muted warm — sepia, deep brown, soft gold.
Portrait framing. Clean serif. Strong single-source contrast.
Slow fades to black.
```
## 4. Heritage Reel — Cassandre
**Mood:** Nostalgic, vintage | **Best for:** History, retrospectives, brand origins
- Faded gold, deep burgundy, navy, cream, sepia wash
- Elegant centered serif like classic film title cards
- Vignetting, softened edges. Film grain, light scratches, gentle jitter
- Iris wipes, film reel flicker
```
STYLE — HERITAGE REEL (Cassandre): Faded gold, burgundy, navy, sepia wash.
Elegant centered serif. Vignetting and aged film grain.
Iris wipe transitions.
```
## 5. Silk Route — Reza Abedini
**Mood:** Flowing, mysterious | **Best for:** Global affairs, cross-cultural, art/design
- Rich jewel tones: deep teal, burgundy, gold, lapis blue, black
- Elegant spaced type along natural visual lines
- Layered compositions — foreground, midground, background all active
- Flowing dissolves, smooth morphs
```
STYLE — SILK ROUTE (Abedini): Jewel tones — deep teal, burgundy, gold, lapis blue.
Layered compositions, all depths active. Elegant spaced type.
Flowing dissolves and smooth morphs.
```
## 6. Swiss Pulse — Josef Müller-Brockmann
**Mood:** Clinical, precise | **Best for:** Data-heavy, analytical, financial, metrics
- Black (#1a1a1a), white, ONE accent: electric blue (#0066FF)
- Helvetica Bold headlines, Regular labels. Numbers LARGE (80-120pt)
- Grid-locked compositions. Every element snaps to 12-column grid
- Animated counters COUNT UP from 0. Diagonal compositions on key moments
- Grid wipes, hard cuts. No dissolves
```
STYLE — SWISS PULSE (Müller-Brockmann): Black/white + electric blue #0066FF.
Grid-locked. Helvetica Bold. Animated counters. Diagonal accents.
Grid wipe transitions.
```
## 7. Geometric Bold — Ikko Tanaka
**Mood:** Minimal, elegant | **Best for:** Clean lifestyle, culture, visual essays, brand profiles
- Maximum 3 flat colors per frame — no gradients
- Bold clean type as primary visual element
- Asymmetric composition, 60% negative space minimum. Single focal point
- Clean cuts on beat, no effects
```
STYLE — GEOMETRIC BOLD (Tanaka): Max 3 flat colors per frame.
60% negative space. Bold type as primary element.
Single focal point. Clean cuts on beat.
```
## 8. Velvet Standard — Massimo Vignelli
**Mood:** Premium, timeless | **Best for:** Luxury, investor updates, keynotes, product showcases
- Black, white, ONE rich accent: deep navy (#1a237e) or gold (#c9a84c)
- Thin sans-serif, ALL CAPS, letter-spaced wide
- Generous negative space. Symmetrical, centered, architectural precision
- Slow, deliberate. Sequential reveals. Elegant cross-dissolves
```
STYLE — VELVET STANDARD (Vignelli): Black, white, one accent: gold #c9a84c.
Thin ALL CAPS, wide spacing. Generous negative space.
Slow elegant cross-dissolves.
```
## 9. Digital Grid — Wim Crouwel
**Mood:** Systematic, technical | **Best for:** Infrastructure, engineering, code, tech
- Dark (#0a0a0a) with cyan (#00E5FF), amber (#FFB300), green (#00FF88)
- Monospaced type throughout. Code-terminal aesthetic
- Pixel grid overlays visible. Everything snaps to system
- Grid nodes light up sequentially. Scan-line effects, cursor blinks
- Clean wipe transitions
```
STYLE — DIGITAL GRID (Crouwel): Monospaced type. Dark #0a0a0a with cyan #00E5FF, amber #FFB300.
Pixel grid overlays. Terminal aesthetic. Clean wipe transitions.
```
## 10. Contact Sheet — Alexey Brodovitch
**Mood:** Editorial, investigative | **Best for:** Journalism, deep dives, research breakdowns
- High contrast B&W with occasional desaturated color accents
- Bold sans-serif captions like editorial annotations
- Photo-editorial framing — multiple images, contact-sheet energy
- Raw grain, imperfect focus. Tight crops on faces and hands
- Hard cuts on beat, snap-zooms
```
STYLE — CONTACT SHEET (Brodovitch): High contrast B&W, desaturated accents.
Photo-editorial framing. Bold sans-serif annotations. Raw grain.
Hard cuts on beat. Snap-zooms.
```
## 11. Folk Frequency — Eduardo Terrazas
**Mood:** Cultural, vivid | **Best for:** Cultural events, food, tradition, heritage
- Vivid folk: hot pink, bright orange, cobalt blue, sun yellow, emerald
- Bold warm rounded type. Pattern and repetition — folk art rhythms
- Rich textures: woven fabrics, painted surfaces, ceramic, handmade
- Colorful wipes, quick cuts on festive rhythm
```
STYLE — FOLK FREQUENCY (Terrazas): Vivid folk — hot pink, cobalt blue, sun yellow, emerald.
Bold rounded type. Folk art rhythms. Rich handmade textures.
Colorful wipes on festive rhythm.
```
## 12. Earth Pulse — Lemi Ghariokwu
**Mood:** Grounded, communal | **Best for:** Community, music/culture, grassroots
- Warm saturated: burnt orange, deep green, rich yellow, terracotta
- Bold expressive type, center-frame. Wide community framing
- Rhythmic editing timed to musical beats
- Rhythmic cuts on beat, freeze-frames for emphasis
```
STYLE — EARTH PULSE (Ghariokwu): Warm saturated — burnt orange, deep green, rich yellow.
Bold expressive type. Wide community framing.
Rhythmic cuts on beat. Freeze-frames.
```
## 13. Dream State — Henryk Tomaszewski
**Mood:** Surreal, poetic | **Best for:** Op-eds, philosophy, think pieces, speculative
- Muted palette with one surreal accent: dusty blues, grey-greens, then shock of red or gold
- Sparse precise text — few words, maximum impact. Thin elegant floating type
- Unusual juxtapositions. Dreamlike quality: soft edges, atmospheric haze
- Slow morph dissolves. NEVER hard cuts
```
STYLE — DREAM STATE (Tomaszewski): Muted palette + one surreal accent.
Thin elegant floating type. Soft edges, atmospheric haze.
Slow morph dissolves — NEVER hard cuts.
```
## 14. Play Mode — Ahn Sang-soo
**Mood:** Playful, irreverent | **Best for:** Entertainment, pop culture, listicles, fun
- Bright candy: electric blue, hot pink, lime green, yellow, white
- Bouncy oversized tilted text. Asymmetric off-kilter compositions
- Quick cuts (1-3 seconds). Score cards, achievement popups, XP bars
- Bouncy spring physics — text overshoots and settles, screen shakes
- Pop cuts, whip pans, bounce effects
```
STYLE — PLAY MODE (Ahn Sang-soo): Electric blue, hot pink, lime green.
Bouncy spring physics. Oversized tilted text. Score cards, XP bars.
Pop cuts, bounce effects.
```
## 15. Carnival Surge — Rico Lins
**Mood:** Euphoric, celebratory | **Best for:** Big announcements, milestones, celebrations, hype
- Maximum color: hot pink (#FF1493), electric yellow (#FFE000), teal (#00CED1), orange, violet
- MASSIVE bold text at ANGLES over footage. Collage-style overlapping
- Rapid 1-2 second clips. Confetti, lights, constant energy
- Smash cuts, flash frames, rapid-fire montage
```
STYLE — CARNIVAL SURGE (Lins): Max color — hot pink #FF1493, yellow #FFE000, teal #00CED1.
Collage layering. Text MASSIVE at ANGLES. Confetti bursts.
Smash cuts, flash frames.
```
## 16. Shadow Cut — Hans Hillmann
**Mood:** Dark, cinematic | **Best for:** Exposés, investigations, controversy, dark deep dives
- Near-monochrome: deep blacks, cold greys, stark white + blood red or toxic green
- Sharp angular text like film noir title cards
- Heavy shadow — faces half-lit, objects emerging from darkness
- Slow creeping push-ins, slow reveals, tension
- Iris to black, slow fade from darkness, hard cuts to silence
```
STYLE — SHADOW CUT (Hillmann): Deep blacks, cold greys + blood red accent.
Sharp angular text. Heavy shadow. Slow creeping push-ins.
Hard cuts to black. Film noir tension.
```
## 17. Deconstructed — Neville Brody
**Mood:** Industrial, raw | **Best for:** Tech news, security, punk energy, counter-culture
- Dark grey (#1a1a1a), black, rust orange (#D4501E), raw white (#f0f0f0)
- Type at angles, overlapping edges, escaping frames. Bold industrial
- High contrast, gritty textures: scratched metal, peeling paint, scan-line glitch
- Text SLAMS, SHATTERS, PUNCHES. Letters scramble then snap
- Smash cuts, glitch transitions, white flash frames
```
STYLE — DECONSTRUCTED (Brody): Dark grey #1a1a1a, rust orange #D4501E.
Type at angles, overlapping. Gritty textures, scan-line glitch.
Smash cuts with flash frames.
```
## 18. Maximalist Type — Paula Scher
**Mood:** Loud, kinetic | **Best for:** Big announcements, launches, high-energy recaps
- Bold saturated: red, yellow, black, white — maximum contrast
- Text IS the visual. Overlapping layers at different scales and angles, 50-80% of frame
- Kinetic energy: everything moving, slamming, sliding. 1-2 second rapid cuts
- Text layered OVER footage — never empty backgrounds
- Smash cuts, text slamming from edges, flash frames
```
STYLE — MAXIMALIST TYPE (Scher): Red, yellow, black, white — max contrast.
Text IS the visual. Overlapping at different scales, 50-80% of frame.
Kinetic everything. Smash cuts, flash frames.
```
## 19. Data Drift — Refik Anadol
**Mood:** Futuristic, immersive | **Best for:** AI/tech, speculative, cutting-edge science
- Iridescent: holographic silver, electric purple (#7c3aed), cyan (#06b6d4), deep black (#0a0a0a)
- Thin futuristic sans-serif — minimal, floating, weightless
- Fluid morphing compositions. Extreme scale shifts: microscopic to cosmic
- Particles coalesce into numbers, light traces data paths
- Liquid dissolves, particles dispersing and reforming
```
STYLE — DATA DRIFT (Anadol): Iridescent — purple #7c3aed, cyan #06b6d4, deep black.
Fluid morphing compositions. Thin futuristic type.
Liquid dissolves. Particles coalesce into numbers.
```
## 20. Red Wire — David Tartakover
**Mood:** Urgent, immediate | **Best for:** Breaking news, crisis updates, alerts
- High alert: red, black, white, emergency yellow — maximum contrast
- Bold condensed all caps — every word screams urgency
- Split screens, ticker-style text bars, timestamp overlays — max information density
- Multiple text elements simultaneously. Handheld energy
- Snap cuts, flash frames, zero breathing room
```
STYLE — RED WIRE (Tartakover): Red, black, white, emergency yellow.
Bold condensed all-caps. Split screens, tickers, timestamps.
Snap cuts, flash frames. Zero breathing room.
```
.claude/skills/create-video/references/webhooks.md
+393
View File
@@ -0,0 +1,393 @@
---
name: webhooks
description: Registering webhook endpoints and event types for HeyGen
---
# Webhooks
Webhooks allow HeyGen to notify your application when events occur, such as video completion. This is more efficient than polling for status updates.
## Overview
Instead of repeatedly checking video status, webhooks push notifications to your server when:
- Video generation completes
- Video generation fails
- Translation completes
- Avatar training completes
- Other async operations finish
## Setting Up a Webhook Endpoint
Your webhook endpoint should:
1. Accept POST requests
2. Return 200 status quickly
3. Handle events asynchronously
### Express.js Example
```typescript
import express from "express";
import crypto from "crypto";
const app = express();
app.use(express.json());
// Webhook endpoint
app.post("/webhook/heygen", async (req, res) => {
// Acknowledge receipt immediately
res.status(200).send("OK");
// Process event asynchronously
processWebhookEvent(req.body).catch(console.error);
});
async function processWebhookEvent(event: HeyGenWebhookEvent) {
console.log(`Received event: ${event.event_type}`);
switch (event.event_type) {
case "avatar_video.success":
await handleVideoSuccess(event);
break;
case "avatar_video.fail":
await handleVideoFailure(event);
break;
case "video_translate.success":
await handleTranslationSuccess(event);
break;
default:
console.log(`Unknown event type: ${event.event_type}`);
}
}
app.listen(3000, () => {
console.log("Webhook server running on port 3000");
});
```
### Python Flask Example
```python
from flask import Flask, request, jsonify
import threading
app = Flask(__name__)
@app.route("/webhook/heygen", methods=["POST"])
def heygen_webhook():
event = request.json
# Acknowledge immediately
response = jsonify({"status": "received"})
# Process asynchronously
thread = threading.Thread(
target=process_webhook_event,
args=(event,)
)
thread.start()
return response, 200
def process_webhook_event(event):
event_type = event.get("event_type")
print(f"Received event: {event_type}")
if event_type == "avatar_video.success":
handle_video_success(event)
elif event_type == "avatar_video.fail":
handle_video_failure(event)
elif event_type == "video_translate.success":
handle_translation_success(event)
if __name__ == "__main__":
app.run(port=3000)
```
## Webhook Event Types
| Event Type | Description |
|------------|-------------|
| `avatar_video.success` | Video generation completed |
| `avatar_video.fail` | Video generation failed |
| `video_translate.success` | Translation completed |
| `video_translate.fail` | Translation failed |
| `instant_avatar.success` | Instant avatar created |
| `instant_avatar.fail` | Instant avatar creation failed |
## Event Payload Structure
### Video Success Event
```typescript
interface VideoSuccessEvent {
event_type: "avatar_video.success";
event_data: {
video_id: string;
video_url: string;
thumbnail_url: string;
duration: number;
callback_id?: string;
};
}
```
```json
{
"event_type": "avatar_video.success",
"event_data": {
"video_id": "abc123",
"video_url": "https://files.heygen.ai/video/abc123.mp4",
"thumbnail_url": "https://files.heygen.ai/thumbnail/abc123.jpg",
"duration": 45.2,
"callback_id": "your_custom_id"
}
}
```
### Video Failure Event
```typescript
interface VideoFailureEvent {
event_type: "avatar_video.fail";
event_data: {
video_id: string;
error: string;
callback_id?: string;
};
}
```
```json
{
"event_type": "avatar_video.fail",
"event_data": {
"video_id": "abc123",
"error": "Script too long for selected avatar",
"callback_id": "your_custom_id"
}
}
```
## Registering a Webhook URL
Configure your webhook URL through the HeyGen dashboard or API:
### Request Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `url` | string | ✓ | Your webhook endpoint URL |
| `events` | array | ✓ | Event types to subscribe to |
| `secret` | string | | Shared secret for signature verification |
### Via API
```bash
curl -X POST "https://api.heygen.com/v1/webhook/endpoint.add" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://your-domain.com/webhook/heygen",
"events": ["avatar_video.success", "avatar_video.fail"]
}'
```
### TypeScript
```typescript
interface WebhookConfig {
url: string; // Required
events: string[]; // Required
secret?: string;
}
async function registerWebhook(config: WebhookConfig): Promise<void> {
const response = await fetch("https://api.heygen.com/v1/webhook/endpoint.add", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
});
const json = await response.json();
if (json.error) {
throw new Error(json.error);
}
}
```
## Using Callback IDs
Track which video triggered a webhook with callback IDs:
### Include Callback ID in Video Generation
```typescript
const videoConfig = {
video_inputs: [...],
callback_id: "order_12345", // Your custom identifier
};
```
### Handle in Webhook
```typescript
async function handleVideoSuccess(event: VideoSuccessEvent) {
const { video_id, video_url, callback_id } = event.event_data;
if (callback_id) {
// Look up your original request
const order = await getOrderByCallbackId(callback_id);
await updateOrderWithVideo(order.id, video_url);
}
}
```
## Webhook Security
### Verify Webhook Signatures
If HeyGen provides signature verification:
```typescript
import crypto from "crypto";
function verifyWebhookSignature(
payload: string,
signature: string,
secret: string
): boolean {
const expectedSignature = crypto
.createHmac("sha256", secret)
.update(payload)
.digest("hex");
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expectedSignature)
);
}
// In your webhook handler
app.post("/webhook/heygen", (req, res) => {
const signature = req.headers["x-heygen-signature"] as string;
const payload = JSON.stringify(req.body);
if (!verifyWebhookSignature(payload, signature, WEBHOOK_SECRET)) {
return res.status(401).send("Invalid signature");
}
// Process event...
});
```
### Validate Event Origin
```typescript
function isValidHeygenEvent(event: any): boolean {
// Check required fields
if (!event.event_type || !event.event_data) {
return false;
}
// Check event type is known
const validEventTypes = [
"avatar_video.success",
"avatar_video.fail",
"video_translate.success",
"video_translate.fail",
];
return validEventTypes.includes(event.event_type);
}
```
## Handling Webhook Failures
Implement retry logic and error handling:
```typescript
async function processWebhookEvent(event: HeyGenWebhookEvent) {
const maxRetries = 3;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await handleEvent(event);
return;
} catch (error) {
console.error(`Attempt ${attempt} failed:`, error);
if (attempt < maxRetries) {
// Exponential backoff
await new Promise((r) => setTimeout(r, Math.pow(2, attempt) * 1000));
}
}
}
// Store failed event for manual review
await storeFailedEvent(event);
}
```
## Webhook vs Polling Comparison
| Aspect | Webhook | Polling |
|--------|---------|---------|
| Latency | Immediate | Depends on interval |
| Efficiency | High (push) | Low (repeated requests) |
| Complexity | Requires endpoint | Simpler to implement |
| Reliability | Needs retry handling | Guaranteed delivery |
| Cost | Lower API usage | Higher API usage |
## Testing Webhooks
### Local Development with ngrok
```bash
# Start ngrok tunnel
ngrok http 3000
# Use ngrok URL as webhook endpoint
# https://abc123.ngrok.io/webhook/heygen
```
### Webhook Testing Tool
```typescript
// Test webhook locally
async function simulateWebhook(event: HeyGenWebhookEvent) {
const response = await fetch("http://localhost:3000/webhook/heygen", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(event),
});
console.log(`Response: ${response.status}`);
}
// Simulate success event
await simulateWebhook({
event_type: "avatar_video.success",
event_data: {
video_id: "test_123",
video_url: "https://example.com/test.mp4",
thumbnail_url: "https://example.com/test.jpg",
duration: 30,
callback_id: "test_callback",
},
});
```
## Best Practices
1. **Respond quickly** - Return 200 within 5 seconds, process async
2. **Handle duplicates** - Same event may be sent multiple times
3. **Implement retries** - Handle temporary processing failures
4. **Log everything** - Store webhook payloads for debugging
5. **Use callback IDs** - Track requests through the system
6. **Secure endpoints** - Verify signatures, use HTTPS
7. **Monitor health** - Track webhook success rates
8. **Queue processing** - Use job queues for heavy processing
.claude/skills/d3-viz/SKILL.md
+820
View File
@@ -0,0 +1,820 @@
---
name: d3-viz
description: Creating interactive data visualisations using d3.js. This skill should be used when creating custom charts, graphs, network diagrams, geographic visualisations, or any complex SVG-based data visualisation that requires fine-grained control over visual elements, transitions, or interactions. Use this for bespoke visualisations beyond standard charting libraries, whether in React, Vue, Svelte, vanilla JavaScript, or any other environment.
---
# D3.js Visualisation
## Overview
This skill provides guidance for creating sophisticated, interactive data visualisations using d3.js. D3.js (Data-Driven Documents) excels at binding data to DOM elements and applying data-driven transformations to create custom, publication-quality visualisations with precise control over every visual element. The techniques work across any JavaScript environment, including vanilla JavaScript, React, Vue, Svelte, and other frameworks.
## When to use d3.js
**Use d3.js for:**
- Custom visualisations requiring unique visual encodings or layouts
- Interactive explorations with complex pan, zoom, or brush behaviours
- Network/graph visualisations (force-directed layouts, tree diagrams, hierarchies, chord diagrams)
- Geographic visualisations with custom projections
- Visualisations requiring smooth, choreographed transitions
- Publication-quality graphics with fine-grained styling control
- Novel chart types not available in standard libraries
**Consider alternatives for:**
- 3D visualisations - use Three.js instead
## Core workflow
### 1. Set up d3.js
Import d3 at the top of your script:
```javascript
import * as d3 from 'd3';
```
Or use the CDN version (7.x):
```html
<script src="https://d3js.org/d3.v7.min.js"></script>
```
All modules (scales, axes, shapes, transitions, etc.) are accessible through the `d3` namespace.
### 2. Choose the integration pattern
**Pattern A: Direct DOM manipulation (recommended for most cases)**
Use d3 to select DOM elements and manipulate them imperatively. This works in any JavaScript environment:
```javascript
function drawChart(data) {
if (!data || data.length === 0) return;
const svg = d3.select('#chart'); // Select by ID, class, or DOM element
// Clear previous content
svg.selectAll("*").remove();
// Set up dimensions
const width = 800;
const height = 400;
const margin = { top: 20, right: 30, bottom: 40, left: 50 };
// Create scales, axes, and draw visualisation
// ... d3 code here ...
}
// Call when data changes
drawChart(myData);
```
**Pattern B: Declarative rendering (for frameworks with templating)**
Use d3 for data calculations (scales, layouts) but render elements via your framework:
```javascript
function getChartElements(data) {
const xScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.value)])
.range([0, 400]);
return data.map((d, i) => ({
x: 50,
y: i * 30,
width: xScale(d.value),
height: 25
}));
}
// In React: {getChartElements(data).map((d, i) => <rect key={i} {...d} fill="steelblue" />)}
// In Vue: v-for directive over the returned array
// In vanilla JS: Create elements manually from the returned data
```
Use Pattern A for complex visualisations with transitions, interactions, or when leveraging d3's full capabilities. Use Pattern B for simpler visualisations or when your framework prefers declarative rendering.
### 3. Structure the visualisation code
Follow this standard structure in your drawing function:
```javascript
function drawVisualization(data) {
if (!data || data.length === 0) return;
const svg = d3.select('#chart'); // Or pass a selector/element
svg.selectAll("*").remove(); // Clear previous render
// 1. Define dimensions
const width = 800;
const height = 400;
const margin = { top: 20, right: 30, bottom: 40, left: 50 };
const innerWidth = width - margin.left - margin.right;
const innerHeight = height - margin.top - margin.bottom;
// 2. Create main group with margins
const g = svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`);
// 3. Create scales
const xScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.x)])
.range([0, innerWidth]);
const yScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.y)])
.range([innerHeight, 0]); // Note: inverted for SVG coordinates
// 4. Create and append axes
const xAxis = d3.axisBottom(xScale);
const yAxis = d3.axisLeft(yScale);
g.append("g")
.attr("transform", `translate(0,${innerHeight})`)
.call(xAxis);
g.append("g")
.call(yAxis);
// 5. Bind data and create visual elements
g.selectAll("circle")
.data(data)
.join("circle")
.attr("cx", d => xScale(d.x))
.attr("cy", d => yScale(d.y))
.attr("r", 5)
.attr("fill", "steelblue");
}
// Call when data changes
drawVisualization(myData);
```
### 4. Implement responsive sizing
Make visualisations responsive to container size:
```javascript
function setupResponsiveChart(containerId, data) {
const container = document.getElementById(containerId);
const svg = d3.select(`#${containerId}`).append('svg');
function updateChart() {
const { width, height } = container.getBoundingClientRect();
svg.attr('width', width).attr('height', height);
// Redraw visualisation with new dimensions
drawChart(data, svg, width, height);
}
// Update on initial load
updateChart();
// Update on window resize
window.addEventListener('resize', updateChart);
// Return cleanup function
return () => window.removeEventListener('resize', updateChart);
}
// Usage:
// const cleanup = setupResponsiveChart('chart-container', myData);
// cleanup(); // Call when component unmounts or element removed
```
Or use ResizeObserver for more direct container monitoring:
```javascript
function setupResponsiveChartWithObserver(svgElement, data) {
const observer = new ResizeObserver(() => {
const { width, height } = svgElement.getBoundingClientRect();
d3.select(svgElement)
.attr('width', width)
.attr('height', height);
// Redraw visualisation
drawChart(data, d3.select(svgElement), width, height);
});
observer.observe(svgElement.parentElement);
return () => observer.disconnect();
}
```
## Common visualisation patterns
### Bar chart
```javascript
function drawBarChart(data, svgElement) {
if (!data || data.length === 0) return;
const svg = d3.select(svgElement);
svg.selectAll("*").remove();
const width = 800;
const height = 400;
const margin = { top: 20, right: 30, bottom: 40, left: 50 };
const innerWidth = width - margin.left - margin.right;
const innerHeight = height - margin.top - margin.bottom;
const g = svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`);
const xScale = d3.scaleBand()
.domain(data.map(d => d.category))
.range([0, innerWidth])
.padding(0.1);
const yScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.value)])
.range([innerHeight, 0]);
g.append("g")
.attr("transform", `translate(0,${innerHeight})`)
.call(d3.axisBottom(xScale));
g.append("g")
.call(d3.axisLeft(yScale));
g.selectAll("rect")
.data(data)
.join("rect")
.attr("x", d => xScale(d.category))
.attr("y", d => yScale(d.value))
.attr("width", xScale.bandwidth())
.attr("height", d => innerHeight - yScale(d.value))
.attr("fill", "steelblue");
}
// Usage:
// drawBarChart(myData, document.getElementById('chart'));
```
### Line chart
```javascript
const line = d3.line()
.x(d => xScale(d.date))
.y(d => yScale(d.value))
.curve(d3.curveMonotoneX); // Smooth curve
g.append("path")
.datum(data)
.attr("fill", "none")
.attr("stroke", "steelblue")
.attr("stroke-width", 2)
.attr("d", line);
```
### Scatter plot
```javascript
g.selectAll("circle")
.data(data)
.join("circle")
.attr("cx", d => xScale(d.x))
.attr("cy", d => yScale(d.y))
.attr("r", d => sizeScale(d.size)) // Optional: size encoding
.attr("fill", d => colourScale(d.category)) // Optional: colour encoding
.attr("opacity", 0.7);
```
### Chord diagram
A chord diagram shows relationships between entities in a circular layout, with ribbons representing flows between them:
```javascript
function drawChordDiagram(data) {
// data format: array of objects with source, target, and value
// Example: [{ source: 'A', target: 'B', value: 10 }, ...]
if (!data || data.length === 0) return;
const svg = d3.select('#chart');
svg.selectAll("*").remove();
const width = 600;
const height = 600;
const innerRadius = Math.min(width, height) * 0.3;
const outerRadius = innerRadius + 30;
// Create matrix from data
const nodes = Array.from(new Set(data.flatMap(d => [d.source, d.target])));
const matrix = Array.from({ length: nodes.length }, () => Array(nodes.length).fill(0));
data.forEach(d => {
const i = nodes.indexOf(d.source);
const j = nodes.indexOf(d.target);
matrix[i][j] += d.value;
matrix[j][i] += d.value;
});
// Create chord layout
const chord = d3.chord()
.padAngle(0.05)
.sortSubgroups(d3.descending);
const arc = d3.arc()
.innerRadius(innerRadius)
.outerRadius(outerRadius);
const ribbon = d3.ribbon()
.source(d => d.source)
.target(d => d.target);
const colourScale = d3.scaleOrdinal(d3.schemeCategory10)
.domain(nodes);
const g = svg.append("g")
.attr("transform", `translate(${width / 2},${height / 2})`);
const chords = chord(matrix);
// Draw ribbons
g.append("g")
.attr("fill-opacity", 0.67)
.selectAll("path")
.data(chords)
.join("path")
.attr("d", ribbon)
.attr("fill", d => colourScale(nodes[d.source.index]))
.attr("stroke", d => d3.rgb(colourScale(nodes[d.source.index])).darker());
// Draw groups (arcs)
const group = g.append("g")
.selectAll("g")
.data(chords.groups)
.join("g");
group.append("path")
.attr("d", arc)
.attr("fill", d => colourScale(nodes[d.index]))
.attr("stroke", d => d3.rgb(colourScale(nodes[d.index])).darker());
// Add labels
group.append("text")
.each(d => { d.angle = (d.startAngle + d.endAngle) / 2; })
.attr("dy", "0.31em")
.attr("transform", d => `rotate(${(d.angle * 180 / Math.PI) - 90})translate(${outerRadius + 30})${d.angle > Math.PI ? "rotate(180)" : ""}`)
.attr("text-anchor", d => d.angle > Math.PI ? "end" : null)
.text((d, i) => nodes[i])
.style("font-size", "12px");
}
```
### Heatmap
A heatmap uses colour to encode values in a two-dimensional grid, useful for showing patterns across categories:
```javascript
function drawHeatmap(data) {
// data format: array of objects with row, column, and value
// Example: [{ row: 'A', column: 'X', value: 10 }, ...]
if (!data || data.length === 0) return;
const svg = d3.select('#chart');
svg.selectAll("*").remove();
const width = 800;
const height = 600;
const margin = { top: 100, right: 30, bottom: 30, left: 100 };
const innerWidth = width - margin.left - margin.right;
const innerHeight = height - margin.top - margin.bottom;
// Get unique rows and columns
const rows = Array.from(new Set(data.map(d => d.row)));
const columns = Array.from(new Set(data.map(d => d.column)));
const g = svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`);
// Create scales
const xScale = d3.scaleBand()
.domain(columns)
.range([0, innerWidth])
.padding(0.01);
const yScale = d3.scaleBand()
.domain(rows)
.range([0, innerHeight])
.padding(0.01);
// Colour scale for values
const colourScale = d3.scaleSequential(d3.interpolateYlOrRd)
.domain([0, d3.max(data, d => d.value)]);
// Draw rectangles
g.selectAll("rect")
.data(data)
.join("rect")
.attr("x", d => xScale(d.column))
.attr("y", d => yScale(d.row))
.attr("width", xScale.bandwidth())
.attr("height", yScale.bandwidth())
.attr("fill", d => colourScale(d.value));
// Add x-axis labels
svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`)
.selectAll("text")
.data(columns)
.join("text")
.attr("x", d => xScale(d) + xScale.bandwidth() / 2)
.attr("y", -10)
.attr("text-anchor", "middle")
.text(d => d)
.style("font-size", "12px");
// Add y-axis labels
svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`)
.selectAll("text")
.data(rows)
.join("text")
.attr("x", -10)
.attr("y", d => yScale(d) + yScale.bandwidth() / 2)
.attr("dy", "0.35em")
.attr("text-anchor", "end")
.text(d => d)
.style("font-size", "12px");
// Add colour legend
const legendWidth = 20;
const legendHeight = 200;
const legend = svg.append("g")
.attr("transform", `translate(${width - 60},${margin.top})`);
const legendScale = d3.scaleLinear()
.domain(colourScale.domain())
.range([legendHeight, 0]);
const legendAxis = d3.axisRight(legendScale)
.ticks(5);
// Draw colour gradient in legend
for (let i = 0; i < legendHeight; i++) {
legend.append("rect")
.attr("y", i)
.attr("width", legendWidth)
.attr("height", 1)
.attr("fill", colourScale(legendScale.invert(i)));
}
legend.append("g")
.attr("transform", `translate(${legendWidth},0)`)
.call(legendAxis);
}
```
### Pie chart
```javascript
const pie = d3.pie()
.value(d => d.value)
.sort(null);
const arc = d3.arc()
.innerRadius(0)
.outerRadius(Math.min(width, height) / 2 - 20);
const colourScale = d3.scaleOrdinal(d3.schemeCategory10);
const g = svg.append("g")
.attr("transform", `translate(${width / 2},${height / 2})`);
g.selectAll("path")
.data(pie(data))
.join("path")
.attr("d", arc)
.attr("fill", (d, i) => colourScale(i))
.attr("stroke", "white")
.attr("stroke-width", 2);
```
### Force-directed network
```javascript
const simulation = d3.forceSimulation(nodes)
.force("link", d3.forceLink(links).id(d => d.id).distance(100))
.force("charge", d3.forceManyBody().strength(-300))
.force("center", d3.forceCenter(width / 2, height / 2));
const link = g.selectAll("line")
.data(links)
.join("line")
.attr("stroke", "#999")
.attr("stroke-width", 1);
const node = g.selectAll("circle")
.data(nodes)
.join("circle")
.attr("r", 8)
.attr("fill", "steelblue")
.call(d3.drag()
.on("start", dragstarted)
.on("drag", dragged)
.on("end", dragended));
simulation.on("tick", () => {
link
.attr("x1", d => d.source.x)
.attr("y1", d => d.source.y)
.attr("x2", d => d.target.x)
.attr("y2", d => d.target.y);
node
.attr("cx", d => d.x)
.attr("cy", d => d.y);
});
function dragstarted(event) {
if (!event.active) simulation.alphaTarget(0.3).restart();
event.subject.fx = event.subject.x;
event.subject.fy = event.subject.y;
}
function dragged(event) {
event.subject.fx = event.x;
event.subject.fy = event.y;
}
function dragended(event) {
if (!event.active) simulation.alphaTarget(0);
event.subject.fx = null;
event.subject.fy = null;
}
```
## Adding interactivity
### Tooltips
```javascript
// Create tooltip div (outside SVG)
const tooltip = d3.select("body").append("div")
.attr("class", "tooltip")
.style("position", "absolute")
.style("visibility", "hidden")
.style("background-color", "white")
.style("border", "1px solid #ddd")
.style("padding", "10px")
.style("border-radius", "4px")
.style("pointer-events", "none");
// Add to elements
circles
.on("mouseover", function(event, d) {
d3.select(this).attr("opacity", 1);
tooltip
.style("visibility", "visible")
.html(`<strong>${d.label}</strong><br/>Value: ${d.value}`);
})
.on("mousemove", function(event) {
tooltip
.style("top", (event.pageY - 10) + "px")
.style("left", (event.pageX + 10) + "px");
})
.on("mouseout", function() {
d3.select(this).attr("opacity", 0.7);
tooltip.style("visibility", "hidden");
});
```
### Zoom and pan
```javascript
const zoom = d3.zoom()
.scaleExtent([0.5, 10])
.on("zoom", (event) => {
g.attr("transform", event.transform);
});
svg.call(zoom);
```
### Click interactions
```javascript
circles
.on("click", function(event, d) {
// Handle click (dispatch event, update app state, etc.)
console.log("Clicked:", d);
// Visual feedback
d3.selectAll("circle").attr("fill", "steelblue");
d3.select(this).attr("fill", "orange");
// Optional: dispatch custom event for your framework/app to listen to
// window.dispatchEvent(new CustomEvent('chartClick', { detail: d }));
});
```
## Transitions and animations
Add smooth transitions to visual changes:
```javascript
// Basic transition
circles
.transition()
.duration(750)
.attr("r", 10);
// Chained transitions
circles
.transition()
.duration(500)
.attr("fill", "orange")
.transition()
.duration(500)
.attr("r", 15);
// Staggered transitions
circles
.transition()
.delay((d, i) => i * 50)
.duration(500)
.attr("cy", d => yScale(d.value));
// Custom easing
circles
.transition()
.duration(1000)
.ease(d3.easeBounceOut)
.attr("r", 10);
```
## Scales reference
### Quantitative scales
```javascript
// Linear scale
const xScale = d3.scaleLinear()
.domain([0, 100])
.range([0, 500]);
// Log scale (for exponential data)
const logScale = d3.scaleLog()
.domain([1, 1000])
.range([0, 500]);
// Power scale
const powScale = d3.scalePow()
.exponent(2)
.domain([0, 100])
.range([0, 500]);
// Time scale
const timeScale = d3.scaleTime()
.domain([new Date(2020, 0, 1), new Date(2024, 0, 1)])
.range([0, 500]);
```
### Ordinal scales
```javascript
// Band scale (for bar charts)
const bandScale = d3.scaleBand()
.domain(['A', 'B', 'C', 'D'])
.range([0, 400])
.padding(0.1);
// Point scale (for line/scatter categories)
const pointScale = d3.scalePoint()
.domain(['A', 'B', 'C', 'D'])
.range([0, 400]);
// Ordinal scale (for colours)
const colourScale = d3.scaleOrdinal(d3.schemeCategory10);
```
### Sequential scales
```javascript
// Sequential colour scale
const colourScale = d3.scaleSequential(d3.interpolateBlues)
.domain([0, 100]);
// Diverging colour scale
const divScale = d3.scaleDiverging(d3.interpolateRdBu)
.domain([-10, 0, 10]);
```
## Best practices
### Data preparation
Always validate and prepare data before visualisation:
```javascript
// Filter invalid values
const cleanData = data.filter(d => d.value != null && !isNaN(d.value));
// Sort data if order matters
const sortedData = [...data].sort((a, b) => b.value - a.value);
// Parse dates
const parsedData = data.map(d => ({
...d,
date: d3.timeParse("%Y-%m-%d")(d.date)
}));
```
### Performance optimisation
For large datasets (>1000 elements):
```javascript
// Use canvas instead of SVG for many elements
// Use quadtree for collision detection
// Simplify paths with d3.line().curve(d3.curveStep)
// Implement virtual scrolling for large lists
// Use requestAnimationFrame for custom animations
```
### Accessibility
Make visualisations accessible:
```javascript
// Add ARIA labels
svg.attr("role", "img")
.attr("aria-label", "Bar chart showing quarterly revenue");
// Add title and description
svg.append("title").text("Quarterly Revenue 2024");
svg.append("desc").text("Bar chart showing revenue growth across four quarters");
// Ensure sufficient colour contrast
// Provide keyboard navigation for interactive elements
// Include data table alternative
```
### Styling
Use consistent, professional styling:
```javascript
// Define colour palettes upfront
const colours = {
primary: '#4A90E2',
secondary: '#7B68EE',
background: '#F5F7FA',
text: '#333333',
gridLines: '#E0E0E0'
};
// Apply consistent typography
svg.selectAll("text")
.style("font-family", "Inter, sans-serif")
.style("font-size", "12px");
// Use subtle grid lines
g.selectAll(".tick line")
.attr("stroke", colours.gridLines)
.attr("stroke-dasharray", "2,2");
```
## Common issues and solutions
**Issue**: Axes not appearing
- Ensure scales have valid domains (check for NaN values)
- Verify axis is appended to correct group
- Check transform translations are correct
**Issue**: Transitions not working
- Call `.transition()` before attribute changes
- Ensure elements have unique keys for proper data binding
- Check that useEffect dependencies include all changing data
**Issue**: Responsive sizing not working
- Use ResizeObserver or window resize listener
- Update dimensions in state to trigger re-render
- Ensure SVG has width/height attributes or viewBox
**Issue**: Performance problems
- Limit number of DOM elements (consider canvas for >1000 items)
- Debounce resize handlers
- Use `.join()` instead of separate enter/update/exit selections
- Avoid unnecessary re-renders by checking dependencies
## Resources
### references/
Contains detailed reference materials:
- `d3-patterns.md` - Comprehensive collection of visualisation patterns and code examples
- `scale-reference.md` - Complete guide to d3 scales with examples
- `colour-schemes.md` - D3 colour schemes and palette recommendations
### assets/
Contains boilerplate templates:
- `chart-template.js` - Starter template for basic chart
- `interactive-template.js` - Template with tooltips, zoom, and interactions
- `sample-data.json` - Example datasets for testing
These templates work with vanilla JavaScript, React, Vue, Svelte, or any other JavaScript environment. Adapt them as needed for your specific framework.
To use these resources, read the relevant files when detailed guidance is needed for specific visualisation types or patterns.
.claude/skills/d3-viz/assets/chart-template.jsx
+106
View File
@@ -0,0 +1,106 @@
import { useEffect, useRef, useState } from 'react';
import * as d3 from 'd3';
function BasicChart({ data }) {
const svgRef = useRef();
useEffect(() => {
if (!data || data.length === 0) return;
// Select SVG element
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove(); // Clear previous content
// Define dimensions and margins
const width = 800;
const height = 400;
const margin = { top: 20, right: 30, bottom: 40, left: 50 };
const innerWidth = width - margin.left - margin.right;
const innerHeight = height - margin.top - margin.bottom;
// Create main group with margins
const g = svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`);
// Create scales
const xScale = d3.scaleBand()
.domain(data.map(d => d.label))
.range([0, innerWidth])
.padding(0.1);
const yScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.value)])
.range([innerHeight, 0])
.nice();
// Create and append axes
const xAxis = d3.axisBottom(xScale);
const yAxis = d3.axisLeft(yScale);
g.append("g")
.attr("class", "x-axis")
.attr("transform", `translate(0,${innerHeight})`)
.call(xAxis);
g.append("g")
.attr("class", "y-axis")
.call(yAxis);
// Bind data and create visual elements (bars in this example)
g.selectAll("rect")
.data(data)
.join("rect")
.attr("x", d => xScale(d.label))
.attr("y", d => yScale(d.value))
.attr("width", xScale.bandwidth())
.attr("height", d => innerHeight - yScale(d.value))
.attr("fill", "steelblue");
// Optional: Add axis labels
g.append("text")
.attr("class", "axis-label")
.attr("x", innerWidth / 2)
.attr("y", innerHeight + margin.bottom - 5)
.attr("text-anchor", "middle")
.text("Category");
g.append("text")
.attr("class", "axis-label")
.attr("transform", "rotate(-90)")
.attr("x", -innerHeight / 2)
.attr("y", -margin.left + 15)
.attr("text-anchor", "middle")
.text("Value");
}, [data]);
return (
<div className="chart-container">
<svg
ref={svgRef}
width="800"
height="400"
style={{ border: '1px solid #ddd' }}
/>
</div>
);
}
// Example usage
export default function App() {
const sampleData = [
{ label: 'A', value: 30 },
{ label: 'B', value: 80 },
{ label: 'C', value: 45 },
{ label: 'D', value: 60 },
{ label: 'E', value: 20 },
{ label: 'F', value: 90 }
];
return (
<div className="p-8">
<h1 className="text-2xl font-bold mb-4">Basic D3.js Chart</h1>
<BasicChart data={sampleData} />
</div>
);
}
.claude/skills/d3-viz/assets/interactive-template.jsx
+227
View File
@@ -0,0 +1,227 @@
import { useEffect, useRef, useState } from 'react';
import * as d3 from 'd3';
function InteractiveChart({ data }) {
const svgRef = useRef();
const tooltipRef = useRef();
const [selectedPoint, setSelectedPoint] = useState(null);
useEffect(() => {
if (!data || data.length === 0) return;
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove();
// Dimensions
const width = 800;
const height = 500;
const margin = { top: 20, right: 30, bottom: 40, left: 50 };
const innerWidth = width - margin.left - margin.right;
const innerHeight = height - margin.top - margin.bottom;
// Create main group
const g = svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`);
// Scales
const xScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.x)])
.range([0, innerWidth])
.nice();
const yScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.y)])
.range([innerHeight, 0])
.nice();
const sizeScale = d3.scaleSqrt()
.domain([0, d3.max(data, d => d.size || 10)])
.range([3, 20]);
const colourScale = d3.scaleOrdinal(d3.schemeCategory10);
// Add zoom behaviour
const zoom = d3.zoom()
.scaleExtent([0.5, 10])
.on("zoom", (event) => {
g.attr("transform", `translate(${margin.left + event.transform.x},${margin.top + event.transform.y}) scale(${event.transform.k})`);
});
svg.call(zoom);
// Axes
const xAxis = d3.axisBottom(xScale);
const yAxis = d3.axisLeft(yScale);
const xAxisGroup = g.append("g")
.attr("class", "x-axis")
.attr("transform", `translate(0,${innerHeight})`)
.call(xAxis);
const yAxisGroup = g.append("g")
.attr("class", "y-axis")
.call(yAxis);
// Grid lines
g.append("g")
.attr("class", "grid")
.attr("opacity", 0.1)
.call(d3.axisLeft(yScale)
.tickSize(-innerWidth)
.tickFormat(""));
g.append("g")
.attr("class", "grid")
.attr("opacity", 0.1)
.attr("transform", `translate(0,${innerHeight})`)
.call(d3.axisBottom(xScale)
.tickSize(-innerHeight)
.tickFormat(""));
// Tooltip
const tooltip = d3.select(tooltipRef.current);
// Data points
const circles = g.selectAll("circle")
.data(data)
.join("circle")
.attr("cx", d => xScale(d.x))
.attr("cy", d => yScale(d.y))
.attr("r", d => sizeScale(d.size || 10))
.attr("fill", d => colourScale(d.category || 'default'))
.attr("stroke", "#fff")
.attr("stroke-width", 2)
.attr("opacity", 0.7)
.style("cursor", "pointer");
// Hover interactions
circles
.on("mouseover", function(event, d) {
// Enlarge circle
d3.select(this)
.transition()
.duration(200)
.attr("opacity", 1)
.attr("stroke-width", 3);
// Show tooltip
tooltip
.style("display", "block")
.style("left", (event.pageX + 10) + "px")
.style("top", (event.pageY - 10) + "px")
.html(`
<strong>${d.label || 'Point'}</strong><br/>
X: ${d.x.toFixed(2)}<br/>
Y: ${d.y.toFixed(2)}<br/>
${d.category ? `Category: ${d.category}<br/>` : ''}
${d.size ? `Size: ${d.size.toFixed(2)}` : ''}
`);
})
.on("mousemove", function(event) {
tooltip
.style("left", (event.pageX + 10) + "px")
.style("top", (event.pageY - 10) + "px");
})
.on("mouseout", function() {
// Restore circle
d3.select(this)
.transition()
.duration(200)
.attr("opacity", 0.7)
.attr("stroke-width", 2);
// Hide tooltip
tooltip.style("display", "none");
})
.on("click", function(event, d) {
// Highlight selected point
circles.attr("stroke", "#fff").attr("stroke-width", 2);
d3.select(this)
.attr("stroke", "#000")
.attr("stroke-width", 3);
setSelectedPoint(d);
});
// Add transition on initial render
circles
.attr("r", 0)
.transition()
.duration(800)
.delay((d, i) => i * 20)
.attr("r", d => sizeScale(d.size || 10));
// Axis labels
g.append("text")
.attr("class", "axis-label")
.attr("x", innerWidth / 2)
.attr("y", innerHeight + margin.bottom - 5)
.attr("text-anchor", "middle")
.style("font-size", "14px")
.text("X Axis");
g.append("text")
.attr("class", "axis-label")
.attr("transform", "rotate(-90)")
.attr("x", -innerHeight / 2)
.attr("y", -margin.left + 15)
.attr("text-anchor", "middle")
.style("font-size", "14px")
.text("Y Axis");
}, [data]);
return (
<div className="relative">
<svg
ref={svgRef}
width="800"
height="500"
style={{ border: '1px solid #ddd', cursor: 'grab' }}
/>
<div
ref={tooltipRef}
style={{
position: 'absolute',
display: 'none',
padding: '10px',
background: 'white',
border: '1px solid #ddd',
borderRadius: '4px',
pointerEvents: 'none',
boxShadow: '0 2px 4px rgba(0,0,0,0.1)',
fontSize: '13px',
zIndex: 1000
}}
/>
{selectedPoint && (
<div className="mt-4 p-4 bg-blue-50 rounded border border-blue-200">
<h3 className="font-bold mb-2">Selected Point</h3>
<pre className="text-sm">{JSON.stringify(selectedPoint, null, 2)}</pre>
</div>
)}
</div>
);
}
// Example usage
export default function App() {
const sampleData = Array.from({ length: 50 }, (_, i) => ({
id: i,
label: `Point ${i + 1}`,
x: Math.random() * 100,
y: Math.random() * 100,
size: Math.random() * 30 + 5,
category: ['A', 'B', 'C', 'D'][Math.floor(Math.random() * 4)]
}));
return (
<div className="p-8">
<h1 className="text-2xl font-bold mb-2">Interactive D3.js Chart</h1>
<p className="text-gray-600 mb-4">
Hover over points for details. Click to select. Scroll to zoom. Drag to pan.
</p>
<InteractiveChart data={sampleData} />
</div>
);
}
.claude/skills/d3-viz/assets/sample-data.json
+115
View File
@@ -0,0 +1,115 @@
{
"timeSeries": [
{ "date": "2024-01-01", "value": 120, "category": "A" },
{ "date": "2024-02-01", "value": 135, "category": "A" },
{ "date": "2024-03-01", "value": 128, "category": "A" },
{ "date": "2024-04-01", "value": 145, "category": "A" },
{ "date": "2024-05-01", "value": 152, "category": "A" },
{ "date": "2024-06-01", "value": 168, "category": "A" },
{ "date": "2024-07-01", "value": 175, "category": "A" },
{ "date": "2024-08-01", "value": 182, "category": "A" },
{ "date": "2024-09-01", "value": 190, "category": "A" },
{ "date": "2024-10-01", "value": 185, "category": "A" },
{ "date": "2024-11-01", "value": 195, "category": "A" },
{ "date": "2024-12-01", "value": 210, "category": "A" }
],
"categorical": [
{ "label": "Product A", "value": 450, "category": "Electronics" },
{ "label": "Product B", "value": 320, "category": "Electronics" },
{ "label": "Product C", "value": 580, "category": "Clothing" },
{ "label": "Product D", "value": 290, "category": "Clothing" },
{ "label": "Product E", "value": 410, "category": "Food" },
{ "label": "Product F", "value": 370, "category": "Food" }
],
"scatterData": [
{ "x": 12, "y": 45, "size": 25, "category": "Group A", "label": "Point 1" },
{ "x": 25, "y": 62, "size": 35, "category": "Group A", "label": "Point 2" },
{ "x": 38, "y": 55, "size": 20, "category": "Group B", "label": "Point 3" },
{ "x": 45, "y": 78, "size": 40, "category": "Group B", "label": "Point 4" },
{ "x": 52, "y": 68, "size": 30, "category": "Group C", "label": "Point 5" },
{ "x": 65, "y": 85, "size": 45, "category": "Group C", "label": "Point 6" },
{ "x": 72, "y": 72, "size": 28, "category": "Group A", "label": "Point 7" },
{ "x": 85, "y": 92, "size": 50, "category": "Group B", "label": "Point 8" }
],
"hierarchical": {
"name": "Root",
"children": [
{
"name": "Category 1",
"children": [
{ "name": "Item 1.1", "value": 100 },
{ "name": "Item 1.2", "value": 150 },
{ "name": "Item 1.3", "value": 80 }
]
},
{
"name": "Category 2",
"children": [
{ "name": "Item 2.1", "value": 200 },
{ "name": "Item 2.2", "value": 120 },
{ "name": "Item 2.3", "value": 90 }
]
},
{
"name": "Category 3",
"children": [
{ "name": "Item 3.1", "value": 180 },
{ "name": "Item 3.2", "value": 140 }
]
}
]
},
"network": {
"nodes": [
{ "id": "A", "group": 1 },
{ "id": "B", "group": 1 },
{ "id": "C", "group": 1 },
{ "id": "D", "group": 2 },
{ "id": "E", "group": 2 },
{ "id": "F", "group": 3 },
{ "id": "G", "group": 3 },
{ "id": "H", "group": 3 }
],
"links": [
{ "source": "A", "target": "B", "value": 1 },
{ "source": "A", "target": "C", "value": 2 },
{ "source": "B", "target": "C", "value": 1 },
{ "source": "C", "target": "D", "value": 3 },
{ "source": "D", "target": "E", "value": 2 },
{ "source": "E", "target": "F", "value": 1 },
{ "source": "F", "target": "G", "value": 2 },
{ "source": "F", "target": "H", "value": 1 },
{ "source": "G", "target": "H", "value": 1 }
]
},
"stackedData": [
{ "group": "Q1", "seriesA": 30, "seriesB": 40, "seriesC": 25 },
{ "group": "Q2", "seriesA": 45, "seriesB": 35, "seriesC": 30 },
{ "group": "Q3", "seriesA": 40, "seriesB": 50, "seriesC": 35 },
{ "group": "Q4", "seriesA": 55, "seriesB": 45, "seriesC": 40 }
],
"geographicPoints": [
{ "city": "London", "latitude": 51.5074, "longitude": -0.1278, "value": 8900000 },
{ "city": "Paris", "latitude": 48.8566, "longitude": 2.3522, "value": 2140000 },
{ "city": "Berlin", "latitude": 52.5200, "longitude": 13.4050, "value": 3645000 },
{ "city": "Madrid", "latitude": 40.4168, "longitude": -3.7038, "value": 3223000 },
{ "city": "Rome", "latitude": 41.9028, "longitude": 12.4964, "value": 2873000 }
],
"divergingData": [
{ "category": "Item A", "value": -15 },
{ "category": "Item B", "value": 8 },
{ "category": "Item C", "value": -22 },
{ "category": "Item D", "value": 18 },
{ "category": "Item E", "value": -5 },
{ "category": "Item F", "value": 25 },
{ "category": "Item G", "value": -12 },
{ "category": "Item H", "value": 14 }
]
}
.claude/skills/d3-viz/references/colour-schemes.md
+564
View File
@@ -0,0 +1,564 @@
# D3.js Colour Schemes and Palette Recommendations
Comprehensive guide to colour selection in data visualisation with d3.js.
## Built-in categorical colour schemes
### Category10 (default)
```javascript
d3.schemeCategory10
// ['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728', '#9467bd',
// '#8c564b', '#e377c2', '#7f7f7f', '#bcbd22', '#17becf']
```
**Characteristics:**
- 10 distinct colours
- Good colour-blind accessibility
- Default choice for most categorical data
- Balanced saturation and brightness
**Use cases:** General purpose categorical encoding, legend items, multiple data series
### Tableau10
```javascript
d3.schemeTableau10
```
**Characteristics:**
- 10 colours optimised for data visualisation
- Professional appearance
- Excellent distinguishability
**Use cases:** Business dashboards, professional reports, presentations
### Accent
```javascript
d3.schemeAccent
// 8 colours with high saturation
```
**Characteristics:**
- Bright, vibrant colours
- High contrast
- Modern aesthetic
**Use cases:** Highlighting important categories, modern web applications
### Dark2
```javascript
d3.schemeDark2
// 8 darker, muted colours
```
**Characteristics:**
- Subdued palette
- Professional appearance
- Good for dark backgrounds
**Use cases:** Dark mode visualisations, professional contexts
### Paired
```javascript
d3.schemePaired
// 12 colours in pairs of similar hues
```
**Characteristics:**
- Pairs of light and dark variants
- Useful for nested categories
- 12 distinct colours
**Use cases:** Grouped bar charts, hierarchical categories, before/after comparisons
### Pastel1 & Pastel2
```javascript
d3.schemePastel1 // 9 colours
d3.schemePastel2 // 8 colours
```
**Characteristics:**
- Soft, low-saturation colours
- Gentle appearance
- Good for large areas
**Use cases:** Background colours, subtle categorisation, calming visualisations
### Set1, Set2, Set3
```javascript
d3.schemeSet1 // 9 colours - vivid
d3.schemeSet2 // 8 colours - muted
d3.schemeSet3 // 12 colours - pastel
```
**Characteristics:**
- Set1: High saturation, maximum distinction
- Set2: Professional, balanced
- Set3: Subtle, many categories
**Use cases:** Varied based on visual hierarchy needs
## Sequential colour schemes
Sequential schemes map continuous data from low to high values using a single hue or gradient.
### Single-hue sequential
**Blues:**
```javascript
d3.interpolateBlues
d3.schemeBlues[9] // 9-step discrete version
```
**Other single-hue options:**
- `d3.interpolateGreens` / `d3.schemeGreens`
- `d3.interpolateOranges` / `d3.schemeOranges`
- `d3.interpolatePurples` / `d3.schemePurples`
- `d3.interpolateReds` / `d3.schemeReds`
- `d3.interpolateGreys` / `d3.schemeGreys`
**Use cases:**
- Simple heat maps
- Choropleth maps
- Density plots
- Single-metric visualisations
### Multi-hue sequential
**Viridis (recommended):**
```javascript
d3.interpolateViridis
```
**Characteristics:**
- Perceptually uniform
- Colour-blind friendly
- Print-safe
- No visual dead zones
- Monotonically increasing perceived lightness
**Other perceptually-uniform options:**
- `d3.interpolatePlasma` - Purple to yellow
- `d3.interpolateInferno` - Black to white through red/orange
- `d3.interpolateMagma` - Black to white through purple
- `d3.interpolateCividis` - Colour-blind optimised
**Colour-blind accessible:**
```javascript
d3.interpolateTurbo // Rainbow-like but perceptually uniform
d3.interpolateCool // Cyan to magenta
d3.interpolateWarm // Orange to yellow
```
**Use cases:**
- Scientific visualisation
- Medical imaging
- Any high-precision data visualisation
- Accessible visualisations
### Traditional sequential
**Yellow-Orange-Red:**
```javascript
d3.interpolateYlOrRd
d3.schemeYlOrRd[9]
```
**Yellow-Green-Blue:**
```javascript
d3.interpolateYlGnBu
d3.schemeYlGnBu[9]
```
**Other multi-hue:**
- `d3.interpolateBuGn` - Blue to green
- `d3.interpolateBuPu` - Blue to purple
- `d3.interpolateGnBu` - Green to blue
- `d3.interpolateOrRd` - Orange to red
- `d3.interpolatePuBu` - Purple to blue
- `d3.interpolatePuBuGn` - Purple to blue-green
- `d3.interpolatePuRd` - Purple to red
- `d3.interpolateRdPu` - Red to purple
- `d3.interpolateYlGn` - Yellow to green
- `d3.interpolateYlOrBr` - Yellow to orange-brown
**Use cases:** Traditional data visualisation, familiar colour associations (temperature, vegetation, water)
## Diverging colour schemes
Diverging schemes highlight deviations from a central value using two distinct hues.
### Red-Blue (temperature)
```javascript
d3.interpolateRdBu
d3.schemeRdBu[11]
```
**Characteristics:**
- Intuitive temperature metaphor
- Strong contrast
- Clear positive/negative distinction
**Use cases:** Temperature, profit/loss, above/below average, correlation
### Red-Yellow-Blue
```javascript
d3.interpolateRdYlBu
d3.schemeRdYlBu[11]
```
**Characteristics:**
- Three-colour gradient
- Softer transition through yellow
- More visual steps
**Use cases:** When extreme values need emphasis and middle needs visibility
### Other diverging schemes
**Traffic light:**
```javascript
d3.interpolateRdYlGn // Red (bad) to green (good)
```
**Spectral (rainbow):**
```javascript
d3.interpolateSpectral // Full spectrum
```
**Other options:**
- `d3.interpolateBrBG` - Brown to blue-green
- `d3.interpolatePiYG` - Pink to yellow-green
- `d3.interpolatePRGn` - Purple to green
- `d3.interpolatePuOr` - Purple to orange
- `d3.interpolateRdGy` - Red to grey
**Use cases:** Choose based on semantic meaning and accessibility needs
## Colour-blind friendly palettes
### General guidelines
1. **Avoid red-green combinations** (most common colour blindness)
2. **Use blue-orange diverging** instead of red-green
3. **Add texture or patterns** as redundant encoding
4. **Test with simulation tools**
### Recommended colour-blind safe schemes
**Categorical:**
```javascript
// Okabe-Ito palette (colour-blind safe)
const okabePalette = [
'#E69F00', // Orange
'#56B4E9', // Sky blue
'#009E73', // Bluish green
'#F0E442', // Yellow
'#0072B2', // Blue
'#D55E00', // Vermillion
'#CC79A7', // Reddish purple
'#000000' // Black
];
const colourScale = d3.scaleOrdinal()
.domain(categories)
.range(okabePalette);
```
**Sequential:**
```javascript
// Use Viridis, Cividis, or Blues
d3.interpolateViridis // Best overall
d3.interpolateCividis // Optimised for CVD
d3.interpolateBlues // Simple, safe
```
**Diverging:**
```javascript
// Use blue-orange instead of red-green
d3.interpolateBrBG
d3.interpolatePuOr
```
## Custom colour palettes
### Creating custom sequential
```javascript
const customSequential = d3.scaleLinear()
.domain([0, 100])
.range(['#e8f4f8', '#006d9c']) // Light to dark blue
.interpolate(d3.interpolateLab); // Perceptually uniform
```
### Creating custom diverging
```javascript
const customDiverging = d3.scaleLinear()
.domain([0, 50, 100])
.range(['#ca0020', '#f7f7f7', '#0571b0']) // Red, grey, blue
.interpolate(d3.interpolateLab);
```
### Creating custom categorical
```javascript
// Brand colours
const brandPalette = [
'#FF6B6B', // Primary red
'#4ECDC4', // Secondary teal
'#45B7D1', // Tertiary blue
'#FFA07A', // Accent coral
'#98D8C8' // Accent mint
];
const colourScale = d3.scaleOrdinal()
.domain(categories)
.range(brandPalette);
```
## Semantic colour associations
### Universal colour meanings
**Red:**
- Danger, error, negative
- High temperature
- Debt, loss
**Green:**
- Success, positive
- Growth, vegetation
- Profit, gain
**Blue:**
- Trust, calm
- Water, cold
- Information, neutral
**Yellow/Orange:**
- Warning, caution
- Energy, warmth
- Attention
**Grey:**
- Neutral, inactive
- Missing data
- Background
### Context-specific palettes
**Financial:**
```javascript
const financialColours = {
profit: '#27ae60',
loss: '#e74c3c',
neutral: '#95a5a6',
highlight: '#3498db'
};
```
**Temperature:**
```javascript
const temperatureScale = d3.scaleSequential(d3.interpolateRdYlBu)
.domain([40, -10]); // Hot to cold (reversed)
```
**Traffic/Status:**
```javascript
const statusColours = {
success: '#27ae60',
warning: '#f39c12',
error: '#e74c3c',
info: '#3498db',
neutral: '#95a5a6'
};
```
## Accessibility best practices
### Contrast ratios
Ensure sufficient contrast between colours and backgrounds:
```javascript
// Good contrast example
const highContrast = {
background: '#ffffff',
text: '#2c3e50',
primary: '#3498db',
secondary: '#e74c3c'
};
```
**WCAG guidelines:**
- Normal text: 4.5:1 minimum
- Large text: 3:1 minimum
- UI components: 3:1 minimum
### Redundant encoding
Never rely solely on colour to convey information:
```javascript
// Add patterns or shapes
const symbols = ['circle', 'square', 'triangle', 'diamond'];
// Add text labels
// Use line styles (solid, dashed, dotted)
// Use size encoding
```
### Testing
Test visualisations for colour blindness:
- Chrome DevTools (Rendering > Emulate vision deficiencies)
- Colour Oracle (free desktop application)
- Coblis (online simulator)
## Professional colour recommendations
### Data journalism
```javascript
// Guardian style
const guardianPalette = [
'#005689', // Guardian blue
'#c70000', // Guardian red
'#7d0068', // Guardian pink
'#951c75', // Guardian purple
];
// FT style
const ftPalette = [
'#0f5499', // FT blue
'#990f3d', // FT red
'#593380', // FT purple
'#262a33', // FT black
];
```
### Academic/Scientific
```javascript
// Nature journal style
const naturePalette = [
'#0071b2', // Blue
'#d55e00', // Vermillion
'#009e73', // Green
'#f0e442', // Yellow
];
// Use Viridis for continuous data
const scientificScale = d3.scaleSequential(d3.interpolateViridis);
```
### Corporate/Business
```javascript
// Professional, conservative
const corporatePalette = [
'#003f5c', // Dark blue
'#58508d', // Purple
'#bc5090', // Magenta
'#ff6361', // Coral
'#ffa600' // Orange
];
```
## Dynamic colour selection
### Based on data range
```javascript
function selectColourScheme(data) {
const extent = d3.extent(data);
const hasNegative = extent[0] < 0;
const hasPositive = extent[1] > 0;
if (hasNegative && hasPositive) {
// Diverging: data crosses zero
return d3.scaleSequentialSymlog(d3.interpolateRdBu)
.domain([extent[0], 0, extent[1]]);
} else {
// Sequential: all positive or all negative
return d3.scaleSequential(d3.interpolateViridis)
.domain(extent);
}
}
```
### Based on category count
```javascript
function selectCategoricalScheme(categories) {
const n = categories.length;
if (n <= 10) {
return d3.scaleOrdinal(d3.schemeTableau10);
} else if (n <= 12) {
return d3.scaleOrdinal(d3.schemePaired);
} else {
// For many categories, use sequential with quantize
return d3.scaleQuantize()
.domain([0, n - 1])
.range(d3.quantize(d3.interpolateRainbow, n));
}
}
```
## Common colour mistakes to avoid
1. **Rainbow gradients for sequential data**
- Problem: Not perceptually uniform, hard to read
- Solution: Use Viridis, Blues, or other uniform schemes
2. **Red-green for diverging (colour blindness)**
- Problem: 8% of males can't distinguish
- Solution: Use blue-orange or purple-green
3. **Too many categorical colours**
- Problem: Hard to distinguish and remember
- Solution: Limit to 5-8 categories, use grouping
4. **Insufficient contrast**
- Problem: Poor readability
- Solution: Test contrast ratios, use darker colours on light backgrounds
5. **Culturally inconsistent colours**
- Problem: Confusing semantic meaning
- Solution: Research colour associations for target audience
6. **Inverted temperature scales**
- Problem: Counterintuitive (red = cold)
- Solution: Red/orange = hot, blue = cold
## Quick reference guide
**Need to show...**
- **Categories (≤10):** `d3.schemeCategory10` or `d3.schemeTableau10`
- **Categories (>10):** `d3.schemePaired` or group categories
- **Sequential (general):** `d3.interpolateViridis`
- **Sequential (scientific):** `d3.interpolateViridis` or `d3.interpolatePlasma`
- **Sequential (temperature):** `d3.interpolateRdYlBu` (inverted)
- **Diverging (zero):** `d3.interpolateRdBu` or `d3.interpolateBrBG`
- **Diverging (good/bad):** `d3.interpolateRdYlGn` (inverted)
- **Colour-blind safe (categorical):** Okabe-Ito palette (shown above)
- **Colour-blind safe (sequential):** `d3.interpolateCividis` or `d3.interpolateBlues`
- **Colour-blind safe (diverging):** `d3.interpolatePuOr` or `d3.interpolateBrBG`
**Always remember:**
1. Test for colour-blindness
2. Ensure sufficient contrast
3. Use semantic colours appropriately
4. Add redundant encoding (patterns, labels)
5. Keep it simple (fewer colours = clearer visualisation)
.claude/skills/d3-viz/references/d3-patterns.md
+869
View File
@@ -0,0 +1,869 @@
# D3.js Visualisation Patterns
This reference provides detailed code patterns for common d3.js visualisation types.
## Hierarchical visualisations
### Tree diagram
```javascript
useEffect(() => {
if (!data) return;
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove();
const width = 800;
const height = 600;
const tree = d3.tree().size([height - 100, width - 200]);
const root = d3.hierarchy(data);
tree(root);
const g = svg.append("g")
.attr("transform", "translate(100,50)");
// Links
g.selectAll("path")
.data(root.links())
.join("path")
.attr("d", d3.linkHorizontal()
.x(d => d.y)
.y(d => d.x))
.attr("fill", "none")
.attr("stroke", "#555")
.attr("stroke-width", 2);
// Nodes
const node = g.selectAll("g")
.data(root.descendants())
.join("g")
.attr("transform", d => `translate(${d.y},${d.x})`);
node.append("circle")
.attr("r", 6)
.attr("fill", d => d.children ? "#555" : "#999");
node.append("text")
.attr("dy", "0.31em")
.attr("x", d => d.children ? -8 : 8)
.attr("text-anchor", d => d.children ? "end" : "start")
.text(d => d.data.name)
.style("font-size", "12px");
}, [data]);
```
### Treemap
```javascript
useEffect(() => {
if (!data) return;
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove();
const width = 800;
const height = 600;
const root = d3.hierarchy(data)
.sum(d => d.value)
.sort((a, b) => b.value - a.value);
d3.treemap()
.size([width, height])
.padding(2)
.round(true)(root);
const colourScale = d3.scaleOrdinal(d3.schemeCategory10);
const cell = svg.selectAll("g")
.data(root.leaves())
.join("g")
.attr("transform", d => `translate(${d.x0},${d.y0})`);
cell.append("rect")
.attr("width", d => d.x1 - d.x0)
.attr("height", d => d.y1 - d.y0)
.attr("fill", d => colourScale(d.parent.data.name))
.attr("stroke", "white")
.attr("stroke-width", 2);
cell.append("text")
.attr("x", 4)
.attr("y", 16)
.text(d => d.data.name)
.style("font-size", "12px")
.style("fill", "white");
}, [data]);
```
### Sunburst diagram
```javascript
useEffect(() => {
if (!data) return;
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove();
const width = 600;
const height = 600;
const radius = Math.min(width, height) / 2;
const root = d3.hierarchy(data)
.sum(d => d.value)
.sort((a, b) => b.value - a.value);
const partition = d3.partition()
.size([2 * Math.PI, radius]);
partition(root);
const arc = d3.arc()
.startAngle(d => d.x0)
.endAngle(d => d.x1)
.innerRadius(d => d.y0)
.outerRadius(d => d.y1);
const colourScale = d3.scaleOrdinal(d3.schemeCategory10);
const g = svg.append("g")
.attr("transform", `translate(${width / 2},${height / 2})`);
g.selectAll("path")
.data(root.descendants())
.join("path")
.attr("d", arc)
.attr("fill", d => colourScale(d.depth))
.attr("stroke", "white")
.attr("stroke-width", 1);
}, [data]);
```
### Chord diagram
```javascript
function drawChordDiagram(data) {
// data format: array of objects with source, target, and value
// Example: [{ source: 'A', target: 'B', value: 10 }, ...]
if (!data || data.length === 0) return;
const svg = d3.select('#chart');
svg.selectAll("*").remove();
const width = 600;
const height = 600;
const innerRadius = Math.min(width, height) * 0.3;
const outerRadius = innerRadius + 30;
// Create matrix from data
const nodes = Array.from(new Set(data.flatMap(d => [d.source, d.target])));
const matrix = Array.from({ length: nodes.length }, () => Array(nodes.length).fill(0));
data.forEach(d => {
const i = nodes.indexOf(d.source);
const j = nodes.indexOf(d.target);
matrix[i][j] += d.value;
matrix[j][i] += d.value;
});
// Create chord layout
const chord = d3.chord()
.padAngle(0.05)
.sortSubgroups(d3.descending);
const arc = d3.arc()
.innerRadius(innerRadius)
.outerRadius(outerRadius);
const ribbon = d3.ribbon()
.source(d => d.source)
.target(d => d.target);
const colourScale = d3.scaleOrdinal(d3.schemeCategory10)
.domain(nodes);
const g = svg.append("g")
.attr("transform", `translate(${width / 2},${height / 2})`);
const chords = chord(matrix);
// Draw ribbons
g.append("g")
.attr("fill-opacity", 0.67)
.selectAll("path")
.data(chords)
.join("path")
.attr("d", ribbon)
.attr("fill", d => colourScale(nodes[d.source.index]))
.attr("stroke", d => d3.rgb(colourScale(nodes[d.source.index])).darker());
// Draw groups (arcs)
const group = g.append("g")
.selectAll("g")
.data(chords.groups)
.join("g");
group.append("path")
.attr("d", arc)
.attr("fill", d => colourScale(nodes[d.index]))
.attr("stroke", d => d3.rgb(colourScale(nodes[d.index])).darker());
// Add labels
group.append("text")
.each(d => { d.angle = (d.startAngle + d.endAngle) / 2; })
.attr("dy", "0.31em")
.attr("transform", d => `rotate(${(d.angle * 180 / Math.PI) - 90})translate(${outerRadius + 30})${d.angle > Math.PI ? "rotate(180)" : ""}`)
.attr("text-anchor", d => d.angle > Math.PI ? "end" : null)
.text((d, i) => nodes[i])
.style("font-size", "12px");
}
// Data format example:
// const data = [
// { source: 'Category A', target: 'Category B', value: 100 },
// { source: 'Category A', target: 'Category C', value: 50 },
// { source: 'Category B', target: 'Category C', value: 75 }
// ];
// drawChordDiagram(data);
```
## Advanced chart types
### Heatmap
```javascript
function drawHeatmap(data) {
// data format: array of objects with row, column, and value
// Example: [{ row: 'A', column: 'X', value: 10 }, ...]
if (!data || data.length === 0) return;
const svg = d3.select('#chart');
svg.selectAll("*").remove();
const width = 800;
const height = 600;
const margin = { top: 100, right: 30, bottom: 30, left: 100 };
const innerWidth = width - margin.left - margin.right;
const innerHeight = height - margin.top - margin.bottom;
// Get unique rows and columns
const rows = Array.from(new Set(data.map(d => d.row)));
const columns = Array.from(new Set(data.map(d => d.column)));
const g = svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`);
// Create scales
const xScale = d3.scaleBand()
.domain(columns)
.range([0, innerWidth])
.padding(0.01);
const yScale = d3.scaleBand()
.domain(rows)
.range([0, innerHeight])
.padding(0.01);
// Colour scale for values (sequential from light to dark red)
const colourScale = d3.scaleSequential(d3.interpolateYlOrRd)
.domain([0, d3.max(data, d => d.value)]);
// Draw rectangles
g.selectAll("rect")
.data(data)
.join("rect")
.attr("x", d => xScale(d.column))
.attr("y", d => yScale(d.row))
.attr("width", xScale.bandwidth())
.attr("height", yScale.bandwidth())
.attr("fill", d => colourScale(d.value));
// Add x-axis labels
svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`)
.selectAll("text")
.data(columns)
.join("text")
.attr("x", d => xScale(d) + xScale.bandwidth() / 2)
.attr("y", -10)
.attr("text-anchor", "middle")
.text(d => d)
.style("font-size", "12px");
// Add y-axis labels
svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`)
.selectAll("text")
.data(rows)
.join("text")
.attr("x", -10)
.attr("y", d => yScale(d) + yScale.bandwidth() / 2)
.attr("dy", "0.35em")
.attr("text-anchor", "end")
.text(d => d)
.style("font-size", "12px");
// Add colour legend
const legendWidth = 20;
const legendHeight = 200;
const legend = svg.append("g")
.attr("transform", `translate(${width - 60},${margin.top})`);
const legendScale = d3.scaleLinear()
.domain(colourScale.domain())
.range([legendHeight, 0]);
const legendAxis = d3.axisRight(legendScale).ticks(5);
// Draw colour gradient in legend
for (let i = 0; i < legendHeight; i++) {
legend.append("rect")
.attr("y", i)
.attr("width", legendWidth)
.attr("height", 1)
.attr("fill", colourScale(legendScale.invert(i)));
}
legend.append("g")
.attr("transform", `translate(${legendWidth},0)`)
.call(legendAxis);
}
// Data format example:
// const data = [
// { row: 'Monday', column: 'Morning', value: 42 },
// { row: 'Monday', column: 'Afternoon', value: 78 },
// { row: 'Tuesday', column: 'Morning', value: 65 },
// { row: 'Tuesday', column: 'Afternoon', value: 55 }
// ];
// drawHeatmap(data);
```
### Area chart with gradient
```javascript
useEffect(() => {
if (!data || data.length === 0) return;
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove();
const width = 800;
const height = 400;
const margin = { top: 20, right: 30, bottom: 40, left: 50 };
const innerWidth = width - margin.left - margin.right;
const innerHeight = height - margin.top - margin.bottom;
// Define gradient
const defs = svg.append("defs");
const gradient = defs.append("linearGradient")
.attr("id", "areaGradient")
.attr("x1", "0%")
.attr("x2", "0%")
.attr("y1", "0%")
.attr("y2", "100%");
gradient.append("stop")
.attr("offset", "0%")
.attr("stop-color", "steelblue")
.attr("stop-opacity", 0.8);
gradient.append("stop")
.attr("offset", "100%")
.attr("stop-color", "steelblue")
.attr("stop-opacity", 0.1);
const g = svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`);
const xScale = d3.scaleTime()
.domain(d3.extent(data, d => d.date))
.range([0, innerWidth]);
const yScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.value)])
.range([innerHeight, 0]);
const area = d3.area()
.x(d => xScale(d.date))
.y0(innerHeight)
.y1(d => yScale(d.value))
.curve(d3.curveMonotoneX);
g.append("path")
.datum(data)
.attr("fill", "url(#areaGradient)")
.attr("d", area);
const line = d3.line()
.x(d => xScale(d.date))
.y(d => yScale(d.value))
.curve(d3.curveMonotoneX);
g.append("path")
.datum(data)
.attr("fill", "none")
.attr("stroke", "steelblue")
.attr("stroke-width", 2)
.attr("d", line);
g.append("g")
.attr("transform", `translate(0,${innerHeight})`)
.call(d3.axisBottom(xScale));
g.append("g")
.call(d3.axisLeft(yScale));
}, [data]);
```
### Stacked bar chart
```javascript
useEffect(() => {
if (!data || data.length === 0) return;
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove();
const width = 800;
const height = 400;
const margin = { top: 20, right: 30, bottom: 40, left: 50 };
const innerWidth = width - margin.left - margin.right;
const innerHeight = height - margin.top - margin.bottom;
const g = svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`);
const categories = Object.keys(data[0]).filter(k => k !== 'group');
const stackedData = d3.stack().keys(categories)(data);
const xScale = d3.scaleBand()
.domain(data.map(d => d.group))
.range([0, innerWidth])
.padding(0.1);
const yScale = d3.scaleLinear()
.domain([0, d3.max(stackedData[stackedData.length - 1], d => d[1])])
.range([innerHeight, 0]);
const colourScale = d3.scaleOrdinal(d3.schemeCategory10);
g.selectAll("g")
.data(stackedData)
.join("g")
.attr("fill", (d, i) => colourScale(i))
.selectAll("rect")
.data(d => d)
.join("rect")
.attr("x", d => xScale(d.data.group))
.attr("y", d => yScale(d[1]))
.attr("height", d => yScale(d[0]) - yScale(d[1]))
.attr("width", xScale.bandwidth());
g.append("g")
.attr("transform", `translate(0,${innerHeight})`)
.call(d3.axisBottom(xScale));
g.append("g")
.call(d3.axisLeft(yScale));
}, [data]);
```
### Grouped bar chart
```javascript
useEffect(() => {
if (!data || data.length === 0) return;
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove();
const width = 800;
const height = 400;
const margin = { top: 20, right: 30, bottom: 40, left: 50 };
const innerWidth = width - margin.left - margin.right;
const innerHeight = height - margin.top - margin.bottom;
const g = svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`);
const categories = Object.keys(data[0]).filter(k => k !== 'group');
const x0Scale = d3.scaleBand()
.domain(data.map(d => d.group))
.range([0, innerWidth])
.padding(0.1);
const x1Scale = d3.scaleBand()
.domain(categories)
.range([0, x0Scale.bandwidth()])
.padding(0.05);
const yScale = d3.scaleLinear()
.domain([0, d3.max(data, d => Math.max(...categories.map(c => d[c])))])
.range([innerHeight, 0]);
const colourScale = d3.scaleOrdinal(d3.schemeCategory10);
const group = g.selectAll("g")
.data(data)
.join("g")
.attr("transform", d => `translate(${x0Scale(d.group)},0)`);
group.selectAll("rect")
.data(d => categories.map(key => ({ key, value: d[key] })))
.join("rect")
.attr("x", d => x1Scale(d.key))
.attr("y", d => yScale(d.value))
.attr("width", x1Scale.bandwidth())
.attr("height", d => innerHeight - yScale(d.value))
.attr("fill", d => colourScale(d.key));
g.append("g")
.attr("transform", `translate(0,${innerHeight})`)
.call(d3.axisBottom(x0Scale));
g.append("g")
.call(d3.axisLeft(yScale));
}, [data]);
```
### Bubble chart
```javascript
useEffect(() => {
if (!data || data.length === 0) return;
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove();
const width = 800;
const height = 600;
const margin = { top: 20, right: 30, bottom: 40, left: 50 };
const innerWidth = width - margin.left - margin.right;
const innerHeight = height - margin.top - margin.bottom;
const g = svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`);
const xScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.x)])
.range([0, innerWidth]);
const yScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.y)])
.range([innerHeight, 0]);
const sizeScale = d3.scaleSqrt()
.domain([0, d3.max(data, d => d.size)])
.range([0, 50]);
const colourScale = d3.scaleOrdinal(d3.schemeCategory10);
g.selectAll("circle")
.data(data)
.join("circle")
.attr("cx", d => xScale(d.x))
.attr("cy", d => yScale(d.y))
.attr("r", d => sizeScale(d.size))
.attr("fill", d => colourScale(d.category))
.attr("opacity", 0.6)
.attr("stroke", "white")
.attr("stroke-width", 2);
g.append("g")
.attr("transform", `translate(0,${innerHeight})`)
.call(d3.axisBottom(xScale));
g.append("g")
.call(d3.axisLeft(yScale));
}, [data]);
```
## Geographic visualisations
### Basic map with points
```javascript
useEffect(() => {
if (!geoData || !pointData) return;
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove();
const width = 800;
const height = 600;
const projection = d3.geoMercator()
.fitSize([width, height], geoData);
const pathGenerator = d3.geoPath().projection(projection);
// Draw map
svg.selectAll("path")
.data(geoData.features)
.join("path")
.attr("d", pathGenerator)
.attr("fill", "#e0e0e0")
.attr("stroke", "#999")
.attr("stroke-width", 0.5);
// Draw points
svg.selectAll("circle")
.data(pointData)
.join("circle")
.attr("cx", d => projection([d.longitude, d.latitude])[0])
.attr("cy", d => projection([d.longitude, d.latitude])[1])
.attr("r", 5)
.attr("fill", "steelblue")
.attr("opacity", 0.7);
}, [geoData, pointData]);
```
### Choropleth map
```javascript
useEffect(() => {
if (!geoData || !valueData) return;
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove();
const width = 800;
const height = 600;
const projection = d3.geoMercator()
.fitSize([width, height], geoData);
const pathGenerator = d3.geoPath().projection(projection);
// Create value lookup
const valueLookup = new Map(valueData.map(d => [d.id, d.value]));
// Colour scale
const colourScale = d3.scaleSequential(d3.interpolateBlues)
.domain([0, d3.max(valueData, d => d.value)]);
svg.selectAll("path")
.data(geoData.features)
.join("path")
.attr("d", pathGenerator)
.attr("fill", d => {
const value = valueLookup.get(d.id);
return value ? colourScale(value) : "#e0e0e0";
})
.attr("stroke", "#999")
.attr("stroke-width", 0.5);
}, [geoData, valueData]);
```
## Advanced interactions
### Brush and zoom
```javascript
useEffect(() => {
if (!data || data.length === 0) return;
const svg = d3.select(svgRef.current);
svg.selectAll("*").remove();
const width = 800;
const height = 400;
const margin = { top: 20, right: 30, bottom: 40, left: 50 };
const innerWidth = width - margin.left - margin.right;
const innerHeight = height - margin.top - margin.bottom;
const xScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.x)])
.range([0, innerWidth]);
const yScale = d3.scaleLinear()
.domain([0, d3.max(data, d => d.y)])
.range([innerHeight, 0]);
const g = svg.append("g")
.attr("transform", `translate(${margin.left},${margin.top})`);
const circles = g.selectAll("circle")
.data(data)
.join("circle")
.attr("cx", d => xScale(d.x))
.attr("cy", d => yScale(d.y))
.attr("r", 5)
.attr("fill", "steelblue");
// Add brush
const brush = d3.brush()
.extent([[0, 0], [innerWidth, innerHeight]])
.on("start brush", (event) => {
if (!event.selection) return;
const [[x0, y0], [x1, y1]] = event.selection;
circles.attr("fill", d => {
const cx = xScale(d.x);
const cy = yScale(d.y);
return (cx >= x0 && cx <= x1 && cy >= y0 && cy <= y1)
? "orange"
: "steelblue";
});
});
g.append("g")
.attr("class", "brush")
.call(brush);
}, [data]);
```
### Linked brushing between charts
```javascript
function LinkedCharts({ data }) {
const [selectedPoints, setSelectedPoints] = useState(new Set());
const svg1Ref = useRef();
const svg2Ref = useRef();
useEffect(() => {
// Chart 1: Scatter plot
const svg1 = d3.select(svg1Ref.current);
svg1.selectAll("*").remove();
// ... create first chart ...
const circles1 = svg1.selectAll("circle")
.data(data)
.join("circle")
.attr("fill", d => selectedPoints.has(d.id) ? "orange" : "steelblue");
// Chart 2: Bar chart
const svg2 = d3.select(svg2Ref.current);
svg2.selectAll("*").remove();
// ... create second chart ...
const bars = svg2.selectAll("rect")
.data(data)
.join("rect")
.attr("fill", d => selectedPoints.has(d.id) ? "orange" : "steelblue");
// Add brush to first chart
const brush = d3.brush()
.on("start brush end", (event) => {
if (!event.selection) {
setSelectedPoints(new Set());
return;
}
const [[x0, y0], [x1, y1]] = event.selection;
const selected = new Set();
data.forEach(d => {
const x = xScale(d.x);
const y = yScale(d.y);
if (x >= x0 && x <= x1 && y >= y0 && y <= y1) {
selected.add(d.id);
}
});
setSelectedPoints(selected);
});
svg1.append("g").call(brush);
}, [data, selectedPoints]);
return (
<div>
<svg ref={svg1Ref} width="400" height="300" />
<svg ref={svg2Ref} width="400" height="300" />
</div>
);
}
```
## Animation patterns
### Enter, update, exit with transitions
```javascript
useEffect(() => {
if (!data || data.length === 0) return;
const svg = d3.select(svgRef.current);
const circles = svg.selectAll("circle")
.data(data, d => d.id); // Key function for object constancy
// EXIT: Remove old elements
circles.exit()
.transition()
.duration(500)
.attr("r", 0)
.remove();
// UPDATE: Modify existing elements
circles
.transition()
.duration(500)
.attr("cx", d => xScale(d.x))
.attr("cy", d => yScale(d.y))
.attr("fill", "steelblue");
// ENTER: Add new elements
circles.enter()
.append("circle")
.attr("cx", d => xScale(d.x))
.attr("cy", d => yScale(d.y))
.attr("r", 0)
.attr("fill", "steelblue")
.transition()
.duration(500)
.attr("r", 5);
}, [data]);
```
### Path morphing
```javascript
useEffect(() => {
if (!data1 || !data2) return;
const svg = d3.select(svgRef.current);
const line = d3.line()
.x(d => xScale(d.x))
.y(d => yScale(d.y))
.curve(d3.curveMonotoneX);
const path = svg.select("path");
// Morph from data1 to data2
path
.datum(data1)
.attr("d", line)
.transition()
.duration(1000)
.attrTween("d", function() {
const previous = d3.select(this).attr("d");
const current = line(data2);
return d3.interpolatePath(previous, current);
});
}, [data1, data2]);
```
.claude/skills/d3-viz/references/scale-reference.md
+509
View File
@@ -0,0 +1,509 @@
# D3.js Scale Reference
Comprehensive guide to all d3 scale types with examples and use cases.
## Continuous scales
### Linear scale
Maps continuous input domain to continuous output range with linear interpolation.
```javascript
const scale = d3.scaleLinear()
.domain([0, 100])
.range([0, 500]);
scale(50); // Returns 250
scale(0); // Returns 0
scale(100); // Returns 500
// Invert scale (get input from output)
scale.invert(250); // Returns 50
```
**Use cases:**
- Most common scale for quantitative data
- Axes, bar lengths, position encoding
- Temperature, prices, counts, measurements
**Methods:**
- `.domain([min, max])` - Set input domain
- `.range([min, max])` - Set output range
- `.invert(value)` - Get domain value from range value
- `.clamp(true)` - Restrict output to range bounds
- `.nice()` - Extend domain to nice round values
### Power scale
Maps continuous input to continuous output with exponential transformation.
```javascript
const sqrtScale = d3.scalePow()
.exponent(0.5) // Square root
.domain([0, 100])
.range([0, 500]);
const squareScale = d3.scalePow()
.exponent(2) // Square
.domain([0, 100])
.range([0, 500]);
// Shorthand for square root
const sqrtScale2 = d3.scaleSqrt()
.domain([0, 100])
.range([0, 500]);
```
**Use cases:**
- Perceptual scaling (human perception is non-linear)
- Area encoding (use square root to map values to circle radii)
- Emphasising differences in small or large values
### Logarithmic scale
Maps continuous input to continuous output with logarithmic transformation.
```javascript
const logScale = d3.scaleLog()
.domain([1, 1000]) // Must be positive
.range([0, 500]);
logScale(1); // Returns 0
logScale(10); // Returns ~167
logScale(100); // Returns ~333
logScale(1000); // Returns 500
```
**Use cases:**
- Data spanning multiple orders of magnitude
- Population, GDP, wealth distributions
- Logarithmic axes
- Exponential growth visualisations
**Important:** Domain values must be strictly positive (>0).
### Time scale
Specialised linear scale for temporal data.
```javascript
const timeScale = d3.scaleTime()
.domain([new Date(2020, 0, 1), new Date(2024, 0, 1)])
.range([0, 800]);
timeScale(new Date(2022, 0, 1)); // Returns 400
// Invert to get date
timeScale.invert(400); // Returns Date object for mid-2022
```
**Use cases:**
- Time series visualisations
- Timeline axes
- Temporal animations
- Date-based interactions
**Methods:**
- `.nice()` - Extend domain to nice time intervals
- `.ticks(count)` - Generate nicely-spaced tick values
- All linear scale methods apply
### Quantize scale
Maps continuous input to discrete output buckets.
```javascript
const quantizeScale = d3.scaleQuantize()
.domain([0, 100])
.range(['low', 'medium', 'high']);
quantizeScale(25); // Returns 'low'
quantizeScale(50); // Returns 'medium'
quantizeScale(75); // Returns 'high'
// Get the threshold values
quantizeScale.thresholds(); // Returns [33.33, 66.67]
```
**Use cases:**
- Binning continuous data
- Heat map colours
- Risk categories (low/medium/high)
- Age groups, income brackets
### Quantile scale
Maps continuous input to discrete output based on quantiles.
```javascript
const quantileScale = d3.scaleQuantile()
.domain([3, 6, 7, 8, 8, 10, 13, 15, 16, 20, 24]) // Sample data
.range(['low', 'medium', 'high']);
quantileScale(8); // Returns based on quantile position
quantileScale.quantiles(); // Returns quantile thresholds
```
**Use cases:**
- Equal-size groups regardless of distribution
- Percentile-based categorisation
- Handling skewed distributions
### Threshold scale
Maps continuous input to discrete output with custom thresholds.
```javascript
const thresholdScale = d3.scaleThreshold()
.domain([0, 10, 20])
.range(['freezing', 'cold', 'warm', 'hot']);
thresholdScale(-5); // Returns 'freezing'
thresholdScale(5); // Returns 'cold'
thresholdScale(15); // Returns 'warm'
thresholdScale(25); // Returns 'hot'
```
**Use cases:**
- Custom breakpoints
- Grade boundaries (A, B, C, D, F)
- Temperature categories
- Air quality indices
## Sequential scales
### Sequential colour scale
Maps continuous input to continuous colour gradient.
```javascript
const colourScale = d3.scaleSequential(d3.interpolateBlues)
.domain([0, 100]);
colourScale(0); // Returns lightest blue
colourScale(50); // Returns mid blue
colourScale(100); // Returns darkest blue
```
**Available interpolators:**
**Single hue:**
- `d3.interpolateBlues`, `d3.interpolateGreens`, `d3.interpolateReds`
- `d3.interpolateOranges`, `d3.interpolatePurples`, `d3.interpolateGreys`
**Multi-hue:**
- `d3.interpolateViridis`, `d3.interpolateInferno`, `d3.interpolateMagma`
- `d3.interpolatePlasma`, `d3.interpolateWarm`, `d3.interpolateCool`
- `d3.interpolateCubehelixDefault`, `d3.interpolateTurbo`
**Use cases:**
- Heat maps, choropleth maps
- Continuous data visualisation
- Temperature, elevation, density
### Diverging colour scale
Maps continuous input to diverging colour gradient with a midpoint.
```javascript
const divergingScale = d3.scaleDiverging(d3.interpolateRdBu)
.domain([-10, 0, 10]);
divergingScale(-10); // Returns red
divergingScale(0); // Returns white/neutral
divergingScale(10); // Returns blue
```
**Available interpolators:**
- `d3.interpolateRdBu` - Red to blue
- `d3.interpolateRdYlBu` - Red, yellow, blue
- `d3.interpolateRdYlGn` - Red, yellow, green
- `d3.interpolatePiYG` - Pink, yellow, green
- `d3.interpolateBrBG` - Brown, blue-green
- `d3.interpolatePRGn` - Purple, green
- `d3.interpolatePuOr` - Purple, orange
- `d3.interpolateRdGy` - Red, grey
- `d3.interpolateSpectral` - Rainbow spectrum
**Use cases:**
- Data with meaningful midpoint (zero, average, neutral)
- Positive/negative values
- Above/below comparisons
- Correlation matrices
### Sequential quantile scale
Combines sequential colour with quantile mapping.
```javascript
const sequentialQuantileScale = d3.scaleSequentialQuantile(d3.interpolateBlues)
.domain([3, 6, 7, 8, 8, 10, 13, 15, 16, 20, 24]);
// Maps based on quantile position
```
**Use cases:**
- Perceptually uniform binning
- Handling outliers
- Skewed distributions
## Ordinal scales
### Band scale
Maps discrete input to continuous bands (rectangles) with optional padding.
```javascript
const bandScale = d3.scaleBand()
.domain(['A', 'B', 'C', 'D'])
.range([0, 400])
.padding(0.1);
bandScale('A'); // Returns start position (e.g., 0)
bandScale('B'); // Returns start position (e.g., 110)
bandScale.bandwidth(); // Returns width of each band (e.g., 95)
bandScale.step(); // Returns total step including padding
bandScale.paddingInner(); // Returns inner padding (between bands)
bandScale.paddingOuter(); // Returns outer padding (at edges)
```
**Use cases:**
- Bar charts (most common use case)
- Grouped elements
- Categorical axes
- Heat map cells
**Padding options:**
- `.padding(value)` - Sets both inner and outer padding (0-1)
- `.paddingInner(value)` - Padding between bands (0-1)
- `.paddingOuter(value)` - Padding at edges (0-1)
- `.align(value)` - Alignment of bands (0-1, default 0.5)
### Point scale
Maps discrete input to continuous points (no width).
```javascript
const pointScale = d3.scalePoint()
.domain(['A', 'B', 'C', 'D'])
.range([0, 400])
.padding(0.5);
pointScale('A'); // Returns position (e.g., 50)
pointScale('B'); // Returns position (e.g., 150)
pointScale('C'); // Returns position (e.g., 250)
pointScale('D'); // Returns position (e.g., 350)
pointScale.step(); // Returns distance between points
```
**Use cases:**
- Line chart categorical x-axis
- Scatter plot with categorical axis
- Node positions in network graphs
- Any point positioning for categories
### Ordinal colour scale
Maps discrete input to discrete output (colours, shapes, etc.).
```javascript
const colourScale = d3.scaleOrdinal(d3.schemeCategory10);
colourScale('apples'); // Returns first colour
colourScale('oranges'); // Returns second colour
colourScale('apples'); // Returns same first colour (consistent)
// Custom range
const customScale = d3.scaleOrdinal()
.domain(['cat1', 'cat2', 'cat3'])
.range(['#FF6B6B', '#4ECDC4', '#45B7D1']);
```
**Built-in colour schemes:**
**Categorical:**
- `d3.schemeCategory10` - 10 colours
- `d3.schemeAccent` - 8 colours
- `d3.schemeDark2` - 8 colours
- `d3.schemePaired` - 12 colours
- `d3.schemePastel1` - 9 colours
- `d3.schemePastel2` - 8 colours
- `d3.schemeSet1` - 9 colours
- `d3.schemeSet2` - 8 colours
- `d3.schemeSet3` - 12 colours
- `d3.schemeTableau10` - 10 colours
**Use cases:**
- Category colours
- Legend items
- Multi-series charts
- Network node types
## Scale utilities
### Nice domain
Extend domain to nice round values.
```javascript
const scale = d3.scaleLinear()
.domain([0.201, 0.996])
.nice();
scale.domain(); // Returns [0.2, 1.0]
// With count (approximate tick count)
const scale2 = d3.scaleLinear()
.domain([0.201, 0.996])
.nice(5);
```
### Clamping
Restrict output to range bounds.
```javascript
const scale = d3.scaleLinear()
.domain([0, 100])
.range([0, 500])
.clamp(true);
scale(-10); // Returns 0 (clamped)
scale(150); // Returns 500 (clamped)
```
### Copy scales
Create independent copies.
```javascript
const scale1 = d3.scaleLinear()
.domain([0, 100])
.range([0, 500]);
const scale2 = scale1.copy();
// scale2 is independent of scale1
```
### Tick generation
Generate nice tick values for axes.
```javascript
const scale = d3.scaleLinear()
.domain([0, 100])
.range([0, 500]);
scale.ticks(10); // Generate ~10 ticks
scale.tickFormat(10); // Get format function for ticks
scale.tickFormat(10, ".2f"); // Custom format (2 decimal places)
// Time scale ticks
const timeScale = d3.scaleTime()
.domain([new Date(2020, 0, 1), new Date(2024, 0, 1)]);
timeScale.ticks(d3.timeYear); // Yearly ticks
timeScale.ticks(d3.timeMonth, 3); // Every 3 months
timeScale.tickFormat(5, "%Y-%m"); // Format as year-month
```
## Colour spaces and interpolation
### RGB interpolation
```javascript
const scale = d3.scaleLinear()
.domain([0, 100])
.range(["blue", "red"]);
// Default: RGB interpolation
```
### HSL interpolation
```javascript
const scale = d3.scaleLinear()
.domain([0, 100])
.range(["blue", "red"])
.interpolate(d3.interpolateHsl);
// Smoother colour transitions
```
### Lab interpolation
```javascript
const scale = d3.scaleLinear()
.domain([0, 100])
.range(["blue", "red"])
.interpolate(d3.interpolateLab);
// Perceptually uniform
```
### HCL interpolation
```javascript
const scale = d3.scaleLinear()
.domain([0, 100])
.range(["blue", "red"])
.interpolate(d3.interpolateHcl);
// Perceptually uniform with hue
```
## Common patterns
### Diverging scale with custom midpoint
```javascript
const scale = d3.scaleLinear()
.domain([min, midpoint, max])
.range(["red", "white", "blue"])
.interpolate(d3.interpolateHcl);
```
### Multi-stop gradient scale
```javascript
const scale = d3.scaleLinear()
.domain([0, 25, 50, 75, 100])
.range(["#d53e4f", "#fc8d59", "#fee08b", "#e6f598", "#66c2a5"]);
```
### Radius scale for circles (perceptual)
```javascript
const radiusScale = d3.scaleSqrt()
.domain([0, d3.max(data, d => d.value)])
.range([0, 50]);
// Use with circles
circle.attr("r", d => radiusScale(d.value));
```
### Adaptive scale based on data range
```javascript
function createAdaptiveScale(data) {
const extent = d3.extent(data);
const range = extent[1] - extent[0];
// Use log scale if data spans >2 orders of magnitude
if (extent[1] / extent[0] > 100) {
return d3.scaleLog()
.domain(extent)
.range([0, width]);
}
// Otherwise use linear
return d3.scaleLinear()
.domain(extent)
.range([0, width]);
}
```
### Colour scale with explicit categories
```javascript
const colourScale = d3.scaleOrdinal()
.domain(['Low Risk', 'Medium Risk', 'High Risk'])
.range(['#2ecc71', '#f39c12', '#e74c3c'])
.unknown('#95a5a6'); // Fallback for unknown values
```
.claude/skills/elevenlabs/SKILL.md
+376
View File
@@ -0,0 +1,376 @@
---
name: elevenlabs
description: Generate AI voiceovers, sound effects, and music using ElevenLabs APIs. Use when creating audio content for videos, podcasts, or games. Triggers include generating voiceovers, narration, dialogue, sound effects from descriptions, background music, soundtrack generation, voice cloning, or any audio synthesis task.
---
# ElevenLabs Audio Generation
Requires `ELEVENLABS_API_KEY` in `.env`.
## Text-to-Speech
```python
from elevenlabs.client import ElevenLabs
from elevenlabs import save, VoiceSettings
import os
client = ElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY"))
audio = client.text_to_speech.convert(
text="Welcome to my video!",
voice_id="JBFqnCBsd6RMkjVDRZzb",
model_id="eleven_multilingual_v2",
voice_settings=VoiceSettings(
stability=0.5,
similarity_boost=0.75,
style=0.5,
speed=1.0
)
)
save(audio, "voiceover.mp3")
```
### Models
| Model | Quality | SSML Support | Notes |
|-------|---------|--------------|-------|
| `eleven_multilingual_v2` | Highest consistency | None | Stable, production-ready, 29 languages |
| `eleven_flash_v2_5` | Good | `<break>`, `<phoneme>` | Fast, supports pause/pronunciation tags |
| `eleven_turbo_v2_5` | Good | `<break>`, `<phoneme>` | Fastest latency |
| `eleven_v3` | Most expressive | None | Alpha — unreliable, needs prompt engineering |
**Choose:** multilingual_v2 for reliability, flash/turbo for SSML control, v3 for maximum expressiveness (expect retakes).
### Voice Settings by Style
| Style | stability | similarity | style | speed |
|-------|-----------|------------|-------|-------|
| Natural/professional | 0.75-0.85 | 0.9 | 0.0-0.1 | 1.0 |
| Conversational | 0.5-0.6 | 0.85 | 0.3-0.4 | 0.9-1.0 |
| Energetic/YouTuber | 0.3-0.5 | 0.75 | 0.5-0.7 | 1.0-1.1 |
### Pauses Between Sections
**With flash/turbo models:** Use SSML break tags inline:
```
...end of section. <break time="1.5s" /> Start of next...
```
Max 3 seconds per break. Excessive breaks can cause speed artifacts.
**With multilingual_v2 / v3:** No SSML support. Options:
- Paragraph breaks (blank lines) — creates ~0.3-0.5s natural pause
- Post-process with ffmpeg: split audio and insert silence
**WARNING:** `...` (ellipsis) is NOT a reliable pause — it can be vocalized as a word/sound. Do not use ellipsis as a pause mechanism.
### Pronunciation Control
**Phonetic spelling (any model):** Write words as you want them pronounced:
- `Janus` → `Jan-us`
- `nginx` → `engine-x`
- Use dashes, capitals, apostrophes to guide pronunciation
**SSML phoneme tags (flash/turbo only):**
```
<phoneme alphabet="ipa" ph="ˈdʒeɪnəs">Janus</phoneme>
```
### Iterative Workflow
1. Generate → listen → identify pronunciation/pacing issues
2. Adjust: phonetic spellings, break tags, voice settings
3. Regenerate. If pauses aren't precise enough, add silence in post with ffmpeg rather than fighting the TTS engine.
## Voice Cloning
### Instant Voice Clone
```python
with open("sample.mp3", "rb") as f:
voice = client.voices.ivc.create(
name="My Voice",
files=[f],
remove_background_noise=True
)
print(f"Voice ID: {voice.voice_id}")
```
- Use `client.voices.ivc.create()` (not `client.voices.clone()`)
- Pass file handles in binary mode (`"rb"`), not paths
- Convert m4a first: `ffmpeg -i input.m4a -codec:a libmp3lame -qscale:a 2 output.mp3`
- Multiple samples (2-3 clips) improve accuracy
- Save voice ID for reuse
**Professional Voice Clone:** Requires Creator plan+, 30+ min audio. See [reference.md](reference.md).
## Sound Effects
Max 22 seconds per generation.
```python
result = client.text_to_sound_effects.convert(
text="Thunder rumbling followed by heavy rain",
duration_seconds=10,
prompt_influence=0.3
)
with open("thunder.mp3", "wb") as f:
for chunk in result:
f.write(chunk)
```
**Prompt tips:** Be specific — "Heavy footsteps on wooden floorboards, slow and deliberate, with creaking"
## Music Generation
10 seconds to 5 minutes. Use `client.music.compose()` (not `.generate()`).
```python
result = client.music.compose(
prompt="Upbeat indie rock, catchy guitar riff, energetic drums, travel vlog",
music_length_ms=60000,
force_instrumental=True
)
with open("music.mp3", "wb") as f:
for chunk in result:
f.write(chunk)
```
**Prompt structure:** Genre, mood, instruments, tempo, use case. Add "no vocals" or use `force_instrumental=True` for background music.
## Remotion Integration
### Complete Workflow: Script to Synchronized Scene
```
VOICEOVER-SCRIPT.md → voiceover.py → public/audio/ → Remotion composition
↓ ↓ ↓ ↓
Scene narration Generate MP3 Audio files <Audio> component
with durations per scene with timing synced to scenes
```
### Step 1: Generate Per-Scene Audio
Use the toolkit's voiceover tool to generate audio for each scene:
```bash
# Generate voiceover files for each scene
python tools/voiceover.py --scene-dir public/audio/scenes --json
# Output:
# public/audio/scenes/
# ├── scene-01-title.mp3
# ├── scene-02-problem.mp3
# ├── scene-03-solution.mp3
# └── manifest.json (durations for each file)
```
The `manifest.json` contains timing info:
```json
{
"scenes": [
{ "file": "scene-01-title.mp3", "duration": 4.2 },
{ "file": "scene-02-problem.mp3", "duration": 12.8 },
{ "file": "scene-03-solution.mp3", "duration": 15.3 }
],
"totalDuration": 32.3
}
```
### Step 2: Use Audio in Remotion Composition
```tsx
// src/Composition.tsx
import { Audio, staticFile, Series, useVideoConfig } from 'remotion';
// Import scene components
import { TitleSlide } from './scenes/TitleSlide';
import { ProblemSlide } from './scenes/ProblemSlide';
import { SolutionSlide } from './scenes/SolutionSlide';
// Scene durations (from manifest.json, converted to frames at 30fps)
const SCENE_DURATIONS = {
title: Math.ceil(4.2 * 30), // 126 frames
problem: Math.ceil(12.8 * 30), // 384 frames
solution: Math.ceil(15.3 * 30), // 459 frames
};
export const MainComposition: React.FC = () => {
return (
<>
{/* Scene sequence */}
<Series>
<Series.Sequence durationInFrames={SCENE_DURATIONS.title}>
<TitleSlide />
</Series.Sequence>
<Series.Sequence durationInFrames={SCENE_DURATIONS.problem}>
<ProblemSlide />
</Series.Sequence>
<Series.Sequence durationInFrames={SCENE_DURATIONS.solution}>
<SolutionSlide />
</Series.Sequence>
</Series>
{/* Audio track - plays continuously across all scenes */}
<Audio src={staticFile('audio/voiceover.mp3')} volume={1} />
{/* Optional: Background music at lower volume */}
<Audio src={staticFile('audio/music.mp3')} volume={0.15} />
</>
);
};
```
### Step 3: Per-Scene Audio (Alternative)
For more control, add audio to each scene individually:
```tsx
// src/scenes/ProblemSlide.tsx
import { Audio, staticFile, useCurrentFrame } from 'remotion';
export const ProblemSlide: React.FC = () => {
const frame = useCurrentFrame();
return (
<div style={{ /* slide styles */ }}>
<h1>The Problem</h1>
{/* Scene content */}
{/* Audio starts when this scene starts (frame 0 of this sequence) */}
<Audio src={staticFile('audio/scenes/scene-02-problem.mp3')} />
</div>
);
};
```
### Syncing Visuals to Voiceover
Calculate scene duration from audio, not the other way around:
```tsx
// src/config/timing.ts
import manifest from '../../public/audio/scenes/manifest.json';
const FPS = 30;
// Convert audio durations to frame counts
export const sceneDurations = manifest.scenes.reduce((acc, scene) => {
const name = scene.file.replace(/^scene-\d+-/, '').replace('.mp3', '');
acc[name] = Math.ceil(scene.duration * FPS);
return acc;
}, {} as Record<string, number>);
// Usage in composition:
// <Series.Sequence durationInFrames={sceneDurations.title}>
```
### Audio Timing Patterns
```tsx
import { Audio, Sequence, interpolate, useCurrentFrame } from 'remotion';
// Fade in audio
export const FadeInAudio: React.FC<{ src: string; fadeFrames?: number }> = ({
src,
fadeFrames = 30
}) => {
const frame = useCurrentFrame();
const volume = interpolate(frame, [0, fadeFrames], [0, 1], {
extrapolateRight: 'clamp',
});
return <Audio src={src} volume={volume} />;
};
// Delayed audio start
export const DelayedAudio: React.FC<{ src: string; delayFrames: number }> = ({
src,
delayFrames
}) => (
<Sequence from={delayFrames}>
<Audio src={src} />
</Sequence>
);
// Usage:
// <FadeInAudio src={staticFile('audio/music.mp3')} fadeFrames={60} />
// <DelayedAudio src={staticFile('audio/sfx/whoosh.mp3')} delayFrames={45} />
```
### Voiceover + Demo Video Sync
When a scene has both voiceover and demo video:
```tsx
import { Audio, OffthreadVideo, staticFile, useVideoConfig } from 'remotion';
export const DemoScene: React.FC = () => {
const { durationInFrames, fps } = useVideoConfig();
// Calculate playback rate to fit demo into voiceover duration
const demoDuration = 45; // seconds (original demo length)
const sceneDuration = durationInFrames / fps; // seconds (from voiceover)
const playbackRate = demoDuration / sceneDuration;
return (
<>
<OffthreadVideo
src={staticFile('demos/feature-demo.mp4')}
playbackRate={playbackRate}
/>
<Audio src={staticFile('audio/scenes/scene-04-demo.mp3')} />
</>
);
};
```
### Error Handling
```tsx
import { Audio, staticFile, delayRender, continueRender } from 'remotion';
import { useEffect, useState } from 'react';
export const SafeAudio: React.FC<{ src: string }> = ({ src }) => {
const [handle] = useState(() => delayRender());
const [audioReady, setAudioReady] = useState(false);
useEffect(() => {
const audio = new window.Audio(src);
audio.oncanplaythrough = () => {
setAudioReady(true);
continueRender(handle);
};
audio.onerror = () => {
console.error(`Failed to load audio: ${src}`);
continueRender(handle); // Continue without audio rather than hang
};
}, [src, handle]);
if (!audioReady) return null;
return <Audio src={src} />;
};
```
### Toolkit Command: /generate-voiceover
The `/generate-voiceover` command handles the full workflow:
```
/generate-voiceover
1. Reads VOICEOVER-SCRIPT.md
2. Extracts narration for each scene
3. Generates audio via ElevenLabs API
4. Saves to public/audio/scenes/
5. Creates manifest.json with durations
6. Updates project.json with timing info
```
## Popular Voices
- George: `JBFqnCBsd6RMkjVDRZzb` (warm narrator)
- Rachel: `21m00Tcm4TlvDq8ikWAM` (clear female)
- Adam: `pNInz6obpgDQGcFmaJgB` (professional male)
List all: `client.voices.get_all()`
For full API docs, see [reference.md](reference.md).
.claude/skills/elevenlabs/reference.md
+167
View File
@@ -0,0 +1,167 @@
# ElevenLabs API Reference
Detailed API documentation for ElevenLabs audio generation services.
## Authentication
```python
from elevenlabs.client import ElevenLabs
client = ElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY"))
```
## Text-to-Speech Models
| Model ID | Description | Languages | Latency |
|----------|-------------|-----------|---------|
| `eleven_flash_v2_5` | Ultra-low latency streaming | 32 | ~75ms |
| `eleven_multilingual_v2` | Highest quality | 32 | Standard |
| `eleven_turbo_v2_5` | Fast, good quality | 32 | Low |
| `eleven_v3` | Best emotional range (alpha) | 32+ | Higher |
## Voice Settings
| Parameter | Range | Default | Effect |
|-----------|-------|---------|--------|
| `stability` | 0.0-1.0 | 0.5 | Lower = more expressive/variable |
| `similarity_boost` | 0.0-1.0 | 0.75 | Higher = closer to original voice |
| `style` | 0.0-1.0 | 0.0 | Style exaggeration (v2 models) |
| `speed` | 0.5-2.0 | 1.0 | Playback speed multiplier |
## Output Formats
| Format Code | Sample Rate | Bitrate | Tier Required |
|-------------|-------------|---------|---------------|
| `mp3_44100_128` | 44.1kHz | 128kbps | Free (default) |
| `mp3_44100_192` | 44.1kHz | 192kbps | Creator+ |
| `pcm_44100` | 44.1kHz | - | Pro+ |
| `ulaw_8000` | 8kHz | - | Free (telephony) |
## Long-form Audio (Stitching)
For continuity across multiple generations:
```python
result1 = client.text_to_speech.convert_with_timestamps(
text="First paragraph...",
voice_id="JBFqnCBsd6RMkjVDRZzb",
model_id="eleven_multilingual_v2"
)
request_id_1 = result1.request_id
result2 = client.text_to_speech.convert(
text="Second paragraph...",
voice_id="JBFqnCBsd6RMkjVDRZzb",
model_id="eleven_multilingual_v2",
previous_request_ids=[request_id_1]
)
```
## Professional Voice Cloning (PVC)
Requires Creator plan+. Creates a fine-tuned model (3-6 hours training).
**Requirements:**
- 30 min minimum, 2-3 hours optimal audio
- Professional XLR mic recommended
- Pop filter, ~20cm distance
- Peak levels: -6dB to -3dB
- Consistent performance style
**Workflow:**
```python
# 1. Create PVC with samples
pvc = client.voices.create_professional_voice_clone(
name="My Pro Voice",
files=["recording1.mp3", "recording2.mp3", ...],
)
# 2. Get verification captcha
captcha = client.voices.get_pvc_verification_captcha(voice_id=pvc.voice_id)
# Read the captcha text aloud and record
# 3. Submit verification
client.voices.verify_pvc(
voice_id=pvc.voice_id,
recording=open("captcha_reading.mp3", "rb")
)
# 4. Start training
client.voices.start_pvc_training(voice_id=pvc.voice_id)
```
## Sound Effects Parameters
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `text` | string | Yes | Description of sound effect |
| `duration_seconds` | float | No | 1-22 seconds (auto if omitted) |
| `prompt_influence` | float | No | 0.0-1.0 (default 0.3) |
**Billing:** 100 chars/generation (auto) or 25 chars/second (fixed duration)
**Example prompts:**
- Environmental: "Rain on a tin roof, steady and rhythmic"
- Action: "Sword being drawn from sheath, metallic ring"
- Mechanical: "Old car engine struggling to start then roaring to life"
## Music Parameters
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `prompt` | string | Yes* | Natural language music description |
| `composition_plan` | object | Yes* | Detailed composition structure |
| `duration_ms` | int | No | 10000-300000 (10s-5min) |
| `instrumental` | bool | No | Force instrumental output |
*Either `prompt` or `composition_plan` required, not both.
**Effective prompts include:**
1. Genre/Style: "indie rock", "lo-fi hip hop", "orchestral"
2. Mood: "uplifting", "melancholic", "tense"
3. Instruments: "acoustic guitar", "synth pads", "strings"
4. Tempo/Energy: "slow", "upbeat", "driving"
5. Context: "for a travel vlog", "podcast intro"
## Rate Limits by Tier
| Tier | TTS Concurrent | SFX Concurrent | Music Concurrent |
|------|---------------|----------------|------------------|
| Free | 2 | 2 | 1 |
| Starter | 3 | 3 | 2 |
| Creator | 5 | 5 | 3 |
| Pro | 10 | 10 | 5 |
| Scale | 15 | 15 | 10 |
## Voice Management
```python
# List voices
voices = client.voices.get_all()
for voice in voices.voices:
print(f"{voice.name}: {voice.voice_id}")
# Delete voice
client.voices.delete(voice_id="your_voice_id")
```
## Error Handling
```python
from elevenlabs.core.api_error import ApiError
try:
audio = client.text_to_speech.convert(...)
except ApiError as e:
if e.status_code == 429:
print("Rate limited - wait and retry")
elif e.status_code == 401:
print("Invalid API key")
```
| Code | Meaning | Action |
|------|---------|--------|
| 401 | Invalid API key | Check API key |
| 403 | Feature not available | Upgrade tier |
| 422 | Invalid parameters | Check request body |
| 429 | Rate limited | Wait and retry |
.claude/skills/faceswap/SKILL.md
+261
View File
@@ -0,0 +1,261 @@
---
name: faceswap
description: |
Swap faces in a video using AI via the HeyGen API. Use when: (1) Replacing a face in a video with another face, (2) Face swapping from a source image onto a target video, (3) Creating personalized videos by swapping in a person's face, (4) Working with HeyGen's /v1/workflows/executions endpoint for face swap processing.
allowed-tools: mcp__heygen__*
metadata:
openclaw:
requires:
env:
- HEYGEN_API_KEY
primaryEnv: HEYGEN_API_KEY
---
# Face Swap (HeyGen API)
Swap a face from a source image into a target video using GPU-accelerated AI processing. The source image provides the face to swap in, and the target video receives the new face.
## Authentication
All requests require the `X-Api-Key` header. Set the `HEYGEN_API_KEY` environment variable.
```bash
curl -X POST "https://api.heygen.com/v1/workflows/executions" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{"workflow_type": "FaceswapNode", "input": {"source_image_url": "https://example.com/face.jpg", "target_video_url": "https://example.com/video.mp4"}}'
```
## Default Workflow
1. Call `POST /v1/workflows/executions` with `workflow_type: "FaceswapNode"`, a source face image, and a target video
2. Receive a `execution_id` in the response
3. Poll `GET /v1/workflows/executions/{id}` every 10 seconds until status is `completed`
4. Use the returned `video_url` from the output
## Execute Face Swap
### Endpoint
`POST https://api.heygen.com/v1/workflows/executions`
### Request Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `workflow_type` | string | Y | Must be `"FaceswapNode"` |
| `input.source_image_url` | string | Y | URL of the face image to swap in |
| `input.target_video_url` | string | Y | URL of the video to apply the face swap to |
### curl
```bash
curl -X POST "https://api.heygen.com/v1/workflows/executions" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"workflow_type": "FaceswapNode",
"input": {
"source_image_url": "https://example.com/face-photo.jpg",
"target_video_url": "https://example.com/original-video.mp4"
}
}'
```
### TypeScript
```typescript
interface FaceswapInput {
source_image_url: string;
target_video_url: string;
}
interface ExecuteResponse {
data: {
execution_id: string;
status: "submitted";
};
}
async function faceswap(input: FaceswapInput): Promise<string> {
const response = await fetch("https://api.heygen.com/v1/workflows/executions", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
workflow_type: "FaceswapNode",
input,
}),
});
const json: ExecuteResponse = await response.json();
return json.data.execution_id;
}
```
### Python
```python
import requests
import os
def faceswap(source_image_url: str, target_video_url: str) -> str:
payload = {
"workflow_type": "FaceswapNode",
"input": {
"source_image_url": source_image_url,
"target_video_url": target_video_url,
},
}
response = requests.post(
"https://api.heygen.com/v1/workflows/executions",
headers={
"X-Api-Key": os.environ["HEYGEN_API_KEY"],
"Content-Type": "application/json",
},
json=payload,
)
data = response.json()
return data["data"]["execution_id"]
```
### Response Format
```json
{
"data": {
"execution_id": "node-gw-f1s2w3p4",
"status": "submitted"
}
}
```
## Check Status
### Endpoint
`GET https://api.heygen.com/v1/workflows/executions/{execution_id}`
### curl
```bash
curl -X GET "https://api.heygen.com/v1/workflows/executions/node-gw-f1s2w3p4" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### Response Format (Completed)
```json
{
"data": {
"execution_id": "node-gw-f1s2w3p4",
"status": "completed",
"output": {
"video_url": "https://resource.heygen.ai/faceswap/output.mp4"
}
}
}
```
## Polling for Completion
```typescript
async function faceswapAndWait(
input: FaceswapInput,
maxWaitMs = 600000,
pollIntervalMs = 10000
): Promise<string> {
const executionId = await faceswap(input);
console.log(`Submitted face swap: ${executionId}`);
const startTime = Date.now();
while (Date.now() - startTime < maxWaitMs) {
const response = await fetch(
`https://api.heygen.com/v1/workflows/executions/${executionId}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
switch (data.status) {
case "completed":
return data.output.video_url;
case "failed":
throw new Error(data.error?.message || "Face swap failed");
case "not_found":
throw new Error("Workflow not found");
default:
await new Promise((r) => setTimeout(r, pollIntervalMs));
}
}
throw new Error("Face swap timed out");
}
```
## Usage Examples
### Basic Face Swap
```bash
curl -X POST "https://api.heygen.com/v1/workflows/executions" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"workflow_type": "FaceswapNode",
"input": {
"source_image_url": "https://example.com/headshot.jpg",
"target_video_url": "https://example.com/presentation.mp4"
}
}'
```
### Chain with Avatar Video
Generate an avatar video first, then swap in a custom face:
```python
import time
# Step 1: Generate avatar video
avatar_execution_id = requests.post(
"https://api.heygen.com/v1/workflows/executions",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"], "Content-Type": "application/json"},
json={
"workflow_type": "AvatarInferenceNode",
"input": {
"avatar": {"avatar_id": "Angela-inblackskirt-20220820"},
"audio_list": [{"audio_url": "https://example.com/speech.mp3"}],
},
},
).json()["data"]["execution_id"]
# Step 2: Wait for avatar video to complete
while True:
status = requests.get(
f"https://api.heygen.com/v1/workflows/executions/{avatar_execution_id}",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]},
).json()["data"]
if status["status"] == "completed":
avatar_video_url = status["output"]["video"]["video_url"]
break
time.sleep(10)
# Step 3: Swap in a custom face
faceswap_execution_id = faceswap(
source_image_url="https://example.com/custom-face.jpg",
target_video_url=avatar_video_url,
)
```
## Best Practices
1. **Use a clear, front-facing face photo** — the source image should show a single face with good lighting
2. **Face swap is GPU-intensive** — expect 1-3 minutes processing time, poll every 10 seconds
3. **Source image quality matters** — higher resolution face photos produce better results
4. **One face per source image** — the source should contain exactly one face to swap in
5. **Works with any video** — the target video can be an avatar video, a recording, or any video with visible faces
6. **Chain with other workflows** — generate an avatar video first, then swap in a custom face for personalization
.claude/skills/ffmpeg/SKILL.md
+432
View File
@@ -0,0 +1,432 @@
---
name: ffmpeg
description: Video and audio processing with FFmpeg. Use for format conversion, resizing, compression, audio extraction, and preparing assets for Remotion. Triggers include converting GIF to MP4, resizing video, extracting audio, compressing files, or any media transformation task.
---
# FFmpeg for Video Production
FFmpeg is the essential tool for video/audio processing. This skill covers common operations for Remotion video projects.
## Quick Reference
### GIF to MP4 (Remotion-compatible)
```bash
ffmpeg -i input.gif -movflags faststart -pix_fmt yuv420p \
-vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" output.mp4
```
**Why these flags:**
- `-movflags faststart` - Moves metadata to start for web streaming
- `-pix_fmt yuv420p` - Ensures compatibility with most players
- `scale=trunc(...)` - Forces even dimensions (required by most codecs)
### Resize Video
```bash
# To 1920x1080 (maintain aspect ratio, add black bars)
ffmpeg -i input.mp4 -vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" output.mp4
# To 1920x1080 (crop to fill)
ffmpeg -i input.mp4 -vf "scale=1920:1080:force_original_aspect_ratio=increase,crop=1920:1080" output.mp4
# Scale to width, auto height
ffmpeg -i input.mp4 -vf "scale=1280:-2" output.mp4
```
### Compress Video
```bash
# Good quality, smaller file (CRF 23 is default, lower = better quality)
ffmpeg -i input.mp4 -c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k output.mp4
# Aggressive compression for web preview
ffmpeg -i input.mp4 -c:v libx264 -crf 28 -preset fast -c:a aac -b:a 96k output.mp4
# Target file size (e.g., ~10MB for 60s video = ~1.3Mbps)
ffmpeg -i input.mp4 -c:v libx264 -b:v 1300k -c:a aac -b:a 128k output.mp4
```
### Extract Audio
```bash
# Extract to MP3
ffmpeg -i input.mp4 -vn -acodec libmp3lame -q:a 2 output.mp3
# Extract to AAC
ffmpeg -i input.mp4 -vn -acodec aac -b:a 192k output.m4a
# Extract to WAV (uncompressed)
ffmpeg -i input.mp4 -vn output.wav
```
### Convert Audio Formats
```bash
# M4A to MP3 (for ElevenLabs voice samples)
ffmpeg -i input.m4a -codec:a libmp3lame -qscale:a 2 output.mp3
# WAV to MP3
ffmpeg -i input.wav -codec:a libmp3lame -b:a 192k output.mp3
# Adjust volume
ffmpeg -i input.mp3 -filter:a "volume=1.5" output.mp3
```
### Trim/Cut Video
```bash
# Cut from timestamp to duration (recommended - reliable)
ffmpeg -i input.mp4 -ss 00:00:30 -t 00:00:15 -c:v libx264 -c:a aac output.mp4
# Cut from timestamp to timestamp
ffmpeg -i input.mp4 -ss 00:00:30 -to 00:00:45 -c:v libx264 -c:a aac output.mp4
# Stream copy (faster but may lose frames at cut points)
# Only use when source has frequent keyframes
ffmpeg -i input.mp4 -ss 00:00:30 -t 00:00:15 -c copy output.mp4
```
**Note:** Re-encoding is recommended for trimming. Stream copy (`-c copy`) can silently drop video if the seek point doesn't align with a keyframe.
### Speed Up / Slow Down
```bash
# 2x speed (video and audio)
ffmpeg -i input.mp4 -filter_complex "[0:v]setpts=0.5*PTS[v];[0:a]atempo=2.0[a]" -map "[v]" -map "[a]" output.mp4
# 0.5x speed (slow motion)
ffmpeg -i input.mp4 -filter_complex "[0:v]setpts=2.0*PTS[v];[0:a]atempo=0.5[a]" -map "[v]" -map "[a]" output.mp4
# Video only (no audio)
ffmpeg -i input.mp4 -filter:v "setpts=0.5*PTS" -an output.mp4
```
### Concatenate Videos
```bash
# Create file list
echo "file 'clip1.mp4'" > list.txt
echo "file 'clip2.mp4'" >> list.txt
echo "file 'clip3.mp4'" >> list.txt
# Concatenate (same codec/resolution)
ffmpeg -f concat -safe 0 -i list.txt -c copy output.mp4
# Concatenate with re-encoding (different sources)
ffmpeg -f concat -safe 0 -i list.txt -c:v libx264 -c:a aac output.mp4
```
### Add Fade In/Out
```bash
# Fade in first 1 second, fade out last 1 second (30fps video)
ffmpeg -i input.mp4 -vf "fade=t=in:st=0:d=1,fade=t=out:st=9:d=1" -c:a copy output.mp4
# Audio fade
ffmpeg -i input.mp4 -af "afade=t=in:st=0:d=1,afade=t=out:st=9:d=1" -c:v copy output.mp4
```
### Get Video Info
```bash
# Duration, resolution, codec info
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 input.mp4
# Full info
ffprobe -v quiet -print_format json -show_format -show_streams input.mp4
```
## Remotion-Specific Patterns
### Video Speed Adjustment for Remotion
**When to use FFmpeg vs Remotion `playbackRate`:**
| Scenario | Use FFmpeg | Use Remotion |
|----------|------------|--------------|
| Constant speed (1.5x, 2x) | Either works | ✅ Simpler |
| Extreme speeds (>4x or <0.25x) | ✅ More reliable | May have issues |
| Variable speed (accelerate over time) | ✅ Pre-process | Complex workaround needed |
| Need perfect audio sync | ✅ Guaranteed | Usually fine |
| Demo needs to fit voiceover timing | ✅ Pre-calculate | Runtime adjustment |
**Remotion limitation:** `playbackRate` must be constant. Dynamic interpolation like `playbackRate={interpolate(frame, [0, 100], [1, 5])}` won't work correctly because Remotion evaluates frames independently.
```bash
# Speed up demo to fit a scene (e.g., 60s demo into 20s = 3x speed)
ffmpeg -i demo-raw.mp4 \
-filter_complex "[0:v]setpts=0.333*PTS[v];[0:a]atempo=3.0[a]" \
-map "[v]" -map "[a]" \
public/demos/demo-fast.mp4
# Slow motion for emphasis (0.5x speed)
ffmpeg -i action.mp4 \
-filter_complex "[0:v]setpts=2.0*PTS[v];[0:a]atempo=0.5[a]" \
-map "[v]" -map "[a]" \
public/demos/action-slow.mp4
# Speed up without audio (common for screen recordings)
ffmpeg -i demo.mp4 -filter:v "setpts=0.5*PTS" -an public/demos/demo-2x.mp4
# Timelapse effect (10x speed, drop audio)
ffmpeg -i long-demo.mp4 -filter:v "setpts=0.1*PTS" -an public/demos/timelapse.mp4
```
**Calculate speed factor:**
- To fit X seconds of video into Y seconds of scene: `speed = X / Y`
- setpts multiplier = `1 / speed` (e.g., 3x speed = setpts=0.333*PTS)
- atempo value = `speed` (e.g., 3x speed = atempo=3.0)
**Extreme speed (>2x audio):** Chain atempo filters (each limited to 0.5-2.0 range):
```bash
# 4x speed audio
-filter_complex "[0:a]atempo=2.0,atempo=2.0[a]"
# 8x speed audio
-filter_complex "[0:a]atempo=2.0,atempo=2.0,atempo=2.0[a]"
```
### Prepare Demo Recording for Remotion
```bash
# Standard 1080p, 30fps, Remotion-ready
ffmpeg -i raw-recording.mp4 \
-vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2,fps=30" \
-c:v libx264 -crf 18 -preset slow \
-c:a aac -b:a 192k \
-movflags faststart \
public/demos/demo.mp4
```
### Screen Recording to Remotion Asset
```bash
# From iPhone/iPad recording (usually 60fps, variable resolution)
ffmpeg -i iphone-recording.mov \
-vf "scale=1920:-2,fps=30" \
-c:v libx264 -crf 20 \
-an \
public/demos/mobile-demo.mp4
```
### Batch Convert GIFs
```bash
for f in assets/*.gif; do
ffmpeg -i "$f" -movflags faststart -pix_fmt yuv420p \
-vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" \
"public/demos/$(basename "$f" .gif).mp4"
done
```
## Common Issues
### "Height not divisible by 2"
Add scale filter: `-vf "scale=trunc(iw/2)*2:trunc(ih/2)*2"`
### Video won't play in browser
Use: `-movflags faststart -pix_fmt yuv420p -c:v libx264`
### Audio out of sync after speed change
Use filter_complex with atempo: `-filter_complex "[0:v]setpts=0.5*PTS[v];[0:a]atempo=2.0[a]"`
### File too large
Increase CRF (23→28) or reduce resolution
## Quality Guidelines
| Use Case | CRF | Preset | Notes |
|----------|-----|--------|-------|
| Archive/Master | 18 | slow | Best quality, large files |
| Production | 20-22 | medium | Good balance |
| Web/Preview | 23-25 | fast | Smaller files |
| Draft/Quick | 28+ | veryfast | Fast encoding |
## Platform-Specific Output Optimization
After Remotion renders your video (typically to `out/video.mp4`), use FFmpeg to optimize for each distribution platform.
### Workflow Integration
```
Remotion render (master) FFmpeg optimization Platform upload
↓ ↓ ↓
out/video.mp4 ────────→ out/video-youtube.mp4 ───→ YouTube
────────→ out/video-twitter.mp4 ───→ Twitter/X
────────→ out/video-linkedin.mp4 ───→ LinkedIn
────────→ out/video-web.mp4 ───→ Website embed
```
### YouTube (Recommended Settings)
YouTube re-encodes everything, so upload high quality:
```bash
# YouTube optimized (1080p)
ffmpeg -i out/video.mp4 \
-c:v libx264 -preset slow -crf 18 \
-profile:v high -level 4.0 \
-bf 2 -g 30 \
-c:a aac -b:a 192k -ar 48000 \
-movflags +faststart \
out/video-youtube.mp4
# YouTube Shorts (vertical 1080x1920)
ffmpeg -i out/video.mp4 \
-vf "scale=1080:1920:force_original_aspect_ratio=decrease,pad=1080:1920:(ow-iw)/2:(oh-ih)/2" \
-c:v libx264 -crf 18 -c:a aac -b:a 192k \
out/video-shorts.mp4
```
### Twitter/X
Twitter has strict limits: max 140s, 512MB, 1920x1200:
```bash
# Twitter optimized (under 15MB target for fast upload)
ffmpeg -i out/video.mp4 \
-c:v libx264 -preset medium -crf 24 \
-profile:v main -level 3.1 \
-vf "scale='min(1280,iw)':'min(720,ih)':force_original_aspect_ratio=decrease" \
-c:a aac -b:a 128k -ar 44100 \
-movflags +faststart \
-fs 15M \
out/video-twitter.mp4
# Check file size and duration
ffprobe -v error -show_entries format=duration,size -of csv=p=0 out/video-twitter.mp4
```
### LinkedIn
LinkedIn prefers MP4 with AAC audio, max 10 minutes:
```bash
# LinkedIn optimized
ffmpeg -i out/video.mp4 \
-c:v libx264 -preset medium -crf 22 \
-profile:v main \
-vf "scale='min(1920,iw)':'min(1080,ih)':force_original_aspect_ratio=decrease" \
-c:a aac -b:a 192k -ar 48000 \
-movflags +faststart \
out/video-linkedin.mp4
```
### Website/Embed (Optimized for Fast Loading)
```bash
# Web-optimized MP4 (small file, progressive loading)
ffmpeg -i out/video.mp4 \
-c:v libx264 -preset medium -crf 26 \
-profile:v baseline -level 3.0 \
-vf "scale=1280:720" \
-c:a aac -b:a 128k \
-movflags +faststart \
out/video-web.mp4
# WebM alternative (better compression, wider browser support)
ffmpeg -i out/video.mp4 \
-c:v libvpx-vp9 -crf 30 -b:v 0 \
-vf "scale=1280:720" \
-c:a libopus -b:a 128k \
-deadline good \
out/video-web.webm
```
### GIF (for Previews/Thumbnails)
```bash
# High-quality GIF (first 5 seconds)
ffmpeg -i out/video.mp4 -t 5 \
-vf "fps=15,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" \
out/preview.gif
# Smaller file GIF
ffmpeg -i out/video.mp4 -t 3 \
-vf "fps=10,scale=320:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" \
out/preview-small.gif
```
### Platform Requirements Quick Reference
| Platform | Max Resolution | Max Size | Max Duration | Audio |
|----------|---------------|----------|--------------|-------|
| YouTube | 8K | 256GB | 12 hours | AAC 48kHz |
| Twitter/X | 1920x1200 | 512MB | 140s | AAC 44.1kHz |
| LinkedIn | 4096x2304 | 5GB | 10 min | AAC 48kHz |
| Instagram Feed | 1080x1350 | 4GB | 60s | AAC 48kHz |
| Instagram Reels | 1080x1920 | 4GB | 90s | AAC 48kHz |
| TikTok | 1080x1920 | 287MB | 10 min | AAC |
### Batch Export for All Platforms
```bash
#!/bin/bash
# save as: export-all-platforms.sh
INPUT="out/video.mp4"
# YouTube (high quality)
ffmpeg -i "$INPUT" -c:v libx264 -preset slow -crf 18 \
-c:a aac -b:a 192k -movflags +faststart \
out/video-youtube.mp4
# Twitter (compressed)
ffmpeg -i "$INPUT" -c:v libx264 -crf 24 \
-vf "scale='min(1280,iw)':'-2'" \
-c:a aac -b:a 128k -movflags +faststart \
out/video-twitter.mp4
# LinkedIn
ffmpeg -i "$INPUT" -c:v libx264 -crf 22 \
-c:a aac -b:a 192k -movflags +faststart \
out/video-linkedin.mp4
# Web embed (small)
ffmpeg -i "$INPUT" -c:v libx264 -crf 26 \
-vf "scale=1280:720" \
-c:a aac -b:a 128k -movflags +faststart \
out/video-web.mp4
echo "Exported:"
ls -lh out/video-*.mp4
```
## Error Handling
Common errors and fixes when processing video:
```bash
# Check if FFmpeg succeeded
ffmpeg -i input.mp4 -c:v libx264 output.mp4 && echo "Success" || echo "Failed: check input file"
# Validate output file is playable
ffprobe -v error -select_streams v:0 -show_entries stream=codec_name -of csv=p=0 output.mp4
# Get detailed error info
ffmpeg -v error -i input.mp4 -f null - 2>&1 | head -20
```
### Handling Common Failures
| Error | Cause | Fix |
|-------|-------|-----|
| "No such file" | Input path wrong | Check path, use quotes for spaces |
| "Invalid data" | Corrupted input | Re-download or re-record source |
| "height not divisible by 2" | Odd dimensions | Add scale filter with trunc |
| "encoder not found" | Missing codec | Install FFmpeg with full codecs |
| Output 0 bytes | Silent failure | Check full ffmpeg output for errors |
---
## Feedback & Contributions
If this skill is missing information or could be improved:
- **Missing a command?** Describe what you needed
- **Found an error?** Let me know what's wrong
- **Want to contribute?** I can help you:
1. Update this skill with improvements
2. Create a PR to github.com/digitalsamba/claude-code-video-toolkit
Just say "improve this skill" and I'll guide you through updating `.claude/skills/ffmpeg/SKILL.md`.
.claude/skills/ffmpeg/reference.md
+173
View File
@@ -0,0 +1,173 @@
# FFmpeg Reference
## Filter Syntax
### Video Filters (-vf)
```bash
# Chain filters with comma
-vf "scale=1920:1080,fps=30,crop=1280:720"
# Complex filters with labels
-filter_complex "[0:v]scale=1920:1080[scaled];[scaled]fps=30[out]" -map "[out]"
```
### Common Video Filters
| Filter | Syntax | Example |
|--------|--------|---------|
| scale | `scale=w:h` | `scale=1920:1080` or `scale=1280:-1` (auto height) |
| crop | `crop=w:h:x:y` | `crop=1280:720:320:180` |
| fps | `fps=N` | `fps=30` |
| pad | `pad=w:h:x:y:color` | `pad=1920:1080:(ow-iw)/2:(oh-ih)/2:black` |
| fade | `fade=t=in/out:st=N:d=N` | `fade=t=in:st=0:d=1` |
| setpts | `setpts=N*PTS` | `setpts=0.5*PTS` (2x speed) |
| drawtext | `drawtext=text='Hi':fontsize=24` | Add text overlay |
| overlay | `overlay=x:y` | Combine videos |
### Common Audio Filters (-af)
| Filter | Syntax | Example |
|--------|--------|---------|
| volume | `volume=N` | `volume=1.5` or `volume=0.5` |
| afade | `afade=t=in/out:st=N:d=N` | `afade=t=in:st=0:d=1` |
| atempo | `atempo=N` | `atempo=2.0` (2x speed, range 0.5-2.0) |
| loudnorm | `loudnorm` | Normalize audio levels |
## Codec Options
### Video Codecs (-c:v)
| Codec | Use Case | Notes |
|-------|----------|-------|
| libx264 | Universal H.264 | Best compatibility |
| libx265 | H.265/HEVC | Better compression, less compatible |
| libvpx-vp9 | WebM | Good for web |
| prores | ProRes | Professional editing |
| copy | Stream copy | No re-encoding, fastest |
### Audio Codecs (-c:a)
| Codec | Use Case | Notes |
|-------|----------|-------|
| aac | MP4 container | Most compatible |
| libmp3lame | MP3 | Universal |
| libvorbis | WebM/OGG | Open source |
| pcm_s16le | WAV | Uncompressed |
| copy | Stream copy | No re-encoding |
## Quality Settings
### CRF (Constant Rate Factor) for x264/x265
| CRF | Quality | Use Case |
|-----|---------|----------|
| 0 | Lossless | Archive |
| 17-18 | Visually lossless | Master |
| 19-22 | High quality | Production |
| 23 | Default | General use |
| 24-27 | Medium | Web delivery |
| 28+ | Low | Preview/draft |
### Presets (-preset)
Faster presets = larger files, quicker encoding
`ultrafast` → `superfast` → `veryfast` → `faster` → `fast` → `medium` → `slow` → `slower` → `veryslow`
## Container Formats
| Format | Extension | Best For |
|--------|-----------|----------|
| MP4 | .mp4 | Universal, web, mobile |
| MOV | .mov | Apple ecosystem, ProRes |
| WebM | .webm | Web (VP9) |
| MKV | .mkv | Archive, multiple streams |
| GIF | .gif | Short animations (no audio) |
## Input/Output Options
### Input Options (before -i)
| Option | Purpose | Example |
|--------|---------|---------|
| -ss | Seek to time | `-ss 00:01:30` |
| -t | Duration limit | `-t 00:00:30` |
| -r | Input framerate | `-r 30` |
| -f | Force format | `-f gif` |
### Output Options (after -i)
| Option | Purpose | Example |
|--------|---------|---------|
| -y | Overwrite output | `-y` |
| -n | Never overwrite | `-n` |
| -movflags faststart | Web streaming | `-movflags faststart` |
| -pix_fmt | Pixel format | `-pix_fmt yuv420p` |
| -an | No audio | `-an` |
| -vn | No video | `-vn` |
## Useful Patterns
### Get Duration in Seconds
```bash
ffprobe -v error -show_entries format=duration -of csv=p=0 input.mp4
```
### Get Resolution
```bash
ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of csv=p=0 input.mp4
```
### Get Frame Count
```bash
ffprobe -v error -select_streams v:0 -count_frames -show_entries stream=nb_read_frames -of csv=p=0 input.mp4
```
### Create Thumbnail
```bash
# At specific time
ffmpeg -i input.mp4 -ss 00:00:05 -vframes 1 thumbnail.jpg
# Best quality
ffmpeg -i input.mp4 -ss 00:00:05 -vframes 1 -q:v 2 thumbnail.jpg
```
### Create GIF from Video
```bash
# Simple (large file)
ffmpeg -i input.mp4 -vf "fps=10,scale=480:-1" output.gif
# With palette (better quality, smaller)
ffmpeg -i input.mp4 -vf "fps=10,scale=480:-1,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" output.gif
```
### Picture-in-Picture
```bash
# Overlay small video in corner
ffmpeg -i main.mp4 -i overlay.mp4 \
-filter_complex "[1:v]scale=320:-1[pip];[0:v][pip]overlay=W-w-20:H-h-20" \
-c:a copy output.mp4
```
### Side-by-Side Videos
```bash
ffmpeg -i left.mp4 -i right.mp4 \
-filter_complex "[0:v][1:v]hstack=inputs=2[v]" \
-map "[v]" -c:v libx264 output.mp4
```
## Remotion Integration Notes
- Remotion uses `<OffthreadVideo>` which handles most formats
- Prefer H.264 (libx264) in MP4 container
- Always use `-movflags faststart` for web playback
- Match fps to composition (usually 30fps)
- Resolution should match composition (1920x1080 typical)
.claude/skills/flux-best-practices/AGENTS.md
+2246
View File
File diff suppressed because it is too large Load Diff
.claude/skills/flux-best-practices/SKILL.md
+82
View File
@@ -0,0 +1,82 @@
---
name: flux-best-practices
description: Comprehensive guide for BFL FLUX image generation models. Covers prompting, T2I, I2I, structured JSON, hex colors, typography, multi-reference editing, and model-specific best practices for FLUX.2 and FLUX.1 families.
metadata:
author: Black Forest Labs
version: "1.0.0"
tags: flux, bfl, image-generation, prompting, t2i, i2i
---
# FLUX Best Practices
Use this skill when generating prompts for any BFL FLUX model to ensure optimal image quality and accurate prompt interpretation.
## When to Use
- Creating prompts for FLUX.2 or FLUX.1 models
- Text-to-image (T2I) generation
- Image-to-image (I2I) editing with FLUX.2 models
- Structured scene generation with JSON
- Typography and text rendering
- Multi-reference style transfer
- Color-accurate brand generations
## Quick Reference
### Prompt Structure Formula
```
[Subject] + [Action/Pose] + [Style/Medium] + [Context/Setting] + [Lighting] + [Camera/Technical]
```
### Model Selection
| Use Case | Recommended Model | Notes |
| ------------------- | ----------------- | -------------------------------------- |
| Fastest generation | FLUX.2 [klein] | 4B or 9B, sub-second |
| Highest quality | FLUX.2 [max] | Best detail, grounding search |
| Production balanced | FLUX.2 [pro] | Quality + speed |
| Typography/text | FLUX.2 [flex] | Best text rendering |
| Local/development | FLUX.2 [dev] | Open weights |
| Image editing | FLUX.2 [pro/max] | Pass image URL directly to input_image |
| Inpainting | FLUX.1 Fill | Object removal/completion |
| Context editing | FLUX.1 Kontext | Older model, prefer FLUX.2 |
### Critical Rules
1. **NO negative prompts** - FLUX does not support negative prompts; describe what you want
2. **Be specific** - Vague prompts produce mediocre results
3. **Use natural language** - Prose/narrative style works best
4. **Specify lighting** - Lighting has the biggest impact on quality
5. **Quote text** - Use "quoted text" for typography rendering
6. **Hex colors** - Use #RRGGBB format with color description
## Related
For API integration (endpoints, polling, webhooks), see the **bfl-api** skill.
## Rules Reference
Read individual rule files for detailed guidance:
- [rules/core-principles.md](rules/core-principles.md) - Universal FLUX prompting principles
- [rules/flux2-models.md](rules/flux2-models.md) - FLUX.2 family: klein, max, pro, flex, dev
- [rules/flux1-models.md](rules/flux1-models.md) - FLUX.1 family: older generation of FLUX.2 models - pro, Kontext, Fill
- [rules/t2i-prompting.md](rules/t2i-prompting.md) - Text-to-image prompting patterns
- [rules/i2i-prompting.md](rules/i2i-prompting.md) - Image-to-image editing with FLUX.2
- [rules/json-structured-prompting.md](rules/json-structured-prompting.md) - Complex scene composition
- [rules/hex-color-prompting.md](rules/hex-color-prompting.md) - Precise color specification
- [rules/typography-text.md](rules/typography-text.md) - Text rendering and typography
- [rules/multi-reference-editing.md](rules/multi-reference-editing.md) - Multi-image references
- [rules/negative-prompt-alternatives.md](rules/negative-prompt-alternatives.md) - Positive alternatives
- [rules/model-selection-guide.md](rules/model-selection-guide.md) - Choosing the right model
## Example Prompt
```
A weathered fisherman in his 70s with deep wrinkles and a salt-and-pepper beard,
wearing a navy cable-knit sweater, standing at the helm of his wooden boat.
Golden hour sunlight from the left creates dramatic rim lighting on his profile.
Shot on Hasselblad with 85mm lens at f/2.8, shallow depth of field with harbor
lights creating soft bokeh in the background. Kodak Portra 400 color science.
```
.claude/skills/flux-best-practices/rules/_sections.md
+51
View File
@@ -0,0 +1,51 @@
# Sections
This file defines all sections, their ordering, impact levels, and descriptions.
The section ID (in parentheses) is the filename prefix used to group rules.
---
## 1. Core Principles (core)
**Impact:** CRITICAL
**Description:** Universal prompting principles that apply to all FLUX models. Master these before diving into specific techniques.
## 2. Model Selection (model, flux2, flux1)
**Impact:** HIGH
**Description:** Choosing the right FLUX model for your use case. Covers both FLUX.2 (latest) and FLUX.1 (legacy) model families.
## 3. Text-to-Image Prompting (t2i)
**Impact:** HIGH
**Description:** Crafting effective prompts for generating images from text descriptions.
## 4. Image-to-Image Editing (i2i)
**Impact:** HIGH
**Description:** Techniques for editing and transforming existing images using FLUX.2 models.
## 5. JSON Structured Prompting (json)
**Impact:** MEDIUM-HIGH
**Description:** Using structured JSON for complex multi-element scene composition.
## 6. Color Specification (hex)
**Impact:** MEDIUM
**Description:** Precise color control using hex codes for brand-accurate generations.
## 7. Typography and Text (typography)
**Impact:** MEDIUM
**Description:** Rendering text and typography within generated images.
## 8. Multi-Reference Editing (multi)
**Impact:** MEDIUM
**Description:** Combining multiple reference images for style transfer and composition.
## 9. Positive Prompt Alternatives (negative)
**Impact:** MEDIUM
**Description:** Strategies for achieving results without negative prompts, which FLUX does not support.
.claude/skills/flux-best-practices/rules/core-principles.md
+138
View File
@@ -0,0 +1,138 @@
---
name: core-principles
description: Universal prompting principles that apply to all FLUX models
---
# Core FLUX Prompting Principles
These principles apply to all FLUX models and form the foundation of effective prompting.
## 1. Positive Descriptions Only
FLUX does NOT support negative prompts. Always describe what you WANT, not what you don't want.
### Wrong Approach
```
a portrait of a woman, no glasses, no hat, no makeup
```
### Correct Approach
```
a portrait of a woman with natural skin, clear face, bare head, visible eyes
```
See [negative-prompt-alternatives.md](negative-prompt-alternatives.md) for comprehensive replacement strategies.
## 2. Prompt Structure Formula
Build prompts using this structure for consistent results:
```
[Subject] + [Action/Pose] + [Style/Medium] + [Context/Setting] + [Lighting] + [Technical Details]
```
### Example
```
A young woman with flowing auburn hair (subject)
dancing gracefully in mid-leap (action)
in the style of classical oil painting (style)
in a moonlit garden with roses (context)
soft diffused moonlight with subtle rim lighting (lighting)
medium shot, shallow depth of field (technical)
```
## 3. Specificity Matters
More specific prompts yield dramatically better results.
### Vague (Poor Results)
```
a cat sitting
```
### Specific (Excellent Results)
```
A fluffy orange tabby cat with bright green eyes sitting regally on a vintage
velvet armchair, afternoon sunlight streaming through lace curtains, warm
golden hour lighting, shallow depth of field, shot on medium format film
```
## 4. Natural Language Works Best
Write prompts as descriptive prose rather than keyword lists.
### Keyword Style (Less Effective)
```
woman, portrait, beautiful, blonde, studio, professional, 8k, detailed
```
### Prose Style (More Effective)
```
A professional studio portrait of a beautiful blonde woman in her thirties,
captured with soft studio lighting that accentuates her features, rendered
in stunning detail with natural skin texture and subtle catchlights in her eyes
```
## 5. Lighting is Critical
Always specify lighting - it has the single greatest impact on image quality.
### Natural Lighting
- Golden hour - warm, soft, directional
- Overcast - soft, diffused, even
- Harsh midday - high contrast, strong shadows
- Dappled forest light - specular, organic patterns
### Studio Lighting
- Softbox - even, professional
- Rim light - edge definition, separation
- Butterfly lighting - beauty, glamour
- Rembrandt lighting - dramatic, classic portraits
### Atmospheric Lighting
- Volumetric fog - depth, mystery
- God rays - dramatic, spiritual
- Neon glow - urban, cyberpunk
- Candlelight - warm, intimate
### Mood-Based Lighting
- Dramatic shadows - tension, noir
- High key - bright, airy, clean
- Low key - moody, mysterious
- Chiaroscuro - strong contrast, painterly
## 6. Word Order Matters
FLUX prioritizes elements that appear earlier in the prompt. Front-load important elements.
### Less Effective
```
A forest background with soft lighting where a knight in shining armor stands
```
### More Effective
```
A knight in shining armor stands in a forest, soft dappled lighting filtering
through the canopy
```
## 7. Medium Prompt Length
Optimal prompt length is typically 30-80 words (FLUX can handle up to 512 tokens).
- Too short: Lacks direction, generic results
- Too long: Can become unfocused
- Sweet spot: Enough detail to guide, not so much it confuses
## 8. Iterative Refinement
Build prompts iteratively:
1. Start with core subject and action
2. Add style and medium
3. Specify lighting and atmosphere
4. Include technical details
5. Refine based on results
Change one element at a time to understand what affects your output.
.claude/skills/flux-best-practices/rules/flux1-models.md
+176
View File
@@ -0,0 +1,176 @@
---
name: flux1-models
description: Prompting guidelines for FLUX.1 model family
---
# FLUX.1 Model Family
> **Tip:** FLUX.2 models are the latest generation and recommended for most use cases. FLUX.1 models are still available for specific needs.
Guide to FLUX.1 models and their specialized capabilities.
## Model Overview
| Model | Purpose | Notes |
|-------|---------|-------|
| FLUX1.1 [pro] | Text-to-image | FLUX.2 [pro] offers improved results |
| FLUX.1 Kontext | Image-to-image | FLUX.2 with references recommended |
| FLUX.1 Kontext Max | Image-to-image | FLUX.2 [max] with references recommended |
| FLUX.1 Fill | Inpainting | Useful for specific inpainting tasks |
## FLUX1.1 [pro]
Fast and reliable text-to-image generation.
### Characteristics
- Strong prompt adherence
- Production-grade architecture
- Consistent, reliable results
- Scalable for high-volume
- Pricing: $0.04 per image
### Prompting Style
Standard descriptive prompts with clear subject and style specification.
### Example Prompt
```
A golden retriever puppy playing in autumn leaves, warm afternoon sunlight,
shallow depth of field with bokeh background, joyful expression, professional
pet photography style
```
## FLUX.1 Kontext - Image Editing
> **Recommendation:** FLUX.2 models with reference images provide improved editing results.
Context-aware image-to-image editing model for transformations and modifications.
### Characteristics
- Understands image context
- Preserves unedited regions
- Style transfer capabilities
- Object modification
- Basic to complex transformations
### Prompting Strategies
#### Basic Edits
Simple, direct instructions work best:
```
Change the color of the car to red
```
```
Make the sky sunset orange and pink
```
#### Controlled Edits
Be explicit about what to preserve:
```
Change the setting to nighttime while maintaining the exact same
composition and the painting's artistic style
```
#### Complex Transformations
For dramatic changes, be specific about preservation:
```
Transform the modern office into a Victorian library with the same
furniture arrangement. Keep the window positions and overall room
proportions identical.
```
#### Style Transfer
Reference specific artistic movements:
```
Transform this photograph into a Bauhaus art style with geometric
shapes and primary colors, maintaining the original composition
and subject positioning
```
#### Text Editing
Describe text placement and integration:
```
Add the text "OPEN" as a neon sign in the window, red glowing letters
with slight reflection on the glass, matching the nighttime atmosphere
```
### Tips for Kontext
- Be explicit about what should NOT change
- Start with simpler edits, build complexity
- Specify style preservation when needed
- Use for incremental refinement
## FLUX.1 Kontext Max
Advanced multi-reference editing for complex compositions.
### Characteristics
- Handles up to 10 reference images
- Best editing consistency across references
- Complex scene composition
- Character consistency maintenance
- Rate limit: 6 concurrent requests
### Multi-Reference Prompting
#### Natural Language References
Describe relationships between images naturally:
```
The person from image 1 is sitting in the cafe from image 2,
wearing the outfit from image 3, with the lighting style of image 4
```
#### Explicit Indexing
Reference images by number for precision:
```
Replace the top half of the person in image 1 with the clothing
from image 2, maintaining the pose and background
```
### Tips for Kontext Max
- Plan your reference images carefully
- Use natural language for relationships
- Specify which elements come from which image
- Leverage for character consistency across scenes
## FLUX.1 Fill - Inpainting
Specialized tool for object removal and area completion.
### Characteristics
- Clean object removal
- Intelligent background completion
- Texture-aware filling
- Seamless blending
### Use Cases
- Remove unwanted objects from photos
- Complete partial images
- Replace specific regions
- Clean up image artifacts
### Prompting for Fill
Describe what should fill the masked area:
```
Fill with continuation of the brick wall texture and ivy
```
```
Complete with matching ocean waves and sandy beach
```
### Tips for Fill
- Provide context about surrounding areas
- Specify texture and pattern continuation
- Describe lighting consistency
- Use for cleanup and removal tasks
.claude/skills/flux-best-practices/rules/flux2-models.md
+202
View File
@@ -0,0 +1,202 @@
---
name: flux2-models
description: Prompting guidelines specific to FLUX.2 model family
---
# FLUX.2 Model Family
Complete guide to FLUX.2 variants and their optimal prompting strategies.
> **Key Feature:** All FLUX.2 models natively support both text-to-image generation AND image-to-image editing via reference images. There's no need to use legacy FLUX.1 Kontext models for editing tasks.
## Model Overview
| Model | Parameters | Best For | Speed | Reference Images |
|-------|------------|----------|-------|------------------|
| [klein] | 4B/9B | Fast iterations, previews, quick edits | Fastest | Up to 4 |
| [max] | - | Highest quality generation & editing | Slowest | Up to 8-10 |
| [pro] | - | Production balanced | Medium | Up to 8 |
| [flex] | - | Typography, text rendering | Medium | Up to 8 |
| [dev] | - | Local development | Varies | Varies |
## FLUX.2 [klein] - Fast Generation
Best for rapid prototyping, previews, and high-volume generation.
### Characteristics
- 4B or 9B parameter versions available
- Sub-second generation times
- Optimized for speed over maximum detail
- **No prompt upsampling** - be descriptive yourself
- Supports up to 4 reference images
### Prompting Style: Narrative Prose
Klein responds best to descriptive, narrative-style prompts with emphasis on lighting and atmosphere.
### Example Prompt
```
A cozy coffee shop interior bathed in warm afternoon light, steam rising lazily
from ceramic cups, worn leather armchairs arranged around small wooden tables,
bookshelves lining exposed brick walls, the soft atmosphere of a quiet afternoon
with dust motes floating in sunbeams through tall windows
```
### Tips for [klein]
- Write like a novelist describing a scene
- Front-load your subject (word order critical)
- Emphasize lighting descriptions heavily
- Keep prompts moderately detailed (40-70 words)
## FLUX.2 [max] - Highest Quality
Premium model for final production assets and maximum detail.
### Characteristics
- Highest detail and coherence
- Best editing consistency
- Vast world knowledge
- Includes grounding search (real-time web data)
- Strongest prompt following
- Supports up to 8 reference images (API), 10 (playground)
### Prompting Style: Technical + Descriptive
[max] excels with detailed technical specifications combined with descriptive prose.
### Example Prompt
```
Portrait of a weathered fisherman, age 70, deep wrinkles telling stories of
decades at sea, salt-and-pepper beard with streaks of white, wearing a navy
cable-knit sweater with visible wool texture. Shot on Hasselblad X2D with
90mm f/2.8 lens at f/4, golden hour natural light from the left creating
strong rim lighting, shallow depth of field with soft bokeh from harbor
lights behind, Kodak Portra 400 color science with natural grain
```
### Tips for [max]
- Include camera and lens specifications for photorealism
- Specify film stock or digital sensor characteristics
- Use technical photography terms (aperture, focal length)
- Leverage grounding search for current events: "news photo of [recent event]"
## FLUX.2 [pro] - Production Balanced
Optimal balance of quality and speed for production workflows.
### Characteristics
- Good quality-to-speed ratio
- Reliable, consistent output
- Suitable for batch processing
- Supports prompt upsampling
- Supports up to 8 reference images
### Prompting Style: Balanced Detail
Standard detailed prompts work well without excessive technical specification.
### Example Prompt
```
A modern minimalist living room with floor-to-ceiling windows overlooking
a city skyline at dusk, clean white furniture with subtle textures, a single
statement plant in the corner, warm ambient lighting from hidden sources,
architectural photography style with clean lines and balanced composition
```
### Tips for [pro]
- Balance specificity with generation speed
- Good for template-based prompt systems
- Enable prompt upsampling for enhanced results
- Consistent quality for production pipelines
## FLUX.2 [flex] - Typography Specialist
Optimized for text rendering and typographic content.
### Characteristics
- Superior text rendering quality
- Handles multiple text elements
- Adjustable steps (1-50) and guidance (1.5-10)
- Best for signage, posters, UI mockups
- Supports up to 8 reference images
### Prompting Style: Typography-Focused
Always quote text and specify font characteristics explicitly.
### Example Prompt
```
A modern minimalist poster design with the headline "DESIGN SUMMIT 2025"
in bold condensed sans-serif typography centered in the upper third,
subtitle "Innovation Meets Creativity" in lighter weight below,
date "MARCH 15-17" in small caps at the bottom, all text in white
on a gradient background transitioning from deep purple #4A0080 to
coral #FF6B6B, clean geometric accent lines, professional print quality
```
### Tips for [flex]
- Always quote exact text: `"Your Text Here"`
- Specify font style: serif, sans-serif, script, display, monospace
- Describe text hierarchy: headline, subhead, body
- Include placement: centered, left-aligned, upper third
- Adjust steps (higher = better quality) and guidance (higher = stricter)
## FLUX.2 [dev] - Local Development
For local development, testing, and non-commercial use.
### Characteristics
- Open weights on Hugging Face
- Runs locally (~13GB VRAM recommended)
- Full customization available
- Free for non-commercial use
- Base variants available (undistilled) for fine-tuning
### Prompting Style: Standard
Same prompting patterns as [pro] work well.
### Tips for [dev]
- Use for development and testing before production
- Experiment with prompt variations
- Good for fine-tuning experiments
- Check license for commercial use restrictions
## Image-to-Image Editing with FLUX.2
All FLUX.2 models support image editing via reference images. This replaces the need for legacy FLUX.1 Kontext models.
### How It Works
1. Provide your source image(s) as reference images
2. Describe the desired changes in your prompt
3. The model preserves context while applying edits
### Example: Style Transfer
```
Reference: [your source image]
Prompt: Transform this image into a watercolor painting style,
maintaining the exact composition and subject positioning
```
### Example: Object Modification
```
Reference: [your source image]
Prompt: Change the car color to red while keeping everything else identical
```
### Example: Character Consistency
```
Reference: [character reference image]
Prompt: The same person from the reference image walking through
a busy Tokyo street at night, neon lights reflecting on wet pavement
```
### Model Selection for Editing
| Use Case | Recommended Model |
|----------|-------------------|
| Quick iterations/previews | FLUX.2 [klein] |
| Production quality edits | FLUX.2 [pro] |
| Maximum quality/complex edits | FLUX.2 [max] |
| Text/typography edits | FLUX.2 [flex] |
.claude/skills/flux-best-practices/rules/hex-color-prompting.md
+210
View File
@@ -0,0 +1,210 @@
---
name: hex-color-prompting
description: Using hex color codes for precise color specification
---
# Hex Color Prompting
FLUX supports hex color codes (#RRGGBB) for precise color specification, essential for brand consistency and exact color matching.
## Syntax
Include hex codes directly in your prompt with descriptive names:
```
A modern living room with walls painted in #2C3E50 (dark blue-gray),
accent pillows in #E74C3C (vibrant red), and a #F39C12 (warm amber)
throw blanket on a #ECF0F1 (off-white) sofa
```
## Signal Keywords
Use these keywords to indicate color specification:
```
color #02eb3c
hex #edfa3c
in #FF5733
using color code #3498DB
```
## Best Practices
### 1. Always Pair with Description
Never use hex codes alone - include the color name:
```
Good: #FF6B6B (coral pink)
Bad: #FF6B6B
```
### 2. Associate with Specific Objects
Clearly connect colors to their targets:
```
A product shot featuring a smartphone with a #1DA1F2 (Twitter blue) case,
resting on a #14171A (near black) matte surface
```
### 3. Limit Color Palette
3-5 colors typically work best. Too many can confuse the model:
```
Color palette for the scene: #2ECC71 (emerald green), #3498DB (sky blue),
#F1C40F (sunflower yellow), #FFFFFF (pure white)
```
## Use Cases
### Brand Colors
```
Corporate office reception with brand colors prominently featured:
walls in #0066CC (company blue), accent furniture in #FF6600 (company orange),
logo displayed in #FFFFFF (white) against the blue backdrop
```
### Interior Design
```
Scandinavian minimalist bedroom with #F5F5F5 (warm white) walls,
#8B4513 (saddle brown) wooden headboard and nightstands,
#708090 (slate gray) linen bedding, and #DAA520 (goldenrod) accent lamp
```
### Fashion
```
Editorial fashion photo: model wearing #000000 (black) cashmere turtleneck,
#FF4500 (orange-red) wide-leg wool pants, #C0C0C0 (silver) geometric earrings,
against a #F0F0F0 (light gray) studio backdrop
```
### Product Design
```
Premium headphones product shot: #1C1C1E (space gray) aluminum body,
#F5F5F7 (silver) mesh ear cups, #FF9500 (iOS orange) accent ring around controls
```
### Digital Art
```
Synthwave cityscape: #FF00FF (magenta) and #00FFFF (cyan) neon signs,
#1A1A2E (deep navy) night sky, #E94560 (hot pink) setting sun on horizon,
#16213E (dark blue) building silhouettes, rain-slicked streets reflecting lights
```
### Data Visualization
```
Infographic showing market share: segments in #2ECC71 (green) for growth,
#E74C3C (red) for decline, #3498DB (blue) for stable, #95A5A6 (gray) for other,
clean #FFFFFF (white) background
```
## Gradient Colors
Specify gradients with start and end colors:
```
Abstract background starting with color #02eb3c (bright green) and
finishing with color #edfa3c (lime yellow), smooth horizontal gradient
```
```
Sunset sky gradient from #FF6B6B (coral) at horizon through
#FFA07A (light salmon) to #87CEEB (sky blue) at top
```
## Color Harmony Patterns
### Complementary (Opposite on color wheel)
```
Scene using complementary colors: #3498DB (blue) dominant with
#E67E22 (orange) accents for visual pop
```
### Analogous (Adjacent colors)
```
Harmonious palette using analogous colors: #9B59B6 (purple),
#8E44AD (deep purple), #3498DB (blue) - flowing naturally together
```
### Triadic (Evenly spaced)
```
Vibrant triadic scheme: #E74C3C (red), #F1C40F (yellow),
#3498DB (blue) - balanced and dynamic
```
### Monochromatic (Single hue variations)
```
Sophisticated monochromatic blue: #1A5276 (dark navy), #2980B9 (medium blue),
#85C1E9 (light blue), #D4E6F1 (pale blue) - elegant depth
```
## Combining with JSON Structured Prompts
```json
{
"scene": {
"setting": "modern tech startup office",
"mood": "innovative, energetic"
},
"colors": {
"primary": "#6C5CE7 (electric purple)",
"secondary": "#00CEC9 (teal)",
"accent": "#FD79A8 (pink)",
"neutral": "#DFE6E9 (light gray)",
"dark": "#2D3436 (charcoal)"
},
"application": {
"walls": "neutral #DFE6E9",
"furniture": "dark #2D3436",
"accent_pieces": "primary #6C5CE7",
"plants": "secondary #00CEC9 pots"
}
}
```
## Common Brand Color References
For reference only - always verify current brand guidelines:
```
# Social Media
Twitter/X Blue: #1DA1F2
Facebook Blue: #1877F2
Instagram Gradient: #833AB4 to #FD1D1D
LinkedIn Blue: #0A66C2
# Tech
Apple Gray: #1C1C1E
Google Blue: #4285F4
Microsoft Blue: #00A4EF
Amazon Orange: #FF9900
# Design
Figma Purple: #A259FF
Dribbble Pink: #EA4C89
Behance Blue: #1769FF
```
## Troubleshooting
### Color Not Accurate
- Add the color name alongside hex
- Specify the exact object the color applies to
- Use fewer total colors in the prompt
### Color Bleeding
- Clearly delineate which objects get which colors
- Use spatial descriptions: "the LEFT chair in #color"
### Muddy Colors
- Check hex code accuracy
- Specify lighting that won't shift colors
- Use "maintaining exact color #XXXXXX" for emphasis
.claude/skills/flux-best-practices/rules/i2i-prompting.md
+293
View File
@@ -0,0 +1,293 @@
---
name: i2i-prompting
description: Image-to-image editing prompts with FLUX models
---
# Image-to-Image (I2I) Prompting
Guide to effective image-to-image editing with FLUX models.
## Overview
All FLUX.2 models support image-to-image editing via reference images:
- **FLUX.2 [klein]**: Up to 4 reference images - fast editing
- **FLUX.2 [pro]**: Up to 8 reference images - balanced quality/speed
- **FLUX.2 [max]**: Up to 8-10 reference images - highest quality editing
- **FLUX.2 [flex]**: Up to 8 reference images - best for typography edits
Simply provide your source image as a reference and describe the desired changes. The model understands image context and can modify specific elements while preserving others.
> **Note:** FLUX.2 models are recommended for image editing. They provide better results than the older FLUX.1 Kontext models.
## Providing Images
**Preferred: Use URLs directly** - simpler and more convenient than base64.
When you have an image URL, pass it directly to `input_image`:
```json
{
"prompt": "Change the background to a beach sunset",
"input_image": "https://example.com/photo.jpg"
}
```
The API fetches URLs automatically. Both URL and base64 work, but URLs are recommended when available.
## Basic Edit Patterns
### Simple Modifications
Direct, single-change instructions:
```
Change the car color to red
```
```
Make the sky a dramatic sunset
```
```
Add snow to the ground
```
### Attribute Changes
Modifying specific characteristics:
```
Change her hair color to platinum blonde
```
```
Make the building taller
```
```
Age the person to appear 20 years older
```
## Controlled Editing
### Explicit Preservation
When you need to keep specific elements unchanged:
```
Change the background to a beach scene while keeping the subject's
pose, clothing, and expression exactly the same
```
```
Transform the daytime photo to nighttime, maintaining the exact
composition, colors of the subject's outfit, and lighting direction
```
### Style Preservation
Preventing unwanted style shifts:
```
Change the season to autumn with falling leaves, but maintain
the photograph's realistic style and color grading
```
```
Add rain effects to the scene while preserving the painting's
impressionist brushwork and color palette
```
## Transformation Types
### Environmental Changes
#### Time of Day
```
Convert to golden hour lighting with warm tones and long shadows,
keeping all other elements identical
```
```
Transform to blue hour with city lights beginning to glow,
maintaining the exact composition
```
#### Weather
```
Add heavy rain with wet reflections on surfaces, dark overcast sky
```
```
Create a foggy atmosphere with reduced visibility, mysterious mood
```
#### Season
```
Transform to winter with snow covering surfaces, bare trees,
cold blue color cast
```
```
Change to spring with cherry blossoms, fresh green leaves,
soft warm lighting
```
### Style Transfer
#### Artistic Movements
```
Transform into Art Nouveau style with flowing organic lines,
decorative patterns, and muted earth tones
```
```
Convert to Pop Art style with bold primary colors, halftone dots,
and high contrast graphic treatment
```
#### Artist References
```
Reimagine in the style of Monet with visible brushstrokes,
soft focus, and impressionist color harmony
```
```
Transform to match Edward Hopper's style with dramatic lighting,
urban isolation feeling, and muted palette
```
#### Medium Conversion
```
Convert this photograph to a detailed pencil sketch with
careful shading and visible line work
```
```
Transform into a watercolor painting with soft edges,
transparent washes, and paper texture visible
```
### Subject Modifications
#### Clothing Changes
```
Change the outfit to a formal black suit with white shirt and red tie
```
```
Replace the casual clothes with traditional Japanese kimono in blue floral pattern
```
#### Expression Changes
```
Change the expression to a warm genuine smile
```
```
Make the expression more serious and contemplative
```
#### Age Modifications
```
Age the subject to appear as a wise elderly person with grey hair and wrinkles
```
```
Make the subject appear younger, around 25 years old
```
### Object Editing
#### Addition
```
Add a vintage leather briefcase in the subject's left hand
```
```
Place a steaming cup of coffee on the table
```
#### Removal
```
Remove the background people, replace with empty street
```
```
Remove the text/logo from the shirt, replace with solid color
```
#### Replacement
```
Replace the modern car with a 1960s vintage Mustang in cherry red
```
```
Swap the coffee mug for an ornate teacup with floral pattern
```
## Text Editing
### Adding Text
```
Add a neon sign reading "OPEN 24 HOURS" in the window,
glowing red letters with blue outline
```
```
Include a wooden sign with hand-painted text "Welcome Home"
mounted above the door
```
### Modifying Text
```
Change the store sign to read "BAKER'S DOZEN" in the same style
```
```
Update the poster text to "SUMMER SALE 2025" maintaining the design
```
## Complex Multi-Step Edits
For dramatic transformations, consider breaking into steps:
### Step-by-Step Approach
Instead of:
```
Transform this modern office into a Victorian library with completely
different furniture, add a fireplace, change the lighting to candlelit,
and age the photograph
```
Try sequential edits:
1. `Change the furniture style to Victorian antique pieces`
2. `Add a stone fireplace on the right wall`
3. `Transform lighting to warm candlelit atmosphere`
4. `Apply vintage photograph aesthetic with sepia tones`
## Common Pitfalls
### Avoid Vague Instructions
```
Bad: Make it look better
Good: Increase contrast, add warm color grading, sharpen details
```
### Be Specific About Scope
```
Bad: Change the background
Good: Replace the office background with a tropical beach at sunset,
maintaining the subject's exact position and lighting direction
```
### Explicit Style Preservation
```
Bad: Make it nighttime
Good: Transform to nighttime while maintaining the photorealistic style,
add appropriate artificial lighting sources
```
## Best Practices
1. **Start Simple** - Begin with single-element changes
2. **Be Explicit** - State what should change AND what should stay
3. **Reference Context** - Mention existing elements when relevant
4. **Iterate** - Refine through multiple small edits rather than one large one
5. **Preserve Deliberately** - Always specify style/composition preservation needs
.claude/skills/flux-best-practices/rules/json-structured-prompting.md
+241
View File
@@ -0,0 +1,241 @@
---
name: json-structured-prompting
description: Using JSON format for complex scene composition
---
# JSON Structured Prompting
For complex scenes with multiple elements, spatial relationships, or production automation, use JSON-structured prompts.
## When to Use
- Multiple characters with distinct attributes
- Precise spatial positioning
- Complex scene composition
- Reproducible, template-based prompts
- Programmatic prompt generation
- Production workflows with variable substitution
## Basic Structure
```json
{
"scene": {
"setting": "description of environment",
"time": "time of day/period",
"mood": "atmospheric quality"
},
"subjects": [
{
"type": "person/object/animal",
"description": "detailed description",
"position": "location in frame",
"action": "what they're doing"
}
],
"style": {
"medium": "photography/painting/illustration",
"technique": "specific style details",
"reference": "artist or style reference"
},
"technical": {
"camera": "camera and lens",
"lighting": "lighting setup",
"composition": "framing details"
},
"colors": ["#hex1", "#hex2"]
}
```
## Single Subject Example
```json
{
"scene": {
"setting": "cozy home office with bookshelves",
"time": "late afternoon",
"mood": "focused, peaceful"
},
"subjects": [
{
"type": "person",
"description": "woman in her 30s, dark curly hair in loose bun, wearing casual cream sweater",
"position": "seated at desk, center frame",
"action": "typing on laptop, slight smile of concentration"
}
],
"style": {
"medium": "photography",
"technique": "lifestyle editorial",
"reference": "kinfolk magazine aesthetic"
},
"technical": {
"camera": "Sony A7III with 50mm f/1.8",
"lighting": "soft natural window light from left",
"composition": "medium shot, rule of thirds"
}
}
```
## Multi-Character Scene
```json
{
"scene": {
"setting": "Victorian-era drawing room with ornate wallpaper and antique furniture",
"time": "evening, candlelit",
"mood": "tense, mysterious"
},
"subjects": [
{
"id": "detective",
"type": "person",
"description": "tall man in his 50s, sharp features, grey at temples, wearing brown tweed suit",
"position": "standing center-left, facing right",
"action": "examining a letter with magnifying glass, intense focus"
},
{
"id": "lady",
"type": "person",
"description": "elegant woman in her 40s, auburn hair in Victorian updo, emerald green evening dress",
"position": "seated on chaise lounge, right side",
"action": "watching the detective with concealed anxiety, hands clasped"
},
{
"id": "butler",
"type": "person",
"description": "elderly man in formal butler attire, stoic expression",
"position": "background, near doorway",
"action": "standing at attention, observing"
}
],
"style": {
"medium": "oil painting",
"technique": "classical realism with dramatic lighting",
"reference": "Victorian narrative painting, John Singer Sargent"
},
"technical": {
"lighting": "warm candlelight as key, cool moonlight through window as fill",
"composition": "triangular arrangement of figures, detective at apex"
}
}
```
## Product Scene with Colors
```json
{
"scene": {
"setting": "minimalist product photography studio",
"mood": "clean, premium, aspirational"
},
"subjects": [
{
"type": "product",
"description": "sleek wireless earbuds in charging case",
"position": "center, slightly angled",
"details": "matte finish, subtle branding"
}
],
"style": {
"medium": "commercial photography",
"technique": "high-end product shot",
"reference": "Apple product photography"
},
"technical": {
"camera": "Phase One with 120mm macro",
"lighting": "large softbox overhead, subtle fill from below",
"composition": "centered, hero product shot"
},
"colors": {
"product": "#1A1A2E",
"accent": "#E94560",
"background": "#FFFFFF"
}
}
```
## Converting JSON to Natural Language
Flatten your JSON into flowing prose for the actual prompt:
### From JSON
```json
{
"subjects": [
{
"type": "person",
"description": "elderly craftsman with weathered hands",
"position": "seated at workbench",
"action": "carefully carving wood"
}
],
"scene": { "setting": "traditional workshop", "time": "morning" },
"technical": { "lighting": "natural window light from right" }
}
```
### To Prompt
```
An elderly craftsman with weathered hands seated at his workbench in a
traditional workshop, carefully carving wood with focused precision.
Morning natural light streams through the window from the right,
illuminating the wood shavings and tools scattered across the worn surface.
```
## Template Variables
Use JSON structure for template-based generation:
```json
{
"template": "product_hero",
"variables": {
"product_name": "{{PRODUCT_NAME}}",
"product_color": "{{PRODUCT_COLOR}}",
"brand_primary": "{{BRAND_HEX_1}}",
"brand_secondary": "{{BRAND_HEX_2}}",
"background_style": "{{BG_STYLE}}"
},
"prompt_template": "Professional product photography of {{PRODUCT_NAME}} in {{PRODUCT_COLOR}}, brand colors {{BRAND_HEX_1}} and {{BRAND_HEX_2}} accents, {{BG_STYLE}} background, commercial quality"
}
```
## Spatial Relationships
Define explicit spatial relationships:
```json
{
"composition": {
"layout": "triangular",
"focal_point": "center-left intersection",
"depth_layers": [
{
"layer": "foreground",
"elements": ["flowers in vase"],
"focus": "soft blur"
},
{
"layer": "midground",
"elements": ["main subject"],
"focus": "sharp"
},
{
"layer": "background",
"elements": ["window", "garden view"],
"focus": "soft blur"
}
]
}
}
```
## Best Practices
1. **Use IDs for References** - Give subjects IDs when they interact
2. **Separate Concerns** - Keep scene, subjects, style, and technical distinct
3. **Be Consistent** - Use the same terminology throughout
4. **Include All Details** - Don't assume, specify everything
5. **Flatten for Execution** - Convert to natural language before sending to model
6. **Version Templates** - Track template versions for reproducibility
.claude/skills/flux-best-practices/rules/model-selection-guide.md
+246
View File
@@ -0,0 +1,246 @@
---
name: model-selection-guide
description: Choosing the right FLUX model for your use case
---
# FLUX Model Selection Guide
Decision guide for selecting the optimal FLUX model based on your requirements.
## Quick Decision Matrix
| Priority | Recommended Model |
| ------------- | ------------------------------- |
| Speed | FLUX.2 [klein] |
| Quality | FLUX.2 [max] |
| Balance | FLUX.2 [pro] |
| Typography | FLUX.2 [flex] |
| Image Editing | FLUX.2 [klein], [pro], or [max] |
| Local/Free | FLUX.2 [dev] |
| Inpainting | FLUX.1 Fill |
## Decision Tree
```
What's your primary need?
├─ Generate images (text-to-image OR image editing)
│ ├─ Need FASTEST possible generation?
│ │ └─ FLUX.2 [klein] (supports up to 4 reference images)
│ │
│ ├─ Need HIGHEST quality output?
│ │ └─ FLUX.2 [max] (supports up to 8-10 reference images)
│ │
│ ├─ Need TEXT/TYPOGRAPHY in image?
│ │ └─ FLUX.2 [flex] (supports up to 8 reference images)
│ │
│ ├─ Need BALANCED speed/quality?
│ │ └─ FLUX.2 [pro] (supports up to 8 reference images)
│ │
│ └─ Need LOCAL/FREE generation?
│ └─ FLUX.2 [dev]
│
├─ Need REAL-TIME web information?
│ └─ FLUX.2 [max] (grounding search)
│
└─ FLUX.1 family (only use when explicitly asked by user)
├─ FLUX.1 Kontext - context-aware editing
└─ FLUX.1 Fill - inpainting/object removal
```
**Note:** All FLUX.2 models natively support image-to-image editing via reference images. Simply provide your source image(s) as references and describe the desired changes.
## Detailed Model Comparisons
### By Speed
| Model | Relative Speed | Best For |
| ----------------- | -------------- | ----------------------- |
| FLUX.2 [klein] 4B | Fastest | Rapid prototyping |
| FLUX.2 [klein] 9B | Very Fast | Better quality previews |
| FLUX.2 [pro] | Medium | Production workflows |
| FLUX.2 [flex] | Medium | Typography tasks |
| FLUX.2 [max] | Slower | Final hero images |
### By Quality
| Model | Quality Level | Trade-off |
| ----------------- | ------------- | -------------------------- |
| FLUX.2 [max] | Highest | Slowest, most expensive |
| FLUX.2 [pro] | High | Good balance |
| FLUX.2 [flex] | High (text) | Specialized for typography |
| FLUX.2 [klein] 9B | Good | Fast, slightly less detail |
| FLUX.2 [klein] 4B | Moderate | Fastest, preview quality |
### By Cost
> **Credit pricing:** 1 credit = $0.01 USD. FLUX.2 uses megapixel-based pricing.
#### FLUX.2 Models
| Model | 1st MP | +MP | 1MP T2I | 1MP I2I | Volume Recommendation |
| ----------------- | ------ | ---- | ------- | ------- | --------------------------- |
| FLUX.2 [klein] 4B | 1.4c | 0.1c | $0.014 | $0.015 | High volume, previews |
| FLUX.2 [klein] 9B | 1.5c | 0.2c | $0.015 | $0.017 | High volume, better quality |
| FLUX.2 [pro] | 3c | 1.5c | $0.03 | $0.045 | Production workloads |
| FLUX.2 [max] | 7c | 3c | $0.07 | $0.10 | Hero images, premium |
| FLUX.2 [flex] | 5c | 5c | $0.05 | $0.10 | Typography |
| FLUX.2 [dev] | - | - | Free | Free | Local dev (non-commercial) |
> **Pricing formula:** `(firstMP + (outputMP-1) * mpPrice) + (inputMP * mpPrice)` in cents
#### FLUX.1 Models
| Model | Price/Image | Use Case |
| -------------------- | ----------- | ----------------------- |
| FLUX.1 Kontext [pro] | $0.04 | Context-aware editing |
| FLUX.1 Kontext [max] | $0.08 | Max quality editing |
| FLUX1.1 [pro] | $0.04 | Standard T2I |
| FLUX1.1 [pro] Ultra | $0.06 | Ultra high-resolution |
| FLUX1.1 [pro] Raw | $0.06 | Candid photography feel |
| FLUX.1 Fill [pro] | $0.05 | Inpainting |
| FLUX.1 [pro] | $0.05 | Original pro model |
> Use [bfl.ai/pricing](https://bfl.ai/pricing) calculator for exact costs at different resolutions.
## Use Case Recommendations
### Creative Exploration / Ideation
**Recommended: FLUX.2 [klein]**
- Fast iterations
- Quick concept testing
- Mood board generation
- Exploring prompt variations
### Production Marketing Assets
**Recommended: FLUX.2 [pro]**
- Consistent quality
- Reasonable speed
- Cost-effective at scale
- Reliable for automation
### Hero Images / Premium Content
**Recommended: FLUX.2 [max]**
- Maximum detail
- Best coherence
- Supports grounding search
- Worth the premium for key visuals
### Typography / Signage / Posters
**Recommended: FLUX.2 [flex]**
- Superior text rendering
- Adjustable quality settings
- Best for readable text
- UI mockups and infographics
### Character Consistency
**Recommended: FLUX.2 [max] or [pro]**
- Multi-reference support (up to 8-10 images)
- Best editing consistency
- Maintains identity across scenes
- Superior quality over FLUX.1 Kontext
### Photo Editing / Retouching
**Recommended: FLUX.2 [klein], [pro], or [max]**
- Native image-to-image support via references
- Style transfer
- Object modification
- Attribute changes
- Better results than FLUX.1 Kontext
### Real-Time Information
**Recommended: FLUX.2 [max]**
- Grounding search feature
- Current events
- Recent news visualization
- Weather/location data
### Local Development / Testing
**Recommended: FLUX.2 [dev]**
- No API costs
- Full control
- Fine-tuning experiments
- Offline capability
### Editorial with Typography
```
1. FLUX.2 [max] - Generate base image (highest quality)
2. FLUX.2 [flex] - Add text overlay pass
```
### Character-Consistent Series
```
1. FLUX.2 [max] - Create character reference
2. FLUX.2 [max]/[pro] - Generate consistent variations using reference images
3. FLUX.2 [klein] - Quick iteration on variations if needed
```
### E-commerce Product Pipeline
```
1. FLUX.2 [pro] - Bulk product generations
2. FLUX.2 [pro]/[klein] - Product variations (colors, angles) using references
3. FLUX.2 [flex] - Add promotional text/pricing
```
## Constraint-Based Selection
### Limited Budget
- **High volume**: FLUX.2 [klein] 4B
- **Quality needed**: FLUX.2 [pro] (best value)
### Tight Deadline
- **Any task**: FLUX.2 [klein]
- **Quality matters**: FLUX.2 [pro]
### Maximum Quality Required
- **Always**: FLUX.2 [max]
### Text Must Be Readable
- **Always**: FLUX.2 [flex]
### Editing Existing Images
- **Fast edits**: FLUX.2 [klein] with reference images
- **Quality edits**: FLUX.2 [max] or [pro] with reference images
- **Alternative**: FLUX.1 Kontext (FLUX.2 preferred)
### Rate Limit Sensitivity
- **Prefer**: FLUX.2 models (24 concurrent limit)
- **Avoid**: FLUX.1 Kontext Max (6 concurrent limit)
## Summary Cheat Sheet
```
Speed? → FLUX.2 [klein]
Quality? → FLUX.2 [max]
Balance? → FLUX.2 [pro]
Text? → FLUX.2 [flex]
Edit? → FLUX.2 [klein/pro/max] with reference images
Free? → FLUX.2 [dev]
```
**Key insight:** All FLUX.2 models support image editing natively via reference images. FLUX.2 is recommended over FLUX.1 Kontext for editing tasks.
.claude/skills/flux-best-practices/rules/multi-reference-editing.md
+241
View File
@@ -0,0 +1,241 @@
---
name: multi-reference-editing
description: Using multiple reference images for complex compositions
---
# Multi-Reference Image Editing
Guide to using multiple reference images for character consistency, style transfer, and complex compositions.
## Overview
FLUX.2 models support multiple reference images for advanced editing:
- **FLUX.2 [klein]**: Up to 4 reference images - fast editing
- **FLUX.2 [pro]**: Up to 8 via API - balanced quality/speed
- **FLUX.2 [max]**: Up to 8 via API, 10 in playground - highest quality
- **FLUX.2 [flex]**: Up to 8 via API - best for typography
> **Note:** FLUX.2 models are recommended over FLUX.1 Kontext Max for better results.
## Providing Images
**Preferred: Use URLs directly** - simpler and more convenient than base64.
Pass image URLs directly to `input_image`, `input_image_2`, etc.:
```json
{
"prompt": "Person from image 1 wearing outfit from image 2",
"input_image": "https://example.com/person.jpg",
"input_image_2": "https://example.com/outfit.jpg"
}
```
The API fetches URLs automatically. Both URL and base64 work, but URLs are recommended when available.
## Reference Methods
### Natural Language Description
Describe relationships between images naturally:
```
The person from image 1 is sitting at the cafe table from image 2,
wearing the outfit from image 3, with the warm lighting style of image 4
```
### Explicit Indexing
Reference images by number for precision:
```
Replace the background of image 1 with the landscape from image 2,
maintaining the subject's exact position and lighting
```
```
Combine the face from image 1 with the hairstyle from image 2
on the body pose from image 3
```
## Use Cases
### Character Consistency
Maintain the same character across multiple scenes:
```
Input: Reference image of character
Prompt: The character from image 1 walking through a busy Tokyo street
at night, neon lights reflecting on wet pavement
```
For sequential consistency:
```
The same person from image 1, now seated at a desk in a modern office,
same clothing and hairstyle, different environment
```
### Style Transfer
Apply the style of one image to another:
```
Transform image 1 into the artistic style of image 2,
maintaining the original composition and subject
```
```
Apply the color grading and mood from image 2 to the scene in image 1
```
### Pose Guidance
Use a reference for body positioning:
```
The person from image 1 in the exact pose shown in image 2,
placed in the environment from image 3
```
### Object Composition
Combine elements from multiple images:
```
Place the product from image 1 on the table setting from image 2,
using the lighting style from image 3
```
### Background Replacement
```
Keep the subject from image 1 exactly as shown, replace the background
with the beach scene from image 2, match the lighting naturally
```
## Multi-Character Scenes
### Two Characters
```
Image 1 (person A) and image 2 (person B) having a conversation
at a coffee shop table, person A on the left gesturing, person B
on the right listening intently
```
### Group Composition
```
The three people from images 1, 2, and 3 standing together for a
group photo, arranged left to right in that order, friendly poses,
outdoor park setting
```
## Attribute Mixing
### Selective Attribute Transfer
```
The face and expression from image 1, the hairstyle from image 2,
wearing the outfit from image 3, in the pose from image 4
```
### Partial Transfer
```
Apply only the color palette from image 2 to image 1,
keeping all other aspects (style, composition, lighting) unchanged
```
## Collage Method
Use a collage input for layout guidance:
```
Arrange the scene using the layout shown in the collage input:
- Person from image 1 in the left position
- Object from image 2 in the center position
- Background element from image 3 filling the right side
```
## Best Practices
### 1. Clear Image Roles
Specify what each reference provides:
```
Image 1: face/identity reference
Image 2: pose/body reference
Image 3: style/aesthetic reference
Image 4: environment/background reference
```
### 2. Quality References
- Use high-quality, clear reference images
- Ensure good lighting in references
- Avoid heavily processed or filtered images
### 3. Consistent Lighting
When combining elements:
```
...ensure the lighting direction matches across all elements,
with main light source from the upper left
```
### 4. Resolution Awareness
For [pro] API with 9MP total limit:
- At 1MP output: up to 8 reference images comfortably
- Calculate: input images + output = total MP
### 5. Explicit Relationships
Don't assume - specify exactly how elements relate:
```
Vague: The person and the background together
Better: The person from image 1 standing in the foreground,
the beach from image 2 visible behind them at a distance
```
## Complex Composition Example
```
Create a scene combining:
- The woman from image 1 (keep exact face, expression, hair)
- Wearing the vintage dress from image 2 (exact pattern and cut)
- In the pose from image 3 (seated position, arm placement)
- Set in the library from image 4 (bookshelves, furniture)
- Using the warm lighting style from image 5 (golden hour quality)
Position her in the center of frame, medium shot, looking slightly
to the right with a thoughtful expression.
```
## Troubleshooting
### Elements Not Transferring
- Be more specific about which element from which image
- Use explicit indexing ("from image 1")
- Reduce the number of references and complexity
### Inconsistent Blending
- Specify lighting consistency
- Describe how elements should interact
- Use style references to unify the composition
### Identity Drift
- Emphasize key identifying features
- Use phrases like "maintaining exact likeness"
- Provide multiple angles of the same subject if available
.claude/skills/flux-best-practices/rules/negative-prompt-alternatives.md
+202
View File
@@ -0,0 +1,202 @@
---
name: negative-prompt-alternatives
description: Positive alternatives to negative prompts
---
# Negative Prompt Alternatives
FLUX does not support negative prompts. This guide provides positive alternatives for common negative prompt patterns.
## Why No Negative Prompts?
Negative prompts can actually make models focus MORE on unwanted elements. Instead, describe exactly what you DO want - this gives clearer direction and better results.
## Replacement Strategy
For any unwanted element:
1. Identify what you don't want
2. Ask: "What would be there instead?"
3. Describe the positive alternative
## Common Replacements
### People/Crowds
| Instead of | Use |
|-----------|-----|
| "no people" | "empty", "deserted", "solitary", "abandoned" |
| "no crowds" | "quiet", "peaceful", "secluded", "private" |
| "without background people" | "isolated subject", "clean background", "solo figure" |
**Example:**
```
Bad: A beach scene, no people
Good: A deserted beach at dawn, pristine untouched sand, solitary seagull
```
### Skin/Appearance
| Instead of | Use |
|-----------|-----|
| "no makeup" | "natural skin", "bare face", "fresh-faced" |
| "no blemishes" | "clear skin", "smooth complexion", "healthy glow" |
| "no wrinkles" | "youthful skin", "smooth features" |
**Example:**
```
Bad: Portrait of woman, no makeup, no blemishes
Good: Portrait of a woman with natural clear skin, fresh-faced with a healthy glow
```
### Accessories
| Instead of | Use |
|-----------|-----|
| "no glasses" | "visible eyes", "unobstructed gaze", "clear eye contact" |
| "no hat" | "bare head", "visible hair", "uncovered head" |
| "no jewelry" | "minimal accessories", "understated", "unadorned" |
**Example:**
```
Bad: Man portrait, no glasses, no hat
Good: Portrait of a man with clear direct gaze, wind-swept visible hair
```
### Colors
| Instead of | Use |
|-----------|-----|
| "no color" | "monochrome", "black and white", "grayscale" |
| "not colorful" | "muted tones", "subdued palette", "desaturated" |
| "no bright colors" | "neutral tones", "earth tones", "soft pastels" |
**Example:**
```
Bad: Landscape photo, no bright colors
Good: Landscape in muted earth tones, soft morning light, desaturated palette
```
### Text/Watermarks
| Instead of | Use |
|-----------|-----|
| "no text" | "clean surfaces", "unmarked", "text-free" |
| "no watermark" | "pristine image", "clean composition" |
| "no logos" | "unbranded", "plain", "logo-free surface" |
**Example:**
```
Bad: Product photo, no watermark, no text
Good: Clean product photography with pristine unmarked surfaces, minimal unbranded design
```
### Style/Era
| Instead of | Use |
|-----------|-----|
| "not modern" | "traditional", "classical", "vintage", "historical" |
| "no CGI look" | "photorealistic", "authentic", "natural", "organic" |
| "not cartoonish" | "realistic", "lifelike", "naturalistic" |
**Example:**
```
Bad: Building design, not modern, no futuristic elements
Good: Traditional Victorian architecture with classical ornate details and period-accurate features
```
### Quality/Artifacts
| Instead of | Use |
|-----------|-----|
| "no blur" | "sharp focus", "crisp details", "tack-sharp" |
| "no noise" | "clean image", "smooth gradients", "low ISO" |
| "no artifacts" | "pristine quality", "clean render", "flawless" |
**Example:**
```
Bad: Portrait, no blur, no noise
Good: Tack-sharp portrait with pristine image quality, smooth skin tones, crisp details
```
### Objects
| Instead of | Use |
|-----------|-----|
| "no cars" | "pedestrian area", "car-free zone", "walking street" |
| "no buildings" | "open landscape", "natural scenery", "wilderness" |
| "no furniture" | "empty room", "bare space", "minimalist interior" |
**Example:**
```
Bad: Street scene, no cars, no modern buildings
Good: Historic cobblestone walking street lined with traditional stone buildings from the 1800s
```
### Weather/Environment
| Instead of | Use |
|-----------|-----|
| "no rain" | "clear sky", "dry weather", "sunny day" |
| "no clouds" | "clear blue sky", "cloudless", "perfect visibility" |
| "not dark" | "well-lit", "bright", "daylight", "illuminated" |
**Example:**
```
Bad: Outdoor portrait, no rain, no clouds, not dark
Good: Outdoor portrait under clear blue sky on a bright sunny day, perfect natural lighting
```
### Composition
| Instead of | Use |
|-----------|-----|
| "no distractions" | "clean composition", "focused framing", "minimal elements" |
| "nothing in background" | "solid background", "isolated subject", "clean backdrop" |
| "no clutter" | "organized", "tidy", "minimal", "streamlined" |
**Example:**
```
Bad: Product shot, no distractions, nothing in background
Good: Product on clean white seamless backdrop, isolated subject, minimal focused composition
```
## Complex Replacement Examples
### Original Negative-Heavy Prompt
```
Portrait of a woman, no glasses, no makeup, no wrinkles, no blemishes,
no bright colors, no distracting background, no harsh lighting
```
### Positive Rewrite
```
Portrait of a youthful woman with clear natural skin and visible bright eyes,
fresh-faced with a healthy glow, wearing muted earth tones against a soft
blurred neutral background, gentle diffused lighting creating soft shadows
```
### Original Negative-Heavy Prompt
```
Landscape photo, no people, no buildings, no power lines, no modern elements,
no overcast sky, no dead trees
```
### Positive Rewrite
```
Pristine wilderness landscape with lush green living forest, clear blue sky,
untouched natural scenery stretching to the horizon, peaceful solitude with
only birdsong and wind, golden hour sunlight filtering through healthy foliage
```
## Quick Reference Card
| Unwanted | Positive Alternative |
|----------|---------------------|
| No people | Empty, solitary, deserted |
| No makeup | Natural, fresh-faced, bare |
| No text | Clean, unmarked, pristine |
| No blur | Sharp, crisp, tack-sharp |
| No modern | Traditional, vintage, classical |
| No dark | Bright, well-lit, luminous |
| No busy | Minimal, clean, focused |
| No artificial | Natural, organic, authentic |
.claude/skills/flux-best-practices/rules/t2i-prompting.md
+230
View File
@@ -0,0 +1,230 @@
---
name: t2i-prompting
description: Text-to-image prompting patterns and techniques
---
# Text-to-Image (T2I) Prompting
Comprehensive guide to crafting effective text-to-image prompts for FLUX models.
## Prompt Structure Framework
### Basic Formula
```
[Subject] + [Action] + [Style] + [Context] + [Lighting] + [Technical]
```
### Expanded Framework
```
[Main Subject] - who/what is the focus
[Attributes] - characteristics, details, clothing
[Action/Pose] - what they're doing
[Environment] - where, setting, background
[Style/Medium] - artistic approach
[Lighting] - light source, quality, mood
[Composition] - framing, camera angle
[Technical] - camera, lens, film stock
```
## Subject Types
### People/Portraits
```
A distinguished professor in his 60s with silver hair and round spectacles,
wearing a tweed jacket with leather elbow patches, deep-set thoughtful eyes,
slight smile suggesting hidden wisdom
```
### Animals
```
A majestic snow leopard with piercing blue-grey eyes, thick spotted fur
dusted with snowflakes, powerful muscular build, alert posture on a
rocky outcrop
```
### Objects/Products
```
A vintage Leica M3 camera with worn brass edges showing decades of use,
black leather covering with patina, sitting on weathered wooden table
```
### Landscapes
```
A dramatic fjord at dawn, steep granite cliffs rising from mirror-still
water, wisps of morning mist, distant snow-capped peaks catching first
golden light
```
### Architecture
```
A brutalist concrete apartment building in late afternoon light, geometric
shadows creating abstract patterns, warm sunlight contrasting with cool
grey concrete
```
## Style Categories
### Photorealistic
#### Modern Digital
```
shot on Sony A7IV, clean and sharp, high dynamic range, professional color grading
```
#### Film Photography
```
shot on Kodak Portra 400, natural film grain, organic colors, slight warmth
```
#### Vintage Digital (2000s)
```
early digital camera aesthetic, slight noise, flash photography, candid feel
```
#### 80s Film
```
80s film photography, film grain, warm color cast, soft focus, nostalgic
```
### Artistic Styles
#### Oil Painting
```
classical oil painting style, visible brushstrokes, rich colors, dramatic lighting
```
#### Watercolor
```
delicate watercolor painting, soft edges, transparent washes, paper texture visible
```
#### Digital Art
```
polished digital illustration, clean lines, vibrant colors, professional concept art
```
#### Anime/Manga
```
anime style, large expressive eyes, clean linework, cel shading, vibrant palette
```
## Lighting Patterns
### Portrait Lighting
```
Rembrandt lighting - 45 degree key light creating triangle shadow on cheek
Butterfly lighting - overhead key creating shadow under nose
Split lighting - 90 degree side light, half face in shadow
Loop lighting - slight angle creating small nose shadow
```
### Natural Lighting
```
Golden hour - warm, soft, directional light 1 hour before sunset
Blue hour - cool, ambient light just after sunset
Overcast - soft, even, diffused lighting
Harsh midday - strong contrast, defined shadows
```
### Atmospheric
```
Volumetric light - visible light rays through fog/dust
Rim lighting - backlight creating edge glow
Practical lighting - visible light sources in scene
Neon glow - colorful artificial urban lighting
```
## Camera and Lens Simulation
### Camera Bodies
```
Shot on Hasselblad X2D - medium format, exceptional detail
Shot on Canon 5D Mark IV - professional DSLR quality
Shot on Leica M10 - rangefinder character, smooth tonality
Shot on iPhone 15 Pro - computational photography look
```
### Lens Characteristics
```
85mm f/1.4 - classic portrait, creamy bokeh
24mm f/2.8 - wide angle, environmental
50mm f/1.2 - natural perspective, shallow DOF
135mm f/2 - compressed perspective, smooth background
Macro lens - extreme close-up detail
Tilt-shift lens - miniature effect or architectural correction
```
### Technical Settings
```
f/1.4 - extremely shallow depth of field
f/2.8 - moderate background blur
f/8 - sharp throughout, landscape
f/16 - maximum sharpness, long exposure
ISO 100 - clean, no noise
ISO 3200 - visible grain, low light
```
## Composition Techniques
### Framing
```
extreme close-up - filling frame with detail
close-up - head and shoulders
medium shot - waist up
full shot - entire body
wide shot - subject in environment
establishing shot - location focus
```
### Angles
```
eye level - natural, relatable
low angle - powerful, imposing
high angle - diminished, overview
Dutch angle - tension, unease
bird's eye - pattern, layout
worm's eye - dramatic upward view
```
### Composition Rules
```
rule of thirds - subject at intersection points
centered composition - symmetry, stability
leading lines - guiding eye to subject
frame within frame - natural framing elements
negative space - minimalist, breathing room
```
## Complete Example Prompts
### Editorial Portrait
```
A fashion editorial portrait of a young woman with striking features and
high cheekbones, wearing an avant-garde geometric collar in silver, dramatic
side lighting creating strong shadows, shot on Hasselblad with 100mm lens
at f/2.8, studio background with subtle gradient, high fashion magazine style
```
### Product Photography
```
A premium wireless headphone product shot, matte black finish with rose gold
accents, floating at slight angle against pure white background, soft even
lighting eliminating harsh shadows, reflection visible on glossy surface below,
commercial catalog style, ultra sharp focus throughout
```
### Landscape
```
A misty morning in ancient redwood forest, towering trees disappearing into
fog above, ferns covering forest floor in layers of green, single shaft of
golden sunlight breaking through canopy, shot on large format camera, rich
detail in bark textures, Ansel Adams inspired black and white with deep tones
```
### Architectural
```
Modern minimalist beach house at golden hour, floor-to-ceiling glass walls
reflecting sunset colors, clean white concrete and natural wood, infinity
pool merging with ocean horizon, architectural photography style, wide angle
showing full structure, warm evening light
```
.claude/skills/flux-best-practices/rules/typography-text.md
+241
View File
@@ -0,0 +1,241 @@
---
name: typography-text
description: Prompting for text rendering and typography in FLUX
---
# Typography and Text Prompting
Guide to rendering text in FLUX images. Use FLUX.2 [flex] for best typography results.
## Basic Syntax
Always quote the exact text you want rendered:
```
A coffee shop chalkboard sign displaying "TODAY'S SPECIAL" in decorative script
```
## Core Rules
### 1. Use Quotation Marks
```
Correct: A poster with "HELLO WORLD" in bold letters
Wrong: A poster with HELLO WORLD in bold letters
```
### 2. Specify Font Style
```
"ADVENTURE" in bold sans-serif typography
"Welcome" in elegant cursive script
"CHAPTER ONE" in classic serif typeface
"CODE" in monospace terminal font
"SALE!" in decorative display lettering
```
### 3. Describe Size Hierarchy
```
Large headline "BREAKING NEWS" above smaller subtext "Details inside"
```
### 4. Indicate Placement
```
"OPEN" sign centered in storefront window
"EXIT" text positioned above doorway
"Page 1" in bottom right corner
```
### 5. Front-Load Text
Place text descriptions early in the prompt for better accuracy:
```
Good: A sign reading "FRESH BREAD" in a bakery window...
Less Good: A bakery window with a sign that says "FRESH BREAD"...
```
## Font Style Categories
### Sans-Serif (Modern/Clean)
```
"MINIMAL" in clean geometric sans-serif, Swiss modernist style
"TECH SUMMIT" in bold condensed grotesque typeface
"future" in thin uppercase sans-serif, contemporary design
```
### Serif (Classic/Elegant)
```
"The New Yorker" in traditional serif typeface, editorial masthead
"LUXURY" in high-contrast Didone serif with thin/thick strokes
"Wisdom" in old-style serif with subtle bracketed serifs
```
### Script/Cursive (Decorative)
```
"With Love" in flowing calligraphic script with flourishes
"Signature" in connected brush script, casual elegance
"Romance" in formal copperplate script, wedding invitation style
```
### Display/Decorative
```
"ROCK CONCERT" in distressed vintage concert poster lettering
"CIRCUS" in ornate Victorian display type with decorative elements
"RETRO" in 1970s rounded bubble letters
```
### Handwritten
```
"Note to self" in casual handwritten style, slightly imperfect
"Thanks!" in quick marker pen handwriting
"ideas" in sketchy pencil handwriting
```
### Monospace
```
"CODE_COMPLETE" in terminal monospace, developer aesthetic
"SYSTEM" in typewriter monospace, vintage tech
"DEBUG" in LCD-style digital monospace
```
## Text Effects
### Neon Signs
```
Glowing neon sign spelling "OPEN 24/7" in pink neon tubes with
blue outline, slight glow and reflection, night scene
```
### Metallic/3D
```
"GOLD" in three-dimensional metallic gold letters with realistic
reflections and subtle shadows, luxury aesthetic
```
### Embossed/Debossed
```
"PREMIUM" embossed into leather surface, subtle shadows showing
the raised letterforms
```
### Outlined
```
"MODERN" in outline-only letters, no fill, thin white stroke
on dark background
```
### Gradient Text
```
"SUMMER" with gradient fill from #FF6B6B (coral) at top to
#4ECDC4 (teal) at bottom
```
## Multi-Text Compositions
### Poster Design
```
Event poster with "SUMMER FEST 2025" as large headline in bold
condensed sans-serif at top, "JULY 15-17" as medium subheading
in regular weight, "Central Park, NYC" as small body text at
bottom, all in white text on #FF6B35 (sunset orange) background
```
### Book Cover
```
Book cover design: "THE GREAT GATSBY" in elegant art deco gold
lettering centered in upper third, author name "F. SCOTT FITZGERALD"
in smaller gold caps below, #1A1A2E (midnight blue) background
with geometric gold accents
```
### Magazine Cover
```
Fashion magazine cover with "VOGUE" in classic serif masthead at top,
cover line "SPRING COLLECTION" in bold sans-serif, "The New Rules of Style"
in lighter weight italic, all in white against dramatic portrait
```
### Signage
```
Vintage diner sign: "MEL'S DINER" in red neon script lettering,
"OPEN" below in separate green neon block letters, chrome border,
1950s Americana aesthetic
```
### Business Card
```
Minimalist business card with "JOHN SMITH" in medium weight sans-serif,
"Creative Director" in lighter weight below, contact details in small
type at bottom, #2C3E50 (dark blue) text on white background
```
## Text Placement Strategies
### Centered Composition
```
Centered text layout: "WELCOME" in large caps at center,
perfectly balanced with equal margins
```
### Left-Aligned
```
Left-aligned text block: "Company Name" as header,
"Tagline goes here" below, flush left alignment
```
### Text on Path
```
"GOING IN CIRCLES" text following a circular path around
the center of the design
```
### Text Overlay
```
"ADVENTURE AWAITS" in bold white text overlaid on landscape
photograph, positioned in lower third with slight shadow for readability
```
## Technical Considerations for [flex]
### Steps Parameter
- Higher steps (30-50) = better text quality
- Lower steps (10-20) = faster, lower quality
### Guidance Parameter
- Higher guidance (6-10) = stricter prompt following
- Lower guidance (1.5-4) = more creative interpretation
### Recommended Settings
```
For clean typography: steps=50, guidance=7
For artistic text: steps=30, guidance=4
```
## Troubleshooting
### Misspelled Words
- Keep text short (1-4 words work best)
- Use common words when possible
- Repeat the exact text in the prompt
### Illegible Text
- Specify larger text size
- Use simpler fonts (sans-serif)
- Ensure high contrast with background
- Use [flex] model
### Wrong Font Style
Be more specific:
```
Instead of: "text in a nice font"
Use: "text in bold geometric sans-serif similar to Futura"
```
### Text Not Appearing
- Front-load text description in prompt
- Put text in quotes
- Specify exact placement
- Reduce other prompt complexity
.claude/skills/framer-motion/SKILL.md
+179
View File
@@ -0,0 +1,179 @@
---
name: framer-motion
description: Use when implementing Disney's 12 animation principles with Framer Motion in React applications
---
# Framer Motion Animation Principles
Implement all 12 Disney animation principles using Framer Motion's declarative React API.
## 1. Squash and Stretch
```jsx
<motion.div
animate={{ scaleX: [1, 1.2, 1], scaleY: [1, 0.8, 1] }}
transition={{ duration: 0.3, times: [0, 0.5, 1] }}
/>
```
## 2. Anticipation
```jsx
<motion.div
variants={{
idle: { y: 0, scaleY: 1 },
anticipate: { y: 10, scaleY: 0.9 },
jump: { y: -200 }
}}
initial="idle"
animate={["anticipate", "jump"]}
transition={{ duration: 0.5, times: [0, 0.2, 1] }}
/>
```
## 3. Staging
```jsx
<motion.div animate={{ filter: "blur(3px)", opacity: 0.6 }} /> {/* bg */}
<motion.div animate={{ scale: 1.1, zIndex: 10 }} /> {/* hero */}
```
## 4. Straight Ahead / Pose to Pose
```jsx
<motion.div
animate={{
x: [0, 100, 200, 300],
y: [0, -50, 0, -30]
}}
transition={{ duration: 1, ease: "easeInOut" }}
/>
```
## 5. Follow Through and Overlapping Action
```jsx
<motion.div animate={{ x: 200 }} transition={{ duration: 0.5 }}>
<motion.span
animate={{ x: 200 }}
transition={{ duration: 0.5, delay: 0.05 }} // hair
/>
<motion.span
animate={{ x: 200 }}
transition={{ duration: 0.6, delay: 0.1 }} // cape
/>
</motion.div>
```
## 6. Slow In and Slow Out
```jsx
<motion.div
animate={{ x: 300 }}
transition={{
duration: 0.6,
ease: [0.42, 0, 0.58, 1] // easeInOut cubic-bezier
}}
/>
// Or use: "easeIn", "easeOut", "easeInOut"
```
## 7. Arc
```jsx
<motion.div
animate={{
x: [0, 100, 200],
y: [0, -100, 0]
}}
transition={{ duration: 1, ease: "easeInOut" }}
/>
```
## 8. Secondary Action
```jsx
<motion.button
whileHover={{ scale: 1.05 }}
whileTap={{ scale: 0.95 }}
>
<motion.span
animate={{ rotate: [0, 10, -10, 0] }}
transition={{ duration: 0.3 }}
>
Icon
</motion.span>
</motion.button>
```
## 9. Timing
```jsx
const timings = {
fast: { duration: 0.15 },
normal: { duration: 0.3 },
slow: { duration: 0.6 },
spring: { type: "spring", stiffness: 300, damping: 20 }
};
```
## 10. Exaggeration
```jsx
<motion.div
animate={{ scale: 1.5, rotate: 720 }}
transition={{
type: "spring",
stiffness: 200,
damping: 10 // low damping = overshoot
}}
/>
```
## 11. Solid Drawing
```jsx
<motion.div
style={{ perspective: 1000 }}
animate={{ rotateX: 45, rotateY: 30 }}
transition={{ duration: 0.5 }}
/>
```
## 12. Appeal
```jsx
<motion.div
whileHover={{
scale: 1.02,
boxShadow: "0 20px 40px rgba(0,0,0,0.2)"
}}
transition={{ duration: 0.3 }}
/>
```
## Stagger Children
```jsx
const container = {
hidden: { opacity: 0 },
show: {
opacity: 1,
transition: { staggerChildren: 0.1 }
}
};
<motion.ul variants={container} initial="hidden" animate="show">
{items.map(item => <motion.li variants={itemVariant} />)}
</motion.ul>
```
## Key Framer Motion Features
- `animate` - Target state
- `variants` - Named animation states
- `whileHover` / `whileTap` - Gesture animations
- `transition` - Timing and easing
- `AnimatePresence` - Exit animations
- `useAnimation` - Programmatic control
- `layout` - Auto-animate layout changes
.claude/skills/heygen/SKILL.md
+103
View File
@@ -0,0 +1,103 @@
---
name: heygen
description: |
[DEPRECATED] Use `create-video` for prompt-based video generation or `avatar-video` for precise avatar/scene control. This legacy skill combines both workflows — the newer focused skills provide clearer guidance.
homepage: https://docs.heygen.com/reference/generate-video-agent
allowed-tools: mcp__heygen__*
metadata:
openclaw:
requires:
env:
- HEYGEN_API_KEY
primaryEnv: HEYGEN_API_KEY
---
# HeyGen API (Deprecated)
> **This skill is deprecated.** Use the focused skills instead:
> - **`create-video`** — Generate videos from a text prompt (Video Agent API)
> - **`avatar-video`** — Build videos with specific avatars, voices, scripts, and scenes (v2 API)
This skill remains for backward compatibility but will be removed in a future release.
---
AI avatar video creation API for generating talking-head videos, explainers, and presentations.
## Tool Selection
If HeyGen MCP tools are available (`mcp__heygen__*`), **prefer them** over direct HTTP API calls — they handle authentication and request formatting automatically.
| Task | MCP Tool | Fallback (Direct API) |
|------|----------|----------------------|
| Generate video from prompt | `mcp__heygen__generate_video_agent` | `POST /v1/video_agent/generate` |
| Check video status / get URL | `mcp__heygen__get_video` | `GET /v2/videos/{video_id}` |
| List account videos | `mcp__heygen__list_videos` | `GET /v2/videos` |
| Delete a video | `mcp__heygen__delete_video` | `DELETE /v2/videos/{video_id}` |
If no HeyGen MCP tools are available, use direct HTTP API calls with `X-Api-Key: $HEYGEN_API_KEY` header as documented in the reference files.
## Default Workflow
**Prefer Video Agent** for most video requests.
Always use [prompt-optimizer.md](references/prompt-optimizer.md) guidelines to structure prompts with scenes, timing, and visual styles.
**With MCP tools:**
1. Write an optimized prompt using [prompt-optimizer.md](references/prompt-optimizer.md) → [visual-styles.md](references/visual-styles.md)
2. Call `mcp__heygen__generate_video_agent` with prompt and config (duration_sec, orientation, avatar_id)
3. Call `mcp__heygen__get_video` with the returned video_id to poll status and get the download URL
**Without MCP tools (direct API):**
1. Write an optimized prompt using [prompt-optimizer.md](references/prompt-optimizer.md) → [visual-styles.md](references/visual-styles.md)
2. `POST /v1/video_agent/generate` — see [video-agent.md](references/video-agent.md)
3. `GET /v2/videos/<id>` — see [video-status.md](references/video-status.md)
Only use v2/video/generate when user explicitly needs:
- Exact script without AI modification
- Specific voice_id selection
- Different avatars/backgrounds per scene
- Precise per-scene timing control
- Programmatic/batch generation with exact specs
## Quick Reference
| Task | MCP Tool | Read |
|------|----------|------|
| Generate video from prompt (easy) | `mcp__heygen__generate_video_agent` | [prompt-optimizer.md](references/prompt-optimizer.md) → [visual-styles.md](references/visual-styles.md) → [video-agent.md](references/video-agent.md) |
| Generate video with precise control | — | [video-generation.md](references/video-generation.md), [avatars.md](references/avatars.md), [voices.md](references/voices.md) |
| Check video status / get download URL | `mcp__heygen__get_video` | [video-status.md](references/video-status.md) |
| Add captions or text overlays | — | [captions.md](references/captions.md), [text-overlays.md](references/text-overlays.md) |
| Transparent video for compositing | — | [video-generation.md](references/video-generation.md) (WebM section) |
| Use with Remotion | — | [remotion-integration.md](references/remotion-integration.md) |
## Reference Files
### Foundation
- [references/authentication.md](references/authentication.md) - API key setup and X-Api-Key header
- [references/quota.md](references/quota.md) - Credit system and usage limits
- [references/video-status.md](references/video-status.md) - Polling patterns and download URLs
- [references/assets.md](references/assets.md) - Uploading images, videos, audio
### Core Video Creation
- [references/avatars.md](references/avatars.md) - Listing avatars, styles, avatar_id selection
- [references/voices.md](references/voices.md) - Listing voices, locales, speed/pitch
- [references/scripts.md](references/scripts.md) - Writing scripts, pauses, pacing
- [references/video-generation.md](references/video-generation.md) - POST /v2/video/generate and multi-scene videos
- [references/video-agent.md](references/video-agent.md) - One-shot prompt video generation
- [references/prompt-optimizer.md](references/prompt-optimizer.md) - Writing effective Video Agent prompts (core workflow + rules)
- [references/visual-styles.md](references/visual-styles.md) - 20 named visual styles with full specs
- [references/prompt-examples.md](references/prompt-examples.md) - Full production prompt example + ready-to-use templates
- [references/dimensions.md](references/dimensions.md) - Resolution and aspect ratios
### Video Customization
- [references/backgrounds.md](references/backgrounds.md) - Solid colors, images, video backgrounds
- [references/text-overlays.md](references/text-overlays.md) - Adding text with fonts and positioning
- [references/captions.md](references/captions.md) - Auto-generated captions and subtitles
### Advanced Features
- [references/templates.md](references/templates.md) - Template listing and variable replacement
- [references/photo-avatars.md](references/photo-avatars.md) - Creating avatars from photos
- [references/webhooks.md](references/webhooks.md) - Webhook endpoints and events
### Integration
- [references/remotion-integration.md](references/remotion-integration.md) - Using HeyGen in Remotion compositions
.claude/skills/heygen/references/assets.md
+339
View File
@@ -0,0 +1,339 @@
---
name: assets
description: Uploading images, videos, and audio for use in HeyGen video generation
---
# Asset Upload and Management
HeyGen allows you to upload custom assets (images, videos, audio) for use in video generation, such as backgrounds, talking photo sources, and custom audio.
## Upload Flow
Asset uploads are a single-step process: POST the raw file binary directly to the upload endpoint. The Content-Type header must match the file's MIME type.
## Uploading an Asset
**Endpoint:** `POST https://upload.heygen.com/v1/asset`
### Request
| Header | Required | Description |
|--------|:--------:|-------------|
| `X-Api-Key` | ✓ | Your HeyGen API key |
| `Content-Type` | ✓ | MIME type of the file (e.g. `image/jpeg`) |
The request body is the raw binary file data. No JSON or form fields are needed.
### Response
| Field | Type | Description |
|-------|------|-------------|
| `code` | number | Status code (`100` = success) |
| `data.id` | string | Unique asset ID for use in video generation |
| `data.name` | string | Asset name |
| `data.file_type` | string | `image`, `video`, or `audio` |
| `data.url` | string | Accessible URL for the uploaded file |
| `data.image_key` | string \| null | Key for creating uploaded photo avatars (images only) |
| `data.folder_id` | string | Folder ID (empty if not in a folder) |
| `data.meta` | string \| null | Asset metadata |
| `data.created_ts` | number | Unix timestamp of creation |
### curl
```bash
curl -X POST "https://upload.heygen.com/v1/asset" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: image/jpeg" \
--data-binary '@./background.jpg'
```
### TypeScript
```typescript
import fs from "fs";
import path from "path";
interface AssetUploadResponse {
code: number;
data: {
id: string;
name: string;
file_type: string;
url: string;
image_key: string | null;
folder_id: string;
meta: string | null;
created_ts: number;
};
msg: string | null;
message: string | null;
}
async function uploadAsset(filePath: string, contentType: string): Promise<AssetUploadResponse["data"]> {
const resolvedPath = path.resolve(filePath);
const fileBuffer = fs.readFileSync(resolvedPath);
const response = await fetch("https://upload.heygen.com/v1/asset", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": contentType,
},
body: fileBuffer,
});
const json: AssetUploadResponse = await response.json();
if (json.code !== 100) {
throw new Error(json.message ?? "Upload failed");
}
return json.data;
}
// Usage
const asset = await uploadAsset("./background.jpg", "image/jpeg");
console.log(`Uploaded asset: ${asset.id}`);
console.log(`Asset URL: ${asset.url}`);
```
### TypeScript (with streams for large files)
```typescript
import fs from "fs";
import path from "path";
import { stat } from "fs/promises";
async function uploadLargeAsset(filePath: string, contentType: string): Promise<AssetUploadResponse["data"]> {
const resolvedPath = path.resolve(filePath);
const fileStats = await stat(resolvedPath);
const fileStream = fs.createReadStream(resolvedPath);
const response = await fetch("https://upload.heygen.com/v1/asset", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": contentType,
"Content-Length": fileStats.size.toString(),
},
body: fileStream as any,
// @ts-ignore - duplex is needed for streaming
duplex: "half",
});
const json: AssetUploadResponse = await response.json();
if (json.code !== 100) {
throw new Error(json.message ?? "Upload failed");
}
return json.data;
}
```
### Python
```python
import requests
import os
def upload_asset(file_path: str, content_type: str) -> dict:
with open(file_path, "rb") as f:
response = requests.post(
"https://upload.heygen.com/v1/asset",
headers={
"X-Api-Key": os.environ["HEYGEN_API_KEY"],
"Content-Type": content_type
},
data=f
)
data = response.json()
if data.get("code") != 100:
raise Exception(data.get("message", "Upload failed"))
return data["data"]
# Usage
asset = upload_asset("./background.jpg", "image/jpeg")
print(f"Uploaded asset: {asset['id']}")
print(f"Asset URL: {asset['url']}")
```
## Supported Content Types
| Type | Content-Type | Use Case |
|------|--------------|----------|
| JPEG | `image/jpeg` | Backgrounds, talking photos |
| PNG | `image/png` | Backgrounds, overlays |
| MP4 | `video/mp4` | Video backgrounds |
| WebM | `video/webm` | Video backgrounds |
| MP3 | `audio/mpeg` | Custom audio input |
| WAV | `audio/wav` | Custom audio input |
## Uploading from URL
If your asset is already hosted online:
```typescript
async function uploadFromUrl(sourceUrl: string, contentType: string): Promise<AssetUploadResponse["data"]> {
// 1. Validate and download the file
const url = new URL(sourceUrl);
if (url.protocol !== "https:") {
throw new Error("Only HTTPS URLs are supported");
}
const sourceResponse = await fetch(sourceUrl);
const buffer = Buffer.from(await sourceResponse.arrayBuffer());
// 2. Upload directly to HeyGen
const response = await fetch("https://upload.heygen.com/v1/asset", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": contentType,
},
body: buffer,
});
const json: AssetUploadResponse = await response.json();
if (json.code !== 100) {
throw new Error(json.message ?? "Upload failed");
}
return json.data;
}
```
## Using Uploaded Assets
### As Background Image
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello, this is a video with a custom background!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: asset.url, // Use the URL from the upload response
},
},
],
};
```
### As Talking Photo Source
```typescript
const talkingPhotoConfig = {
video_inputs: [
{
character: {
type: "talking_photo",
talking_photo_id: asset.id, // Use the ID from the upload response
},
voice: {
type: "text",
input_text: "Hello from my talking photo!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
};
```
### As Audio Input
```typescript
const audioConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "audio",
audio_url: asset.url, // Use the URL from the upload response
},
},
],
};
```
## Complete Upload Workflow
```typescript
async function createVideoWithCustomBackground(
backgroundPath: string,
script: string
): Promise<string> {
// 1. Upload background
console.log("Uploading background...");
const background = await uploadAsset(backgroundPath, "image/jpeg");
// 2. Create video config
const config = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: background.url,
},
},
],
dimension: { width: 1920, height: 1080 },
};
// 3. Generate video
console.log("Generating video...");
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
});
const { data } = await response.json();
return data.video_id;
}
```
## Asset Limitations
- **File size**: 10MB maximum
- **Image dimensions**: Recommended to match video dimensions
- **Audio duration**: Should match expected video length
- **Retention**: Assets may be deleted after a period of inactivity
## Best Practices
1. **Optimize images** - Resize to match video dimensions before uploading
2. **Use appropriate formats** - JPEG for photos, PNG for graphics with transparency
3. **Validate before upload** - Check file type and size locally first
4. **Handle upload errors** - Implement retry logic for failed uploads
5. **Cache asset IDs** - Reuse assets across multiple video generations
.claude/skills/heygen/references/authentication.md
+232
View File
@@ -0,0 +1,232 @@
---
name: authentication
description: API key setup, X-Api-Key header, and authentication patterns for HeyGen
---
# HeyGen Authentication
All HeyGen API requests require authentication using an API key passed in the `X-Api-Key` header.
## Getting Your API Key
1. Go to https://app.heygen.com/settings?from=&nav=API
2. Log in if prompted
3. Copy your API key
## Environment Setup
Store your API key securely as an environment variable:
```bash
export HEYGEN_API_KEY="your-api-key-here"
```
For `.env` files:
```
HEYGEN_API_KEY=your-api-key-here
```
## Making Authenticated Requests
### curl
```bash
curl -X GET "https://api.heygen.com/v2/avatars" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript/JavaScript (fetch)
```typescript
const response = await fetch("https://api.heygen.com/v2/avatars", {
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
},
});
const { data } = await response.json();
```
### TypeScript/JavaScript (axios)
```typescript
import axios from "axios";
const client = axios.create({
baseURL: "https://api.heygen.com",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY,
},
});
const { data } = await client.get("/v2/avatars");
```
### Python (requests)
```python
import os
import requests
response = requests.get(
"https://api.heygen.com/v2/avatars",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()
```
### Python (httpx)
```python
import os
import httpx
async with httpx.AsyncClient() as client:
response = await client.get(
"https://api.heygen.com/v2/avatars",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()
```
## Creating a Reusable API Client
### TypeScript
```typescript
class HeyGenClient {
private baseUrl = "https://api.heygen.com";
private apiKey: string;
constructor(apiKey: string) {
this.apiKey = apiKey;
}
async request<T>(endpoint: string, options: RequestInit = {}): Promise<T> {
const response = await fetch(`${this.baseUrl}${endpoint}`, {
...options,
headers: {
"X-Api-Key": this.apiKey,
"Content-Type": "application/json",
...options.headers,
},
});
if (!response.ok) {
const error = await response.json();
throw new Error(error.message || `HTTP ${response.status}`);
}
return response.json();
}
get<T>(endpoint: string): Promise<T> {
return this.request<T>(endpoint);
}
post<T>(endpoint: string, body: unknown): Promise<T> {
return this.request<T>(endpoint, {
method: "POST",
body: JSON.stringify(body),
});
}
}
// Usage
const client = new HeyGenClient(process.env.HEYGEN_API_KEY!);
const avatars = await client.get("/v2/avatars");
```
## API Response Format
All HeyGen API responses follow this structure:
```typescript
interface ApiResponse<T> {
error: null | string;
data: T;
}
```
Successful response example:
```json
{
"error": null,
"data": {
"avatars": [...]
}
}
```
Error response example:
```json
{
"error": "Invalid API key",
"data": null
}
```
## Error Handling
Common authentication errors:
| Status Code | Error | Cause |
|-------------|-------|-------|
| 401 | Invalid API key | API key is missing or incorrect |
| 403 | Forbidden | API key doesn't have required permissions |
| 429 | Rate limit exceeded | Too many requests |
### Handling Errors
```typescript
async function makeRequest(endpoint: string) {
const response = await fetch(`https://api.heygen.com${endpoint}`, {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const json = await response.json();
if (!response.ok || json.error) {
throw new Error(json.error || `HTTP ${response.status}`);
}
return json.data;
}
```
## Rate Limiting
HeyGen enforces rate limits on API requests:
- Standard rate limits apply per API key
- Some endpoints (like video generation) have stricter limits
- Use exponential backoff when receiving 429 errors
```typescript
async function requestWithRetry(
fn: () => Promise<Response>,
maxRetries = 3
): Promise<Response> {
for (let i = 0; i < maxRetries; i++) {
const response = await fn();
if (response.status === 429) {
const waitTime = Math.pow(2, i) * 1000;
await new Promise((resolve) => setTimeout(resolve, waitTime));
continue;
}
return response;
}
throw new Error("Max retries exceeded");
}
```
## Security Best Practices
1. **Never expose API keys in client-side code** - Always make API calls from a backend server
2. **Use environment variables** - Don't hardcode API keys in source code
3. **Rotate keys periodically** - Generate new API keys regularly
4. **Monitor usage** - Check your HeyGen dashboard for unusual activity
.claude/skills/heygen/references/avatars.md
+586
View File
@@ -0,0 +1,586 @@
---
name: avatars
description: Listing avatars, avatar styles, and avatar_id selection for HeyGen
---
# HeyGen Avatars
Avatars are the AI-generated presenters in HeyGen videos. You can use public avatars provided by HeyGen or create custom avatars.
## Previewing Avatars Before Generation
Always preview avatars before generating a video to ensure they match user preferences. Each avatar has preview URLs that can be opened directly in the browser - no downloading required.
### List Avatars and Show Previews
```typescript
async function listAndPreviewAvatars(openInBrowser = true): Promise<void> {
const response = await fetch("https://api.heygen.com/v2/avatars", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const { data } = await response.json();
for (const avatar of data.avatars.slice(0, 5)) {
console.log(`\n${avatar.avatar_name} (${avatar.gender})`);
console.log(` ID: ${avatar.avatar_id}`);
console.log(` Preview: ${avatar.preview_image_url}`);
}
// Preview URLs can be opened directly in any browser
for (const avatar of data.avatars.slice(0, 3)) {
console.log(`Open in browser: ${avatar.preview_image_url}`);
}
}
```
### Workflow: Preview Before Generate
1. **List available avatars** - get names, genders, and preview URLs
2. **Show preview URLs to user** - share `preview_image_url` for visual check
3. **User selects** preferred avatar by name or ID
4. **Get avatar details** for `default_voice_id`
5. **Generate video** with selected avatar
### Preview Fields in API Response
| Field | Description |
|-------|-------------|
| `preview_image_url` | Static image of the avatar (JPG) - publicly accessible URL |
| `preview_video_url` | Short video clip showing avatar animation |
Both URLs are publicly accessible - no authentication needed to view.
## Listing Available Avatars
### curl
```bash
curl -X GET "https://api.heygen.com/v2/avatars" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface Avatar {
avatar_id: string;
avatar_name: string;
gender: "male" | "female";
preview_image_url: string;
preview_video_url: string;
}
interface AvatarsResponse {
error: null | string;
data: {
avatars: Avatar[];
talking_photos: TalkingPhoto[];
};
}
async function listAvatars(): Promise<Avatar[]> {
const response = await fetch("https://api.heygen.com/v2/avatars", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const json: AvatarsResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.avatars;
}
```
### Python
```python
import requests
import os
def list_avatars() -> list:
response = requests.get(
"https://api.heygen.com/v2/avatars",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]["avatars"]
```
## Response Format
```json
{
"error": null,
"data": {
"avatars": [
{
"avatar_id": "josh_lite3_20230714",
"avatar_name": "Josh",
"gender": "male",
"preview_image_url": "https://files.heygen.ai/...",
"preview_video_url": "https://files.heygen.ai/..."
},
{
"avatar_id": "angela_expressive_20231010",
"avatar_name": "Angela",
"gender": "female",
"preview_image_url": "https://files.heygen.ai/...",
"preview_video_url": "https://files.heygen.ai/..."
}
],
"talking_photos": []
}
}
```
## Avatar Types
### Public Avatars
HeyGen provides a library of public avatars that anyone can use:
```typescript
// List only public avatars
const avatars = await listAvatars();
const publicAvatars = avatars.filter((a) => !a.avatar_id.startsWith("custom_"));
```
### Private/Custom Avatars
Custom avatars created from your own training footage:
```typescript
const customAvatars = avatars.filter((a) => a.avatar_id.startsWith("custom_"));
```
## Avatar Styles
Avatars support different rendering styles:
| Style | Description |
|-------|-------------|
| `normal` | Full body shot, standard framing |
| `closeUp` | Close-up on face, more expressive |
| `circle` | Avatar in circular frame (talking head) |
| `voice_only` | Audio only, no video rendering |
### When to Use Each Style
| Use Case | Recommended Style |
|----------|-------------------|
| Full-screen presenter video | `normal` |
| Personal/intimate content | `closeUp` |
| Picture-in-picture overlay | `circle` |
| Small corner widget | `circle` |
| Podcast/audio content | `voice_only` |
| Motion graphics with avatar overlay | `normal` or `closeUp` + transparent bg |
### Using Avatar Styles
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal", // "normal" | "closeUp" | "circle" | "voice_only"
},
voice: {
type: "text",
input_text: "Hello, world!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
};
```
### Circle Style for Talking Heads
Circle style is ideal for overlay compositions:
```typescript
// Circle avatar for picture-in-picture
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "circle",
},
voice: { ... },
background: {
type: "color",
value: "#00FF00", // Green for chroma key, or use webm endpoint
},
}
```
## Searching and Filtering Avatars
### By Gender
```typescript
function filterByGender(avatars: Avatar[], gender: "male" | "female"): Avatar[] {
return avatars.filter((a) => a.gender === gender);
}
const maleAvatars = filterByGender(avatars, "male");
const femaleAvatars = filterByGender(avatars, "female");
```
### By Name
```typescript
function searchByName(avatars: Avatar[], query: string): Avatar[] {
const lowerQuery = query.toLowerCase();
return avatars.filter((a) =>
a.avatar_name.toLowerCase().includes(lowerQuery)
);
}
const results = searchByName(avatars, "josh");
```
## Avatar Groups
Avatars are organized into groups for better management.
### List Avatar Groups
```bash
curl -X GET "https://api.heygen.com/v2/avatar_group.list?include_public=true" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
#### Query Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `include_public` | bool | false | Include public avatars in results |
#### TypeScript
```typescript
interface AvatarGroupItem {
id: string;
name: string;
created_at: number;
num_looks: number;
preview_image: string;
group_type: string;
train_status: string;
default_voice_id: string | null;
}
interface AvatarGroupListResponse {
error: null | string;
data: {
avatar_group_list: AvatarGroupItem[];
};
}
async function listAvatarGroups(
includePublic = true
): Promise<AvatarGroupListResponse["data"]> {
const params = new URLSearchParams({
include_public: includePublic.toString(),
});
const response = await fetch(
`https://api.heygen.com/v2/avatar_group.list?${params}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json: AvatarGroupListResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data;
}
```
### Get Avatars in a Group
```bash
curl -X GET "https://api.heygen.com/v2/avatar_group/{group_id}/avatars" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
## Using Avatars in Video Generation
### Basic Avatar Usage
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Welcome to our product demo!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
dimension: { width: 1920, height: 1080 },
};
```
### Multiple Scenes with Different Avatars
```typescript
const multiSceneConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hi, I'm Josh. Let me introduce my colleague.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
{
character: {
type: "avatar",
avatar_id: "angela_expressive_20231010",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello! I'm Angela. Nice to meet you!",
voice_id: "2d5b0e6a8c3f47d9a1b2c3d4e5f60718",
},
},
],
};
```
## Using Avatar's Default Voice
Many avatars have a `default_voice_id` that's pre-matched for natural results. **This is the recommended approach** rather than manually selecting voices.
### Recommended Flow
```
1. GET /v2/avatars → Get list of avatar_ids
2. GET /v2/avatar/{id}/details → Get default_voice_id for chosen avatar
3. POST /v2/video/generate → Use avatar_id + default_voice_id
```
### Get Avatar Details (v2 API)
Given an `avatar_id`, fetch its details including the default voice:
```bash
curl -X GET "https://api.heygen.com/v2/avatar/{avatar_id}/details" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
#### Response Format
```json
{
"error": null,
"data": {
"type": "avatar",
"id": "josh_lite3_20230714",
"name": "Josh",
"gender": "male",
"preview_image_url": "https://files.heygen.ai/...",
"preview_video_url": "https://files.heygen.ai/...",
"premium": false,
"is_public": true,
"default_voice_id": "1bd001e7e50f421d891986aad5158bc8",
"tags": ["AVATAR_IV"]
}
}
```
#### TypeScript
```typescript
interface AvatarDetails {
type: "avatar";
id: string;
name: string;
gender: "male" | "female";
preview_image_url: string;
preview_video_url: string;
premium: boolean;
is_public: boolean;
default_voice_id: string | null;
tags: string[];
}
async function getAvatarDetails(avatarId: string): Promise<AvatarDetails> {
const response = await fetch(
`https://api.heygen.com/v2/avatar/${avatarId}/details`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data;
}
// Usage: Get default voice for a known avatar
const details = await getAvatarDetails("josh_lite3_20230714");
if (details.default_voice_id) {
console.log(`Using ${details.name} with default voice: ${details.default_voice_id}`);
} else {
console.log(`${details.name} has no default voice, select manually`);
}
```
#### Complete Example: Generate Video with Any Avatar's Default Voice
```typescript
async function generateWithAvatarDefaultVoice(
avatarId: string,
script: string
): Promise<string> {
// 1. Get avatar details to find default voice
const avatar = await getAvatarDetails(avatarId);
if (!avatar.default_voice_id) {
throw new Error(`Avatar ${avatar.name} has no default voice`);
}
// 2. Generate video with the avatar's default voice
const videoId = await generateVideo({
video_inputs: [{
character: {
type: "avatar",
avatar_id: avatar.id,
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: avatar.default_voice_id,
},
}],
dimension: { width: 1920, height: 1080 },
});
return videoId;
}
```
### Why Use Default Voice?
1. **Guaranteed gender match** - Avatar and voice are pre-paired
2. **Natural lip sync** - Default voices are optimized for the avatar
3. **Simpler code** - No need to fetch and match voices separately
4. **Better quality** - HeyGen has tested this combination
## Selecting the Right Avatar
### Avatar Categories
HeyGen avatars fall into distinct categories. Match the category to your use case:
| Category | Examples | Best For |
|----------|----------|----------|
| **Business/Professional** | Josh, Angela, Wayne | Corporate videos, product demos, training |
| **Casual/Friendly** | Lily, various lifestyle avatars | Social media, informal content |
| **Themed/Seasonal** | Holiday-themed, costume avatars | Specific campaigns, seasonal content |
| **Expressive** | Avatars with "expressive" in name | Engaging storytelling, dynamic content |
### Selection Guidelines
**For business/professional content:**
- Choose avatars with neutral attire (business casual or formal)
- Avoid themed or seasonal avatars (holiday costumes, casual clothing)
- Preview the avatar to verify professional appearance
- Consider your audience demographics when selecting gender and appearance
**For casual/social content:**
- More flexibility in avatar choice
- Themed avatars can work for specific campaigns
- Match avatar energy to content tone
### Common Mistakes to Avoid
1. **Using themed avatars for business content** - A holiday-themed avatar looks unprofessional in a product demo
2. **Not previewing before generation** - Always check the preview URL to verify appearance
3. **Ignoring avatar style** - A `circle` style avatar may not work for full-screen presentations
4. **Mismatched voice gender** - Always use the avatar's `default_voice_id` or match genders manually
### Selection Checklist
Before generating a video:
- [ ] Previewed avatar image/video in browser
- [ ] Avatar appearance matches content tone (professional vs casual)
- [ ] Avatar style (`normal`, `closeUp`, `circle`) fits the video format
- [ ] Voice gender matches avatar gender
- [ ] Using `default_voice_id` when available
## Helper Functions
### Get Avatar by ID
```typescript
async function getAvatarById(avatarId: string): Promise<Avatar | null> {
const avatars = await listAvatars();
return avatars.find((a) => a.avatar_id === avatarId) || null;
}
```
### Validate Avatar ID
```typescript
async function isValidAvatarId(avatarId: string): Promise<boolean> {
const avatar = await getAvatarById(avatarId);
return avatar !== null;
}
```
### Get Random Avatar
```typescript
async function getRandomAvatar(gender?: "male" | "female"): Promise<Avatar> {
let avatars = await listAvatars();
if (gender) {
avatars = avatars.filter((a) => a.gender === gender);
}
const randomIndex = Math.floor(Math.random() * avatars.length);
return avatars[randomIndex];
}
```
## Common Avatar IDs
Some commonly used public avatar IDs (availability may vary):
| Avatar ID | Name | Gender |
|-----------|------|--------|
| `josh_lite3_20230714` | Josh | Male |
| `angela_expressive_20231010` | Angela | Female |
| `wayne_20240422` | Wayne | Male |
| `lily_20230614` | Lily | Female |
Always verify avatar availability by calling the list endpoint before using.
.claude/skills/heygen/references/backgrounds.md
+295
View File
@@ -0,0 +1,295 @@
---
name: backgrounds
description: Solid colors, images, and video backgrounds for HeyGen videos
---
# Video Backgrounds
HeyGen supports various background types to customize the appearance of your avatar videos.
## Background Types
| Type | Description |
|------|-------------|
| `color` | Solid color background |
| `image` | Static image background |
| `video` | Looping video background |
## Color Backgrounds
The simplest option - use a solid color:
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello with a colored background!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#FFFFFF", // White background
},
},
],
};
```
### Common Color Values
| Color | Hex Value | Use Case |
|-------|-----------|----------|
| White | `#FFFFFF` | Clean, professional |
| Black | `#000000` | Dramatic, cinematic |
| Blue | `#0066CC` | Corporate, trustworthy |
| Green | `#00FF00` | Chroma key (for compositing) |
| Gray | `#808080` | Neutral, modern |
### Using Transparent/Green Screen
For compositing in post-production:
```typescript
background: {
type: "color",
value: "#00FF00", // Green screen
}
```
## Image Backgrounds
Use a static image as background:
### From URL
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Check out this custom background!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: "https://example.com/my-background.jpg",
},
},
],
};
```
### From Uploaded Asset
First upload your image, then use the asset URL:
```typescript
// 1. Upload the image
const assetId = await uploadFile("./background.jpg", "image/jpeg");
// 2. Use in video config
const videoConfig = {
video_inputs: [
{
character: {...},
voice: {...},
background: {
type: "image",
url: `https://files.heygen.ai/asset/${assetId}`,
},
},
],
};
```
### Image Requirements
- **Formats**: JPEG, PNG
- **Recommended size**: Match video dimensions (e.g., 1920x1080 for 1080p)
- **Aspect ratio**: Should match video aspect ratio
- **File size**: Under 10MB recommended
## Video Backgrounds
Use a looping video as background:
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Dynamic video background!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "video",
url: "https://example.com/background-loop.mp4",
},
},
],
};
```
### Video Requirements
- **Format**: MP4 (H.264 codec recommended)
- **Looping**: Video will loop if shorter than avatar content
- **Audio**: Background video audio is typically muted
- **File size**: Under 100MB recommended
## Different Backgrounds Per Scene
Use different backgrounds for each scene:
```typescript
const multiBackgroundConfig = {
video_inputs: [
// Scene 1: Office background
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Let me start with an introduction.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: "https://example.com/office-bg.jpg",
},
},
// Scene 2: Product showcase
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "closeUp",
},
voice: {
type: "text",
input_text: "Now let me show you our product.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: "https://example.com/product-bg.jpg",
},
},
// Scene 3: Call to action
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Get started today!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#1a1a2e",
},
},
],
};
```
## Background Helper Functions
### TypeScript
```typescript
type BackgroundType = "color" | "image" | "video";
interface Background {
type: BackgroundType;
value?: string;
url?: string;
}
function createColorBackground(hexColor: string): Background {
return { type: "color", value: hexColor };
}
function createImageBackground(imageUrl: string): Background {
return { type: "image", url: imageUrl };
}
function createVideoBackground(videoUrl: string): Background {
return { type: "video", url: videoUrl };
}
// Preset backgrounds
const backgrounds = {
white: createColorBackground("#FFFFFF"),
black: createColorBackground("#000000"),
greenScreen: createColorBackground("#00FF00"),
corporate: createColorBackground("#0066CC"),
};
```
## Best Practices
1. **Match dimensions** - Background should match video dimensions
2. **Consider avatar position** - Leave space where avatar will appear
3. **Use contrasting colors** - Ensure avatar is visible against background
4. **Optimize file sizes** - Compress images/videos for faster processing
5. **Test with green screen** - For professional post-production workflows
6. **Keep backgrounds simple** - Avoid distracting elements behind the avatar
## Common Issues
### Background Not Showing
```typescript
// Wrong: missing url/value
background: {
type: "image"
}
// Correct
background: {
type: "image",
url: "https://example.com/bg.jpg"
}
```
### Aspect Ratio Mismatch
If your background doesn't match the video dimensions, it may be cropped or stretched. Always match your background aspect ratio to your video dimensions:
```typescript
// For 1920x1080 video
// Use 1920x1080 background image
// For 1080x1920 portrait video
// Use 1080x1920 background image
```
### Video Background Audio
Background video audio is typically muted to avoid conflicting with the avatar's voice. If you need background music, add it as a separate audio track in post-production.
.claude/skills/heygen/references/captions.md
+277
View File
@@ -0,0 +1,277 @@
---
name: captions
description: Auto-generated captions and subtitle options for HeyGen videos
---
# Video Captions
HeyGen can automatically generate captions (subtitles) for your videos, improving accessibility and engagement.
## Enabling Captions
Captions can be enabled when generating a video:
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello! This video will have automatic captions.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
// Caption settings (availability varies by plan)
caption: true,
};
```
## Caption Configuration Options
```typescript
interface CaptionConfig {
// Enable/disable captions
enabled: boolean;
// Caption style
style?: {
font_family?: string;
font_size?: number;
font_color?: string;
background_color?: string;
position?: "top" | "bottom";
};
// Language for caption generation
language?: string;
}
```
## Caption Styles
### Basic Captions
```typescript
const config = {
video_inputs: [...],
caption: true, // Enable with default styling
};
```
### Styled Captions
```typescript
const config = {
video_inputs: [...],
caption: {
enabled: true,
style: {
font_family: "Arial",
font_size: 32,
font_color: "#FFFFFF",
background_color: "rgba(0, 0, 0, 0.7)",
position: "bottom",
},
},
};
```
## Multi-Language Captions
For videos in different languages, captions are generated based on the voice language:
```typescript
// Spanish video with Spanish captions
const spanishConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "¡Hola! Este video tendrá subtítulos en español.",
voice_id: "spanish_voice_id",
},
},
],
caption: true,
};
```
## Working with SRT Files
### SRT File Format
Standard SRT format:
```srt
1
00:00:00,000 --> 00:00:03,000
Hello! This video will have
2
00:00:03,000 --> 00:00:06,000
automatic captions generated.
3
00:00:06,000 --> 00:00:09,000
They sync with the audio.
```
### Using Custom SRT
For video translation, you can provide your own SRT:
```typescript
const translationConfig = {
input_video_id: "original_video_id",
output_languages: ["es-ES", "fr-FR"],
srt_key: "path/to/custom.srt", // Custom SRT file
srt_role: "input", // "input" or "output"
};
```
## Caption Positioning
### Bottom (Default)
Standard position for most videos:
```typescript
caption: {
enabled: true,
style: {
position: "bottom"
}
}
```
### Top
For videos where bottom space is occupied:
```typescript
caption: {
enabled: true,
style: {
position: "top"
}
}
```
## Accessibility Best Practices
1. **Always enable captions** - Improves accessibility for deaf/hard-of-hearing viewers
2. **Use high contrast** - White text on dark background or vice versa
3. **Readable font size** - At least 24px for standard video, larger for mobile
4. **Don't cover important content** - Position captions away from key visual elements
5. **Sync timing** - Ensure captions match audio timing accurately
## Caption Helper Functions
```typescript
interface CaptionStyle {
font_family: string;
font_size: number;
font_color: string;
background_color: string;
position: "top" | "bottom";
}
const captionPresets: Record<string, CaptionStyle> = {
default: {
font_family: "Arial",
font_size: 32,
font_color: "#FFFFFF",
background_color: "rgba(0, 0, 0, 0.7)",
position: "bottom",
},
minimal: {
font_family: "Arial",
font_size: 28,
font_color: "#FFFFFF",
background_color: "transparent",
position: "bottom",
},
bold: {
font_family: "Arial",
font_size: 36,
font_color: "#FFFFFF",
background_color: "rgba(0, 0, 0, 0.9)",
position: "bottom",
},
branded: {
font_family: "Roboto",
font_size: 30,
font_color: "#00D1FF",
background_color: "rgba(26, 26, 46, 0.9)",
position: "bottom",
},
};
function createCaptionConfig(preset: keyof typeof captionPresets) {
return {
enabled: true,
style: captionPresets[preset],
};
}
```
## Social Media Caption Considerations
### TikTok / Instagram Reels
- Position captions in center or upper portion
- Avoid bottom 20% (covered by UI elements)
- Use larger font sizes for mobile viewing
```typescript
const socialCaptions = {
enabled: true,
style: {
font_size: 42,
position: "top", // Avoid bottom UI elements
},
};
```
### YouTube
- Standard bottom captions work well
- YouTube also supports closed captions upload
### LinkedIn
- Captions highly recommended (many watch without sound)
- Professional styling preferred
## Limitations
- Caption styles may be limited depending on your subscription tier
- Some advanced caption features may require the web interface
- Multi-speaker caption detection may have limited availability
- Caption accuracy depends on audio quality and speech clarity
## Integration with Video Translation
When using video translation, captions are automatically handled:
```typescript
// Video translation includes caption generation
const translationConfig = {
input_video_id: "original_video_id",
output_languages: ["es-ES"],
// Captions generated in target language
};
```
See [video-translation.md](video-translation.md) for more details.
.claude/skills/heygen/references/dimensions.md
+314
View File
@@ -0,0 +1,314 @@
---
name: dimensions
description: Resolution options (720p/1080p) and aspect ratios for HeyGen videos
---
# Video Dimensions and Resolution
HeyGen supports various video dimensions and aspect ratios to fit different platforms and use cases.
## Standard Resolutions
### Landscape (16:9)
| Resolution | Width | Height | Use Case |
|------------|-------|--------|----------|
| 720p | 1280 | 720 | Standard quality, faster processing |
| 1080p | 1920 | 1080 | High quality, most common |
### Portrait (9:16)
| Resolution | Width | Height | Use Case |
|------------|-------|--------|----------|
| 720p | 720 | 1280 | Mobile-first content |
| 1080p | 1080 | 1920 | High quality vertical |
### Square (1:1)
| Resolution | Width | Height | Use Case |
|------------|-------|--------|----------|
| 720p | 720 | 720 | Social media posts |
| 1080p | 1080 | 1080 | High quality square |
## Setting Dimensions
### TypeScript
```typescript
// Landscape 1080p
const landscapeConfig = {
video_inputs: [...],
dimension: {
width: 1920,
height: 1080
}
};
// Portrait 1080p
const portraitConfig = {
video_inputs: [...],
dimension: {
width: 1080,
height: 1920
}
};
// Square 1080p
const squareConfig = {
video_inputs: [...],
dimension: {
width: 1080,
height: 1080
}
};
```
### curl
```bash
# Landscape 1080p
curl -X POST "https://api.heygen.com/v2/video/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"video_inputs": [...],
"dimension": {
"width": 1920,
"height": 1080
}
}'
```
## Dimension Helper Functions
```typescript
type AspectRatio = "16:9" | "9:16" | "1:1" | "4:3" | "4:5";
type Quality = "720p" | "1080p";
interface Dimensions {
width: number;
height: number;
}
function getDimensions(aspectRatio: AspectRatio, quality: Quality): Dimensions {
const configs: Record<AspectRatio, Record<Quality, Dimensions>> = {
"16:9": {
"720p": { width: 1280, height: 720 },
"1080p": { width: 1920, height: 1080 },
},
"9:16": {
"720p": { width: 720, height: 1280 },
"1080p": { width: 1080, height: 1920 },
},
"1:1": {
"720p": { width: 720, height: 720 },
"1080p": { width: 1080, height: 1080 },
},
"4:3": {
"720p": { width: 960, height: 720 },
"1080p": { width: 1440, height: 1080 },
},
"4:5": {
"720p": { width: 576, height: 720 },
"1080p": { width: 864, height: 1080 },
},
};
return configs[aspectRatio][quality];
}
// Usage
const youTubeDimensions = getDimensions("16:9", "1080p");
const tikTokDimensions = getDimensions("9:16", "1080p");
const instagramDimensions = getDimensions("1:1", "1080p");
```
## Platform-Specific Recommendations
### YouTube
```typescript
const youtubeConfig = {
video_inputs: [...],
dimension: { width: 1920, height: 1080 }, // 16:9 landscape
};
```
### TikTok / Instagram Reels / YouTube Shorts
```typescript
const shortFormConfig = {
video_inputs: [...],
dimension: { width: 1080, height: 1920 }, // 9:16 portrait
};
```
### Instagram Feed Post
```typescript
const instagramFeedConfig = {
video_inputs: [...],
dimension: { width: 1080, height: 1080 }, // 1:1 square
};
```
### LinkedIn
```typescript
const linkedinConfig = {
video_inputs: [...],
dimension: { width: 1920, height: 1080 }, // 16:9 landscape preferred
};
```
### Twitter/X
```typescript
const twitterConfig = {
video_inputs: [...],
dimension: { width: 1280, height: 720 }, // 16:9, 720p is common
};
```
## Avatar IV Dimensions
For Avatar IV (photo-based avatars), dimensions are set via orientation:
```typescript
type VideoOrientation = "portrait" | "landscape" | "square";
function getAvatarIVDimensions(orientation: VideoOrientation): Dimensions {
switch (orientation) {
case "portrait":
return { width: 720, height: 1280 };
case "landscape":
return { width: 1280, height: 720 };
case "square":
return { width: 720, height: 720 };
}
}
```
## Custom Dimensions
HeyGen supports custom dimensions within limits:
```typescript
const customConfig = {
video_inputs: [...],
dimension: {
width: 1600,
height: 900 // Custom 16:9 at non-standard resolution
}
};
```
### Dimension Constraints
- **Minimum**: 128px on any side
- **Maximum**: 4096px on any side
- **Must be even numbers**: Both width and height must be divisible by 2
```typescript
function validateDimensions(width: number, height: number): boolean {
if (width < 128 || height < 128) {
throw new Error("Dimensions must be at least 128px");
}
if (width > 4096 || height > 4096) {
throw new Error("Dimensions cannot exceed 4096px");
}
if (width % 2 !== 0 || height % 2 !== 0) {
throw new Error("Dimensions must be even numbers");
}
return true;
}
```
## Resolution vs. Credit Cost
Higher resolutions may consume more credits:
| Resolution | Relative Cost |
|------------|---------------|
| 720p | Base rate |
| 1080p | ~1.5x base rate |
Consider using 720p for drafts and testing, then 1080p for final output.
## Background Considerations
Match background image/video dimensions to your video dimensions:
```typescript
// For 1080p landscape video
const config = {
video_inputs: [
{
character: {...},
voice: {...},
background: {
type: "image",
url: "https://example.com/1920x1080-background.jpg" // Match video dimensions
}
}
],
dimension: { width: 1920, height: 1080 }
};
```
## Creating a Video Config Factory
```typescript
interface VideoConfigOptions {
script: string;
avatarId: string;
voiceId: string;
platform: "youtube" | "tiktok" | "instagram_feed" | "instagram_story" | "linkedin";
quality?: "720p" | "1080p";
}
function createVideoConfig(options: VideoConfigOptions) {
const platformDimensions: Record<string, Dimensions> = {
youtube: { width: 1920, height: 1080 },
tiktok: { width: 1080, height: 1920 },
instagram_feed: { width: 1080, height: 1080 },
instagram_story: { width: 1080, height: 1920 },
linkedin: { width: 1920, height: 1080 },
};
const dimension = platformDimensions[options.platform];
// Scale down for 720p if requested
if (options.quality === "720p") {
dimension.width = Math.round((dimension.width * 720) / 1080);
dimension.height = Math.round((dimension.height * 720) / 1080);
}
return {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: options.avatarId,
avatar_style: "normal",
},
voice: {
type: "text",
input_text: options.script,
voice_id: options.voiceId,
},
},
],
dimension,
};
}
// Usage
const tiktokVideo = createVideoConfig({
script: "Hey everyone! Check this out!",
avatarId: "josh_lite3_20230714",
voiceId: "1bd001e7e50f421d891986aad5158bc8",
platform: "tiktok",
quality: "1080p",
});
```
.claude/skills/heygen/references/photo-avatars.md
+853
View File
@@ -0,0 +1,853 @@
---
name: photo-avatars
description: Creating avatars from photos (talking photos) for HeyGen
---
# Photo Avatars (Talking Photos)
Photo avatars allow you to animate a static photo and make it speak. This is useful for creating personalized video content from portraits, headshots, or any suitable image.
## Creating a Photo Avatar from an Uploaded Image
The workflow is: **Upload Image → Create Avatar Group → Use in Video**
### Step 1: Upload the Image
Upload a portrait photo using the asset upload endpoint. The response includes an `image_key` which you'll use in the next step.
```bash
curl -X POST "https://upload.heygen.com/v1/asset" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: image/jpeg" \
--data-binary '@./portrait.jpg'
```
Response:
```json
{
"code": 100,
"data": {
"id": "741299e941764988b432ed3a6757878f",
"name": "741299e941764988b432ed3a6757878f",
"file_type": "image",
"url": "https://resource2.heygen.ai/image/.../original.jpg",
"image_key": "image/741299e941764988b432ed3a6757878f/original.jpg"
}
}
```
> **Important:** Save the `image_key` field (not the `id`). The `image_key` is the S3 path used to create the photo avatar.
See [assets.md](assets.md) for full upload details.
### Step 2: Create Photo Avatar Group
Use the `image_key` from the upload response to create a photo avatar group. This processes the image and creates a usable photo avatar.
**Endpoint:** `POST https://api.heygen.com/v2/photo_avatar/avatar_group/create`
```bash
curl -X POST "https://api.heygen.com/v2/photo_avatar/avatar_group/create" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"image_key": "image/741299e941764988b432ed3a6757878f/original.jpg",
"name": "My Photo Avatar"
}'
```
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `image_key` | string | ✓ | S3 image key from upload response |
| `name` | string | ✓ | Display name for the avatar |
| `generation_id` | string | | If using AI-generated photo (see below) |
Response:
```json
{
"error": null,
"data": {
"id": "045c260bc0364727b2cbe50442c3a5bf",
"image_url": "https://files2.heygen.ai/...",
"created_at": 1771798135.777256,
"name": "My Photo Avatar",
"status": "pending",
"group_id": "045c260bc0364727b2cbe50442c3a5bf",
"is_motion": false,
"business_type": "uploaded"
}
}
```
The `id` (same as `group_id`) is your `talking_photo_id` for video generation.
### Step 3: Wait for Processing
The photo avatar starts with `status: "pending"` and transitions to `"completed"` within seconds. Poll the status endpoint:
**Endpoint:** `GET https://api.heygen.com/v2/photo_avatar/{id}`
```bash
curl "https://api.heygen.com/v2/photo_avatar/045c260bc0364727b2cbe50442c3a5bf" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
Wait until `status` is `"completed"` before using in video generation.
### Step 4: Use in Video Generation
Use the photo avatar `id` as `talking_photo_id`:
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "talking_photo",
talking_photo_id: "045c260bc0364727b2cbe50442c3a5bf",
},
voice: {
type: "text",
input_text: "Hello! This is my photo avatar speaking.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
dimension: { width: 1920, height: 1080 },
};
```
## TypeScript: Complete Workflow
```typescript
import fs from "fs";
import path from "path";
interface AssetUploadResponse {
code: number;
data: {
id: string;
image_key: string;
url: string;
};
}
interface PhotoAvatarResponse {
error: string | null;
data: {
id: string;
group_id: string;
image_url: string;
name: string;
status: string;
is_motion: boolean;
business_type: string;
};
}
async function createPhotoAvatar(
imagePath: string,
name: string
): Promise<string> {
// 1. Upload image
const resolvedPath = path.resolve(imagePath);
const fileBuffer = fs.readFileSync(resolvedPath);
const uploadResponse = await fetch("https://upload.heygen.com/v1/asset", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "image/jpeg",
},
body: fileBuffer,
});
const uploadJson: AssetUploadResponse = await uploadResponse.json();
if (uploadJson.code !== 100) {
throw new Error("Upload failed");
}
const imageKey = uploadJson.data.image_key;
// 2. Create avatar group
const createResponse = await fetch(
"https://api.heygen.com/v2/photo_avatar/avatar_group/create",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({ image_key: imageKey, name }),
}
);
const createJson: PhotoAvatarResponse = await createResponse.json();
if (createJson.error) {
throw new Error(createJson.error);
}
const photoAvatarId = createJson.data.id;
// 3. Wait for processing
await waitForPhotoAvatar(photoAvatarId);
return photoAvatarId;
}
async function waitForPhotoAvatar(id: string): Promise<void> {
for (let i = 0; i < 30; i++) {
const response = await fetch(
`https://api.heygen.com/v2/photo_avatar/${id}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json: PhotoAvatarResponse = await response.json();
if (json.data.status === "completed") return;
if (json.data.status === "failed") {
throw new Error("Photo avatar processing failed");
}
await new Promise((r) => setTimeout(r, 2000));
}
throw new Error("Photo avatar processing timed out");
}
async function createVideoFromPhoto(
photoPath: string,
script: string,
voiceId: string
): Promise<string> {
// 1. Create photo avatar
const talkingPhotoId = await createPhotoAvatar(photoPath, "Video Avatar");
// 2. Generate video
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
video_inputs: [
{
character: {
type: "talking_photo",
talking_photo_id: talkingPhotoId,
},
voice: {
type: "text",
input_text: script,
voice_id: voiceId,
},
},
],
dimension: { width: 1920, height: 1080 },
}),
});
const { data } = await response.json();
return data.video_id;
}
```
## Python: Complete Workflow
```python
import requests
import os
import time
def create_photo_avatar(image_path: str, name: str) -> str:
api_key = os.environ["HEYGEN_API_KEY"]
# 1. Upload image
with open(image_path, "rb") as f:
upload_resp = requests.post(
"https://upload.heygen.com/v1/asset",
headers={
"X-Api-Key": api_key,
"Content-Type": "image/jpeg",
},
data=f,
)
upload_data = upload_resp.json()
if upload_data.get("code") != 100:
raise Exception("Upload failed")
image_key = upload_data["data"]["image_key"]
# 2. Create avatar group
create_resp = requests.post(
"https://api.heygen.com/v2/photo_avatar/avatar_group/create",
headers={
"X-Api-Key": api_key,
"Content-Type": "application/json",
},
json={"image_key": image_key, "name": name},
)
create_data = create_resp.json()
if create_data.get("error"):
raise Exception(create_data["error"])
photo_avatar_id = create_data["data"]["id"]
# 3. Wait for processing
for _ in range(30):
status_resp = requests.get(
f"https://api.heygen.com/v2/photo_avatar/{photo_avatar_id}",
headers={"X-Api-Key": api_key},
)
status = status_resp.json()["data"]["status"]
if status == "completed":
return photo_avatar_id
if status == "failed":
raise Exception("Photo avatar processing failed")
time.sleep(2)
raise Exception("Photo avatar processing timed out")
```
## Listing Existing Talking Photos
Retrieve all talking photos in your account:
**Endpoint:** `GET https://api.heygen.com/v1/talking_photo.list`
```bash
curl "https://api.heygen.com/v1/talking_photo.list" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
Response:
```json
{
"code": 100,
"data": [
{
"id": "ef0ed70f72c6497793e5e36e434d2aea",
"image_url": "https://files2.heygen.ai/talking_photo/.../image.WEBP",
"circle_image": ""
}
]
}
```
Each `id` can be used as `talking_photo_id` in video generation.
## Adding Photos to an Existing Group
Add additional photo looks to an existing avatar group:
**Endpoint:** `POST https://api.heygen.com/v2/photo_avatar/avatar_group/add`
```typescript
async function addPhotosToGroup(
groupId: string,
imageKeys: string[],
name: string
): Promise<void> {
const response = await fetch(
"https://api.heygen.com/v2/photo_avatar/avatar_group/add",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
group_id: groupId,
image_keys: imageKeys,
name,
}),
}
);
const json = await response.json();
if (json.error) {
throw new Error(json.error);
}
}
```
## Training a Photo Avatar Group
Train the avatar group for improved animation quality:
**Endpoint:** `POST https://api.heygen.com/v2/photo_avatar/train`
```bash
curl -X POST "https://api.heygen.com/v2/photo_avatar/train" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{"group_id": "045c260bc0364727b2cbe50442c3a5bf"}'
```
Check training status:
**Endpoint:** `GET https://api.heygen.com/v2/photo_avatar/train/status/{group_id}`
## Avatar IV Video Generation
Avatar IV is HeyGen's latest photo avatar technology with improved quality and natural motion. It generates a video directly from an uploaded image, bypassing the avatar group creation step.
**Endpoint:** `POST https://api.heygen.com/v2/video/av4/generate`
```bash
curl -X POST "https://api.heygen.com/v2/video/av4/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"image_key": "image/741299e941764988b432ed3a6757878f/original.jpg",
"script": "Hello! This is Avatar IV with enhanced quality.",
"voice_id": "1bd001e7e50f421d891986aad5158bc8",
"video_orientation": "landscape",
"video_title": "My Avatar IV Video"
}'
```
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `image_key` | string | ✓ | S3 image key from asset upload |
| `script` | string | ✓ | Text for the avatar to speak |
| `voice_id` | string | ✓ | Voice to use |
| `video_orientation` | string | | `"portrait"`, `"landscape"`, or `"square"` |
| `video_title` | string | | Title for the video |
| `fit` | string | | `"cover"` or `"contain"` |
| `custom_motion_prompt` | string | | Motion/expression description |
| `enhance_custom_motion_prompt` | boolean | | Enhance the motion prompt with AI |
### TypeScript
```typescript
interface AvatarIVRequest {
image_key: string;
script: string;
voice_id: string;
video_orientation?: "portrait" | "landscape" | "square";
video_title?: string;
fit?: "cover" | "contain";
custom_motion_prompt?: string;
enhance_custom_motion_prompt?: boolean;
}
interface AvatarIVResponse {
error: null | string;
data: {
video_id: string;
};
}
async function generateAvatarIVVideo(
config: AvatarIVRequest
): Promise<string> {
const response = await fetch(
"https://api.heygen.com/v2/video/av4/generate",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
}
);
const json: AvatarIVResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.video_id;
}
```
### Avatar IV Options
| Orientation | Dimensions | Use Case |
|-------------|------------|----------|
| `portrait` | 720x1280 | TikTok, Stories |
| `landscape` | 1280x720 | YouTube, Web |
| `square` | 720x720 | Instagram Feed |
| Fit | Description |
|-----|-------------|
| `cover` | Fill the frame, may crop edges |
| `contain` | Fit entire image, may show background |
### Custom Motion Prompts
```typescript
const videoId = await generateAvatarIVVideo({
image_key: "image/.../original.jpg",
script: "Let me tell you about our product.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
custom_motion_prompt: "nodding head and smiling",
enhance_custom_motion_prompt: true,
});
```
## Generating AI Photo Avatars
Generate synthetic photo avatars from text descriptions instead of uploading a photo.
**Endpoint:** `POST https://api.heygen.com/v2/photo_avatar/photo/generate`
> **IMPORTANT: All 8 fields are REQUIRED.** The API will reject requests missing any field.
> When a user asks to "generate an AI avatar of a professional man", you need to ask for or select values for ALL fields below.
### Required Fields (ALL must be provided)
| Field | Type | Allowed Values |
|-------|------|----------------|
| `name` | string | Name for the generated avatar |
| `age` | enum | `"Young Adult"`, `"Early Middle Age"`, `"Late Middle Age"`, `"Senior"`, `"Unspecified"` |
| `gender` | enum | `"Woman"`, `"Man"`, `"Unspecified"` |
| `ethnicity` | enum | `"White"`, `"Black"`, `"Asian American"`, `"East Asian"`, `"South East Asian"`, `"South Asian"`, `"Middle Eastern"`, `"Pacific"`, `"Hispanic"`, `"Unspecified"` |
| `orientation` | enum | `"square"`, `"horizontal"`, `"vertical"` |
| `pose` | enum | `"half_body"`, `"close_up"`, `"full_body"` |
| `style` | enum | `"Realistic"`, `"Pixar"`, `"Cinematic"`, `"Vintage"`, `"Noir"`, `"Cyberpunk"`, `"Unspecified"` |
| `appearance` | string | Text prompt describing appearance (clothing, mood, lighting, etc). Max 1000 chars |
### curl Example
```bash
curl -X POST "https://api.heygen.com/v2/photo_avatar/photo/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Sarah Product Demo",
"age": "Young Adult",
"gender": "Woman",
"ethnicity": "White",
"orientation": "horizontal",
"pose": "half_body",
"style": "Realistic",
"appearance": "Professional woman with a friendly smile, wearing a navy blue blazer over a white blouse, soft studio lighting, clean neutral background"
}'
```
Response:
```json
{
"error": null,
"data": {
"generation_id": "6a7f7f2795de4599bec7cf1e06babe30"
}
}
```
### Check Generation Status
**Endpoint:** `GET https://api.heygen.com/v2/photo_avatar/generation/{generation_id}`
The response includes multiple generated images to choose from:
```json
{
"error": null,
"data": {
"id": "6a7f7f2795de4599bec7cf1e06babe30",
"status": "success",
"image_url_list": [
"https://resource2.heygen.ai/photo_generation/.../image1.jpg",
"https://resource2.heygen.ai/photo_generation/.../image2.jpg",
"https://resource2.heygen.ai/photo_generation/.../image3.jpg",
"https://resource2.heygen.ai/photo_generation/.../image4.jpg"
],
"image_key_list": [
"photo_generation/.../image1.jpg",
"photo_generation/.../image2.jpg",
"photo_generation/.../image3.jpg",
"photo_generation/.../image4.jpg"
]
}
}
```
### TypeScript
```typescript
interface GeneratePhotoAvatarRequest {
name: string;
age: "Young Adult" | "Early Middle Age" | "Late Middle Age" | "Senior" | "Unspecified";
gender: "Woman" | "Man" | "Unspecified";
ethnicity: "White" | "Black" | "Asian American" | "East Asian" | "South East Asian" | "South Asian" | "Middle Eastern" | "Pacific" | "Hispanic" | "Unspecified";
orientation: "square" | "horizontal" | "vertical";
pose: "half_body" | "close_up" | "full_body";
style: "Realistic" | "Pixar" | "Cinematic" | "Vintage" | "Noir" | "Cyberpunk" | "Unspecified";
appearance: string;
}
interface GeneratePhotoAvatarResponse {
error: string | null;
data: {
generation_id: string;
};
}
interface PhotoGenerationStatus {
error: string | null;
data: {
id: string;
status: "pending" | "processing" | "success" | "failed";
msg: string | null;
image_url_list?: string[];
image_key_list?: string[];
};
}
async function generatePhotoAvatar(
config: GeneratePhotoAvatarRequest
): Promise<string> {
const response = await fetch(
"https://api.heygen.com/v2/photo_avatar/photo/generate",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
}
);
const json: GeneratePhotoAvatarResponse = await response.json();
if (json.error) {
throw new Error(`Photo avatar generation failed: ${json.error}`);
}
return json.data.generation_id;
}
async function waitForPhotoGeneration(
generationId: string
): Promise<string[]> {
for (let i = 0; i < 60; i++) {
const response = await fetch(
`https://api.heygen.com/v2/photo_avatar/generation/${generationId}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json: PhotoGenerationStatus = await response.json();
if (json.error) throw new Error(json.error);
if (json.data.status === "success") {
return json.data.image_key_list!;
}
if (json.data.status === "failed") {
throw new Error(json.data.msg ?? "Photo generation failed");
}
await new Promise((r) => setTimeout(r, 5000));
}
throw new Error("Photo generation timed out");
}
```
### AI Photo → Avatar Group → Video
Use a generated AI photo to create an avatar group, then generate a video:
```typescript
// 1. Generate AI photo
const generationId = await generatePhotoAvatar({
name: "Product Demo Host",
age: "Young Adult",
gender: "Woman",
ethnicity: "Unspecified",
orientation: "horizontal",
pose: "half_body",
style: "Realistic",
appearance: "Professional woman, navy blazer, friendly smile, soft lighting",
});
// 2. Wait for generation and pick first result
const imageKeys = await waitForPhotoGeneration(generationId);
const selectedImageKey = imageKeys[0];
// 3. Create avatar group from the AI photo
const createResponse = await fetch(
"https://api.heygen.com/v2/photo_avatar/avatar_group/create",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
image_key: selectedImageKey,
name: "Product Demo Host",
generation_id: generationId,
}),
}
);
const { data } = await createResponse.json();
const talkingPhotoId = data.id;
// 4. Generate video (after status is "completed")
const videoId = await generateVideo({
video_inputs: [{
character: {
type: "talking_photo",
talking_photo_id: talkingPhotoId,
},
voice: {
type: "text",
input_text: "Welcome to our product demo!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
}],
dimension: { width: 1920, height: 1080 },
});
```
### Pre-Generation Checklist
Before calling the AI generation API, ensure you have values for ALL fields:
| # | Field | Question to Ask / Default |
|---|-------|---------------------------|
| 1 | `name` | What should we call this avatar? |
| 2 | `age` | Young Adult / Early Middle Age / Late Middle Age / Senior? |
| 3 | `gender` | Woman / Man? |
| 4 | `ethnicity` | Which ethnicity? (see enum values above) |
| 5 | `orientation` | horizontal (landscape) / vertical (portrait) / square? |
| 6 | `pose` | half_body (recommended) / close_up / full_body? |
| 7 | `style` | Realistic (recommended) / Cinematic / other? |
| 8 | `appearance` | Describe clothing, expression, lighting, background |
**If the user only provides a vague request** like "create a professional looking man", ask them to specify the missing fields OR make reasonable defaults (e.g., "Early Middle Age", "Realistic" style, "half_body" pose, "horizontal" orientation).
### Appearance Prompt Tips
The `appearance` field is a text prompt - be descriptive:
**Good prompts:**
- "Professional woman with shoulder-length brown hair, wearing a light blue button-down shirt, warm friendly smile, soft studio lighting, clean white background"
- "Young man with short black hair, casual tech startup style, wearing a dark hoodie, confident expression, modern office background with plants"
**Avoid:**
- Vague descriptions: "a nice person"
- Conflicting attributes
- Requesting specific real people
## Managing Photo Avatars
### Get Photo Avatar Details
**Endpoint:** `GET https://api.heygen.com/v2/photo_avatar/{id}`
```typescript
async function getPhotoAvatar(id: string): Promise<PhotoAvatarResponse> {
const response = await fetch(
`https://api.heygen.com/v2/photo_avatar/${id}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
return response.json();
}
```
### Delete Photo Avatar
**Endpoint:** `DELETE https://api.heygen.com/v2/photo_avatar/{id}`
```typescript
async function deletePhotoAvatar(id: string): Promise<void> {
const response = await fetch(
`https://api.heygen.com/v2/photo_avatar/${id}`,
{
method: "DELETE",
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
}
);
if (!response.ok) {
throw new Error("Failed to delete photo avatar");
}
}
```
### Delete Photo Avatar Group
**Endpoint:** `DELETE https://api.heygen.com/v2/photo_avatar_group/{group_id}`
```typescript
async function deletePhotoAvatarGroup(groupId: string): Promise<void> {
const response = await fetch(
`https://api.heygen.com/v2/photo_avatar_group/${groupId}`,
{
method: "DELETE",
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
}
);
if (!response.ok) {
throw new Error("Failed to delete photo avatar group");
}
}
```
## API Reference
| Endpoint | Method | Description |
|----------|--------|-------------|
| `upload.heygen.com/v1/asset` | POST | Upload image (returns `image_key`) |
| `/v2/photo_avatar/avatar_group/create` | POST | Create photo avatar from `image_key` |
| `/v2/photo_avatar/avatar_group/add` | POST | Add photos to existing group |
| `/v2/photo_avatar/train` | POST | Train avatar group |
| `/v2/photo_avatar/train/status/{group_id}` | GET | Check training status |
| `/v2/photo_avatar/{id}` | GET | Get photo avatar details/status |
| `/v2/photo_avatar/{id}` | DELETE | Delete photo avatar |
| `/v2/photo_avatar_group/{id}` | DELETE | Delete avatar group |
| `/v2/photo_avatar/photo/generate` | POST | Generate AI photo from text |
| `/v2/photo_avatar/generation/{id}` | GET | Check AI generation status |
| `/v2/video/av4/generate` | POST | Avatar IV video from `image_key` |
| `/v1/talking_photo.list` | GET | List all existing talking photos |
| `/v2/video/generate` | POST | Generate video with `talking_photo_id` |
## Photo Requirements
### Technical Requirements
| Aspect | Requirement |
|--------|-------------|
| Format | JPEG, PNG |
| Resolution | Minimum 512x512px |
| File size | Under 10MB |
| Face visibility | Clear, front-facing |
### Quality Guidelines
1. **Lighting** - Even, natural lighting on face
2. **Expression** - Neutral or slight smile
3. **Background** - Simple, uncluttered
4. **Face position** - Centered, not cut off
5. **Clarity** - Sharp, in focus
6. **Angle** - Straight-on or slight angle
## Best Practices
1. **Use high-quality photos** - Better input = better output
2. **Front-facing portraits** - Work best for animation
3. **Neutral expressions** - Allow for more natural animation
4. **Use Avatar IV for best quality** - Latest generation technology
5. **Train avatar groups** - Improves animation quality
6. **Reuse photo avatar IDs** - Once created, use the same `talking_photo_id` across multiple videos
## Limitations
- Photo quality significantly affects output
- Side-profile photos have limited support
- Full-body photos may not animate properly
- Some expressions may look unnatural
- Processing time varies by complexity
.claude/skills/heygen/references/prompt-examples.md
+206
View File
@@ -0,0 +1,206 @@
---
name: prompt-examples
description: Full production prompt examples and ready-to-use templates for Video Agent
---
# Video Agent Prompt Examples
## Full Example: Brief to Production Prompt
### Input Brief
```
Topic: Monthly company report for a SaaS startup
Key data: $141M ARR (up from $54M), 1.85M signups (+28%), 3M paid videos/month
Customer story: Creator built AI character, 2.5M followers, 20 min/video
Challenge: Organic traffic volatile, -16% last week
Duration: ~90 seconds
Tone: Confident CEO, data-backed
```
### Output Prompt
```
FORMAT: Bloomberg-style company report. 90 seconds. Fast-paced, data-dense.
Record-breaking month. Proud but analytical.
TONE: Confident, direct, data-backed. Highlights hit hard with numbers.
Customer stories are the emotional core. Challenges are honest — no spin.
AVATAR: Man in simple black crew-neck tee, standing in a modern glass-walled
office at golden hour. Behind him, a wall-mounted display shows the company logo
in soft blue glow. Monitor to his right shows a dashboard with upward-trending
charts. Desk beside him: laptop, half-empty flat white, scattered sticky notes.
Warm afternoon light through floor-to-ceiling windows, long shadows on polished
concrete. Minimal, focused startup HQ.
STYLE — SWISS PULSE (Müller-Brockmann): Grid-locked compositions. Black (#1a1a1a),
white, electric blue (#0066FF), warm amber (#FF9500) for records. Helvetica Bold
headlines, Regular labels. Numbers LARGE. Animated counters count up from 0.
Diagonal compositions on accent moments. Grid wipe transitions. No dissolves.
CRITICAL ON-SCREEN TEXT (display literally):
- "1.85M SIGNUPS — +28% MoM"
- "$2.12M NEW SUBSCRIPTION REVENUE"
- "$54M → $141M ARR"
- "2.5M FOLLOWERS" and "20 MIN / VIDEO"
- Quote: "Use technology to serve the message, not distract from it."
- "ORGANIC: 65% OF SUBS — VOLATILE"
MUSIC: Upbeat electronic with a driving beat. Tycho meets Bloomberg opening theme.
Builds through highlights, warms for customer story, softens for challenges, peaks
on close.
---
SCENE 1 — A-ROLL (8s)
[Avatar center-frame, energetic, leaning slightly forward]
VOICEOVER: "January was a record month. New highs across acquisition, revenue,
and product velocity. Here's the full picture."
Lower-third SLIDES in: "COMPANY NAME | JANUARY 2026" white on blue bar.
Grid wipe.
SCENE 2 — FULL SCREEN B-ROLL (12s)
[NO AVATAR — motion graphic only]
VOICEOVER: "One-point-eight-five million signups — twenty-eight percent month
over month. Two-point-one-two million in new subscription revenue. Both all-time
highs."
LAYER 1: Dark #1a1a1a background with thin grid lines pulsing at 8% opacity.
LAYER 2: "1.85M" SLAMS in from left, white Bold 140pt. "SIGNUPS" types on
in electric blue 32pt uppercase. "+28% MoM" appears in amber.
LAYER 3: Three stat cards CASCADE from top-right, staggered 0.3s:
"$2.12M New Revenue" — "$3.4M Business ARR" — "$3M Pro ARR."
Each number COUNTS UP from 0.
LAYER 4: Bottom ticker scrolls: "Non-brand search +36% • Brand impressions 9.2M
• Weekly subs +20.5%"
LAYER 5: Grid lines RIPPLE outward on "1.85M" slam. Diagonal amber bar behind
stat cards.
Hard cut.
SCENE 3 — FULL SCREEN B-ROLL (12s)
[NO AVATAR — motion graphic only]
VOICEOVER: "Zoom out. Twelve months ago — fifty-four million ARR. Today —
one hundred forty-one million. Nearly three X in a single year."
LAYER 1: Dark background, subtle grid scrolling upward.
LAYER 2: Animated line chart DRAWS ITSELF left to right. Y-axis: $50M to $150M.
Final point "$140.84M" glows amber and pulses.
LAYER 3: Milestone annotations float in at key data points.
LAYER 4: Second smaller chart below — "Paid Videos" 0.91M to 2.97M, same style.
LAYER 5: Thin grid lines converge toward final data point. Scan line sweeps.
Grid wipe.
SCENE 4 — A-ROLL (8s)
[Avatar center-frame, warm tone, genuine smile]
VOICEOVER: "But the numbers only tell half the story. The other half is the
people building on the platform."
Lower-third: "Customer Spotlight"
SCENE 5 — FULL SCREEN B-ROLL (12s)
[NO AVATAR — warm palette]
VOICEOVER: "An AI character built entirely on the platform. Twenty minutes
per video. Two-point-five million Instagram followers. The creator's principle:
use technology to serve the message, not distract from it."
LAYER 1: Dark background with warm amber grid lines at low opacity.
LAYER 2: "CHARACTER NAME" in large white, center-top, 80pt.
LAYER 3: Stats cascade from right: "2.5M Followers" COUNTS UP in amber —
"20 min/video" — "7x Faster." Each a glowing node.
LAYER 4: Quote card SLIDES UP: "Use technology to serve the message, not
distract from it." Types on word by word.
LAYER 5: Warm light bloom. Grid lines soften into curved arcs.
Grid wipe.
SCENE 6 — A-ROLL (10s)
[Avatar center-frame, serious/candid]
VOICEOVER: "Now the honest part. Organic drives sixty-five percent of
subscriptions and it's volatile. Non-brand traffic dropped sixteen percent
last week. We've rebuilt attribution and we're investing in SEO."
Lower-third: "Challenges"
SCENE 7 — A-ROLL (7s)
[Avatar center-frame, energy lifts, direct eye contact]
VOICEOVER: "Fifty-four million to one-forty-one in twelve months. Three million
paid videos a month. January set the bar — now we raise it."
End card: Logo centered, blue glow fade-in. Grid lines converge. Music peaks.
---
NARRATION STYLE: CEO energy — conviction backed by data. Fast on highlights.
Warm on customer stories. Candid on challenges. Close with forward momentum.
```
## Ready-to-Use Templates
### Tech News Briefing
```
FORMAT: 75-second high-energy tech briefing. Think: Bloomberg meets Vice.
AVATAR: [Presenter in tech-casual at a multi-monitor station.
Describe clothing, monitor content, desk items, lighting.]
STYLE — DECONSTRUCTED (Brody): Dark grey #1a1a1a, rust orange #D4501E.
Type at angles, overlapping. Gritty textures. Smash cut transitions.
CRITICAL ON-SCREEN TEXT:
- [List every stat, quote, handle that must appear]
SCENE 1 — A-ROLL (8s): Hook with energy. State what's happening.
SCENE 2 — B-ROLL (12s): First story with layered visuals (L1-L5).
SCENE 3 — A-ROLL + OVERLAY (10s): Second story, split frame.
SCENE 4 — B-ROLL (10s): Third story or dramatic data point.
SCENE 5 — A-ROLL (8s): Wrap-up and forward look.
```
### Product Comparison
```
FORMAT: 60-second comparison. [Product A] vs [Product B]. Data-driven.
AVATAR: [Presenter in review studio. Desk with both products visible.]
STYLE — DIGITAL GRID (Crouwel): Dark #0a0a0a, cyan #00D4FF and amber #FFB800.
Two-color coding: cyan = Product A, amber = Product B. Monospaced type.
CRITICAL ON-SCREEN TEXT:
- [Key stats for each product]
- [Pricing, features, differentiators]
Use SPLIT FRAME B-roll: Product A left, Product B right.
```
### Strategy Presentation
```
FORMAT: 90-second strategy briefing. Bloomberg meets board meeting.
AVATAR: [Executive in blazer over tee. Conference room with whiteboard frameworks.]
STYLE — SWISS PULSE (Müller-Brockmann): Black/white + blue #0066FF.
Grid-locked. Helvetica. Animated counters. Grid wipe transitions.
CRITICAL ON-SCREEN TEXT:
- [Framework labels, quadrant labels, key quotes]
Build frameworks visually: draw axes, plot positions, animate labels.
```
### Social Ad (30 seconds)
```
FORMAT: 30-second social ad. Maximum energy. Portrait 9:16.
AVATAR: [Creator-style presenter. Ring light, colorful background.]
STYLE — CARNIVAL SURGE (Lins): Hot pink, yellow, teal. Collage layering.
Text MASSIVE at angles. Confetti. Smash cuts.
Three scenes: Hook (8s) → Value prop (12s) → CTA (10s).
Text fills 50-80% of every frame. Numbers SLAM.
```
### Premium Report
```
FORMAT: 120-second investor-grade report. Understated authority.
AVATAR: [Tailored merino sweater. Architectural room, diffused natural light.]
STYLE — VELVET STANDARD (Vignelli): Black, white, gold #c9a84c.
Thin ALL CAPS, wide spacing. Generous negative space.
Slow cross-dissolves. Numbers fade in with weight.
```
.claude/skills/heygen/references/prompt-optimizer.md
+289
View File
@@ -0,0 +1,289 @@
---
name: prompt-optimizer
description: Write production-quality prompts for HeyGen Video Agent — from basic ideas to fully art-directed scene-by-scene scripts
---
# Video Agent Prompt Optimizer
Write effective prompts for the HeyGen Video Agent API. Based on patterns from 40+ produced videos.
**The core insight: Video Agent is an HTML interpreter.** It renders layouts, typography, and structured content natively. Describe B-roll as layered text motion graphics with action verbs ("slams in," "types on," "counts up") — not layout specs ("upper-left, 48pt").
## Reference Files
| File | Load when... |
|------|-------------|
| [visual-styles.md](visual-styles.md) | Choosing a visual style (20 styles with full specs) |
| [prompt-examples.md](prompt-examples.md) | Writing a prompt from scratch (full production example + templates) |
## Workflow: Brief to Prompt
1. **Pull data** — Research the topic: web search, APIs, internal docs. Gather real quotes, stats, handles
2. **Synthesize a thesis** — Not a list. A story. *"X is happening because Y — here's the proof."* Group into 3-5 themes with a narrative arc
3. **Choose a style** — Match mood first, content second. Ask: *"What should the viewer FEEL?"* See [visual-styles.md](visual-styles.md)
4. **Write the avatar** — Thematic wardrobe matching content's emotional context. Brand logos and content-specific props in the set (see Avatar Guide below)
5. **Extract critical text** — List every number, quote, handle, and label that must appear literally
6. **Break into scenes** — One concept per scene. Rotate scene types. Never 3+ of same type in a row. At least 2 pure B-roll scenes
7. **Write voiceover** — Spell out numbers in VO ("one-point-eight-five million"), use figures on screen ("1.85M"). Narration on EVERY scene including B-roll
8. **Layer each B-roll scene** — L1 background, L2 hero, L3 supporting, L4 info bar, L5 effects. Every element must MOVE
9. **Add music direction** — Reference artists, describe energy arc
10. **Add narration style** — How to deliver: fast/slow, where to pause, emotional register per section
## Prompt Anatomy
Every production-quality prompt follows this structure:
```
FORMAT: What kind of video, how long, what energy
TONE: Emotional register, references
AVATAR: Detailed physical + environment description (60-100 words)
STYLE: Named aesthetic with colors, typography, motion rules, transitions
CRITICAL ON-SCREEN TEXT: Exact strings that must appear
SCENE-BY-SCENE: Individual scene breakdowns with VO and layered visuals
MUSIC: Genre, reference artists, energy arc
NARRATION STYLE: How to deliver the voiceover
```
### FORMAT
```
FORMAT: 75-second high-energy tech daily briefing. Think: a creator who just got amazing news.
FORMAT: Bloomberg-style strategy briefing. 100-120 seconds. CEO-delivered.
```
### TONE
```
TONE: Confident, direct, data-backed. Highlights hit hard. Lowlights are honest — no spin.
TONE: Edgy, punk tech commentary. Vice News meets The Face magazine — raw, confrontational.
```
### CRITICAL ON-SCREEN TEXT
List every exact string that must appear on screen. Without this, the agent may summarize, round numbers, or rephrase quotes.
```
CRITICAL ON-SCREEN TEXT (display literally):
- "$141M ARR — All-Time High"
- "1.85M Signups — +28% MoM"
- Quote: "Use technology to serve the message, not distract from it." — Shalev Hani
- "@username" — exact social handle
```
### MUSIC & NARRATION
```
MUSIC: Driving electronic, heavy bass drops on key numbers. Run the Jewels meets
a tech keynote. Builds relentlessly, only softens for customer stories.
NARRATION STYLE: High energy throughout. Let numbers PUNCH — pause before big ones,
then deliver hard. Customer stories get warmth. The close should feel like a mic drop.
```
## Avatar Description Guide
**The avatar is NOT a fixed headshot** — design it for each video like a movie character. Think costume designer + set designer.
### Thematic Wardrobe Rule
The avatar's outfit and environment MUST match the content's emotional/cultural context:
| Content Type | Avatar Design | NOT This |
|---|---|---|
| Chinese New Year | Red qipao with gold embroidery, lantern-lit courtyard | "Reporter in a blazer" |
| Breaking tech news | Field reporter, windswept hair, earpiece, city skyline | "Anchor at a desk" |
| Sleep science | Oversized cream knit, cross-legged on bed, warm lamp | "Analyst in a lab" |
| Reddit community | Messy desk, Reddit alien on monitors, upvote arrows on wall | "Researcher in a studio" |
### What to Specify
| Element | Weak | Strong |
|---------|------|--------|
| Clothing | "Business casual" | "Black ribbed merino turtleneck, high collar framing jaw" |
| Environment | "An office" | "Glass-walled conference room. Whiteboard with hand-drawn tier pyramid" |
| Monitor content | "Computer screens" | "Monitor shows scrolling green terminal text and red security alerts" |
| Lighting | "Well lit" | "Cool blue monitor glow from left, warm amber desk lamp from right" |
### Template
```
AVATAR: [Clothing — fabric, color, fit, accessories, posture].
[Setting — specific props, brand logos, what's on the walls].
[Monitors/desk — content visible on screens, items on desk].
[Lighting — direction, color temperature]. [Mood of the space].
60-100 words. 3+ content-specific props. Brand elements visible.
```
## Scene Types
| Type | Format | When to Use |
|------|--------|-------------|
| **A-ROLL** | Avatar speaking to camera | Intros, key insights, CTAs, emotional beats |
| **FULL SCREEN B-ROLL** | No avatar — motion graphics only | Data visualization, information-dense content |
| **A-ROLL + OVERLAY** | Split frame: avatar + content | Presenting data while maintaining human connection |
**Rotation is mandatory.** Never 3+ of the same type in a row. Every prompt needs at least 2 pure B-roll scenes.
**Voiceover on EVERY scene.** Every B-roll scene MUST include a `VOICEOVER:` line. Silent B-roll = broken video.
### Scene Anatomy
**A-ROLL:**
```
SCENE 1 — A-ROLL (10s)
[Avatar center-frame, excited, hands gesturing]
VOICEOVER: "The exact script for this scene."
Lower-third: "TITLE TEXT" white on blue bar.
```
**B-ROLL with layers:**
```
SCENE 2 — FULL SCREEN B-ROLL (12s)
[NO AVATAR — motion graphic only]
VOICEOVER: "The exact script for this scene."
LAYER 1: Dark #1a1a1a background with subtle grid lines pulsing.
LAYER 2: "HEADLINE" SLAMS in from left in white Bold 100pt at -5 degrees.
LAYER 3: Three data cards CASCADE from right, staggered 0.3s.
LAYER 4: Bottom ticker SLIDES in: "supporting text scrolling continuously."
LAYER 5: Grid lines RIPPLE outward from impact point.
Hard cut.
```
**A-ROLL + OVERLAY:**
```
SCENE 3 — A-ROLL + OVERLAY (10s)
[SPLIT — Avatar LEFT 35%. Content RIGHT 65%. NO overlap.]
Avatar gestures toward content side.
VOICEOVER: "The exact script for this scene."
RIGHT SIDE: "HEADLINE" in cyan 60pt. Three stats COUNT UP below.
```
Alternate which side the avatar appears on between overlay scenes.
## The Visual Layer System
Break B-roll into 5 stacked layers. This is the most powerful technique for motion graphics scenes.
| Layer | Purpose | Examples |
|-------|---------|---------|
| **L1** | Background | Textured surface, grid, gradient, color field |
| **L2** | Hero content | Main headline/number that dominates the frame |
| **L3** | Supporting data | Cards, stats, bullet points, secondary information |
| **L4** | Information bar | Tickers, labels, source attributions, quotes |
| **L5** | Effects | Particles, glitches, grid animations, ambient motion |
Every B-roll: 4+ layers. Every overlay content side: 3+ layers. **Every element must MOVE.**
## Motion Vocabulary
### High Energy
| Verb | Example |
|------|---------|
| **SLAMS** | `"$95M" SLAMS in from left at -5 degrees` |
| **CRASHES** | `Title CRASHES in from right, screen-shake on impact` |
| **PUNCHES** | `Quote card PUNCHES up from bottom` |
| **STAMPS** | `Data blocks STAMP in staggered 0.4s` |
| **SHATTERS** | `Text SHATTERS after 1.5s, revealing number underneath` |
### Medium Energy
| Verb | Example |
|------|---------|
| **CASCADE** | `Three cards CASCADE from top, staggered 0.3s` |
| **SLIDES** | `Ticker SLIDES in from right — continuous scroll` |
| **DROPS** | `"TIER 1" DROPS in with white flash` |
| **FILLS** | `Progress bar FILLS 0 to 90% in orange` |
| **DRAWS** | `Chart line DRAWS itself left to right` |
### Low Energy
| Verb | Example |
|------|---------|
| **types on** | `Quote types on word by word in italic white` |
| **fades in** | `Logo fades in at center, held for 3 seconds` |
| **FLOATS** | `Bokeh orbs FLOAT across frame at different speeds` |
| **morphs** | `Number morphs from 17 to 18.9` |
| **COUNTS UP** | `"1.85M" COUNTS UP from 0 in amber 96pt` |
## Transition Types
| Transition | Energy | Styles It Fits |
|------------|--------|---------------|
| Smash cut | Aggressive | Deconstructed, Maximalist, Carnival Surge |
| White flash frame | Punchy | Deconstructed, Maximalist |
| Grid wipe | Systematic | Swiss Pulse, Digital Grid |
| Hard cut | Clean | Swiss Pulse, Shadow Cut |
| Liquid dissolve | Elegant | Data Drift, Dream State |
| Slow cross-dissolve | Refined | Velvet Standard |
| Pop cut / bounce | Fun | Play Mode, Carnival Surge |
| Snap cut | Urgent | Red Wire, Contact Sheet |
| Soft dissolve | Warm | Soft Signal, Warm Grain, Quiet Drama |
| Iris wipe | Nostalgic | Heritage Reel |
## Timing Guidelines
| Content Type | Duration |
|--------------|----------|
| Hook/Intro (A-roll) | 6-10 seconds |
| Data-heavy B-roll | 10-15 seconds (NEVER ≤5s — causes black frames) |
| A-roll + Overlay | 8-12 seconds |
| CTA / Close (A-roll) | 6-8 seconds |
**Common video lengths:** Social clip: 30-45s (5-7 scenes) | Briefing: 60-75s (7-9 scenes) | Deep dive: 90-120s (10-13 scenes)
**Speaking pace:** ~150 words/minute. Calculate: `words / 150 * 60 = seconds`
## What Doesn't Work
Patterns that consistently produce poor results:
**Layout language** — Screen coordinates cause empty/black B-roll:
```
❌ "UPPER-LEFT: headline in 48pt Helvetica"
❌ "CENTER-SCREEN: display at coordinates (400, 300)"
✅ "135K" SLAMS in from left, white Impact 120pt, fills 40% of frame.
```
**Named artists without specs** — "Ikko Tanaka style" means nothing to Video Agent. Translate to concrete rules:
```
❌ "Use an Ikko Tanaka style"
✅ "Flat color blocks, maximum 3 colors per frame, 60% negative space, typography as primary element"
```
**Style examples injected into prompts** — Full example scenes from a style library confuse the agent. Use the style's **rules**, not example scenes.
**Forced short B-roll (≤5 seconds)** — Too short for rendering. Every tested video with 5s B-roll had empty/black screens. Use 10-15s.
**Content as a list, not a story** — "Here are 5 tweets" produces flat videos. Always synthesize: *"X is happening because Y — here's the proof."*
## Production Insights
### Style Performance (from 40+ videos)
| Rank | Style | Strength |
|------|-------|----------|
| 1 | Deconstructed (Brody) | Most reliable across all topics |
| 2 | Swiss Pulse (Müller-Brockmann) | Best for data-heavy content |
| 3 | Digital Grid (Crouwel) | Strong for tech topics |
| 4 | Geometric Bold (Tanaka) | Elegant and versatile |
| 5 | Maximalist Type (Scher) | High energy, use sparingly |
### Duration by Approach
| Approach | Avg Duration | Quality |
|----------|-------------|---------|
| Natural storyboard + custom avatar | ~106s | Best |
| Natural storyboard, no custom avatar | ~69s | Good |
| Forced short scenes + custom avatar | ~71s | Mixed |
| Layout language prompts | ~48s | Poor |
## Quality Checklist
- [ ] Thesis-driven — story, not bullet points
- [ ] Style named with colors, typography, motion, transitions (see [visual-styles.md](visual-styles.md))
- [ ] Avatar has thematic wardrobe + branded environment (60-100 words)
- [ ] Critical text listed — every stat, quote, label
- [ ] Scenes rotate types — never 3+ same type. At least 2 B-roll scenes
- [ ] Every scene has VOICEOVER — including B-roll
- [ ] B-roll scenes have 4+ layers, every element has motion verbs
- [ ] B-roll scenes are 10-15 seconds (never ≤5s)
- [ ] Brand logos appear when discussing companies
- [ ] Every element moves — no static frames
.claude/skills/heygen/references/quota.md
+195
View File
@@ -0,0 +1,195 @@
---
name: quota
description: Credit system, usage limits, and checking remaining quota for HeyGen
---
# HeyGen Quota and Credits
HeyGen uses a credit-based system for video generation. Understanding quota management helps prevent failed video generation requests.
## Checking Remaining Quota
### curl
```bash
curl -X GET "https://api.heygen.com/v2/user/remaining_quota" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface QuotaResponse {
error: null | string;
data: {
remaining_quota: number;
used_quota: number;
};
}
const response = await fetch("https://api.heygen.com/v2/user/remaining_quota", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const { data }: QuotaResponse = await response.json();
console.log(`Remaining credits: ${data.remaining_quota}`);
```
### Python
```python
import requests
import os
response = requests.get(
"https://api.heygen.com/v2/user/remaining_quota",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()["data"]
print(f"Remaining credits: {data['remaining_quota']}")
```
## Response Format
```json
{
"error": null,
"data": {
"remaining_quota": 450,
"used_quota": 50
}
}
```
## Credit Consumption
Different operations consume different amounts of credits:
| Operation | Credit Cost | Notes |
|-----------|-------------|-------|
| Standard video (1 min) | ~1 credit per minute | Varies by resolution |
| 720p video | Base rate | Standard quality |
| 1080p video | ~1.5x base rate | Higher quality |
| Video translation | Varies | Depends on video length |
| Streaming avatar | Per session | Real-time usage |
## Pre-Generation Quota Check
Always verify sufficient quota before generating videos:
```typescript
async function generateVideoWithQuotaCheck(videoConfig: VideoConfig) {
// Check quota first
const quotaResponse = await fetch(
"https://api.heygen.com/v2/user/remaining_quota",
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data: quota } = await quotaResponse.json();
// Estimate required credits (rough estimate: 1 credit per minute)
const estimatedMinutes = videoConfig.estimatedDuration / 60;
const requiredCredits = Math.ceil(estimatedMinutes);
if (quota.remaining_quota < requiredCredits) {
throw new Error(
`Insufficient credits. Need ${requiredCredits}, have ${quota.remaining_quota}`
);
}
// Proceed with video generation
return generateVideo(videoConfig);
}
```
## Quota Management Best Practices
### 1. Monitor Usage Regularly
```typescript
async function logQuotaUsage() {
const response = await fetch(
"https://api.heygen.com/v2/user/remaining_quota",
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
console.log({
remaining: data.remaining_quota,
used: data.used_quota,
percentUsed: (
(data.used_quota / (data.remaining_quota + data.used_quota)) *
100
).toFixed(1),
});
}
```
### 2. Set Up Alerts
```typescript
const QUOTA_WARNING_THRESHOLD = 50;
async function checkQuotaWithAlert() {
const response = await fetch(
"https://api.heygen.com/v2/user/remaining_quota",
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
if (data.remaining_quota < QUOTA_WARNING_THRESHOLD) {
// Send alert (email, Slack, etc.)
await sendAlert(`Low HeyGen quota: ${data.remaining_quota} credits remaining`);
}
return data;
}
```
### 3. Use Test Mode for Development
When available, use test mode to avoid consuming credits during development:
```typescript
const videoConfig = {
test: true, // Use test mode during development
video_inputs: [...],
};
// Test videos may have watermarks but don't consume credits
```
## Subscription Tiers
Different subscription tiers have different quota allocations and features:
| Tier | Features |
|------|----------|
| Free | Limited credits, basic features |
| Creator | More credits, standard avatars |
| Team | Higher limits, team collaboration |
| Enterprise | Custom limits, API access, priority support |
API access typically requires Enterprise tier or higher.
## Error Handling for Quota Issues
```typescript
async function handleQuotaError(error: any) {
if (error.message.includes("quota") || error.message.includes("credit")) {
console.error("Quota exceeded. Consider:");
console.error("1. Upgrading your subscription");
console.error("2. Waiting for quota reset");
console.error("3. Purchasing additional credits");
// Check current quota
const quota = await getQuota();
console.error(`Current remaining: ${quota.remaining_quota}`);
}
throw error;
}
```
.claude/skills/heygen/references/remotion-integration.md
+705
View File
@@ -0,0 +1,705 @@
---
name: remotion-integration
description: Using HeyGen avatar videos in Remotion compositions
---
# HeyGen + Remotion Integration
This guide covers workflows for generating HeyGen avatar videos and using them in Remotion compositions.
## Quick Start
```typescript
// 1. Get avatar with default voice
const avatar = await getAvatarDetails(avatarId);
// 2. Generate video (MP4 with background - most common)
const videoId = await generateVideo({
video_inputs: [{
character: { type: "avatar", avatar_id: avatar.id, avatar_style: "normal" },
voice: { type: "text", input_text: script, voice_id: avatar.default_voice_id },
background: { type: "color", value: "#1a1a2e" },
}],
dimension: { width: 1920, height: 1080 },
});
// 3. Poll for completion (10-15+ min)
// 4. Use in Remotion with motion graphics overlaid on top
```
## Overview
A typical workflow:
1. Generate avatar video with HeyGen
2. Wait for completion and get video URL
3. Download or use URL directly in Remotion
4. Compose with other elements (backgrounds, overlays, animations)
## Choosing the Right Output Format
| Your Composition | Recommended | Why |
|------------------|-------------|-----|
| Avatar as presenter with overlays | MP4 + background | Simpler, overlays go on top |
| Loom-style (avatar over screen recording) | WebM + `closeUp`, mask in Remotion | Need transparency, apply circle mask in CSS |
| Avatar overlaid ON other video/content | WebM (transparent) | Need to see through to content behind |
| Full-screen avatar | MP4 + background | Standard approach |
**Use MP4 with background for most cases.** Use WebM when you need to see content *behind* the avatar.
**Note:** WebM only supports `normal` and `closeUp` styles. For circular framing, use CSS `border-radius: 50%` in Remotion.
## Recommended: Parallel Development Workflow
HeyGen video generation takes **10-15+ minutes**. Don't wait - work in parallel:
1. **Start HeyGen generation** - save `video_id` to a file, exit immediately
2. **Build Remotion composition** - use a placeholder or the avatar's `preview_video_url` (a short loop)
3. **Check HeyGen status** periodically or when done building
4. **Swap placeholder** for real video URL once ready
**Estimate duration from script**: ~150 words/minute speech rate, so `wordCount / 150 * 60 * fps` gives approximate frames.
**Composition tip**: Design components to work with or without the avatar video, so motion graphics can be tested independently.
## Dimension Alignment
**Critical**: Match HeyGen output dimensions to your Remotion composition.
### Common Dimension Presets
```typescript
// Shared dimension constants for both HeyGen and Remotion
const DIMENSIONS = {
landscape_1080p: { width: 1920, height: 1080 },
landscape_720p: { width: 1280, height: 720 },
portrait_1080p: { width: 1080, height: 1920 },
portrait_720p: { width: 720, height: 1280 },
square_1080p: { width: 1080, height: 1080 },
square_720p: { width: 720, height: 720 },
} as const;
type DimensionPreset = keyof typeof DIMENSIONS;
```
### HeyGen Video Generation
```typescript
// Generate HeyGen video with specific dimensions
async function generateHeyGenVideo(
script: string,
avatarId: string,
voiceId: string,
preset: DimensionPreset
): Promise<string> {
const dimension = DIMENSIONS[preset];
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
video_inputs: [
{
character: {
type: "avatar",
avatar_id: avatarId,
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: voiceId,
},
background: {
type: "color",
value: "#00FF00", // Green screen for compositing
},
},
],
dimension,
}),
});
const { data } = await response.json();
return data.video_id;
}
```
### Remotion Composition Setup
```tsx
// remotion/src/Root.tsx
import { Composition } from "remotion";
import { AvatarComposition } from "./AvatarComposition";
const DIMENSIONS = {
landscape_1080p: { width: 1920, height: 1080 },
// ... same as above
};
export const RemotionRoot: React.FC = () => {
return (
<>
<Composition
id="AvatarVideo"
component={AvatarComposition}
durationInFrames={300} // Will be set dynamically
fps={30}
width={DIMENSIONS.landscape_1080p.width}
height={DIMENSIONS.landscape_1080p.height}
defaultProps={{
avatarVideoUrl: "",
}}
/>
</>
);
};
```
## Generating Avatar Video for Remotion
### Standard: MP4 with Background
Most Remotion compositions work best with MP4 + background. Overlays and motion graphics go on top:
```typescript
async function generateAvatarForRemotion(
script: string,
avatarId: string,
voiceId: string,
options: {
style?: "normal" | "closeUp" | "circle";
backgroundColor?: string;
} = {}
): Promise<string> {
const { style = "normal", backgroundColor = "#1a1a2e" } = options;
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
video_inputs: [{
character: {
type: "avatar",
avatar_id: avatarId,
avatar_style: style,
},
voice: {
type: "text",
input_text: script,
voice_id: voiceId,
},
background: {
type: "color",
value: backgroundColor,
},
}],
dimension: { width: 1920, height: 1080 },
}),
});
const { data } = await response.json();
return data.video_id;
}
```
### Transparent Background (WebM)
Only use when you need to see content *behind* the avatar (e.g., avatar overlaid on screen recording):
```typescript
// Use /v1/video.webm endpoint for transparent background
// Note: Different structure than /v2/video/generate
const response = await fetch("https://api.heygen.com/v1/video.webm", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
avatar_pose_id: avatarPoseId, // Required: avatar pose ID
avatar_style: "normal", // Required: "normal" or "closeUp" only
input_text: script, // Required (with voice_id)
voice_id: voiceId, // Required (with input_text)
dimension: { width: 1920, height: 1080 },
}),
});
```
## Using HeyGen Video in Remotion
### Important: Use OffthreadVideo for Frame-Accurate Rendering
**Always use `OffthreadVideo` instead of `Video`** for HeyGen avatar videos. The basic `Video` component uses the browser's video decoder which isn't frame-accurate, causing jitter during rendering. `OffthreadVideo` extracts frames via FFmpeg for smooth, accurate playback.
`OffthreadVideo` is included in the core `remotion` package - no additional install needed.
### Basic Usage
```tsx
// remotion/src/AvatarComposition.tsx
import { OffthreadVideo, useVideoConfig } from "remotion";
interface AvatarCompositionProps {
avatarVideoUrl: string;
}
export const AvatarComposition: React.FC<AvatarCompositionProps> = ({
avatarVideoUrl,
}) => {
return (
<div style={{ flex: 1, backgroundColor: "#1a1a2e" }}>
<OffthreadVideo
src={avatarVideoUrl}
style={{
width: "100%",
height: "100%",
objectFit: "contain",
}}
/>
</div>
);
};
```
### WebM with Transparent Background (Recommended)
Using WebM from `/v1/video.webm` - no chroma keying needed:
```tsx
import { OffthreadVideo, AbsoluteFill, Sequence } from "remotion";
export const AvatarWithMotionGraphics: React.FC<{
avatarWebmUrl: string
}> = ({ avatarWebmUrl }) => {
return (
<AbsoluteFill>
{/* Layer 1: Your background/content */}
<AbsoluteFill style={{ backgroundColor: "#1a1a2e" }}>
<YourMotionGraphics />
</AbsoluteFill>
{/* Layer 2: Avatar with transparent background - use OffthreadVideo for frame-accurate rendering */}
<OffthreadVideo
src={avatarWebmUrl}
transparent
style={{
position: "absolute",
bottom: 0,
right: 0,
width: "50%",
height: "auto",
}}
/>
{/* Layer 3: Overlays on top of avatar */}
<Sequence from={30}>
<AnimatedTitle text="Welcome!" />
</Sequence>
</AbsoluteFill>
);
};
```
### Loom-Style: Circle Avatar Over Screen Recording
Use `closeUp` style + WebM, then apply circular mask in Remotion:
```tsx
import { OffthreadVideo, AbsoluteFill } from "remotion";
export const LoomStyleComposition: React.FC<{
screenRecordingUrl: string;
avatarWebmUrl: string; // Generated with avatar_style: "closeUp" via /v1/video.webm
}> = ({ screenRecordingUrl, avatarWebmUrl }) => {
return (
<AbsoluteFill>
{/* Screen recording fills the frame */}
<OffthreadVideo src={screenRecordingUrl} style={{ width: "100%", height: "100%" }} />
{/* Avatar with circular mask - transparent bg shows screen behind */}
<OffthreadVideo
src={avatarWebmUrl}
transparent
style={{
position: "absolute",
bottom: 40,
left: 40,
width: 180,
height: 180,
borderRadius: "50%", // Circular mask applied in CSS
overflow: "hidden",
objectFit: "cover",
}}
/>
</AbsoluteFill>
);
};
```
**Note:** WebM doesn't support `circle` style - use `normal` or `closeUp` and apply circular masking via CSS.
### Legacy: Green Screen with Chroma Key
If using MP4 with green background (not recommended - use WebM instead):
```tsx
// Note: True chroma key requires WebGL or post-processing
// WebM transparent background is much simpler
<OffthreadVideo
src={avatarVideoUrl}
style={{
mixBlendMode: "multiply", // Basic compositing only
}}
/>
```
### Layered Composition
```tsx
import { OffthreadVideo, Sequence, useVideoConfig, Img } from "remotion";
interface LayeredAvatarProps {
avatarVideoUrl: string;
backgroundUrl: string;
logoUrl: string;
title: string;
}
export const LayeredAvatarComposition: React.FC<LayeredAvatarProps> = ({
avatarVideoUrl,
backgroundUrl,
logoUrl,
title,
}) => {
const { fps } = useVideoConfig();
return (
<div style={{ position: "relative", width: "100%", height: "100%" }}>
{/* Layer 1: Background */}
<Img
src={backgroundUrl}
style={{
position: "absolute",
width: "100%",
height: "100%",
objectFit: "cover",
}}
/>
{/* Layer 2: Avatar video - use OffthreadVideo to prevent jitter */}
<OffthreadVideo
src={avatarVideoUrl}
style={{
position: "absolute",
bottom: 0,
right: 0,
width: "40%",
height: "auto",
}}
/>
{/* Layer 3: Title (appears after 1 second) */}
<Sequence from={fps}>
<div
style={{
position: "absolute",
top: 50,
left: 50,
color: "white",
fontSize: 48,
fontWeight: "bold",
}}
>
{title}
</div>
</Sequence>
{/* Layer 4: Logo */}
<Img
src={logoUrl}
style={{
position: "absolute",
top: 20,
right: 20,
width: 100,
height: "auto",
}}
/>
</div>
);
};
```
## Complete Workflow
### Generate and Compose
```typescript
import { bundle } from "@remotion/bundler";
import { renderMedia, selectComposition } from "@remotion/renderer";
async function generateAvatarVideoForRemotion(
script: string,
outputPath: string
) {
// 1. Generate HeyGen video
console.log("Generating HeyGen avatar video...");
const videoId = await generateHeyGenVideo(
script,
"josh_lite3_20230714",
"1bd001e7e50f421d891986aad5158bc8",
"landscape_1080p"
);
// 2. Wait for completion
console.log("Waiting for HeyGen video...");
const avatarVideoUrl = await waitForVideo(videoId);
console.log(`HeyGen video ready: ${avatarVideoUrl}`);
// 3. Get video duration for Remotion
const avatarDuration = await getVideoDuration(avatarVideoUrl);
const durationInFrames = Math.ceil(avatarDuration * 30); // 30 fps
// 4. Bundle Remotion project
console.log("Bundling Remotion project...");
const bundleLocation = await bundle({
entryPoint: "./remotion/src/index.ts",
});
// 5. Select composition
const composition = await selectComposition({
serveUrl: bundleLocation,
id: "AvatarVideo",
inputProps: {
avatarVideoUrl,
},
});
// 6. Render final video
console.log("Rendering final composition...");
await renderMedia({
composition: {
...composition,
durationInFrames,
},
serveUrl: bundleLocation,
codec: "h264",
outputLocation: outputPath,
inputProps: {
avatarVideoUrl,
},
});
console.log(`Final video rendered: ${outputPath}`);
return outputPath;
}
```
### Dynamic Duration with calculateMetadata
```tsx
// remotion/src/AvatarComposition.tsx
import { CalculateMetadataFunction } from "remotion";
export const calculateAvatarMetadata: CalculateMetadataFunction<
AvatarCompositionProps
> = async ({ props }) => {
// Fetch video duration from HeyGen video
const duration = await getVideoDurationInSeconds(props.avatarVideoUrl);
return {
durationInFrames: Math.ceil(duration * 30),
fps: 30,
width: 1920,
height: 1080,
};
};
// In Root.tsx
<Composition
id="AvatarVideo"
component={AvatarComposition}
calculateMetadata={calculateAvatarMetadata}
defaultProps={{
avatarVideoUrl: "",
}}
/>
```
## Best Practices
### 1. Use Green Screen for Flexibility
Generate HeyGen videos with green screen background when you want to composite:
```typescript
background: {
type: "color",
value: "#00FF00", // Pure green for chroma key
}
```
### 2. Match Frame Rates
HeyGen default is 25 fps. Consider this when setting Remotion fps:
```typescript
// Option 1: Match HeyGen's 25 fps
fps: 25
// Option 2: Use 30 fps with playback rate adjustment
<OffthreadVideo
src={avatarVideoUrl}
playbackRate={25/30} // Slow down slightly to match
/>
```
### 3. URL vs Download: When to Use Each
**Use URL directly** when:
- Previewing in Remotion Studio (`npm run dev`)
- URL won't expire before render completes
- You want faster iteration during development
```tsx
// Direct URL usage - simpler, faster for dev
<OffthreadVideo src={avatarVideoUrl} />
```
**Download first** when:
- URL has expiration (HeyGen URLs expire after ~24 hours)
- Rendering will happen later or repeatedly
- Network reliability is a concern
- You need offline rendering
```typescript
// Download with retry for reliability
async function downloadVideoWithRetry(
url: string,
outputPath: string,
maxRetries = 5
): Promise<string> {
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await fetch(url);
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const buffer = await response.arrayBuffer();
await fs.promises.writeFile(outputPath, Buffer.from(buffer));
return outputPath;
} catch (error) {
const delay = 2000 * Math.pow(2, attempt);
console.log(`Retry ${attempt + 1}/${maxRetries} in ${delay}ms...`);
await new Promise((r) => setTimeout(r, delay));
}
}
throw new Error("Download failed after retries");
}
// Use local file in Remotion
const localPath = await downloadVideoWithRetry(avatarVideoUrl, "./public/avatar.mp4");
```
**Hybrid approach** (recommended for production):
```typescript
// Save both URL and local path in metadata
const metadata = {
videoUrl: result.video_url, // For quick preview
localPath: "./public/avatar.mp4", // For reliable rendering
expiresAt: Date.now() + 24 * 60 * 60 * 1000, // URL expiration
};
// In Remotion component, prefer local if available
const videoSrc = fs.existsSync(localPath) ? staticFile("avatar.mp4") : avatarVideoUrl;
```
### 4. Handle Avatar Positioning
Common avatar positions in compositions:
```typescript
const AVATAR_POSITIONS = {
fullscreen: { width: "100%", height: "100%", position: "center" },
bottomRight: { width: "40%", bottom: 0, right: 0 },
bottomLeft: { width: "40%", bottom: 0, left: 0 },
pictureInPicture: { width: "25%", bottom: 20, right: 20 },
leftThird: { width: "33%", left: 0, height: "100%" },
};
```
## Output Formats
### HeyGen Output
- Format: MP4 (H.264)
- Audio: AAC
- Resolution: As specified in request
### Remotion Output
- Codec: H.264 (default), VP8, VP9, ProRes
- Match or exceed HeyGen quality settings
```typescript
await renderMedia({
codec: "h264",
crf: 18, // High quality
// ...
});
```
## Troubleshooting
### Video Not Playing in Remotion
1. Check URL accessibility (CORS issues)
2. Verify video format compatibility
3. Try downloading locally first
### Dimension Mismatch
Ensure both HeyGen and Remotion use identical dimensions:
```typescript
// Shared config
const VIDEO_CONFIG = {
width: 1920,
height: 1080,
fps: 30,
};
// HeyGen
dimension: { width: VIDEO_CONFIG.width, height: VIDEO_CONFIG.height }
// Remotion
<Composition width={VIDEO_CONFIG.width} height={VIDEO_CONFIG.height} />
```
### Video Jitter During Rendering
If avatar video appears jittery or stuttery in rendered output:
1. **Use `OffthreadVideo` instead of `Video`** - The basic `Video` component uses the browser's video decoder which isn't frame-accurate
2. Update imports (no additional install needed - it's in core `remotion`):
```tsx
// Before (causes jitter)
import { Video } from "remotion";
// After (frame-accurate)
import { OffthreadVideo } from "remotion";
```
3. For WebM with transparency, add the `transparent` prop:
```tsx
<OffthreadVideo src={avatarWebmUrl} transparent />
```
### Audio Sync Issues
If avatar audio drifts:
- Verify source video frame rate
- Check for encoding issues
- Consider re-encoding with consistent settings
.claude/skills/heygen/references/scripts.md
+322
View File
@@ -0,0 +1,322 @@
---
name: scripts
description: Writing effective scripts for HeyGen AI avatar videos
---
# Writing Scripts for HeyGen Videos
Scripts for AI avatar videos have different requirements than scripts for human presenters. This guide covers best practices for writing scripts that sound natural and render well.
## Script Basics
### Speech Rate and Duration
Typical speech is approximately **150 words per minute** at normal speed (1.0x). Use this as a rough estimate for planning script length.
| Script Length | Approximate Duration |
|---------------|---------------------|
| 75 words | 30 seconds |
| 150 words | 1 minute |
| 300 words | 2 minutes |
| 450 words | 3 minutes |
| 750 words | 5 minutes |
```typescript
// Estimate video duration from script
function estimateDuration(script: string, speed: number = 1.0): number {
const words = script.split(/\s+/).filter(w => w.length > 0).length;
const wordsPerMinute = 150 * speed;
return words / wordsPerMinute * 60; // seconds
}
// Estimate frames for Remotion
function estimateFrames(script: string, fps: number = 30, speed: number = 1.0): number {
const durationSeconds = estimateDuration(script, speed);
return Math.ceil(durationSeconds * fps);
}
```
### Sentence Structure
**Keep sentences short.** AI voices handle shorter sentences more naturally.
| Guideline | Example |
|-----------|---------|
| **Good**: 10-20 words per sentence | "Our platform helps teams collaborate. It syncs in real-time across all devices." |
| **Avoid**: 30+ word run-on sentences | "Our platform helps teams collaborate more effectively by providing real-time synchronization across all devices while also offering offline support and automatic conflict resolution." |
### Punctuation Affects Delivery
| Punctuation | Effect |
|-------------|--------|
| Period `.` | Full stop, natural pause |
| Comma `,` | Brief pause |
| Question mark `?` | Rising intonation |
| Exclamation `!` | Emphasis (use sparingly) |
| Ellipsis `...` | Trailing off, slight pause |
## Adding Pauses with Break Tags
Use SSML-style `<break>` tags for precise pause control:
```
<break time="Xs"/>
```
Where `X` is seconds (e.g., `0.5s`, `1s`, `1.5s`, `2s`).
### Formatting Rules
| Rule | Correct | Incorrect |
|------|---------|-----------|
| Space before tag | `word <break time="1s"/>` | `word<break time="1s"/>` |
| Space after tag | `<break time="1s"/> word` | `<break time="1s"/>word` |
| Use seconds with "s" | `<break time="1.5s"/>` | `<break time="1500ms"/>` |
| Self-closing tag | `<break time="1s"/>` | `<break time="1s"></break>` |
### When to Use Pauses
| Situation | Recommended Pause | Example |
|-----------|-------------------|---------|
| After greeting | 0.5-1s | `Hello! <break time="0.5s"/> Welcome to...` |
| Between sections | 1-1.5s | `...that's feature one. <break time="1.5s"/> Now let's look at...` |
| Before key point | 0.5s | `The most important thing is <break time="0.5s"/> consistency.` |
| For dramatic effect | 1.5-2s | `And the winner is... <break time="2s"/> you!` |
| After question | 1s | `Sound good? <break time="1s"/> Let's get started.` |
| List items | 0.5s | `First, speed. <break time="0.5s"/> Second, reliability.` |
### Pause Duration Guide
| Duration | Feel | Use For |
|----------|------|---------|
| 0.3-0.5s | Brief breath | Between clauses, light emphasis |
| 0.5-1s | Natural pause | Sentence breaks, transitions |
| 1-1.5s | Deliberate pause | Section changes, setup for key points |
| 1.5-2s | Dramatic | Reveals, important announcements |
| 2s+ | Long pause | Use sparingly, can feel unnatural |
### Examples
```typescript
// Section transitions
const script = `
Welcome to our product overview. <break time="1s"/>
Today I'll cover three key features. <break time="0.5s"/>
First, let's look at the dashboard. <break time="1.5s"/>
As you can see, it's designed for simplicity. <break time="0.5s"/>
Every action is just one click away.
`;
// Building suspense
const announcement = `
We've been working on something special. <break time="1s"/>
After months of development... <break time="1.5s"/>
I'm excited to announce <break time="0.5s"/> our new AI assistant.
`;
// List with rhythm
const features = `
Our platform offers three core benefits. <break time="0.5s"/>
Speed. <break time="0.5s"/>
Reliability. <break time="0.5s"/>
And simplicity. <break time="1s"/>
Let me show you each one.
`;
```
### Consecutive Breaks
Multiple consecutive breaks are combined:
```typescript
// These two breaks:
"Hello <break time=\"1s\"/> <break time=\"0.5s\"/> world"
// Are treated as a single 1.5s pause
```
## Script Structure Templates
### Product Demo (60 seconds, ~150 words)
```typescript
const productDemo = `
Hi, I'm [Name], and I'm excited to show you [Product]. <break time="1s"/>
[Product] helps you [main benefit] in just [timeframe]. <break time="0.5s"/>
Here's how it works. <break time="1s"/>
First, [step 1]. <break time="0.5s"/>
Then, [step 2]. <break time="0.5s"/>
And finally, [step 3]. <break time="1s"/>
What used to take [old time] now takes [new time]. <break time="0.5s"/>
Ready to get started? <break time="0.5s"/>
Visit [website] today.
`;
```
### Tutorial Introduction (90 seconds, ~225 words)
```typescript
const tutorial = `
Welcome to this tutorial on [topic]. <break time="0.5s"/>
I'm [Name], and I'll guide you through everything you need to know. <break time="1s"/>
By the end of this video, you'll be able to [outcome 1], [outcome 2], and [outcome 3]. <break time="1s"/>
Let's start with the basics. <break time="1.5s"/>
[Section 1 content - 2-3 sentences] <break time="1s"/>
Now that you understand [concept], let's move on to [next topic]. <break time="1.5s"/>
[Section 2 content - 2-3 sentences] <break time="1s"/>
And finally, let's cover [last topic]. <break time="1.5s"/>
[Section 3 content - 2-3 sentences] <break time="1s"/>
That's everything you need to get started. <break time="0.5s"/>
If you have questions, leave a comment below. <break time="0.5s"/>
Thanks for watching!
`;
```
### Announcement (30 seconds, ~75 words)
```typescript
const announcement = `
Big news! <break time="0.5s"/>
We're thrilled to announce [announcement]. <break time="1s"/>
This means [benefit 1] and [benefit 2] for all our users. <break time="0.5s"/>
Starting [date], you'll be able to [new capability]. <break time="1s"/>
Head to [location] to learn more. <break time="0.5s"/>
We can't wait to hear what you think!
`;
```
## Writing Tips for AI Voices
### Do
- **Write conversationally** - Read it aloud to check flow
- **Use contractions** - "We're" not "We are", "It's" not "It is"
- **Break up long sentences** - Split at natural pause points
- **Spell out abbreviations** - "API" may sound like "a pee eye"
- **Add pauses for emphasis** - Guide the listener's attention
- **End sections clearly** - Don't trail off mid-thought
### Avoid
- **Jargon without context** - Explain technical terms
- **Long parentheticals** - Move to separate sentences
- **Ambiguous pronunciations** - "read" (present) vs "read" (past)
- **Excessive exclamation marks** - One per script is usually enough
- **Run-on sentences** - Break into digestible chunks
- **Dense information** - Space out facts with pauses
### Pronunciation Hints
For words that might be mispronounced, spell phonetically or add hints:
```typescript
// Technical terms
const script1 = "Our API (A-P-I) handles authentication...";
// Ambiguous words
const script2 = "I read (red) the documentation yesterday...";
// Brand names
const script3 = "Welcome to HeyGen (hey-jen)...";
```
## Multi-Scene Scripts
When splitting scripts across scenes (for different backgrounds or avatars):
```typescript
const multiSceneVideo = {
video_inputs: [
{
// Scene 1: Introduction
character: { type: "avatar", avatar_id: "josh_lite3_20230714", avatar_style: "normal" },
voice: {
type: "text",
input_text: "Welcome to our quarterly update. <break time=\"1s\"/> I'm Josh, and I'll walk you through the highlights.",
voice_id: "voice_id_here",
},
background: { type: "color", value: "#1a1a2e" },
},
{
// Scene 2: Main content (different background)
character: { type: "avatar", avatar_id: "josh_lite3_20230714", avatar_style: "normal" },
voice: {
type: "text",
input_text: "Let's start with revenue. <break time=\"0.5s\"/> We grew 25 percent quarter over quarter. <break time=\"1s\"/> Here's what drove that growth.",
voice_id: "voice_id_here",
},
background: { type: "image", url: "https://..." },
},
// ... more scenes
],
};
```
### Scene Transition Tips
- End each scene with a complete thought
- Start new scenes with brief context
- Maintain consistent tone across scenes
- Use pauses at scene starts to let visuals register
## Testing Your Script
Before generating the full video:
1. **Read aloud** - Time yourself, check for awkward phrasing
2. **Count words** - Verify expected duration
3. **Check break tags** - Ensure proper spacing and syntax
4. **Preview with short clip** - Generate a 10-second test if unsure about pronunciation
```typescript
// Test a small portion first
const testScript = script.split('.').slice(0, 2).join('.') + '.';
const testVideoId = await generateVideo({
video_inputs: [{
character: { type: "avatar", avatar_id: avatarId, avatar_style: "normal" },
voice: { type: "text", input_text: testScript, voice_id: voiceId },
}],
dimension: { width: 1280, height: 720 }, // Lower res for test
});
```
## Voice Speed Adjustment
Adjust delivery speed in the voice configuration:
```typescript
voice: {
type: "text",
input_text: script,
voice_id: "voice_id",
speed: 1.1, // Slightly faster (range: 0.5 - 2.0)
}
```
| Speed | Effect | Use Case |
|-------|--------|----------|
| 0.8-0.9 | Slower, deliberate | Complex topics, older audiences |
| 1.0 | Normal | General use |
| 1.1-1.2 | Slightly faster | Energetic content, younger audiences |
| 1.3+ | Fast | Use sparingly, may reduce clarity |
See [voices.md](voices.md) for full voice configuration options.
.claude/skills/heygen/references/templates.md
+433
View File
@@ -0,0 +1,433 @@
---
name: templates
description: Template listing and variable replacement for HeyGen videos
---
# Video Templates
HeyGen templates allow you to create reusable video structures with variable placeholders, enabling personalized video generation at scale.
## Listing Templates
### curl
```bash
curl -X GET "https://api.heygen.com/v2/templates" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface Template {
template_id: string;
name: string;
thumbnail_url: string;
variables: TemplateVariable[];
}
interface TemplateVariable {
name: string;
type: "text" | "image" | "audio";
properties?: {
max_length?: number;
default_value?: string;
};
}
interface TemplatesResponse {
error: null | string;
data: {
templates: Template[];
};
}
async function listTemplates(): Promise<Template[]> {
const response = await fetch("https://api.heygen.com/v2/templates", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const json: TemplatesResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.templates;
}
```
### Python
```python
import requests
import os
def list_templates() -> list:
response = requests.get(
"https://api.heygen.com/v2/templates",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]["templates"]
```
## Response Format
```json
{
"error": null,
"data": {
"templates": [
{
"template_id": "template_abc123",
"name": "Product Announcement",
"thumbnail_url": "https://files.heygen.ai/...",
"variables": [
{
"name": "product_name",
"type": "text",
"properties": {
"max_length": 50
}
},
{
"name": "presenter_script",
"type": "text",
"properties": {
"max_length": 500
}
},
{
"name": "product_image",
"type": "image"
}
]
}
]
}
}
```
## Getting Template Details
### curl
```bash
curl -X GET "https://api.heygen.com/v2/template/{template_id}" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
async function getTemplate(templateId: string): Promise<Template> {
const response = await fetch(
`https://api.heygen.com/v2/template/${templateId}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data;
}
```
## Generating Video from Template
### Request Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `variables` | object | ✓ | Key-value pairs matching template variables |
| `test` | boolean | | Test mode (watermarked, no credits) |
| `title` | string | | Video name for organization |
| `callback_id` | string | | Custom ID for webhook tracking |
| `callback_url` | string | | URL for completion notification |
**Note:** The `variables` object keys must match the template's defined variable names. Check template details to see which variables are defined.
### curl
```bash
curl -X POST "https://api.heygen.com/v2/template/{template_id}/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"test": false,
"variables": {
"product_name": "SuperWidget Pro",
"presenter_script": "Introducing our latest innovation!",
"product_image": "https://example.com/product.jpg"
}
}'
```
### TypeScript
```typescript
interface TemplateGenerateRequest {
variables: Record<string, string>; // Required
test?: boolean;
title?: string;
callback_id?: string;
callback_url?: string;
}
interface TemplateGenerateResponse {
error: null | string;
data: {
video_id: string;
};
}
async function generateFromTemplate(
templateId: string,
variables: Record<string, string>,
test: boolean = false
): Promise<string> {
const response = await fetch(
`https://api.heygen.com/v2/template/${templateId}/generate`,
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({ test, variables }),
}
);
const json: TemplateGenerateResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.video_id;
}
```
### Python
```python
def generate_from_template(template_id: str, variables: dict, test: bool = False) -> str:
response = requests.post(
f"https://api.heygen.com/v2/template/{template_id}/generate",
headers={
"X-Api-Key": os.environ["HEYGEN_API_KEY"],
"Content-Type": "application/json"
},
json={
"test": test,
"variables": variables
}
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]["video_id"]
```
## Variable Types
### Text Variables
For dynamic text content:
```typescript
const variables = {
customer_name: "John Smith",
product_name: "SuperWidget Pro",
price: "$99.99",
cta_text: "Order Now!",
};
```
### Image Variables
For dynamic images (backgrounds, product shots):
```typescript
const variables = {
product_image: "https://example.com/product.jpg",
logo: "https://example.com/logo.png",
background: "https://example.com/bg.jpg",
};
```
### Audio Variables
For custom audio content:
```typescript
const variables = {
background_music: "https://example.com/music.mp3",
custom_voiceover: "https://example.com/voiceover.mp3",
};
```
## Batch Video Generation
Generate multiple personalized videos from a template:
```typescript
interface PersonalizationData {
name: string;
email: string;
company: string;
customMessage: string;
}
async function batchGenerateVideos(
templateId: string,
recipients: PersonalizationData[]
): Promise<string[]> {
const videoIds: string[] = [];
for (const recipient of recipients) {
const variables = {
recipient_name: recipient.name,
company_name: recipient.company,
personalized_message: recipient.customMessage,
};
const videoId = await generateFromTemplate(templateId, variables);
videoIds.push(videoId);
// Rate limiting: add delay between requests
await new Promise((r) => setTimeout(r, 1000));
}
return videoIds;
}
// Usage
const recipients = [
{
name: "John Smith",
email: "john@example.com",
company: "Acme Inc",
customMessage: "Thanks for your interest in our product!",
},
{
name: "Jane Doe",
email: "jane@example.com",
company: "Tech Corp",
customMessage: "We'd love to show you a demo!",
},
];
const videoIds = await batchGenerateVideos("template_abc123", recipients);
```
## Template Validation
Validate variables before generating:
```typescript
function validateTemplateVariables(
template: Template,
variables: Record<string, string>
): { valid: boolean; errors: string[] } {
const errors: string[] = [];
for (const templateVar of template.variables) {
const value = variables[templateVar.name];
// Check if required variable is provided
if (!value) {
errors.push(`Missing required variable: ${templateVar.name}`);
continue;
}
// Check text length limits
if (templateVar.type === "text" && templateVar.properties?.max_length) {
if (value.length > templateVar.properties.max_length) {
errors.push(
`Variable "${templateVar.name}" exceeds max length of ${templateVar.properties.max_length}`
);
}
}
// Validate image URLs
if (templateVar.type === "image") {
try {
new URL(value);
} catch {
errors.push(`Variable "${templateVar.name}" is not a valid URL`);
}
}
}
return {
valid: errors.length === 0,
errors,
};
}
```
## Complete Template Workflow
```typescript
async function createPersonalizedVideo(
templateId: string,
personalization: Record<string, string>
): Promise<string> {
// 1. Get template details
const template = await getTemplate(templateId);
console.log(`Using template: ${template.name}`);
// 2. Validate variables
const validation = validateTemplateVariables(template, personalization);
if (!validation.valid) {
throw new Error(`Validation errors: ${validation.errors.join(", ")}`);
}
// 3. Generate video
console.log("Generating video...");
const videoId = await generateFromTemplate(templateId, personalization);
console.log(`Video ID: ${videoId}`);
// 4. Wait for completion
const videoUrl = await waitForVideo(videoId);
console.log(`Video ready: ${videoUrl}`);
return videoUrl;
}
// Usage
const videoUrl = await createPersonalizedVideo("template_abc123", {
customer_name: "John Smith",
product_name: "SuperWidget Pro",
offer_details: "Get 20% off your first order!",
});
```
## Best Practices
1. **Design for flexibility** - Create templates with generic placeholders
2. **Set reasonable limits** - Define max lengths for text variables
3. **Validate inputs** - Check variable values before generating
4. **Use test mode** - Test with `test: true` to verify before production
5. **Implement rate limiting** - Add delays for batch generation
6. **Cache template data** - Reduce API calls by caching template details
7. **Error handling** - Gracefully handle generation failures
## Use Cases
- **Sales outreach** - Personalized prospect videos
- **Customer onboarding** - Welcome videos with customer name
- **Product updates** - Announcements with dynamic content
- **Training** - Customized training modules
- **Marketing campaigns** - Targeted promotional videos
.claude/skills/heygen/references/text-overlays.md
+316
View File
@@ -0,0 +1,316 @@
---
name: text-overlays
description: Adding text overlays with fonts and positioning to HeyGen videos
---
# Text Overlays
Add text overlays to your HeyGen videos for titles, captions, lower thirds, and other on-screen text elements.
## Basic Text Overlay
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Welcome to our presentation!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#1a1a2e",
},
},
],
// Text overlay configuration (if supported in your API tier)
// Note: Availability varies by plan
};
```
## Text Overlay Configuration
Text overlays typically support these properties:
```typescript
interface TextOverlay {
text: string;
x: number; // X position (pixels or percentage)
y: number; // Y position (pixels or percentage)
width?: number; // Text box width
height?: number; // Text box height
font_family?: string;
font_size?: number;
font_color?: string;
background_color?: string;
text_align?: "left" | "center" | "right";
duration?: {
start: number; // Start time in seconds
end: number; // End time in seconds
};
}
```
## Positioning Text
### Coordinate System
- **Origin**: Top-left corner (0, 0)
- **X-axis**: Increases to the right
- **Y-axis**: Increases downward
- **Units**: Typically pixels or percentage of video dimensions
### Common Positions
For a 1920x1080 video:
| Position | X | Y | Description |
|----------|---|---|-------------|
| Top-left | 50 | 50 | Upper left corner |
| Top-center | 960 | 50 | Top center |
| Top-right | 1870 | 50 | Upper right corner |
| Center | 960 | 540 | Dead center |
| Bottom-left | 50 | 1030 | Lower third left |
| Bottom-center | 960 | 1030 | Lower third center |
### Position Helper Function
```typescript
interface Position {
x: number;
y: number;
}
function getTextPosition(
location: "top-left" | "top-center" | "top-right" | "center" | "bottom-left" | "bottom-center" | "bottom-right",
videoWidth: number,
videoHeight: number,
padding: number = 50
): Position {
const positions: Record<string, Position> = {
"top-left": { x: padding, y: padding },
"top-center": { x: videoWidth / 2, y: padding },
"top-right": { x: videoWidth - padding, y: padding },
"center": { x: videoWidth / 2, y: videoHeight / 2 },
"bottom-left": { x: padding, y: videoHeight - padding },
"bottom-center": { x: videoWidth / 2, y: videoHeight - padding },
"bottom-right": { x: videoWidth - padding, y: videoHeight - padding },
};
return positions[location];
}
```
## Font Styling
### Available Font Properties
```typescript
const textStyle = {
font_family: "Arial",
font_size: 48,
font_color: "#FFFFFF",
font_weight: "bold",
background_color: "rgba(0, 0, 0, 0.5)",
text_align: "center",
};
```
### Common Font Families
| Font | Style | Use Case |
|------|-------|----------|
| Arial | Sans-serif | Clean, universal |
| Helvetica | Sans-serif | Modern, professional |
| Times New Roman | Serif | Traditional, formal |
| Georgia | Serif | Elegant, readable |
| Roboto | Sans-serif | Modern, digital |
| Open Sans | Sans-serif | Friendly, accessible |
## Common Text Overlay Patterns
### Title Card
```typescript
const titleOverlay = {
text: "Product Demo",
x: 960,
y: 540,
font_family: "Arial",
font_size: 72,
font_color: "#FFFFFF",
text_align: "center",
duration: {
start: 0,
end: 3,
},
};
```
### Lower Third (Name/Title)
```typescript
const lowerThirdOverlay = {
text: "John Smith\nCEO, Company Inc.",
x: 100,
y: 900,
font_family: "Arial",
font_size: 36,
font_color: "#FFFFFF",
background_color: "rgba(0, 102, 204, 0.9)",
text_align: "left",
duration: {
start: 2,
end: 8,
},
};
```
### Call to Action
```typescript
const ctaOverlay = {
text: "Visit example.com",
x: 960,
y: 1000,
font_family: "Arial",
font_size: 42,
font_color: "#FFD700",
text_align: "center",
duration: {
start: 25,
end: 30,
},
};
```
## Creating Text Overlay Templates
```typescript
interface TextOverlayTemplate {
name: string;
style: Partial<TextOverlay>;
}
const templates: TextOverlayTemplate[] = [
{
name: "title",
style: {
font_family: "Arial",
font_size: 72,
font_color: "#FFFFFF",
text_align: "center",
},
},
{
name: "subtitle",
style: {
font_family: "Arial",
font_size: 42,
font_color: "#CCCCCC",
text_align: "center",
},
},
{
name: "lower-third",
style: {
font_family: "Arial",
font_size: 36,
font_color: "#FFFFFF",
background_color: "rgba(0, 0, 0, 0.7)",
text_align: "left",
},
},
{
name: "caption",
style: {
font_family: "Arial",
font_size: 32,
font_color: "#FFFFFF",
background_color: "rgba(0, 0, 0, 0.5)",
text_align: "center",
},
},
];
function createTextOverlay(
text: string,
templateName: string,
position: Position,
duration?: { start: number; end: number }
): TextOverlay {
const template = templates.find((t) => t.name === templateName);
if (!template) {
throw new Error(`Template "${templateName}" not found`);
}
return {
text,
x: position.x,
y: position.y,
...template.style,
duration,
};
}
```
## Timing Text Overlays
Coordinate text appearance with your script:
```typescript
// Script with timing markers
const script = `
Hello and welcome. [0:00 - 0:03]
Let me show you our features. [0:03 - 0:08]
First, we have analytics. [0:08 - 0:15]
Get started today! [0:15 - 0:20]
`;
// Matching text overlays
const overlays = [
{
text: "Welcome",
duration: { start: 0, end: 3 },
...titleStyle,
},
{
text: "Feature Overview",
duration: { start: 3, end: 8 },
...subtitleStyle,
},
{
text: "Analytics Dashboard",
duration: { start: 8, end: 15 },
...lowerThirdStyle,
},
{
text: "www.example.com",
duration: { start: 15, end: 20 },
...ctaStyle,
},
];
```
## Best Practices
1. **Readability** - Use sufficient contrast between text and background
2. **Size** - Ensure text is large enough to read on mobile devices
3. **Duration** - Give viewers enough time to read (rule of thumb: 3 seconds minimum)
4. **Positioning** - Don't overlap with the avatar's face
5. **Consistency** - Use consistent fonts and styles throughout
6. **Accessibility** - Consider color-blind friendly palettes
## Limitations
- Text overlay support varies by subscription tier
- Some advanced styling options may not be available via API
- Complex animations may require post-production tools
- For auto-generated captions, see [captions.md](captions.md)
.claude/skills/heygen/references/video-agent.md
+347
View File
@@ -0,0 +1,347 @@
---
name: video-agent
description: One-shot prompt video generation with HeyGen Video Agent API
---
# Video Agent API
The Video Agent API generates complete videos from a single text prompt. Unlike the standard video generation API which requires detailed scene-by-scene configuration, Video Agent automatically handles script writing, avatar selection, visuals, voiceover, pacing, and captions.
## MCP Tool (Preferred)
If the HeyGen MCP server is connected, use `mcp__heygen__generate_video_agent` instead of direct API calls:
```
Tool: mcp__heygen__generate_video_agent
Parameters:
prompt: "<optimized prompt from prompt-optimizer.md>"
config:
duration_sec: 90 # optional, 5-300
avatar_id: "avatar_id" # optional, agent selects if omitted
orientation: "landscape" # optional, "landscape" or "portrait"
files: # optional
- asset_id: "uploaded_asset_id"
```
Then check status with `mcp__heygen__get_video` using the returned `video_id`.
The prompt quality is still the critical factor — always follow [prompt-optimizer.md](prompt-optimizer.md) regardless of whether you use MCP or direct API.
## When to Use Video Agent vs Standard API
| Use Case | Recommended API |
|----------|-----------------|
| Quick video from idea | Video Agent |
| Precise control over scenes, avatars, timing | Standard v2/video/generate |
| Automated content generation at scale | Video Agent |
| Specific avatar with exact script | Standard v2/video/generate |
| Prototype or draft video | Video Agent |
| Brand-consistent production video | Standard v2/video/generate |
## Before You Call This API
**Required step:** Optimize your prompt using [prompt-optimizer.md](prompt-optimizer.md) before generating a video. The difference between mediocre and professional results depends entirely on prompt quality.
Quick checklist:
1. Define visual style (colors, aesthetic) — see [visual-styles.md](visual-styles.md)
2. Structure scenes with specific scene types
3. Write VO script at ~150 words/minute
4. Specify media types for each scene (Motion Graphics, Stock, AI-generated)
## Direct API Endpoint
```
POST https://api.heygen.com/v1/video_agent/generate
```
## Request Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `prompt` | string | ✓ | Text prompt describing the video you want |
| `config` | object | | Configuration options (see below) |
| `files` | array | | Asset files to reference in generation |
| `callback_id` | string | | Custom ID for tracking. **Requires `callback_url` to also be set** — omit both if you don't need webhooks |
| `callback_url` | string | | Webhook URL for completion notification |
### Config Object
| Field | Type | Description |
|-------|------|-------------|
| `duration_sec` | integer | Approximate duration in seconds (5-300) |
| `avatar_id` | string | Specific avatar to use (optional - agent selects if not provided) |
| `orientation` | string | `"portrait"` or `"landscape"` |
### Files Array
| Field | Type | Description |
|-------|------|-------------|
| `asset_id` | string | Asset ID of uploaded file to reference |
## Response Format
```json
{
"error": null,
"data": {
"video_id": "abc123"
}
}
```
## curl Example
```bash
curl -X POST "https://api.heygen.com/v1/video_agent/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "Create a 60-second product demo video for a new AI-powered calendar app. The tone should be professional but friendly, targeting busy professionals. Highlight the smart scheduling feature and time zone handling."
}'
```
## TypeScript
```typescript
interface VideoAgentConfig {
duration_sec?: number; // 5-300 seconds
avatar_id?: string; // Optional: specific avatar
orientation?: "portrait" | "landscape";
}
interface VideoAgentFile {
asset_id: string;
}
interface VideoAgentRequest {
prompt: string; // Required
config?: VideoAgentConfig;
files?: VideoAgentFile[];
callback_id?: string; // Requires callback_url if set
callback_url?: string;
}
interface VideoAgentResponse {
error: string | null;
data: {
video_id: string;
};
}
async function generateWithVideoAgent(
prompt: string,
config?: VideoAgentConfig
): Promise<string> {
const request: VideoAgentRequest = { prompt };
if (config) {
request.config = config;
}
const response = await fetch(
"https://api.heygen.com/v1/video_agent/generate",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(request),
}
);
const json: VideoAgentResponse = await response.json();
if (json.error) {
throw new Error(`Video Agent failed: ${json.error}`);
}
return json.data.video_id;
}
```
## Python
```python
import requests
import os
from typing import Optional
def generate_with_video_agent(
prompt: str,
duration_sec: Optional[int] = None,
avatar_id: Optional[str] = None,
orientation: Optional[str] = None
) -> str:
request_body = {"prompt": prompt}
config = {}
if duration_sec:
config["duration_sec"] = duration_sec
if avatar_id:
config["avatar_id"] = avatar_id
if orientation:
config["orientation"] = orientation
if config:
request_body["config"] = config
response = requests.post(
"https://api.heygen.com/v1/video_agent/generate",
headers={
"X-Api-Key": os.environ["HEYGEN_API_KEY"],
"Content-Type": "application/json"
},
json=request_body
)
data = response.json()
if data.get("error"):
raise Exception(f"Video Agent failed: {data['error']}")
return data["data"]["video_id"]
```
## Examples
### Basic: Prompt Only
```typescript
const videoId = await generateWithVideoAgent(
"Create a 30-second welcome video for new employees at a tech startup. Keep it energetic and modern."
);
```
### With Duration and Orientation
```typescript
const videoId = await generateWithVideoAgent(
"Explain the benefits of cloud computing for small businesses. Use simple language and real-world examples.",
{
duration_sec: 90,
orientation: "landscape"
}
);
```
### With Specific Avatar
```typescript
const videoId = await generateWithVideoAgent(
"Present quarterly sales results. Professional tone, data-focused.",
{
duration_sec: 120,
avatar_id: "josh_lite3_20230714",
orientation: "landscape"
}
);
```
### With Reference Files
Upload assets first, then reference them:
```typescript
// 1. Upload reference materials (see assets.md)
const logoAssetId = await uploadFile("./company-logo.png", "image/png");
const productImageId = await uploadFile("./product-screenshot.png", "image/png");
// 2. Generate video with references
const response = await fetch(
"https://api.heygen.com/v1/video_agent/generate",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
prompt: "Create a product demo video showcasing our new dashboard feature. Use the uploaded screenshots as visual references.",
config: {
duration_sec: 60,
orientation: "landscape"
},
files: [
{ asset_id: logoAssetId },
{ asset_id: productImageId }
]
}),
}
);
```
## Writing Effective Prompts
See **[prompt-optimizer.md](prompt-optimizer.md)** for comprehensive prompt writing guidance.
The prompt optimizer covers:
- Prompt complexity levels (basic → scene-by-scene)
- Visual style taxonomy and color specification
- Media type selection (Motion Graphics vs Stock vs AI-generated)
- Scene structure and timing calculations
- Ready-to-use templates for common video types
## Checking Video Status
Video Agent returns a `video_id` - use the standard status endpoint to check progress:
```typescript
// Same polling as standard video generation
const videoUrl = await waitForVideo(videoId);
```
See [video-status.md](video-status.md) for polling implementation.
## Comparison: Video Agent vs Standard API
### Video Agent Request
```typescript
// Simple: describe what you want
const videoId = await generateWithVideoAgent(
"Create a 60-second tutorial on setting up two-factor authentication. Professional tone, step-by-step."
);
```
### Equivalent Standard API Request
```typescript
// Complex: specify every detail
const videoId = await generateVideo({
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Welcome to this tutorial on two-factor authentication...",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#1a1a2e",
},
},
// ... more scenes for each step
],
dimension: { width: 1920, height: 1080 },
});
```
## Limitations
- Less control over exact script wording
- Avatar selection may vary if not specified
- Scene composition is automated
- May not match precise brand guidelines
- Duration is approximate, not exact
## Best Practices
1. **Be specific in prompts** - More detail = better results
2. **Specify duration** - Use `config.duration_sec` for predictable length
3. **Lock avatar if needed** - Use `config.avatar_id` for consistency
4. **Upload reference files** - Help agent understand your brand/product
5. **Iterate on prompts** - Refine based on results
6. **Use for drafts** - Video Agent is great for quick iterations before final production
.claude/skills/heygen/references/video-generation.md
+770
View File
@@ -0,0 +1,770 @@
---
name: video-generation
description: POST /v2/video/generate workflow and multi-scene videos for HeyGen
---
# Video Generation
## Table of Contents
- [Video Output Formats](#video-output-formats)
- [Basic Video Generation](#basic-video-generation)
- [Request Fields](#request-fields)
- [Video Configuration Options](#video-configuration-options)
- [Multi-Scene Videos](#multi-scene-videos)
- [Using Different Character Types](#using-different-character-types)
- [Voice Input Types](#voice-input-types)
- [Complete Workflow Example](#complete-workflow-example)
- [Error Handling](#error-handling)
- [Script Length Limits](#script-length-limits)
- [Adding Pauses to Scripts](#adding-pauses-to-scripts)
- [Test Mode](#test-mode)
- [Production-Ready Workflow](#production-ready-workflow)
- [Transparent Background Videos (WebM)](#transparent-background-videos-webm)
- [Best Practices](#best-practices)
---
The `/v2/video/generate` endpoint is the primary way to create AI avatar videos with HeyGen.
## Video Output Formats
| Endpoint | Format | Use Case |
|----------|--------|----------|
| `/v2/video/generate` | MP4 | **Standard** - videos with background (most common) |
| `/v1/video.webm` | WebM | Transparent background - only when needed |
Use MP4 with background for most cases. WebM is only needed when you want to see content *behind* the avatar (e.g., overlaying avatar on a screen recording).
## Basic Video Generation
### curl
```bash
curl -X POST "https://api.heygen.com/v2/video/generate" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"video_inputs": [
{
"character": {
"type": "avatar",
"avatar_id": "josh_lite3_20230714",
"avatar_style": "normal"
},
"voice": {
"type": "text",
"input_text": "Hello! Welcome to HeyGen.",
"voice_id": "1bd001e7e50f421d891986aad5158bc8"
}
}
],
"dimension": {
"width": 1920,
"height": 1080
}
}'
```
## Request Fields
### Top-Level Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `video_inputs` | array | ✓ | Array of 1-50 video input objects |
| `dimension` | object | | Video dimensions `{width, height}` |
| `title` | string | | Video name for organization |
| `test` | boolean | | Test mode (watermarked, no credits) |
| `caption` | boolean | | Enable auto-captions |
| `callback_id` | string | | Custom ID for webhook tracking |
| `callback_url` | string | | URL for completion notification |
| `folder_id` | string | | Storage folder ID |
### video_inputs[].character Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `type` | string | ✓ | `"avatar"` or `"talking_photo"` |
| `avatar_id` | string | ✓* | Avatar ID (*required when type is "avatar") |
| `talking_photo_id` | string | ✓* | Photo ID (*required when type is "talking_photo") |
| `avatar_style` | string | | `"normal"`, `"closeUp"`, or `"circle"` |
| `scale` | number | | Avatar scale factor |
| `offset` | object | | Position offset `{x, y}` |
### video_inputs[].voice Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `type` | string | ✓ | `"text"`, `"audio"`, or `"silence"` |
| `voice_id` | string | ✓* | Voice ID (*required when type is "text") |
| `input_text` | string | ✓* | Script text (*required when type is "text") |
| `audio_url` | string | ✓* | Audio URL (*required when type is "audio") |
| `duration` | number | ✓* | Duration in seconds (*required when type is "silence") |
| `speed` | number | | Speech speed 0.5-2.0 (default 1.0) |
| `pitch` | number | | Voice pitch -20 to 20 (default 0) |
### video_inputs[].background Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `type` | string | | `"color"`, `"image"`, or `"video"` |
| `value` | string | | Hex color (when type is "color") |
| `url` | string | | Image/video URL (when type is "image"/"video") |
| `fit` | string | | `"cover"` or `"contain"` |
### TypeScript
```typescript
// Required fields have no '?' - optional fields have '?'
interface VideoInput {
character: {
type: "avatar" | "talking_photo"; // Required
avatar_id?: string; // Required when type="avatar"
talking_photo_id?: string; // Required when type="talking_photo"
avatar_style?: "normal" | "closeUp" | "circle";
scale?: number;
offset?: { x: number; y: number };
};
voice: {
type: "text" | "audio" | "silence"; // Required
input_text?: string; // Required when type="text"
voice_id?: string; // Required when type="text"
audio_url?: string; // Required when type="audio"
duration?: number; // Required when type="silence"
speed?: number;
pitch?: number;
};
background?: {
type?: "color" | "image" | "video";
value?: string;
url?: string;
fit?: "cover" | "contain";
};
}
interface VideoGenerateRequest {
video_inputs: VideoInput[]; // Required
dimension?: { width: number; height: number };
test?: boolean;
title?: string;
caption?: boolean;
callback_id?: string;
callback_url?: string;
folder_id?: string;
}
interface VideoGenerateResponse {
error: null | string;
data: {
video_id: string;
};
}
async function generateVideo(config: VideoGenerateRequest): Promise<string> {
const response = await fetch("https://api.heygen.com/v2/video/generate", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
});
const json: VideoGenerateResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.video_id;
}
```
### Python
```python
import requests
import os
def generate_video(config: dict) -> str:
response = requests.post(
"https://api.heygen.com/v2/video/generate",
headers={
"X-Api-Key": os.environ["HEYGEN_API_KEY"],
"Content-Type": "application/json"
},
json=config
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]["video_id"]
```
## Video Configuration Options
### Full Configuration Example
```typescript
const fullConfig: VideoGenerateRequest = {
// Test mode (no credits consumed, watermarked output)
test: false,
// Video title (for organization)
title: "Product Demo Video",
// Video dimensions
dimension: {
width: 1920,
height: 1080,
},
// Video scenes/inputs
video_inputs: [
{
// Avatar configuration
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
// Voice configuration
voice: {
type: "text",
input_text: "Welcome to our product demonstration!",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
speed: 1.0,
pitch: 0,
},
// Background configuration
background: {
type: "color",
value: "#FFFFFF",
},
},
],
};
```
## Multi-Scene Videos
Create videos with multiple scenes:
```typescript
const multiSceneConfig = {
video_inputs: [
// Scene 1: Introduction
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello! Today I'll show you three key features.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#1a1a2e",
},
},
// Scene 2: Feature 1
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "closeUp",
},
voice: {
type: "text",
input_text: "First, let's look at our dashboard.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "image",
url: "https://example.com/dashboard-bg.jpg",
},
},
// Scene 3: Conclusion
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Thanks for watching! Try it today.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
background: {
type: "color",
value: "#1a1a2e",
},
},
],
dimension: { width: 1920, height: 1080 },
};
```
## Using Different Character Types
### Avatar
```typescript
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal"
}
}
```
### Talking Photo
```typescript
{
character: {
type: "talking_photo",
talking_photo_id: "your_talking_photo_id"
}
}
```
## Voice Input Types
### Text-to-Speech
```typescript
{
voice: {
type: "text",
input_text: "Your script here",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
speed: 1.0, // 0.5 - 2.0
pitch: 0 // -20 to 20
}
}
```
### Custom Audio
```typescript
{
voice: {
type: "audio",
audio_url: "https://example.com/your-audio.mp3"
}
}
```
## Complete Workflow Example
```typescript
async function createVideo(script: string, avatarId: string, voiceId: string) {
// 1. Generate video
console.log("Starting video generation...");
const videoId = await generateVideo({
video_inputs: [
{
character: {
type: "avatar",
avatar_id: avatarId,
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: voiceId,
},
background: {
type: "color",
value: "#FFFFFF",
},
},
],
dimension: { width: 1920, height: 1080 },
});
console.log(`Video ID: ${videoId}`);
// 2. Poll for completion
console.log("Waiting for video completion...");
const videoUrl = await waitForVideo(videoId);
console.log(`Video ready: ${videoUrl}`);
return videoUrl;
}
// Helper function for polling
async function waitForVideo(videoId: string): Promise<string> {
const maxAttempts = 60;
const pollInterval = 10000; // 10 seconds
for (let i = 0; i < maxAttempts; i++) {
const response = await fetch(
`https://api.heygen.com/v2/videos/${videoId}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
if (data.status === "completed") {
return data.video_url;
} else if (data.status === "failed") {
throw new Error(data.failure_message || "Video generation failed");
}
await new Promise((r) => setTimeout(r, pollInterval));
}
throw new Error("Video generation timed out");
}
```
## Error Handling
```typescript
async function generateVideoSafe(config: VideoGenerateRequest) {
try {
const videoId = await generateVideo(config);
return { success: true, videoId };
} catch (error) {
// Common errors
if (error.message.includes("quota")) {
console.error("Insufficient credits");
} else if (error.message.includes("avatar")) {
console.error("Invalid avatar ID");
} else if (error.message.includes("voice")) {
console.error("Invalid voice ID");
} else if (error.message.includes("script")) {
console.error("Script too long or invalid");
}
return { success: false, error: error.message };
}
}
```
## Script Length Limits
| Tier | Max Characters |
|------|----------------|
| Free | ~500 |
| Creator | ~1,500 |
| Team | ~3,000 |
| Enterprise | ~5,000+ |
## Adding Pauses to Scripts
Use `<break>` tags to add pauses in your script:
```typescript
const script = "Welcome to our demo. <break time=\"1s\"/> Let me show you the features.";
```
**Format:** `<break time="Xs"/>` where X is seconds (e.g., `1s`, `1.5s`, `0.5s`)
**Important:** Break tags must have spaces before and after them.
See [voices.md](voices.md) for detailed break tag documentation.
## Test Mode
Use test mode during development:
```typescript
const config = {
test: true, // Watermarked output, no credits consumed
video_inputs: [...],
};
```
## Production-Ready Workflow
Complete example using avatar's default voice (recommended), proper timeouts, and retry logic:
```typescript
interface VideoGenerationResult {
videoId: string;
videoUrl: string;
duration: number;
avatarId: string;
voiceId: string;
avatarName: string;
}
async function generateAvatarVideo(
script: string,
options: {
avatarId?: string; // Specific avatar, or will pick first available
width?: number;
height?: number;
} = {}
): Promise<VideoGenerationResult> {
const { width = 1920, height = 1080 } = options;
let { avatarId } = options;
// 1. List avatars if no specific one provided
if (!avatarId) {
console.log("Listing available avatars...");
const listResponse = await fetch("https://api.heygen.com/v2/avatars", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const listData = await listResponse.json();
if (!listData.data?.avatars?.length) {
throw new Error("No avatars available");
}
avatarId = listData.data.avatars[0].avatar_id;
}
// 2. Get avatar details including default_voice_id
console.log(`Getting details for avatar: ${avatarId}`);
const detailsResponse = await fetch(
`https://api.heygen.com/v2/avatar/${avatarId}/details`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data: avatar } = await detailsResponse.json();
if (!avatar.default_voice_id) {
throw new Error(`Avatar ${avatar.name} has no default voice - select voice manually`);
}
console.log(`Using avatar: ${avatar.name} with default voice: ${avatar.default_voice_id}`);
// 3. Generate video using avatar's default voice
const videoId = await generateVideo({
video_inputs: [{
character: {
type: "avatar",
avatar_id: avatar.id, // from details response
avatar_style: "normal",
},
voice: {
type: "text",
input_text: script,
voice_id: avatar.default_voice_id, // pre-matched default voice
speed: 1.0,
},
background: {
type: "color",
value: "#1a1a2e",
},
}],
dimension: { width, height },
});
console.log(`Video ID: ${videoId}`);
// 3. Wait for completion (20 minute timeout - generation can take 15+ min)
console.log("Waiting for video generation (typically 5-15 minutes, can be longer)...");
const result = await waitForVideo(
videoId,
process.env.HEYGEN_API_KEY!,
(status, elapsed) => {
console.log(` [${Math.round(elapsed / 1000)}s] ${status}`);
},
1200000 // 20 minute timeout for safety
);
return {
videoId,
videoUrl: result.video_url!,
duration: result.duration!,
avatarId: avatar.id,
voiceId: avatar.default_voice_id,
avatarName: avatar.name,
};
}
// Usage - let it pick an avatar automatically
const result = await generateAvatarVideo(
"Hello! Welcome to our product demonstration."
);
console.log(`Video ready: ${result.videoUrl}`);
// Or specify a known avatar_id
const result2 = await generateAvatarVideo(
"Hello! Welcome to our product demonstration.",
{ avatarId: "josh_lite3_20230714" }
);
```
## Transparent Background Videos (WebM)
Use WebM **only when you need transparency** - i.e., when the avatar should be overlaid on other video content and you need to see through to what's behind.
**Don't need WebM for:**
- Avatar with motion graphics/text overlaid ON TOP of avatar
- Picture-in-picture with solid background
- Standard presenter videos
**Do need WebM for:**
- Avatar overlaid on screen recording
- Avatar floating over video background
- True alpha-channel compositing
### WebM Request Fields
**Note:** The WebM endpoint (`/v1/video.webm`) uses a different structure than `/v2/video/generate`.
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `avatar_pose_id` | string | ✓ | Avatar pose ID (from avatar details) |
| `avatar_style` | string | ✓ | `"normal"` or `"closeUp"` only (no circle) |
| `input_text` | string | ✓* | Script text (*required if not using input_audio) |
| `voice_id` | string | ✓* | Voice ID (*required with input_text) |
| `input_audio` | string | ✓* | Audio URL (*required if not using input_text) |
| `dimension` | object | | `{width, height}` (default: 1280x720) |
**Either** (`input_text` + `voice_id`) **OR** `input_audio` must be provided, but not both.
### curl
```bash
curl -X POST "https://api.heygen.com/v1/video.webm" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"avatar_pose_id": "josh_lite3_20230714",
"avatar_style": "normal",
"input_text": "Hello! This video has a transparent background.",
"voice_id": "1bd001e7e50f421d891986aad5158bc8",
"dimension": {
"width": 1920,
"height": 1080
}
}'
```
### TypeScript
```typescript
interface WebMVideoRequest {
avatar_pose_id: string; // Required
avatar_style: "normal" | "closeUp"; // Required (no circle support)
input_text?: string; // Required if not using input_audio
voice_id?: string; // Required with input_text
input_audio?: string; // Required if not using input_text
dimension?: { width: number; height: number };
}
async function generateTransparentVideo(
script: string,
avatarPoseId: string,
voiceId: string
): Promise<string> {
const response = await fetch("https://api.heygen.com/v1/video.webm", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
avatar_pose_id: avatarPoseId, // Required
avatar_style: "normal", // Required: "normal" or "closeUp"
input_text: script, // Required (with voice_id)
voice_id: voiceId, // Required (with input_text)
dimension: { width: 1920, height: 1080 },
}),
});
const { data } = await response.json();
return data.video_id;
}
```
### When to Use WebM vs MP4
| Scenario | Format | Why |
|----------|--------|-----|
| Avatar with overlays on top | **MP4** | Overlays go on top, don't need transparency |
| Standard presenter | **MP4** | Simpler, more compatible |
| Loom-style (avatar over screen recording) | **WebM** + `normal`/`closeUp` | Need transparency, crop to circle in post |
| Avatar floating over video content | **WebM** | Need to see content behind avatar |
**Note:** WebM only supports `normal` and `closeUp` styles. Circle style is not supported for WebM - apply circular masking in your video editor/Remotion instead.
### WebM Example: Loom-Style (Avatar Over Screen Recording)
Generate with `normal` or `closeUp` style (circle not supported for WebM):
```typescript
// Generate avatar with transparent background
const videoId = await fetch("https://api.heygen.com/v1/video.webm", {
method: "POST",
headers: { "X-Api-Key": apiKey, "Content-Type": "application/json" },
body: JSON.stringify({
avatar_pose_id: avatarPoseId, // Required
avatar_style: "closeUp", // Required: "normal" or "closeUp" only
input_text: script, // Required (with voice_id)
voice_id: voiceId, // Required (with input_text)
dimension: { width: 1920, height: 1080 },
}),
}).then(r => r.json()).then(d => d.data.video_id);
```
Apply circular masking in Remotion:
```tsx
import { Video, AbsoluteFill } from "remotion";
export const LoomStyleVideo: React.FC<{
screenRecordingUrl: string;
avatarWebmUrl: string;
}> = ({ screenRecordingUrl, avatarWebmUrl }) => {
return (
<AbsoluteFill>
{/* Screen recording as base layer */}
<Video src={screenRecordingUrl} style={{ width: "100%", height: "100%" }} />
{/* Avatar with circular mask applied in CSS */}
<Video
src={avatarWebmUrl}
style={{
position: "absolute",
bottom: 20,
left: 20,
width: 150,
height: 150,
borderRadius: "50%", // Circular mask
overflow: "hidden",
objectFit: "cover",
}}
/>
</AbsoluteFill>
);
};
```
### Note on Status Polling
WebM videos use the same status endpoint as MP4:
```typescript
// Same polling as regular videos
const status = await getVideoStatus(videoId);
// status.video_url will be a .webm file
```
## Best Practices
1. **Preview avatars before generating** - Download `preview_image_url` so user can see what the avatar looks like before committing to a video (see [avatars.md](avatars.md))
2. **Use avatar's default voice** - Most avatars have a `default_voice_id` that's pre-matched for natural results (see [avatars.md](avatars.md))
2. **Fallback: match gender manually** - If no default voice, ensure avatar and voice genders match (see [voices.md](voices.md))
3. **Validate inputs** - Check avatar and voice IDs before generating
4. **Use test mode** - Test configurations without consuming credits
5. **Set generous timeouts** - Use 15-20 minutes; generation often takes 10-15 min, sometimes longer
6. **Consider async patterns** - For long videos, save video_id and check status later (see [video-status.md](video-status.md))
7. **Handle errors gracefully** - Implement proper error handling
8. **Monitor progress** - Implement polling with progress feedback
9. **Optimize scripts** - Keep scripts concise and natural
10. **Consider dimensions** - Match dimensions to your use case (see [dimensions.md](dimensions.md))
.claude/skills/heygen/references/video-status.md
+502
View File
@@ -0,0 +1,502 @@
---
name: video-status
description: Polling patterns, status types, and retrieving download URLs for HeyGen videos
---
# Video Status and Polling
After generating a video, you need to poll for status until the video is complete. HeyGen processes videos asynchronously.
## MCP Tool (Preferred)
If the HeyGen MCP server is connected, use `mcp__heygen__get_video` with the `videoId` parameter. It returns status, video_url, thumbnail_url, duration, title, gif_url, captioned_video_url, and other metadata in a single call.
## Checking Video Status (Direct API)
### curl
```bash
curl -X GET "https://api.heygen.com/v2/videos/YOUR_VIDEO_ID" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface VideoStatusResponse {
error: null | string;
data: {
id: string;
status: "pending" | "processing" | "completed" | "failed";
video_url?: string;
thumbnail_url?: string;
duration?: number;
title?: string;
created_at?: string;
completed_at?: string;
gif_url?: string;
captioned_video_url?: string;
subtitle_url?: string;
folder_id?: string;
output_language?: string;
failure_code?: string;
failure_message?: string;
};
}
async function getVideoStatus(videoId: string): Promise<VideoStatusResponse["data"]> {
const response = await fetch(
`https://api.heygen.com/v2/videos/${videoId}`,
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const json: VideoStatusResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data;
}
```
### Python
```python
import requests
import os
def get_video_status(video_id: str) -> dict:
response = requests.get(
f"https://api.heygen.com/v2/videos/{video_id}",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]
```
## Video Status Types
| Status | Description |
|--------|-------------|
| `pending` | Video is queued for processing |
| `processing` | Video is being generated |
| `completed` | Video is ready for download |
| `failed` | Video generation failed |
## Expected Generation Times
Video generation typically takes **5-15 minutes**, but can exceed 20 minutes during peak load or for longer scripts.
| Factor | Impact |
|--------|--------|
| Script length | Longer scripts = significantly longer processing |
| Resolution | 1080p takes longer than 720p |
| Avatar complexity | Some avatars render faster |
| Queue load | Peak hours may cause 15-20+ minute waits |
| Multiple scenes | Each scene adds processing time |
**Recommendations**:
- Set timeout to **15-20 minutes** (900,000-1,200,000 ms) for safety
- For scripts > 2 minutes of speech, expect 15+ minutes
- Consider async patterns (save video_id, check later) for long videos
## Response Format
### Completed Video
```json
{
"error": null,
"data": {
"id": "abc123",
"status": "completed",
"video_url": "https://files.heygen.ai/video/abc123.mp4",
"thumbnail_url": "https://files.heygen.ai/thumbnail/abc123.jpg",
"duration": 45.2,
"title": "My Video",
"created_at": "2024-01-15T10:30:00Z",
"completed_at": "2024-01-15T10:38:00Z",
"gif_url": "https://files.heygen.ai/gif/abc123.gif",
"captioned_video_url": null,
"subtitle_url": null,
"folder_id": null,
"output_language": "en"
}
}
```
### Failed Video
```json
{
"error": null,
"data": {
"id": "abc123",
"status": "failed",
"failure_code": "script_too_long",
"failure_message": "Script too long for selected avatar"
}
}
```
## Polling Implementation
### Basic Polling
```typescript
async function waitForVideo(
videoId: string,
maxWaitMs = 600000, // 10 minutes
pollIntervalMs = 5000 // 5 seconds
): Promise<string> {
const startTime = Date.now();
while (Date.now() - startTime < maxWaitMs) {
const status = await getVideoStatus(videoId);
switch (status.status) {
case "completed":
return status.video_url!;
case "failed":
throw new Error(status.failure_message || "Video generation failed");
case "pending":
case "processing":
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
break;
}
}
throw new Error("Video generation timed out");
}
```
### Polling with Progress Callback
```typescript
type ProgressCallback = (status: string, elapsed: number) => void;
async function waitForVideoWithProgress(
videoId: string,
onProgress?: ProgressCallback,
maxWaitMs = 600000,
pollIntervalMs = 5000
): Promise<string> {
const startTime = Date.now();
while (Date.now() - startTime < maxWaitMs) {
const elapsed = Date.now() - startTime;
const status = await getVideoStatus(videoId);
onProgress?.(status.status, elapsed);
switch (status.status) {
case "completed":
return status.video_url!;
case "failed":
throw new Error(status.failure_message || "Video generation failed");
default:
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
}
}
throw new Error("Video generation timed out");
}
// Usage
const videoUrl = await waitForVideoWithProgress(
videoId,
(status, elapsed) => {
console.log(`Status: ${status}, Elapsed: ${Math.round(elapsed / 1000)}s`);
}
);
```
### Python Polling
```python
import time
from typing import Optional, Callable
def wait_for_video(
video_id: str,
max_wait_seconds: int = 600,
poll_interval: int = 5,
on_progress: Optional[Callable[[str, int], None]] = None
) -> str:
start_time = time.time()
while time.time() - start_time < max_wait_seconds:
elapsed = int(time.time() - start_time)
status_data = get_video_status(video_id)
status = status_data["status"]
if on_progress:
on_progress(status, elapsed)
if status == "completed":
return status_data["video_url"]
elif status == "failed":
raise Exception(status_data.get("failure_message", "Video generation failed"))
time.sleep(poll_interval)
raise Exception("Video generation timed out")
# Usage
def progress_callback(status: str, elapsed: int):
print(f"Status: {status}, Elapsed: {elapsed}s")
video_url = wait_for_video(video_id, on_progress=progress_callback)
```
## Downloading the Video
Once the video is complete, download it. **Important**: The video URL may not be immediately available after status shows "completed". Use retry logic with backoff.
### TypeScript (with retry)
```typescript
import fs from "fs";
import path from "path";
async function downloadVideoWithRetry(
videoUrl: string,
outputPath = "./output/video.mp4",
maxRetries = 5,
initialDelayMs = 2000
): Promise<void> {
let lastError: Error | null = null;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await fetch(videoUrl);
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
const arrayBuffer = await response.arrayBuffer();
fs.writeFileSync(path.resolve(outputPath), Buffer.from(arrayBuffer));
console.log(`Video downloaded to ${outputPath}`);
return;
} catch (error) {
lastError = error as Error;
const delay = initialDelayMs * Math.pow(2, attempt); // Exponential backoff
console.log(`Download attempt ${attempt + 1} failed, retrying in ${delay}ms...`);
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
throw new Error(`Failed to download after ${maxRetries} attempts: ${lastError?.message}`);
}
```
### Python (with retry)
```python
import requests
import time
def download_video_with_retry(
video_url: str,
output_path: str,
max_retries: int = 5,
initial_delay: float = 2.0
) -> None:
last_error = None
for attempt in range(max_retries):
try:
response = requests.get(video_url, stream=True, timeout=60)
response.raise_for_status()
with open(output_path, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
print(f"Video downloaded to {output_path}")
return
except Exception as e:
last_error = e
delay = initial_delay * (2 ** attempt) # Exponential backoff
print(f"Download attempt {attempt + 1} failed, retrying in {delay}s...")
time.sleep(delay)
raise Exception(f"Failed to download after {max_retries} attempts: {last_error}")
```
### Simple Download (no retry)
For quick scripts where you'll retry manually:
```typescript
async function downloadVideo(videoUrl: string, outputPath = "./output/video.mp4") {
const response = await fetch(videoUrl);
if (!response.ok) {
throw new Error(`Failed to download: ${response.status}`);
}
const arrayBuffer = await response.arrayBuffer();
fs.writeFileSync(path.resolve(outputPath), Buffer.from(arrayBuffer));
}
```
## Complete Workflow Example
```typescript
async function generateAndDownloadVideo(config: VideoConfig): Promise<string> {
// 1. Generate video
const generateResponse = await fetch(
"https://api.heygen.com/v2/video/generate",
{
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
}
);
const { data: generateData } = await generateResponse.json();
const videoId = generateData.video_id;
console.log(`Video ID: ${videoId}`);
// 2. Poll for completion
const videoUrl = await waitForVideoWithProgress(
videoId,
(status, elapsed) => {
console.log(`[${Math.round(elapsed / 1000)}s] Status: ${status}`);
}
);
// 3. Download
const outputPath = `./output/${videoId}.mp4`;
await downloadVideo(videoUrl, outputPath);
return outputPath;
}
```
## Resumable Status Checking
For long-running generations, save the video_id and check status later rather than keeping a process waiting.
### Save State After Generation
```typescript
interface PendingVideo {
videoId: string;
createdAt: string;
script: string;
avatarId: string;
voiceId: string;
}
async function startVideoGeneration(config: VideoGenerateRequest): Promise<PendingVideo> {
const videoId = await generateVideo(config);
const pending: PendingVideo = {
videoId,
createdAt: new Date().toISOString(),
script: config.video_inputs[0].voice.input_text!,
avatarId: config.video_inputs[0].character.avatar_id!,
voiceId: config.video_inputs[0].voice.voice_id!,
};
// Save to file for later retrieval
fs.writeFileSync("pending-video.json", JSON.stringify(pending, null, 2));
console.log(`Video generation started. ID: ${videoId}`);
console.log("Check status later with: checkVideoStatus()");
return pending;
}
```
### Check Status Later
```typescript
async function checkVideoStatus(): Promise<void> {
if (!fs.existsSync("pending-video.json")) {
console.log("No pending video found");
return;
}
const pending: PendingVideo = JSON.parse(
fs.readFileSync("pending-video.json", "utf-8")
);
const elapsed = Date.now() - new Date(pending.createdAt).getTime();
console.log(`Checking video ${pending.videoId} (started ${Math.round(elapsed / 60000)} min ago)...`);
const status = await getVideoStatus(pending.videoId);
switch (status.status) {
case "completed":
console.log(`Video ready: ${status.video_url}`);
console.log(`Duration: ${status.duration}s`);
// Clean up pending file
fs.unlinkSync("pending-video.json");
// Save result
fs.writeFileSync("video-result.json", JSON.stringify({
...pending,
videoUrl: status.video_url,
thumbnailUrl: status.thumbnail_url,
duration: status.duration,
title: status.title,
createdAt: status.created_at,
completedAt: status.completed_at,
}, null, 2));
break;
case "failed":
console.error(`Video failed: ${status.failure_message}`);
fs.unlinkSync("pending-video.json");
break;
default:
console.log(`Status: ${status.status} - check again in a few minutes`);
}
}
```
### CLI-Friendly Pattern
```typescript
// generate-video.ts - Start generation and exit
async function main() {
const pending = await startVideoGeneration(config);
console.log(`\nVideo ID saved. Run 'npx tsx check-status.ts' to check progress.`);
process.exit(0); // Exit immediately, don't wait
}
// check-status.ts - Check and optionally wait
async function main() {
const args = process.argv.slice(2);
const shouldWait = args.includes("--wait");
if (shouldWait) {
// Poll until complete (with 20 min timeout)
const result = await waitForVideo(pending.videoId, apiKey, onProgress, 1200000);
console.log(`Done: ${result.video_url}`);
} else {
// Just check once and report
await checkVideoStatus();
}
}
```
## Alternative: Using Webhooks
Instead of polling, you can use webhooks to receive notifications when videos complete. See [webhooks.md](webhooks.md) for details. Webhooks are ideal for production systems where you don't want to maintain polling connections.
## Best Practices
1. **Use exponential backoff** - Increase poll intervals for long-running jobs
2. **Set reasonable timeouts** - Most videos complete within 10 minutes
3. **Handle failures gracefully** - Check error messages for actionable feedback
4. **Consider webhooks** - For production systems, webhooks are more efficient than polling
5. **Cache video URLs** - Downloaded video URLs are valid for a limited time
.claude/skills/heygen/references/visual-styles.md
+367
View File
@@ -0,0 +1,367 @@
---
name: visual-styles
description: 20 named visual styles for Video Agent prompts — each with colors, typography, motion, and transitions
---
# Visual Style Library — 20 Styles
Named visual styles for Video Agent prompts. Each is inspired by a real graphic designer. Ordered by mood intensity.
**Picking a style:** Match mood first, content second. Ask: *"What should the viewer FEEL?"*
**Using a style:** Copy the style block into your prompt's STYLE section. Use the visual language rules — don't inject the example B-roll scenes (they confuse the agent).
**Custom styles:** These are examples. Create your own by combining elements, referencing other designers, art movements, or cultural aesthetics. The pattern: **named style + designer reference + color palette + typography + motion rules + transitions.**
## Quick Reference
| # | Style | Artist | Mood | Best For |
|---|---|---|---|---|
| 1 | Soft Signal | Sagmeister | Intimate, warm | Personal stories, wellness |
| 2 | Warm Grain | Eksell | Organic, friendly | Environmental, sustainability |
| 3 | Quiet Drama | Ray | Humanist, contemplative | Profiles, biographical |
| 4 | Heritage Reel | Cassandre | Nostalgic, vintage | History, retrospectives |
| 5 | Silk Route | Abedini | Flowing, mysterious | Global affairs, cross-cultural |
| 6 | Swiss Pulse | Müller-Brockmann | Clinical, precise | Data-heavy, analytical |
| 7 | Geometric Bold | Tanaka | Minimal, elegant | Lifestyle, visual essays |
| 8 | Velvet Standard | Vignelli | Premium, timeless | Luxury, investor updates |
| 9 | Digital Grid | Crouwel | Systematic, technical | Infrastructure, engineering |
| 10 | Contact Sheet | Brodovitch | Editorial, investigative | Journalism, deep dives |
| 11 | Folk Frequency | Terrazas | Cultural, vivid | Festivals, food, heritage |
| 12 | Earth Pulse | Ghariokwu | Grounded, communal | Community, grassroots |
| 13 | Dream State | Tomaszewski | Surreal, poetic | Op-eds, philosophy |
| 14 | Play Mode | Ahn Sang-soo | Playful, irreverent | Entertainment, pop culture |
| 15 | Carnival Surge | Lins | Euphoric, celebratory | Milestones, hype |
| 16 | Shadow Cut | Hillmann | Dark, cinematic | Exposés, investigations |
| 17 | Deconstructed | Brody | Industrial, raw | Tech news, punk energy |
| 18 | Maximalist Type | Scher | Loud, kinetic | Big announcements, launches |
| 19 | Data Drift | Anadol | Futuristic, immersive | AI/tech, innovation |
| 20 | Red Wire | Tartakover | Urgent, immediate | Breaking news, crisis |
## Mood-to-Style Guide
| Content feels... | Use... |
|---|---|
| Personal, intimate | Soft Signal, Quiet Drama |
| Natural, earthy | Warm Grain, Earth Pulse |
| Nostalgic, historical | Heritage Reel |
| Data-driven, analytical | Swiss Pulse, Digital Grid |
| Elegant, premium | Velvet Standard, Geometric Bold |
| Cultural, global | Silk Route, Folk Frequency |
| Investigative, serious | Contact Sheet, Shadow Cut |
| Fun, lighthearted | Play Mode, Carnival Surge |
| Philosophical, abstract | Dream State |
| Punk, grassroots, raw | Deconstructed |
| Hype, loud, high-energy | Maximalist Type |
| Tech-forward, futuristic | Data Drift |
| Breaking, urgent | Red Wire |
---
## 1. Soft Signal — Stefan Sagmeister
**Mood:** Intimate, warm | **Best for:** Personal stories, wellness, reflections
- Warm amber and cream with dusty rose, sage green, honey gold accents
- Handwritten-style text overlays — personal, lowercase, delicate
- Close-up framing: hands, faces, textures. Macro lens feel
- Slow drifts and floats, never snaps. Soft dissolves, warm light leaks
```
STYLE — SOFT SIGNAL (Sagmeister): Warm amber/cream, dusty rose, sage green.
Handwritten-style text. Close-up framing. Slow drifts and floats.
Soft dissolves with warm light leaks.
```
## 2. Warm Grain — Olle Eksell
**Mood:** Organic, friendly | **Best for:** Environmental, sustainability, community
- Earth tones: ochre, forest green, terracotta, cream, soft brown
- Rounded sans-serif type. Organic rounded compositions — nothing angular
- 16mm film grain, slightly desaturated. Natural textures: wood, linen, stone
- Gentle wipes, soft cuts, unhurried
```
STYLE — WARM GRAIN (Eksell): Earth tones — ochre, forest green, terracotta, cream.
Organic rounded compositions. 16mm film grain. Rounded sans-serif.
Gentle wipes and soft cuts.
```
## 3. Quiet Drama — Satyajit Ray
**Mood:** Humanist, contemplative | **Best for:** Profiles, biographical, cultural
- Muted warm: sepia, deep brown, soft gold, off-white, charcoal
- Clean serif type, positioned with care. Portrait framing
- Strong single-source contrast: window light, single lamp
- Deliberate pacing, longer holds. Slow fades to black
```
STYLE — QUIET DRAMA (Ray): Muted warm — sepia, deep brown, soft gold.
Portrait framing. Clean serif. Strong single-source contrast.
Slow fades to black.
```
## 4. Heritage Reel — Cassandre
**Mood:** Nostalgic, vintage | **Best for:** History, retrospectives, brand origins
- Faded gold, deep burgundy, navy, cream, sepia wash
- Elegant centered serif like classic film title cards
- Vignetting, softened edges. Film grain, light scratches, gentle jitter
- Iris wipes, film reel flicker
```
STYLE — HERITAGE REEL (Cassandre): Faded gold, burgundy, navy, sepia wash.
Elegant centered serif. Vignetting and aged film grain.
Iris wipe transitions.
```
## 5. Silk Route — Reza Abedini
**Mood:** Flowing, mysterious | **Best for:** Global affairs, cross-cultural, art/design
- Rich jewel tones: deep teal, burgundy, gold, lapis blue, black
- Elegant spaced type along natural visual lines
- Layered compositions — foreground, midground, background all active
- Flowing dissolves, smooth morphs
```
STYLE — SILK ROUTE (Abedini): Jewel tones — deep teal, burgundy, gold, lapis blue.
Layered compositions, all depths active. Elegant spaced type.
Flowing dissolves and smooth morphs.
```
## 6. Swiss Pulse — Josef Müller-Brockmann
**Mood:** Clinical, precise | **Best for:** Data-heavy, analytical, financial, metrics
- Black (#1a1a1a), white, ONE accent: electric blue (#0066FF)
- Helvetica Bold headlines, Regular labels. Numbers LARGE (80-120pt)
- Grid-locked compositions. Every element snaps to 12-column grid
- Animated counters COUNT UP from 0. Diagonal compositions on key moments
- Grid wipes, hard cuts. No dissolves
```
STYLE — SWISS PULSE (Müller-Brockmann): Black/white + electric blue #0066FF.
Grid-locked. Helvetica Bold. Animated counters. Diagonal accents.
Grid wipe transitions.
```
## 7. Geometric Bold — Ikko Tanaka
**Mood:** Minimal, elegant | **Best for:** Clean lifestyle, culture, visual essays, brand profiles
- Maximum 3 flat colors per frame — no gradients
- Bold clean type as primary visual element
- Asymmetric composition, 60% negative space minimum. Single focal point
- Clean cuts on beat, no effects
```
STYLE — GEOMETRIC BOLD (Tanaka): Max 3 flat colors per frame.
60% negative space. Bold type as primary element.
Single focal point. Clean cuts on beat.
```
## 8. Velvet Standard — Massimo Vignelli
**Mood:** Premium, timeless | **Best for:** Luxury, investor updates, keynotes, product showcases
- Black, white, ONE rich accent: deep navy (#1a237e) or gold (#c9a84c)
- Thin sans-serif, ALL CAPS, letter-spaced wide
- Generous negative space. Symmetrical, centered, architectural precision
- Slow, deliberate. Sequential reveals. Elegant cross-dissolves
```
STYLE — VELVET STANDARD (Vignelli): Black, white, one accent: gold #c9a84c.
Thin ALL CAPS, wide spacing. Generous negative space.
Slow elegant cross-dissolves.
```
## 9. Digital Grid — Wim Crouwel
**Mood:** Systematic, technical | **Best for:** Infrastructure, engineering, code, tech
- Dark (#0a0a0a) with cyan (#00E5FF), amber (#FFB300), green (#00FF88)
- Monospaced type throughout. Code-terminal aesthetic
- Pixel grid overlays visible. Everything snaps to system
- Grid nodes light up sequentially. Scan-line effects, cursor blinks
- Clean wipe transitions
```
STYLE — DIGITAL GRID (Crouwel): Monospaced type. Dark #0a0a0a with cyan #00E5FF, amber #FFB300.
Pixel grid overlays. Terminal aesthetic. Clean wipe transitions.
```
## 10. Contact Sheet — Alexey Brodovitch
**Mood:** Editorial, investigative | **Best for:** Journalism, deep dives, research breakdowns
- High contrast B&W with occasional desaturated color accents
- Bold sans-serif captions like editorial annotations
- Photo-editorial framing — multiple images, contact-sheet energy
- Raw grain, imperfect focus. Tight crops on faces and hands
- Hard cuts on beat, snap-zooms
```
STYLE — CONTACT SHEET (Brodovitch): High contrast B&W, desaturated accents.
Photo-editorial framing. Bold sans-serif annotations. Raw grain.
Hard cuts on beat. Snap-zooms.
```
## 11. Folk Frequency — Eduardo Terrazas
**Mood:** Cultural, vivid | **Best for:** Cultural events, food, tradition, heritage
- Vivid folk: hot pink, bright orange, cobalt blue, sun yellow, emerald
- Bold warm rounded type. Pattern and repetition — folk art rhythms
- Rich textures: woven fabrics, painted surfaces, ceramic, handmade
- Colorful wipes, quick cuts on festive rhythm
```
STYLE — FOLK FREQUENCY (Terrazas): Vivid folk — hot pink, cobalt blue, sun yellow, emerald.
Bold rounded type. Folk art rhythms. Rich handmade textures.
Colorful wipes on festive rhythm.
```
## 12. Earth Pulse — Lemi Ghariokwu
**Mood:** Grounded, communal | **Best for:** Community, music/culture, grassroots
- Warm saturated: burnt orange, deep green, rich yellow, terracotta
- Bold expressive type, center-frame. Wide community framing
- Rhythmic editing timed to musical beats
- Rhythmic cuts on beat, freeze-frames for emphasis
```
STYLE — EARTH PULSE (Ghariokwu): Warm saturated — burnt orange, deep green, rich yellow.
Bold expressive type. Wide community framing.
Rhythmic cuts on beat. Freeze-frames.
```
## 13. Dream State — Henryk Tomaszewski
**Mood:** Surreal, poetic | **Best for:** Op-eds, philosophy, think pieces, speculative
- Muted palette with one surreal accent: dusty blues, grey-greens, then shock of red or gold
- Sparse precise text — few words, maximum impact. Thin elegant floating type
- Unusual juxtapositions. Dreamlike quality: soft edges, atmospheric haze
- Slow morph dissolves. NEVER hard cuts
```
STYLE — DREAM STATE (Tomaszewski): Muted palette + one surreal accent.
Thin elegant floating type. Soft edges, atmospheric haze.
Slow morph dissolves — NEVER hard cuts.
```
## 14. Play Mode — Ahn Sang-soo
**Mood:** Playful, irreverent | **Best for:** Entertainment, pop culture, listicles, fun
- Bright candy: electric blue, hot pink, lime green, yellow, white
- Bouncy oversized tilted text. Asymmetric off-kilter compositions
- Quick cuts (1-3 seconds). Score cards, achievement popups, XP bars
- Bouncy spring physics — text overshoots and settles, screen shakes
- Pop cuts, whip pans, bounce effects
```
STYLE — PLAY MODE (Ahn Sang-soo): Electric blue, hot pink, lime green.
Bouncy spring physics. Oversized tilted text. Score cards, XP bars.
Pop cuts, bounce effects.
```
## 15. Carnival Surge — Rico Lins
**Mood:** Euphoric, celebratory | **Best for:** Big announcements, milestones, celebrations, hype
- Maximum color: hot pink (#FF1493), electric yellow (#FFE000), teal (#00CED1), orange, violet
- MASSIVE bold text at ANGLES over footage. Collage-style overlapping
- Rapid 1-2 second clips. Confetti, lights, constant energy
- Smash cuts, flash frames, rapid-fire montage
```
STYLE — CARNIVAL SURGE (Lins): Max color — hot pink #FF1493, yellow #FFE000, teal #00CED1.
Collage layering. Text MASSIVE at ANGLES. Confetti bursts.
Smash cuts, flash frames.
```
## 16. Shadow Cut — Hans Hillmann
**Mood:** Dark, cinematic | **Best for:** Exposés, investigations, controversy, dark deep dives
- Near-monochrome: deep blacks, cold greys, stark white + blood red or toxic green
- Sharp angular text like film noir title cards
- Heavy shadow — faces half-lit, objects emerging from darkness
- Slow creeping push-ins, slow reveals, tension
- Iris to black, slow fade from darkness, hard cuts to silence
```
STYLE — SHADOW CUT (Hillmann): Deep blacks, cold greys + blood red accent.
Sharp angular text. Heavy shadow. Slow creeping push-ins.
Hard cuts to black. Film noir tension.
```
## 17. Deconstructed — Neville Brody
**Mood:** Industrial, raw | **Best for:** Tech news, security, punk energy, counter-culture
- Dark grey (#1a1a1a), black, rust orange (#D4501E), raw white (#f0f0f0)
- Type at angles, overlapping edges, escaping frames. Bold industrial
- High contrast, gritty textures: scratched metal, peeling paint, scan-line glitch
- Text SLAMS, SHATTERS, PUNCHES. Letters scramble then snap
- Smash cuts, glitch transitions, white flash frames
```
STYLE — DECONSTRUCTED (Brody): Dark grey #1a1a1a, rust orange #D4501E.
Type at angles, overlapping. Gritty textures, scan-line glitch.
Smash cuts with flash frames.
```
## 18. Maximalist Type — Paula Scher
**Mood:** Loud, kinetic | **Best for:** Big announcements, launches, high-energy recaps
- Bold saturated: red, yellow, black, white — maximum contrast
- Text IS the visual. Overlapping layers at different scales and angles, 50-80% of frame
- Kinetic energy: everything moving, slamming, sliding. 1-2 second rapid cuts
- Text layered OVER footage — never empty backgrounds
- Smash cuts, text slamming from edges, flash frames
```
STYLE — MAXIMALIST TYPE (Scher): Red, yellow, black, white — max contrast.
Text IS the visual. Overlapping at different scales, 50-80% of frame.
Kinetic everything. Smash cuts, flash frames.
```
## 19. Data Drift — Refik Anadol
**Mood:** Futuristic, immersive | **Best for:** AI/tech, speculative, cutting-edge science
- Iridescent: holographic silver, electric purple (#7c3aed), cyan (#06b6d4), deep black (#0a0a0a)
- Thin futuristic sans-serif — minimal, floating, weightless
- Fluid morphing compositions. Extreme scale shifts: microscopic to cosmic
- Particles coalesce into numbers, light traces data paths
- Liquid dissolves, particles dispersing and reforming
```
STYLE — DATA DRIFT (Anadol): Iridescent — purple #7c3aed, cyan #06b6d4, deep black.
Fluid morphing compositions. Thin futuristic type.
Liquid dissolves. Particles coalesce into numbers.
```
## 20. Red Wire — David Tartakover
**Mood:** Urgent, immediate | **Best for:** Breaking news, crisis updates, alerts
- High alert: red, black, white, emergency yellow — maximum contrast
- Bold condensed all caps — every word screams urgency
- Split screens, ticker-style text bars, timestamp overlays — max information density
- Multiple text elements simultaneously. Handheld energy
- Snap cuts, flash frames, zero breathing room
```
STYLE — RED WIRE (Tartakover): Red, black, white, emergency yellow.
Bold condensed all-caps. Split screens, tickers, timestamps.
Snap cuts, flash frames. Zero breathing room.
```
.claude/skills/heygen/references/voices.md
+505
View File
@@ -0,0 +1,505 @@
---
name: voices
description: Listing voices, locales, speed/pitch configuration for HeyGen
---
# HeyGen Voices
HeyGen provides a wide variety of AI voices for different languages, accents, and styles. Voices convert your text script into natural-sounding speech.
## Listing Available Voices
### curl
```bash
curl -X GET "https://api.heygen.com/v2/voices" \
-H "X-Api-Key: $HEYGEN_API_KEY"
```
### TypeScript
```typescript
interface Voice {
voice_id: string;
name: string;
language: string;
gender: "male" | "female";
preview_audio: string;
support_pause: boolean;
emotion_support: boolean;
}
interface VoicesResponse {
error: null | string;
data: {
voices: Voice[];
};
}
async function listVoices(): Promise<Voice[]> {
const response = await fetch("https://api.heygen.com/v2/voices", {
headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! },
});
const json: VoicesResponse = await response.json();
if (json.error) {
throw new Error(json.error);
}
return json.data.voices;
}
```
### Python
```python
import requests
import os
def list_voices() -> list:
response = requests.get(
"https://api.heygen.com/v2/voices",
headers={"X-Api-Key": os.environ["HEYGEN_API_KEY"]}
)
data = response.json()
if data.get("error"):
raise Exception(data["error"])
return data["data"]["voices"]
```
## Response Format
```json
{
"error": null,
"data": {
"voices": [
{
"voice_id": "1bd001e7e50f421d891986aad5158bc8",
"name": "Sara",
"language": "English",
"gender": "female",
"preview_audio": "https://files.heygen.ai/...",
"support_pause": true,
"emotion_support": true
},
{
"voice_id": "de8b5d78f2e0485f88d1e9f5c8e7f9a6",
"name": "Paul",
"language": "English",
"gender": "male",
"preview_audio": "https://files.heygen.ai/...",
"support_pause": true,
"emotion_support": false
}
]
}
}
```
## Supported Languages
HeyGen supports many languages including:
| Language | Code | Notes |
|----------|------|-------|
| English (US) | en-US | Multiple voice options |
| English (UK) | en-GB | British accent |
| Spanish | es-ES | Spain Spanish |
| Spanish (Latin) | es-MX | Mexican Spanish |
| French | fr-FR | France French |
| German | de-DE | Standard German |
| Portuguese | pt-BR | Brazilian Portuguese |
| Chinese (Mandarin) | zh-CN | Simplified Chinese |
| Japanese | ja-JP | Standard Japanese |
| Korean | ko-KR | Standard Korean |
| Italian | it-IT | Standard Italian |
| Dutch | nl-NL | Standard Dutch |
| Polish | pl-PL | Standard Polish |
| Arabic | ar-SA | Saudi Arabic |
## Using Voices in Video Generation
### Basic Voice Usage
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello! Welcome to our presentation.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
};
```
### Voice with Speed Adjustment
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "This is spoken at a faster pace.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
speed: 1.2, // 1.0 is normal, range: 0.5 - 2.0
},
},
],
};
```
### Voice with Pitch Adjustment
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "This has a higher pitch.",
voice_id: "1bd001e7e50f421d891986aad5158bc8",
pitch: 10, // Range: -20 to 20
},
},
],
};
```
## Adding Pauses with Break Tags
HeyGen supports SSML-style `<break>` tags to add pauses in scripts.
### Break Tag Format
```
<break time="Xs"/>
```
Where `X` is the duration in seconds (e.g., `1s`, `1.5s`, `0.5s`).
### Requirements
| Rule | Example |
|------|---------|
| Use seconds with "s" suffix | `<break time="1.5s"/>` ✓ |
| Must have space before tag | `word <break time="1s"/>` ✓ |
| Must have space after tag | `<break time="1s"/> word` ✓ |
| Self-closing tag | `<break time="1s"/>` ✓ |
**Incorrect:** `word<break time="1s"/>word` (no spaces)
**Correct:** `word <break time="1s"/> word`
### Examples
```typescript
// Single pause
const script1 = "Hello and welcome. <break time=\"1s\"/> Let me introduce our product.";
// Multiple pauses
const script2 = "First point. <break time=\"1.5s\"/> Second point. <break time=\"1s\"/> Third point.";
// Pause at start (dramatic opening)
const script3 = "<break time=\"0.5s\"/> Welcome to our presentation.";
// Longer pause for emphasis
const script4 = "And the winner is... <break time=\"2s\"/> You!";
```
### Full Example
```typescript
const scriptWithPauses = `
Welcome to our product demo. <break time="1s"/>
Today I'll show you three key features. <break time="0.5s"/>
First, let's look at the dashboard. <break time="1.5s"/>
As you can see, it's incredibly intuitive.
`;
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: scriptWithPauses,
voice_id: "1bd001e7e50f421d891986aad5158bc8",
},
},
],
};
```
### Consecutive Breaks
Multiple consecutive break tags are automatically combined:
```typescript
// These two breaks:
"Hello <break time=\"1s\"/> <break time=\"0.5s\"/> world"
// Are treated as a single 1.5s pause
```
### Best Practices
1. **Use for emphasis** - Add pauses before important points
2. **Keep pauses reasonable** - 0.5s to 2s is typical; longer feels unnatural
3. **Match natural speech** - Add pauses where a human would breathe or pause
4. **Test the output** - Listen to generated audio to verify timing feels right
## Using Custom Audio Instead of TTS
Instead of text-to-speech, you can provide your own audio:
```typescript
const videoConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "audio",
audio_url: "https://example.com/my-audio.mp3",
},
},
],
};
```
## Filtering Voices
### By Language
```typescript
function filterByLanguage(voices: Voice[], language: string): Voice[] {
return voices.filter((v) =>
v.language.toLowerCase().includes(language.toLowerCase())
);
}
const englishVoices = filterByLanguage(voices, "english");
const spanishVoices = filterByLanguage(voices, "spanish");
```
### By Gender
```typescript
function filterByGender(voices: Voice[], gender: "male" | "female"): Voice[] {
return voices.filter((v) => v.gender === gender);
}
const femaleVoices = filterByGender(voices, "female");
```
### By Features
```typescript
function filterByFeatures(
voices: Voice[],
options: { supportPause?: boolean; emotionSupport?: boolean }
): Voice[] {
return voices.filter((v) => {
if (options.supportPause !== undefined && v.support_pause !== options.supportPause) {
return false;
}
if (options.emotionSupport !== undefined && v.emotion_support !== options.emotionSupport) {
return false;
}
return true;
});
}
const expressiveVoices = filterByFeatures(voices, { emotionSupport: true });
```
## Voice Selection Helper
```typescript
interface VoiceSelectionCriteria {
language?: string;
gender?: "male" | "female";
supportPause?: boolean;
emotionSupport?: boolean;
}
async function findVoice(criteria: VoiceSelectionCriteria): Promise<Voice | null> {
const voices = await listVoices();
const filtered = voices.filter((v) => {
if (criteria.language && !v.language.toLowerCase().includes(criteria.language.toLowerCase())) {
return false;
}
if (criteria.gender && v.gender !== criteria.gender) {
return false;
}
if (criteria.supportPause !== undefined && v.support_pause !== criteria.supportPause) {
return false;
}
if (criteria.emotionSupport !== undefined && v.emotion_support !== criteria.emotionSupport) {
return false;
}
return true;
});
return filtered[0] || null;
}
// Usage
const voice = await findVoice({
language: "english",
gender: "female",
emotionSupport: true,
});
```
## Multi-Language Videos
Create videos with different languages per scene:
```typescript
const multiLanguageConfig = {
video_inputs: [
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hello! Welcome to our global product launch.",
voice_id: "english_voice_id",
},
},
{
character: {
type: "avatar",
avatar_id: "josh_lite3_20230714",
avatar_style: "normal",
},
voice: {
type: "text",
input_text: "Hola! Bienvenidos al lanzamiento global de nuestro producto.",
voice_id: "spanish_voice_id",
},
},
],
};
```
## Matching Voice to Avatar
### Recommended: Use Avatar's Default Voice
Many avatars have a `default_voice_id` that's pre-matched. **This is the best approach.**
```typescript
// Using v2 API to get avatar with default voice
const response = await fetch(
"https://api.heygen.com/v2/avatar_group.list?include_public=true",
{ headers: { "X-Api-Key": process.env.HEYGEN_API_KEY! } }
);
const { data } = await response.json();
// Find avatar with a default voice
const avatar = data.avatar_group_list.find((a: any) => a.default_voice_id);
if (avatar) {
const videoConfig = {
video_inputs: [{
character: { type: "avatar", avatar_id: avatar.id },
voice: {
type: "text",
input_text: script,
voice_id: avatar.default_voice_id, // Pre-matched voice
},
}],
};
}
```
See [avatars.md](avatars.md) for complete examples.
### Fallback: Match Gender Manually
If avatar has no default voice, match genders manually:
```typescript
interface AvatarVoicePair {
avatarId: string;
voiceId: string;
gender: "male" | "female";
}
async function findMatchingAvatarAndVoice(
preferredGender?: "male" | "female"
): Promise<AvatarVoicePair> {
const [avatars, voices] = await Promise.all([
listAvatars(),
listVoices(),
]);
// Default to male if no preference
const gender = preferredGender || "male";
// Find avatar with matching gender
const avatar = avatars.find((a) => a.gender === gender);
if (!avatar) {
throw new Error(`No ${gender} avatar available`);
}
// Find voice with matching gender AND language
const voice = voices.find(
(v) => v.gender === gender && v.language.toLowerCase().includes("english")
);
if (!voice) {
throw new Error(`No ${gender} English voice available`);
}
return {
avatarId: avatar.avatar_id,
voiceId: voice.voice_id,
gender,
};
}
```
## Best Practices
1. **Match voice gender to avatar** - Always pair male voices with male avatars, female with female
2. **Match voice to content** - Use professional voices for business content
3. **Test voice previews** - Listen to preview audio before selecting
4. **Consider locale** - Match voice accent to target audience
5. **Use natural pacing** - Adjust speed for clarity, typically 0.9-1.1x
6. **Add pauses** - Use SSML breaks for more natural speech flow
7. **Validate availability** - Always verify voice_id exists before using
.claude/skills/heygen/references/webhooks.md
+393
View File
@@ -0,0 +1,393 @@
---
name: webhooks
description: Registering webhook endpoints and event types for HeyGen
---
# Webhooks
Webhooks allow HeyGen to notify your application when events occur, such as video completion. This is more efficient than polling for status updates.
## Overview
Instead of repeatedly checking video status, webhooks push notifications to your server when:
- Video generation completes
- Video generation fails
- Translation completes
- Avatar training completes
- Other async operations finish
## Setting Up a Webhook Endpoint
Your webhook endpoint should:
1. Accept POST requests
2. Return 200 status quickly
3. Handle events asynchronously
### Express.js Example
```typescript
import express from "express";
import crypto from "crypto";
const app = express();
app.use(express.json());
// Webhook endpoint
app.post("/webhook/heygen", async (req, res) => {
// Acknowledge receipt immediately
res.status(200).send("OK");
// Process event asynchronously
processWebhookEvent(req.body).catch(console.error);
});
async function processWebhookEvent(event: HeyGenWebhookEvent) {
console.log(`Received event: ${event.event_type}`);
switch (event.event_type) {
case "avatar_video.success":
await handleVideoSuccess(event);
break;
case "avatar_video.fail":
await handleVideoFailure(event);
break;
case "video_translate.success":
await handleTranslationSuccess(event);
break;
default:
console.log(`Unknown event type: ${event.event_type}`);
}
}
app.listen(3000, () => {
console.log("Webhook server running on port 3000");
});
```
### Python Flask Example
```python
from flask import Flask, request, jsonify
import threading
app = Flask(__name__)
@app.route("/webhook/heygen", methods=["POST"])
def heygen_webhook():
event = request.json
# Acknowledge immediately
response = jsonify({"status": "received"})
# Process asynchronously
thread = threading.Thread(
target=process_webhook_event,
args=(event,)
)
thread.start()
return response, 200
def process_webhook_event(event):
event_type = event.get("event_type")
print(f"Received event: {event_type}")
if event_type == "avatar_video.success":
handle_video_success(event)
elif event_type == "avatar_video.fail":
handle_video_failure(event)
elif event_type == "video_translate.success":
handle_translation_success(event)
if __name__ == "__main__":
app.run(port=3000)
```
## Webhook Event Types
| Event Type | Description |
|------------|-------------|
| `avatar_video.success` | Video generation completed |
| `avatar_video.fail` | Video generation failed |
| `video_translate.success` | Translation completed |
| `video_translate.fail` | Translation failed |
| `instant_avatar.success` | Instant avatar created |
| `instant_avatar.fail` | Instant avatar creation failed |
## Event Payload Structure
### Video Success Event
```typescript
interface VideoSuccessEvent {
event_type: "avatar_video.success";
event_data: {
video_id: string;
video_url: string;
thumbnail_url: string;
duration: number;
callback_id?: string;
};
}
```
```json
{
"event_type": "avatar_video.success",
"event_data": {
"video_id": "abc123",
"video_url": "https://files.heygen.ai/video/abc123.mp4",
"thumbnail_url": "https://files.heygen.ai/thumbnail/abc123.jpg",
"duration": 45.2,
"callback_id": "your_custom_id"
}
}
```
### Video Failure Event
```typescript
interface VideoFailureEvent {
event_type: "avatar_video.fail";
event_data: {
video_id: string;
error: string;
callback_id?: string;
};
}
```
```json
{
"event_type": "avatar_video.fail",
"event_data": {
"video_id": "abc123",
"error": "Script too long for selected avatar",
"callback_id": "your_custom_id"
}
}
```
## Registering a Webhook URL
Configure your webhook URL through the HeyGen dashboard or API:
### Request Fields
| Field | Type | Req | Description |
|-------|------|:---:|-------------|
| `url` | string | ✓ | Your webhook endpoint URL |
| `events` | array | ✓ | Event types to subscribe to |
| `secret` | string | | Shared secret for signature verification |
### Via API
```bash
curl -X POST "https://api.heygen.com/v1/webhook/endpoint.add" \
-H "X-Api-Key: $HEYGEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://your-domain.com/webhook/heygen",
"events": ["avatar_video.success", "avatar_video.fail"]
}'
```
### TypeScript
```typescript
interface WebhookConfig {
url: string; // Required
events: string[]; // Required
secret?: string;
}
async function registerWebhook(config: WebhookConfig): Promise<void> {
const response = await fetch("https://api.heygen.com/v1/webhook/endpoint.add", {
method: "POST",
headers: {
"X-Api-Key": process.env.HEYGEN_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify(config),
});
const json = await response.json();
if (json.error) {
throw new Error(json.error);
}
}
```
## Using Callback IDs
Track which video triggered a webhook with callback IDs:
### Include Callback ID in Video Generation
```typescript
const videoConfig = {
video_inputs: [...],
callback_id: "order_12345", // Your custom identifier
};
```
### Handle in Webhook
```typescript
async function handleVideoSuccess(event: VideoSuccessEvent) {
const { video_id, video_url, callback_id } = event.event_data;
if (callback_id) {
// Look up your original request
const order = await getOrderByCallbackId(callback_id);
await updateOrderWithVideo(order.id, video_url);
}
}
```
## Webhook Security
### Verify Webhook Signatures
If HeyGen provides signature verification:
```typescript
import crypto from "crypto";
function verifyWebhookSignature(
payload: string,
signature: string,
secret: string
): boolean {
const expectedSignature = crypto
.createHmac("sha256", secret)
.update(payload)
.digest("hex");
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expectedSignature)
);
}
// In your webhook handler
app.post("/webhook/heygen", (req, res) => {
const signature = req.headers["x-heygen-signature"] as string;
const payload = JSON.stringify(req.body);
if (!verifyWebhookSignature(payload, signature, WEBHOOK_SECRET)) {
return res.status(401).send("Invalid signature");
}
// Process event...
});
```
### Validate Event Origin
```typescript
function isValidHeygenEvent(event: any): boolean {
// Check required fields
if (!event.event_type || !event.event_data) {
return false;
}
// Check event type is known
const validEventTypes = [
"avatar_video.success",
"avatar_video.fail",
"video_translate.success",
"video_translate.fail",
];
return validEventTypes.includes(event.event_type);
}
```
## Handling Webhook Failures
Implement retry logic and error handling:
```typescript
async function processWebhookEvent(event: HeyGenWebhookEvent) {
const maxRetries = 3;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await handleEvent(event);
return;
} catch (error) {
console.error(`Attempt ${attempt} failed:`, error);
if (attempt < maxRetries) {
// Exponential backoff
await new Promise((r) => setTimeout(r, Math.pow(2, attempt) * 1000));
}
}
}
// Store failed event for manual review
await storeFailedEvent(event);
}
```
## Webhook vs Polling Comparison
| Aspect | Webhook | Polling |
|--------|---------|---------|
| Latency | Immediate | Depends on interval |
| Efficiency | High (push) | Low (repeated requests) |
| Complexity | Requires endpoint | Simpler to implement |
| Reliability | Needs retry handling | Guaranteed delivery |
| Cost | Lower API usage | Higher API usage |
## Testing Webhooks
### Local Development with ngrok
```bash
# Start ngrok tunnel
ngrok http 3000
# Use ngrok URL as webhook endpoint
# https://abc123.ngrok.io/webhook/heygen
```
### Webhook Testing Tool
```typescript
// Test webhook locally
async function simulateWebhook(event: HeyGenWebhookEvent) {
const response = await fetch("http://localhost:3000/webhook/heygen", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(event),
});
console.log(`Response: ${response.status}`);
}
// Simulate success event
await simulateWebhook({
event_type: "avatar_video.success",
event_data: {
video_id: "test_123",
video_url: "https://example.com/test.mp4",
thumbnail_url: "https://example.com/test.jpg",
duration: 30,
callback_id: "test_callback",
},
});
```
## Best Practices
1. **Respond quickly** - Return 200 within 5 seconds, process async
2. **Handle duplicates** - Same event may be sent multiple times
3. **Implement retries** - Handle temporary processing failures
4. **Log everything** - Store webhook payloads for debugging
5. **Use callback IDs** - Track requests through the system
6. **Secure endpoints** - Verify signatures, use HTTPS
7. **Monitor health** - Track webhook success rates
8. **Queue processing** - Use job queues for heavy processing
.claude/skills/lottie-bodymovin/SKILL.md
+163
View File
@@ -0,0 +1,163 @@
---
name: lottie-bodymovin
description: Use when implementing Disney's 12 animation principles with Lottie animations exported from After Effects
---
# Lottie Animation Principles
Implement all 12 Disney animation principles using Lottie (Bodymovin) for vector animations.
## 1. Squash and Stretch
In After Effects before export:
- Animate Scale X and Y inversely
- Use expression: `s = transform.scale[1]; [100 + (100-s), s]`
```javascript
// Control at runtime
lottie.setSpeed(1.5); // affect squash timing
```
## 2. Anticipation
Structure your AE composition:
1. **Frames 0-10**: Wind-up pose
2. **Frames 10-40**: Main action
3. **Frames 40-50**: Settle
```javascript
// Play anticipation segment
anim.playSegments([0, 10], true);
setTimeout(() => anim.playSegments([10, 50], true), 200);
```
## 3. Staging
```javascript
// Layer multiple Lotties
<div className="scene">
<Lottie animationData={background} style={{ opacity: 0.6 }} />
<Lottie animationData={hero} style={{ zIndex: 10 }} />
</div>
```
## 4. Straight Ahead / Pose to Pose
Pose to pose in AE:
- Set keyframes at key poses
- Let AE interpolate between
- Use Easy Ease for smoothing
```javascript
// Jump to specific poses
anim.goToAndStop(25, true); // frame 25
```
## 5. Follow Through and Overlapping Action
In After Effects:
- Offset child layer keyframes by 2-4 frames
- Use parenting with delayed expressions
- `thisComp.layer("Parent").transform.position.valueAtTime(time - 0.05)`
## 6. Slow In and Slow Out
AE Keyframe settings:
- Select keyframes > Easy Ease (F9)
- Use Graph Editor to adjust curves
- Bezier handles control acceleration
```javascript
// Adjust playback speed dynamically
anim.setSpeed(0.5); // slower
anim.setSpeed(2); // faster
```
## 7. Arc
In After Effects:
- Use motion paths (position property)
- Convert keyframes to Bezier
- Pull handles to create arcs
- Or use "Auto-Orient to Path"
## 8. Secondary Action
```javascript
// Trigger secondary animation
mainAnim.addEventListener('complete', () => {
secondaryAnim.play();
});
// Or sync with frame
mainAnim.addEventListener('enterFrame', (e) => {
if (e.currentTime > 15) particleAnim.play();
});
```
## 9. Timing
```javascript
anim.setSpeed(0.5); // half speed - dramatic
anim.setSpeed(1); // normal
anim.setSpeed(2); // double speed - snappy
// Or control frame rate in AE export
// 24fps = cinematic, 30fps = smooth, 60fps = fluid
```
## 10. Exaggeration
In After Effects:
- Push scale beyond 100% (120-150%)
- Overshoot rotation
- Use Overshoot expression
- `amp = 15; freq = 3; decay = 5; n = 0; time_start = key(1).time; if (time > time_start) { n = (time - time_start) / thisComp.frameDuration; amp * Math.sin(freq*n) / Math.exp(decay*n/100); } else { 0; }`
## 11. Solid Drawing
In After Effects:
- Use 3D layers
- Apply perspective camera
- Animate Z position and rotation
- Use depth of field
## 12. Appeal
Design principles in AE:
- Smooth curves over sharp angles
- Consistent timing patterns
- Pleasing color palette
- Clean vector shapes
```javascript
// React Lottie with hover
<Lottie
animationData={data}
onMouseEnter={() => anim.setDirection(1)}
onMouseLeave={() => anim.setDirection(-1)}
/>
```
## Lottie Implementation
```javascript
import Lottie from 'lottie-react';
import animationData from './animation.json';
<Lottie
animationData={animationData}
loop={true}
autoplay={true}
style={{ width: 200, height: 200 }}
/>
```
## Key Lottie Features
- `playSegments([start, end])` - Play frame range
- `setSpeed(n)` - Control timing
- `setDirection(1/-1)` - Forward/reverse
- `goToAndStop(frame)` - Pose control
- `addEventListener` - Frame events
- Interactivity via `lottie-interactivity`
.claude/skills/ltx2/SKILL.md
+176
View File
@@ -0,0 +1,176 @@
---
name: ltx2
description: AI video generation with LTX-2.3 22B — text-to-video, image-to-video clips for video production. Use when generating video clips, animating images, creating b-roll, animated backgrounds, or motion content. Triggers include video generation, animate image, b-roll, motion, video clip, text-to-video, image-to-video.
---
# LTX-2.3 Video Generation
Generate ~5 second video clips from text prompts or images using the LTX-2.3 22B DiT model.
Runs on Modal (A100-80GB). Requires `MODAL_LTX2_ENDPOINT_URL` in `.env`.
## Quick Reference
```bash
# Text-to-video
python3 tools/ltx2.py --prompt "A sunset over the ocean, golden light on waves, cinematic" --output sunset.mp4
# Image-to-video (animate a still image)
python3 tools/ltx2.py --prompt "Gentle camera drift, soft ambient motion" --input photo.jpg --output animated.mp4
# Custom resolution and duration
python3 tools/ltx2.py --prompt "..." --width 1024 --height 576 --num-frames 161 --output wide.mp4
# Fast mode (fewer steps, quicker)
python3 tools/ltx2.py --prompt "..." --quality fast --output quick.mp4
# Reproducible output
python3 tools/ltx2.py --prompt "..." --seed 42 --output reproducible.mp4
```
## Parameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `--prompt` | (required) | Text description of the video |
| `--input` | - | Input image for image-to-video |
| `--width` | 768 | Video width (divisible by 64) |
| `--height` | 512 | Video height (divisible by 64) |
| `--num-frames` | 121 | Frame count, must satisfy `(n-1) % 8 == 0` |
| `--fps` | 24 | Frames per second |
| `--quality` | standard | `standard` (30 steps) or `fast` (15 steps) |
| `--steps` | 30 | Override inference steps directly |
| `--seed` | random | Seed for reproducibility |
| `--output` | auto | Output file path |
| `--negative-prompt` | sensible default | What to avoid |
## Valid Frame Counts
`(n - 1) % 8 == 0`: 25 (~1s), 49 (~2s), 73 (~3s), 97 (~4s), **121 (~5s default)**, 161 (~6.7s), 193 (~8s max practical).
## Common Resolutions
| Resolution | Ratio | Notes |
|------------|-------|-------|
| 768x512 | 3:2 | Default, good balance |
| 512x512 | 1:1 | Square, fastest |
| 1024x576 | 16:9 | Widescreen |
| 576x1024 | 9:16 | Portrait/vertical |
## Prompting Guide
LTX-2 responds well to cinematographic descriptions. Layer these dimensions:
- **Camera:** "Slow dolly forward", "Aerial drone shot", "Tracking shot", "Static wide angle"
- **Lighting:** "Golden hour", "Cinematic lighting", "Neon-lit", "Soft diffused light"
- **Motion:** "Timelapse of...", "Slow motion", "Gentle camera drift", "Gradually transitions"
- **Style:** "Shot on 35mm film", "Documentary style", "Clean minimal aesthetic"
- **Negative:** Always implicitly avoids "worst quality, blurry, jittery, watermark, text, logo"
Keep prompts under 200 words. Be specific about the scene.
### Good Prompts
```
# Atmospheric b-roll
"Aerial drone shot slowly flying over turquoise ocean waves breaking on white sand, golden hour sunlight, cinematic"
# Product/tech scene
"Close-up of hands typing on a mechanical keyboard, shallow depth of field, soft desk lamp lighting, cozy atmosphere"
# Abstract background
"Dark moody abstract background with flowing blue light streaks, subtle geometric grid, bokeh particles floating, cinematic tech atmosphere"
# Animate a portrait
"Professional headshot, subtle natural head movement, confident warm expression, studio lighting, shallow depth of field"
# Animate a slide/screenshot
"Gentle subtle particle effects floating across a presentation slide, soft ambient light shifts, very slight camera drift"
```
### Bad Prompts
```
# Too vague
"A cool video"
# Too many competing ideas
"A cat riding a skateboard while juggling fire on the moon during a thunderstorm"
# Describing text/UI (model can't render text reliably)
"A website showing the text 'Welcome to our platform'"
```
## Video Production Use Cases
### B-Roll Clips
Generate atmospheric 5s shots for cutaways between narrated scenes:
```bash
python3 tools/ltx2.py --prompt "Futuristic holographic interface, glowing data visualizations, clean workspace, cinematic" --output broll_tech.mp4
python3 tools/ltx2.py --prompt "Aerial view of European city at golden hour, modern architecture" --output broll_europe.mp4
```
### Animated Slide Backgrounds
Feed a slide screenshot and add subtle motion:
```bash
python3 tools/ltx2.py --prompt "Gentle particle effects, soft ambient light shifts, very slight camera drift" --input slide.png --output animated_slide.mp4
```
### Animated Portraits
Bring still headshots to life:
```bash
python3 tools/ltx2.py --prompt "Subtle natural head movement, warm expression, professional lighting" --input headshot.png --output animated_portrait.mp4
```
### Branded Intro/Outro
Generate abstract motion backgrounds for title cards:
```bash
python3 tools/ltx2.py --prompt "Dark moody background with flowing blue and coral light streaks, bokeh particles, cinematic tech atmosphere, no text" --output intro_bg.mp4
```
### Combining with Other Tools
LTX-2 generates raw clips. Combine with the rest of the toolkit:
| Workflow | Tools |
|----------|-------|
| Generate clip → upscale | `ltx2.py` → `upscale.py` |
| Generate clip → add to Remotion | `ltx2.py` → use as `<OffthreadVideo>` in composition |
| Generate image → animate | `flux2.py` → `ltx2.py --input` |
| Generate clip → extract audio | `ltx2.py` → `ffmpeg -i clip.mp4 -vn audio.wav` |
| Generate clip → add voiceover | `ltx2.py` → mix with `qwen3_tts.py` output |
## Technical Details
- **Model:** LTX-2.3 22B DiT (Lightricks), bf16
- **GPU:** A100-80GB on Modal (~$4.68/hr)
- **Inference:** ~2.5 min per clip (768x512, 121 frames, 30 steps)
- **Cost:** ~$0.20-0.25 per 5s clip
- **Cold start:** ~60-90s (loading ~55GB weights)
- **Output:** H.264 MP4 with synchronized ambient audio (24fps)
- **Max duration:** ~8s (193 frames) per clip
### Known Limitations
- **Training data artifacts:** ~30% of generations may have unwanted logos/text from training data. Re-run with different `--seed`.
- **Text rendering:** Cannot reliably generate readable text in video. Use Remotion overlays instead.
- **Max duration:** ~8s per clip. Longer content needs stitching.
- **Audio:** Generated audio is ambient/environmental only. Use voiceover/music tools for speech and music.
- **License:** Community License — free under $10M revenue, commercial license needed above that.
## Setup
```bash
# 1. Create Modal secret for HuggingFace (one-time)
modal secret create huggingface-token HF_TOKEN=hf_your_token
# 2. Deploy (downloads ~55GB of weights, takes ~10 min)
modal deploy docker/modal-ltx2/app.py
# 3. Save endpoint URL to .env
echo "MODAL_LTX2_ENDPOINT_URL=https://yourname--video-toolkit-ltx2-ltx2-generate.modal.run" >> .env
# 4. Test
python3 tools/ltx2.py --prompt "A candle flickering on a dark table, cinematic" --output test.mp4
```
**Important:** HuggingFace token needs read-access scope. Accept the [Gemma 3 license](https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-unquantized) before deploying. Unauthenticated downloads are severely rate-limited.
.claude/skills/manim-composer/SKILL.md
+138
View File
@@ -0,0 +1,138 @@
---
name: manim-composer
description: |
Trigger when: (1) User wants to create an educational/explainer video, (2) User has a vague concept they want visualized, (3) User mentions "3b1b style" or "explain like 3Blue1Brown", (4) User wants to plan a Manim video or animation sequence, (5) User asks to "compose" or "plan" a math/science visualization.
Transforms vague video ideas into detailed scene-by-scene plans (scenes.md). Conducts research, asks clarifying questions about audience/scope/focus, and outputs comprehensive scene specifications ready for implementation with ManimCE or ManimGL.
Use this BEFORE writing any Manim code. This skill plans the video; use manimce-best-practices or manimgl-best-practices for implementation.
---
## Workflow
### Phase 1: Understand the Concept
1. **Research the topic** deeply before asking questions
- Use web search to understand the core concepts
- Identify the key insights that make this topic interesting
- Find the "aha moment" - what makes this click for learners
- Note common misconceptions to address
2. **Identify the narrative hook**
- What question does this video answer?
- Why should the viewer care?
- What's the surprising or counterintuitive element?
### Phase 2: Clarify with User
Ask targeted questions (not all at once - adapt based on responses):
**Audience & Scope**
- What math/science background should I assume? (e.g., "knows calculus" or "high school algebra")
- Target video length? (short: 5-10min, medium: 15-20min, long: 30min+)
- Should this be self-contained or part of a series?
**Focus & Depth**
- Any specific aspects to emphasize or skip?
- Proof-heavy or intuition-focused?
- Real-world applications to include?
**Style Preferences**
- Color scheme preferences?
- Narration style? (casual, formal, playful)
- Any specific visual metaphors you have in mind?
### Phase 3: Create scenes.md
Output a comprehensive `scenes.md` file with this structure:
```markdown
# [Video Title]
## Overview
- **Topic**: [Core concept]
- **Hook**: [Opening question/mystery]
- **Target Audience**: [Prerequisites]
- **Estimated Length**: [X minutes]
- **Key Insight**: [The "aha moment"]
## Narrative Arc
[2-3 sentences describing the journey from confusion to understanding]
---
## Scene 1: [Scene Name]
**Duration**: ~X seconds
**Purpose**: [What this scene accomplishes]
### Visual Elements
- [List of mobjects needed]
- [Animations to use]
- [Camera movements]
### Content
[Detailed description of what happens, what's shown, what's explained]
### Narration Notes
[Key points to convey, tone, pacing notes]
### Technical Notes
- [Specific Manim classes/methods to use]
- [Any tricky implementations to note]
---
## Scene 2: [Scene Name]
...
---
## Transitions & Flow
[Notes on how scenes connect, recurring visual motifs]
## Color Palette
- Primary: [color] - used for [purpose]
- Secondary: [color] - used for [purpose]
- Accent: [color] - used for [purpose]
- Background: [color]
## Mathematical Content
[List of equations, formulas, or mathematical objects that need to be rendered]
## Implementation Order
[Suggested order for implementing scenes, noting dependencies]
```
## 3b1b Style Principles
Apply these principles when composing scenes:
### Visual Storytelling
- **Show, don't just tell** - Every concept needs a visual representation
- **Progressive revelation** - Build complexity gradually, don't show everything at once
- **Visual continuity** - Transform objects rather than replacing them when possible
### Pacing & Rhythm
- **Pause for insight** - Give viewers time to absorb key moments
- **Vary the pace** - Mix quick sequences with slower explanations
- **End scenes with resolution** - Each scene should feel complete
### Mathematical Beauty
- **Emphasize elegance** - Highlight when math is surprisingly simple or beautiful
- **Connect representations** - Show the same concept multiple ways (algebraic, geometric, intuitive)
- **Embrace abstraction gradually** - Start concrete, then generalize
### Engagement Techniques
- **Pose questions** - Make viewers curious before revealing answers
- **Acknowledge difficulty** - "This might seem confusing at first..."
- **Celebrate insight** - Make the "aha moment" feel earned
## References
- [references/narrative-patterns.md](references/narrative-patterns.md) - Common 3b1b narrative structures
- [references/visual-techniques.md](references/visual-techniques.md) - Effective visualization patterns
- [references/scene-examples.md](references/scene-examples.md) - Example scenes.md excerpts
## Templates
- [templates/scenes-template.md](templates/scenes-template.md) - Blank scenes.md template
.claude/skills/manim-composer/references/narrative-patterns.md
+125
View File
@@ -0,0 +1,125 @@
# Narrative Patterns for Math Explainers
Common structures used in effective 3Blue1Brown-style videos.
## Pattern 1: Mystery → Investigation → Resolution
**Structure:**
1. Present a puzzling result or paradox
2. Investigate why it's true through visual exploration
3. Reveal the underlying principle
4. Show how the principle generalizes
**Example topics:** Euler's identity, Bayes theorem, infinite series paradoxes
**Opening hooks:**
- "What does it even mean to raise a number to an imaginary power?"
- "This equation looks wrong, but it's actually true..."
- "Most people get this probability question wrong..."
---
## Pattern 2: Build Up → Payoff
**Structure:**
1. Introduce simple building blocks
2. Combine them to create something complex
3. Show the beautiful/surprising result
4. Reflect on why it works
**Example topics:** Fourier series, neural networks, linear algebra
**Opening hooks:**
- "Let's start with something simple..."
- "Each piece here is easy, but together they do something remarkable..."
---
## Pattern 3: Two Perspectives → Unity
**Structure:**
1. Show concept from perspective A (e.g., algebraic)
2. Show same concept from perspective B (e.g., geometric)
3. Reveal they're the same thing
4. Explore implications of this connection
**Example topics:** Dot product, determinants, complex multiplication
**Opening hooks:**
- "There are two ways to think about this..."
- "These seem like completely different ideas, but..."
---
## Pattern 4: Wrong → Less Wrong → Right
**Structure:**
1. Present common misconception or naive approach
2. Show why it fails
3. Refine the approach
4. Arrive at correct understanding
**Example topics:** Limits, probability distributions, definitions
**Opening hooks:**
- "Your first instinct here is probably wrong..."
- "The obvious approach doesn't quite work..."
---
## Pattern 5: Specific → General
**Structure:**
1. Solve a specific concrete example
2. Notice patterns in the solution
3. Abstract to general principle
4. Apply to new situations
**Example topics:** Derivatives, group theory, algorithm analysis
**Opening hooks:**
- "Let's work through a specific example..."
- "Once you see the pattern here, it shows up everywhere..."
---
## Pattern 6: History as Narrative
**Structure:**
1. Present the problem as historically encountered
2. Follow the journey of discovery
3. Show key insights that led to breakthroughs
4. Connect to modern understanding
**Example topics:** Calculus, cryptography, quantum mechanics
**Opening hooks:**
- "Imagine you're a mathematician in the 1600s..."
- "This problem stumped the greatest minds for centuries..."
---
## Combining Patterns
Most effective videos combine multiple patterns:
- Mystery hook + Build Up explanation
- Two Perspectives + Specific → General examples
- Wrong → Right + History narrative
## Pacing Guidelines
| Video Length | Intro Hook | Main Content | Recap/Implications |
|--------------|------------|--------------|-------------------|
| 5-10 min | 30-60s | 4-8 min | 30-60s |
| 15-20 min | 1-2 min | 12-16 min | 1-2 min |
| 30+ min | 2-3 min | 24-26 min | 2-4 min |
## Emotional Arc
Every video should have emotional beats:
1. **Curiosity** (opening) - Why should I care?
2. **Confusion** (early) - This is harder than it looks
3. **Partial clarity** (middle) - I'm starting to see...
4. **Aha moment** (climax) - Oh! That's beautiful!
5. **Satisfaction** (end) - Now I truly understand

Some files were not shown because too many files have changed in this diff Show More