Initial release — OpenMontage: the first open-source agentic video production system
11 production pipelines, 47 tools, 124 agent skills. Supports cloud APIs (fal.ai, OpenAI, ElevenLabs, Suno, HeyGen, Runway) and free local providers (diffusers, Piper TTS, WAN 2.1, Hunyuan, CogVideo). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
calesthio
@@ -0,0 +1,127 @@
|
||||
# Animation & Motion Graphics Pipeline
|
||||
|
||||
> Sources: School of Motion curriculum, After Effects documentation, Remotion documentation,
|
||||
> Disney's 12 Principles of Animation (Frank Thomas & Ollie Johnston), Motion Design School,
|
||||
> The Animator's Survival Kit (Richard Williams)
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
FRAME RATE: 30fps for web video | 24fps for cinematic feel | 60fps for UI/smooth motion
|
||||
EASE DEFAULT: easeInOutCubic (0.65, 0, 0.35, 1) — never use linear
|
||||
TRANSITION: 0.5-1.0s between scenes
|
||||
ANTICIPATION: 2-3 frames before main action
|
||||
OVERSHOOT: 10-15% past target, settle back in 3-5 frames
|
||||
HOLD FRAMES: 6-12 frames (0.2-0.4s) on key poses
|
||||
COLOR: Max 5 colors from playbook palette
|
||||
EXPORT: H.264 CRF 18-20 for web, ProRes 422 for editing
|
||||
```
|
||||
|
||||
## Frame Rate Selection
|
||||
|
||||
| Style | FPS | When to Use |
|
||||
|-------|-----|-------------|
|
||||
| **Cinematic animation** | 24 | Film-like feel, character animation, organic motion |
|
||||
| **Web/explainer standard** | 30 | Default for YouTube/web video. OpenMontage default. |
|
||||
| **Smooth UI animation** | 60 | Software demos, UI transitions, scrolling |
|
||||
| **Stylized/limited** | 12-15 on 2s/3s | Deliberately choppy, artistic choice |
|
||||
|
||||
**OpenMontage default:** 30fps. Render Manim at 60fps and transcode to 30fps for smoother motion at delivery frame rate.
|
||||
|
||||
## Timing Principles (Applied to Motion Graphics)
|
||||
|
||||
### The 4 Most Important Principles
|
||||
|
||||
| Principle | Application | Timing |
|
||||
|-----------|------------|--------|
|
||||
| **Ease In/Out** | Every movement starts slow, ends slow | Use cubic or quart easing, never linear |
|
||||
| **Anticipation** | Brief movement opposite to the main action | 2-3 frames (66-100ms at 30fps) |
|
||||
| **Overshoot** | Object passes target, bounces back | 10-15% past target, settle in 3-5 frames |
|
||||
| **Staging** | Only one thing moves at a time | Stagger animations by 3-6 frames |
|
||||
|
||||
### Easing Curves
|
||||
|
||||
| Curve | Cubic Bezier | Use For |
|
||||
|-------|-------------|---------|
|
||||
| **easeOutCubic** | `(0.33, 1, 0.68, 1)` | Objects entering the scene |
|
||||
| **easeInCubic** | `(0.32, 0, 0.67, 0)` | Objects leaving the scene |
|
||||
| **easeInOutCubic** | `(0.65, 0, 0.35, 1)` | Position changes within scene |
|
||||
| **easeOutBack** | `(0.34, 1.56, 0.64, 1)` | Bouncy pop-in (playful) |
|
||||
| **easeOutElastic** | spring simulation | Attention-grabbing reveals |
|
||||
| **linear** | `(0, 0, 1, 1)` | **NEVER for motion** — only for opacity or color |
|
||||
|
||||
### Hold Frames
|
||||
|
||||
After a movement completes, **hold the pose** before the next animation:
|
||||
|
||||
| Context | Hold Duration |
|
||||
|---------|--------------|
|
||||
| Key information on screen | 1.0-2.0s (narration dependent) |
|
||||
| Between animation beats | 0.3-0.5s (8-15 frames at 30fps) |
|
||||
| After a reveal | 1.5-3.0s (let it register) |
|
||||
| Quick transition | 0.1-0.2s (3-6 frames) |
|
||||
|
||||
## Scene Transitions
|
||||
|
||||
| Transition | Duration | When to Use |
|
||||
|-----------|----------|-------------|
|
||||
| **Hard cut** | Instant | Same topic, different angle/zoom |
|
||||
| **Crossfade** | 0.5-1.0s | Topic change, gentle shift |
|
||||
| **Wipe/slide** | 0.5-0.8s | Sequential steps, progression |
|
||||
| **Zoom in** | 0.8-1.2s | Diving deeper into detail |
|
||||
| **Zoom out** | 0.8-1.2s | Revealing bigger picture |
|
||||
| **Match cut** | Instant | Same shape/position, different content |
|
||||
| **Morph/transform** | 1.0-2.0s | Concept evolution, before/after |
|
||||
|
||||
### Transition Rules
|
||||
|
||||
1. **Consistent transitions** — pick 2-3 types and stick with them throughout the video
|
||||
2. **Transition = meaning** — a wipe means "next step," a zoom means "deeper detail"
|
||||
3. **Don't over-transition** — a hard cut is the most invisible and most professional transition
|
||||
4. **Audio leads visual** — start transition sound 10-20ms before the visual change
|
||||
|
||||
## Composition for Motion Graphics
|
||||
|
||||
### Layout
|
||||
|
||||
- **Rule of thirds** — place focal elements on intersection points
|
||||
- **Visual hierarchy** — largest/brightest element = most important
|
||||
- **White space** — minimum 10% margin on all sides (within title-safe)
|
||||
- **Direction of motion** — left-to-right = forward/progress, right-to-left = reverse/back
|
||||
|
||||
### Color
|
||||
|
||||
- **Max 5 colors** from the style playbook palette
|
||||
- **1 accent color** for emphasis — used sparingly
|
||||
- **Background** should be the least saturated color
|
||||
- **Contrast** between foreground elements and background: minimum 3:1
|
||||
|
||||
### Stagger and Choreography
|
||||
|
||||
When multiple elements enter:
|
||||
- Stagger entry by **3-6 frames** (100-200ms) between elements
|
||||
- Enter from the same direction for grouped elements
|
||||
- Use `LaggedStart` (Manim) or staggered `delay` (Remotion) with `lag_ratio=0.1-0.2`
|
||||
|
||||
## Export Settings
|
||||
|
||||
| Target | Codec | Settings |
|
||||
|--------|-------|----------|
|
||||
| YouTube/web final | H.264 | CRF 18-20, `-pix_fmt yuv420p`, `-movflags +faststart` |
|
||||
| Editing intermediate | ProRes 422 | For further editing/compositing |
|
||||
| Transparent overlay | ProRes 4444 | When compositing over other footage |
|
||||
| GIF preview | GIF | 480px wide, 15fps, 256 colors |
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When building animation/motion graphics content:
|
||||
|
||||
1. **Render at 30fps** (OpenMontage default) — Manim at 60fps, transcode down
|
||||
2. **Never use linear easing** — default to `easeInOutCubic` for all motion
|
||||
3. **Stagger multi-element entrances** by 100-200ms — don't reveal everything at once
|
||||
4. **Hold key frames** for 1.0-2.0s after reveals (synced to narration)
|
||||
5. **Use 2-3 transition types** consistently — hard cut + crossfade covers most needs
|
||||
6. **Audio leads visual** — SFX starts 10-20ms before transition (see sound-design.md)
|
||||
7. **Max 5 palette colors** — enforce from the style playbook
|
||||
8. **Anticipation + overshoot** on important movements for polish
|
||||
9. **Export H.264 CRF 18-20** for final output via `video_compose`
|
||||
@@ -0,0 +1,113 @@
|
||||
# Background Removal Usage for OpenMontage
|
||||
|
||||
> Sources: rembg library documentation, U2Net paper (Qin et al. 2020), IS-Net paper
|
||||
> (Qin et al. 2022), OpenMontage `tools/bg_remove.py` implementation
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
DEFAULT MODEL: u2net (general purpose, fast)
|
||||
FOR PEOPLE: u2net_human_seg (optimized for human silhouettes)
|
||||
FINE EDGES: Enable alpha_matting (hair, fur, leaves)
|
||||
OUTPUT: Transparent PNG by default; set bg_color for solid replacement
|
||||
RUNTIME: ~1-3s per image (CPU), <0.5s (GPU with onnxruntime-gpu)
|
||||
INSTALL: pip install rembg (CPU) | pip install rembg[gpu] (CUDA)
|
||||
```
|
||||
|
||||
## When to Use bg_remove
|
||||
|
||||
Background removal is an **asset-prep** step. Use it before the compose stage.
|
||||
|
||||
- **Product demos / e-commerce videos** -- isolate a product on a clean background
|
||||
- **Compositing** -- layer a speaker over generated backgrounds or diagrams
|
||||
- **Thumbnail generation** -- clean cutouts for YouTube thumbnails
|
||||
- **Green-screen replacement** -- achieve green-screen results without an actual green screen
|
||||
- **B-roll preparation** -- clean up raw photos for overlay use
|
||||
|
||||
## Model Selection Guide
|
||||
|
||||
| Model | Best For | Speed | Notes |
|
||||
|-------|----------|-------|-------|
|
||||
| `u2net` | General objects, products, scenes | Fast | Default; good all-rounder |
|
||||
| `u2net_human_seg` | People, portraits, speakers | Fast | More accurate masks for human silhouettes |
|
||||
| `isnet-general-use` | Complex edges, hair, fur | Slower | Higher detail on fine boundaries |
|
||||
|
||||
**Decision rule:** If the subject is a person, use `u2net_human_seg`. If the subject has intricate edges (hair, fur, foliage) and you need maximum quality, use `isnet-general-use`. Otherwise, use the default `u2net`.
|
||||
|
||||
## Alpha Matting
|
||||
|
||||
Alpha matting refines the edge mask by computing soft transparency at boundaries. It produces more natural edges but costs approximately 2x processing time.
|
||||
|
||||
| Subject Type | Alpha Matting | Reason |
|
||||
|-------------|---------------|--------|
|
||||
| Hair, fur, feathers | Enable | Fine semi-transparent strands need soft edges |
|
||||
| Leaves, trees, grass | Enable | Irregular organic boundaries benefit from matting |
|
||||
| Products, devices | Disable | Clean geometric edges; matting adds no value |
|
||||
| Text, logos, shapes | Disable | Hard edges are correct for these subjects |
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### 1. Speaker Cutout for Compositing
|
||||
|
||||
Extract a speaker from their background and layer over a diagram or slide.
|
||||
|
||||
```
|
||||
bg_remove(input_path="speaker.png", model="u2net_human_seg")
|
||||
--> speaker_nobg.png (transparent)
|
||||
--> compose over diagram/slide in compose stage
|
||||
```
|
||||
|
||||
### 2. Product Isolation
|
||||
|
||||
Isolate a product and optionally place on a brand-colored background.
|
||||
|
||||
```
|
||||
bg_remove(input_path="product.jpg", model="u2net")
|
||||
--> product_nobg.png (transparent)
|
||||
|
||||
# Or with brand background:
|
||||
bg_remove(input_path="product.jpg", model="u2net", bg_color="#FFFFFF")
|
||||
--> product_nobg.png (white background)
|
||||
```
|
||||
|
||||
### 3. Thumbnail Prep
|
||||
|
||||
Remove background, upscale, then compose with text overlays.
|
||||
|
||||
```
|
||||
bg_remove(input_path="subject.png", model="u2net_human_seg", alpha_matting=True)
|
||||
--> subject_nobg.png
|
||||
--> upscale --> compose with text overlays in compose stage
|
||||
```
|
||||
|
||||
### 4. Batch Frame Processing
|
||||
|
||||
When preparing multiple frames for a compositing sequence, process all source frames before entering the compose stage.
|
||||
|
||||
```
|
||||
for each source frame:
|
||||
bg_remove(input_path=frame, model="u2net_human_seg")
|
||||
--> frame_nobg.png
|
||||
then: compose all transparent frames over background sequence
|
||||
```
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before moving to the compose stage, verify each bg_remove output:
|
||||
|
||||
- [ ] **Edge quality is clean** -- no halo artifacts around the subject
|
||||
- [ ] **Fine details preserved** -- hair, fingers, and thin features are intact
|
||||
- [ ] **Transparency is complete** -- no residual background bleed in transparent areas
|
||||
- [ ] **Subject integrity** -- no parts of the subject were incorrectly removed
|
||||
- [ ] **Compositing test** -- when layered over the target background, the subject blends naturally
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When using the `bg_remove` tool in asset preparation:
|
||||
|
||||
1. **Use `u2net_human_seg` for any frame containing people** -- it produces tighter masks around human silhouettes than the general model
|
||||
2. **Enable `alpha_matting` only for subjects with complex edges** like hair, fur, or foliage -- skip it for clean-edged subjects to save processing time
|
||||
3. **For compositing workflows, output transparent PNG** (omit `bg_color`) and layer in the compose stage -- this preserves maximum flexibility
|
||||
4. **For solid-background replacements, set `bg_color`** to match the playbook's background color token -- keeps outputs consistent with the project style
|
||||
5. **Process source frames BEFORE the compose stage** -- bg_remove is an asset-prep step, not a compose-time operation
|
||||
6. **Check output edges at full resolution before compositing** -- halo artifacts and edge bleed are visible in final video and must be caught early
|
||||
@@ -0,0 +1,140 @@
|
||||
# B-Roll Planning for OpenMontage
|
||||
|
||||
> How to plan B-roll needs from a script, decide between stock and generated footage,
|
||||
> construct effective search queries, and evaluate footage quality.
|
||||
|
||||
## When to Use
|
||||
|
||||
You are planning visual assets for a video and need supplementary footage (B-roll) to accompany
|
||||
narration, establish context, or add visual variety. This skill teaches you when to reach for
|
||||
stock footage vs. AI generation, and how to get good results from each.
|
||||
|
||||
## The Decision Matrix: Stock vs. Generated
|
||||
|
||||
| Scene Need | Prefer Stock | Prefer Generated |
|
||||
|------------|-------------|-----------------|
|
||||
| Real-world establishing shot (city, office, nature) | **Yes** — stock excels here | Only if no good stock match |
|
||||
| People in realistic settings | **Yes** — generated humans often look uncanny | Only with high-quality models |
|
||||
| Abstract concept visualization | No | **Yes** — AI can create what doesn't exist |
|
||||
| Custom diagrams/infographics | No | **Yes** — use `diagram_gen` or `image_selector` |
|
||||
| Branded/stylized imagery | No | **Yes** — AI matches your playbook style |
|
||||
| Historical/archival footage | **Yes** — stock libraries have archives | No |
|
||||
| Specific technical equipment | **Yes** — real photos are more credible | Only if equipment doesn't exist |
|
||||
| Motion/action clips (waves, traffic, clouds) | **Yes** — stock video is perfect for this | AI video is catching up |
|
||||
| Metaphorical imagery (growth, connection) | Either works | **Yes** — more creative control |
|
||||
|
||||
**Rule of thumb:** If the scene needs to look _real_, use stock. If it needs to look _specific to your concept_, generate it.
|
||||
|
||||
## Extracting B-Roll Needs from a Script
|
||||
|
||||
Walk the script section by section. For each section, ask:
|
||||
|
||||
1. **What is the narrator talking about?** — The subject suggests the visual.
|
||||
2. **Is there an enhancement cue?** — The script writer may have embedded `[B-ROLL: ...]` cues.
|
||||
3. **Does this section reference something concrete?** — "servers in a data center" → stock footage of servers.
|
||||
4. **Does this section explain an abstract concept?** — "the algorithm weighs each factor" → generated diagram.
|
||||
5. **How long is this section?** — Determines clip duration needed.
|
||||
|
||||
### Output: B-Roll Brief
|
||||
|
||||
For each identified need, create an entry:
|
||||
|
||||
```
|
||||
Scene: s3 (15s-22s)
|
||||
Need: Establishing shot of a modern data center
|
||||
Source: stock
|
||||
Keywords: ["data center", "server room", "rack servers blue light"]
|
||||
Duration: 4-6 seconds
|
||||
Orientation: landscape
|
||||
Mood: cool, technological, clean
|
||||
Fallback: AI-generated image of server racks
|
||||
```
|
||||
|
||||
## Constructing Effective Stock Search Queries
|
||||
|
||||
### Query Construction Rules
|
||||
|
||||
1. **Be specific but not too specific.** "aerial city skyline sunset" works. "aerial shot of downtown San Francisco financial district at 6:47pm golden hour" returns nothing.
|
||||
|
||||
2. **Use 2-4 keywords.** Stock search is keyword-based, not semantic. More words = fewer results.
|
||||
|
||||
3. **Lead with the subject.** "ocean waves" not "beautiful calm serene ocean waves at dawn."
|
||||
|
||||
4. **Include the visual quality you need:**
|
||||
- Add "aerial" or "drone" for overhead shots
|
||||
- Add "close-up" or "macro" for detail shots
|
||||
- Add "timelapse" for time-lapse footage
|
||||
- Add "slow motion" for slow-mo clips
|
||||
|
||||
5. **Try synonyms on failure.** If "programmer coding" returns poor results, try "developer laptop" or "software engineer workspace."
|
||||
|
||||
### Query Templates by Scene Type
|
||||
|
||||
| Scene Type | Query Template | Example |
|
||||
|-----------|---------------|---------|
|
||||
| Establishing | `[place] [time of day]` | "tokyo skyline night" |
|
||||
| Activity | `[person] [action]` | "scientist microscope" |
|
||||
| Object | `[object] [style]` | "circuit board closeup" |
|
||||
| Nature | `[element] [quality]` | "ocean waves aerial" |
|
||||
| Abstract motion | `[movement] [style]` | "light trails timelapse" |
|
||||
| Workplace | `[setting] [activity]` | "modern office meeting" |
|
||||
|
||||
## Evaluating Stock Footage Quality
|
||||
|
||||
When the stock tool returns results, evaluate before using:
|
||||
|
||||
### Image Criteria
|
||||
- **Resolution:** Meets target (1080p minimum for video frames)
|
||||
- **Relevance:** Actually depicts what the scene needs (not just keyword match)
|
||||
- **Style compatibility:** Doesn't clash with the playbook's visual style
|
||||
- **No watermarks:** Pexels/Pixabay are license-free, but verify
|
||||
- **Composition:** Subject is well-framed, not cut off awkwardly
|
||||
|
||||
### Video Criteria (all image criteria plus)
|
||||
- **Duration:** At least as long as the scene needs (can trim, can't extend)
|
||||
- **Motion:** Smooth, no jarring camera movement (unless that's the intent)
|
||||
- **Frame rate:** Matches target output (24/30fps standard)
|
||||
- **Audio:** Stock video audio is usually discarded — don't factor it in
|
||||
|
||||
### Scoring Heuristic
|
||||
|
||||
Rate each result 1-5:
|
||||
- **5:** Perfect match, use immediately
|
||||
- **4:** Good match, minor crop or trim needed
|
||||
- **3:** Acceptable, would benefit from color grading to match playbook
|
||||
- **2:** Marginal — try different keywords first
|
||||
- **1:** Wrong — doesn't match the scene at all
|
||||
|
||||
**Threshold:** Use results scoring 3+. Below 3, refine the query or switch to generated.
|
||||
|
||||
## Failure Escalation
|
||||
|
||||
When stock search fails (no results or all score below 3):
|
||||
|
||||
1. **Retry with different keywords** — try synonyms, broader terms, or different angles
|
||||
2. **Try the other stock provider** — Pexels and Pixabay have different libraries
|
||||
3. **Switch to AI generation** — use `flux_image` or `openai_image` with the scene description
|
||||
4. **Escalate to user** — "I couldn't find good stock footage for [scene]. Here are the best options: [show results]. Or I can generate an image instead. What do you prefer?"
|
||||
|
||||
The agent should only ask the user when both stock search AND generation fallback would produce suboptimal results. For most cases, the fallback chain handles it silently.
|
||||
|
||||
## Attribution Tracking
|
||||
|
||||
Both Pexels and Pixabay are free for commercial use with no required attribution.
|
||||
However, best practice is to track sources in the asset manifest:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "broll-scene-3",
|
||||
"type": "image",
|
||||
"source_tool": "pexels_image",
|
||||
"provider": "pexels",
|
||||
"attribution": {
|
||||
"photographer": "Joey Farina",
|
||||
"source_url": "https://www.pexels.com/photo/...",
|
||||
"license": "Pexels License"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This data is available in the tool's response (`photographer`, `pexels_url` / `page_url`). Include it in the asset manifest for transparency.
|
||||
@@ -0,0 +1,129 @@
|
||||
# Cinematic Video Pipeline
|
||||
|
||||
> Sources: No Film School editorial guides, StudioBinder filmmaking resources, Film Riot
|
||||
> production tutorials, CinematographyDB shot databases, Walter Murch "In the Blink of an Eye"
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
ASPECT RATIO: 2.39:1 (widescreen cinematic) or 16:9 with letterbox
|
||||
LETTERBOX: Black bars at top/bottom — 1920x800 active area in 1920x1080 frame
|
||||
FRAME RATE: 24fps (cinematic standard)
|
||||
SHOT DURATION: 4-8 seconds average (longer than explainer, shorter than documentary)
|
||||
COLOR GRADE: cinematic_warm or cinematic_cool profile
|
||||
AUDIO: Layered: dialogue + ambient + Foley + score
|
||||
MUSIC: 60-90 BPM, orchestral or ambient, dynamic (not loop-based)
|
||||
TARGET LUFS: -14 LUFS integrated, -24 LUFS for quiet moments
|
||||
```
|
||||
|
||||
## Aspect Ratios
|
||||
|
||||
| Ratio | Resolution (in 1080p frame) | Feel | When to Use |
|
||||
|-------|---------------------------|------|-------------|
|
||||
| **2.39:1** (anamorphic) | 1920x803 (138px bars each) | Epic, cinematic, grand | Cinematic explainers, brand films |
|
||||
| **2.35:1** (scope) | 1920x817 (131px bars each) | Classic film | Similar to 2.39:1, slightly taller |
|
||||
| **1.85:1** (flat) | 1920x1038 (21px bars each) | Moderate cinematic | Subtle letterbox, less dramatic |
|
||||
| **16:9** (no letterbox) | 1920x1080 | Standard | Default, no cinematic treatment |
|
||||
|
||||
### Implementing Letterbox in FFmpeg
|
||||
|
||||
```bash
|
||||
# Add 2.39:1 letterbox (138px black bars top and bottom)
|
||||
ffmpeg -i input.mp4 -vf "pad=1920:1080:0:138:black,crop=1920:1080:0:0" output.mp4
|
||||
|
||||
# Or render at native ratio and pad:
|
||||
ffmpeg -i input.mp4 -vf "scale=1920:803,pad=1920:1080:0:138:black" output.mp4
|
||||
```
|
||||
|
||||
**Rule:** Only use letterbox when the content genuinely benefits from cinematic framing. Don't letterbox a screen recording or talking head — it just wastes pixels.
|
||||
|
||||
## Shot Duration and Pacing
|
||||
|
||||
### Average Shot Length by Style
|
||||
|
||||
| Style | Average Shot | Cuts/Minute |
|
||||
|-------|-------------|-------------|
|
||||
| Action/intense | 2-4s | 15-30 |
|
||||
| Standard cinematic | 4-8s | 8-15 |
|
||||
| Documentary | 6-12s | 5-10 |
|
||||
| Contemplative | 10-20s | 3-6 |
|
||||
| Montage sequence | 1-3s | 20-40 |
|
||||
|
||||
### Pacing Rhythm
|
||||
|
||||
Cinematic pacing follows a **breathing rhythm** — vary shot length deliberately:
|
||||
|
||||
```
|
||||
Long (8s) → Medium (5s) → Short (3s) → Short (2s) → LONG (10s) → Medium (6s)
|
||||
```
|
||||
|
||||
**Never use the same shot length 3 times in a row** — it creates monotony.
|
||||
|
||||
### The Murch Rule
|
||||
|
||||
Walter Murch's editing priorities (in order of importance):
|
||||
1. **Emotion** — does the cut serve the emotional arc?
|
||||
2. **Story** — does the cut advance the narrative?
|
||||
3. **Rhythm** — does the cut feel right in the pacing?
|
||||
4. **Eye trace** — where is the viewer looking?
|
||||
5. **2D plane** — screen geography (180-degree rule)
|
||||
6. **3D space** — spatial continuity
|
||||
|
||||
For OpenMontage explainers using cinematic style: prioritize rhythm and story over spatial concerns (since we're often cutting between generated images, not continuous footage).
|
||||
|
||||
## Audio Layering
|
||||
|
||||
Cinematic audio has **4 layers** (not just voiceover + music):
|
||||
|
||||
| Layer | Level | Content |
|
||||
|-------|-------|---------|
|
||||
| **Dialogue/narration** | -12 dB peak | Primary voice |
|
||||
| **Music/score** | -24 to -18 dB | Orchestral, ambient, dynamic |
|
||||
| **Ambient/room tone** | -30 to -24 dB | Environmental sound bed |
|
||||
| **Foley/SFX** | -18 to -12 dB | Specific action sounds |
|
||||
|
||||
### Music for Cinematic
|
||||
|
||||
| Characteristic | Value |
|
||||
|---------------|-------|
|
||||
| BPM | 60-90 (slower than standard explainer) |
|
||||
| Genre | Orchestral, ambient, piano, cinematic electronic |
|
||||
| Dynamics | Dynamic (crescendos, swells, quiet moments) — NOT loop-based |
|
||||
| Key changes | At narrative turning points |
|
||||
| Silence | Deliberately remove music for 3-5s at key reveals |
|
||||
|
||||
### Ambient Sound
|
||||
|
||||
Add a subtle ambient layer to fill silence and create depth:
|
||||
- Room tone / air conditioning hum (very low, -35 dB)
|
||||
- Environmental sounds matching the topic (city, nature, lab)
|
||||
- Generates "presence" even during narration pauses
|
||||
|
||||
## Color Grading for Cinematic
|
||||
|
||||
| Look | Profile | Intensity | Characteristics |
|
||||
|------|---------|-----------|----------------|
|
||||
| **Warm cinematic** | `cinematic_warm` | 0.85 | Orange highlights, lifted shadows |
|
||||
| **Teal & orange** | `cinematic_cool` | 0.7 | Classic Hollywood blockbuster look |
|
||||
| **Moody dark** | `moody_dark` | 0.6 | Crushed blacks, low saturation |
|
||||
| **Vintage film** | `vintage_film` | 0.7 | Faded, warm tint, reduced contrast |
|
||||
|
||||
**Cinematic grading rules:**
|
||||
- Shadows should be slightly lifted (never pure black)
|
||||
- Highlights should be slightly rolled off (never pure white)
|
||||
- Skin tones must stay on the vectorscope skin tone line
|
||||
- Consistency across all clips — one LUT/profile for the entire video
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When building cinematic-style content:
|
||||
|
||||
1. **Set aspect ratio** — use 2.39:1 letterbox for true cinematic, or 16:9 with `cinematic_warm` grade for subtle
|
||||
2. **Render at 24fps** if the content is purely generated/animated (set in `video_compose`)
|
||||
3. **Shot duration 4-8 seconds average** — vary deliberately, never same length 3x
|
||||
4. **Layer audio** — narration + music + ambient minimum; add Foley SFX at key moments
|
||||
5. **Music at 60-90 BPM**, dynamic (not looping) — use `music_gen` with "cinematic orchestral" prompt
|
||||
6. **Remove music for 3-5 seconds** at key reveals — silence is powerful
|
||||
7. **Color grade with `cinematic_warm` or `cinematic_cool`** at 0.7-0.85 intensity
|
||||
8. **Image prompts** should include "cinematic lighting, shallow depth of field, film grain" for matching aesthetic
|
||||
9. **Slower narration** — 140-150 WPM (slower than standard 155 WPM explainer pace)
|
||||
@@ -0,0 +1,344 @@
|
||||
# Data Visualization Strategy Skill
|
||||
|
||||
## When to Use
|
||||
|
||||
Apply this skill when a scene requires presenting data visually: statistics, comparisons,
|
||||
trends, compositions, or key metrics. This skill guides chart type selection, animation
|
||||
sequencing, label placement, data density, and color usage to produce charts that are
|
||||
clear, accurate, and effective in video.
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | Role |
|
||||
|------|------|
|
||||
| `diagram_gen` | Generate charts via Mermaid or D3 |
|
||||
| `image_selector` | Generate stylized chart illustrations (FLUX/DALL-E) |
|
||||
| Remotion | Animated chart components (bar grow, line draw, pie fill) |
|
||||
| Manim | Mathematical plots, coordinate systems, function graphs |
|
||||
|
||||
## Chart Type Decision Tree
|
||||
|
||||
Follow this tree top-to-bottom. Stop at the first match.
|
||||
|
||||
```
|
||||
Is there data to visualize?
|
||||
NO -> Use a text card or stat card instead
|
||||
YES -> How many data points?
|
||||
< 3 -> Use text or stat card (charts look empty with 1-2 points)
|
||||
3-9 -> Continue to "What story does the data tell?"
|
||||
> 12 -> Simplify first: aggregate into top-N + "Other", then continue
|
||||
|
||||
What story does the data tell?
|
||||
|
|
||||
|-- Comparing quantities across categories?
|
||||
| -> BAR CHART (horizontal if labels are long)
|
||||
|
|
||||
|-- Showing a trend over time?
|
||||
| -> LINE CHART (area chart if showing volume)
|
||||
|
|
||||
|-- Showing parts of a whole?
|
||||
| -> PIE / DONUT CHART (max 5-6 slices)
|
||||
| (If > 6 categories, aggregate smallest into "Other")
|
||||
|
|
||||
|-- Showing key metrics / KPIs?
|
||||
| -> KPI GRID (3-6 stat cards in a grid layout)
|
||||
|
|
||||
|-- Showing ranking or ordered list?
|
||||
| -> HORIZONTAL BAR CHART (sorted descending)
|
||||
|
|
||||
|-- Showing before/after or change?
|
||||
| -> PAIRED BAR CHART or STAT CARD with delta arrow
|
||||
|
|
||||
|-- Showing correlation between two variables?
|
||||
| -> LINE CHART with dual series (avoid scatter in video -- too dense)
|
||||
|
|
||||
|-- None of the above?
|
||||
| -> Default to BAR CHART (most universally readable)
|
||||
```
|
||||
|
||||
### When NOT to Use a Chart
|
||||
|
||||
| Situation | Do This Instead |
|
||||
|-----------|----------------|
|
||||
| Fewer than 3 data points | Stat card or text overlay: "Revenue grew 40% to $2.1M" |
|
||||
| More than 12 categories | Aggregate into top 5-7 + "Other", then chart |
|
||||
| Single number to emphasize | Full-screen stat card with impact animation |
|
||||
| Qualitative comparison | Side-by-side images or text table |
|
||||
| Data requires 30+ seconds to read | Split into multiple simpler charts across scenes |
|
||||
|
||||
## Animation Sequencing
|
||||
|
||||
Every chart in video should be animated. Static charts feel like slides, not video.
|
||||
|
||||
### Pattern: Build-Up (Default)
|
||||
|
||||
Show empty axes/frame, then animate data in.
|
||||
|
||||
```
|
||||
Frame 0.0s: Empty chart frame (axes, title, gridlines visible)
|
||||
Frame 0.3s: First data element begins animating in
|
||||
Frame 2.0s: All data elements fully rendered
|
||||
Frame 2.0-5.0s: Hold for readability
|
||||
```
|
||||
|
||||
- **Bar charts:** Bars grow upward from baseline (stagger left-to-right, 0.1s delay each)
|
||||
- **Line charts:** Line draws left-to-right following the data path
|
||||
- **Pie/donut charts:** Slices fill clockwise from 12 o'clock, largest slice first
|
||||
- **KPI grid:** Numbers count up from 0 to final value (odometer effect)
|
||||
|
||||
### Pattern: Narrative Highlight
|
||||
|
||||
Highlight one element at a time as narration mentions it.
|
||||
|
||||
```
|
||||
Frame 0.0s: Full chart visible but all elements at 30% opacity (desaturated)
|
||||
Frame 0.5s: First highlighted element goes full color + slight scale-up
|
||||
Frame 3.0s: First element returns to normal, second element highlights
|
||||
...continue for each narrated point
|
||||
```
|
||||
|
||||
Use when the narrator walks through specific data points. Keeps viewer focus synchronized
|
||||
with the voiceover.
|
||||
|
||||
### Pattern: Comparison Reveal
|
||||
|
||||
Show baseline, then animate the change.
|
||||
|
||||
```
|
||||
Frame 0.0s: Baseline data visible (e.g., "Before" bars)
|
||||
Frame 2.0s: Hold baseline for comprehension
|
||||
Frame 2.5s: Animate change (bars grow/shrink to "After" values)
|
||||
Frame 3.5s: Delta labels appear (+40%, -15%, etc.)
|
||||
Frame 3.5-7.0s: Hold for readability
|
||||
```
|
||||
|
||||
Use for before/after, year-over-year, or A/B comparisons.
|
||||
|
||||
### Timing Rules
|
||||
|
||||
| Element | Animation Duration | Hold Duration |
|
||||
|---------|-------------------|---------------|
|
||||
| Chart build-up | 2-4 seconds | 3-5 seconds |
|
||||
| Single element highlight | 0.3-0.5 seconds | 2-3 seconds |
|
||||
| Comparison transition | 1-2 seconds | 3-5 seconds |
|
||||
| KPI counter | 1.5-2 seconds | 2-3 seconds |
|
||||
| Label/annotation appear | 0.2-0.3 seconds | Remains on screen |
|
||||
|
||||
**Critical rule:** The chart must be fully built and held for at least 3 seconds before the
|
||||
scene transitions. Viewers need time to read. If the narration moves on before the chart is
|
||||
readable, either extend the scene or simplify the chart.
|
||||
|
||||
## Label Placement Rules
|
||||
|
||||
### Bar Charts
|
||||
|
||||
```
|
||||
Vertical bars:
|
||||
- Value labels: ABOVE each bar (or INSIDE if bar is tall enough for legible text)
|
||||
- Category labels: Below on x-axis, horizontal text
|
||||
- If labels overlap: rotate 45 degrees or use horizontal bars instead
|
||||
- Y-axis: include gridlines, omit axis label if title makes it obvious
|
||||
|
||||
Horizontal bars:
|
||||
- Value labels: TO THE RIGHT of each bar
|
||||
- Category labels: Left-aligned on y-axis
|
||||
- Preferred when category names are longer than 2 words
|
||||
```
|
||||
|
||||
### Line Charts
|
||||
|
||||
```
|
||||
- Endpoint labels: Show value at the last data point (right end)
|
||||
- Start label: Show value at the first data point (left end) for context
|
||||
- Dense data (>7 points): Label only start, end, and notable peaks/valleys
|
||||
- Avoid: Labels on every point (creates clutter in video)
|
||||
- Legend: Top-right or inline (label next to the line) for multi-series
|
||||
```
|
||||
|
||||
### Pie / Donut Charts
|
||||
|
||||
```
|
||||
- Large slices (>= 10%): Label INSIDE the slice (percentage + category)
|
||||
- Small slices (< 10%): Label OUTSIDE with leader line connecting to slice
|
||||
- Center of donut: Use for total value or key metric label
|
||||
- Maximum: 5-6 slices. Combine anything under 5% into "Other"
|
||||
- Always show percentages, not just raw values
|
||||
```
|
||||
|
||||
### KPI Grid
|
||||
|
||||
```
|
||||
- Large number: Center of each card, using stat_card font (3-4x body size)
|
||||
- Label: Below the number, smaller font, describes the metric
|
||||
- Delta indicator: Small arrow + percentage showing change (green up, red down)
|
||||
- Grid: 2x2 or 3x2 layout, evenly spaced, consistent card sizing
|
||||
```
|
||||
|
||||
### Universal Label Rules
|
||||
|
||||
- **Title visible:** Every chart must have a clear title (top-left or top-center)
|
||||
- **Source citation:** If data is from an external source, show "Source: [name]" in small text at bottom
|
||||
- **Units:** Always show units (%, $, seconds, etc.) either in the title or on the axis
|
||||
- **No orphan labels:** Every visual element must be labeled or explained by the narration
|
||||
|
||||
## Data Density vs Readability
|
||||
|
||||
### The Video Rule: Less Is More
|
||||
|
||||
Video is not a spreadsheet. The viewer cannot pause, scroll, or zoom. Every data point
|
||||
competes for attention in a 5-7 second window.
|
||||
|
||||
```
|
||||
Ideal data points per chart type:
|
||||
Bar chart: 5-7 bars (max 9)
|
||||
Line chart: 5-12 points (max 15, but label sparsely)
|
||||
Pie chart: 3-5 slices (max 6)
|
||||
KPI grid: 3-6 metrics (max 6)
|
||||
```
|
||||
|
||||
### Simplification Strategies
|
||||
|
||||
| Problem | Solution |
|
||||
|---------|----------|
|
||||
| Too many categories (>9) | Show top 5-7, aggregate rest into "Other" |
|
||||
| Too many time periods | Aggregate (monthly -> quarterly, daily -> weekly) |
|
||||
| Multiple metrics to show | Split into separate charts across scenes |
|
||||
| Wide value ranges | Use normalized/percentage view instead of absolute |
|
||||
| Decimal precision | Round aggressively: $1,234,567 -> $1.2M |
|
||||
|
||||
### Font Size Minimums (at 1080p)
|
||||
|
||||
These are non-negotiable for readability on screens including mobile:
|
||||
|
||||
| Element | Minimum Size | Recommended |
|
||||
|---------|-------------|-------------|
|
||||
| Chart title | 32px | 36-40px |
|
||||
| Axis labels | 24px | 28px |
|
||||
| Value labels | 24px | 28px |
|
||||
| Annotations | 20px | 24px |
|
||||
| Source citation | 16px | 18px |
|
||||
|
||||
**Scaling rule:** For 4K output, multiply by 2x. For 720p, these minimums still apply
|
||||
(they are the floor).
|
||||
|
||||
## Color Usage
|
||||
|
||||
### Deriving Chart Colors from the Playbook
|
||||
|
||||
Charts must look like they belong to the video. Always derive colors from the active
|
||||
style playbook.
|
||||
|
||||
```
|
||||
Color derivation priority:
|
||||
1. playbook.visual_language.color_palette.chart_palette (if the playbook defines one)
|
||||
2. Derive from primary + accent colors:
|
||||
- Bar/slice 1: primary[0]
|
||||
- Bar/slice 2: accent[0]
|
||||
- Bar/slice 3: primary[1]
|
||||
- Bar/slice 4: accent[1]
|
||||
- Bar/slice 5+: generate by adjusting lightness of primary[0]
|
||||
3. Background: use playbook background color
|
||||
4. Text/labels: use playbook text color
|
||||
5. Gridlines: use playbook muted color at 50% opacity
|
||||
```
|
||||
|
||||
### Highlight and Focus
|
||||
|
||||
```
|
||||
Highlighting strategy:
|
||||
- KEY data point: Full saturation of accent[0], slight scale-up (1.05x)
|
||||
- FOCUS data points: Full saturation of their assigned color
|
||||
- NON-FOCUS points: Desaturate to 30% opacity or use muted color
|
||||
- BASELINE/CONTEXT: Dashed lines using muted color
|
||||
```
|
||||
|
||||
### Accessibility Rules
|
||||
|
||||
Never rely on color alone to convey meaning:
|
||||
|
||||
- **Add patterns:** Use hatching, dots, or stripes on bars/slices in addition to color
|
||||
- **Add labels:** Every bar/slice/line must have a text label, not just a legend
|
||||
- **Contrast:** Minimum 3:1 contrast ratio between adjacent chart elements
|
||||
- **Colorblind-safe:** Avoid red-green as the only differentiator. Prefer blue-orange or
|
||||
blue-yellow pairings when showing positive/negative
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Misleading Charts
|
||||
|
||||
| Pitfall | Why It Misleads | Fix |
|
||||
|---------|----------------|-----|
|
||||
| Truncated y-axis (not starting at 0) | Small differences look enormous | Always start bar chart y-axis at 0 |
|
||||
| 3D charts | Perspective distorts size perception | Always use 2D flat charts |
|
||||
| Dual y-axes with different scales | Implies false correlation | Use two separate charts side by side |
|
||||
| Cherry-picked time range | Hides broader context | Show full relevant range or acknowledge truncation |
|
||||
| Pie chart with too many slices | Impossible to compare small angles | Max 5-6 slices, aggregate rest |
|
||||
|
||||
### Animation Mistakes
|
||||
|
||||
| Pitfall | Fix |
|
||||
|---------|-----|
|
||||
| Animation too fast (< 1.5s) | Viewers cannot track what appeared. Minimum 2s build-up |
|
||||
| No hold time after animation | Scene cuts away before chart is readable. Hold 3-5s minimum |
|
||||
| All elements appear at once | Loses the narrative. Stagger element entrance |
|
||||
| Gratuitous bouncing/spinning | Distracts from data. Use clean ease-in-out per playbook |
|
||||
|
||||
### Design Mistakes
|
||||
|
||||
| Pitfall | Fix |
|
||||
|---------|-----|
|
||||
| Too many colors (>5 in one chart) | Limit to 4-5 distinct colors. Aggregate or split charts |
|
||||
| Missing title | Every chart needs a title. Viewers have no other context |
|
||||
| Tiny font on mobile | Enforce minimums: 32px title, 24px labels at 1080p |
|
||||
| Decorative gridlines | Use light gridlines or none. They should aid reading, not decorate |
|
||||
| Dark text on dark background | Use playbook text color on playbook background. Check contrast |
|
||||
|
||||
## Integration with Scene Director
|
||||
|
||||
When the Scene Director identifies a data visualization need, apply this skill as follows:
|
||||
|
||||
1. **Determine chart type** using the decision tree above
|
||||
2. **Specify animation pattern** in the scene's `movement` field (e.g., "build-up: bars grow from baseline over 2s, hold 4s")
|
||||
3. **Include label specifications** in `overlay_notes` (e.g., "value labels above bars, title top-left, source bottom-right")
|
||||
4. **Reference playbook colors** in `required_assets` description (e.g., "bar chart using primary[0] #2563EB for main bars, accent[0] #F59E0B for highlight bar")
|
||||
5. **Set scene duration** to accommodate animation (2-4s) + hold (3-5s) = minimum 5s per chart scene
|
||||
|
||||
### Example Scene Specification
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "scene-7",
|
||||
"type": "animation",
|
||||
"description": "Horizontal bar chart comparing response times: Traditional DB 450ms, Vector DB 12ms, Cached 3ms. Bars grow left-to-right with stagger. Vector DB bar highlighted in accent color. Hold for readability.",
|
||||
"start_seconds": 32,
|
||||
"end_seconds": 40,
|
||||
"script_section_id": "s5",
|
||||
"framing": "full-screen chart, centered with generous padding",
|
||||
"movement": "build-up: bars grow from left over 2.5s with 0.3s stagger, hold 5s",
|
||||
"transition_in": "fade",
|
||||
"transition_out": "dissolve",
|
||||
"overlay_notes": "Title: 'Query Response Time Comparison'. Value labels right of bars (ms units). Source: 'Benchmark 2024' bottom-right 16px. Vector DB bar uses accent[0], others use primary[0] at 50% opacity.",
|
||||
"required_assets": [
|
||||
{
|
||||
"type": "chart_data",
|
||||
"description": "Horizontal bar chart data: Traditional DB 450ms, Vector DB 12ms, Cached 3ms. Use playbook primary #2563EB at 50% for context bars, accent #F59E0B for Vector DB highlight bar.",
|
||||
"source": "generate"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
- [ ] Chart type matches the data story (not just "default to bar chart")
|
||||
- [ ] Data points within limits: 5-7 bars, 3-5 pie slices, 5-12 line points
|
||||
- [ ] Animation duration: 2-4s build, 3-5s hold minimum
|
||||
- [ ] All text meets size minimums: 32px title, 24px labels at 1080p
|
||||
- [ ] Colors derived from active playbook palette
|
||||
- [ ] Key data point has visual emphasis (highlight color, scale, or annotation)
|
||||
- [ ] No reliance on color alone for meaning (labels + patterns for accessibility)
|
||||
- [ ] Y-axis starts at 0 for bar charts
|
||||
- [ ] No 3D effects or perspective distortion
|
||||
- [ ] Title is visible and descriptive
|
||||
- [ ] Source cited if using external data
|
||||
- [ ] Chart is readable when paused at any frame during the hold period
|
||||
@@ -0,0 +1,111 @@
|
||||
# Diagram Generation Usage for OpenMontage
|
||||
|
||||
> Sources: Mermaid.js documentation, existing Layer 3 skill at `.agents/skills/beautiful-mermaid/`,
|
||||
> Mermaid-Sonar complexity analysis research, Mermaid GitHub issues #651 (scaling), #3029 (animation)
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
MAX NODES (1080p): 15-20 nodes, 20-25 edges
|
||||
MAX NODES (4K): 25-35 nodes, 35-45 edges
|
||||
MAX NODES (vert): 10-12 nodes, 12-15 edges
|
||||
MIN FONT SIZE: 16px at 1080p, 14px at 4K
|
||||
RENDER WIDTH: Minimum 1200px
|
||||
RENDER VIEWPORT: 3840x2160 (4K) for high-res PNG export
|
||||
THEME (dark bg): tokyo-night or dracula
|
||||
THEME (light bg): github-light or catppuccin-latte
|
||||
```
|
||||
|
||||
## Diagram Type Selection
|
||||
|
||||
| Type | Video Suitability | Best For |
|
||||
|------|------------------|----------|
|
||||
| **Flowchart (TD)** | Excellent | Process flows, decision trees, algorithms |
|
||||
| **Sequence diagram** | Good | API calls, user interactions, message flows |
|
||||
| **State diagram** | Good | State machines, lifecycle, workflow status |
|
||||
| **Class diagram** | Fair | Architecture (limit to 3-5 classes) |
|
||||
| **ER diagram** | Poor for video | Too dense — simplify to key entities only |
|
||||
| **Gantt chart** | Fair | Timelines, project phases |
|
||||
| **Mindmap** | Good | Concept overviews, topic breakdowns |
|
||||
|
||||
**Default:** Use flowcharts (top-down `TD`) unless the content specifically requires another type. They read naturally and build well step by step.
|
||||
|
||||
## Complexity Limits for Video
|
||||
|
||||
Video is transient — viewers can't zoom or scroll. Cut complexity in half compared to static documentation.
|
||||
|
||||
| Target Resolution | Max Nodes | Max Edges | Min Font Size (CSS) |
|
||||
|------------------|-----------|-----------|---------------------|
|
||||
| 1920x1080 (HD) | 15-20 | 20-25 | 16px |
|
||||
| 3840x2160 (4K) | 25-35 | 35-45 | 14px |
|
||||
| 1080x1920 (vertical) | 10-12 | 12-15 | 18px |
|
||||
|
||||
**If your diagram exceeds these limits:** Split it into multiple frames, each showing a subset. This also creates a natural "building" animation for the video.
|
||||
|
||||
## Color Themes for Video
|
||||
|
||||
| Use Case | Theme | Why |
|
||||
|----------|-------|-----|
|
||||
| Dark video background | `tokyo-night` or `dracula` | High contrast, readable |
|
||||
| Light video background | `github-light` or `catppuccin-latte` | Soft, professional |
|
||||
| Code/developer content | `one-dark` | Familiar to dev audience |
|
||||
| Maximum contrast | `zinc-dark` | Neutral, no color bias |
|
||||
| Corporate/presentation | `nord-light` | Calm, professional |
|
||||
|
||||
**Match the playbook:** The diagram theme should complement the style playbook's color palette.
|
||||
|
||||
## Progressive Building for Video
|
||||
|
||||
Mermaid doesn't animate natively. Use progressive rendering to create a "building" effect:
|
||||
|
||||
### Approach: Multi-Stage Renders
|
||||
|
||||
1. Render diagram in stages — first 2 nodes, then 4, then full diagram
|
||||
2. Each stage is a separate Mermaid render → SVG → PNG
|
||||
3. Crossfade or cut between stages in FFmpeg
|
||||
4. Viewers follow the logic step by step
|
||||
|
||||
### Highlighting Current Step
|
||||
|
||||
Use `classDef` to highlight the active node and dim completed ones:
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Input Data]:::completed --> B[Process]:::highlight
|
||||
B --> C[Output]:::dimmed
|
||||
classDef highlight fill:#f96,stroke:#333,stroke-width:3px
|
||||
classDef completed fill:#6c6,stroke:#333,stroke-width:1px
|
||||
classDef dimmed fill:#555,stroke:#333,opacity:0.5
|
||||
```
|
||||
|
||||
Generate one PNG per step with different `classDef` assignments, then sequence them in the compose stage.
|
||||
|
||||
## Styling for Video Readability
|
||||
|
||||
### Node Sizing
|
||||
- Minimum node width: 150px at 1080p
|
||||
- Padding inside nodes: 15-20px
|
||||
- Keep text to 3-5 words per node — use abbreviations if needed
|
||||
|
||||
### Edge Labels
|
||||
- Keep to 1-2 words maximum
|
||||
- Use edge labels only when the relationship isn't obvious from context
|
||||
- Prefer labeled nodes over labeled edges
|
||||
|
||||
### Layout Direction
|
||||
- **Top-down (TD):** Best for processes, hierarchies, flows
|
||||
- **Left-right (LR):** Best for timelines, sequences, pipelines
|
||||
- Avoid bottom-up (BT) — counterintuitive for most viewers
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When using the `diagram_gen` tool:
|
||||
|
||||
1. **Check complexity** — max 15-20 nodes at 1080p. Split larger diagrams into multiple frames
|
||||
2. **Choose theme** to match the video's style playbook and background
|
||||
3. **Use progressive building** — render stages and crossfade for "building" effect in video
|
||||
4. **Highlight with classDef** — show the current step in orange/red, completed in green, upcoming in grey
|
||||
5. **Keep text minimal** — 3-5 words per node, 1-2 words per edge label
|
||||
6. **Default to flowchart TD** unless the content specifically requires another diagram type
|
||||
7. **Render at 4K viewport** (3840x2160) even for 1080p output — ensures crisp text when scaled
|
||||
8. **Test readability** — view the rendered PNG at actual video frame size before composing
|
||||
@@ -0,0 +1,111 @@
|
||||
# Enhancement Strategy Skill
|
||||
|
||||
## When to Use
|
||||
|
||||
Apply this skill when deciding how to enhance talking-head footage: which
|
||||
face/color/audio presets to use, what overlays to add, and how to balance
|
||||
enhancement visibility with naturalness.
|
||||
|
||||
## Enhancement Tools
|
||||
|
||||
| Tool | What It Does | Recommended Preset |
|
||||
|------|-------------|-------------------|
|
||||
| `face_enhance` | Skin smoothing, sharpening, tone correction | `talking_head_standard` |
|
||||
| `color_grade` | Cinematic color look with intensity control | `cinematic_warm` at 0.85 |
|
||||
| `audio_enhance` | Loudness normalization, noise reduction, EQ | `clean_speech` |
|
||||
| `code_snippet` | Render code as styled overlay image | `monokai` theme |
|
||||
| `diagram_gen` | Generate box/flow diagrams as overlay images | `dark` theme |
|
||||
| `image_selector` | AI-generated illustrations (requires API key) | — |
|
||||
|
||||
## Enhancement Chain
|
||||
|
||||
Apply in this order — each step is optional and gracefully skipped on failure:
|
||||
|
||||
```
|
||||
raw footage
|
||||
→ subtitle burn (video_compose)
|
||||
→ face enhance (face_enhance)
|
||||
→ color grade (color_grade)
|
||||
→ audio enhance (audio_enhance)
|
||||
→ final encode (video_compose)
|
||||
```
|
||||
|
||||
### Face Enhancement Presets
|
||||
|
||||
| Preset | When to Use |
|
||||
|--------|-------------|
|
||||
| `talking_head_standard` | Default for any talking head — smoothing + sharpening + warm |
|
||||
| `soft_skin` | Webcam footage with visible pores — gentle smoothing |
|
||||
| `sharpen` | Soft/blurry camera — adds edge definition |
|
||||
| `brighten` | Dark/underlit footage — lifts shadows and midtones |
|
||||
| `denoise` | Grainy footage (low light, high ISO) — temporal noise reduction |
|
||||
|
||||
### Color Grade Profiles
|
||||
|
||||
| Profile | Look | Intensity |
|
||||
|---------|------|-----------|
|
||||
| `cinematic_warm` | Warm highlights, lifted shadows, slight saturation | 0.85 |
|
||||
| `cinematic_cool` | Teal shadows, orange highlights | 0.7 |
|
||||
| `bright_clean` | Vivid, lifted, YouTube-style | 0.8 |
|
||||
| `moody_dark` | Crushed blacks, desaturated — dramatic | 0.6 |
|
||||
| `neutral` | Minimal correction — just normalizes levels | 1.0 |
|
||||
|
||||
### Audio Enhancement Presets
|
||||
|
||||
| Preset | When to Use | Target |
|
||||
|--------|-------------|--------|
|
||||
| `clean_speech` | Default talking head — full processing chain | -16 LUFS |
|
||||
| `voice_clarity` | Speaker sounds muddy — boosts 3kHz/5kHz presence | -16 LUFS |
|
||||
| `podcast` | Interview/podcast — heavier compression | -16 LUFS |
|
||||
| `noise_reduce` | Noisy environment — aggressive FFT denoising | -16 LUFS |
|
||||
| `normalize_only` | Clean source that just needs loudness matching | -16 LUFS |
|
||||
|
||||
## Overlay Enhancement Types
|
||||
|
||||
| Type | When to Use | Tool | Placement |
|
||||
|------|-------------|------|-----------|
|
||||
| **Text overlay** | Key terms, statistics, quotes | video_compose overlay | Upper or lower third |
|
||||
| **Code snippet** | Technical content, API examples | code_snippet → overlay | Side of frame or full-screen |
|
||||
| **Diagram** | Explaining a concept visually | diagram_gen → overlay | Side of frame or full-screen |
|
||||
| **Lower third** | Speaker name, topic label | video_compose overlay | Bottom 20% of frame |
|
||||
|
||||
## Overlay Density Guidelines
|
||||
|
||||
### Short-form (< 60 seconds)
|
||||
- High density: overlay every 3-5 seconds
|
||||
- Quick visual changes, bold text
|
||||
- Subtitles **mandatory** (most viewers watch muted)
|
||||
|
||||
### Medium-form (1-10 minutes)
|
||||
- Moderate density: overlay every 10-20 seconds
|
||||
- Let the speaker carry sections without visual competition
|
||||
|
||||
### Long-form (> 10 minutes)
|
||||
- Low density: overlay every 30-60 seconds
|
||||
- Only enhance when the content benefits (key points, complex topics)
|
||||
|
||||
## Placement Rules
|
||||
|
||||
1. **Never cover the speaker's face** — eyes, nose, mouth must remain visible
|
||||
2. **Subtitles go in the bottom 20%** — margin_v: 50 for vertical, 40 for horizontal
|
||||
3. **Consistent positioning** — once you place overlays on the left, keep them there
|
||||
4. **Text overlays: 2-5 seconds on screen** — long enough to read, short enough to not feel stuck
|
||||
|
||||
## Deciding What to Enhance
|
||||
|
||||
For each section of the script, ask:
|
||||
|
||||
1. Is the speaker explaining something visual? → Add a diagram (`diagram_gen`)
|
||||
2. Is there a key statistic or quote? → Add a text overlay
|
||||
3. Is there code or technical content? → Add a code screenshot (`code_snippet`)
|
||||
4. Has the speaker been on camera > 30 seconds straight? → Consider B-roll or overlay
|
||||
5. Is this the intro or conclusion? → Bold text overlay with the key message
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
- [ ] Face enhancement looks natural — not over-smoothed or orange
|
||||
- [ ] Color grade is visible but subtle — skin tones look healthy
|
||||
- [ ] Audio is normalized to target LUFS — consistent volume throughout
|
||||
- [ ] Subtitles are readable on mobile, positioned below the face
|
||||
- [ ] Overlays add value (not just decoration)
|
||||
- [ ] Enhancement density matches content length and platform
|
||||
@@ -0,0 +1,96 @@
|
||||
# Face Restoration Usage for OpenMontage
|
||||
|
||||
> Sources: CodeFormer paper (Zhou et al. 2022), GFPGAN documentation, Real-ESRGAN upsampling docs,
|
||||
> existing Layer 2 skill at `skills/creative/enhancement-strategy.md`
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
DEFAULT MODEL: CodeFormer with fidelity 0.5
|
||||
ALTERNATIVE: GFPGAN (faster, less controllable)
|
||||
FIDELITY RANGE: 0 = max quality enhancement, 1 = max faithfulness to input
|
||||
BG UPSAMPLER: Enable to also upscale the background (Real-ESRGAN)
|
||||
PROCESSING ORDER: face_restore BEFORE face_enhance — restore first, polish second
|
||||
```
|
||||
|
||||
## CRITICAL DISTINCTION — face_restore vs face_enhance
|
||||
|
||||
| Tool | What It Does | When to Use |
|
||||
|------|-------------|-------------|
|
||||
| `face_enhance` | FFmpeg filter chains — skin smoothing, color balance, sharpening | Good-quality footage that needs polish |
|
||||
| `face_restore` | AI model reconstruction — rebuilds degraded face detail | Bad-quality footage with blur, compression, low-res faces |
|
||||
|
||||
**Decision rule:** If the face is recognizable and just needs polish, use `face_enhance`. If the face is degraded, blurry, or compressed beyond recognition, use `face_restore`.
|
||||
|
||||
## Model Selection
|
||||
|
||||
| Model | Strengths | Fidelity Control | Speed |
|
||||
|-------|----------|------------------|-------|
|
||||
| CodeFormer | Better quality, identity preservation, controllable | Yes (0-1 slider) | Slower |
|
||||
| GFPGAN | Good baseline, simpler | No | Faster |
|
||||
|
||||
### Fidelity Tuning (CodeFormer Only)
|
||||
|
||||
| Fidelity | Effect | Use Case |
|
||||
|----------|--------|----------|
|
||||
| 0.0 | Maximum enhancement — best visual quality but may alter identity | Unrecognizable faces, artistic use |
|
||||
| 0.3 | Strong restoration — good for very degraded faces | Old footage, heavy compression artifacts |
|
||||
| 0.5 | Balanced (default) — restoration + identity preservation | General-purpose restoration |
|
||||
| 0.7 | Conservative — mild cleanup, strong identity preservation | Webcam footage, light degradation |
|
||||
| 1.0 | Minimal change — essentially passthrough | Testing, comparison baseline |
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### 1. Old Footage Restoration
|
||||
|
||||
```
|
||||
face_restore (fidelity 0.3) → color_grade → compose
|
||||
```
|
||||
|
||||
Heavy restoration for archival/vintage footage where faces are significantly degraded.
|
||||
|
||||
### 2. Webcam Cleanup
|
||||
|
||||
```
|
||||
face_restore (fidelity 0.7) → face_enhance (talking_head_standard) → compose
|
||||
```
|
||||
|
||||
Light restoration followed by polish — best for modern but low-quality webcam footage.
|
||||
|
||||
### 3. Low-Res Face + Background Upscale
|
||||
|
||||
```
|
||||
face_restore (bg_upsampler=true) → compose
|
||||
```
|
||||
|
||||
Single-step restoration when both face and background need improvement.
|
||||
|
||||
### 4. Archival Photo for Talking Head
|
||||
|
||||
```
|
||||
face_restore → talking_head tool (SadTalker)
|
||||
```
|
||||
|
||||
Restore the source face image before feeding into the talking-head animation pipeline.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before accepting face_restore output, verify:
|
||||
|
||||
- [ ] Restored face is sharper and cleaner than input
|
||||
- [ ] Identity is preserved — the person is still recognizable
|
||||
- [ ] No hallucinated features (extra eyes, wrong skin texture, teeth artifacts)
|
||||
- [ ] Skin texture looks natural, not plastic/over-smoothed
|
||||
- [ ] Consistent across frames (for video) — no flickering between restored/unrestored quality
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When using the `face_restore` tool:
|
||||
|
||||
1. **Use face_restore BEFORE face_enhance** in the processing chain — restore first, polish second
|
||||
2. **Start with fidelity 0.5** and adjust based on visual inspection
|
||||
3. **For talking-head pipelines with poor source footage**, apply face_restore in the assets stage
|
||||
4. **Enable `bg_upsampler` only when both face AND background need improvement**
|
||||
5. **NEVER use face_restore on already-good footage** — it can introduce subtle artifacts
|
||||
6. **Compare input and output side-by-side** — the face should be recognizably the same person
|
||||
7. **For video, extract key frames and test face_restore settings** before processing full video
|
||||
@@ -0,0 +1,110 @@
|
||||
# Image Generation Usage for OpenMontage
|
||||
|
||||
> Sources: OpenAI DALL-E 3 documentation, FLUX/BFL API documentation, existing Layer 3 skills
|
||||
> at `.agents/skills/flux-best-practices/` and `.agents/skills/bfl-api/`
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
FLUX RESOLUTION: 1920x1088 (16:9) | 1088x1920 (9:16) — must be multiples of 16
|
||||
MAX TOTAL: 4 megapixels (width x height)
|
||||
CONSISTENCY: Use hero image as input_image for subsequent frames
|
||||
STYLE PREFIX: Set in playbook, prepend to every prompt
|
||||
BATCH STRATEGY: Hero at max quality → iterate with klein → final pass with pro
|
||||
```
|
||||
|
||||
## Resolution for Video Frames
|
||||
|
||||
All FLUX dimensions **must be multiples of 16**. Maximum total is 4MP.
|
||||
|
||||
| Target | FLUX Resolution | Cost (FLUX.2 pro) |
|
||||
|--------|----------------|-------------------|
|
||||
| YouTube 16:9 | `1920x1088` | $0.03/image |
|
||||
| YouTube 4K | `3840x2160` | Requires pro/max |
|
||||
| TikTok/Reels 9:16 | `1088x1920` | $0.03/image |
|
||||
| Square 1:1 | `1024x1024` | $0.03/image |
|
||||
| Thumbnail | `1280x720` | $0.03/image |
|
||||
|
||||
## Maintaining Visual Consistency
|
||||
|
||||
The biggest challenge: making 8-12 generated images look like they belong in the same video.
|
||||
|
||||
### Strategy 1 — Style Prefix (Always Use)
|
||||
|
||||
Prepend the playbook's `image_prompt_prefix` to every prompt. Example from `clean-professional`:
|
||||
|
||||
```
|
||||
"Clean, minimal illustration with soft shadows, muted color palette,
|
||||
white background, professional vector art style. [YOUR SCENE DESCRIPTION]"
|
||||
```
|
||||
|
||||
### Strategy 2 — Hero Reference Image (Recommended)
|
||||
|
||||
1. Generate one "hero" image at maximum quality (`FLUX.2 [max]`, $0.07)
|
||||
2. Use it as `input_image` for all subsequent frames:
|
||||
|
||||
```
|
||||
Frame 1: T2I with detailed prompt → hero.png
|
||||
Frame 2: I2I with hero.png + "Same style, camera pans right to show..."
|
||||
Frame 3: I2I with hero.png + "Same style, zoomed in on..."
|
||||
```
|
||||
|
||||
FLUX.2 supports up to 4 references (klein) or 8 references (pro/max/flex). Reference by number: "The character from image 1 in the environment from image 2."
|
||||
|
||||
### Strategy 3 — Seed Locking
|
||||
|
||||
Use the same `seed` parameter across generations with similar prompts. Produces similar compositions but is fragile to prompt changes — use as supplement, not primary strategy.
|
||||
|
||||
## Prompt Template
|
||||
|
||||
```
|
||||
[STYLE PREFIX from playbook].
|
||||
[SCENE DESCRIPTION: subject, action, environment].
|
||||
[LIGHTING: golden hour / overcast / studio softbox / dramatic side-light].
|
||||
[COMPOSITION: wide shot / medium shot / close-up / overhead / isometric].
|
||||
[CAMERA: Shot on [camera] with [lens] at [aperture]] (for photorealistic only).
|
||||
16:9 aspect ratio.
|
||||
```
|
||||
|
||||
### Style-Specific Prompt Patterns
|
||||
|
||||
| Style | Prompt Pattern |
|
||||
|-------|---------------|
|
||||
| **Flat illustration** | "Flat vector illustration, bold colors, clean edges, no gradients, white background" |
|
||||
| **Isometric** | "Isometric 3D illustration, 30-degree angle, clean geometric shapes, soft shadows" |
|
||||
| **Photorealistic** | "Photorealistic, shot on Canon EOS R5 with 85mm f/1.4, shallow depth of field" |
|
||||
| **Diagram-style** | "Technical diagram, labeled components, clean lines, minimal color, white background" |
|
||||
| **Watercolor** | "Soft watercolor illustration, muted tones, visible brush strokes, paper texture" |
|
||||
|
||||
## Batch Generation Strategy
|
||||
|
||||
| Phase | Model | Cost/Image | Purpose |
|
||||
|-------|-------|-----------|---------|
|
||||
| 1. Style guide | FLUX.2 [max] | $0.07 | One hero image, maximum quality |
|
||||
| 2. Storyboard iteration | FLUX.2 [klein] 9B | $0.015 | Rapid variations during planning |
|
||||
| 3. Final frames | FLUX.2 [pro] | $0.03 | Re-generate finals with hero as reference |
|
||||
|
||||
**Rate limit:** 24 concurrent requests max. Pipeline accordingly.
|
||||
|
||||
**Budget for 8-image explainer:** $0.07 (hero) + $0.12 (8x klein iterations) + $0.24 (8x pro finals) = ~$0.43
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
1. **Text in images** — AI image generators are unreliable with text. Never include text in prompts; add text as overlays in the compose stage
|
||||
2. **Hands and fingers** — DALL-E 3 and FLUX still struggle. Avoid prompts requiring detailed hand poses
|
||||
3. **Inconsistent characters** — Without reference images, the same character will look different each time. Always use the hero reference strategy
|
||||
4. **Over-prompting** — Long, complex prompts produce unpredictable results. Keep to 2-3 sentences
|
||||
5. **Ignoring the playbook** — Every image must match the style playbook. The style prefix is not optional
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When using the `image_selector` tool in the asset stage:
|
||||
|
||||
1. **Always prepend the playbook's style prefix** to every prompt
|
||||
2. **Generate a hero image first** at highest quality, use as reference for all others
|
||||
3. **Use `1920x1088`** for 16:9 video frames (FLUX multiple-of-16 requirement)
|
||||
4. **Never request text in images** — add text overlays in the compose stage
|
||||
5. **Budget check** — estimate total image cost before generating; switch to local diffusers if over budget
|
||||
6. **Iterate with klein** during planning, finalize with pro
|
||||
7. **Keep prompts to 2-3 sentences** — style prefix + scene description + composition
|
||||
8. **Match the scene plan** — each image maps to a specific scene in the script
|
||||
@@ -0,0 +1,111 @@
|
||||
# Image Provider Usage for OpenMontage
|
||||
|
||||
> How to choose between image generation and stock providers, and how to use each effectively.
|
||||
> Supplements the existing `image-gen-usage.md` (which covers FLUX prompting in depth).
|
||||
|
||||
## Provider Landscape
|
||||
|
||||
### Generation Providers (AI creates the image)
|
||||
|
||||
| Tool | Provider | Cost | Speed | Best For |
|
||||
|------|----------|------|-------|----------|
|
||||
| `flux_image` | FLUX 2 Pro via fal.ai | ~$0.03-0.05 | ~5-10s | Photorealism, general purpose, workhorse |
|
||||
| `openai_image` | GPT Image 1 (OpenAI) | ~$0.01-0.17 | ~5-15s | Complex instructions, text in images, multi-element |
|
||||
| `recraft_image` | Recraft V4 via fal.ai | ~$0.04-0.25 | ~5-10s | Logos, SVG vectors, brand assets, text rendering |
|
||||
| `local_diffusion` | Stable Diffusion (local) | Free | ~30s+ | Offline, privacy, free |
|
||||
| `image_gen` | Multi (legacy, deprecated) | Varies | Varies | **Deprecated** — use `image_selector` or per-provider tools |
|
||||
|
||||
### Stock Providers (search and download existing images)
|
||||
|
||||
| Tool | Provider | Cost | Speed | Best For |
|
||||
|------|----------|------|-------|----------|
|
||||
| `pexels_image` | Pexels | Free | ~2-5s | High-quality photography, color filtering |
|
||||
| `pixabay_image` | Pixabay | Free | ~2-5s | Large library, category filtering, illustrations |
|
||||
|
||||
### Selector
|
||||
|
||||
| Tool | Purpose |
|
||||
|------|---------|
|
||||
| `image_selector` | Routes to the best available provider based on preference and availability |
|
||||
|
||||
## Provider Selection by Scene Type
|
||||
|
||||
| Scene Type | Primary Provider | Why | Fallback |
|
||||
|-----------|-----------------|-----|----------|
|
||||
| **Real-world photo** (city, nature, people) | `pexels_image` | Real photos > AI for realism | `pixabay_image` → `flux_image` |
|
||||
| **Technical diagram** | `diagram_gen` | Structured, editable | `flux_image` with diagram prompt |
|
||||
| **Abstract/conceptual illustration** | `flux_image` | AI excels at custom concepts | `openai_image` |
|
||||
| **Logo or brand asset** | `recraft_image` | SVG support, text accuracy | `openai_image` |
|
||||
| **Image with text/labels** | `openai_image` | Best text rendering (GPT Image 1) | `recraft_image` |
|
||||
| **Complex multi-element composition** | `openai_image` | Best instruction following | `flux_image` |
|
||||
| **Hero image (key visual)** | `flux_image` | Highest visual quality | `openai_image` |
|
||||
| **Thumbnail** | `flux_image` or `recraft_image` | Needs to be eye-catching | — |
|
||||
| **Budget/free project** | `pexels_image` or `pixabay_image` | Free, immediate | `local_diffusion` |
|
||||
| **Offline/air-gapped** | `local_diffusion` | No network needed | — |
|
||||
|
||||
## Cost-Quality Tradeoff
|
||||
|
||||
```
|
||||
PRODUCTION PATH: Premium
|
||||
├── Hero images: flux_image ($0.05/img)
|
||||
├── Supporting visuals: flux_image ($0.03/img)
|
||||
├── Text overlays: openai_image ($0.04/img)
|
||||
├── B-roll stills: pexels_image ($0.00)
|
||||
└── Total for 10 images: ~$0.35
|
||||
|
||||
PRODUCTION PATH: Standard
|
||||
├── All generated: flux_image ($0.03/img)
|
||||
├── B-roll stills: pexels_image ($0.00)
|
||||
└── Total for 10 images: ~$0.25
|
||||
|
||||
PRODUCTION PATH: Budget
|
||||
├── All stock: pexels_image + pixabay_image ($0.00)
|
||||
├── Diagrams: diagram_gen ($0.00)
|
||||
└── Total: $0.00
|
||||
|
||||
PRODUCTION PATH: Offline
|
||||
├── All generated: local_diffusion ($0.00)
|
||||
├── Diagrams: diagram_gen ($0.00)
|
||||
└── Total: $0.00 (but slower, lower quality)
|
||||
```
|
||||
|
||||
## Using the Image Selector
|
||||
|
||||
For most cases, use `image_selector` and let it route:
|
||||
|
||||
```python
|
||||
# The selector finds the best available provider
|
||||
result = image_selector.execute({
|
||||
"prompt": "aerial view of a modern data center",
|
||||
"preferred_provider": "auto", # or "flux", "pexels", etc.
|
||||
"output_path": "assets/images/scene-3.png"
|
||||
})
|
||||
```
|
||||
|
||||
Override with `preferred_provider` when you know which provider is best for the scene type.
|
||||
Use `allowed_providers` to restrict to free or local options:
|
||||
|
||||
```python
|
||||
# Budget mode: only free providers
|
||||
result = image_selector.execute({
|
||||
"prompt": "server room interior",
|
||||
"allowed_providers": ["pexels", "pixabay", "local_diffusion"],
|
||||
"output_path": "assets/images/scene-3.jpg"
|
||||
})
|
||||
```
|
||||
|
||||
## Consistency Across Mixed Sources
|
||||
|
||||
When mixing stock and generated images in the same video, visual consistency is the challenge.
|
||||
|
||||
### Strategy: Color Grade Everything
|
||||
Apply the playbook's color grading LUT to both stock and generated images in the compose stage.
|
||||
This unifies the look. The `color_grade` enhancement tool handles this.
|
||||
|
||||
### Strategy: Match Playbook Style in Prompts
|
||||
When generating images, always prepend the playbook's `image_prompt_prefix`. When searching stock,
|
||||
use the playbook's color names in search filters (Pexels supports color filtering).
|
||||
|
||||
### Strategy: Avoid Mixing Styles Within a Scene
|
||||
Don't use a stock photo for one element and an AI illustration for another in the same scene.
|
||||
Keep each scene internally consistent — all stock or all generated.
|
||||
@@ -0,0 +1,133 @@
|
||||
# Lip Sync Usage for OpenMontage
|
||||
|
||||
> Sources: Wav2Lip paper (Prajwal et al. 2020), Wav2Lip-GAN documentation, OpenMontage
|
||||
> `tools/lip_sync.py` implementation
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
DEFAULT MODEL: wav2lip (faster, good sync accuracy)
|
||||
HIGHER QUALITY: wav2lip_gan (better visual quality, slower)
|
||||
FACE PADDING: [0, 10, 0, 0] (top, bottom, left, right)
|
||||
INPUT: Video with visible face + audio to sync to
|
||||
RESIZE FACTOR: 1 = full res (best), 2 = half res (recommended for drafts)
|
||||
KEY RULE: Use lip_sync for VIDEO input; use talking_head for PHOTO input
|
||||
```
|
||||
|
||||
## When to Use lip_sync
|
||||
|
||||
Lip sync is a **post-production** step. Use it after generating replacement audio.
|
||||
|
||||
- **Dubbing / localization** -- replace original speech with translated audio and match lips
|
||||
- **Audio replacement** -- re-record narration and sync to existing video
|
||||
- **Voice-over correction** -- fix mismatched audio/video timing
|
||||
- **NOT for photo-to-video** -- use the `talking_head` tool instead
|
||||
|
||||
## CRITICAL DISTINCTION -- lip_sync vs talking_head
|
||||
|
||||
| | `lip_sync` | `talking_head` |
|
||||
|---|---|---|
|
||||
| **Input** | Existing VIDEO + new audio | Still PHOTO + audio |
|
||||
| **Output** | Video with synced lips | New video from photo |
|
||||
| **Use Case** | Dubbing, audio replacement | Avatar generation, spokesperson |
|
||||
|
||||
**Decision rule:** If you already have video footage of the person speaking, use `lip_sync`. If you only have a photograph and want to make it talk, use `talking_head`.
|
||||
|
||||
## Model Selection Guide
|
||||
|
||||
| Model | Quality | Speed | Best For |
|
||||
|-------|---------|-------|----------|
|
||||
| `wav2lip` | Good lip sync, may blur chin | Faster | Quick dubbing, drafts |
|
||||
| `wav2lip_gan` | Better visual quality around mouth | Slower | Final renders, close-ups |
|
||||
|
||||
**Decision rule:** Use `wav2lip` for iteration and drafts. Switch to `wav2lip_gan` for final renders or any shot where the face is prominent (close-ups, medium shots). The quality difference is most visible in the mouth and chin region.
|
||||
|
||||
## Input Requirements
|
||||
|
||||
- Video must contain a clearly visible face throughout
|
||||
- Face should be front-facing or at most 30-degree angle
|
||||
- Minimum face size: ~100px across
|
||||
- Audio should be clean speech (not music or noise)
|
||||
- Audio length should roughly match video length (within 10%)
|
||||
|
||||
## Face Padding
|
||||
|
||||
Face padding controls how much area around the detected face is included in the sync region. Format: `[top, bottom, left, right]`.
|
||||
|
||||
| Scenario | Padding | Reason |
|
||||
|----------|---------|--------|
|
||||
| Default talking-head shot | `[0, 10, 0, 0]` | Works for 90% of footage |
|
||||
| Chin being cut off | Increase index 1 (bottom) | Extends the mask below the chin |
|
||||
| Forehead getting cropped | Increase index 0 (top) | Extends the mask above the forehead |
|
||||
| Face off-center in frame | Adjust indices 2, 3 (left, right) | Compensates for lateral offset |
|
||||
|
||||
Keep left/right at 0 unless the face is noticeably off-center in the frame.
|
||||
|
||||
## Resize Factor
|
||||
|
||||
| Value | Resolution | Quality | Speed | Use Case |
|
||||
|-------|-----------|---------|-------|----------|
|
||||
| 1 | Full | Best | Slowest | Final renders |
|
||||
| 2 | Half | Good | Faster | Drafts, iteration |
|
||||
| 3+ | Reduced | Degraded | Fastest | Quick previews only |
|
||||
|
||||
**Recommendation:** Use `resize_factor=2` during iteration, `resize_factor=1` for final output.
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### 1. Localization Dubbing
|
||||
|
||||
Translate a video into another language with matched lip movements.
|
||||
|
||||
```
|
||||
transcriber(video) --> transcript
|
||||
--> translate script to target language
|
||||
--> tts_selector(translated_script, target_language_voice)
|
||||
--> lip_sync(original_video, translated_audio)
|
||||
```
|
||||
|
||||
### 2. Audio Re-Record
|
||||
|
||||
Replace narration audio and re-sync the speaker's lips.
|
||||
|
||||
```
|
||||
new_audio_recording
|
||||
--> lip_sync(original_video, new_audio)
|
||||
--> face_enhance (post-sync cleanup)
|
||||
--> compose
|
||||
```
|
||||
|
||||
### 3. Multi-Language Output
|
||||
|
||||
Produce multiple language versions from a single source video.
|
||||
|
||||
```
|
||||
source_video
|
||||
--> lip_sync(source_video, english_audio) --> english_output
|
||||
--> lip_sync(source_video, spanish_audio) --> spanish_output
|
||||
--> lip_sync(source_video, french_audio) --> french_output
|
||||
```
|
||||
|
||||
Keep the original video as the source for each language -- do not chain lip_sync outputs.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before moving to the compose stage, verify each lip_sync output:
|
||||
|
||||
- [ ] **Lip movements match the new audio naturally** -- no desync or lag
|
||||
- [ ] **No visual artifacts around the mouth/chin area** -- no blurring, smearing, or color mismatch
|
||||
- [ ] **Face region blends seamlessly with the rest of the frame** -- no visible boundary
|
||||
- [ ] **No temporal flickering between frames** -- smooth frame-to-frame transitions
|
||||
- [ ] **Audio-visual sync is tight** -- no perceptible delay between mouth movement and sound
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When using the `lip_sync` tool in post-production:
|
||||
|
||||
1. **Generate the replacement audio FIRST** (`tts_selector`, `elevenlabs_tts`, `openai_tts`, or `piper_tts`), then lip sync -- lip_sync requires finished audio as input
|
||||
2. **Use `wav2lip` for drafts and iteration, `wav2lip_gan` for final renders** -- save processing time during the creative loop
|
||||
3. **Apply `face_enhance` AFTER lip_sync, not before** -- lip_sync modifies the face region, so enhancing before sync is wasted work
|
||||
4. **For localization workflows, keep the original video as source** and sync each language separately -- never chain lip_sync outputs
|
||||
5. **Check that audio length matches video length before syncing** -- trim or pad audio if needed to stay within 10% of video duration
|
||||
6. **Face padding `[0, 10, 0, 0]` works for 90% of talking-head footage** -- only adjust if you see cropping artifacts
|
||||
7. **For close-up shots, always use `wav2lip_gan`** -- the quality difference is visible at this framing
|
||||
@@ -0,0 +1,239 @@
|
||||
# Long-Form Video Pipeline (10+ Minutes)
|
||||
|
||||
> Sources: YouTube Creator Academy, VidIQ analytics research, Think Media production guides,
|
||||
> Paddy Galloway retention analytics, Retention Rabbit 2025 Benchmark Report, AIR Media-Tech
|
||||
> retention editing guide, Epidemic Sound mixing guide, Sweetwater YouTube mastering
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
DURATION: 8-15 min (sweet spot for most topics)
|
||||
HOOK: Complete by 0:30 — survive the 30-second cliff
|
||||
PATTERN INTERRUPT: Every 45-90 seconds
|
||||
RETENTION TARGET: 40-60% average view duration
|
||||
CHAPTER LENGTH: 2-4 minutes per chapter
|
||||
NARRATION: 150-160 WPM
|
||||
MUSIC BED: Continuous, ducked 18-20 dB below speech
|
||||
TARGET LUFS: -14 LUFS integrated
|
||||
END SCREEN: Last 20 seconds (YouTube end screen cards)
|
||||
```
|
||||
|
||||
## Retention Benchmarks (2025-2026 Data)
|
||||
|
||||
| Video Duration | Good Retention | Excellent Retention |
|
||||
|---------------|---------------|-------------------|
|
||||
| 1-3 min | 60%+ | 75%+ |
|
||||
| 3-5 min | 50%+ | 65%+ |
|
||||
| 5-10 min | 45%+ | 60%+ |
|
||||
| **10-20 min** | **40%+** | **55%+** |
|
||||
| 20-60 min | 35%+ | 50%+ |
|
||||
|
||||
- Platform average: **23.7%** across all YouTube videos
|
||||
- Only **16.8%** of videos exceed 50% retention
|
||||
- Only **16%** of viewers reach the final 10 seconds
|
||||
- **Improving retention by 10 percentage points** correlates with 25%+ increase in impressions
|
||||
|
||||
### AI-Generated Content Warning
|
||||
|
||||
- AI-generated video shows **70% lower retention** vs human-fronted content
|
||||
- AI narration triggers **35% viewer drop-off** within the first 45 seconds vs human narration
|
||||
- **Implication for OpenMontage:** Prioritize natural-sounding TTS (ElevenLabs over Piper), and avoid detectable AI visual artifacts. The processing chain in `sound-design.md` is essential.
|
||||
|
||||
## Retention Curve Management
|
||||
|
||||
### The Critical Points
|
||||
|
||||
| Timestamp | What Happens | How to Survive |
|
||||
|-----------|-------------|----------------|
|
||||
| 0:00-0:03 | Thumbnail-to-video match | First frame must match thumbnail promise |
|
||||
| 0:00-0:30 | **55%+ leave in first 60s** | Hook + tension must be complete by 0:30. Must retain 70%+ here. |
|
||||
| 2:00-3:00 | **Retention valley** — initial curiosity spent | Deliver first major payoff BEFORE 2:00, pattern interrupt at 1:45 |
|
||||
| 55-65% mark | **Secondary exodus** in long-form | Re-engage with burst sequence + open loop resolution |
|
||||
| Last 20s | End screen opportunity | CTA + end screen cards |
|
||||
|
||||
### Survival Tactics for the 2-3 Minute Valley
|
||||
|
||||
1. **Open loops in first 60 seconds** — raise a question early, hold the answer until later
|
||||
2. **First major payoff before 2:00** — the hook's promise must have a down-payment
|
||||
3. **Pattern interrupt at 1:45-2:00** — camera angle shift, B-roll burst, music change
|
||||
4. **"Burst sequence" at the valley** — 5-10 quick cuts lasting 10-15 seconds, then return to calm
|
||||
5. **Foreshadowing cue** — "But the really surprising part is coming up in a minute"
|
||||
|
||||
### Pattern Interrupts
|
||||
|
||||
Deploy **major interrupts** every **60-90 seconds** and **minor interrupts** every **20-30 seconds**:
|
||||
|
||||
| Technique | Type | When to Use |
|
||||
|-----------|------|-------------|
|
||||
| B-roll cut | Minor | Every 30-60s of talking head |
|
||||
| Visual style change | Major | New section, new concept |
|
||||
| On-screen text/graphic | Minor | Key stat, definition, emphasis |
|
||||
| Music energy shift | Major | Section transitions |
|
||||
| Direct address | Minor | "Now here's what's interesting..." |
|
||||
| Burst sequence (5-10 rapid cuts) | Major | Every 2-3 minutes |
|
||||
| Sound effect | Minor | Transition whoosh, pop for text |
|
||||
|
||||
**Impact:** Videos using pattern interrupts in the first 5 seconds achieve **23% higher average retention**.
|
||||
|
||||
### Re-Engagement Hooks
|
||||
|
||||
Place a **re-hook** at the 2-minute mark and every 3-4 minutes after:
|
||||
|
||||
```
|
||||
"But that's not even the interesting part..."
|
||||
"Now here's where it gets weird..."
|
||||
"Most people stop here, but if you keep watching..."
|
||||
"This next part changes everything..."
|
||||
```
|
||||
|
||||
These verbal signposts give viewers a reason to stay through the next segment.
|
||||
|
||||
## Content Structure
|
||||
|
||||
### Chapter Template
|
||||
|
||||
```
|
||||
[INTRO] 0:00 - 0:30 Hook + stakes + preview
|
||||
[CHAPTER 1] 0:30 - 3:00 Foundation concept
|
||||
[RE-HOOK] 3:00 - 3:15 Curiosity gap for next section
|
||||
[CHAPTER 2] 3:15 - 6:00 Complication / deeper layer
|
||||
[PALETTE CLEANSER] 6:00 - 6:15 Visual break, humor, or "let that sink in"
|
||||
[CHAPTER 3] 6:15 - 9:00 Key insight / "aha" moment
|
||||
[PROOF] 9:00 - 10:30 Demonstration / example
|
||||
[CONCLUSION] 10:30 - 11:30 Implications + reframe
|
||||
[OUTRO] 11:30 - 12:00 CTA + end screen
|
||||
```
|
||||
|
||||
### Chapter Length Rules
|
||||
|
||||
| Chapter Content | Ideal Length | Notes |
|
||||
|----------------|-------------|-------|
|
||||
| Simple concept | 2-3 minutes | One idea, one visual set |
|
||||
| Complex concept | 3-4 minutes | Multi-step, needs examples |
|
||||
| Demonstration | 2-3 minutes | Show, don't just tell |
|
||||
| Story / narrative | 3-5 minutes | Needs setup + payoff |
|
||||
|
||||
**Max 5-6 chapters** for a 10-15 minute video. More chapters = too fragmented.
|
||||
|
||||
### YouTube Chapters (Timestamps)
|
||||
|
||||
Add chapter markers in the description:
|
||||
```
|
||||
0:00 Introduction
|
||||
0:30 Why This Matters
|
||||
3:15 The Key Mechanism
|
||||
6:15 The Breakthrough
|
||||
9:00 Real-World Example
|
||||
10:30 What This Means For You
|
||||
```
|
||||
|
||||
Chapters improve navigation and can boost retention by letting viewers skip to relevant sections.
|
||||
|
||||
## Audio Consistency
|
||||
|
||||
### Music Bed Management
|
||||
|
||||
| Rule | Value |
|
||||
|------|-------|
|
||||
| Music presence | Continuous throughout (no silent gaps) |
|
||||
| Ducking during speech | -18 to -20 dB below narration |
|
||||
| Music transitions | 2-3 second crossfade between sections |
|
||||
| Energy matching | Shift music energy at chapter boundaries |
|
||||
| BPM consistency | Stay within ±10 BPM across the video |
|
||||
|
||||
### LUFS Over Long Duration
|
||||
|
||||
- Target: **-14 LUFS integrated** (YouTube standard)
|
||||
- Dynamic range: **6-12 dB** for speech-heavy content
|
||||
- Check LUFS per chapter — variation between chapters should be < 2 LUFS
|
||||
- Use a limiter at **-1.5 dBTP** on the final mix
|
||||
|
||||
### Narration Pacing
|
||||
|
||||
| Section | WPM | Energy |
|
||||
|---------|-----|--------|
|
||||
| Hook | 160-170 | High energy, urgent |
|
||||
| Explanation | 150-160 | Steady, clear |
|
||||
| Key insight | 140-150 | Slower, deliberate |
|
||||
| Silence after reveal | 0 WPM (1-3s pause) | Let it land |
|
||||
| Conclusion | 155-165 | Energized, resolved |
|
||||
|
||||
## Visual Pacing
|
||||
|
||||
### Cut Frequency by Video Phase
|
||||
|
||||
| Phase | Timing | Cut Interval | Notes |
|
||||
|-------|--------|-------------|-------|
|
||||
| Hook | 0:00-0:30 | Every 3-5s | Rapid changes signal momentum |
|
||||
| Early body | 0:30-3:00 | Every 10-15s | High energy, frequent B-roll |
|
||||
| Mid body | 3:00-7:00 | Every 15-25s | Stabilize; fewer cuts, more contextual B-roll |
|
||||
| Late body | 8:00+ | 15-25s calm + burst sequences | Alternate calm with 5-10 quick-cut bursts every 2-3 min |
|
||||
|
||||
### B-Roll Strategy
|
||||
|
||||
- **Individual B-roll clip length:** 5-8 seconds
|
||||
- **B-roll as percentage of total video:** 35-50% for educational content
|
||||
- **Watch time impact:** Strategic B-roll at 35-50% increases watch time by **15-25%**
|
||||
- **Shot absorption time:** Viewers need ~3 seconds; beyond 5 seconds without change, attention fades
|
||||
|
||||
### The "Something Must Happen" Rule
|
||||
|
||||
| Rule | Value |
|
||||
|------|-------|
|
||||
| Visual/audio change | Every 3-5 seconds |
|
||||
| Substantive frame change | Every 20-30 seconds |
|
||||
| Max without any change | 15 seconds (expect drop-off beyond this) |
|
||||
|
||||
## End Screen & Cards
|
||||
|
||||
### End Screen (Last 20 Seconds)
|
||||
|
||||
- YouTube allows end screen elements in the **last 5-20 seconds**
|
||||
- Include: subscribe button, next video recommendation, playlist link
|
||||
- **Do NOT put critical content in the last 20 seconds** — it gets covered
|
||||
- Verbal CTA: "If you found this helpful, check out this next video on..."
|
||||
|
||||
### Info Cards
|
||||
|
||||
- Place at moments when a related topic is mentioned
|
||||
- Max 1 card per 2 minutes — too many feels spammy
|
||||
- Best placement: when you reference a concept covered in another video
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When building long-form content:
|
||||
|
||||
1. **Structure with chapters** — 2-4 minutes each, max 5-6 chapters
|
||||
2. **Complete the hook by 0:30** — follow the storytelling.md Explainer Arc template
|
||||
3. **Re-hook at 2:00-3:00** — this is the retention valley
|
||||
4. **Pattern interrupt every 45-90 seconds** — B-roll, text overlay, visual change
|
||||
5. **Continuous music bed** — use `music_gen` for full-length track, duck 18-20 dB
|
||||
6. **Narrate at 150-160 WPM** — slower than short-form, clearer for learning
|
||||
7. **Check LUFS per chapter** — should be consistent (< 2 LUFS variation)
|
||||
8. **Reserve last 20 seconds** for end screen — no essential content there
|
||||
9. **Add chapter timestamps** — include in publish stage metadata
|
||||
10. **Target 40-60% average view duration** — if retention drops below 30% at any point, that section needs a pattern interrupt
|
||||
|
||||
## Timing Cheat Sheet (12-Minute Video)
|
||||
|
||||
```
|
||||
0:00-0:03 Visual hook (most compelling shot)
|
||||
0:03-0:08 Verbal hook (promise/question)
|
||||
0:08-0:15 Stakes ("here's why this matters")
|
||||
0:15-0:30 Value preview + open loop planted
|
||||
0:30-0:35 Branded intro (5 sec max)
|
||||
0:35-1:45 Body segment 1 (high energy, cuts every 10-15s)
|
||||
1:45-2:00 Pattern interrupt to bridge retention valley
|
||||
2:00-3:00 First major payoff delivered
|
||||
3:00-3:05 Chapter 2 mini-hook + bridging sentence
|
||||
3:00-5:30 Body segment 2 (stabilized pacing, 15-25s cuts)
|
||||
~5:00 Mid-roll CTA (subscribe ask, after earning value)
|
||||
5:30-8:00 Body segment 3 (B-roll heavy, callbacks)
|
||||
7:00-7:15 Burst sequence (5-10 quick cuts to re-engage)
|
||||
8:00-10:00 Body segment 4 (mix calm + energy bursts)
|
||||
9:30 Open loop resolution / major callback payoff
|
||||
10:00-11:20 Final segment + main reveal
|
||||
11:00 Card placement (last 20% of video)
|
||||
11:20-11:40 Outro: tease next content, do NOT say goodbye
|
||||
11:40-12:00 End screen (last 20 seconds), 1-2 elements
|
||||
```
|
||||
@@ -0,0 +1,102 @@
|
||||
# ManimCE Usage for OpenMontage
|
||||
|
||||
> Sources: ManimCE documentation, 3Blue1Brown FAQ/conventions, Theorem of Beethoven tutorials,
|
||||
> existing Layer 3 skill at `.agents/skills/manimce-best-practices/`
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
RENDER QUALITY: -qh (1080p60) for YouTube | -qm (720p30) for drafts
|
||||
BACKGROUND: Dark (#1a1a2e or BLACK)
|
||||
MAX ELEMENTS: 3-4 new visual elements revealed simultaneously
|
||||
PACING: One concept per scene, build incrementally
|
||||
EQUATION WRITE: 1.5-2.0s run_time
|
||||
SHAPE CREATE: 0.8-1.2s run_time
|
||||
WAIT AFTER: 1.0-2.0s (longer for complex equations)
|
||||
2D vs 3D: Default to 2D. 3D only when spatial relationship IS the concept.
|
||||
```
|
||||
|
||||
## Render Settings for OpenMontage
|
||||
|
||||
| Flag | Resolution | FPS | Use Case |
|
||||
|------|-----------|-----|----------|
|
||||
| `-ql` | 480x360 | 15 | Development/testing |
|
||||
| `-qm` | 1280x720 | 30 | Draft review |
|
||||
| `-qh` | 1920x1080 | 60 | Standard YouTube upload |
|
||||
| `-qp` | 2560x1440 | 60 | High-quality export |
|
||||
| `-qk` | 3840x2160 | 60 | 4K archival/premium |
|
||||
|
||||
For OpenMontage's YouTube landscape profile (1920x1080/30fps), render at `-qh` and transcode to 30fps, or set custom config:
|
||||
|
||||
```ini
|
||||
[CLI]
|
||||
pixel_width = 1920
|
||||
pixel_height = 1080
|
||||
frame_rate = 30
|
||||
```
|
||||
|
||||
## Animation Timing
|
||||
|
||||
| Animation Type | `run_time` | Rate Function | Notes |
|
||||
|---------------|-----------|---------------|-------|
|
||||
| Equation write (`Write`) | 1.5-2.0s | `smooth` (default) | Give viewers time to parse LaTeX |
|
||||
| Equation transform | 1.5s | `smooth` | Use `TransformMatchingTex` for derivations |
|
||||
| Shape creation (`Create`) | 0.8-1.2s | `smooth` | `Create()` or `DrawBorderThenFill()` |
|
||||
| Color highlight | 0.5s | `smooth` | Brief attention call |
|
||||
| Camera zoom | 1.5-2.0s | `ease_in_out_cubic` | Smooth entry/exit |
|
||||
| Staggered reveals | `lag_ratio=0.1-0.2` | — | `LaggedStart` for grid/list reveals |
|
||||
| Wait after reveal | 1.0-2.0s | — | Longer for complex equations |
|
||||
| Fast cut / punctuation | 0.3-0.5s | `rush_from` | Between concepts |
|
||||
|
||||
## Scene Composition
|
||||
|
||||
### Pacing Rule (3Blue1Brown Convention)
|
||||
|
||||
- **One concept per scene** — build incrementally
|
||||
- Show the simple version first, then `Transform` it into the complex version
|
||||
- Never reveal more than **3-4 new visual elements** simultaneously
|
||||
- Use `self.wait(1.5)` after every major reveal
|
||||
|
||||
### 2D vs 3D Decision
|
||||
|
||||
**Use 2D** (`Scene` or `MovingCameraScene`) for:
|
||||
- Equation derivations, graph plots, number lines, matrices
|
||||
- 2D vector spaces (even for "high dimensions" — project down)
|
||||
- State diagrams, flowcharts, timelines
|
||||
|
||||
**Use 3D** (`ThreeDScene`) only when:
|
||||
- Visualizing surfaces (`z = f(x,y)`), volumes, or 3D vector fields
|
||||
- The spatial relationship IS the concept (cross products, surface normals)
|
||||
- You need camera orbit to reveal hidden structure
|
||||
|
||||
**Performance:** 3D uses CPU-only Cairo rendering — 5-10x slower than 2D.
|
||||
|
||||
## Color Usage
|
||||
|
||||
| Semantic Role | Color | Manim Constant |
|
||||
|--------------|-------|----------------|
|
||||
| Variable being solved | Yellow | `YELLOW` |
|
||||
| Matrix / operator | Red | `RED` |
|
||||
| Eigenvector / result | Teal | `TEAL` |
|
||||
| Known constant | Blue | `BLUE_C` |
|
||||
| Annotation / label | Green | `GREEN` |
|
||||
| De-emphasis / background | Grey 50% | `GREY`, `opacity=0.5` |
|
||||
| Error / wrong path | Dark red | `RED_E` |
|
||||
|
||||
**Accessibility:** Avoid red-green only distinctions. Use brightness variation (`_A` through `_E` shades) alongside hue changes.
|
||||
|
||||
**Background:** Always use dark backgrounds (`BLACK` or `#1a1a2e`) for video output.
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When using the `math_animate` tool:
|
||||
|
||||
1. **Render at `-qh`** (1080p60) for final output, `-qm` for drafts
|
||||
2. **One concept per scene** — break complex proofs into multiple Manim scenes
|
||||
3. **Use timing table above** — don't rush equations (1.5-2.0s for writes)
|
||||
4. **Wait after reveals** — `self.wait(1.5)` minimum after key insights
|
||||
5. **Dark background** — set `background_color=BLACK` in config
|
||||
6. **Use color semantically** — yellow for unknowns, blue for knowns, red for operators
|
||||
7. **Default to 2D** — only use `ThreeDScene` when 3D is essential to understanding
|
||||
8. **Stagger complex reveals** — `LaggedStart` with `lag_ratio=0.15` for lists/grids
|
||||
9. **Sync to narration** — the scene's total duration should match the narration segment timing from the script
|
||||
@@ -0,0 +1,135 @@
|
||||
# Music Generation Usage for OpenMontage
|
||||
|
||||
> Sources: ElevenLabs Music API documentation, ElevenLabs best practices guide, Artlist BPM
|
||||
> guide, existing Layer 3 skills at `.agents/skills/music/` and `.agents/skills/elevenlabs/`
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
API MODEL: music_v1
|
||||
MIN DURATION: 3,000ms (3s)
|
||||
MAX DURATION: 600,000ms (10 min)
|
||||
INSTRUMENTAL: Always set force_instrumental=true for video background
|
||||
COST: ~$0.05 per 30 seconds
|
||||
KEY RULE: Music must be 18-20 dB below narration (see sound-design.md)
|
||||
```
|
||||
|
||||
## BPM Selection by Video Type
|
||||
|
||||
| Video Type | BPM Range | Prompt Fragment |
|
||||
|-----------|-----------|-----------------|
|
||||
| Educational explainer | 80-100 | "gentle ambient electronic, 90 BPM" |
|
||||
| Corporate / tech | 100-120 | "upbeat corporate pop, 110 BPM, positive" |
|
||||
| Epic / dramatic reveal | 60-80 | "cinematic orchestral, 70 BPM, building tension" |
|
||||
| Fast-paced montage | 120-140 | "energetic electronic, 130 BPM, driving beat" |
|
||||
| Meditation / calm | 50-70 | "ambient drone, 60 BPM, peaceful" |
|
||||
| Comedy / lighthearted | 100-130 | "playful ukulele pop, 120 BPM, whimsical" |
|
||||
| Sad / reflective | 60-80 | "melancholic piano, 65 BPM, minor key" |
|
||||
| Action / hype | 140-170 | "high-intensity drum and bass, 160 BPM" |
|
||||
|
||||
## Key and Mood Mapping
|
||||
|
||||
| Mood | Key | Musical Characteristics |
|
||||
|------|-----|----------------------|
|
||||
| Happy / upbeat | C major, G major | Bright, resolved, energetic |
|
||||
| Serious / professional | D minor, A minor | Grounded, authoritative |
|
||||
| Mysterious / curious | E minor, B minor | Tension, anticipation |
|
||||
| Triumphant / inspiring | D major, Bb major | Expansive, climactic |
|
||||
| Melancholic / thoughtful | F minor, C minor | Reflective, emotional |
|
||||
| Neutral / ambient | C major, Am (no strong key) | Unobtrusive, background |
|
||||
|
||||
## Prompt Engineering
|
||||
|
||||
### Structure
|
||||
|
||||
```
|
||||
[GENRE/STYLE], [BPM], [KEY/MOOD], [INSTRUMENTS], [ENERGY LEVEL], [PURPOSE]
|
||||
```
|
||||
|
||||
### Examples
|
||||
|
||||
**Educational explainer:**
|
||||
```
|
||||
Gentle lo-fi ambient electronic, 90 BPM, C major, soft synth pads and light
|
||||
percussion, calm and steady energy, background music for narration
|
||||
```
|
||||
|
||||
**Corporate product demo:**
|
||||
```
|
||||
Modern upbeat corporate pop, 110 BPM, G major, acoustic guitar and light drums,
|
||||
positive energy building gradually, underscore for product walkthrough
|
||||
```
|
||||
|
||||
**Technical deep-dive:**
|
||||
```
|
||||
Minimal ambient electronic, 80 BPM, A minor, soft Rhodes piano and subtle
|
||||
bass, contemplative and focused, background music for technical explanation
|
||||
```
|
||||
|
||||
### Key Prompting Rules
|
||||
|
||||
1. **Always include "background" or "underscore"** — tells the model to stay dynamically even
|
||||
2. **Always use `force_instrumental=true`** — lyrics compete with narration
|
||||
3. **Specify BPM explicitly** — don't rely on genre to set tempo
|
||||
4. **Avoid "bright hi-hats" or "prominent vocals"** — high-frequency busy elements compete with speech in the 2-4 kHz intelligibility band
|
||||
5. **Include energy direction** — "steady energy" for explainers, "building gradually" for reveals
|
||||
|
||||
## Duration Matching
|
||||
|
||||
### Exact Duration
|
||||
|
||||
```python
|
||||
result = music_gen.execute({
|
||||
"prompt": "Gentle ambient, 90 BPM, background underscore",
|
||||
"duration_seconds": 150, # Match video length
|
||||
"output_path": "assets/music/background.mp3"
|
||||
})
|
||||
```
|
||||
|
||||
### Section-Mapped (Advanced)
|
||||
|
||||
For videos with distinct acts, generate sections separately:
|
||||
|
||||
| Video Section | Duration | Music Style |
|
||||
|--------------|----------|-------------|
|
||||
| Intro / hook | 8-10s | Soft, building |
|
||||
| Main explanation | 90-120s | Steady, neutral |
|
||||
| Key reveal | 20-30s | Intensified, fuller |
|
||||
| Outro | 10-15s | Fading, gentle |
|
||||
|
||||
Generate each as a separate track and crossfade in the `audio_mixer`.
|
||||
|
||||
## Looping for Long Videos
|
||||
|
||||
For videos longer than the generated track:
|
||||
|
||||
1. Generate a track 30-60% of video length
|
||||
2. Use FFmpeg to create a seamless loop:
|
||||
```bash
|
||||
ffmpeg -stream_loop 2 -i music.mp3 -c copy music_looped.mp3
|
||||
```
|
||||
3. Add a 2-3 second crossfade at loop points in `audio_mixer`
|
||||
|
||||
**Better approach:** Generate at the exact video duration. ElevenLabs supports up to 10 minutes per generation.
|
||||
|
||||
## Stem Isolation
|
||||
|
||||
For cleaner ducking control, generate isolated stems:
|
||||
|
||||
- `"solo electric guitar in E minor, 90 BPM"` — guitar-only track
|
||||
- `"soft ambient pad in C major, 80 BPM"` — synth pad only
|
||||
- Layer stems in FFmpeg during composition for precise ducking control
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When using the `music_gen` tool:
|
||||
|
||||
1. **Match BPM to content type** using the table above — don't default to a generic prompt
|
||||
2. **Always set `force_instrumental=true`** — no lyrics under narration
|
||||
3. **Include "background" or "underscore"** in every prompt
|
||||
4. **Set duration to match video length** — avoid looping when possible
|
||||
5. **Budget check** — at $0.05/30s, a 3-minute video costs ~$0.30 for music
|
||||
6. **Duck music 18-20 dB below narration** — see `skills/creative/sound-design.md` for ducking rules
|
||||
7. **Cut 2-4 kHz on the music bed** in `audio_mixer` to clear the speech intelligibility band
|
||||
8. **Test on phone speakers** — if narration disappears behind music, duck more aggressively
|
||||
9. **One track per video** — avoid switching music styles mid-video unless there's a clear narrative shift
|
||||
@@ -0,0 +1,88 @@
|
||||
# HunyuanVideo 1.5 — Prompting Guide
|
||||
|
||||
> Source: [Tencent Prompt Handbook](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5/blob/main/assets/HunyuanVideo_1_5_Prompt_Handbook_EN.md)
|
||||
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
|
||||
|
||||
## HunyuanVideo Prompt Formula
|
||||
|
||||
### Text-to-Video
|
||||
```
|
||||
Subject + Motion + Scene + [Shot Type] + [Camera Movement] + [Lighting] + [Style] + [Atmosphere]
|
||||
```
|
||||
|
||||
### Image-to-Video
|
||||
```
|
||||
Subject Motion Dynamics + Scene Motion Dynamics + [Camera Movement]
|
||||
```
|
||||
|
||||
For I2V, focus on describing MOTION, not appearance (the image provides appearance).
|
||||
|
||||
## HunyuanVideo-Specific Strengths
|
||||
|
||||
### Lighting as Atmosphere
|
||||
Tencent emphasizes: **"Light is the soul of atmosphere."**
|
||||
|
||||
Describe lighting with multiple dimensions:
|
||||
- **Style**: soft, hard, neon, ambient
|
||||
- **Direction**: side-lit, backlit, overhead, underlighting
|
||||
- **Quality**: harsh spotlight, diffuse glow
|
||||
- **Shadows**: long dramatic shadows, soft shadow edges
|
||||
- **Color temperature**: golden hour warmth, cool daylight blue
|
||||
- **Reflections**: wet surface reflections, metallic glints
|
||||
|
||||
### Camera Movement Library
|
||||
|
||||
| Movement | Type | HunyuanVideo Prompt |
|
||||
|----------|------|-------------------|
|
||||
| Crane / Pedestal | Vertical | "camera rises vertically" |
|
||||
| Truck / Tracking | Horizontal | "camera tracks left alongside subject" |
|
||||
| Dolly In | Push | "camera pushes forward toward subject" |
|
||||
| Dolly Out | Pull | "camera pulls back from subject" |
|
||||
| Pan | Rotation | "camera pans right across the scene" |
|
||||
| Orbit | Circular | "camera orbits around subject" |
|
||||
| Follow | Lock-on | "camera follows subject from behind" |
|
||||
| Static | Fixed | "static camera, no movement" |
|
||||
|
||||
### Style Keywords
|
||||
|
||||
**Photorealistic / Cinematic**:
|
||||
- Film noir, hard sci-fi, cinematic photography
|
||||
- Period drama, war documentary, nature documentary
|
||||
|
||||
**Animation / Illustration**:
|
||||
- 2D animation, Japanese anime
|
||||
- Watercolor painting, Chinese ink wash
|
||||
- Low-poly 3D, pixel art
|
||||
|
||||
## I2V Best Practice
|
||||
|
||||
When using image-to-video, the input image defines appearance. Your prompt should ONLY describe:
|
||||
1. How the subject moves
|
||||
2. How the environment changes
|
||||
3. Camera motion
|
||||
|
||||
**Good I2V prompt**: "The woman's hair blows in the wind as she turns to face the camera. Leaves scatter across the path. Camera slowly dollies in."
|
||||
|
||||
**Bad I2V prompt**: "A beautiful woman in a red dress standing in a forest" — this repeats what the image already shows.
|
||||
|
||||
## Example (T2V)
|
||||
|
||||
```
|
||||
A young woman in a flowing white dress walks barefoot along
|
||||
a deserted beach at golden hour. She trails her hand through
|
||||
the shallow surf, leaving ripples. Her hair catches the warm
|
||||
side-light from the setting sun. Medium tracking shot, camera
|
||||
follows alongside at knee height. Soft golden lighting with
|
||||
long shadows stretching toward the camera. Cinematic
|
||||
photography style, shallow depth of field. Peaceful,
|
||||
contemplative atmosphere.
|
||||
```
|
||||
|
||||
## Example (I2V)
|
||||
|
||||
```
|
||||
The cat stretches lazily, then leaps from the windowsill
|
||||
to the floor. Dust motes scatter in the shaft of light.
|
||||
Camera remains static, slight rack focus from window to
|
||||
landing spot.
|
||||
```
|
||||
@@ -0,0 +1,77 @@
|
||||
# LTX-2 — Prompting Guide
|
||||
|
||||
> Source: [LTX Official Prompting Guide](https://docs.ltx.video/api-documentation/prompting-guide)
|
||||
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
|
||||
|
||||
## LTX-Specific 6-Element Structure
|
||||
|
||||
LTX-2 uses a clean, focused prompt structure:
|
||||
|
||||
1. **Establish the shot** — cinematography terms matching your genre
|
||||
2. **Set the scene** — lighting, color palette, textures, atmosphere
|
||||
3. **Describe the action** — natural sequence flowing from beginning to end
|
||||
4. **Define the character(s)** — physical cues (age, hair, clothes), not abstract labels
|
||||
5. **Camera movement(s)** — specify how and when; describe what appears AFTER the movement
|
||||
6. **Describe the audio** — ambient sound, music, speech, or singing
|
||||
|
||||
## LTX-Specific Tips
|
||||
|
||||
### Post-Movement Description
|
||||
LTX renders camera movements more accurately when you describe the result:
|
||||
- Instead of: "Camera pans left"
|
||||
- Write: "Camera pans left to reveal a bustling market square"
|
||||
|
||||
### Audio Prompting (Unique to LTX-2)
|
||||
LTX-2 generates synchronized audio. Use specific descriptors:
|
||||
|
||||
| Category | Examples |
|
||||
|----------|---------|
|
||||
| **Ambient** | "coffeeshop noise", "wind and rain", "forest with birdsong" |
|
||||
| **Voice style** | "energetic announcer", "resonant voice with gravitas", "childlike curiosity" |
|
||||
| **Volume** | "whisper", "mutter", "shout", "scream" |
|
||||
| **Music** | "soft acoustic guitar", "electronic beat building" |
|
||||
|
||||
Dialogue goes in quotes: `The narrator says: "Welcome to the future."`
|
||||
Specify language/accent: `speaks in British English with a warm tone`
|
||||
|
||||
### Style Categories
|
||||
LTX organizes styles into three families:
|
||||
|
||||
**Animation**: stop-motion, 2D animation, 3D animation, claymation, hand-drawn
|
||||
**Stylized**: comic book, cyberpunk, 8-bit pixel, surreal, minimalist, painterly
|
||||
**Cinematic**: period drama, film noir, fantasy, thriller, documentary, arthouse
|
||||
|
||||
## What to Avoid (LTX-Specific)
|
||||
|
||||
| Avoid | Reason |
|
||||
|-------|--------|
|
||||
| Internal emotional states ("sad", "confused") | Use visual cues: tears, slumped posture, furrowed brow |
|
||||
| Readable text and logos | Not reliably rendered |
|
||||
| Complex physics (explosions, splashing) | Causes artifacts; simple motion is fine |
|
||||
| Overloaded scenes | Many characters/actions reduces coherence |
|
||||
| Conflicting lighting descriptions | Pick one setup, commit to it |
|
||||
| Starting complex | Build up: simple prompt first, add layers |
|
||||
|
||||
## LTX Technical Notes
|
||||
|
||||
- **Duration**: ~5-8 seconds per generation
|
||||
- **Audio**: Generated automatically; describe what you want to hear
|
||||
- **~30% of outputs have artifacts** — re-run with a different seed
|
||||
- **Cannot render readable text** — don't include signs or titles
|
||||
- **Frame count must satisfy** `(n-1) % 8 == 0`: valid counts are 25, 49, 73, 97, 121, 161, 193
|
||||
|
||||
## Example
|
||||
|
||||
```
|
||||
A wide establishing shot captures a misty morning harbor.
|
||||
Weathered fishing boats bob gently, their paint peeling in
|
||||
patches of red and blue. A grey-haired fisherman in a dark
|
||||
wool peacoat steps onto the dock, carrying a heavy net over
|
||||
one shoulder. He pauses, looks out at the fog bank, then
|
||||
walks toward the nearest boat with steady, deliberate steps.
|
||||
The camera tracks alongside him at waist height, slowly
|
||||
pushing in as he reaches the boat and tosses the net aboard.
|
||||
Soft overcast light with a warm break in the clouds near
|
||||
the horizon. Ambient sound of water lapping, rope creaking,
|
||||
and distant foghorn.
|
||||
```
|
||||
@@ -0,0 +1,91 @@
|
||||
# Sora 2 — Prompting Guide
|
||||
|
||||
> Source: [OpenAI Sora 2 Cookbook](https://developers.openai.com/cookbook/examples/sora/sora2_prompting_guide)
|
||||
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
|
||||
|
||||
## Sora-Specific Prompt Template
|
||||
|
||||
Sora responds best to a structured format with prose + cinematography block + action beats:
|
||||
|
||||
```
|
||||
[Prose scene description — characters, costumes, scenery, weather, details.
|
||||
Be as descriptive as possible to match your vision.]
|
||||
|
||||
Cinematography:
|
||||
Camera shot: [framing and angle]
|
||||
Lens: [focal length, type]
|
||||
Lighting: [key, fill, rim, practical sources with color temp]
|
||||
Mood: [overall tone]
|
||||
|
||||
Actions:
|
||||
- [Beat 1: specific gesture or movement]
|
||||
- [Beat 2: another distinct beat]
|
||||
- [Beat 3: reaction or dialogue]
|
||||
|
||||
Dialogue:
|
||||
[Short natural lines, kept brief for clip length]
|
||||
```
|
||||
|
||||
## Advanced Optional Fields
|
||||
|
||||
Sora uniquely responds to these production-level details that most models ignore:
|
||||
|
||||
| Field | Example |
|
||||
|-------|---------|
|
||||
| **Lens spec** | "40mm spherical", "85mm", "Anamorphic 2.0x" |
|
||||
| **Filtration** | "Black Pro-Mist 1/4", "slight CPL rotation" |
|
||||
| **Grade / palette** | "Warm Kodak-inspired grade", "teal-and-orange LUT" |
|
||||
| **Film stock emulation** | "16mm black-and-white", "35mm photochemical contrast" |
|
||||
| **Diegetic sound** | "faint rail screech, rain patters window, clock ticks" |
|
||||
| **Wardrobe** | "navy coat, sleeves rolled, suspenders loose" |
|
||||
| **Finishing** | "fine-grain overlay, mild halation, gate weave, soft vignette" |
|
||||
| **Shutter** | "180° shutter angle" |
|
||||
|
||||
## What Sora Does Differently
|
||||
|
||||
- **Prose-first**: Write a rich paragraph, then add technical blocks. Don't lead with camera specs.
|
||||
- **Character references**: Can lock onto up to 2 uploaded character IDs via API.
|
||||
- **Dialogue sync**: Short lines work. Complex multi-character dialogue does not.
|
||||
- **Edit commands**: "Same shot, switch to 85mm" or "Same lighting, new palette: teal, sand, rust" — Sora supports iterative refinement on existing generations.
|
||||
- **Creative freedom**: Shorter prompts → more creative latitude. Longer → more control.
|
||||
|
||||
## Color Palette Technique
|
||||
|
||||
Name 3-5 anchor colors instead of vague "warm tones":
|
||||
- "Amber, cream, walnut brown" (vintage warmth)
|
||||
- "Teal, sand, rust" (coastal desert)
|
||||
- "Cool blues with warm tungsten accents" (noir)
|
||||
|
||||
## Sora API Parameters (cannot be set in prompt)
|
||||
|
||||
- `model`: `sora-2` or `sora-2-pro`
|
||||
- `size`: 720x1280, 1280x720, 1080x1920, 1920x1080, 1024x1792, 1792x1024
|
||||
- `seconds`: 4, 8, 12, 16, 20
|
||||
|
||||
## Example
|
||||
|
||||
```
|
||||
Style: Hand-painted 2D/3D hybrid animation with soft brush textures,
|
||||
warm tungsten lighting, tactile stop-motion feel. Subtle watercolor wash;
|
||||
warm-cool balance; filmic motion blur.
|
||||
|
||||
Inside a cluttered workshop, shelves overflow with gears and yellowing
|
||||
blueprints. Small round robot sits on wooden bench, dented body patched
|
||||
with mismatched plates. Large glowing blue eyes flicker as it fiddles
|
||||
with a humming light bulb.
|
||||
|
||||
Cinematography:
|
||||
Camera: medium close-up, slow push-in with gentle parallax from hanging tools
|
||||
Lens: 35mm virtual; shallow depth of field
|
||||
Lighting: warm key from overhead practical; cool spill from window
|
||||
Mood: gentle, whimsical, touch of suspense
|
||||
|
||||
Actions:
|
||||
- Robot taps bulb; sparks crackle
|
||||
- Flinches, dropping bulb, eyes widening
|
||||
- Bulb tumbles in slow motion; catches it just in time
|
||||
- Puff of steam escapes chest — relief and pride
|
||||
|
||||
Background Sound:
|
||||
Rain, ticking clock, soft mechanical hum, faint bulb sizzle
|
||||
```
|
||||
@@ -0,0 +1,73 @@
|
||||
# VEO 3.1 / VEO 3 — Prompting Guide
|
||||
|
||||
> Source: [Vertex AI Video Gen Prompt Guide](https://cloud.google.com/vertex-ai/generative-ai/docs/video/video-gen-prompt-guide)
|
||||
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
|
||||
|
||||
## VEO-Specific 14-Component Structure
|
||||
|
||||
VEO responds to the most comprehensive prompt structure of any model:
|
||||
|
||||
1. **Subject** — who/what the action revolves around
|
||||
2. **Action** — movements, interactions, expressions
|
||||
3. **Scene / Context** — location, time, weather, period
|
||||
4. **Camera Angles** — shot type and perspective
|
||||
5. **Camera Movements** — dynamic motion
|
||||
6. **Lens / Optical Effects** — how the camera "sees"
|
||||
7. **Lighting** — source, direction, quality
|
||||
8. **Tone / Mood** — emotional register
|
||||
9. **Artistic Style** — photorealistic, cinematic, animation, art movement
|
||||
10. **Ambiance** — color palettes, atmospheric effects, textures
|
||||
11. **Temporal Elements** — pacing, time flow, rhythm
|
||||
12. **Audio** — sound effects, ambient, dialogue (VEO 3 generates dialogue)
|
||||
13. **Cinematic Terms** — editing techniques (match cut, montage, split diopter)
|
||||
14. **Negative Prompt** — what to exclude
|
||||
|
||||
## VEO-Specific Strengths
|
||||
|
||||
- **Dialogue generation**: VEO 3 natively generates character speech. Write dialogue naturally.
|
||||
- **Audio integration**: Ambient sound, music, and voice are generated together with video.
|
||||
- **Negative prompts**: Explicitly supported — "no text overlays, no watermarks, no lens flare"
|
||||
- **Editing vocabulary**: Understands "match cut", "jump cut", "montage", "split diopter" as prompt terms.
|
||||
|
||||
## VEO Lens Effects (Unique)
|
||||
|
||||
VEO specifically responds to optical effects most models ignore:
|
||||
|
||||
| Effect | Prompt Language |
|
||||
|--------|----------------|
|
||||
| **Rack focus** | "rack focus from foreground flower to background figure" |
|
||||
| **Dolly zoom (vertigo)** | "vertigo effect as character realizes the truth" |
|
||||
| **Fisheye** | "fisheye lens distortion, skatepark POV" |
|
||||
| **Lens flare** | "anamorphic lens flare from setting sun" |
|
||||
|
||||
## VEO Art Movement References
|
||||
|
||||
VEO responds well to specific art movements as style anchors:
|
||||
- "Van Gogh-inspired swirling sky"
|
||||
- "Surrealist Dalí-esque melting landscape"
|
||||
- "Art Deco geometric patterns in the architecture"
|
||||
- "Bauhaus clean lines and primary colors"
|
||||
- "Gritty graphic novel illustration style"
|
||||
- "Chinese ink wash painting animation"
|
||||
|
||||
## Subtitle Prevention
|
||||
|
||||
VEO may add subtitles by default for dialogue. To prevent:
|
||||
- Add to negative prompt: "no subtitles, no captions, no text overlays"
|
||||
|
||||
## Example
|
||||
|
||||
```
|
||||
Subject: A lone astronaut in a weathered white spacesuit
|
||||
Action: Slowly turns to face the camera, visor reflecting a dying star
|
||||
Scene: Surface of a barren moon, cracked grey terrain, massive ringed
|
||||
planet filling the horizon
|
||||
Camera: Low-angle medium shot, slow arc around subject
|
||||
Lens: Wide-angle, deep focus keeping both astronaut and planet sharp
|
||||
Lighting: Harsh rim light from the star behind, cool blue fill from
|
||||
planet reflection, no atmosphere diffusion
|
||||
Mood: Awe, isolation, quiet grandeur
|
||||
Style: Photorealistic sci-fi cinematography, IMAX-scale
|
||||
Audio: Breathing inside helmet, faint radio static, low rumble
|
||||
Negative: No text, no HUD overlay, no lens flare
|
||||
```
|
||||
@@ -0,0 +1,108 @@
|
||||
# Scene Detection Usage for OpenMontage
|
||||
|
||||
> Sources: PySceneDetect documentation, FFmpeg scenedetect filter docs, PySceneDetect
|
||||
> GitHub issues #187 (threshold tuning) and #226 (adaptive discussion)
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
DEFAULT METHOD: content (ContentDetector) — works for most content
|
||||
DEFAULT THRESH: 27.0 (range 0-255)
|
||||
MIN SCENE LEN: 1.0s default, 2.0-3.0s for educational video
|
||||
TUNING: Generate stats CSV first, inspect content_val column
|
||||
HARD CUTS: Use content detector
|
||||
FADE TO BLACK: Use threshold detector
|
||||
MIXED CONTENT: Use adaptive detector
|
||||
```
|
||||
|
||||
## Algorithm Selection
|
||||
|
||||
| Method | Default Threshold | Best For | How It Works |
|
||||
|--------|------------------|----------|-------------|
|
||||
| `content` | 27.0 | Hard cuts between shots | HSV color difference between adjacent frames (0-255) |
|
||||
| `threshold` | 12.0 | Fades to/from black | Average pixel intensity; detects transitions through black |
|
||||
| `adaptive` | 3.0 | Mixed content with camera motion | Rolling average of frame differences; adapts to local pace |
|
||||
|
||||
## Threshold Tuning Guide
|
||||
|
||||
### ContentDetector (Default, Start Here)
|
||||
|
||||
| Symptom | Action | New Threshold |
|
||||
|---------|--------|---------------|
|
||||
| Too many false cuts | Raise threshold | 35-45 |
|
||||
| Missing real cuts | Lower threshold | 20-22 |
|
||||
| Fast-paced content (music videos, action) | Raise | 35-40 |
|
||||
| Slow/static content (talking heads, presentations) | Lower | 20-25 |
|
||||
| Animated content (Manim, motion graphics) | Raise | 30-35 |
|
||||
|
||||
### AdaptiveDetector
|
||||
|
||||
- Multiplier on rolling average (default 3.0)
|
||||
- Better than ContentDetector when there's fast camera motion causing false positives
|
||||
- Good default for OpenMontage explainers where Manim segments are static but live-action may have motion
|
||||
|
||||
### ThresholdDetector
|
||||
|
||||
- Only for videos with deliberate fade-to-black transitions
|
||||
- Most AI-generated video does NOT use fades — prefer `content` or `adaptive`
|
||||
|
||||
## Tuning Workflow
|
||||
|
||||
1. **Generate stats file first:**
|
||||
```bash
|
||||
scenedetect -i video.mp4 --stats stats.csv detect-content
|
||||
```
|
||||
|
||||
2. **Inspect `stats.csv`** — look at the `content_val` column. Peaks = scene changes.
|
||||
|
||||
3. **Set threshold** just below the smallest real peak.
|
||||
|
||||
4. **Set `min_scene_length`** to suppress micro-scenes:
|
||||
- Educational video: 2.0-3.0s minimum
|
||||
- Fast-paced content: 0.5-1.0s
|
||||
- Default: 1.0s
|
||||
|
||||
### Component Weights (Advanced)
|
||||
|
||||
ContentDetector score = weighted sum of HSV + edge differences:
|
||||
|
||||
```
|
||||
weights = (delta_hue, delta_sat, delta_lum, delta_edges)
|
||||
Default: (1.0, 1.0, 1.0, 0.0)
|
||||
```
|
||||
|
||||
For animated content with color transitions but few actual cuts:
|
||||
```
|
||||
weights=(1.0, 0.5, 1.0, 0.2), threshold=32
|
||||
```
|
||||
|
||||
## Post-Processing Detected Scenes
|
||||
|
||||
After detection, clean up the scene list:
|
||||
|
||||
1. **Merge too-short segments** — any scene under `min_scene_length` should be merged with the adjacent scene
|
||||
2. **Validate boundaries** — check that scene boundaries align with narration pauses (for explainers)
|
||||
3. **Label scenes** — map detected scenes to script sections for the edit stage
|
||||
|
||||
## Content-Type Presets
|
||||
|
||||
| Content Type | Method | Threshold | Min Scene Length |
|
||||
|-------------|--------|-----------|-----------------|
|
||||
| Talking head (single camera) | content | 22 | 3.0s |
|
||||
| Talking head (multi-camera) | content | 27 | 1.0s |
|
||||
| Screen recording | content | 30 | 2.0s |
|
||||
| Animated explainer | adaptive | 3.0 | 2.0s |
|
||||
| Fast-paced montage | content | 40 | 0.5s |
|
||||
| Documentary with fades | threshold | 12 | 2.0s |
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When using the `scene_detect` tool:
|
||||
|
||||
1. **Start with `content` method, threshold 27** — it works for most content
|
||||
2. **For talking-head pipeline**, lower threshold to 22 and set min_scene_length to 3.0s
|
||||
3. **For animated-explainer pipeline**, use `adaptive` with default threshold 3.0
|
||||
4. **Always generate stats CSV first** when tuning — don't guess thresholds
|
||||
5. **Set min_scene_length to 2.0s** for educational content to avoid micro-scenes
|
||||
6. **Use detected scenes to inform the edit stage** — map scenes to script sections
|
||||
7. **For AI-generated video clips**, use `content` not `threshold` — AI video rarely uses fade-to-black
|
||||
@@ -0,0 +1,123 @@
|
||||
# Screen Recording Pipeline
|
||||
|
||||
> Sources: OBS Studio documentation, Loom production guidelines, Fireship production
|
||||
> methodology, Kevin Powell CSS tutorial techniques, Theo Browne dev content guides
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
RESOLUTION: 1920x1080 at 2x display (record at 3840x2160, deliver at 1080p)
|
||||
FRAME RATE: 60fps for UI/scrolling, 30fps for static code
|
||||
CURSOR: Enlarged (1.5-2x), highlighted with ring or glow
|
||||
ZOOM: 1.5-2x for code focus, 0.8s ease-in-out transition
|
||||
SPEED RAMP: 1.5x for navigation, 2x for repetitive actions, 1.0x for key moments
|
||||
DEAD AIR: Remove pauses > 1.5 seconds
|
||||
FONT SIZE (IDE): 18-22px minimum for readability at 1080p delivery
|
||||
```
|
||||
|
||||
## Recording Settings
|
||||
|
||||
### Resolution Strategy
|
||||
|
||||
| Approach | Record At | Deliver At | Why |
|
||||
|----------|----------|-----------|-----|
|
||||
| **Recommended** | 3840x2160 (4K) | 1920x1080 | Enables 2x zoom into code without quality loss |
|
||||
| Budget | 1920x1080 | 1920x1080 | Direct capture, limited zoom headroom |
|
||||
| Vertical | 2160x3840 | 1080x1920 | Short-form screen recording |
|
||||
|
||||
### Frame Rate
|
||||
|
||||
| Content Type | FPS | Why |
|
||||
|-------------|-----|-----|
|
||||
| Code editing (mostly static) | 30 | Smaller file size, no visible difference |
|
||||
| UI interaction, scrolling | 60 | Smooth scrolling and cursor movement |
|
||||
| Animation/demo with motion | 60 | Motion clarity |
|
||||
| Terminal output | 30 | Text updates don't need 60fps |
|
||||
|
||||
### IDE/Editor Setup
|
||||
|
||||
- **Font size:** 18-22px minimum (must be readable at 1080p delivery)
|
||||
- **Theme:** Dark theme preferred (easier on eyes, looks better in video)
|
||||
- **Line numbers:** ON (helps viewers follow along)
|
||||
- **Minimap:** OFF (wastes screen space, distracting)
|
||||
- **Sidebar:** Collapsed unless showing file structure is the point
|
||||
- **Status bar:** Consider hiding (clutters bottom of frame)
|
||||
- **Zoom level:** 150-175% for readability
|
||||
|
||||
## Cursor Management
|
||||
|
||||
### Visibility
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| Cursor size | 1.5-2x default system size |
|
||||
| Highlight | Yellow or white ring/glow (50px radius) |
|
||||
| Click indicator | Brief flash or ripple on click |
|
||||
| Smoothing | Light smoothing to reduce jitter |
|
||||
|
||||
### Cursor Behavior
|
||||
|
||||
- **Move deliberately** — no random wandering
|
||||
- **Pause on target** for 0.5s before clicking
|
||||
- **Avoid circling** — don't circle the cursor around what you're talking about
|
||||
- **Hide cursor** when it's not needed (during code explanation)
|
||||
|
||||
## Zoom and Pan
|
||||
|
||||
### Zoom Levels
|
||||
|
||||
| Context | Zoom | Duration of Transition |
|
||||
|---------|------|----------------------|
|
||||
| Full screen overview | 1.0x (100%) | — |
|
||||
| Code focus | 1.5-2.0x | 0.8s ease-in-out |
|
||||
| Terminal focus | 1.5x | 0.6s ease-in-out |
|
||||
| UI element highlight | 2.0-2.5x | 0.8s ease-in-out |
|
||||
| Return to overview | 1.0x | 0.6s ease-in-out |
|
||||
|
||||
### Pan Rules
|
||||
|
||||
- Pan to follow the active area — don't make viewers search
|
||||
- Smooth pan (ease-in-out), not instant jump
|
||||
- Hold position for at least **3 seconds** before next pan
|
||||
- Announce what you're zooming into: "Let's look at this function..."
|
||||
|
||||
## Post-Processing
|
||||
|
||||
### Speed Ramping
|
||||
|
||||
| Action | Speed | Notes |
|
||||
|--------|-------|-------|
|
||||
| Typing boilerplate | 2-3x | Viewers don't need to watch you type imports |
|
||||
| File navigation | 1.5-2x | Opening files, switching tabs |
|
||||
| Package install / build | 2-4x or cut | Show start + end, skip the wait |
|
||||
| Key code writing | 1.0x | Important moments at real speed |
|
||||
| Debugging / thinking | 1.0x with cuts | Remove dead pauses, keep the reasoning |
|
||||
|
||||
### Dead Air Removal
|
||||
|
||||
- Remove **all pauses > 1.5 seconds** unless deliberate
|
||||
- Remove "um", "uh", typing mistakes and backspaces (when possible)
|
||||
- Jump cuts are acceptable and expected in screen recording content
|
||||
- Add a subtle **zoom shift** (1.0x → 1.02x) at each jump cut to mask the edit
|
||||
|
||||
### Audio Enhancement
|
||||
|
||||
- Apply `clean_speech` preset from `audio_enhance`
|
||||
- HPF at 80Hz to remove keyboard/desk rumble
|
||||
- Compress at 3:1 to even out speaking volume
|
||||
- Target -16 LUFS for screen recording content (slightly quieter than -14, more comfortable for long viewing)
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When processing screen recordings in the talking-head pipeline:
|
||||
|
||||
1. **Record at 4K** if possible — enables quality zoom in post
|
||||
2. **Set IDE font to 20px+** before recording
|
||||
3. **Use `scene_detect`** with threshold 30, min_scene_length 2.0s to find natural segments
|
||||
4. **Apply zoom/pan** in compose stage — 1.5-2x on code, 0.8s transitions
|
||||
5. **Speed ramp navigation** to 1.5-2x, keep key moments at 1.0x
|
||||
6. **Remove dead air** > 1.5s with `video_trimmer`
|
||||
7. **Add cursor highlight** in post if not captured in recording
|
||||
8. **Target -16 LUFS** (slightly below YouTube standard for comfortable viewing)
|
||||
9. **Subtitles recommended** — use `subtitle_gen` for accessibility
|
||||
10. **Dark theme** looks best in video — recommend to users before recording
|
||||
@@ -0,0 +1,205 @@
|
||||
# Short-Form Video Pipeline (TikTok / Reels / Shorts)
|
||||
|
||||
> Sources: TikTok Creator Portal, Instagram for Business blog, YouTube Shorts documentation,
|
||||
> Hootsuite Social Trends Report 2025, OpusClip retention data 2025, Shortimize 35B Shorts
|
||||
> analysis, PostPlanify safe zones 2026, Kreatli platform guides, TTS Vibes hook statistics
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
ASPECT RATIO: 9:16 vertical (1080x1920)
|
||||
SAFE ZONE: 900x1400px centered (universal cross-platform)
|
||||
DURATION: 15s (highest completion) | 30s (best engagement) | 60s (most flexible)
|
||||
HOOK: First 1-2 seconds — visual or text pattern interrupt
|
||||
CAPTIONS: Mandatory (85% watch muted on mobile)
|
||||
TEXT SIZE: 42px+ minimum, bold sans-serif
|
||||
PACING: Visual change every 1-3 seconds
|
||||
TARGET LUFS: -14 LUFS, true peak -1 dBTP
|
||||
MUSIC: 120-140 BPM for energetic, 90-110 for explainers
|
||||
```
|
||||
|
||||
## Platform Safe Zones (1080x1920)
|
||||
|
||||
| Platform | Safe Zone | Top Dead | Bottom Dead | Right Dead |
|
||||
|----------|-----------|----------|-------------|------------|
|
||||
| TikTok | 900x1492 | 108px | 320px | 120px |
|
||||
| Instagram Reels | 996x1400 | 210px | 310px | 84px |
|
||||
| YouTube Shorts | 984x1500 | 120px | 300px | 96px |
|
||||
| Facebook Reels | 1080x1520 | 100px | 300px | 60px |
|
||||
|
||||
**Universal safe zone: 900x1400px centered** — works across all platforms.
|
||||
|
||||
**Bottom dead zones are critical** — platform UI (comments, share buttons, captions) covers the bottom 300-320px. Never put important content there.
|
||||
|
||||
## Upload Specs
|
||||
|
||||
```
|
||||
CODEC: H.264 High Profile, Level 4.2
|
||||
BITRATE: 8-15 Mbps VBR (below 5 Mbps triggers quality downgrade)
|
||||
FORMAT: .mp4 preferred
|
||||
MAX SIZE: 500 MB (desktop), 287.6 MB (iOS), 72 MB (Android)
|
||||
```
|
||||
|
||||
## Duration Strategy
|
||||
|
||||
| Duration | Avg Completion Rate | Best For |
|
||||
|----------|-------------------|----------|
|
||||
| 0-15s | 92% | Single fact, quick tip, visual gag |
|
||||
| 16-30s | 84% | One concept explained, before/after |
|
||||
| 31-60s | 68% | Mini tutorial, step-by-step, story arc |
|
||||
| 60s+ | 48% | Deep explainer, only if retention structure is strong |
|
||||
|
||||
**Platform sweet spots:**
|
||||
- TikTok: 21-34 seconds for completion; 60-180s for maximum total watch time
|
||||
- Reels: 15-30 seconds for viral reach; 60-90s for highest engagement
|
||||
- Shorts: Bimodal — ~13 seconds OR full 60 seconds (Shortimize 35B views analysis)
|
||||
|
||||
**Key formula:** A 45s video with 70% completion (31.5s watch time) outperforms a 15s video with 40% completion (6s). Total watch time is what the algorithm rewards.
|
||||
|
||||
## The 1-Second Hook
|
||||
|
||||
**70%+ of TikTok users decide to scroll or stay within 3 seconds** (average decision point: 1.7 seconds). The hook must be immediate.
|
||||
|
||||
### 3-Second Retention and Algorithmic Impact
|
||||
|
||||
| 3-Second Retention | Algorithmic Effect | View Multiplier |
|
||||
|-------------------|-------------------|-----------------|
|
||||
| Below 60% | Minimal promotion | 1.0x (baseline) |
|
||||
| 60-70% | Average distribution | 1.6x |
|
||||
| 70-85% | Optimal reach | 2.2x |
|
||||
| 85%+ | Viral potential | 2.8x |
|
||||
|
||||
### Retention Checkpoints
|
||||
|
||||
| Timestamp | Target Retention |
|
||||
|-----------|-----------------|
|
||||
| 3 seconds | 70%+ |
|
||||
| 15 seconds | 60%+ |
|
||||
| 30 seconds | 50%+ |
|
||||
|
||||
### Hook Techniques
|
||||
|
||||
| Technique | Example | When to Use |
|
||||
|-----------|---------|-------------|
|
||||
| **Bold text on screen** | "STOP doing this..." (text appears frame 1) | Always — text hooks work even muted |
|
||||
| **Pattern interrupt** | Unexpected visual, jump cut, color flash | Attention-grabbing |
|
||||
| **Question** | "Why does X happen?" (text + voiceover) | Educational |
|
||||
| **Result first** | Show the finished result, then explain how | Tutorial/how-to |
|
||||
| **Controversy** | "Everyone gets this wrong" | Engagement bait |
|
||||
|
||||
### Hook Rules
|
||||
|
||||
1. **Frame 1 must have visual interest** — no blank intros, no logos, no "hey guys"
|
||||
2. **Text appears in the first 0.5 seconds** — viewers scan text before listening
|
||||
3. **Voice starts immediately** — no silent buildup
|
||||
4. **Movement in frame 1** — static opening frames get scrolled past
|
||||
|
||||
## Pacing
|
||||
|
||||
| Rule | Value | Why |
|
||||
|------|-------|-----|
|
||||
| Visual change frequency | Every 1-3 seconds | Mobile attention span |
|
||||
| Cuts per minute | 20-40 | 2-3x faster than long-form |
|
||||
| Text on screen | 2-4 seconds per text block | Fast reading pace |
|
||||
| No static holds | Max 3 seconds | Anything longer feels frozen |
|
||||
| Speed ramp | 1.2-1.5x for setup, 1.0x for payoff | Compress boring parts |
|
||||
|
||||
**Impact of pacing on retention:**
|
||||
- Pattern interrupts every 2-4s: **58% average retention**
|
||||
- Static talking head (no interrupts): **41% average retention**
|
||||
- That's a **41% relative improvement** from pacing alone
|
||||
|
||||
### Script Word Counts
|
||||
|
||||
| Duration | Word Count |
|
||||
|----------|-----------|
|
||||
| 15 seconds | 35-40 words |
|
||||
| 30 seconds | 70-80 words |
|
||||
| 60 seconds | 125-150 words |
|
||||
|
||||
## Text & Captions
|
||||
|
||||
### Mandatory Captions
|
||||
|
||||
**80% of short-form viewers watch without sound** (mid-2025 data). Videos with accurate captions average **12% higher retention**. Captions are not optional.
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Font size | 42px+ at 1080p |
|
||||
| Font weight | Bold |
|
||||
| Font family | Sans-serif (Inter, Montserrat, Poppins) |
|
||||
| Background | Semi-transparent black (75% opacity) or text stroke (3px) |
|
||||
| Position | Center or lower-center, within safe zone |
|
||||
| Max chars/line | 30 |
|
||||
| Max lines | 2 |
|
||||
| Word-by-word highlight | Recommended for engagement |
|
||||
|
||||
### On-Screen Text (Non-Caption)
|
||||
|
||||
- Position in the **top 40%** of the safe zone (above center)
|
||||
- Bold, high contrast (white on dark or colored background box)
|
||||
- 3-5 words maximum per text block
|
||||
- Animate entrance (scale pop or fade, 0.2-0.3s)
|
||||
|
||||
## Audio
|
||||
|
||||
| Element | Level | Notes |
|
||||
|---------|-------|-------|
|
||||
| Voiceover | -12 to -14 dB peak | Primary |
|
||||
| Music | -22 to -26 dB | Lower than long-form — less room |
|
||||
| SFX | -18 to -14 dB | Brief pops/whooshes only |
|
||||
| Target LUFS | -14 LUFS | Same as long-form YouTube |
|
||||
| True peak | -1 dBTP | TikTok/Instagram spec |
|
||||
|
||||
### Music Selection
|
||||
- **Energetic content:** 120-140 BPM
|
||||
- **Explainer content:** 90-110 BPM
|
||||
- **Match trending audio patterns** — short-form audiences expect music-forward content
|
||||
- **Music should start immediately** — no silent intro
|
||||
|
||||
### Voiceover Pacing
|
||||
- **180-200 WPM** for short-form (faster than long-form's 150-160)
|
||||
- Speak with energy and urgency
|
||||
- No long pauses — dead air = scroll
|
||||
|
||||
## Structure Templates
|
||||
|
||||
### 15-Second Quick Tip
|
||||
```
|
||||
[0-1s] HOOK: Bold text + voice starts immediately
|
||||
[1-3s] CONTEXT: One sentence setup
|
||||
[3-12s] CONTENT: The tip/fact/technique (show, don't tell)
|
||||
[12-15s] PAYOFF: Result or CTA text overlay
|
||||
```
|
||||
|
||||
### 30-Second Explainer
|
||||
```
|
||||
[0-1s] HOOK: Pattern interrupt or question
|
||||
[1-5s] PROBLEM: Why this matters
|
||||
[5-22s] SOLUTION: Step-by-step with visual changes every 2-3s
|
||||
[22-28s] RESULT: Show the outcome
|
||||
[28-30s] CTA: Follow/share/comment prompt
|
||||
```
|
||||
|
||||
### 60-Second Mini Tutorial
|
||||
```
|
||||
[0-2s] HOOK: Show finished result first
|
||||
[2-8s] SETUP: "Here's how to do X in Y steps"
|
||||
[8-45s] STEPS: 3-5 steps, ~8s each, visual change per step
|
||||
[45-55s] RESULT: Before/after or final demo
|
||||
[55-60s] CTA + LOOP: End connects back to start for replay
|
||||
```
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When building short-form content:
|
||||
|
||||
1. **Set output resolution to 1080x1920** (9:16) in the compose stage
|
||||
2. **Keep all text within 900x1400px safe zone** — centered in frame
|
||||
3. **Captions are mandatory** — use `subtitle_gen` with word-by-word timing
|
||||
4. **Hook in frame 1** — text overlay + voice starts immediately, no intro
|
||||
5. **Visual change every 1-3 seconds** — use quick cuts, zooms, text pops
|
||||
6. **Voiceover at 180-200 WPM** — faster than long-form
|
||||
7. **Music starts immediately** — set `music_gen` to energetic BPM (110-140)
|
||||
8. **Target 15-30 seconds** for maximum completion rate
|
||||
9. **Test on phone** — view at actual mobile size before publishing
|
||||
@@ -0,0 +1,141 @@
|
||||
# Sound Design for Video Production
|
||||
|
||||
> Sources: W3C accessibility standards, BBC audio guidelines, YouTube/TikTok platform specs,
|
||||
> Sweetwater mastering guides, ElevenLabs documentation, Boris FX, HookSounds, Artlist
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
DIALOGUE: -12 dB peak | -16 to -14 LUFS integrated
|
||||
MUSIC BED: -30 to -20 dB (18-20 dB below dialogue)
|
||||
SFX: -18 to -12 dB (6 dB below dialogue minimum)
|
||||
WHOOSH TIMING: Start 10-20ms before visual, duration 400-500ms
|
||||
MUSIC BPM: Calm 60-80 | Standard 90-110 | Upbeat 120-140
|
||||
TRUE PEAK: Never exceed -1.5 dBTP
|
||||
VOICE EQ: HPF 80Hz, cut 500Hz, boost 2-5kHz, cut 6-8kHz
|
||||
VOICE COMP: 3:1 ratio, 1-5ms attack, 10-20ms release
|
||||
TARGET LUFS: -14 LUFS (YouTube/TikTok/IG) | -16 LUFS (podcasts)
|
||||
```
|
||||
|
||||
## Audio Ducking Levels
|
||||
|
||||
| Element | Peak Level | Notes |
|
||||
|---------|-----------|-------|
|
||||
| Dialogue / Narration | -6 dB to -12 dB | Primary element |
|
||||
| Background music (during speech) | -18 dB to -20 dB | 18-20 dB below dialogue |
|
||||
| Sound effects | -12 dB to -18 dB | Between dialogue and music |
|
||||
| Final mix | -10 dB to -20 dB | Never exceed 0 dB |
|
||||
|
||||
**Ducking rules:**
|
||||
- W3C accessibility: music must be **20 dB lower** than foreground speech
|
||||
- BBC guideline: lower music by an additional **4 dB** from where you think it sounds right
|
||||
- Duck music **6-12 dB** when narration is active; for complex educational topics, duck up to **22 dB**
|
||||
- EQ trick: cut **2-4 kHz** on background music to make room for speech clarity
|
||||
- When testing, adjust in **1 dB increments** from a -20 dB baseline upward
|
||||
|
||||
## Music Selection by Content Type
|
||||
|
||||
| Content Type | BPM Range | Mood |
|
||||
|-------------|-----------|------|
|
||||
| Calm explainer / tutorial | 60-80 | Contemplative, focused, trust-building |
|
||||
| Corporate / testimonial | 60-100 | Professional, calm, credible |
|
||||
| Standard explainer / educational | 90-110 | Steady, engaging, not distracting |
|
||||
| Upbeat explainer / promo | 110-130 | Enthusiastic, approachable |
|
||||
| High-energy / product demo | 120-140 | Exciting, urgent, dynamic |
|
||||
| Action / fast-paced | 140-200 | Adrenaline, intensity |
|
||||
|
||||
**Genre recommendations for explainers:**
|
||||
- Lo-fi (steady, non-distracting, modern feel)
|
||||
- Ambient (atmospheric, stays in background)
|
||||
- Light acoustic guitar instrumentals (warm, approachable)
|
||||
- Contemporary pop instrumentals (upbeat, familiar)
|
||||
- Inspiring soundtrack / cinematic light (builds emotion without overwhelming)
|
||||
|
||||
**Key rules:**
|
||||
- Always use **instrumental** tracks when voiceover is present — lyrics compete with narration
|
||||
- Choose dynamically **even** tracks — avoid dramatic crescendos or beat drops
|
||||
- Match energy to the learning context: upbeat for "exciting new concept," gentle for serious topics
|
||||
|
||||
## Sound Effects (SFX) Placement
|
||||
|
||||
### SFX Categories for Explainer Videos
|
||||
|
||||
| SFX Type | Use Case | Duration | Level |
|
||||
|----------|----------|----------|-------|
|
||||
| Whoosh / Swish | Scene transitions, slide changes | 400-500ms | -18 to -12 dB |
|
||||
| Pop / Pluck | Text appearing, bullet points | <200ms | -15 to -12 dB |
|
||||
| Click / Tap | UI interactions, button presses | <100ms | -20 to -15 dB |
|
||||
| Riser / Swell | Building to a reveal or key point | 1-3s | -18 to -12 dB |
|
||||
| Impact / Hit | Key reveal, important stat | <300ms | -12 to -6 dB |
|
||||
| Subtle whoosh | Element sliding in/out | 200-400ms | -20 to -15 dB |
|
||||
|
||||
### Timing rules
|
||||
- Start whoosh **10-20ms before** the visual transition (brain processes audio faster)
|
||||
- Peak of whoosh energy should coincide with the **moment of greatest visual change**
|
||||
- Fine-tune in **1-frame increments** for sync
|
||||
- When stacking whooshes, keep them in different frequency bands
|
||||
|
||||
## Platform Loudness Targets (2025)
|
||||
|
||||
| Platform | Integrated LUFS | True Peak | Notes |
|
||||
|----------|----------------|-----------|-------|
|
||||
| YouTube | -14 LUFS | -1.5 dBTP | Normalizes down, not up |
|
||||
| YouTube Shorts | -14 LUFS | -1.5 dBTP | Same as long-form |
|
||||
| TikTok | -14 LUFS | -1 dBTP | Prioritize 2-4 kHz for phone speakers |
|
||||
| Instagram Reels | -14 LUFS | -1 dBTP | Same mobile optimization |
|
||||
| Spotify | -14 LUFS | -2 dBTP | Stricter true peak |
|
||||
| Apple Podcasts | -16 LUFS | -1 dBTP | More headroom for speech |
|
||||
|
||||
### Content-type LUFS
|
||||
|
||||
| Content Type | Integrated LUFS | Dynamic Range |
|
||||
|-------------|----------------|---------------|
|
||||
| Dialogue-heavy / educational | -16 to -14 LUFS | 6-12 dB |
|
||||
| Music videos | -14 to -12 LUFS | 6-10 dB |
|
||||
| Gaming content | -14 to -12 LUFS | 8-12 dB |
|
||||
|
||||
### Technical specs
|
||||
- Sample rate: **48 kHz** preferred
|
||||
- Bit depth: **24-bit** preferred
|
||||
- Bitrate: **192 kbps** minimum
|
||||
- Noise floor: below **-60 dB**
|
||||
- Headroom: at least **-6 dB** in the final mix
|
||||
|
||||
## AI TTS (ElevenLabs) Mixing
|
||||
|
||||
### Processing Chain
|
||||
|
||||
1. **High-pass filter:** 80-100 Hz (24 dB/oct slope) — removes rumble and low-frequency TTS artifacts
|
||||
2. **EQ:**
|
||||
- Cut ~500 Hz: removes muddiness/boxy quality
|
||||
- Boost 2-5 kHz (+2-3 dB): adds presence and clarity
|
||||
- Cut 6-8 kHz (gentle): reduces sibilance/harshness common in AI voices
|
||||
- Optional: boost 120-250 Hz for thinner AI voices
|
||||
3. **Compression:**
|
||||
- Ratio: **3:1** (range 2:1 to 4:1)
|
||||
- Attack: **1-5 ms**
|
||||
- Release: **10-20 ms** (increase to 30ms if pumping)
|
||||
- Threshold: **-26 dB** (target -4 to -6 dB gain reduction)
|
||||
- Output gain: **+6 dB**
|
||||
4. **De-esser:** target **6-8 kHz** if sibilance remains
|
||||
5. **Limiter:** ceiling at **-1.5 dBTP**
|
||||
|
||||
### AI-specific tips
|
||||
- AI TTS has inconsistent dynamics — compression is more important than for human speech
|
||||
- ElevenLabs may have subtle artifacts in 4-6 kHz; use narrow notch cut if detected
|
||||
- Sidechain background music to voiceover track for automatic ducking
|
||||
- Cut 2-4 kHz on the music bed to clear the "intelligibility band" for voice
|
||||
- Always test on phone speakers — if voice disappears, boost 2-4 kHz more aggressively
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When the **audio_mixer** tool is used in the compose stage:
|
||||
|
||||
1. Set narration as primary track, music as secondary
|
||||
2. Apply ducking: music -18 to -20 dB below narration during speech
|
||||
3. Select music BPM from the table above based on the playbook mood
|
||||
4. Place SFX at transition points with 10-20ms audio lead
|
||||
5. Target -14 LUFS integrated for YouTube output
|
||||
6. Keep true peak below -1.5 dBTP
|
||||
7. For AI TTS narration, apply the processing chain above before mixing
|
||||
8. Test the final mix on phone speakers — most viewers watch on mobile
|
||||
@@ -0,0 +1,127 @@
|
||||
# Stock Sourcing Usage for OpenMontage
|
||||
|
||||
> How to use the stock image and video tools effectively — query construction,
|
||||
> provider selection, license awareness, and integration with the asset pipeline.
|
||||
|
||||
## Available Stock Tools
|
||||
|
||||
| Tool | Provider | Content | Cost | Rate Limit | Best For |
|
||||
|------|----------|---------|------|-----------|----------|
|
||||
| `pexels_image` | Pexels | Photos | Free | 200/hr | High-quality photography, diverse library |
|
||||
| `pixabay_image` | Pixabay | Photos, illustrations, vectors | Free | 100/min | Category filtering, large library (5M+) |
|
||||
| `pexels_video` | Pexels | Video clips | Free | 200/hr | HD/4K real-world footage |
|
||||
| `pixabay_video` | Pixabay | Video clips | Free | 100/min | Category-filtered video, animation clips |
|
||||
|
||||
## Provider Selection Guide
|
||||
|
||||
### When to Use Pexels
|
||||
- Need **high-quality photography** (curated, professional)
|
||||
- Need **video** (larger video library than Pixabay)
|
||||
- Want **orientation filtering** (landscape/portrait/square)
|
||||
- Want **color filtering** (match playbook palette)
|
||||
- Need results in **multiple languages** (28 locales)
|
||||
|
||||
### When to Use Pixabay
|
||||
- Need **category-based filtering** (nature, business, science, etc.)
|
||||
- Want **illustrations or vectors** in addition to photos
|
||||
- Want **editor's choice** curated results
|
||||
- Need **higher rate limits** (100/min vs 200/hr)
|
||||
- Need **video type filtering** (film vs animation)
|
||||
|
||||
### Decision Flow
|
||||
```
|
||||
Need stock image?
|
||||
├── Need specific category (science, business, etc.)? → pixabay_image
|
||||
├── Need illustration/vector? → pixabay_image
|
||||
├── Need color matching? → pexels_image
|
||||
└── General photo? → pexels_image (higher quality curation)
|
||||
|
||||
Need stock video?
|
||||
├── Need 4K? → pexels_video (supports 4K via size="large")
|
||||
├── Need animation clips? → pixabay_video (video_type="animation")
|
||||
├── Need category filter? → pixabay_video
|
||||
└── General footage? → pexels_video (better HD quality)
|
||||
```
|
||||
|
||||
## Input Parameters Guide
|
||||
|
||||
### pexels_image / pexels_video
|
||||
```python
|
||||
{
|
||||
"query": "city skyline sunset", # Required: search term
|
||||
"orientation": "landscape", # Optional: landscape/portrait/square
|
||||
"size": "large", # Optional: large/medium/small
|
||||
"color": "FF6B35", # Optional: hex without # or color name
|
||||
"per_page": 5, # Results per page (1-80)
|
||||
"download_size": "large2x", # Image: original/large2x/large/medium
|
||||
"preferred_quality": "hd", # Video: hd/sd
|
||||
"output_path": "assets/images/s3.jpg" # Where to save
|
||||
}
|
||||
```
|
||||
|
||||
### pixabay_image / pixabay_video
|
||||
```python
|
||||
{
|
||||
"query": "server room", # Required: search term (max 100 chars)
|
||||
"image_type": "photo", # Image: all/photo/illustration/vector
|
||||
"video_type": "film", # Video: all/film/animation
|
||||
"orientation": "horizontal", # all/horizontal/vertical
|
||||
"category": "computer", # One of 20 categories
|
||||
"colors": "blue,gray", # Comma-separated color names
|
||||
"editors_choice": true, # Curated high-quality only
|
||||
"safesearch": true, # Always true for production
|
||||
"output_path": "assets/video/s5.mp4" # Where to save
|
||||
}
|
||||
```
|
||||
|
||||
## Gotchas and Best Practices
|
||||
|
||||
### 1. Pixabay URLs Expire
|
||||
Pixabay download URLs contain embedded tokens that expire. **Always download immediately** after searching. The tools handle this automatically, but never cache Pixabay URLs for later use.
|
||||
|
||||
### 2. Pixabay Resolution Limit
|
||||
Standard Pixabay API users get max 1280px wide images (`largeImageURL`). Full resolution requires approved API access. For most video production overlays, 1280px is sufficient.
|
||||
|
||||
### 3. Pexels Auth Header
|
||||
Pexels uses a bare API key in the `Authorization` header (NOT `Bearer`). The tool handles this, but be aware if debugging.
|
||||
|
||||
### 4. Search Results Vary by Locale
|
||||
Pexels supports 28 locales. If searching for culturally specific content, set the locale parameter.
|
||||
|
||||
### 5. Stock Images Are Deterministic
|
||||
Unlike AI generation, searching "ocean waves" twice returns the same results. If the first result isn't good enough, try different keywords — don't retry the same query.
|
||||
|
||||
### 6. Duration Filtering for Video
|
||||
Both stock video tools support `min_duration` and `max_duration` parameters. Use these to avoid downloading 30-second clips when you only need 4 seconds — it saves bandwidth and time.
|
||||
|
||||
## Integration with Asset Pipeline
|
||||
|
||||
Stock tools integrate exactly like generation tools. In the asset manifest:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "broll-s3",
|
||||
"type": "image",
|
||||
"subtype": "broll",
|
||||
"path": "assets/images/broll-s3.jpg",
|
||||
"source_tool": "pexels_image",
|
||||
"scene_id": "scene-3",
|
||||
"cost_usd": 0.00,
|
||||
"metadata": {
|
||||
"photographer": "Joey Farina",
|
||||
"source_url": "https://www.pexels.com/photo/2014422/",
|
||||
"license": "Pexels License (free, no attribution required)"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The Edit Director and Compose Director treat stock assets identically to generated ones — they just reference the file path from the manifest.
|
||||
|
||||
## Licensing Summary
|
||||
|
||||
| Provider | Commercial Use | Attribution | Restrictions |
|
||||
|----------|---------------|-------------|-------------|
|
||||
| Pexels | Yes, free | Not required (appreciated) | Cannot sell unaltered; cannot imply endorsement |
|
||||
| Pixabay | Yes, free | Not required | Cannot sell unaltered; cannot create competing stock service |
|
||||
|
||||
Both are safe for all OpenMontage use cases. No licensing fees, no per-use royalties, no attribution obligations.
|
||||
@@ -0,0 +1,149 @@
|
||||
# Storytelling & Narrative Structure for Explainer Videos
|
||||
|
||||
> Sources: YouTube Creator Academy, Derek Muller PhD thesis (U. Sydney 2008), Kurzgesagt production
|
||||
> methodology (Philipp Dettmer), 3Blue1Brown (Grant Sanderson), Richard Mayer "Multimedia Learning"
|
||||
> (Cambridge UP, 2001/2020)
|
||||
|
||||
## The Explainer Arc Template
|
||||
|
||||
For a **3-minute explainer video** (scale proportionally for other lengths):
|
||||
|
||||
```
|
||||
[0:00 - 0:08] HOOK
|
||||
Pattern interrupt or counterintuitive claim. 1-2 sentences max.
|
||||
Visual: striking image or animation that creates curiosity.
|
||||
|
||||
[0:08 - 0:30] TENSION / INFORMATION GAP
|
||||
"Here's what most people think... but that's not quite right."
|
||||
Establish stakes: why should I care?
|
||||
Visual: show the misconception or the puzzle.
|
||||
|
||||
[0:30 - 0:50] CONCEPT 1 (Foundation)
|
||||
Simplest building block needed. ONE idea, ONE visual.
|
||||
End with a "but" or "therefore" transition.
|
||||
|
||||
[0:50 - 1:15] CONCEPT 2 (Complication)
|
||||
Build on Concept 1. Introduce the wrinkle.
|
||||
Visual: transform/evolve the previous visual.
|
||||
|
||||
[1:15 - 1:20] PALETTE CLEANSER
|
||||
Brief pause, visual gag, or "let that sink in" moment.
|
||||
Gives working memory a beat to consolidate.
|
||||
|
||||
[1:20 - 1:50] CONCEPT 3 (Key Insight)
|
||||
The "aha" moment. Core of the video.
|
||||
1-3 seconds of deliberate silence after the reveal.
|
||||
Visual: the most polished animation in the video.
|
||||
|
||||
[1:50 - 2:20] PROOF / EXAMPLE
|
||||
Concrete demonstration: "Watch what happens when..."
|
||||
Visual: show the insight working in a specific case.
|
||||
|
||||
[2:20 - 2:45] IMPLICATIONS / "SO WHAT?"
|
||||
Connect back to the real world. "This means that..."
|
||||
Scale from specific back to general.
|
||||
|
||||
[2:45 - 3:00] REFRAME + CLOSE
|
||||
Callback to the hook. Restate the core insight in one sentence.
|
||||
Optional: open a new curiosity gap.
|
||||
```
|
||||
|
||||
## Scaling by Duration
|
||||
|
||||
| Length | Concepts | Hook | Tension | Core | Proof | Close |
|
||||
|--------|----------|------|---------|------|-------|-------|
|
||||
| 1 min | 1-2 | 5s | 10s | 30s | 10s | 5s |
|
||||
| 2 min | 2-3 | 8s | 15s | 60s | 25s | 12s |
|
||||
| 3 min | 3-5 | 8s | 22s | 100s | 30s | 15s |
|
||||
| 5 min | 5-8 | 10s | 30s | 180s | 50s | 20s |
|
||||
|
||||
## Hook Types
|
||||
|
||||
| Type | Pattern | Best For |
|
||||
|------|---------|----------|
|
||||
| **Contrarian** | "Everything you've been told about X is wrong." | Veritasium-style science/myth-busting |
|
||||
| **Outcome** | "By the end of this video, you'll understand X." | 3Blue1Brown-style math/concept |
|
||||
| **Mystery** | "In 1987, something impossible happened..." | Kurzgesagt-style story-driven |
|
||||
| **Stakes** | "This one mistake costs people X every year." | Practical/how-to content |
|
||||
|
||||
## The 30-Second Rule
|
||||
|
||||
YouTube data shows **50% of viewer drop-off happens in the first 30 seconds**. The hook + tension
|
||||
setup MUST be complete by second 30. Retention curves that survive the 30-second cliff typically
|
||||
retain 40-60% through the full video.
|
||||
|
||||
## The "But-Therefore" Method
|
||||
|
||||
Never connect sections with "and then." Always use **"but"** or **"therefore."**
|
||||
|
||||
**Bad:** "Atoms have electrons, AND THEN those electrons have energy levels, AND THEN..."
|
||||
|
||||
**Good:** "Atoms have electrons, BUT they don't behave like tiny planets, THEREFORE we need a
|
||||
completely new model..."
|
||||
|
||||
Applied structure:
|
||||
```
|
||||
SETUP: Here's what you think you know about X.
|
||||
BUT: Here's why that's wrong / incomplete / surprising.
|
||||
THEREFORE: We need to understand Y (the real mechanism).
|
||||
BUT: Y creates a new puzzle...
|
||||
THEREFORE: The actual answer is Z.
|
||||
THEREFORE: This changes how you should think about X.
|
||||
```
|
||||
|
||||
## Misconception-First Approach (Research-Backed)
|
||||
|
||||
Derek Muller's PhD research (University of Sydney, 2008) showed that **videos presenting common
|
||||
misconceptions FIRST, then refuting them, produce significantly higher learning gains** than videos
|
||||
that simply present correct information. Viewers who watched "misconception-first" videos scored
|
||||
higher on post-tests and reported higher engagement.
|
||||
|
||||
Apply this: always consider opening with what the audience *thinks* is true before revealing what
|
||||
*actually* is.
|
||||
|
||||
## Guided Discovery (3Blue1Brown Method)
|
||||
|
||||
Don't explain the answer. **Reconstruct the reasoning path** so the viewer feels they discovered it.
|
||||
|
||||
1. **The Question** — Pose a specific, concrete question
|
||||
2. **The Naive Attempt** — Show the obvious approach; let it partially work, then break
|
||||
3. **The Key Insight** — Introduce ONE new idea. Pause visually for 2-3 seconds of silence.
|
||||
4. **The Build** — Apply the insight step by step. Each step feels inevitable.
|
||||
5. **The Generalization** — "Notice this pattern works beyond our specific example..."
|
||||
|
||||
**Progressive Revelation:** Never show the full picture at once. Build visuals layer by layer.
|
||||
Each layer arrives exactly when the narration references it.
|
||||
|
||||
## Pacing Rules
|
||||
|
||||
| Rule | Value | Source |
|
||||
|------|-------|--------|
|
||||
| Narration speed | 150-160 wpm | Kurzgesagt standard (conversational is 170-190) |
|
||||
| New visual element | Every 3-5 seconds | Kurzgesagt production rules |
|
||||
| Concept density | Max 1 new concept per 30-45 seconds | Mayer's Segmenting Principle |
|
||||
| Pattern interrupt | Every 45-90 seconds | YouTube retention data |
|
||||
| Deliberate silence | 1-3 seconds after key insights | 3Blue1Brown technique |
|
||||
| Palette cleanser | Every 45-60 seconds | Kurzgesagt production rules |
|
||||
|
||||
## Mayer's Multimedia Learning Principles (Applied)
|
||||
|
||||
These are the most relevant research-backed rules from cognitive science:
|
||||
|
||||
1. **Segmenting** — Max 1 new concept per 30-45 seconds. A 3-min video = 4-6 concept segments.
|
||||
2. **Signaling** — Use verbal signposts every 30-45 seconds ("Here's where it gets interesting").
|
||||
3. **Temporal Contiguity** — Narration and visuals must be simultaneous. Learning drops ~30% when offset even by a few seconds.
|
||||
4. **Coherence** — Remove interesting-but-irrelevant content. "Seductive details" reduce learning by 20-30% on transfer tests.
|
||||
5. **Modality** — Use narration (audio) + visuals (animation), NOT on-screen text + visuals. Spoken words + pictures outperform written words + pictures.
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When writing a **script artifact** for the animated-explainer pipeline:
|
||||
|
||||
1. Choose a hook type from the table above based on the topic
|
||||
2. Structure sections using the Explainer Arc template
|
||||
3. Apply "but-therefore" connectors between sections
|
||||
4. Consider the misconception-first approach for science/technical topics
|
||||
5. Set `narration_wpm: 155` in the script to calculate accurate timing
|
||||
6. Plan visual changes every 3-5 seconds in the scene_plan
|
||||
7. Mark "silence" beats in the script for key insights
|
||||
8. Validate: total concepts should not exceed the scaling table above
|
||||
@@ -0,0 +1,137 @@
|
||||
# Talking Head Generation Usage for OpenMontage
|
||||
|
||||
> Sources: SadTalker paper (Zhang et al. 2023), MuseTalk documentation, existing Layer 2 skills
|
||||
> at `skills/creative/face-restore-usage.md` and `skills/creative/enhancement-strategy.md`
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
DEFAULT MODEL: sadtalker
|
||||
INPUT: One face photo + one audio file → animated talking video
|
||||
EXPRESSION: expression_scale=1.0 (0.5 = subtle, 1.5 = expressive)
|
||||
STILL MODE: false (true = mouth-only animation, head stays fixed)
|
||||
PREPROCESS: crop (default — crops face, animates, pastes back)
|
||||
KEY RULE: Generate audio FIRST, then pass to talking_head
|
||||
```
|
||||
|
||||
## When to Use the talking_head Tool
|
||||
|
||||
| Scenario | Use talking_head? |
|
||||
|----------|-------------------|
|
||||
| Avatar spokesperson video from a single photo | Yes |
|
||||
| Personalized message — animate a headshot with custom narration | Yes |
|
||||
| No video footage exists but a photo is available | Yes |
|
||||
| Multi-language avatar — same face, different audio tracks | Yes |
|
||||
| Existing video footage needs processing | No — use the talking-head pipeline |
|
||||
| Lip-syncing existing video to new audio | No — use the `lip_sync` tool |
|
||||
|
||||
## Input Requirements
|
||||
|
||||
### Photo
|
||||
|
||||
- Clear, front-facing face with good lighting
|
||||
- Minimum resolution: 256x256px
|
||||
- Best results: 512x512 or larger
|
||||
- Neutral expression, direct eye contact
|
||||
- Avoid: extreme angles, accessories covering the face (large sunglasses, masks), multiple faces in the image
|
||||
|
||||
### Audio
|
||||
|
||||
- Clean speech audio — WAV or MP3
|
||||
- Sample rate: 16kHz or higher
|
||||
- Audio duration determines output video duration
|
||||
- Remove background noise before feeding into talking_head — clean audio produces cleaner lip sync
|
||||
|
||||
## Model Selection
|
||||
|
||||
| Model | Strengths | Weaknesses |
|
||||
|-------|----------|------------|
|
||||
| sadtalker | Natural head motion, good expression range, well-tested | Can struggle with extreme expressions |
|
||||
| musetalk | Higher quality lip sync, sharper mouth region | More constrained head motion |
|
||||
|
||||
**Default to `sadtalker`** unless lip sync precision is the top priority.
|
||||
|
||||
## Settings Reference
|
||||
|
||||
### Preprocess Modes
|
||||
|
||||
| Mode | What It Does | When to Use |
|
||||
|------|-------------|-------------|
|
||||
| `crop` | Crops face region, animates, pastes back into original frame | Default — best for headshots and portraits |
|
||||
| `resize` | Resizes full input to model dimensions | When you want full-frame output at model resolution |
|
||||
| `full` | No preprocessing — input passed directly | Advanced — input must already be correctly sized for the model |
|
||||
|
||||
### expression_scale Tuning
|
||||
|
||||
| Value | Effect | Use Case |
|
||||
|-------|--------|----------|
|
||||
| 0.5 | Subtle, minimal head movement | Corporate, formal, conservative |
|
||||
| 0.7 | Calm, professional | Business presentations, news-style |
|
||||
| 1.0 | Natural conversational (default) | General-purpose, explainers |
|
||||
| 1.5 | Expressive, energetic | Social media, engaging content |
|
||||
| >1.5 | Risk of artifacts | Avoid unless intentionally stylized |
|
||||
|
||||
### still_mode
|
||||
|
||||
| Value | Effect | Use Case |
|
||||
|-------|--------|----------|
|
||||
| `false` (default) | Head moves naturally while speaking | More realistic, conversational feel |
|
||||
| `true` | Only mouth animates, head stays fixed | Formal/corporate look, or when head motion causes artifacts |
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### 1. Avatar Spokesperson
|
||||
|
||||
```
|
||||
photo + elevenlabs_tts → talking_head → face_enhance → compose
|
||||
```
|
||||
|
||||
Standard avatar video: generate speech from script, animate the photo, polish the face, compose into final video.
|
||||
|
||||
### 2. Multi-Language Avatar
|
||||
|
||||
```
|
||||
photo + tts per language → talking_head per language → compose variants
|
||||
```
|
||||
|
||||
Same face photo, different audio tracks per language. Each produces a separate talking-head video for localized content.
|
||||
|
||||
### 3. Quick Social Content
|
||||
|
||||
```
|
||||
headshot + script → piper_tts → talking_head → subtitle_gen → compose
|
||||
```
|
||||
|
||||
Fast turnaround social video: generate speech locally, animate, add subtitles, compose.
|
||||
|
||||
### 4. Photo-to-Explainer
|
||||
|
||||
```
|
||||
talking_head output → compose with diagram overlays
|
||||
```
|
||||
|
||||
Use the talking-head video as a presenter layer, then overlay diagrams, charts, or screen recordings during composition.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before accepting talking_head output, verify:
|
||||
|
||||
- [ ] Lip movements match the audio naturally
|
||||
- [ ] Head motion looks organic, not robotic
|
||||
- [ ] No visual artifacts around face edges or jaw
|
||||
- [ ] Eyes blink naturally (not frozen or blinking too fast)
|
||||
- [ ] Output resolution is acceptable for the target platform
|
||||
- [ ] Expression intensity matches the tone of the narration
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When using the `talking_head` tool:
|
||||
|
||||
1. **Generate audio FIRST** (via `tts_selector`, `elevenlabs_tts`, `openai_tts`, or `piper_tts`), then pass to talking_head
|
||||
2. **Use `expression_scale=1.0` as baseline** — only increase for high-energy content
|
||||
3. **Always apply `face_enhance` AFTER talking_head** to polish the output
|
||||
4. **For corporate/professional content**, use `still_mode=true` and `expression_scale=0.7`
|
||||
5. **Source photo quality directly impacts output quality** — use the best available photo
|
||||
6. **Crop mode is the safest default** — only use `resize` or `full` if crop produces bad framing
|
||||
7. **Preview a 5-second clip before generating the full video** — catch artifacts early
|
||||
8. **Fallback strategy:** if SadTalker is unavailable but Wav2Lip is, record a simple static video from the photo and lip-sync it with the `lip_sync` tool instead
|
||||
@@ -0,0 +1,226 @@
|
||||
# Typography for Video Production
|
||||
|
||||
> Sources: School of Motion typography guides, legibility.info video text rules, Wave.video font
|
||||
> pairing research, EBU/SMPTE broadcast standards, Netflix subtitle spec, BBC subtitle guidelines,
|
||||
> WCAG 2.1 contrast requirements, Easings.net, postplanify.com safe zone data (2026)
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
TITLE SIZE: 60-90px at 1080p | 120-180px at 4K
|
||||
BODY SIZE: 40-60px at 1080p | 80-120px at 4K
|
||||
SUBTITLE SIZE: 42px+ at 1080p | 3-5% of video height
|
||||
MAX CHARS/LINE: 32-42 (subtitles) | 30 (overlays)
|
||||
MAX LINES: 2 (subtitles) | 3 (overlays)
|
||||
READING SPEED: 21 chars/sec | 160-200 WPM
|
||||
TITLE SAFE: 80% of frame (192px margin at 1080p)
|
||||
ACTION SAFE: 90% of frame (96px margin at 1080p)
|
||||
FONT FAMILIES: 1-2 per video maximum
|
||||
CONTRAST: 4.5:1 minimum, 7:1 optimal
|
||||
FADE DURATION: 0.3s opacity | 0.5-1.0s slide/scale
|
||||
```
|
||||
|
||||
## Font Selection
|
||||
|
||||
### Recommended Video Fonts
|
||||
|
||||
| Category | Fonts | Use For |
|
||||
|----------|-------|---------|
|
||||
| **Body / Captions** | Inter, Open Sans, Roboto, Source Sans Pro, Lato, DM Sans | All body text, subtitles, captions |
|
||||
| **Headlines** | Montserrat Bold, Bebas Neue, Oswald Bold, Poppins Bold | Titles, section headers, key stats |
|
||||
| **Editorial** | Playfair Display, Roboto Slab | Luxury, cinematic, documentary |
|
||||
| **System Safe** | Helvetica Neue, Arial, Avenir Next | When custom fonts unavailable |
|
||||
|
||||
### Font Pairing Rules
|
||||
|
||||
- Limit to **1-2 font families** per video — more creates visual noise
|
||||
- Pair a **display/bold heading** font with a **neutral body** font
|
||||
- Size difference between title and body: at least **50% larger**
|
||||
- **Sans-serif** for motion graphics and captions (holds up in motion)
|
||||
- **Serif** only for cinematic title cards and editorial content
|
||||
- **Script/decorative** fonts: hero titles only, never body, never in motion
|
||||
|
||||
### Proven Pairings
|
||||
|
||||
| Heading | Body | Style |
|
||||
|---------|------|-------|
|
||||
| Bebas Neue | Open Sans | High-impact, social ads |
|
||||
| Montserrat Bold | Lato | Clean modern |
|
||||
| Oswald Bold | Raleway | Strong contrast |
|
||||
| Playfair Display | Inter | Editorial |
|
||||
| Poppins Bold | Poppins Light | Single-family hierarchy |
|
||||
|
||||
## Text Sizing
|
||||
|
||||
### Minimum Readable Sizes
|
||||
|
||||
| Element | 1080p (px) | 4K (px) | Notes |
|
||||
|---------|-----------|---------|-------|
|
||||
| Title / Hero text | 60-90 | 120-180 | Must be readable as thumbnail |
|
||||
| Body text | 40-60 | 80-120 | Absolute minimum for readability |
|
||||
| Subtitles | 42+ | 84+ | Accessibility requirement |
|
||||
| Lower third name | 48-60 | 96-120 | Bold weight |
|
||||
| Lower third role | 36-44 | 72-88 | Light/regular weight |
|
||||
| Thumbnail text | — | — | Must read at 120-160px wide display |
|
||||
|
||||
## Safe Zones
|
||||
|
||||
### Broadcast Standard
|
||||
|
||||
| Zone | Coverage | Margin at 1080p | Purpose |
|
||||
|------|----------|----------------|---------|
|
||||
| **Title Safe** | 80% of frame | 192px H, 108px V | All text must stay within |
|
||||
| **Action Safe** | 90% of frame | 96px H, 54px V | All important content |
|
||||
|
||||
At 1920x1080: Title Safe = inner **1536x864px**
|
||||
At 3840x2160: Title Safe = inner **3072x1728px**
|
||||
|
||||
### Platform-Specific Safe Zones (Vertical 1080x1920)
|
||||
|
||||
| Platform | Safe Zone | Top Dead | Bottom Dead | Right Dead |
|
||||
|----------|-----------|----------|-------------|------------|
|
||||
| **TikTok** | 900x1492 | 108px | 320px | 120px |
|
||||
| **Instagram Reels** | 996x1400 | 210px | 310px | 84px |
|
||||
| **YouTube Shorts** | 984x1500 | 120px | 300px | 96px |
|
||||
| **Facebook Reels** | 1080x1520 | 100px | 300px | 60px |
|
||||
| **Instagram Stories** | 1080x1620 | 100px | 200px | — |
|
||||
|
||||
**Universal cross-platform safe zone: 900x1400px centered** — works on all platforms.
|
||||
|
||||
## Text Animation Timing
|
||||
|
||||
### Duration on Screen
|
||||
|
||||
- Reading speed: **13 characters per second** minimum dwell time
|
||||
- 30-character line: minimum **2.3 seconds**
|
||||
- General rule: **3 seconds per 63 characters**
|
||||
- Title cards: **3-6 seconds**
|
||||
- After animation completes, hold motionless for **1 second per 13 characters**
|
||||
|
||||
### Animation Durations
|
||||
|
||||
| Animation Type | Duration | Use Case |
|
||||
|---------------|----------|----------|
|
||||
| Fade in/out | 0.3-0.5s | Subtle, universal |
|
||||
| Slide / scale entrance | 0.5-1.0s | Standard motion graphics |
|
||||
| Kinetic text entrance | 1.0-2.0s | Bold, energetic |
|
||||
| Lower third entrance | 1.0-2.0s | Speaker identification |
|
||||
| Lower third exit | 0.5-1.0s | Quick departure |
|
||||
|
||||
### Easing Curves
|
||||
|
||||
| Easing | Cubic Bezier | Use For |
|
||||
|--------|-------------|---------|
|
||||
| **easeOutCubic** | `(0.33, 1, 0.68, 1)` | Text entrances (decelerates into place) — **default choice** |
|
||||
| **easeOutQuart** | `(0.25, 1, 0.5, 1)` | Snappier entrance, kinetic type |
|
||||
| **easeInOutQuad** | `(0.45, 0, 0.55, 1)` | Smooth position transitions |
|
||||
| **easeInOutCubic** | `(0.65, 0, 0.35, 1)` | Scale and opacity changes |
|
||||
| **easeInCubic** | `(0.32, 0, 0.67, 0)` | Exits (accelerates out) |
|
||||
|
||||
**Never use linear easing** for text animations — it feels robotic.
|
||||
|
||||
### Reveal Techniques
|
||||
|
||||
| Technique | Feel | Best For |
|
||||
|-----------|------|----------|
|
||||
| Mask reveal | Cinematic | Title cards, premium content |
|
||||
| Scale pop | Energetic | Social media, short-form |
|
||||
| Character stagger | Natural flow | Kinetic typography |
|
||||
| Word-by-word sync | Engaging | Talking-head captions, TikTok |
|
||||
| Fade | Subtle | Professional, corporate |
|
||||
|
||||
## Subtitle & Caption Typography
|
||||
|
||||
### Specifications
|
||||
|
||||
| Parameter | Value | Source |
|
||||
|-----------|-------|--------|
|
||||
| Font size | 42px+ at 1080p | Accessibility standard |
|
||||
| Max characters per line | 32-42 | Platform dependent (see below) |
|
||||
| Max lines | 2 per block | Universal standard |
|
||||
| Line spacing | 1.3x | Readability standard |
|
||||
| Background | Semi-transparent black, 70-80% opacity | Contrast requirement |
|
||||
| Alternative style | White text + 2-4px dark stroke | No-box style |
|
||||
| Minimum contrast | 4.5:1 (white on black = 21:1) | WCAG AA |
|
||||
| Bottom margin | 60px from edge minimum | Mobile gesture clearance |
|
||||
| Within frame width | 90% maximum | Title safe compliance |
|
||||
|
||||
### Character Limits by Platform
|
||||
|
||||
| Platform | Max Chars/Line |
|
||||
|----------|---------------|
|
||||
| YouTube | 42 |
|
||||
| Netflix | 42 |
|
||||
| BBC | 37 |
|
||||
| TV broadcast | 37-42 |
|
||||
| Cinema | 40-45 |
|
||||
|
||||
### Caption Timing
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Minimum duration | 1 second |
|
||||
| Maximum duration | 6-7 seconds |
|
||||
| Reading speed | 21 characters/second |
|
||||
| Fade-in transition | 0.3 seconds |
|
||||
| Gap between captions | 2 frames |
|
||||
| Sync tolerance | 3 frames of audio |
|
||||
|
||||
### Reading Speed by Platform
|
||||
|
||||
| Platform | WPM |
|
||||
|----------|-----|
|
||||
| TikTok / Instagram Reels | 180-200 |
|
||||
| YouTube | 160-180 |
|
||||
| LinkedIn | 140-160 |
|
||||
| Educational content | 120-140 |
|
||||
|
||||
## Lower Thirds
|
||||
|
||||
### Standard Specs (1080p)
|
||||
|
||||
- Overlay region: **1920x360px** (bottom third)
|
||||
- Sans-serif fonts (Helvetica, Open Sans, Roboto)
|
||||
- White text with drop shadow or semi-transparent background bar
|
||||
- Name: bold, larger weight
|
||||
- Role/subtitle: lighter weight, smaller
|
||||
|
||||
### Timing
|
||||
|
||||
| Phase | Duration |
|
||||
|-------|----------|
|
||||
| Entrance animation | 1-2 seconds |
|
||||
| Display | 3-6 seconds |
|
||||
| Exit animation | 0.5-1 second |
|
||||
|
||||
## Contrast & Readability
|
||||
|
||||
### WCAG Requirements
|
||||
|
||||
| Element | Min Ratio | Standard |
|
||||
|---------|----------|----------|
|
||||
| Body text | 4.5:1 | WCAG AA |
|
||||
| Large text (>18pt) | 3:1 | WCAG AA |
|
||||
| Enhanced body | 7:1 | WCAG AAA |
|
||||
| UI components | 3:1 | WCAG 2.1 |
|
||||
|
||||
### Text-Over-Video Techniques
|
||||
|
||||
1. **Semi-transparent box** — 70-80% black opacity behind text (most reliable)
|
||||
2. **Text stroke** — 2-4px dark outline around light text
|
||||
3. **Drop shadow** — subtle shadow for depth (less reliable on busy backgrounds)
|
||||
4. **Darkened region** — gradient overlay behind text area
|
||||
5. **Full-screen overlay** — 30-50% dark overlay for text-heavy screens
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When generating text for video in the compose/asset stages:
|
||||
|
||||
1. **Font selection** — use the recommended video fonts above; prefer Inter or Open Sans for body, Montserrat Bold for titles
|
||||
2. **Size check** — never go below 40px at 1080p for any text element
|
||||
3. **Safe zones** — all text within 80% title-safe area; for vertical/short-form, use the 900x1400px universal safe zone
|
||||
4. **Subtitle styling** — 42px+, max 2 lines, max 42 chars/line, semi-transparent background at 75% opacity
|
||||
5. **Animation** — use easeOutCubic for entrances, hold text for at least 1 second per 13 characters after animation
|
||||
6. **Contrast** — verify 4.5:1 minimum on a representative graded frame; prefer white-on-dark-background (21:1)
|
||||
7. **Platform targeting** — check the platform safe zone table above and adjust text placement accordingly
|
||||
8. **Remotion rendering** — all font families must be loaded via `@import` or `fontFamily` in the component; test that fonts render in the Docker/Lambda environment
|
||||
@@ -0,0 +1,130 @@
|
||||
# Upscaling Usage for OpenMontage
|
||||
|
||||
> Sources: Real-ESRGAN documentation, GFPGAN face enhancement docs, Real-ESRGAN paper
|
||||
> (Wang et al., 2021), practical upscaling benchmarks
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
DEFAULT MODEL: RealESRGAN_x4plus — real-world photos and video frames
|
||||
DEFAULT SCALE: 4x (480p→1080p, 720p→4K)
|
||||
ANIME MODEL: RealESRGAN_x4plus_anime_6B — flat color areas, illustrations
|
||||
FACE ENHANCE: Enable face_enhance for footage with people (uses GFPGAN)
|
||||
DENOISE: 0.5 default, raise to 0.8 for very noisy inputs
|
||||
```
|
||||
|
||||
## When to Upscale
|
||||
|
||||
| Situation | Upscale? | Notes |
|
||||
|-----------|----------|-------|
|
||||
| User-provided footage is 480p or 720p, target is 1080p/4K | Yes | Most common use case |
|
||||
| Generated images need higher resolution for video frames | Yes | AI image output is often 512-1024px |
|
||||
| Thumbnail or still frames need crisp detail | Yes | Single-frame upscale is fast |
|
||||
| Old/archival footage restoration | Yes | Combine with higher denoise_strength |
|
||||
| Source is already 1080p+ and target is 1080p | **No** | Wastes compute, can introduce artifacts |
|
||||
| Source is already 4K | **No** | Over-sharpening degrades quality |
|
||||
|
||||
## Model Selection
|
||||
|
||||
| Model | Best For | Notes |
|
||||
|-------|----------|-------|
|
||||
| `RealESRGAN_x4plus` | Real-world photos, video frames | Default choice |
|
||||
| `RealESRGAN_x4plus_anime_6B` | Anime, illustrations, motion graphics | Preserves flat color areas |
|
||||
| `RealESRNet_x4plus` | Fastest option, slightly lower quality | When speed matters |
|
||||
|
||||
## Scale Factor Guidance
|
||||
|
||||
| Scale | Use Case | Example |
|
||||
|-------|----------|---------|
|
||||
| 4x | Standard upscale for low-res sources | 480p→1080p, 720p→4K |
|
||||
| 2x | Moderate upscale when 4x is overkill | 720p→1080p |
|
||||
|
||||
- **4x** is the most common choice. Use it for 480p sources targeting 1080p, or 720p targeting 4K.
|
||||
- **2x** is appropriate when the source is already 720p and the target is 1080p — avoids unnecessary processing and potential artifacts.
|
||||
- **Never upscale beyond 4x in a single pass.** Quality degrades sharply, and hallucinated details become obvious.
|
||||
|
||||
## Face Enhancement
|
||||
|
||||
- Enable `face_enhance` when the video contains human faces
|
||||
- Uses GFPGAN internally to enhance face regions while Real-ESRGAN handles the rest
|
||||
- Particularly valuable for webcam footage and old video
|
||||
- Do NOT enable for content without faces — adds processing time with no benefit
|
||||
|
||||
## Denoising Strength
|
||||
|
||||
| Source Quality | denoise_strength | Rationale |
|
||||
|---------------|-----------------|-----------|
|
||||
| Clean digital source | 0.5 (default) | Minimal denoising needed |
|
||||
| Slight compression artifacts | 0.6 | Light cleanup without over-smoothing |
|
||||
| Old/noisy footage | 0.7-0.8 | Aggressive denoising for archival content |
|
||||
| Very noisy / low-light footage | 0.8 | Maximum practical denoising |
|
||||
|
||||
Do not exceed 0.8 — higher values destroy legitimate detail.
|
||||
|
||||
## Video Upscaling Notes
|
||||
|
||||
- Video upscaling extracts frames, upscales each, reassembles
|
||||
- This is **SLOW** — budget 5-10x real-time on GPU
|
||||
- For long videos, consider upscaling only key scenes/clips rather than the full video
|
||||
- Audio is preserved from the original
|
||||
- Output file size will be significantly larger (~16x for 4x upscale)
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Workflow 1 — User-Provided Low-Res Footage
|
||||
|
||||
```
|
||||
1. Assess source resolution (e.g., 480p webcam recording)
|
||||
2. Choose scale factor: 4x for 480p→1080p, 2x for 720p→1080p
|
||||
3. Enable face_enhance if footage contains people
|
||||
4. Set denoise_strength based on source quality
|
||||
5. Upscale → inspect output → proceed to compose stage
|
||||
```
|
||||
|
||||
### Workflow 2 — AI-Generated Image Frames
|
||||
|
||||
```
|
||||
1. Generate images at native model resolution (512-1024px)
|
||||
2. Upscale with RealESRGAN_x4plus to target video resolution
|
||||
3. Keep denoise_strength at 0.5 — AI output is clean
|
||||
4. Do NOT enable face_enhance unless faces are prominent
|
||||
```
|
||||
|
||||
### Workflow 3 — Manim / Motion Graphics Frames
|
||||
|
||||
```
|
||||
1. Render Manim at default resolution
|
||||
2. Upscale with RealESRGAN_x4plus_anime_6B (preserves flat colors)
|
||||
3. Keep denoise_strength at 0.5
|
||||
4. Verify text and line art remain sharp
|
||||
```
|
||||
|
||||
### Workflow 4 — Archival Footage Restoration
|
||||
|
||||
```
|
||||
1. Assess noise level and resolution
|
||||
2. Set denoise_strength to 0.7-0.8
|
||||
3. Enable face_enhance for footage with people
|
||||
4. Use RealESRGAN_x4plus at 4x
|
||||
5. Carefully inspect output for hallucinated details
|
||||
```
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
- [ ] Upscaled output is sharp without visible artifacts
|
||||
- [ ] Faces look natural (no over-smoothing or distortion)
|
||||
- [ ] Text/UI elements in screen recordings remain readable
|
||||
- [ ] No hallucinated details in flat color areas
|
||||
- [ ] File size is reasonable (4x upscale = ~16x file size)
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When using the `upscale` tool in the asset stage:
|
||||
|
||||
1. **Upscale BEFORE the compose stage** — it is an asset-prep step, not a post-processing step
|
||||
2. **Use `face_enhance=true` for any talking-head footage** — GFPGAN dramatically improves face quality
|
||||
3. **Use `RealESRGAN_x4plus_anime_6B` model for Manim outputs** or flat illustration frames — preserves clean edges and flat color areas
|
||||
4. **For budget-conscious pipelines**, upscale only hero shots and thumbnails rather than every frame
|
||||
5. **Set `denoise_strength` to 0.7-0.8 for old/noisy footage**, keep at 0.5 for clean digital sources
|
||||
6. **Check upscaled output for artifacts** — over-sharpening, hallucinated texture, face distortion
|
||||
7. **Prefer 2x over 4x when the source is already 720p and target is 1080p** — less compute, fewer artifacts
|
||||
@@ -0,0 +1,61 @@
|
||||
# Video Editing Skill
|
||||
|
||||
## When to Use
|
||||
|
||||
Apply this skill when making editorial decisions for talking-head content:
|
||||
where to cut, what to remove, how to pace, and how to structure the final edit.
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | Role |
|
||||
|------|------|
|
||||
| `transcriber` | Analyze speech for filler words, dead air, false starts |
|
||||
| `video_trimmer` | Execute cuts and speed adjustments |
|
||||
| `frame_sampler` | Sample frames to evaluate visual quality at potential cut points |
|
||||
| `video_compose` | Assemble the final edit |
|
||||
|
||||
## Editing Principles for Talking Heads
|
||||
|
||||
### What to Cut
|
||||
|
||||
1. **Filler words:** "um", "uh", "like", "you know" — cut at word boundaries using word timestamps.
|
||||
2. **False starts:** When the speaker restarts a sentence, keep only the final take.
|
||||
3. **Dead air:** Silence longer than 1.5 seconds should be trimmed to ~0.5 seconds.
|
||||
4. **Off-topic tangents:** If the speaker wanders, cut to the next relevant segment.
|
||||
5. **Repeated points:** Keep the best delivery, remove redundant takes.
|
||||
|
||||
### What NOT to Cut
|
||||
|
||||
- **Breath pauses:** Natural 0.3-0.8 second pauses between sentences. These sound natural.
|
||||
- **Emphasis pauses:** Intentional pauses for dramatic effect.
|
||||
- **Reactions and transitions:** Verbal bridges like "So..." or "Now..." that provide flow.
|
||||
|
||||
### Cut Technique
|
||||
|
||||
- **J-cut:** Audio from the next segment starts ~0.5s before the visual cut. Makes transitions feel smooth.
|
||||
- **L-cut:** Audio from the current segment continues ~0.5s after the visual cut. Maintains continuity.
|
||||
- **Hard cut:** Instant transition. Use at major topic changes.
|
||||
|
||||
### Pacing
|
||||
|
||||
- **Short-form (< 60s):** Aggressive cuts. Minimal dead air. High energy.
|
||||
- **Medium-form (1-10 min):** Balanced. Keep natural pauses for breathing room.
|
||||
- **Long-form (> 10 min):** Let scenes breathe. Only cut obvious problems.
|
||||
|
||||
## Edit Decision Structure
|
||||
|
||||
The `edit_decisions` artifact should include:
|
||||
|
||||
- **cuts:** Ordered list of segments to keep (source, in/out points, speed)
|
||||
- **overlays:** Timed overlay placements (images, diagrams, lower thirds)
|
||||
- **subtitles:** Subtitle configuration (enabled, style, source file)
|
||||
- **music:** Background music settings (asset, volume, ducking, fades)
|
||||
- **transitions:** Transition type and timing between cuts
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
- [ ] No visible jump cuts (smooth transitions between segments)
|
||||
- [ ] Audio doesn't pop or click at cut points
|
||||
- [ ] Pacing matches the content energy and target platform
|
||||
- [ ] Speaker's face is never covered by overlays
|
||||
- [ ] All cuts are at word boundaries (not mid-word)
|
||||
@@ -0,0 +1,195 @@
|
||||
# Video Generation Prompting — Universal Guide
|
||||
|
||||
## When to Use
|
||||
|
||||
When writing prompts for the video generation family (`video_selector`, `heygen_video`,
|
||||
`wan_video`, `hunyuan_video`, `ltx_video_local`, `ltx_video_modal`, `cogvideo_video`).
|
||||
This skill covers the universal prompt vocabulary that works across all video generation models.
|
||||
|
||||
For model-specific tips, see the linked guides below.
|
||||
|
||||
## Model-Specific Guides
|
||||
|
||||
| Model | Guide | Key Insight |
|
||||
|-------|-------|-------------|
|
||||
| **Sora 2 / Sora 2 Pro** | [OpenAI Sora 2 Cookbook](https://developers.openai.com/cookbook/examples/sora/sora2_prompting_guide) | Richest structured template. Advanced fields: lenses, filtration, grade, diegetic sound, wardrobe, finishing. |
|
||||
| **VEO 3.1 / VEO 3** | [Vertex AI Prompt Guide](https://cloud.google.com/vertex-ai/generative-ai/docs/video/video-gen-prompt-guide) | Best vocabulary reference tables. 14-component prompt structure. |
|
||||
| **LTX-2** | [LTX Prompting Guide](https://docs.ltx.video/api-documentation/prompting-guide) | 6-element structure. Audio/voice prompting. Strong "what to avoid" section. |
|
||||
| **HunyuanVideo 1.5** | [Tencent Prompt Handbook](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5/blob/main/assets/HunyuanVideo_1_5_Prompt_Handbook_EN.md) | Formula: Subject + Motion + Scene + [Shot] + [Camera] + [Lighting] + [Style] + [Atmosphere]. |
|
||||
| **Runway Gen-4** | [Runway Prompting Guide](https://help.runwayml.com/hc/en-us/articles/39789879462419-Gen-4-Video-Prompting-Guide) | "Focus on motion, not appearance." One scene per clip. Simplicity wins. |
|
||||
| **Kling 2.6** | [Kling Prompt Guide](https://fal.ai/learn/devs/kling-2-6-pro-prompt-guide) | 4-part structure. Supports `++emphasis++` syntax for key elements. |
|
||||
| **Wan 2.1 / CogVideoX** | Use this generic guide | No official prompt guide. Standard cinematographic vocabulary works well. |
|
||||
|
||||
## Universal Prompt Formula
|
||||
|
||||
All video generation models respond to this structure. Include what's relevant, omit what's not.
|
||||
|
||||
```
|
||||
[Shot type/framing] + [Camera movement] + [Subject description] +
|
||||
[Action/motion in beats] + [Setting/environment] + [Lighting] +
|
||||
[Style/aesthetic] + [Audio/atmosphere]
|
||||
```
|
||||
|
||||
**Shorter prompts = more creative freedom. Longer prompts = more control.**
|
||||
|
||||
---
|
||||
|
||||
## Camera Shot Types
|
||||
|
||||
| Shot | When to Use |
|
||||
|------|-------------|
|
||||
| **Wide / establishing shot** | Open a scene, show location context |
|
||||
| **Full / long shot** | Subject head-to-toe with environment |
|
||||
| **Medium shot** | Waist up, balances detail with context |
|
||||
| **Medium close-up** | Chest up, conversational intimacy |
|
||||
| **Close-up** | Face or key object, emphasize emotion |
|
||||
| **Extreme close-up** | Isolated detail (eye, drop, texture) |
|
||||
| **Over-the-shoulder** | Conversation framing, connection |
|
||||
| **Point-of-view (POV)** | Viewer becomes the character |
|
||||
| **Bird's-eye / top-down** | Map-like overview, omniscient feel |
|
||||
| **Worm's-eye view** | Looking straight up, emphasize height |
|
||||
| **Dutch / canted angle** | Tilted horizon, unease or tension |
|
||||
| **Low-angle** | Subject appears powerful, dominant |
|
||||
| **High-angle** | Subject appears small, vulnerable |
|
||||
|
||||
## Camera Movements
|
||||
|
||||
| Movement | What It Does | Best For |
|
||||
|----------|-------------|----------|
|
||||
| **Static / fixed** | No movement | Dialogue, contemplation, stability |
|
||||
| **Pan** (left/right) | Rotates horizontally | Revealing a scene, following action |
|
||||
| **Tilt** (up/down) | Rotates vertically | Revealing height, slow reveal |
|
||||
| **Dolly in / out** | Physically moves toward/away | Building tension, emphasis |
|
||||
| **Truck** (left/right) | Moves sideways | Parallels subject movement |
|
||||
| **Pedestal** (up/down) | Moves vertically | Smooth elevation changes |
|
||||
| **Crane shot** | Sweeping vertical arcs | Epic reveals, transitions |
|
||||
| **Tracking / follow** | Follows subject | Action sequences, walk-and-talk |
|
||||
| **Arc shot** | Circles around subject | Dramatic emphasis, 360° reveal |
|
||||
| **Zoom** (in/out) | Lens focal length change | Quick emphasis (cheaper than dolly) |
|
||||
| **Whip pan** | Extremely fast pan (blurs) | Transitions, energy, surprise |
|
||||
| **Handheld / shaky cam** | Unstable, human feel | Documentary, urgency, realism |
|
||||
| **Aerial / drone** | High altitude, smooth | Landscapes, establishing shots |
|
||||
| **Slow push-in** | Gradual forward movement | Building intimacy or tension |
|
||||
| **Dolly zoom (vertigo)** | Dolly one way, zoom opposite | Disorientation, revelation |
|
||||
|
||||
## Lighting Vocabulary
|
||||
|
||||
| Term | Effect |
|
||||
|------|--------|
|
||||
| **Natural light** | Soft, realistic (morning sun, overcast, moonlight) |
|
||||
| **Golden hour** | Warm sunlight, long shadows, romantic |
|
||||
| **High-key** | Bright, even, cheerful — comedy, lifestyle |
|
||||
| **Low-key** | Dark, high contrast — thriller, drama |
|
||||
| **Rembrandt** | Triangle of light on cheek, classic portrait |
|
||||
| **Film noir** | Deep shadows, stark highlights |
|
||||
| **Volumetric** | Visible light rays through atmosphere (fog, dust) |
|
||||
| **Backlighting** | Light behind subject, silhouette effect |
|
||||
| **Side lighting** | Strong directional, dramatic shadows |
|
||||
| **Practical lights** | In-frame sources (lamps, candles, neon signs) |
|
||||
| **Rim / edge light** | Highlights subject outline, separates from background |
|
||||
|
||||
**Lighting direction modifiers**: key light, fill light, bounce, rim, spill, negative fill.
|
||||
|
||||
**Color temperature**: warm (tungsten, amber), cool (daylight, blue), mixed.
|
||||
|
||||
## Lens & Optical Effects
|
||||
|
||||
| Effect | Result |
|
||||
|--------|--------|
|
||||
| **Shallow depth of field** | Subject sharp, background bokeh |
|
||||
| **Deep focus** | Everything sharp, foreground to background |
|
||||
| **Wide-angle lens** (24-35mm) | Broader view, exaggerated perspective |
|
||||
| **Telephoto** (85mm+) | Compressed perspective, subject isolation |
|
||||
| **Anamorphic** | Stretched aspect, signature lens flares |
|
||||
| **Lens flare** | Streaks from bright light hitting lens |
|
||||
| **Rack focus** | Shift focus between subjects in-shot |
|
||||
| **Fisheye** | Ultra-wide, barrel distortion |
|
||||
|
||||
## Style & Aesthetic References
|
||||
|
||||
### Cinematic Styles
|
||||
- Film noir, period drama, thriller, modern romance
|
||||
- Documentary, arthouse, experimental film
|
||||
- Epic space opera, fantasy, horror
|
||||
- 1970s romantic drama, 90s documentary-style
|
||||
|
||||
### Animation Styles
|
||||
- Studio Ghibli / Japanese anime
|
||||
- Classic Disney, Pixar-like 3D
|
||||
- Stop-motion, claymation
|
||||
- Hand-painted 2D/3D hybrid
|
||||
- Cel-shaded, low-poly 3D
|
||||
|
||||
### Art Movements
|
||||
- Impressionistic, surrealist, Art Deco, Bauhaus
|
||||
- Watercolor, charcoal sketch, ink wash
|
||||
- Graphic novel, blueprint schematic
|
||||
|
||||
### Film Stock / Grade
|
||||
- Kodak warm grade, Fuji cool tones
|
||||
- 16mm black-and-white, 35mm photochemical contrast
|
||||
- Vintage grain overlay, halation on speculars
|
||||
- Teal-and-orange color grade
|
||||
|
||||
## Temporal Effects
|
||||
|
||||
| Effect | Use |
|
||||
|--------|-----|
|
||||
| **Slow motion** | Emphasis, beauty, impact |
|
||||
| **Time-lapse** | Passage of time, processes |
|
||||
| **Freeze-frame** | Dramatic pause |
|
||||
| **Rapid cuts** | Energy, urgency |
|
||||
| **Continuous / long take** | Immersion, tension |
|
||||
| **Fade in / fade out** | Scene transitions |
|
||||
| **Match cut** | Visual continuity between scenes |
|
||||
|
||||
## Audio Descriptions
|
||||
|
||||
Models that support audio generation (LTX-2, Sora 2, VEO 3) respond to:
|
||||
|
||||
**Ambient**: wind, rain, traffic, crowd murmur, forest birds, mechanical hum
|
||||
**Diegetic sound**: footsteps, door creaking, glass clinking, keyboard typing
|
||||
**Voice style**: whisper, calm narration, energetic announcer, gravitas
|
||||
**Music mood**: "soft piano in background", "upbeat electronic"
|
||||
|
||||
Put dialogue in quotation marks: `Character says: "Hello world."`
|
||||
|
||||
## What to Avoid
|
||||
|
||||
| Don't | Why | Do Instead |
|
||||
|-------|-----|-----------|
|
||||
| "Beautiful scene" | Too vague, no visual info | "Wet cobblestone street, warm streetlamp glow reflecting in puddles" |
|
||||
| "Person moves quickly" | No visible action | "Woman sprints three steps and vaults over the railing" |
|
||||
| "Cinematic look" | Every model already tries this | Specify: "anamorphic lens, shallow DOF, golden hour lighting" |
|
||||
| "Sad character" | Internal states aren't visible | "Tears on cheek, shoulders slumped, staring at empty chair" |
|
||||
| Readable text / logos | Models can't render text reliably | Avoid signs with text, or accept imperfect rendering |
|
||||
| Complex physics | Chaotic motion causes artifacts | Keep physics simple; dancing/walking OK, explosions risky |
|
||||
| Multiple characters talking | Multi-person dialogue breaks sync | One speaker per clip, or use reaction shots |
|
||||
| Overloaded prompts | Too many elements = incoherent | Start simple, layer complexity one element at a time |
|
||||
| Conflicting lighting | "Bright noon" + "dark shadows" | Pick one lighting setup and commit |
|
||||
|
||||
## Prompt Iteration Strategy
|
||||
|
||||
1. **Start simple** — subject + action + setting. See what the model gives you.
|
||||
2. **Add one element at a time** — camera, then lighting, then style.
|
||||
3. **If a shot misfires** — strip back. Freeze camera, simplify action, try again.
|
||||
4. **For consistency across clips** — repeat the same style/lighting/grade description.
|
||||
5. **Use seed values** — when you find a good result, save the seed for variations.
|
||||
|
||||
## Example: Generic Prompt Template
|
||||
|
||||
```
|
||||
[Shot]: Medium close-up, slight low angle
|
||||
[Camera]: Slow dolly-in
|
||||
[Subject]: A weathered fisherman in his 60s, salt-and-pepper beard,
|
||||
dark wool sweater, calloused hands gripping a rope
|
||||
[Action]: He pulls the rope hand-over-hand, muscles straining,
|
||||
then pauses and looks out to sea
|
||||
[Setting]: Wooden dock at dawn, calm grey ocean, distant fog bank,
|
||||
seagulls wheeling overhead
|
||||
[Lighting]: Soft overcast with warm break in clouds on the horizon,
|
||||
gentle rim light from the rising sun
|
||||
[Style]: Documentary cinematography, 35mm film grain,
|
||||
muted earth tones with a cold blue-grey palette
|
||||
[Audio]: Rope creaking, water lapping, distant gull cries, wind
|
||||
```
|
||||
@@ -0,0 +1,307 @@
|
||||
# Video Stitching Strategy Skill
|
||||
|
||||
## When to Use
|
||||
|
||||
Apply this skill when assembling multiple video clips into a unified output:
|
||||
sequential narrative assembly, multi-take compilation, AI-generated clip chaining
|
||||
(e.g., LTX-2 produces max ~8s per clip), or spatial compositions like side-by-side
|
||||
comparisons and picture-in-picture commentary.
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | Role |
|
||||
|------|------|
|
||||
| `video_trimmer` | Cut segments to precise in/out points, concatenate clips (`concat` operation) |
|
||||
| `video_compose` | Full composition with overlays, subtitles, audio mixing, spatial layouts |
|
||||
| `frame_sampler` | Inspect frames at stitch boundaries for visual continuity |
|
||||
| `audio_mixer` | Mix, duck, and crossfade audio tracks across stitch points |
|
||||
| `scene_detect` | Find natural scene boundaries in source footage |
|
||||
|
||||
## When to Stitch — Decision Tree
|
||||
|
||||
```
|
||||
Do you have multiple clips that need to become one video?
|
||||
├── YES: Are they sequential (play one after another)?
|
||||
│ ├── YES: Are they from the same shoot / same scene?
|
||||
│ │ ├── YES → Multi-take assembly (pick best takes, stitch)
|
||||
│ │ └── NO → Sequential narrative (match cuts, handle transitions)
|
||||
│ └── NO: Do clips need to appear simultaneously on screen?
|
||||
│ ├── YES → Spatial composition (side-by-side, PIP, stack)
|
||||
│ └── MIXED → Hybrid (sequential with spatial inserts)
|
||||
├── AI-generated clips (LTX-2, CogVideo)?
|
||||
│ └── YES → AI clip chaining (handle 8s boundaries, maintain continuity)
|
||||
└── NO → No stitching needed. Use video_trimmer for single-clip edits.
|
||||
```
|
||||
|
||||
## Stitch Strategies
|
||||
|
||||
### 1. Sequential Stitching
|
||||
|
||||
Clips play one after another in timeline order. This is the most common strategy.
|
||||
|
||||
**When:** Narrative videos, multi-section explainers, compiled takes.
|
||||
|
||||
**Process:**
|
||||
1. Order clips by narrative sequence (not filename)
|
||||
2. Trim each clip to precise in/out points via `video_trimmer` (operation: `cut`)
|
||||
3. Select transition type for each junction (see Transition Selection below)
|
||||
4. Concatenate via `video_trimmer` (operation: `concat`) for hard cuts, or `video_compose` for transitions requiring filters
|
||||
5. Verify audio continuity across all stitch points
|
||||
|
||||
**Audio continuity rules:**
|
||||
- Match audio levels across clips before stitching (normalize to -16 LUFS)
|
||||
- If background music spans multiple clips, mix it as a single track via `audio_mixer` and mux post-concat
|
||||
- Never let music cut abruptly at a stitch point — crossfade or duck instead
|
||||
|
||||
### 2. Spatial Stitching
|
||||
|
||||
Multiple clips visible simultaneously on screen.
|
||||
|
||||
**When:** Reactions, comparisons, commentary, multi-angle coverage.
|
||||
|
||||
| Layout | FFmpeg Filter | Use Case |
|
||||
|--------|---------------|----------|
|
||||
| Side-by-side (duet) | `hstack` or `xstack` | Reaction videos, before/after |
|
||||
| Vertical stack | `vstack` or `xstack` | Comparison (top vs bottom) |
|
||||
| Picture-in-picture (PIP) | `overlay=x:y` via `video_compose` | Commentary, webcam + screen |
|
||||
| Grid (2x2, 3x3) | `xstack` with layout string | Multi-angle, compilation |
|
||||
|
||||
**Spatial layout decision tree:**
|
||||
```
|
||||
What relationship do the clips have?
|
||||
├── Reaction / response → Side-by-side (duet), main clip 70% width
|
||||
├── Before / after → Side-by-side, equal 50/50 split
|
||||
├── Comparison (A vs B) → Vertical stack or side-by-side depending on aspect ratio
|
||||
├── Commentary over content → PIP, speaker in corner (20-25% frame size)
|
||||
├── Multi-angle same event → Grid layout, synced to same timecode
|
||||
└── Screen recording + face → PIP, face cam in bottom-right corner
|
||||
```
|
||||
|
||||
**PIP placement rules:**
|
||||
- Default position: bottom-right with 20px padding
|
||||
- Size: 20-25% of frame width for commentary, 30-35% for equal importance
|
||||
- Always ensure PIP does not cover critical content (subtitles, key visuals)
|
||||
- Add a 2px border or subtle shadow to separate PIP from background
|
||||
|
||||
### 3. AI Clip Chaining (LTX-2 / CogVideo)
|
||||
|
||||
AI video generators produce short clips (LTX-2: ~8 seconds max). Stitching them
|
||||
into longer sequences requires special care to maintain visual continuity.
|
||||
|
||||
**Process:**
|
||||
1. Generate clips with overlapping prompts — last frame description of clip N should match first frame description of clip N+1
|
||||
2. Use `frame_sampler` to extract the last frame of clip N and first frame of clip N+1
|
||||
3. Visually inspect the pair for continuity breaks (color shift, subject position, background change)
|
||||
4. If discontinuity is minor → use a 0.5-1.0s crossfade to smooth the junction
|
||||
5. If discontinuity is major → insert a fade-through-black (0.5s out + 0.5s in) to signal scene transition
|
||||
6. After stitching, apply a global color grade to unify the visual tone across clips
|
||||
|
||||
**AI clip chaining pitfalls:**
|
||||
- AI clips may have inconsistent FPS — normalize all clips to the same FPS before stitching
|
||||
- Color temperature often shifts between generations — apply consistent color grade post-stitch
|
||||
- Motion direction may not match — review last/first frames for jarring movement reversals
|
||||
- Audio (if any) will not be continuous — strip AI audio and use a single music/narration track
|
||||
|
||||
### 4. Hybrid Stitching
|
||||
|
||||
Sequential flow with spatial inserts at specific moments.
|
||||
|
||||
**When:** Explainer that switches to side-by-side for comparisons, tutorial that
|
||||
shows PIP during demonstrations, documentary with occasional split-screen.
|
||||
|
||||
**Process:**
|
||||
1. Plan the timeline: mark which segments are sequential and which are spatial
|
||||
2. Render each spatial segment as a standalone composed clip via `video_compose` (overlay operation)
|
||||
3. Treat the rendered spatial clips as regular clips in the sequential stitch
|
||||
4. Concatenate everything in order using the sequential stitching process
|
||||
|
||||
## Transition Selection
|
||||
|
||||
### Decision Tree
|
||||
|
||||
```
|
||||
What is the relationship between clip N and clip N+1?
|
||||
│
|
||||
├── Same scene, continuous action?
|
||||
│ └── HARD CUT (0ms)
|
||||
│
|
||||
├── Same topic, different angle or take?
|
||||
│ └── HARD CUT (0ms) — use J-cut or L-cut for audio smoothing
|
||||
│
|
||||
├── Topic change or new section?
|
||||
│ └── CROSSFADE (0.5-1.0s)
|
||||
│
|
||||
├── Time passage or mood shift?
|
||||
│ └── CROSSFADE (1.0-1.5s)
|
||||
│
|
||||
├── Major section break (intro→body, body→outro)?
|
||||
│ └── FADE THROUGH BLACK (0.5-1.0s)
|
||||
│
|
||||
├── Dialogue transition between speakers?
|
||||
│ └── L-CUT or J-CUT (audio leads or trails by 0.3-0.5s)
|
||||
│
|
||||
└── AI clip boundary (LTX-2 chain)?
|
||||
├── Continuity is good → HARD CUT or short CROSSFADE (0.3-0.5s)
|
||||
└── Continuity is broken → FADE THROUGH BLACK (0.5s)
|
||||
```
|
||||
|
||||
### Transition Reference
|
||||
|
||||
| Transition | Duration | Implementation | Best For |
|
||||
|-----------|----------|----------------|----------|
|
||||
| Hard cut | 0ms | `video_trimmer` concat (codec: copy) | Same scene, fast pace, continuation |
|
||||
| Crossfade | 0.5-1.5s | `video_compose` with `xfade` filter | Topic change, time passage, mood shift |
|
||||
| Fade through black | 0.5-1.0s each | `video_compose`: fade out → black → fade in | Major section break, intro/outro |
|
||||
| L-cut | 0.3-0.5s | Audio from clip N continues into clip N+1's video | Smooth dialogue exit, lingering emotion |
|
||||
| J-cut | 0.3-0.5s | Audio from clip N+1 starts under clip N's video | Dialogue anticipation, building tension |
|
||||
|
||||
### Transition Duration by Content Pace
|
||||
|
||||
| Pacing | Crossfade | Fade Through Black |
|
||||
|--------|-----------|-------------------|
|
||||
| Fast (short-form, < 60s) | 0.3-0.5s | 0.3-0.5s |
|
||||
| Medium (1-10 min) | 0.5-1.0s | 0.5-0.8s |
|
||||
| Slow (documentary, > 10 min) | 1.0-1.5s | 0.8-1.0s |
|
||||
|
||||
## Audio Coordination
|
||||
|
||||
### Audio at Stitch Points
|
||||
|
||||
```
|
||||
What audio exists at the stitch boundary?
|
||||
│
|
||||
├── Both clips have narration/dialogue?
|
||||
│ ├── Hard cut → Ensure no audio pop (cut at zero-crossing or apply 5ms fade)
|
||||
│ ├── Crossfade → Duck outgoing audio -6dB during overlap, bring in incoming
|
||||
│ └── L-cut/J-cut → Blend: outgoing audio fades -∞dB over 0.3-0.5s
|
||||
│
|
||||
├── Music spans the stitch?
|
||||
│ ├── Same track continues → Do not re-encode audio; use stream copy
|
||||
│ ├── Track changes → Crossfade music 1.0-2.0s centered on the cut point
|
||||
│ └── Music + narration → Duck music -12dB under narration at all times
|
||||
│
|
||||
├── One clip has audio, the other is silent?
|
||||
│ └── Add a 0.3s fade-in/fade-out to avoid abrupt silence transitions
|
||||
│
|
||||
└── No audio on either clip?
|
||||
└── No audio coordination needed. Add music/narration as a single track post-stitch.
|
||||
```
|
||||
|
||||
### Audio Level Targets
|
||||
|
||||
| Content Type | Target LUFS | Headroom |
|
||||
|-------------|-------------|----------|
|
||||
| Narration / dialogue | -16 LUFS | -1 dB true peak |
|
||||
| Background music (under narration) | -28 to -24 LUFS | -1 dB true peak |
|
||||
| Music only (no narration) | -14 LUFS | -1 dB true peak |
|
||||
| Sound effects | -20 LUFS | -1 dB true peak |
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before declaring a stitch complete, verify every item:
|
||||
|
||||
- [ ] **Resolution match:** All input clips have the same resolution (or are scaled to match before stitching)
|
||||
- [ ] **FPS match:** All input clips share the same frame rate (or are conformed with `fps` filter)
|
||||
- [ ] **Aspect ratio consistency:** No mixed 16:9 / 9:16 / 4:3 unless intentional spatial layout
|
||||
- [ ] **Color consistency:** No visible color temperature or exposure jumps at stitch boundaries
|
||||
- [ ] **Audio level consistency:** All clips normalized to target LUFS before stitching
|
||||
- [ ] **No audio pops or clicks:** Stitch points have micro-fades or are at zero-crossings
|
||||
- [ ] **Transition appropriateness:** Transition type matches the content relationship (see decision tree)
|
||||
- [ ] **Total duration check:** Final output duration matches expected sum (accounting for transition overlaps)
|
||||
- [ ] **Codec consistency:** All clips use the same codec to allow stream copy; re-encode only if necessary
|
||||
- [ ] **Playback test:** Scrub through every stitch point in the output and confirm smooth playback
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Codec Mismatch Causing Full Re-encode
|
||||
|
||||
**Problem:** Mixing clips encoded with different codecs (e.g., H.264 + H.265) or different
|
||||
encoding parameters forces FFmpeg to re-encode everything during concat.
|
||||
|
||||
**Solution:** Before stitching, probe all clips with `ffprobe`. If codecs differ, re-encode
|
||||
the minority clips to match the majority codec. This is faster than re-encoding everything.
|
||||
|
||||
```
|
||||
Check: ffprobe -v error -select_streams v:0 -show_entries stream=codec_name,width,height,r_frame_rate -of csv=p=0 input.mp4
|
||||
```
|
||||
|
||||
### Audio Drift in Long Stitches
|
||||
|
||||
**Problem:** When concatenating many short clips (10+), tiny timing mismatches accumulate,
|
||||
causing audio to drift out of sync by the end.
|
||||
|
||||
**Solution:**
|
||||
1. Re-encode each clip with constant frame rate before concatenation (`-vsync cfr`)
|
||||
2. If using a separate audio track, align it to the final video's duration post-stitch
|
||||
3. For AI clip chains, use `-async 1` to resync audio on concatenation
|
||||
|
||||
### Aspect Ratio Mixing
|
||||
|
||||
**Problem:** Stitching a 16:9 clip with a 9:16 clip creates letterboxing or stretching.
|
||||
|
||||
**Solution:** Decide on a target aspect ratio up front. Pad non-conforming clips with black
|
||||
bars (`pad` filter) or crop them (`crop` filter) — never stretch.
|
||||
|
||||
### Variable Frame Rate (VFR) Sources
|
||||
|
||||
**Problem:** Screen recordings and phone footage often use VFR, which causes
|
||||
desync and stuttering when stitched with CFR content.
|
||||
|
||||
**Solution:** Convert VFR sources to CFR before stitching:
|
||||
`ffmpeg -i vfr_input.mp4 -vsync cfr -r 30 cfr_output.mp4`
|
||||
|
||||
### Concatenation with Stream Copy Fails
|
||||
|
||||
**Problem:** `video_trimmer` concat with `codec: copy` fails or produces glitchy output
|
||||
when clips have different GOP structures or encoding parameters.
|
||||
|
||||
**Solution:** If stream copy fails, fall back to re-encoding with consistent parameters:
|
||||
`-c:v libx264 -crf 18 -preset medium -c:a aac -b:a 192k`
|
||||
Use CRF 18 (near-lossless) to avoid quality loss from the re-encode.
|
||||
|
||||
## Stitch Planning Template
|
||||
|
||||
When planning a stitch, produce this structure as part of `edit_decisions`:
|
||||
|
||||
```yaml
|
||||
stitch_plan:
|
||||
strategy: sequential | spatial | hybrid | ai_chain
|
||||
target_resolution: "1920x1080"
|
||||
target_fps: 30
|
||||
target_codec: libx264
|
||||
|
||||
clips:
|
||||
- id: clip_01
|
||||
source: "assets/intro.mp4"
|
||||
in_seconds: 0.0
|
||||
out_seconds: 5.0
|
||||
transition_out: crossfade
|
||||
transition_duration: 0.8
|
||||
|
||||
- id: clip_02
|
||||
source: "assets/section_1.mp4"
|
||||
in_seconds: 0.0
|
||||
out_seconds: 8.0
|
||||
transition_out: hard_cut
|
||||
|
||||
- id: clip_03
|
||||
source: "assets/section_2.mp4"
|
||||
in_seconds: 0.0
|
||||
out_seconds: 8.0
|
||||
transition_out: fade_black
|
||||
transition_duration: 0.5
|
||||
|
||||
audio:
|
||||
narration: "assets/narration_full.wav"
|
||||
music: "assets/bg_music.mp3"
|
||||
music_volume: -24 # LUFS
|
||||
ducking: true
|
||||
|
||||
spatial_inserts: # Only for hybrid strategy
|
||||
- at_clip: clip_02
|
||||
at_seconds: 3.0
|
||||
layout: pip
|
||||
overlay_source: "assets/webcam.mp4"
|
||||
position: bottom_right
|
||||
size_percent: 25
|
||||
```
|
||||
@@ -0,0 +1,124 @@
|
||||
# Video Understanding Usage for OpenMontage
|
||||
|
||||
> Sources: OpenMontage video_understand tool implementation, CLIP/BLIP2/LLaVA model
|
||||
> documentation, OpenCV image quality metrics
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
DEFAULT MODE: describe — generates captions for frames
|
||||
FOR REVIEW: quality — assesses blur, brightness, contrast
|
||||
FOR Q&A: qa mode with a query — "Is the speaker visible?" "Is the text readable?"
|
||||
DEFAULT MODEL: clip (fastest, good for classification)
|
||||
FOR DETAIL: blip2 or llava (slower, richer descriptions)
|
||||
MAX FRAMES: 5 default for video — sample strategically, not exhaustively
|
||||
```
|
||||
|
||||
## When to Use video_understand
|
||||
|
||||
- **Visual QA during review** — check rendered output quality before delivering
|
||||
- **Footage analysis** — understand what's in user-provided footage before planning
|
||||
- **Highlight extraction** — identify the most visually interesting frames
|
||||
- **Quality gating** — programmatic check for blur, exposure, scene coherence
|
||||
- **Scene classification** — categorize footage by content type
|
||||
- **Asset validation** — verify generated images match the intended scene description
|
||||
|
||||
## Mode Selection
|
||||
|
||||
| Mode | What It Does | When to Use |
|
||||
|------|-------------|-------------|
|
||||
| `describe` | Generates a text description of the frame | Understanding footage content, logging |
|
||||
| `qa` | Answers a specific question about the frame | Targeted checks ("Is text readable?", "Is face visible?") |
|
||||
| `quality` | Measures blur, brightness, contrast numerically | Automated quality gating, comparing takes |
|
||||
| `classify` | Categorizes the scene type | Sorting footage, pipeline routing |
|
||||
|
||||
### Quality Mode Metrics
|
||||
|
||||
| Metric | What It Measures | Bad | Good |
|
||||
|--------|-----------------|-----|------|
|
||||
| `blur_score` | Laplacian variance | Below 100 = blurry | Above 500 = sharp |
|
||||
| `brightness` | Mean pixel value (0-255) | Below 50 = too dark, above 200 = overexposed | 50-200 |
|
||||
| `contrast` | Pixel standard deviation | Below 30 = flat/washed out | Above 80 = good contrast |
|
||||
|
||||
## Model Selection
|
||||
|
||||
| Model | Speed | Capabilities | Best For |
|
||||
|-------|-------|-------------|----------|
|
||||
| `clip` | Fast | Classification, similarity matching | Quick scene categorization, batch processing |
|
||||
| `blip2` | Medium | Detailed captions, visual QA | Understanding complex scenes, answering questions |
|
||||
| `llava` | Slow | Most detailed understanding, reasoning | Deep analysis, subjective quality assessment |
|
||||
|
||||
### Model Selection Rules
|
||||
|
||||
- Use `clip` for batch operations and classification tasks
|
||||
- Use `blip2` for describe and qa modes when detail matters
|
||||
- Use `llava` only when you need the most thorough understanding
|
||||
|
||||
## Frame Selection for Video
|
||||
|
||||
- Default samples `max_frames` (5) evenly across the video
|
||||
- Use `frame_indices` to target specific frames (e.g., check quality at specific timestamps)
|
||||
- For quality review, sample the first frame, middle frame, and last frame minimum
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### 1. Pre-Edit Footage Review
|
||||
|
||||
```
|
||||
video_understand (describe, 10 frames) → inform scene_plan
|
||||
```
|
||||
|
||||
Analyze user-provided footage before planning cuts or edits. Use `blip2` for detailed descriptions that inform the scene plan.
|
||||
|
||||
### 2. Post-Render Quality Gate
|
||||
|
||||
```
|
||||
video_understand (quality) → pass/fail → re-render if needed
|
||||
```
|
||||
|
||||
Run after composing the final video. Fail if any frame has blur_score < 100, brightness outside 50-200, or contrast < 30.
|
||||
|
||||
### 3. Highlight Selection
|
||||
|
||||
```
|
||||
video_understand (describe, 20 frames) → rank by visual interest → select clips
|
||||
```
|
||||
|
||||
Sample many frames, describe each, then select the most visually compelling segments for a montage or trailer.
|
||||
|
||||
### 4. Asset Validation
|
||||
|
||||
```
|
||||
video_understand (qa, "Does this match: [scene description]?") → confirm or regenerate
|
||||
```
|
||||
|
||||
After generating an image or video clip, verify it matches the intended scene description before proceeding.
|
||||
|
||||
### 5. Talking-Head Analysis
|
||||
|
||||
```
|
||||
video_understand (qa, "Is the speaker's face clearly visible?") → face_enhance if needed
|
||||
```
|
||||
|
||||
Check face visibility and framing before applying lip-sync or face restoration tools.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
- Descriptions accurately match what's in the frame
|
||||
- Quality scores correlate with visual inspection (manually spot-check)
|
||||
- QA answers are consistent across similar frames
|
||||
- Classification categories are stable across adjacent frames
|
||||
- No false positives in quality gating (good frames passing, bad frames failing)
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
When using the `video_understand` tool:
|
||||
|
||||
1. **Use `quality` mode as a post-render gate in the compose stage** — reject outputs below quality thresholds
|
||||
2. **Use `describe` mode to analyze user-provided footage** at the start of the talking-head pipeline
|
||||
3. **For batch quality checks, use `clip` model** (fastest) — switch to `blip2` only for detailed review
|
||||
4. **Sample at least 3 frames for quality assessment** — beginning, middle, end
|
||||
5. **Quality thresholds for passing:** blur_score > 100, brightness 50-200, contrast > 30
|
||||
6. **Use `qa` mode to validate generated assets:** "Does this image show [expected content]?"
|
||||
7. **In the review stage**, combine video_understand quality data with the reviewer skill's rubric
|
||||
8. **Do NOT run video_understand on every frame of a long video** — sample strategically
|
||||
Reference in New Issue
Block a user