Harden talking-head pipeline: Watch & Propose creative overlays, fix Round 1 gaps

Scene-director rewrite: agent now watches footage, understands content, and proposes
creative overlays (charts, stats, key terms, comparisons) before building anything.
Presents enhancement plan to user for approval before proceeding.

Compose-director fixes from Round 1 verification:
- eye_enhance: now explicitly required, not silently skippable
- Caption positioning: explicit MarginV=160 for 9:16, never center
- Final encode: mandatory with target file size table
- ASR corrections: new Step 2b to scan transcript and build corrections dict
- Overlay compositing: new Step 3b for burning approved graphics onto footage

Asset-director rewrite: generates Remotion overlay assets (callouts, stat cards,
charts, comparisons) from scene plan. Includes overlay type → Remotion cut mapping
table and dark theme requirements.

Bug fixes found during subagent verification:
- remotion_caption_burn.py: fix run_command API, add npx.cmd for Windows
- visual_qa.py: fix run_command API (3 places), Windows /dev/null → NUL
This commit is contained in:
calesthio
2026-04-01 11:03:15 -07:00
parent aee9379810
commit 942244ca54
6 changed files with 327 additions and 50 deletions
+1
View File
@@ -64,3 +64,4 @@ remotion-composer/public/*
# Ignore test/scratch compositions in demo-props (keep only curated demos)
remotion-composer/public/demo-props/test-*
remotion-composer/public/demo-props/talking-head-*
remotion-composer/public/demo-props/caption-burn-*
+103 -11
View File
@@ -2,7 +2,7 @@
## When to Use
You have a scene plan and script. Your job is to generate the supporting assets for a talking-head video: subtitles, extracted audio, and any overlay graphics.
You have a scene plan and script. Your job is to generate the supporting assets for a talking-head video: subtitles, extracted audio, overlay graphics (charts, text cards, stat reveals), and any supplementary visuals.
## Prerequisites
@@ -10,7 +10,9 @@ You have a scene plan and script. Your job is to generate the supporting assets
|-------|----------|---------|
| Schema | `schemas/artifacts/asset_manifest.schema.json` | Artifact validation |
| Prior artifacts | Scene plan, Script | What assets to create |
| Tools | `subtitle_gen`, `audio_mixer`, `image_selector` (optional) | Asset generation |
| Tools | `subtitle_gen`, `audio_mixer` | Subtitle and audio generation |
| Tools | `image_selector` (optional) | Stock images for overlays |
| Tools | `pixabay_music` (optional) | Royalty-free background music |
## Process
@@ -20,31 +22,121 @@ Use the transcription data from the script stage to create:
- SRT or ASS subtitle file with word-level timing
- Style subtitles per the playbook (font, size, color, position)
If the scene plan includes a `corrections` dict, pass it to `subtitle_gen`:
```
subtitle_gen.execute({
"segments": <transcript_segments>,
"corrections": {"cloud": "Claude"},
"max_words_per_line": 5,
"output_path": "<project>/assets/subtitles/subtitles.srt"
})
```
### Step 2: Extract and Process Audio
- Extract audio track from raw footage
- Apply noise reduction if needed (via `audio_mixer`)
- Normalize audio levels
### Step 3: Generate Overlays (Optional)
### Step 3: Source Background Music
If the scene plan includes overlay scenes:
- Generate text card images
- Generate lower third graphics
- Create any B-roll placeholders
If the scene plan includes background music:
### Step 4: Build Asset Manifest
1. **Check local pixabay music library** — look for downloaded MP3s matching the mood
2. **Use `pixabay_music` tool** — search by mood/genre keywords from the scene plan
3. **Run `audio_energy` analysis** on the selected track to find optimal start offset (skip quiet intros)
Document all generated assets with paths, types, and tool references.
Record the music path, offset, and whether looping is needed in the asset manifest.
### Step 5: Self-Evaluate
### Step 4: Generate Overlay Assets
If the scene plan includes overlay scenes (from the scene-director's Watch & Propose step), generate the assets for each.
**For Remotion-rendered overlays** (charts, comparisons, KPI grids, stat cards):
Create a composition JSON snippet for each overlay. These will be rendered by the compose-director. Each overlay needs:
```json
{
"overlay_id": "overlay_1",
"remotion_cut": {
"id": "term-agentic-ai",
"type": "callout",
"text": "Agentic AI: software that acts autonomously toward goals",
"in_seconds": 0,
"out_seconds": 4,
"backgroundColor": "#0F172A",
"accentColor": "#22D3EE",
"icon": "💡"
},
"overlay_timestamp": 22.0,
"position": "lower_third"
}
```
**Overlay type → Remotion cut mapping:**
| Scene Plan Overlay | Remotion `type` | Required Props |
|-------------------|-----------------|----------------|
| Key term definition | `callout` | `text`, `icon` (optional) |
| Statistic/number | `stat_card` | `stat` (the number), `text` (label) |
| Comparison | `comparison` | `leftLabel`, `rightLabel`, `leftValue`, `rightValue` |
| Data chart | `bar_chart` | `chartData` (array of `{label, value}`) |
| Pie chart | `pie_chart` | `chartData` (array of `{label, value}`) |
| Line chart | `line_chart` | `chartSeries` (array of `{name, data: number[]}`) |
| KPI dashboard | `kpi_grid` | `chartData` (array of `{label, value}`) — keep numbers small with suffix (e.g. "2.4M") |
| Progress indicator | `progress_bar` | `progress` (0-100), `text` |
| Section title | `hero_title` | `text`, `subtitle` (optional) |
| Callout/quote | `callout` | `text`, `icon` |
| Lower third | `text_card` | `text` |
**Dark theme for all overlays** — use dark backgrounds (`#0F172A`, `#1E293B`) with light text. This ensures overlays are legible when composited on top of talking-head footage.
**For simple text overlays** (if Remotion is overkill):
Generate PNG images using FFmpeg or PIL, stored at `<project>/assets/overlays/overlay_<id>.png`.
### Step 5: Build Asset Manifest
Document all generated assets with paths, types, and tool references:
```json
{
"subtitles": {
"path": "assets/subtitles/subtitles.srt",
"format": "srt",
"word_count": 208
},
"music": {
"path": "assets/audio/bg_music.mp3",
"offset_seconds": 3.5,
"needs_loop": true
},
"overlays": [
{
"overlay_id": "overlay_1",
"type": "callout",
"timestamp": 22.0,
"duration": 4.0,
"remotion_cut": { ... },
"position": "lower_third"
}
],
"transcript_segments": "assets/audio/transcript.json"
}
```
### Step 6: Self-Evaluate
| Criterion | Question |
|-----------|----------|
| **Subtitles** | Do subtitles exist and match speech timing? |
| **Audio** | Is audio clean and normalized? |
| **Music** | Was audio_energy run on the music to find optimal offset? |
| **Overlays** | Does every overlay from the scene plan have a generated asset? |
| **Overlay content** | Is the data in overlays accurate to what the speaker actually says? |
| **Files** | Do all asset paths point to existing files? |
### Step 6: Submit
### Step 7: Submit
Validate the asset_manifest against the schema and persist via checkpoint.
@@ -17,15 +17,14 @@ You have edit decisions and an asset manifest. Your job is to render the final t
### Step 1: Run Enhancement Chain
Apply video enhancements in order:
1. **Face enhancement** (if `face_enhance` tool available) — apply `talking_head_standard` preset
2. **Eye enhancement** (if `eye_enhance` tool available) — under-eye dark circle removal + eye brightening
3. **Color grading** (if `color_grade` tool available) — apply a profile
4. **Audio enhancement** (if `audio_enhance` tool available) — noise reduction, normalization
Apply video enhancements in this exact order. **Attempt every step** if the tool is available — do not skip steps without a reason.
Each step is optional — check tool availability first.
1. **Face enhancement** — apply `talking_head_standard` preset
2. **Eye enhancement** — under-eye dark circle removal + eye brightening
3. **Color grading** — apply a profile
4. **Audio enhancement** — noise reduction, normalization
**Eye enhancement**removes under-eye dark circles and brightens eyes using MediaPipe Face Mesh landmark detection:
**Eye enhancement**always attempt this after face_enhance. It makes a visible difference on webcam/phone footage:
```
eye_enhance.execute({
"input_path": "<face_enhanced_video>",
@@ -35,7 +34,7 @@ eye_enhance.execute({
"eye_brighten_intensity": 0.3,
})
```
**Important:** Keep intensities low (0.2-0.5). Over-processing makes eyes look unnatural. Always compare before/after.
**Important:** Keep intensities low (0.2-0.5). Over-processing makes eyes look unnatural. If the tool fails (e.g. MediaPipe not installed), log the fallback and continue with the face_enhanced video.
### Step 1b: Speed Adjustment (if requested)
@@ -86,6 +85,25 @@ The tool automatically runs face detection and keeps the speaker centered. If Me
**Important:** Run auto_reframe AFTER face_enhance and color_grade but BEFORE burning subtitles. Subtitles need to be positioned for the final aspect ratio.
### Step 2b: Build ASR Corrections Dictionary
Before burning captions, scan the transcript for likely ASR misrecognitions. Common issues:
- Product/brand names: "cloud" → "Claude", "co-pilot" → "Copilot", "remotion" → "Remotion"
- Technical terms: "pythonic" misheard as "pathonic", "API" as "a pie"
- Speaker's name or company name
- Domain-specific jargon
Build a corrections dict:
```python
corrections = {
"cloud": "Claude",
"co pilot": "Copilot",
"open montage": "OpenMontage",
}
```
Pass this dict to both `subtitle_gen` (if generating SRT) and `remotion_caption_burn` (if using Remotion captions). Even if you find zero corrections needed, explicitly pass an empty dict `{}` to confirm you checked.
### Step 3: Burn Subtitles
**Preferred: Remotion captions** (if `remotion_caption_burn` tool available):
@@ -106,11 +124,40 @@ Remotion renders animated word-by-word captions at the bottom of the frame with
Use `video_compose` with `burn_subtitles` operation:
- Input: reframed video (or enhanced video if no reframe needed)
- Subtitle file from asset manifest
- Style from playbook
- For vertical (9:16) output: position subtitles in the lower 20% of frame with `MarginV=100`
- **Never** position subtitles in the center of the frame — they will occlude the face
### Step 3b: Build Showcase Cards (if multi-clip reel)
**CRITICAL: Caption positioning for 9:16 vertical video.**
Captions MUST be in the lower 20% of the frame. On a 1920-high frame, that means `MarginV=160` or higher. The default FFmpeg subtitle position is center — this WILL occlude the face. You MUST override it.
FFmpeg subtitle style string for vertical talking-head:
```
"FontName=Arial,FontSize=22,Bold=1,PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,Outline=2,Shadow=0,MarginV=160,Alignment=2"
```
**Never** use the default subtitle position. **Never** position subtitles in the center or upper half of the frame. If you see captions on the face during visual QA, the video must be re-rendered with corrected positioning.
### Step 3b: Burn Overlay Graphics (if scene plan includes overlays)
If the scene plan includes overlay scenes (text_cards, stat_cards, charts, comparisons, callouts), render them onto the video.
**How overlay compositing works:**
1. Each overlay is a short Remotion composition (3-5 seconds) rendered as a transparent video clip or composited directly
2. Use `video_compose` with `picture_in_picture` or `overlay` operation to place each overlay at the correct timestamp
3. Respect the overlay's `position` field from the scene plan:
- `lower_third` → bottom 30% of frame
- `upper_third` → top 30% of frame
- `side_panel` → left or right 40%
- `full_overlay` → centered, brief (1-2s)
**For Remotion-based overlays:** Create a composition JSON with the overlay cuts, render to a transparent clip, then composite onto the talking-head video using FFmpeg.
**For simple text overlays:** Use FFmpeg's drawtext filter directly:
```
ffmpeg -i captioned.mp4 -vf "drawtext=text='Key Term':fontsize=48:fontcolor=white:borderw=3:bordercolor=black:x=(w-text_w)/2:y=h*0.75:enable='between(t,22,26)'" -c:a copy output.mp4
```
**Important:** Time each overlay to match the scene plan timestamps. After speed adjustment, recalculate overlay timestamps: `adjusted_time = original_time / speed_factor`.
### Step 3c: Build Showcase Cards (if multi-clip reel)
If the output is a reel with showcase clips, use `showcase_card` for each:
```
@@ -166,12 +213,35 @@ audio_mixer.execute({
- Apply ducking if music is present
- Normalize final levels
### Step 6: Final Encode
### Step 6: Final Encode — MANDATORY
**Do not skip this step.** Without a final encode, the output will be oversized and may not play correctly on the target platform.
Use `video_compose` with `encode` operation:
- Apply target media profile (youtube_landscape, tiktok, instagram_reels, etc.)
- Two-pass encoding for quality
**Target file sizes:**
| Platform | Max Duration | Target Size |
|----------|-------------|-------------|
| Instagram Reels | 90s | < 50 MB |
| TikTok | 10 min | < 100 MB |
| YouTube Shorts | 60s | < 40 MB |
| YouTube | unlimited | < 25 MB/min |
If the output exceeds the target, re-encode with a lower bitrate. A 66-second Instagram Reel at 76 MB is unacceptable — it should be under 30 MB.
```
video_compose.execute({
"operation": "encode",
"input_path": "<mixed_video>",
"output_path": "<project>/renders/final.mp4",
"media_profile": "instagram_reels",
"video_bitrate": "4M",
"audio_bitrate": "192k",
})
```
### Step 7: Visual QA
Use `visual_qa` to verify the output before declaring success:
+127 -19
View File
@@ -2,7 +2,9 @@
## When to Use
You have a script (from transcription) and raw footage. Your job is to create a scene plan — mostly simple since talking-head footage is a single continuous shot, but you still need to plan where overlays, text cards, and B-roll might appear.
You have a script (from transcription) and raw footage. Your job is to **watch the footage, understand the content, and propose a creative enhancement plan** — then build a scene plan that transforms raw talking-head footage into an engaging, visually rich video.
You are not just a processor. You are a creative director. Your job is to figure out what the speaker is saying and propose visual enhancements that make the content more engaging and easier to understand.
## Prerequisites
@@ -16,7 +18,87 @@ You have a script (from transcription) and raw footage. Your job is to create a
## Process
### Step 1: Analyze Footage (if tools available)
### Step 1: Watch & Listen — Understand the Content
**This is the most important step. Do not skip it.**
Read the full transcript carefully. Understand:
- What is the speaker's **main topic**?
- What are the **key concepts** they explain?
- Where do they use **numbers, statistics, or data**?
- Where do they **compare things** (A vs B, before/after, old vs new)?
- Where do they **list items** (3 tips, 5 steps, etc.)?
- Where do they introduce **technical terms** or jargon?
- Where are the **section transitions** (topic changes)?
- What is the **emotional arc** (excitement, serious, humorous)?
If `frame_sampler` is available, extract 5-8 representative frames to see the speaker's setup, background, lighting, and gestures.
### Step 2: Propose Creative Overlays
Based on your content analysis, propose **on-screen graphics** that will appear alongside the speaker at key moments. These are Remotion components that get composited on top of or next to the talking-head footage.
**Available overlay types:**
| Overlay Type | Remotion Component | Best For |
|-------------|-------------------|----------|
| **Key term definition** | `text_card` | When the speaker introduces a technical term — show the term + short definition |
| **Statistic/number** | `stat_card` | When the speaker mentions a number or percentage — animate it on screen |
| **Comparison** | `comparison` | When the speaker compares two things (A vs B) — show side-by-side |
| **Data chart** | `bar_chart` / `pie_chart` / `line_chart` | When the speaker references data or rankings |
| **KPI dashboard** | `kpi_grid` | When multiple numbers are mentioned together |
| **Progress indicator** | `progress_bar` | When the speaker describes a process or percentage |
| **Section title** | `hero_title` | At major topic transitions — show the new section title |
| **Callout/quote** | `callout` | When the speaker makes a key point worth emphasizing |
| **Lower third** | `text_card` | Speaker identification at the start |
**Overlay planning rules:**
- **Don't over-overlay.** 3-6 overlays per minute of final video is the sweet spot. More than that is distracting.
- **Time overlays to speech.** Each overlay should appear when the speaker says the relevant words, not before or after.
- **Keep overlays brief.** 3-5 seconds each. They support the speaker, not compete with them.
- **Vary the types.** Don't use 5 text_cards in a row — mix in charts, comparisons, callouts.
- **Use overlays at natural pauses.** When the speaker pauses for emphasis, that's a good overlay moment.
- **Match the vibe.** Professional talk = clean stat cards and charts. Casual talk = callouts and bold key terms.
### Step 3: Present Your Plan to the User
**MANDATORY: Present your enhancement plan before proceeding.**
Format your proposal clearly:
```
## Enhancement Plan for [Video Title/Topic]
**Content Summary:** [1-2 sentences about what the speaker covers]
**Proposed Overlays:**
| Time | Type | Content | Why |
|------|------|---------|-----|
| 0:05 | lower_third | "Speaker Name — Title" | Speaker intro |
| 0:22 | text_card | "Agentic AI: software that acts autonomously" | Key term definition |
| 0:45 | comparison | Traditional Software vs Agentic Software | Speaker is comparing the two |
| 1:10 | stat_card | "73% of developers..." | Statistic mentioned |
| 1:35 | bar_chart | [Framework popularity data] | Speaker references rankings |
| 2:00 | callout | "The key insight is..." | Speaker's main takeaway |
**Enhancement Chain:**
- Silence removal: ~X seconds of dead air detected
- Speed: 1.25x (user requested)
- Face + eye enhancement
- Animated captions (bottom of frame)
- Background music: [recommendation]
**Estimated final duration:** ~Xs (from Xs raw)
```
Wait for user approval before proceeding. The user may:
- Approve as-is
- Add/remove overlays
- Change overlay content
- Adjust the enhancement plan
### Step 4: Analyze Footage (if tools available)
**Face tracking** — If `face_tracker` is available, run it on the raw footage:
```
@@ -29,6 +111,7 @@ This outputs per-frame face bounding boxes. Use this data to:
- Decide if reframing is needed (e.g. speaker is off-center for vertical crop)
- Identify sections where the speaker moves significantly (needs dynamic crop)
- Note face position for auto_reframe in the compose stage
- **Determine overlay safe zones** — where to place graphics without occluding the face
**Silence detection** — If `silence_cutter` is available, run in `mark` mode:
```
@@ -43,20 +126,38 @@ This outputs silence/speech segment timestamps. Use this to:
- Plan which segments should be jump-cut or sped up
- Identify dead air, false starts, and long pauses
- Estimate the final video duration after cuts
- Present the user with a summary: "Found X seconds of silence across Y segments — recommend removing?"
### Step 2: Plan Base Scenes
### Step 5: Plan Base Scenes
For talking-head, the base is simple: one scene per script section, all type `talking_head`. The raw footage IS the scene.
### Step 3: Plan Enhancement Scenes
### Step 6: Build Overlay Scenes
Based on script enhancement cues, plan overlay scenes:
- Text cards for key terms or statistics
- Lower thirds for speaker identification
- B-roll suggestions for topic illustrations
For each approved overlay from Step 3, create an overlay scene entry:
```json
{
"id": "overlay_1",
"type": "overlay",
"overlay_type": "text_card",
"start_seconds": 22.0,
"duration_seconds": 4.0,
"content": {
"text": "Agentic AI",
"subtext": "Software that acts autonomously toward goals",
"backgroundColor": "#0F172A",
"accentColor": "#22D3EE"
},
"position": "lower_third"
}
```
### Step 4: Plan Reframing & Cuts
**Overlay position options:**
- `lower_third` — bottom 30% of frame (safest, doesn't occlude face)
- `upper_third` — top 30% of frame (good for titles)
- `side_panel` — left or right 40% (for charts/comparisons, speaker shifts to other side)
- `full_overlay` — brief full-screen graphic (1-2s max, for dramatic emphasis)
### Step 7: Plan Reframing & Cuts
If the target platform requires a different aspect ratio (e.g. Instagram Reels = 9:16):
- Note `auto_reframe` should be applied in the compose stage
@@ -67,21 +168,28 @@ If silence detection found segments to cut:
- Record the recommended cut mode (`remove` or `speed_up`) in the scene plan
- Note padding preferences (default 0.08s to avoid clipping words)
### Step 5: Build Scene Plan
### Step 8: Build Scene Plan
Create a scene per section with:
- Type: `talking_head` (primary)
- Timing from script sections
- Required assets: subtitle file, any overlay images
Assemble the full scene plan with:
- Base scenes (one per script section, type `talking_head`)
- Overlay scenes (from Step 6, type `overlay`)
- Enhancement chain decisions (silence cut mode, speed factor, reframe target)
- Music recommendation
- Estimated final duration
### Step 6: Self-Evaluate
### Step 9: Self-Evaluate
| Criterion | Question |
|-----------|----------|
| **Content understanding** | Did I actually understand what the speaker is talking about? |
| **Overlay relevance** | Does every overlay directly relate to what's being said at that moment? |
| **Overlay density** | Am I in the 3-6 per minute range? Not too sparse, not too cluttered? |
| **Overlay variety** | Am I using different types, not just text_cards? |
| **Timing** | Are overlays timed to the speaker's words, not arbitrary moments? |
| **Coverage** | Every script section has a scene? |
| **Enhancement** | Are overlay opportunities identified? |
| **Feasibility** | Can all required assets be generated? |
| **Feasibility** | Can all overlays be rendered with available Remotion components? |
| **User approved** | Did the user approve the enhancement plan? |
### Step 7: Submit
### Step 10: Submit
Validate the scene_plan against the schema and persist via checkpoint.
+7 -4
View File
@@ -215,7 +215,8 @@ class VisualQA(BaseTool):
input_path,
]
import json
probe_out = self.run_command(cmd, capture=True)
probe_result = self.run_command(cmd)
probe_out = probe_result.stdout
probe_data = json.loads(probe_out)
# Extract key info
@@ -302,10 +303,11 @@ class VisualQA(BaseTool):
"-t", "3",
"-i", input_path,
"-vn", "-af", "volumedetect",
"-f", "null", "/dev/null",
"-f", "null", "NUL" if __import__("sys").platform == "win32" else "/dev/null",
]
try:
output = self.run_command(cmd, capture=True, stderr=True)
cmd_result = self.run_command(cmd)
output = cmd_result.stderr # volumedetect outputs to stderr
mean_vol = None
max_vol = None
for line in output.split("\n"):
@@ -340,4 +342,5 @@ class VisualQA(BaseTool):
"-of", "csv=p=0",
path,
]
return float(self.run_command(cmd, capture=True).strip().split("\n")[0])
dur_result = self.run_command(cmd)
return float(dur_result.stdout.strip().split("\n")[0])
+6 -3
View File
@@ -257,7 +257,8 @@ class RemotionCaptionBurn(BaseTool):
"-of", "csv=p=0",
input_path,
]
dur_out = self.run_command(dur_cmd, capture=True)
dur_result = self.run_command(dur_cmd)
dur_out = dur_result.stdout
duration_s = float(dur_out.strip().split("\n")[0])
total_frames = math.ceil(duration_s * 30)
@@ -281,9 +282,11 @@ class RemotionCaptionBurn(BaseTool):
props_file = props_dir / f"caption-burn-{Path(input_path).stem}.json"
props_file.write_text(json.dumps(props, indent=2), encoding="utf-8")
# Render
# Render (use npx.cmd on Windows for subprocess compatibility)
import sys
npx_bin = "npx.cmd" if sys.platform == "win32" else "npx"
render_cmd = [
"npx", "remotion", "render",
npx_bin, "remotion", "render",
"src/index.tsx", "TalkingHead",
f"--props={props_file.relative_to(root)}",
"--width=1080", "--height=1920", "--fps=30",