Harden talking-head pipeline: Watch & Propose creative overlays, fix Round 1 gaps
Scene-director rewrite: agent now watches footage, understands content, and proposes creative overlays (charts, stats, key terms, comparisons) before building anything. Presents enhancement plan to user for approval before proceeding. Compose-director fixes from Round 1 verification: - eye_enhance: now explicitly required, not silently skippable - Caption positioning: explicit MarginV=160 for 9:16, never center - Final encode: mandatory with target file size table - ASR corrections: new Step 2b to scan transcript and build corrections dict - Overlay compositing: new Step 3b for burning approved graphics onto footage Asset-director rewrite: generates Remotion overlay assets (callouts, stat cards, charts, comparisons) from scene plan. Includes overlay type → Remotion cut mapping table and dark theme requirements. Bug fixes found during subagent verification: - remotion_caption_burn.py: fix run_command API, add npx.cmd for Windows - visual_qa.py: fix run_command API (3 places), Windows /dev/null → NUL
This commit is contained in:
@@ -64,3 +64,4 @@ remotion-composer/public/*
|
||||
# Ignore test/scratch compositions in demo-props (keep only curated demos)
|
||||
remotion-composer/public/demo-props/test-*
|
||||
remotion-composer/public/demo-props/talking-head-*
|
||||
remotion-composer/public/demo-props/caption-burn-*
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## When to Use
|
||||
|
||||
You have a scene plan and script. Your job is to generate the supporting assets for a talking-head video: subtitles, extracted audio, and any overlay graphics.
|
||||
You have a scene plan and script. Your job is to generate the supporting assets for a talking-head video: subtitles, extracted audio, overlay graphics (charts, text cards, stat reveals), and any supplementary visuals.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -10,7 +10,9 @@ You have a scene plan and script. Your job is to generate the supporting assets
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/asset_manifest.schema.json` | Artifact validation |
|
||||
| Prior artifacts | Scene plan, Script | What assets to create |
|
||||
| Tools | `subtitle_gen`, `audio_mixer`, `image_selector` (optional) | Asset generation |
|
||||
| Tools | `subtitle_gen`, `audio_mixer` | Subtitle and audio generation |
|
||||
| Tools | `image_selector` (optional) | Stock images for overlays |
|
||||
| Tools | `pixabay_music` (optional) | Royalty-free background music |
|
||||
|
||||
## Process
|
||||
|
||||
@@ -20,31 +22,121 @@ Use the transcription data from the script stage to create:
|
||||
- SRT or ASS subtitle file with word-level timing
|
||||
- Style subtitles per the playbook (font, size, color, position)
|
||||
|
||||
If the scene plan includes a `corrections` dict, pass it to `subtitle_gen`:
|
||||
```
|
||||
subtitle_gen.execute({
|
||||
"segments": <transcript_segments>,
|
||||
"corrections": {"cloud": "Claude"},
|
||||
"max_words_per_line": 5,
|
||||
"output_path": "<project>/assets/subtitles/subtitles.srt"
|
||||
})
|
||||
```
|
||||
|
||||
### Step 2: Extract and Process Audio
|
||||
|
||||
- Extract audio track from raw footage
|
||||
- Apply noise reduction if needed (via `audio_mixer`)
|
||||
- Normalize audio levels
|
||||
|
||||
### Step 3: Generate Overlays (Optional)
|
||||
### Step 3: Source Background Music
|
||||
|
||||
If the scene plan includes overlay scenes:
|
||||
- Generate text card images
|
||||
- Generate lower third graphics
|
||||
- Create any B-roll placeholders
|
||||
If the scene plan includes background music:
|
||||
|
||||
### Step 4: Build Asset Manifest
|
||||
1. **Check local pixabay music library** — look for downloaded MP3s matching the mood
|
||||
2. **Use `pixabay_music` tool** — search by mood/genre keywords from the scene plan
|
||||
3. **Run `audio_energy` analysis** on the selected track to find optimal start offset (skip quiet intros)
|
||||
|
||||
Document all generated assets with paths, types, and tool references.
|
||||
Record the music path, offset, and whether looping is needed in the asset manifest.
|
||||
|
||||
### Step 5: Self-Evaluate
|
||||
### Step 4: Generate Overlay Assets
|
||||
|
||||
If the scene plan includes overlay scenes (from the scene-director's Watch & Propose step), generate the assets for each.
|
||||
|
||||
**For Remotion-rendered overlays** (charts, comparisons, KPI grids, stat cards):
|
||||
|
||||
Create a composition JSON snippet for each overlay. These will be rendered by the compose-director. Each overlay needs:
|
||||
|
||||
```json
|
||||
{
|
||||
"overlay_id": "overlay_1",
|
||||
"remotion_cut": {
|
||||
"id": "term-agentic-ai",
|
||||
"type": "callout",
|
||||
"text": "Agentic AI: software that acts autonomously toward goals",
|
||||
"in_seconds": 0,
|
||||
"out_seconds": 4,
|
||||
"backgroundColor": "#0F172A",
|
||||
"accentColor": "#22D3EE",
|
||||
"icon": "💡"
|
||||
},
|
||||
"overlay_timestamp": 22.0,
|
||||
"position": "lower_third"
|
||||
}
|
||||
```
|
||||
|
||||
**Overlay type → Remotion cut mapping:**
|
||||
|
||||
| Scene Plan Overlay | Remotion `type` | Required Props |
|
||||
|-------------------|-----------------|----------------|
|
||||
| Key term definition | `callout` | `text`, `icon` (optional) |
|
||||
| Statistic/number | `stat_card` | `stat` (the number), `text` (label) |
|
||||
| Comparison | `comparison` | `leftLabel`, `rightLabel`, `leftValue`, `rightValue` |
|
||||
| Data chart | `bar_chart` | `chartData` (array of `{label, value}`) |
|
||||
| Pie chart | `pie_chart` | `chartData` (array of `{label, value}`) |
|
||||
| Line chart | `line_chart` | `chartSeries` (array of `{name, data: number[]}`) |
|
||||
| KPI dashboard | `kpi_grid` | `chartData` (array of `{label, value}`) — keep numbers small with suffix (e.g. "2.4M") |
|
||||
| Progress indicator | `progress_bar` | `progress` (0-100), `text` |
|
||||
| Section title | `hero_title` | `text`, `subtitle` (optional) |
|
||||
| Callout/quote | `callout` | `text`, `icon` |
|
||||
| Lower third | `text_card` | `text` |
|
||||
|
||||
**Dark theme for all overlays** — use dark backgrounds (`#0F172A`, `#1E293B`) with light text. This ensures overlays are legible when composited on top of talking-head footage.
|
||||
|
||||
**For simple text overlays** (if Remotion is overkill):
|
||||
|
||||
Generate PNG images using FFmpeg or PIL, stored at `<project>/assets/overlays/overlay_<id>.png`.
|
||||
|
||||
### Step 5: Build Asset Manifest
|
||||
|
||||
Document all generated assets with paths, types, and tool references:
|
||||
|
||||
```json
|
||||
{
|
||||
"subtitles": {
|
||||
"path": "assets/subtitles/subtitles.srt",
|
||||
"format": "srt",
|
||||
"word_count": 208
|
||||
},
|
||||
"music": {
|
||||
"path": "assets/audio/bg_music.mp3",
|
||||
"offset_seconds": 3.5,
|
||||
"needs_loop": true
|
||||
},
|
||||
"overlays": [
|
||||
{
|
||||
"overlay_id": "overlay_1",
|
||||
"type": "callout",
|
||||
"timestamp": 22.0,
|
||||
"duration": 4.0,
|
||||
"remotion_cut": { ... },
|
||||
"position": "lower_third"
|
||||
}
|
||||
],
|
||||
"transcript_segments": "assets/audio/transcript.json"
|
||||
}
|
||||
```
|
||||
|
||||
### Step 6: Self-Evaluate
|
||||
|
||||
| Criterion | Question |
|
||||
|-----------|----------|
|
||||
| **Subtitles** | Do subtitles exist and match speech timing? |
|
||||
| **Audio** | Is audio clean and normalized? |
|
||||
| **Music** | Was audio_energy run on the music to find optimal offset? |
|
||||
| **Overlays** | Does every overlay from the scene plan have a generated asset? |
|
||||
| **Overlay content** | Is the data in overlays accurate to what the speaker actually says? |
|
||||
| **Files** | Do all asset paths point to existing files? |
|
||||
|
||||
### Step 6: Submit
|
||||
### Step 7: Submit
|
||||
|
||||
Validate the asset_manifest against the schema and persist via checkpoint.
|
||||
|
||||
@@ -17,15 +17,14 @@ You have edit decisions and an asset manifest. Your job is to render the final t
|
||||
|
||||
### Step 1: Run Enhancement Chain
|
||||
|
||||
Apply video enhancements in order:
|
||||
1. **Face enhancement** (if `face_enhance` tool available) — apply `talking_head_standard` preset
|
||||
2. **Eye enhancement** (if `eye_enhance` tool available) — under-eye dark circle removal + eye brightening
|
||||
3. **Color grading** (if `color_grade` tool available) — apply a profile
|
||||
4. **Audio enhancement** (if `audio_enhance` tool available) — noise reduction, normalization
|
||||
Apply video enhancements in this exact order. **Attempt every step** if the tool is available — do not skip steps without a reason.
|
||||
|
||||
Each step is optional — check tool availability first.
|
||||
1. **Face enhancement** — apply `talking_head_standard` preset
|
||||
2. **Eye enhancement** — under-eye dark circle removal + eye brightening
|
||||
3. **Color grading** — apply a profile
|
||||
4. **Audio enhancement** — noise reduction, normalization
|
||||
|
||||
**Eye enhancement** — removes under-eye dark circles and brightens eyes using MediaPipe Face Mesh landmark detection:
|
||||
**Eye enhancement** — always attempt this after face_enhance. It makes a visible difference on webcam/phone footage:
|
||||
```
|
||||
eye_enhance.execute({
|
||||
"input_path": "<face_enhanced_video>",
|
||||
@@ -35,7 +34,7 @@ eye_enhance.execute({
|
||||
"eye_brighten_intensity": 0.3,
|
||||
})
|
||||
```
|
||||
**Important:** Keep intensities low (0.2-0.5). Over-processing makes eyes look unnatural. Always compare before/after.
|
||||
**Important:** Keep intensities low (0.2-0.5). Over-processing makes eyes look unnatural. If the tool fails (e.g. MediaPipe not installed), log the fallback and continue with the face_enhanced video.
|
||||
|
||||
### Step 1b: Speed Adjustment (if requested)
|
||||
|
||||
@@ -86,6 +85,25 @@ The tool automatically runs face detection and keeps the speaker centered. If Me
|
||||
|
||||
**Important:** Run auto_reframe AFTER face_enhance and color_grade but BEFORE burning subtitles. Subtitles need to be positioned for the final aspect ratio.
|
||||
|
||||
### Step 2b: Build ASR Corrections Dictionary
|
||||
|
||||
Before burning captions, scan the transcript for likely ASR misrecognitions. Common issues:
|
||||
- Product/brand names: "cloud" → "Claude", "co-pilot" → "Copilot", "remotion" → "Remotion"
|
||||
- Technical terms: "pythonic" misheard as "pathonic", "API" as "a pie"
|
||||
- Speaker's name or company name
|
||||
- Domain-specific jargon
|
||||
|
||||
Build a corrections dict:
|
||||
```python
|
||||
corrections = {
|
||||
"cloud": "Claude",
|
||||
"co pilot": "Copilot",
|
||||
"open montage": "OpenMontage",
|
||||
}
|
||||
```
|
||||
|
||||
Pass this dict to both `subtitle_gen` (if generating SRT) and `remotion_caption_burn` (if using Remotion captions). Even if you find zero corrections needed, explicitly pass an empty dict `{}` to confirm you checked.
|
||||
|
||||
### Step 3: Burn Subtitles
|
||||
|
||||
**Preferred: Remotion captions** (if `remotion_caption_burn` tool available):
|
||||
@@ -106,11 +124,40 @@ Remotion renders animated word-by-word captions at the bottom of the frame with
|
||||
Use `video_compose` with `burn_subtitles` operation:
|
||||
- Input: reframed video (or enhanced video if no reframe needed)
|
||||
- Subtitle file from asset manifest
|
||||
- Style from playbook
|
||||
- For vertical (9:16) output: position subtitles in the lower 20% of frame with `MarginV=100`
|
||||
- **Never** position subtitles in the center of the frame — they will occlude the face
|
||||
|
||||
### Step 3b: Build Showcase Cards (if multi-clip reel)
|
||||
**CRITICAL: Caption positioning for 9:16 vertical video.**
|
||||
Captions MUST be in the lower 20% of the frame. On a 1920-high frame, that means `MarginV=160` or higher. The default FFmpeg subtitle position is center — this WILL occlude the face. You MUST override it.
|
||||
|
||||
FFmpeg subtitle style string for vertical talking-head:
|
||||
```
|
||||
"FontName=Arial,FontSize=22,Bold=1,PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,Outline=2,Shadow=0,MarginV=160,Alignment=2"
|
||||
```
|
||||
|
||||
**Never** use the default subtitle position. **Never** position subtitles in the center or upper half of the frame. If you see captions on the face during visual QA, the video must be re-rendered with corrected positioning.
|
||||
|
||||
### Step 3b: Burn Overlay Graphics (if scene plan includes overlays)
|
||||
|
||||
If the scene plan includes overlay scenes (text_cards, stat_cards, charts, comparisons, callouts), render them onto the video.
|
||||
|
||||
**How overlay compositing works:**
|
||||
1. Each overlay is a short Remotion composition (3-5 seconds) rendered as a transparent video clip or composited directly
|
||||
2. Use `video_compose` with `picture_in_picture` or `overlay` operation to place each overlay at the correct timestamp
|
||||
3. Respect the overlay's `position` field from the scene plan:
|
||||
- `lower_third` → bottom 30% of frame
|
||||
- `upper_third` → top 30% of frame
|
||||
- `side_panel` → left or right 40%
|
||||
- `full_overlay` → centered, brief (1-2s)
|
||||
|
||||
**For Remotion-based overlays:** Create a composition JSON with the overlay cuts, render to a transparent clip, then composite onto the talking-head video using FFmpeg.
|
||||
|
||||
**For simple text overlays:** Use FFmpeg's drawtext filter directly:
|
||||
```
|
||||
ffmpeg -i captioned.mp4 -vf "drawtext=text='Key Term':fontsize=48:fontcolor=white:borderw=3:bordercolor=black:x=(w-text_w)/2:y=h*0.75:enable='between(t,22,26)'" -c:a copy output.mp4
|
||||
```
|
||||
|
||||
**Important:** Time each overlay to match the scene plan timestamps. After speed adjustment, recalculate overlay timestamps: `adjusted_time = original_time / speed_factor`.
|
||||
|
||||
### Step 3c: Build Showcase Cards (if multi-clip reel)
|
||||
|
||||
If the output is a reel with showcase clips, use `showcase_card` for each:
|
||||
```
|
||||
@@ -166,12 +213,35 @@ audio_mixer.execute({
|
||||
- Apply ducking if music is present
|
||||
- Normalize final levels
|
||||
|
||||
### Step 6: Final Encode
|
||||
### Step 6: Final Encode — MANDATORY
|
||||
|
||||
**Do not skip this step.** Without a final encode, the output will be oversized and may not play correctly on the target platform.
|
||||
|
||||
Use `video_compose` with `encode` operation:
|
||||
- Apply target media profile (youtube_landscape, tiktok, instagram_reels, etc.)
|
||||
- Two-pass encoding for quality
|
||||
|
||||
**Target file sizes:**
|
||||
| Platform | Max Duration | Target Size |
|
||||
|----------|-------------|-------------|
|
||||
| Instagram Reels | 90s | < 50 MB |
|
||||
| TikTok | 10 min | < 100 MB |
|
||||
| YouTube Shorts | 60s | < 40 MB |
|
||||
| YouTube | unlimited | < 25 MB/min |
|
||||
|
||||
If the output exceeds the target, re-encode with a lower bitrate. A 66-second Instagram Reel at 76 MB is unacceptable — it should be under 30 MB.
|
||||
|
||||
```
|
||||
video_compose.execute({
|
||||
"operation": "encode",
|
||||
"input_path": "<mixed_video>",
|
||||
"output_path": "<project>/renders/final.mp4",
|
||||
"media_profile": "instagram_reels",
|
||||
"video_bitrate": "4M",
|
||||
"audio_bitrate": "192k",
|
||||
})
|
||||
```
|
||||
|
||||
### Step 7: Visual QA
|
||||
|
||||
Use `visual_qa` to verify the output before declaring success:
|
||||
|
||||
@@ -2,7 +2,9 @@
|
||||
|
||||
## When to Use
|
||||
|
||||
You have a script (from transcription) and raw footage. Your job is to create a scene plan — mostly simple since talking-head footage is a single continuous shot, but you still need to plan where overlays, text cards, and B-roll might appear.
|
||||
You have a script (from transcription) and raw footage. Your job is to **watch the footage, understand the content, and propose a creative enhancement plan** — then build a scene plan that transforms raw talking-head footage into an engaging, visually rich video.
|
||||
|
||||
You are not just a processor. You are a creative director. Your job is to figure out what the speaker is saying and propose visual enhancements that make the content more engaging and easier to understand.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -16,7 +18,87 @@ You have a script (from transcription) and raw footage. Your job is to create a
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Analyze Footage (if tools available)
|
||||
### Step 1: Watch & Listen — Understand the Content
|
||||
|
||||
**This is the most important step. Do not skip it.**
|
||||
|
||||
Read the full transcript carefully. Understand:
|
||||
- What is the speaker's **main topic**?
|
||||
- What are the **key concepts** they explain?
|
||||
- Where do they use **numbers, statistics, or data**?
|
||||
- Where do they **compare things** (A vs B, before/after, old vs new)?
|
||||
- Where do they **list items** (3 tips, 5 steps, etc.)?
|
||||
- Where do they introduce **technical terms** or jargon?
|
||||
- Where are the **section transitions** (topic changes)?
|
||||
- What is the **emotional arc** (excitement, serious, humorous)?
|
||||
|
||||
If `frame_sampler` is available, extract 5-8 representative frames to see the speaker's setup, background, lighting, and gestures.
|
||||
|
||||
### Step 2: Propose Creative Overlays
|
||||
|
||||
Based on your content analysis, propose **on-screen graphics** that will appear alongside the speaker at key moments. These are Remotion components that get composited on top of or next to the talking-head footage.
|
||||
|
||||
**Available overlay types:**
|
||||
|
||||
| Overlay Type | Remotion Component | Best For |
|
||||
|-------------|-------------------|----------|
|
||||
| **Key term definition** | `text_card` | When the speaker introduces a technical term — show the term + short definition |
|
||||
| **Statistic/number** | `stat_card` | When the speaker mentions a number or percentage — animate it on screen |
|
||||
| **Comparison** | `comparison` | When the speaker compares two things (A vs B) — show side-by-side |
|
||||
| **Data chart** | `bar_chart` / `pie_chart` / `line_chart` | When the speaker references data or rankings |
|
||||
| **KPI dashboard** | `kpi_grid` | When multiple numbers are mentioned together |
|
||||
| **Progress indicator** | `progress_bar` | When the speaker describes a process or percentage |
|
||||
| **Section title** | `hero_title` | At major topic transitions — show the new section title |
|
||||
| **Callout/quote** | `callout` | When the speaker makes a key point worth emphasizing |
|
||||
| **Lower third** | `text_card` | Speaker identification at the start |
|
||||
|
||||
**Overlay planning rules:**
|
||||
- **Don't over-overlay.** 3-6 overlays per minute of final video is the sweet spot. More than that is distracting.
|
||||
- **Time overlays to speech.** Each overlay should appear when the speaker says the relevant words, not before or after.
|
||||
- **Keep overlays brief.** 3-5 seconds each. They support the speaker, not compete with them.
|
||||
- **Vary the types.** Don't use 5 text_cards in a row — mix in charts, comparisons, callouts.
|
||||
- **Use overlays at natural pauses.** When the speaker pauses for emphasis, that's a good overlay moment.
|
||||
- **Match the vibe.** Professional talk = clean stat cards and charts. Casual talk = callouts and bold key terms.
|
||||
|
||||
### Step 3: Present Your Plan to the User
|
||||
|
||||
**MANDATORY: Present your enhancement plan before proceeding.**
|
||||
|
||||
Format your proposal clearly:
|
||||
|
||||
```
|
||||
## Enhancement Plan for [Video Title/Topic]
|
||||
|
||||
**Content Summary:** [1-2 sentences about what the speaker covers]
|
||||
|
||||
**Proposed Overlays:**
|
||||
|
||||
| Time | Type | Content | Why |
|
||||
|------|------|---------|-----|
|
||||
| 0:05 | lower_third | "Speaker Name — Title" | Speaker intro |
|
||||
| 0:22 | text_card | "Agentic AI: software that acts autonomously" | Key term definition |
|
||||
| 0:45 | comparison | Traditional Software vs Agentic Software | Speaker is comparing the two |
|
||||
| 1:10 | stat_card | "73% of developers..." | Statistic mentioned |
|
||||
| 1:35 | bar_chart | [Framework popularity data] | Speaker references rankings |
|
||||
| 2:00 | callout | "The key insight is..." | Speaker's main takeaway |
|
||||
|
||||
**Enhancement Chain:**
|
||||
- Silence removal: ~X seconds of dead air detected
|
||||
- Speed: 1.25x (user requested)
|
||||
- Face + eye enhancement
|
||||
- Animated captions (bottom of frame)
|
||||
- Background music: [recommendation]
|
||||
|
||||
**Estimated final duration:** ~Xs (from Xs raw)
|
||||
```
|
||||
|
||||
Wait for user approval before proceeding. The user may:
|
||||
- Approve as-is
|
||||
- Add/remove overlays
|
||||
- Change overlay content
|
||||
- Adjust the enhancement plan
|
||||
|
||||
### Step 4: Analyze Footage (if tools available)
|
||||
|
||||
**Face tracking** — If `face_tracker` is available, run it on the raw footage:
|
||||
```
|
||||
@@ -29,6 +111,7 @@ This outputs per-frame face bounding boxes. Use this data to:
|
||||
- Decide if reframing is needed (e.g. speaker is off-center for vertical crop)
|
||||
- Identify sections where the speaker moves significantly (needs dynamic crop)
|
||||
- Note face position for auto_reframe in the compose stage
|
||||
- **Determine overlay safe zones** — where to place graphics without occluding the face
|
||||
|
||||
**Silence detection** — If `silence_cutter` is available, run in `mark` mode:
|
||||
```
|
||||
@@ -43,20 +126,38 @@ This outputs silence/speech segment timestamps. Use this to:
|
||||
- Plan which segments should be jump-cut or sped up
|
||||
- Identify dead air, false starts, and long pauses
|
||||
- Estimate the final video duration after cuts
|
||||
- Present the user with a summary: "Found X seconds of silence across Y segments — recommend removing?"
|
||||
|
||||
### Step 2: Plan Base Scenes
|
||||
### Step 5: Plan Base Scenes
|
||||
|
||||
For talking-head, the base is simple: one scene per script section, all type `talking_head`. The raw footage IS the scene.
|
||||
|
||||
### Step 3: Plan Enhancement Scenes
|
||||
### Step 6: Build Overlay Scenes
|
||||
|
||||
Based on script enhancement cues, plan overlay scenes:
|
||||
- Text cards for key terms or statistics
|
||||
- Lower thirds for speaker identification
|
||||
- B-roll suggestions for topic illustrations
|
||||
For each approved overlay from Step 3, create an overlay scene entry:
|
||||
```json
|
||||
{
|
||||
"id": "overlay_1",
|
||||
"type": "overlay",
|
||||
"overlay_type": "text_card",
|
||||
"start_seconds": 22.0,
|
||||
"duration_seconds": 4.0,
|
||||
"content": {
|
||||
"text": "Agentic AI",
|
||||
"subtext": "Software that acts autonomously toward goals",
|
||||
"backgroundColor": "#0F172A",
|
||||
"accentColor": "#22D3EE"
|
||||
},
|
||||
"position": "lower_third"
|
||||
}
|
||||
```
|
||||
|
||||
### Step 4: Plan Reframing & Cuts
|
||||
**Overlay position options:**
|
||||
- `lower_third` — bottom 30% of frame (safest, doesn't occlude face)
|
||||
- `upper_third` — top 30% of frame (good for titles)
|
||||
- `side_panel` — left or right 40% (for charts/comparisons, speaker shifts to other side)
|
||||
- `full_overlay` — brief full-screen graphic (1-2s max, for dramatic emphasis)
|
||||
|
||||
### Step 7: Plan Reframing & Cuts
|
||||
|
||||
If the target platform requires a different aspect ratio (e.g. Instagram Reels = 9:16):
|
||||
- Note `auto_reframe` should be applied in the compose stage
|
||||
@@ -67,21 +168,28 @@ If silence detection found segments to cut:
|
||||
- Record the recommended cut mode (`remove` or `speed_up`) in the scene plan
|
||||
- Note padding preferences (default 0.08s to avoid clipping words)
|
||||
|
||||
### Step 5: Build Scene Plan
|
||||
### Step 8: Build Scene Plan
|
||||
|
||||
Create a scene per section with:
|
||||
- Type: `talking_head` (primary)
|
||||
- Timing from script sections
|
||||
- Required assets: subtitle file, any overlay images
|
||||
Assemble the full scene plan with:
|
||||
- Base scenes (one per script section, type `talking_head`)
|
||||
- Overlay scenes (from Step 6, type `overlay`)
|
||||
- Enhancement chain decisions (silence cut mode, speed factor, reframe target)
|
||||
- Music recommendation
|
||||
- Estimated final duration
|
||||
|
||||
### Step 6: Self-Evaluate
|
||||
### Step 9: Self-Evaluate
|
||||
|
||||
| Criterion | Question |
|
||||
|-----------|----------|
|
||||
| **Content understanding** | Did I actually understand what the speaker is talking about? |
|
||||
| **Overlay relevance** | Does every overlay directly relate to what's being said at that moment? |
|
||||
| **Overlay density** | Am I in the 3-6 per minute range? Not too sparse, not too cluttered? |
|
||||
| **Overlay variety** | Am I using different types, not just text_cards? |
|
||||
| **Timing** | Are overlays timed to the speaker's words, not arbitrary moments? |
|
||||
| **Coverage** | Every script section has a scene? |
|
||||
| **Enhancement** | Are overlay opportunities identified? |
|
||||
| **Feasibility** | Can all required assets be generated? |
|
||||
| **Feasibility** | Can all overlays be rendered with available Remotion components? |
|
||||
| **User approved** | Did the user approve the enhancement plan? |
|
||||
|
||||
### Step 7: Submit
|
||||
### Step 10: Submit
|
||||
|
||||
Validate the scene_plan against the schema and persist via checkpoint.
|
||||
|
||||
@@ -215,7 +215,8 @@ class VisualQA(BaseTool):
|
||||
input_path,
|
||||
]
|
||||
import json
|
||||
probe_out = self.run_command(cmd, capture=True)
|
||||
probe_result = self.run_command(cmd)
|
||||
probe_out = probe_result.stdout
|
||||
probe_data = json.loads(probe_out)
|
||||
|
||||
# Extract key info
|
||||
@@ -302,10 +303,11 @@ class VisualQA(BaseTool):
|
||||
"-t", "3",
|
||||
"-i", input_path,
|
||||
"-vn", "-af", "volumedetect",
|
||||
"-f", "null", "/dev/null",
|
||||
"-f", "null", "NUL" if __import__("sys").platform == "win32" else "/dev/null",
|
||||
]
|
||||
try:
|
||||
output = self.run_command(cmd, capture=True, stderr=True)
|
||||
cmd_result = self.run_command(cmd)
|
||||
output = cmd_result.stderr # volumedetect outputs to stderr
|
||||
mean_vol = None
|
||||
max_vol = None
|
||||
for line in output.split("\n"):
|
||||
@@ -340,4 +342,5 @@ class VisualQA(BaseTool):
|
||||
"-of", "csv=p=0",
|
||||
path,
|
||||
]
|
||||
return float(self.run_command(cmd, capture=True).strip().split("\n")[0])
|
||||
dur_result = self.run_command(cmd)
|
||||
return float(dur_result.stdout.strip().split("\n")[0])
|
||||
|
||||
@@ -257,7 +257,8 @@ class RemotionCaptionBurn(BaseTool):
|
||||
"-of", "csv=p=0",
|
||||
input_path,
|
||||
]
|
||||
dur_out = self.run_command(dur_cmd, capture=True)
|
||||
dur_result = self.run_command(dur_cmd)
|
||||
dur_out = dur_result.stdout
|
||||
duration_s = float(dur_out.strip().split("\n")[0])
|
||||
total_frames = math.ceil(duration_s * 30)
|
||||
|
||||
@@ -281,9 +282,11 @@ class RemotionCaptionBurn(BaseTool):
|
||||
props_file = props_dir / f"caption-burn-{Path(input_path).stem}.json"
|
||||
props_file.write_text(json.dumps(props, indent=2), encoding="utf-8")
|
||||
|
||||
# Render
|
||||
# Render (use npx.cmd on Windows for subprocess compatibility)
|
||||
import sys
|
||||
npx_bin = "npx.cmd" if sys.platform == "win32" else "npx"
|
||||
render_cmd = [
|
||||
"npx", "remotion", "render",
|
||||
npx_bin, "remotion", "render",
|
||||
"src/index.tsx", "TalkingHead",
|
||||
f"--props={props_file.relative_to(root)}",
|
||||
"--width=1080", "--height=1920", "--fps=30",
|
||||
|
||||
Reference in New Issue
Block a user