freedak/OpenMontage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Green screen pipeline: new tools, AnimatedBackground, caption burn fixes

Browse Source 
Add green_screen_processor (auto-detect + rembg fallback) and
green_screen_composite (4 layout presets with alpha compositing) tools
to automate the full keying-to-composite pipeline.

Remotion: add AnimatedBackground with gradient mesh and floating orbs
to Explainer, fix caption burn tool (remove entry point arg, auto-detect
dimensions, extend TalkingHead duration to 300s).

Update scene-director, compose-director, and asset-director skill docs
with green screen workflow steps and component constraints.
This commit is contained in:
calesthio
2026-04-02 11:54:30 -07:00
parent 942244ca54
commit a7e5f7498b
11 changed files with 2087 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/pipelines/talking-head/asset-director.md
+28 -1
View File
@@ -90,7 +90,34 @@ Create a composition JSON snippet for each overlay. These will be rendered by th
| Callout/quote | `callout` | `text`, `icon` |
| Lower third | `text_card` | `text` |
**Dark theme for all overlays** — use dark backgrounds (`#0F172A`, `#1E293B`) with light text. This ensures overlays are legible when composited on top of talking-head footage.
**Remotion AnimatedBackground:**
The Explainer composition now includes an `AnimatedBackground` component that renders an animated gradient mesh, floating orbs, and a subtle grid pattern. This provides a far more professional look than flat solid colors.
- Scene backgrounds should use `backgroundColor: "#0F172A"` which auto-converts to transparent, letting the animated gradient show through.
- Do NOT use flat solid colors for backgrounds -- always let the AnimatedBackground render behind overlays.
- When compositing green screen footage, render the AnimatedBackground as the replacement background (see compose-director Step 3c).
**Component constraints:**
| Component | Min Width | 720px Portrait? | Value Type |
|-----------|-----------|-----------------|------------|
| comparison | 900px | NO -> use 2x stat_card | string |
| kpi_grid | 720px | YES | numeric ONLY (no "15+") |
| bar_chart | 500px | YES | numeric |
| stat_card | 300px | YES | string OK |
| callout | 400px | YES | string |
| hero_title | 400px | YES | string |
| line_chart | 500px | YES | numeric |
| progress_bar | 600px | YES | numeric |
| stat_reveal | 300px | YES | string OK |
Key rules:
- `comparison` requires 900px+ width. In 720px portrait frames, substitute with two sequential `stat_card` components instead.
- `kpi_grid` values MUST be purely numeric (e.g., `4.8`, `73`, `2400`). Formatted strings like `"15+"`, `"$4.8B"`, or `"2.4M"` will cause rendering errors. Use `stat_card` for string-formatted numbers instead.
- Always check the target frame width before choosing a component. Portrait (720px) excludes `comparison`.
**Dark theme for all overlays** -- use dark backgrounds (`#0F172A`, `#1E293B`) with light text. This ensures overlays are legible when composited on top of talking-head footage. The `#0F172A` backgroundColor is treated as transparent by the AnimatedBackground system, so overlays will show the animated gradient behind them.
**For simple text overlays** (if Remotion is overkill):
skills/pipelines/talking-head/compose-director.md
+190 -23
View File
@@ -15,6 +15,45 @@ You have edit decisions and an asset manifest. Your job is to render the final t
## Process
### Step 0: Pre-flight Checks
Before rendering anything, validate the inputs and catch issues that are expensive to fix later.
1. **Silence detection** -- Run `silence_cutter` in mark mode:
```
silence_cutter.execute({
"input_path": "<raw_footage>",
"mode": "mark",
"silence_threshold_db": -35,
"min_silence_duration": 0.5
})
```
- Report all gaps > 0.5s with timestamps.
- If total silence > 5s, **recommend cutting before proceeding**. Long silences waste render time and produce dead spots in the final video.
2. **ASR confidence check** -- Scan word-level transcript for low-confidence words:
- Flag any word with probability < 0.7.
- List flagged words with timestamps so the user can verify correct transcription.
- Common misrecognitions to watch for: proper nouns, brand names, domain jargon.
3. **Auto-build corrections dictionary** from common ASR error patterns:
```python
corrections = {
# Indian finance context
"DMI": "EMI",
"AMI": "EMI",
# Common brand misspellings
"open montage": "OpenMontage",
"remotion": "Remotion",
# Numbers that got split by ASR
"4 -5": "4-5",
"10 -15": "10-15",
}
```
Extend this dict with domain-specific corrections based on the video topic. Present the corrections to the user for review before applying.
4. **Green screen flag** -- Check if scene-director Step 0 flagged green/blue screen footage. If yes, note that Step 3c (Green Screen Composite) will be needed.
### Step 1: Run Enhancement Chain
Apply video enhancements in this exact order. **Attempt every step** if the tool is available — do not skip steps without a reason.
@@ -106,7 +145,14 @@ Pass this dict to both `subtitle_gen` (if generating SRT) and `remotion_caption_
### Step 3: Burn Subtitles
**Preferred: Remotion captions** (if `remotion_caption_burn` tool available):
**ALWAYS use Remotion TikTok-style captions** (word-by-word highlighting). This is the default and preferred method. Do NOT fall back to FFmpeg ASS subtitles unless Remotion is completely unavailable.
**Remotion caption requirements:**
- **Auto-detect video dimensions** -- do NOT hardcode width/height. Use `visual_qa` probe or ffprobe to get actual dimensions, then pass them to the render.
- **Set `--frames` based on actual video duration** -- calculate from probe: `frames = duration_seconds * fps`. Never use a hardcoded frame count.
- Word-by-word highlighting with active word color (`highlight_color`).
- Captions positioned at the bottom of frame, away from the face.
```
remotion_caption_burn.execute({
"input_path": "<reframed_or_enhanced_video>",
@@ -118,15 +164,11 @@ remotion_caption_burn.execute({
"highlight_color": "#22D3EE",
})
```
Remotion renders animated word-by-word captions at the bottom of the frame with active word highlighting. Captions are positioned away from the face.
**Fallback: FFmpeg subtitles** (if Remotion unavailable):
Use `video_compose` with `burn_subtitles` operation:
- Input: reframed video (or enhanced video if no reframe needed)
- Subtitle file from asset manifest
**Fallback ONLY if Remotion is completely unavailable:** Use `video_compose` with `burn_subtitles` operation. This is a degraded experience -- warn the user that word-by-word highlighting won't be available.
**CRITICAL: Caption positioning for 9:16 vertical video.**
Captions MUST be in the lower 20% of the frame. On a 1920-high frame, that means `MarginV=160` or higher. The default FFmpeg subtitle position is center — this WILL occlude the face. You MUST override it.
**CRITICAL: Caption positioning for 9:16 vertical video (FFmpeg fallback only).**
Captions MUST be in the lower 20% of the frame. On a 1920-high frame, that means `MarginV=160` or higher. The default FFmpeg subtitle position is center -- this WILL occlude the face. You MUST override it.
FFmpeg subtitle style string for vertical talking-head:
```
@@ -137,27 +179,152 @@ FFmpeg subtitle style string for vertical talking-head:
### Step 3b: Burn Overlay Graphics (if scene plan includes overlays)
If the scene plan includes overlay scenes (text_cards, stat_cards, charts, comparisons, callouts), render them onto the video.
If the scene plan includes overlay scenes (text_cards, stat_cards, charts, comparisons, callouts), pass them to `remotion_caption_burn` alongside captions. **Both captions and overlays render in a single Remotion pass** — no separate FFmpeg compositing needed.
**How overlay compositing works:**
1. Each overlay is a short Remotion composition (3-5 seconds) rendered as a transparent video clip or composited directly
2. Use `video_compose` with `picture_in_picture` or `overlay` operation to place each overlay at the correct timestamp
3. Respect the overlay's `position` field from the scene plan:
- `lower_third` → bottom 30% of frame
- `upper_third` → top 30% of frame
- `side_panel` → left or right 40%
- `full_overlay` → centered, brief (1-2s)
**How it works:** The TalkingHead Remotion composition renders three layers:
1. **Video** (bottom) — the talking-head footage
2. **Overlays** (middle) — positioned charts, stats, callouts with fade in/out
3. **Captions** (top) — word-by-word highlighting, always visible
**For Remotion-based overlays:** Create a composition JSON with the overlay cuts, render to a transparent clip, then composite onto the talking-head video using FFmpeg.
**For simple text overlays:** Use FFmpeg's drawtext filter directly:
**Combine Step 3 and 3b into one `remotion_caption_burn` call:**
```
ffmpeg -i captioned.mp4 -vf "drawtext=text='Key Term':fontsize=48:fontcolor=white:borderw=3:bordercolor=black:x=(w-text_w)/2:y=h*0.75:enable='between(t,22,26)'" -c:a copy output.mp4
remotion_caption_burn.execute({
"input_path": "<reframed_or_enhanced_video>",
"output_path": "<project>/assets/video/captioned.mp4",
"segments": <transcript_segments>,
"corrections": {"cloud": "Claude"},
"words_per_page": 4,
"font_size": 52,
"highlight_color": "#22D3EE",
"overlays": [
{
"id": "term-agentic-ai",
"type": "callout",
"text": "Agentic AI: software that acts autonomously toward goals",
"callout_type": "info",
"in_seconds": 22.0,
"out_seconds": 26.0,
"position": "lower_third",
"backgroundColor": "#0F172A",
"accentColor": "#22D3EE"
},
{
"id": "stat-market-size",
"type": "stat_card",
"stat": "$4.8B",
"subtitle": "Global AI Agent Market (2026)",
"in_seconds": 35.0,
"out_seconds": 39.0,
"position": "upper_third",
"accentColor": "#A78BFA"
},
{
"id": "chart-growth",
"type": "bar_chart",
"chartData": [
{"label": "2023", "value": 1.2},
{"label": "2024", "value": 2.1},
{"label": "2025", "value": 3.5},
{"label": "2026", "value": 4.8}
],
"title": "AI Agent Market ($B)",
"in_seconds": 40.0,
"out_seconds": 45.0,
"position": "lower_third",
"chartColors": ["#22D3EE", "#A78BFA", "#F472B6", "#34D399"]
}
]
})
```
**Important:** Time each overlay to match the scene plan timestamps. After speed adjustment, recalculate overlay timestamps: `adjusted_time = original_time / speed_factor`.
**Overlay position options:**
- `lower_third` → bottom area, above captions (default — safest for most overlays)
- `upper_third` → top area (good for stats while speaker is center/lower)
- `left_panel` → left 45% of frame (side-by-side with speaker)
- `right_panel` → right 45% of frame
- `full_overlay` → full frame with dark backdrop (use sparingly, 1-2s max)
### Step 3c: Build Showcase Cards (if multi-clip reel)
**Overlay type → required props** (same as asset-director mapping):
| Type | Required Props |
|------|---------------|
| `text_card` | `text` |
| `stat_card` | `stat`, `subtitle` (optional) |
| `callout` | `text`, `callout_type` (info/warning/tip/quote) |
| `comparison` | `leftLabel`, `rightLabel`, `leftValue`, `rightValue` |
| `bar_chart` | `chartData` (array of `{label, value}`) |
| `line_chart` | `chartSeries` (array of `{name, data: number[]}`) |
| `pie_chart` | `chartData` (array of `{label, value}`) |
| `kpi_grid` | `chartData` (array of `{label, value}`) |
| `hero_title` | `text`, `subtitle` (optional) |
| `section_title` | `text`, `subtitle` (optional) |
| `stat_reveal` | `text` (the stat), `subtitle` (label) |
**Important:** After speed adjustment, recalculate overlay timestamps: `adjusted_time = original_time / speed_factor`.
**Fallback (no Remotion):** If Remotion is unavailable, `remotion_caption_burn` falls back to FFmpeg for captions only. Overlays are NOT rendered in FFmpeg fallback mode — warn the user that overlays require Remotion.
### Step 3c: Green Screen Composite (if green screen footage)
If the footage has a green/blue screen (detected in scene-director Step 0), follow this pipeline:
1. **Run `green_screen_processor` tool** to remove the green/blue screen:
```
green_screen_processor.execute({
"input_path": "<enhanced_video>",
"output_path": "<project>/assets/video/greenscreen_removed.mp4",
"method": "auto"
})
```
The `auto` method detects whether the background is green or blue and applies the appropriate chroma key.
2. **Render Remotion animated background** using the Explainer composition:
```
# Render an AnimatedBackground clip (gradient mesh, floating orbs, subtle grid)
# Use the Explainer composition — NOT a flat #0F172A solid color
npx remotion render src/index.ts Explainer --props='{"duration":VIDEO_DURATION}' --output=<project>/assets/video/animated_bg.mp4
```
The AnimatedBackground provides a professional gradient mesh with floating orbs and a subtle grid pattern. This is far superior to a flat solid color.
3. **Run `green_screen_composite` tool** to layer the speaker onto the animated background:
```
green_screen_composite.execute({
"foreground_path": "<greenscreen_removed_video>",
"background_path": "<animated_bg>",
"output_path": "<project>/assets/video/composited.mp4",
"layout": "news_anchor"
})
```
Default layout is `news_anchor` (speaker center-bottom, background fills frame). Adjust layout based on speaker position detected in Step 0.
4. **Burn captions via Remotion TalkingHead composition** (NOT FFmpeg ASS subtitles):
```
remotion_caption_burn.execute({
"input_path": "<composited_video>",
"output_path": "<project>/assets/video/captioned.mp4",
"segments": <transcript_segments>,
"corrections": <corrections_dict>,
"words_per_page": 4,
"font_size": 52,
"highlight_color": "#22D3EE",
"overlays": <overlay_list_from_scene_plan>
})
```
5. **Mix background music** (ducked at 15% volume under speech):
```
audio_mixer.execute({
"operation": "duck",
"video_path": "<captioned_video>",
"music_path": "<bg_music>",
"music_volume": 0.15,
"output_path": "<project>/assets/video/with_music.mp4"
})
```
6. **Final encode** to target platform specs (see Step 6 below).
### Step 3d: Build Showcase Cards (if multi-clip reel)
If the output is a reel with showcase clips, use `showcase_card` for each:
```
skills/pipelines/talking-head/scene-director.md
+48
View File
@@ -18,6 +18,34 @@ You are not just a processor. You are a creative director. Your job is to figure
## Process
### Step 0: Footage Analysis
Before watching the content, analyze the raw footage to understand the physical setup.
1. **Sample 5 frames** from the footage using ffmpeg (evenly spaced across the duration):
```
ffmpeg -i <footage> -vf "select='not(mod(n\,TOTAL_FRAMES/5))'" -vsync vfr -frames:v 5 frame_%02d.png
```
2. **Run visual_qa or histogram analysis** on each sampled frame to detect:
- **Background type:** Green screen / blue screen / natural background. Green/blue screens show a dominant narrow-band color spike in the histogram. Use `visual_qa` with prompt "Is this a green screen or blue screen background?" for confirmation.
- **Speaker position:** Center, left, or right of frame. Estimate the approximate bounding box (e.g., "speaker occupies center 40% of frame, from x=30% to x=70%").
- **Lighting quality:** Even studio lighting, harsh shadows, backlit, mixed color temperature. Note any issues that may affect chroma keying.
3. **Green screen detected?**
- If yes, note that `green_screen_processor` tool will be needed in the compose stage.
- Record the detected screen color (green or blue) and estimated uniformity.
- The compose-director will use this to run chroma key removal and composite onto an animated background.
4. **Measure speaker safe zone:**
- From the speaker's bounding box, determine where graphics can be placed WITHOUT overlapping the speaker.
- For a centered speaker: left panel and right panel are safe for overlays.
- For a left-positioned speaker: right panel is the primary safe zone.
- For a right-positioned speaker: left panel is the primary safe zone.
- Upper third and lower third are generally safe regardless of speaker position.
Record all findings in the scene plan metadata for downstream stages.
### Step 1: Watch & Listen — Understand the Content
**This is the most important step. Do not skip it.**
@@ -52,6 +80,24 @@ Based on your content analysis, propose **on-screen graphics** that will appear
| **Callout/quote** | `callout` | When the speaker makes a key point worth emphasizing |
| **Lower third** | `text_card` | Speaker identification at the start |
**Remotion Component Constraints:**
| Component | Min Width | 720px Portrait? | Value Type |
|-----------|-----------|-----------------|------------|
| comparison | 900px | NO -> use 2x stat_card | string |
| kpi_grid | 720px | YES | numeric ONLY (no "15+") |
| bar_chart | 500px | YES | numeric |
| stat_card | 300px | YES | string OK |
| callout | 400px | YES | string |
| hero_title | 400px | YES | string |
| line_chart | 500px | YES | numeric |
| progress_bar | 600px | YES | numeric |
| stat_reveal | 300px | YES | string OK |
When a component's minimum width exceeds the available space (e.g., `comparison` at 900px won't fit in a 720px portrait frame), substitute with the recommended alternative. For `comparison`, use two `stat_card` components shown sequentially instead.
For `kpi_grid`, values MUST be numeric (e.g., `4.8`, `73`, `2400`). String values like `"15+"` or `"$4.8B"` will cause rendering errors. Use `stat_card` for string-formatted numbers.
**Overlay planning rules:**
- **Don't over-overlay.** 3-6 overlays per minute of final video is the sweet spot. More than that is distracting.
- **Time overlays to speech.** Each overlay should appear when the speaker says the relevant words, not before or after.
@@ -92,6 +138,8 @@ Format your proposal clearly:
**Estimated final duration:** ~Xs (from Xs raw)
```
**IMPORTANT: When outputting the overlay plan, ALSO generate the actual Remotion JSON props file (`greenscreen-bg.json`) -- do not just describe scenes in prose.** The JSON props file should be a complete, valid input for the Remotion TalkingHead composition, including all overlay definitions, timing, colors, and content. Save it to `<project>/public/demo-props/` or the project's props directory.
Wait for user approval before proceeding. The user may:
- Approve as-is
- Add/remove overlays