freedak/OpenMontage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Talking-head pipeline: 8 new tools, Remotion TalkingHead composition, and skill rewrites

Browse Source 
New tools: face_tracker, visual_qa, eye_enhance, auto_reframe, remotion_caption_burn, showcase_card, silence_cutter. Updated audio_mixer with segmented_music operation and subtitle_gen with ASR corrections. Registered TalkingHead composition in Root.tsx. Rewrote compose/edit/scene director skills for full enhancement chain, Remotion captions, multi-clip assembly, and visual QA. Gitignore cleanup: exclude test demo-props, downloaded music, and generated images.
This commit is contained in:
calesthio
2026-04-01 10:00:15 -07:00
parent 237af7fb5c
commit 358b8647f5
17 changed files with 3467 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/pipelines/talking-head/compose-director.md
+184 -20
View File
@@ -18,51 +18,215 @@ You have edit decisions and an asset manifest. Your job is to render the final t
### Step 1: Run Enhancement Chain
Apply video enhancements in order:
1. **Face enhancement** (if face_enhance tool available) — sharpen faces
2. **Color grading** (if color_grade tool available) — apply a profile
3. **Audio enhancement** (if audio_enhance tool available) — noise reduction, normalization
1. **Face enhancement** (if `face_enhance` tool available) — apply `talking_head_standard` preset
2. **Eye enhancement** (if `eye_enhance` tool available) — under-eye dark circle removal + eye brightening
3. **Color grading** (if `color_grade` tool available) — apply a profile
4. **Audio enhancement** (if `audio_enhance` tool available) — noise reduction, normalization
Each step is optional — check tool availability first.
### Step 2: Burn Subtitles
**Eye enhancement** — removes under-eye dark circles and brightens eyes using MediaPipe Face Mesh landmark detection:
```
eye_enhance.execute({
"input_path": "<face_enhanced_video>",
"output_path": "<project>/assets/video/eye_enhanced.mp4",
"operations": ["dark_circles", "brighten_eyes"],
"dark_circle_intensity": 0.4, # 0-1, subtle is better
"eye_brighten_intensity": 0.3,
})
```
**Important:** Keep intensities low (0.2-0.5). Over-processing makes eyes look unnatural. Always compare before/after.
### Step 1b: Speed Adjustment (if requested)
If the user wants the video sped up or slowed down, use `video_trimmer`:
```
video_trimmer.execute({
"operation": "speed",
"input_path": "<enhanced_video>",
"output_path": "<project>/assets/video/speed_adjusted.mp4",
"speed_factor": 1.25 # 0.5x (slow), 1.25x, 1.5x, 2x (fast)
})
```
Common speed factors:
| Factor | Use Case |
|--------|----------|
| `0.5` | Slow-mo for dramatic effect |
| `1.0` | Normal (no change) |
| `1.25` | Slightly faster — tighter pacing without sounding unnatural |
| `1.5` | Noticeably faster — good for recaps or condensed content |
| `2.0` | Double speed — time-lapse effect |
Apply speed AFTER enhancements, BEFORE reframing.
### Step 2: Auto-Reframe (if target platform requires it)
If the target platform requires a different aspect ratio (e.g. Instagram Reels = 9:16), use `auto_reframe`:
```
auto_reframe.execute({
"input_path": "<enhanced_video>",
"output_path": "<project>/renders/reframed.mp4",
"target_aspect": "portrait", # 9:16 for Reels/TikTok/Shorts
"smoothing_window": 15, # smooth camera pan
"face_padding": 0.4, # 40% padding around face
})
```
**Aspect ratio presets:**
| Preset | Ratio | Platform |
|--------|-------|----------|
| `portrait` | 9:16 | Instagram Reels, TikTok, YouTube Shorts |
| `square` | 1:1 | Instagram Feed |
| `landscape` | 16:9 | YouTube, LinkedIn |
| `vertical_4_5` | 4:5 | Instagram portrait post |
The tool automatically runs face detection and keeps the speaker centered. If MediaPipe is not installed, falls back to center-crop.
**Important:** Run auto_reframe AFTER face_enhance and color_grade but BEFORE burning subtitles. Subtitles need to be positioned for the final aspect ratio.
### Step 3: Burn Subtitles
**Preferred: Remotion captions** (if `remotion_caption_burn` tool available):
```
remotion_caption_burn.execute({
"input_path": "<reframed_or_enhanced_video>",
"output_path": "<project>/assets/video/captioned.mp4",
"segments": <transcript_segments_from_asset_manifest>,
"corrections": {"cloud": "Claude", "co-pilot": "Copilot"},
"words_per_page": 4,
"font_size": 52,
"highlight_color": "#22D3EE",
})
```
Remotion renders animated word-by-word captions at the bottom of the frame with active word highlighting. Captions are positioned away from the face.
**Fallback: FFmpeg subtitles** (if Remotion unavailable):
Use `video_compose` with `burn_subtitles` operation:
- Input: enhanced video (or raw if no enhancements)
- Input: reframed video (or enhanced video if no reframe needed)
- Subtitle file from asset manifest
- Style from playbook
- For vertical (9:16) output: position subtitles in the lower 20% of frame with `MarginV=100`
- **Never** position subtitles in the center of the frame — they will occlude the face
### Step 3: Mix Audio
### Step 3b: Build Showcase Cards (if multi-clip reel)
Use `audio_mixer` to:
- Layer original audio with any background music
If the output is a reel with showcase clips, use `showcase_card` for each:
```
showcase_card.execute({
"input_path": "<showcase_video>",
"output_path": "<project>/assets/video/sc_<name>.mp4",
"title": "VIDEO TITLE",
"subtitle": "Description | Style | Cost: $0.15",
"background_color": "0x0A0F1A",
})
```
This creates letterboxed 9:16 cards with typography.
### Step 4: Assemble Multi-Clip (if applicable)
If the output has multiple segments (e.g. talking head + showcase clips), use `video_stitch`:
```
video_stitch.execute({
"operation": "stitch",
"clips": ["intro.mp4", "showcase1.mp4", ..., "outro.mp4"],
"output_path": "<project>/renders/assembled.mp4",
"transition": "crossfade", # or "fade" for fade-through-black
"transition_duration": 0.5,
})
```
**Transition guidance:**
- `crossfade` (fade): smooth blend between talking head and showcase
- `fade` (fade-through-black): brief dip to black between showcase clips
- Mix transition types: use `crossfade` for talk→showcase, `fade` between showcases
### Step 5: Mix Audio
Use `audio_mixer` to layer background music:
**For multi-clip reels** — use `segmented_music` to play music only during talking head sections:
```
audio_mixer.execute({
"operation": "segmented_music",
"video_path": "<assembled_video>",
"music_path": "<bg_music>",
"music_volume": 0.20,
"segments": [
{"start": 0, "end": 17.0}, # intro speech
{"start": 167.0, "end": 175.0} # outro speech
],
"fade_duration": 0.5,
"output_path": "<project>/renders/final.mp4",
})
```
**For single talking-head videos** — use `duck` or `full_mix`:
- Layer original audio with background music
- Apply ducking if music is present
- Normalize final levels
### Step 4: Final Encode
### Step 6: Final Encode
Use `video_compose` with `encode` operation:
- Apply target media profile (youtube_landscape, tiktok, etc.)
- Apply target media profile (youtube_landscape, tiktok, instagram_reels, etc.)
- Two-pass encoding for quality
### Step 5: Verify Output
### Step 7: Visual QA
- Check file exists and is playable
- Verify duration matches expectations
- Check audio is present
Use `visual_qa` to verify the output before declaring success:
```
visual_qa.execute({
"operation": "review",
"input_path": "<final_video>",
"timestamps": [3.0, 10.0, 25.0, 50.0, 100.0, 170.0],
})
```
Then **read each extracted frame** to verify:
- Captions are visible and positioned at the bottom (not on the face)
- Face enhancement is applied (skin looks smooth, not over-processed)
- Transitions are clean (no artifacts at transition points)
- Showcase cards have readable typography
### Step 6: Build Render Report
Also run probe validation:
```
visual_qa.execute({
"operation": "probe",
"input_path": "<final_video>",
"expected": {
"width": 1080, "height": 1920,
"has_audio": true,
"pixel_format": "yuv420p"
},
})
```
Document output: path, format, resolution, duration, file size.
And check audio levels:
```
visual_qa.execute({
"operation": "audio_levels",
"input_path": "<final_video>",
"timestamps": [5.0, 50.0, 170.0],
})
```
Verify: speech sections have higher volume than showcase sections (confirms music placement).
### Step 7: Self-Evaluate
### Step 8: Build Render Report
Document output: path, format, resolution, duration, file size, QA results.
### Step 9: Self-Evaluate
| Criterion | Question |
|-----------|----------|
| **Playability** | Does the video play without errors? |
| **Quality** | Are enhancements applied correctly? |
| **Audio** | Is speech clear with balanced levels? |
| **Subtitles** | Are subtitles visible and synced? |
| **Framing** | If reframed — is the face centered? No important content cropped? |
| **Audio** | Is speech clear with balanced levels? Music only during intended segments? |
| **Subtitles** | Are captions visible at the bottom? Not occluding the face? Word highlighting working? |
| **Transitions** | Are transitions clean? Correct type (crossfade vs fadeblack)? |
| **Showcase** | Are showcase cards properly letterboxed with readable typography? |
### Step 8: Submit
### Step 10: Submit
Validate the render_report against the schema and persist via checkpoint.
skills/pipelines/talking-head/edit-director.md
+31 -7
View File
@@ -14,39 +14,63 @@ You have a scene plan and asset manifest. Your job is to assemble the edit decis
## Process
### Step 1: Define Primary Cut
### Step 1: Apply Silence Cuts (if planned)
If the scene plan includes silence removal, run `silence_cutter` before defining cuts:
```
silence_cutter.execute({
"input_path": "<raw_footage>",
"mode": "remove", # or "speed_up" for less jarring result
"silence_threshold_db": -35,
"min_silence_duration": 0.5,
"padding_seconds": 0.08, # prevents clipped words
"output_path": "<project>/assets/video/footage_cut.mp4"
})
```
**Choosing the mode:**
- `remove` — Hard jump cuts. Best for fast-paced social content (Reels, TikTok, Shorts)
- `speed_up` — Fast-forwards through silence at 6x. Less jarring for longer-form content (YouTube, LinkedIn)
Present the result to the user: "Removed X seconds of silence (Y%) — output is now Z seconds."
Use the cut footage as the source for all subsequent steps.
### Step 2: Define Primary Cut
For talking-head, the primary cut is usually the full footage (or trimmed segments). Create cuts that:
- Reference the raw footage as source
- Reference the raw footage (or silence-cut footage) as source
- Use timestamps from the script sections
- Apply any trim decisions (cut dead air, false starts)
### Step 2: Configure Subtitles
### Step 3: Configure Subtitles
- Enable subtitles with playbook-compatible styling
- Reference the subtitle asset from the manifest
- Set position (usually bottom-center)
### Step 3: Configure Audio
### Step 4: Configure Audio
- Set narration to the raw footage audio
- If background music is desired, configure ducking
- Set music volume per playbook
### Step 4: Plan Enhancements
### Step 5: Plan Enhancements
If the scene plan includes overlays:
- Add overlay cuts for text cards, lower thirds
- Time them to match speech content
### Step 5: Self-Evaluate
### Step 6: Self-Evaluate
| Criterion | Question |
|-----------|----------|
| **Coverage** | Do cuts span the full intended duration? |
| **Silence** | Were silence cuts applied if planned? What % was removed? |
| **Subtitles** | Are subtitles enabled and styled? |
| **Audio** | Is audio configuration complete? |
### Step 6: Submit
### Step 7: Submit
Validate the edit_decisions against the schema and persist via checkpoint.
skills/pipelines/talking-head/scene-director.md
+47 -5
View File
@@ -11,28 +11,70 @@ You have a script (from transcription) and raw footage. Your job is to create a
| Schema | `schemas/artifacts/scene_plan.schema.json` | Artifact validation |
| Prior artifacts | Script, Brief | Section timing and context |
| Tools | `frame_sampler` (optional) | Extract representative frames |
| Tools | `face_tracker` (optional) | Analyze speaker face position for reframing |
| Tools | `silence_cutter` (optional) | Detect silence for jump cut planning |
## Process
### Step 1: Plan Base Scenes
### Step 1: Analyze Footage (if tools available)
**Face tracking** — If `face_tracker` is available, run it on the raw footage:
```
face_tracker.execute({
"input_path": "<raw_footage>",
"sample_fps": 5
})
```
This outputs per-frame face bounding boxes. Use this data to:
- Decide if reframing is needed (e.g. speaker is off-center for vertical crop)
- Identify sections where the speaker moves significantly (needs dynamic crop)
- Note face position for auto_reframe in the compose stage
**Silence detection** — If `silence_cutter` is available, run in `mark` mode:
```
silence_cutter.execute({
"input_path": "<raw_footage>",
"mode": "mark",
"silence_threshold_db": -35,
"min_silence_duration": 0.5
})
```
This outputs silence/speech segment timestamps. Use this to:
- Plan which segments should be jump-cut or sped up
- Identify dead air, false starts, and long pauses
- Estimate the final video duration after cuts
- Present the user with a summary: "Found X seconds of silence across Y segments — recommend removing?"
### Step 2: Plan Base Scenes
For talking-head, the base is simple: one scene per script section, all type `talking_head`. The raw footage IS the scene.
### Step 2: Plan Enhancement Scenes
### Step 3: Plan Enhancement Scenes
Based on script enhancement cues, plan overlay scenes:
- Text cards for key terms or statistics
- Lower thirds for speaker identification
- B-roll suggestions for topic illustrations
### Step 3: Build Scene Plan
### Step 4: Plan Reframing & Cuts
If the target platform requires a different aspect ratio (e.g. Instagram Reels = 9:16):
- Note `auto_reframe` should be applied in the compose stage
- Record the target aspect ratio in the scene plan
- If face tracking data shows significant speaker movement, note that dynamic crop is needed
If silence detection found segments to cut:
- Record the recommended cut mode (`remove` or `speed_up`) in the scene plan
- Note padding preferences (default 0.08s to avoid clipping words)
### Step 5: Build Scene Plan
Create a scene per section with:
- Type: `talking_head` (primary)
- Timing from script sections
- Required assets: subtitle file, any overlay images
### Step 4: Self-Evaluate
### Step 6: Self-Evaluate
| Criterion | Question |
|-----------|----------|
@@ -40,6 +82,6 @@ Create a scene per section with:
| **Enhancement** | Are overlay opportunities identified? |
| **Feasibility** | Can all required assets be generated? |
### Step 5: Submit
### Step 7: Submit
Validate the scene_plan against the schema and persist via checkpoint.