Initial release — OpenMontage: the first open-source agentic video production system
11 production pipelines, 47 tools, 124 agent skills. Supports cloud APIs (fal.ai, OpenAI, ElevenLabs, Suno, HeyGen, Runway) and free local providers (diffusers, Piper TTS, WAN 2.1, Hunyuan, CogVideo). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
calesthio
@@ -0,0 +1,50 @@
|
||||
# Asset Director — Talking Head Pipeline
|
||||
|
||||
## When to Use
|
||||
|
||||
You have a scene plan and script. Your job is to generate the supporting assets for a talking-head video: subtitles, extracted audio, and any overlay graphics.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/asset_manifest.schema.json` | Artifact validation |
|
||||
| Prior artifacts | Scene plan, Script | What assets to create |
|
||||
| Tools | `subtitle_gen`, `audio_mixer`, `image_selector` (optional) | Asset generation |
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Generate Subtitles
|
||||
|
||||
Use the transcription data from the script stage to create:
|
||||
- SRT or ASS subtitle file with word-level timing
|
||||
- Style subtitles per the playbook (font, size, color, position)
|
||||
|
||||
### Step 2: Extract and Process Audio
|
||||
|
||||
- Extract audio track from raw footage
|
||||
- Apply noise reduction if needed (via `audio_mixer`)
|
||||
- Normalize audio levels
|
||||
|
||||
### Step 3: Generate Overlays (Optional)
|
||||
|
||||
If the scene plan includes overlay scenes:
|
||||
- Generate text card images
|
||||
- Generate lower third graphics
|
||||
- Create any B-roll placeholders
|
||||
|
||||
### Step 4: Build Asset Manifest
|
||||
|
||||
Document all generated assets with paths, types, and tool references.
|
||||
|
||||
### Step 5: Self-Evaluate
|
||||
|
||||
| Criterion | Question |
|
||||
|-----------|----------|
|
||||
| **Subtitles** | Do subtitles exist and match speech timing? |
|
||||
| **Audio** | Is audio clean and normalized? |
|
||||
| **Files** | Do all asset paths point to existing files? |
|
||||
|
||||
### Step 6: Submit
|
||||
|
||||
Validate the asset_manifest against the schema and persist via checkpoint.
|
||||
@@ -0,0 +1,68 @@
|
||||
# Compose Director — Talking Head Pipeline
|
||||
|
||||
## When to Use
|
||||
|
||||
You have edit decisions and an asset manifest. Your job is to render the final talking-head video: apply the enhancement chain, burn subtitles, mix audio, and encode to the target profile.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/render_report.schema.json` | Artifact validation |
|
||||
| Prior artifacts | Edit decisions, Asset manifest | Render inputs |
|
||||
| Tools | `video_compose`, `audio_mixer` | Rendering |
|
||||
| Media profiles | `lib/media_profiles.py` | Output format |
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Run Enhancement Chain
|
||||
|
||||
Apply video enhancements in order:
|
||||
1. **Face enhancement** (if face_enhance tool available) — sharpen faces
|
||||
2. **Color grading** (if color_grade tool available) — apply a profile
|
||||
3. **Audio enhancement** (if audio_enhance tool available) — noise reduction, normalization
|
||||
|
||||
Each step is optional — check tool availability first.
|
||||
|
||||
### Step 2: Burn Subtitles
|
||||
|
||||
Use `video_compose` with `burn_subtitles` operation:
|
||||
- Input: enhanced video (or raw if no enhancements)
|
||||
- Subtitle file from asset manifest
|
||||
- Style from playbook
|
||||
|
||||
### Step 3: Mix Audio
|
||||
|
||||
Use `audio_mixer` to:
|
||||
- Layer original audio with any background music
|
||||
- Apply ducking if music is present
|
||||
- Normalize final levels
|
||||
|
||||
### Step 4: Final Encode
|
||||
|
||||
Use `video_compose` with `encode` operation:
|
||||
- Apply target media profile (youtube_landscape, tiktok, etc.)
|
||||
- Two-pass encoding for quality
|
||||
|
||||
### Step 5: Verify Output
|
||||
|
||||
- Check file exists and is playable
|
||||
- Verify duration matches expectations
|
||||
- Check audio is present
|
||||
|
||||
### Step 6: Build Render Report
|
||||
|
||||
Document output: path, format, resolution, duration, file size.
|
||||
|
||||
### Step 7: Self-Evaluate
|
||||
|
||||
| Criterion | Question |
|
||||
|-----------|----------|
|
||||
| **Playability** | Does the video play without errors? |
|
||||
| **Quality** | Are enhancements applied correctly? |
|
||||
| **Audio** | Is speech clear with balanced levels? |
|
||||
| **Subtitles** | Are subtitles visible and synced? |
|
||||
|
||||
### Step 8: Submit
|
||||
|
||||
Validate the render_report against the schema and persist via checkpoint.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Edit Director — Talking Head Pipeline
|
||||
|
||||
## When to Use
|
||||
|
||||
You have a scene plan and asset manifest. Your job is to assemble the edit decision list for a talking-head video: primarily keeping the full footage with subtitle overlay and optional enhancements.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/edit_decisions.schema.json` | Artifact validation |
|
||||
| Prior artifacts | Scene plan, Asset manifest, Script | Edit inputs |
|
||||
| Playbook | Active style playbook | Transition and pacing rules |
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Define Primary Cut
|
||||
|
||||
For talking-head, the primary cut is usually the full footage (or trimmed segments). Create cuts that:
|
||||
- Reference the raw footage as source
|
||||
- Use timestamps from the script sections
|
||||
- Apply any trim decisions (cut dead air, false starts)
|
||||
|
||||
### Step 2: Configure Subtitles
|
||||
|
||||
- Enable subtitles with playbook-compatible styling
|
||||
- Reference the subtitle asset from the manifest
|
||||
- Set position (usually bottom-center)
|
||||
|
||||
### Step 3: Configure Audio
|
||||
|
||||
- Set narration to the raw footage audio
|
||||
- If background music is desired, configure ducking
|
||||
- Set music volume per playbook
|
||||
|
||||
### Step 4: Plan Enhancements
|
||||
|
||||
If the scene plan includes overlays:
|
||||
- Add overlay cuts for text cards, lower thirds
|
||||
- Time them to match speech content
|
||||
|
||||
### Step 5: Self-Evaluate
|
||||
|
||||
| Criterion | Question |
|
||||
|-----------|----------|
|
||||
| **Coverage** | Do cuts span the full intended duration? |
|
||||
| **Subtitles** | Are subtitles enabled and styled? |
|
||||
| **Audio** | Is audio configuration complete? |
|
||||
|
||||
### Step 6: Submit
|
||||
|
||||
Validate the edit_decisions against the schema and persist via checkpoint.
|
||||
@@ -0,0 +1,446 @@
|
||||
# Executive Producer — Talking Head Pipeline
|
||||
|
||||
## When to Use
|
||||
|
||||
You are the **Executive Producer (EP)** for a talking-head video project. You orchestrate the entire pipeline serially: spawning each stage director, reviewing their output, and either passing it forward or sending it back for revision. You are the stateful brain; the directors are stateless workers.
|
||||
|
||||
**You replace the default parallel/sequential execution model.** Instead of running all stages blindly, you exercise judgment at every gate.
|
||||
|
||||
## Why This Exists
|
||||
|
||||
The talking-head pipeline transforms raw footage of a person speaking into a polished, subtitled video. Without an EP:
|
||||
- Transcript errors propagate silently through all downstream stages
|
||||
- Subtitle timing drifts from speech with no feedback to correct it
|
||||
- Scene coverage gaps leave dead air in the final output
|
||||
- No A/V sync validation before the final render
|
||||
- No ability to send a single stage back without re-running everything
|
||||
- Enhancement decisions (face, color, audio) are made without context of the full picture
|
||||
|
||||
The EP solves all of these by maintaining cumulative state and applying judgment at each gate.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Pipeline | `pipeline_defs/talking-head.yaml` | Stage definitions, review focus, success criteria |
|
||||
| Skills | All 7 director skills + `meta/reviewer` | Stage execution knowledge |
|
||||
| Schemas | All artifact schemas | Validation |
|
||||
| Playbook | `clean-professional` (or user-selected) | Quality constraints |
|
||||
| Tools | Full tool registry | Available capabilities |
|
||||
|
||||
## Key Difference from Explainer EP
|
||||
|
||||
The talking-head pipeline is **footage-first**, not idea-first:
|
||||
|
||||
| Aspect | Explainer EP | Talking-Head EP |
|
||||
|--------|-------------|-----------------|
|
||||
| Source material | None — generates everything | Raw footage provided up front |
|
||||
| Script stage | Writes from scratch | Extracts from transcription |
|
||||
| Core challenge | Creative generation quality | Transcript accuracy + timing |
|
||||
| Budget model | Moderate (TTS + image gen) | Low (mostly processing, optional overlays) |
|
||||
| Duration source | Target set in proposal | Determined by raw footage length |
|
||||
| Critical sync | Narration ↔ visual duration | Subtitles ↔ speech timing |
|
||||
| Pre-production | Research + proposal (2 stages) | Idea (1 stage) — no research needed |
|
||||
|
||||
## Cumulative State
|
||||
|
||||
The EP maintains a running state object that flows through the entire pipeline:
|
||||
|
||||
```
|
||||
EP_STATE:
|
||||
pipeline: talking-head
|
||||
playbook: <selected playbook name, default: clean-professional>
|
||||
raw_footage_path: <path to source footage>
|
||||
raw_footage_duration_seconds: <from ffprobe>
|
||||
raw_footage_resolution: <from ffprobe>
|
||||
target_duration_seconds: <from brief, may be shorter than raw>
|
||||
budget_total_usd: <from user or default: $0.50>
|
||||
budget_spent_usd: 0.0
|
||||
budget_remaining_usd: <budget_total>
|
||||
|
||||
# Accumulated from each stage (7 stages)
|
||||
artifacts:
|
||||
idea: null # → brief
|
||||
script: null # → script (transcript-based)
|
||||
scene_plan: null # → scene_plan
|
||||
assets: null # → asset_manifest
|
||||
edit: null # → edit_decisions
|
||||
compose: null # → render_report
|
||||
publish: null # → publish_log
|
||||
|
||||
# Transcript tracking (the core of talking-head quality)
|
||||
transcript_segments: [] # word-level timestamped segments from transcriber
|
||||
transcript_confidence: null # average word confidence score
|
||||
transcript_language: null # detected language
|
||||
subtitle_sync_offsets: {} # section_id → drift_seconds (positive = subtitle late)
|
||||
|
||||
# Cross-stage tracking
|
||||
total_footage_seconds: 0
|
||||
total_edit_seconds: 0 # may differ from footage if trimmed
|
||||
style_anchors: {} # consistency tokens for overlays
|
||||
revision_counts: {} # stage_name → number of revisions
|
||||
issues_log: [] # all issues found, with resolution status
|
||||
|
||||
# Enhancement tracking
|
||||
enhancements_applied: [] # face_enhance, color_grade, audio_enhance
|
||||
audio_profile: # from raw footage analysis
|
||||
has_background_noise: null
|
||||
audio_channels: null
|
||||
sample_rate: null
|
||||
```
|
||||
|
||||
## Execution Protocol
|
||||
|
||||
### Phase 0: Initialize
|
||||
|
||||
1. Load the pipeline manifest (`talking-head.yaml`)
|
||||
2. Load the playbook (from user selection or default `clean-professional`)
|
||||
3. Set budget from configuration or user input (default: $0.50 — talking-head is mostly processing)
|
||||
4. Probe the raw footage with ffprobe: duration, resolution, fps, audio channels, codec
|
||||
5. Store footage metadata in EP_STATE
|
||||
6. Initialize EP_STATE
|
||||
|
||||
### Phase 1: Execute Stages Serially
|
||||
|
||||
For each stage in order: `idea → script → scene_plan → assets → edit → compose → publish`
|
||||
|
||||
```
|
||||
EXECUTE_STAGE(stage_name):
|
||||
|
||||
1. PREPARE
|
||||
- Load the director skill for this stage
|
||||
- Inject EP_STATE as context (prior artifacts, budget remaining, style anchors)
|
||||
- Inject any EP feedback from previous revision attempts
|
||||
|
||||
2. SPAWN DIRECTOR
|
||||
- The director executes its full process (as defined in its skill MD)
|
||||
- Director produces an artifact
|
||||
|
||||
3. REVIEW (EP performs this, not a separate reviewer)
|
||||
- Schema validation against artifact schema
|
||||
- Check review_focus items from pipeline manifest
|
||||
- Check success_criteria from pipeline manifest
|
||||
- Cross-check against playbook constraints
|
||||
- Run EP-SPECIFIC CROSS-STAGE CHECKS (see below)
|
||||
|
||||
4. GATE DECISION
|
||||
If PASS:
|
||||
- Store artifact in EP_STATE
|
||||
- Update cumulative tracking (budget, durations, etc.)
|
||||
- Log: "[stage] PASSED — moving to next stage"
|
||||
- Continue to next stage
|
||||
|
||||
If REVISE:
|
||||
- Increment revision_counts[stage_name]
|
||||
- If revision_counts[stage_name] >= 3:
|
||||
- PASS WITH WARNINGS (never block forever)
|
||||
- Log unresolved issues
|
||||
- Else:
|
||||
- Compose specific feedback for the director
|
||||
- Re-run SPAWN DIRECTOR with feedback injected
|
||||
- Re-run REVIEW
|
||||
|
||||
If SEND_BACK(target_stage):
|
||||
- This is the EP's special power: send work BACK to a prior stage
|
||||
- Only used when a downstream discovery invalidates upstream work
|
||||
- Example: Subtitle sync check reveals transcript has wrong timestamps
|
||||
→ Send back to script director: "Re-transcribe section 3. Timestamps are off."
|
||||
- Re-execute from target_stage forward (artifacts after target are invalidated)
|
||||
- Max 1 send-back per stage pair (prevent infinite loops)
|
||||
```
|
||||
|
||||
### Phase 2: Final Quality Assurance
|
||||
|
||||
After all 7 stages complete, the EP performs a holistic review:
|
||||
|
||||
```
|
||||
FINAL_QA:
|
||||
1. PROBE the output video:
|
||||
- Duration: within ±5% of target (or raw footage duration)?
|
||||
- Resolution: matches target or raw footage resolution?
|
||||
- Audio: speech audible throughout? No clipping? Balanced levels?
|
||||
- File: valid container, reasonable size?
|
||||
|
||||
2. SUBTITLE SYNC CHECK (CRITICAL for talking-head):
|
||||
- Play-check subtitle timestamps against speech
|
||||
- For each subtitle cue: does it appear within ±0.3s of the spoken word?
|
||||
- Flag any section where subtitles are visibly out of sync
|
||||
- Tolerance: ±0.3 seconds (tighter than explainer because speech is the content)
|
||||
|
||||
3. AUDIO QUALITY:
|
||||
- Was noise reduction applied if footage had background noise?
|
||||
- Are audio levels normalized? (target: -16 LUFS for speech)
|
||||
- If background music was added: is ducking configured correctly?
|
||||
|
||||
4. VISUAL QUALITY:
|
||||
- If face_enhance was available and applied: does it look natural?
|
||||
- If color_grade was available and applied: is it consistent?
|
||||
- If overlays were added: do they appear at the right timestamps?
|
||||
|
||||
5. BUDGET RECONCILIATION:
|
||||
- Total actual spend vs. budget
|
||||
- Log per-stage cost breakdown
|
||||
|
||||
6. DECISION:
|
||||
If all checks pass → APPROVE for publish stage
|
||||
If issues found → Send back to the specific stage(s) that can fix them
|
||||
- Subtitle timing → asset director (regenerate subtitles)
|
||||
- Audio issues → compose director (remix)
|
||||
- Visual enhancement issues → compose director (re-render)
|
||||
- Coverage gaps → scene director (replan) or edit director (re-cut)
|
||||
- Transcript errors → script director (re-transcribe)
|
||||
```
|
||||
|
||||
## EP-Specific Cross-Stage Checks
|
||||
|
||||
These checks use information accumulated across stages — something no individual director can do.
|
||||
|
||||
### After IDEA stage:
|
||||
```
|
||||
CHECK: Footage viability
|
||||
- Does the footage have audio? (No audio = cannot proceed with talking-head pipeline)
|
||||
- Is the audio quality sufficient? (Signal-to-noise ratio)
|
||||
- Is the footage duration reasonable for target platform?
|
||||
- If duration > 3x target: flag that significant trimming is needed
|
||||
- Note: Idea stage DOES checkpoint with user — this is the approval gate
|
||||
```
|
||||
|
||||
### After SCRIPT stage:
|
||||
```
|
||||
CHECK: Transcript quality (CRITICAL — everything downstream depends on this)
|
||||
- Average word confidence score (from transcriber output)
|
||||
- If avg_confidence < 0.8:
|
||||
REVISE: "Transcript confidence is low ({X}). Try model: large-v3 if not already used.
|
||||
If still low, flag specific low-confidence sections for manual review."
|
||||
- Spot-check: do timestamps increase monotonically?
|
||||
- Spot-check: are there gaps > 2 seconds with no words? (may indicate missed speech)
|
||||
- Store transcript_segments in EP_STATE for downstream subtitle generation
|
||||
|
||||
CHECK: Section boundaries
|
||||
- Do sections align with natural topic changes?
|
||||
- Are timestamps within the raw footage duration?
|
||||
- Any section longer than 60s? (May need splitting for better scene planning)
|
||||
```
|
||||
|
||||
### After SCENE_PLAN stage:
|
||||
```
|
||||
CHECK: Full coverage
|
||||
- Sum all scene durations
|
||||
- Compare to raw footage duration (or target edit duration)
|
||||
- Gaps > 1 second: REVISE scene_plan
|
||||
- Overlaps: REVISE scene_plan
|
||||
|
||||
CHECK: Enhancement feasibility
|
||||
- For each planned enhancement (face, color, overlay):
|
||||
Verify the required tool exists in the registry
|
||||
- If face_enhance planned but unavailable: remove from plan, log warning
|
||||
- If overlay images planned: verify image tools are available
|
||||
|
||||
CHECK: Overlay alignment
|
||||
- If overlays are planned at specific timestamps, verify those timestamps
|
||||
fall within actual scene boundaries from the transcript
|
||||
```
|
||||
|
||||
### After ASSETS stage:
|
||||
```
|
||||
CHECK: Subtitle sync (CRITICAL for talking-head)
|
||||
- Compare subtitle cue timestamps to transcript word timestamps
|
||||
- For each cue: |subtitle_start - word_start| < 0.3s
|
||||
- Store sync offsets in EP_STATE.subtitle_sync_offsets
|
||||
- If any offset > 0.5s: REVISE assets: "Subtitle cue {id} is {X}s off.
|
||||
Re-generate from original transcript segments."
|
||||
|
||||
CHECK: Audio extraction
|
||||
- Was audio extracted from raw footage?
|
||||
- Was noise reduction applied if needed?
|
||||
- Are audio levels in a reasonable range?
|
||||
|
||||
CHECK: Budget gate
|
||||
- If budget_spent > budget_total * 0.8 and stages remain:
|
||||
Alert: "80% budget consumed with {N} stages remaining"
|
||||
Adjust remaining stages to skip optional enhancements
|
||||
```
|
||||
|
||||
### After EDIT stage:
|
||||
```
|
||||
CHECK: Timeline completeness
|
||||
- Verify edit decisions cover 0 to total_edit_duration with no gaps
|
||||
- Verify all cut source files reference existing paths from asset_manifest
|
||||
- Verify subtitle configuration is present and points to valid subtitle file
|
||||
|
||||
CHECK: Trim validation
|
||||
- If footage was trimmed (edit is shorter than raw): are the right sections kept?
|
||||
- Do the kept sections match the scene_plan?
|
||||
- Are transitions between cuts smooth (no jump cuts unless intentional)?
|
||||
```
|
||||
|
||||
### After COMPOSE stage:
|
||||
```
|
||||
CHECK: Output validation
|
||||
- ffprobe the output: duration, resolution, codec, audio channels
|
||||
- Duration drift > 5%: investigate which stage caused it
|
||||
- Audio missing: check audio extraction and mixing
|
||||
- Resolution wrong: check if face_enhance or color_grade changed it
|
||||
- Subtitles: if burn-in was requested, verify they're visible in output
|
||||
```
|
||||
|
||||
## Feedback Message Templates
|
||||
|
||||
When sending work back to a director, use these structured feedback messages:
|
||||
|
||||
### To Script Director:
|
||||
```
|
||||
EP FEEDBACK — Script Revision Required
|
||||
Reason: {reason}
|
||||
Specific issue: {transcript_quality / timestamp_error / section_boundary}
|
||||
Affected sections: {section_ids}
|
||||
Action: {re-transcribe / re-segment / re-align}
|
||||
Transcriber settings: {model / language hints if applicable}
|
||||
```
|
||||
|
||||
### To Scene Director:
|
||||
```
|
||||
EP FEEDBACK — Scene Plan Revision Required
|
||||
Reason: {reason}
|
||||
Affected scenes: {scene_ids}
|
||||
Constraint: {coverage / feasibility / timing}
|
||||
Available tools: {current tool registry status}
|
||||
```
|
||||
|
||||
### To Asset Director:
|
||||
```
|
||||
EP FEEDBACK — Asset Regeneration Required
|
||||
Reason: {reason}
|
||||
Affected assets: {asset_ids}
|
||||
Specific fix: {subtitle_resync / audio_renormalize / overlay_regen}
|
||||
Transcript reference: {original transcript segments for re-alignment}
|
||||
Budget remaining: ${remaining}
|
||||
```
|
||||
|
||||
### To Edit Director:
|
||||
```
|
||||
EP FEEDBACK — Edit Revision Required
|
||||
Reason: {reason}
|
||||
Specific issue: {gap_at_timestamp / invalid_reference / missing_subtitle_config}
|
||||
Asset manifest: {current valid asset paths}
|
||||
```
|
||||
|
||||
### To Compose Director:
|
||||
```
|
||||
EP FEEDBACK — Re-render Required
|
||||
Reason: {reason}
|
||||
Specific issue: {subtitle_sync / audio_quality / resolution / duration}
|
||||
Expected: {what the output should be}
|
||||
Actual: {what was produced}
|
||||
Enhancement adjustments: {skip/add face_enhance, color_grade, etc.}
|
||||
```
|
||||
|
||||
## Quality Gates Summary
|
||||
|
||||
| Gate | After Stage | What's Checked | Fail Action |
|
||||
|------|-------------|---------------|-------------|
|
||||
| G1 | idea | Footage viability, audio presence, user approval | Revise brief OR stop pipeline |
|
||||
| G2 | script | Transcript confidence, timestamps, section boundaries | Revise script (re-transcribe) |
|
||||
| G3 | scene_plan | Full coverage, enhancement feasibility, overlay alignment | Revise scene_plan |
|
||||
| G4 | assets | Subtitle sync, audio extraction, budget | Revise assets OR send-back to script |
|
||||
| G5 | edit | Timeline completeness, trim validation, subtitle config | Revise edit |
|
||||
| G6 | compose | Output probe, duration, audio, subtitle burn-in | Revise compose OR send-back to edit/assets |
|
||||
| G7 | publish | Metadata, packaging | Revise publish |
|
||||
| FINAL | all | Subtitle sync, audio quality, visual quality | Send-back to specific stage |
|
||||
|
||||
## Execution Limits (Anti-Loop Protection)
|
||||
|
||||
| Limit | Value | Rationale |
|
||||
|-------|-------|-----------|
|
||||
| Max revisions per stage | 3 | Prevent perfectionism loops |
|
||||
| Max send-backs per stage pair | 1 | Prevent ping-pong between stages |
|
||||
| Max total send-backs | 3 | Cap total pipeline re-work |
|
||||
| Max total budget | Configurable (default $0.50) | Hard stop on spending |
|
||||
| Max total wall-time | 10 minutes | Timeout for entire pipeline (shorter than explainer — less generation) |
|
||||
|
||||
After any limit is hit: **proceed with warnings**, never block indefinitely.
|
||||
|
||||
## Integration with Existing Skills
|
||||
|
||||
The EP doesn't replace any director skill — it wraps them. Each director skill continues to work exactly as documented. The EP adds:
|
||||
|
||||
1. **Context injection**: Directors receive EP_STATE with cross-stage information they couldn't access before
|
||||
2. **Feedback injection**: Directors receive specific revision instructions when sent back
|
||||
3. **Budget awareness**: Directors receive remaining budget and can adjust tool choices accordingly
|
||||
4. **Transcript continuity**: The EP carries transcript data forward, ensuring subtitle generation and edit decisions use the same source of truth
|
||||
|
||||
## Example EP Run (Abbreviated)
|
||||
|
||||
```
|
||||
[EP] Starting pipeline: talking-head v2.0
|
||||
[EP] Default budget: $0.50 | Playbook: clean-professional
|
||||
|
||||
[EP] Probing raw footage: interview_raw.mp4
|
||||
[EP] → Duration: 4m22s | Resolution: 1920x1080 | FPS: 30 | Audio: stereo AAC
|
||||
[EP] Footage looks viable. Audio present. Proceeding.
|
||||
|
||||
[EP] === STAGE 1: idea ===
|
||||
[EP] Spawning idea-director... Footage: interview_raw.mp4
|
||||
[EP] Brief: "Interview with CTO on API security" | Target: 3m00s (trim from 4m22s)
|
||||
[EP] Platform: YouTube Shorts → wait, that's < 60s. User said LinkedIn.
|
||||
[EP] G1 PASS — Brief references footage, duration target realistic, user approved.
|
||||
|
||||
[EP] === STAGE 2: script ===
|
||||
[EP] Spawning script-director with brief...
|
||||
[EP] Transcriber: WhisperX large-v3. Processing 4m22s audio...
|
||||
[EP] Transcript: 612 words, avg confidence 0.91. Language: en.
|
||||
[EP] 8 sections identified. Timestamps monotonic. ✓
|
||||
[EP] G2 PASS — Confidence good, sections align with topic changes.
|
||||
|
||||
[EP] === STAGE 3: scene_plan ===
|
||||
[EP] Spawning scene-director with script...
|
||||
[EP] 8 scenes planned. Total duration: 3m02s (target 3m00s).
|
||||
[EP] Enhancements: face_enhance on all scenes, color_grade, lower-third overlay at 0:00-0:05.
|
||||
[EP] face_enhance: checking registry... AVAILABLE ✓
|
||||
[EP] G3 PASS — Full coverage, enhancements feasible.
|
||||
|
||||
[EP] === STAGE 4: assets ===
|
||||
[EP] Spawning asset-director with scene_plan + script...
|
||||
[EP] Subtitles generated: 82 cues, SRT format.
|
||||
[EP] Sync check: max offset 0.18s. All within 0.3s tolerance. ✓
|
||||
[EP] Audio extracted and normalized to -16 LUFS. ✓
|
||||
[EP] Lower-third overlay generated via recraft_image. Cost: $0.02.
|
||||
[EP] Budget: $0.02 spent, $0.48 remaining.
|
||||
[EP] G4 PASS — Subtitles synced, audio clean, assets on disk.
|
||||
|
||||
[EP] === STAGE 5: edit ===
|
||||
[EP] Spawning edit-director with scene_plan + asset_manifest...
|
||||
[EP] Timeline: 3m02s with 7 cuts. Subtitles enabled.
|
||||
[EP] Trim: removed 0:00-0:12 (dead air) and 3:45-4:22 (off-topic).
|
||||
[EP] G5 PASS — Timeline complete, all references valid.
|
||||
|
||||
[EP] === STAGE 6: compose ===
|
||||
[EP] Spawning compose-director with edit_decisions + asset_manifest...
|
||||
[EP] face_enhance applied: 8 scenes processed.
|
||||
[EP] color_grade applied: unified warm tone.
|
||||
[EP] audio_enhance: noise reduction applied.
|
||||
[EP] video_compose: final render → output/talking-head-final.mp4
|
||||
[EP] Output probe: 3m01s, 1920x1080, stereo audio, H.264. ✓
|
||||
[EP] Budget: $0.18 spent (face_enhance + color_grade + overlays).
|
||||
[EP] G6 PASS
|
||||
|
||||
[EP] === STAGE 7: publish ===
|
||||
[EP] Spawning publish-director with render_report...
|
||||
[EP] G7 PASS — Title, description, chapters, thumbnail configured.
|
||||
|
||||
[EP] === FINAL QA ===
|
||||
[EP] Duration: 3m01s ✓ | Subtitle sync: max drift 0.18s ✓ | Audio: -16.2 LUFS ✓
|
||||
[EP] Face enhance: natural ✓ | Color: consistent ✓ | Overlays: timed correctly ✓
|
||||
[EP] Budget: $0.18 / $0.50 ✓
|
||||
[EP] PIPELINE COMPLETE — 0 revisions, 0 send-backs
|
||||
[EP] Output: output/talking-head-final.mp4
|
||||
```
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Ignoring transcript quality**: Everything downstream depends on the transcript. If confidence is low, fix it in the script stage — don't let bad timestamps propagate to subtitles and edits.
|
||||
- **Over-enhancing**: Face enhance and color grade are optional. If the raw footage looks good, skip them. Don't add processing for the sake of it.
|
||||
- **Subtitle style mismatch**: The subtitle style must come from the playbook. Don't let the asset director use default SRT styling when the playbook specifies font/color/position.
|
||||
- **Not probing raw footage**: Always ffprobe before starting. A video with no audio track or a corrupt container will waste every downstream stage.
|
||||
- **Trimming too aggressively**: The edit director may cut sections that seem off-topic but contain valuable context. The EP should verify that trimmed content is genuinely unnecessary by checking against the brief.
|
||||
- **Losing transcript data**: The EP must carry `transcript_segments` from the script stage all the way to asset generation. Subtitle timing depends on the exact same word-level data the transcriber produced.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Idea Director — Talking Head Pipeline
|
||||
|
||||
## When to Use
|
||||
|
||||
You are starting a talking-head video project. You have raw footage of a person speaking. Your job is to analyze the footage, understand what it contains, and build a brief that captures the content's essence and production goals.
|
||||
|
||||
Unlike the explainer pipeline (which starts from a topic), you start from existing footage. The brief documents what you're working with and what the final video should look like.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/brief.schema.json` | Artifact validation |
|
||||
| Inputs | Raw footage file path | Source material |
|
||||
| Tools | `ffprobe` (via shell) | Footage metadata extraction |
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Inspect the Footage
|
||||
|
||||
Use ffprobe to extract metadata:
|
||||
- Duration
|
||||
- Resolution
|
||||
- Frame rate
|
||||
- Audio channels and codec
|
||||
- File size
|
||||
|
||||
This tells you what you're working with — quality, length, format.
|
||||
|
||||
### Step 2: Quick Content Assessment
|
||||
|
||||
Watch/scan the footage mentally (or sample frames if frame_sampler is available):
|
||||
- What is the person talking about?
|
||||
- How long is the raw footage?
|
||||
- What's the intended platform? (Ask the user if unclear)
|
||||
- Is there good audio? Background noise?
|
||||
|
||||
### Step 3: Build the Brief
|
||||
|
||||
Create a brief artifact documenting:
|
||||
- **Title**: Descriptive title based on footage content
|
||||
- **Hook**: What makes this worth watching?
|
||||
- **Key points**: Main topics covered in the footage
|
||||
- **Tone**: Match the speaker's actual tone (casual, professional, educational)
|
||||
- **Style**: Usually `clean-professional` for talking head
|
||||
- **Target platform**: Where this will be published
|
||||
- **Target duration**: May be shorter than raw footage (trimmed)
|
||||
|
||||
### Step 4: Self-Evaluate
|
||||
|
||||
| Criterion | Question |
|
||||
|-----------|----------|
|
||||
| **Accuracy** | Does the brief reflect what's actually in the footage? |
|
||||
| **Completeness** | Are all required brief fields present? |
|
||||
| **Platform fit** | Is the target platform appropriate for this content? |
|
||||
|
||||
### Step 5: Submit
|
||||
|
||||
Validate the brief against the schema and persist via checkpoint.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Publish Director — Talking Head Pipeline
|
||||
|
||||
## When to Use
|
||||
|
||||
You have a render report with the final video. Your job is to prepare metadata, thumbnails, and an export package for publishing.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/publish_log.schema.json` | Artifact validation |
|
||||
| Prior artifacts | Render report, Brief | Video file and context |
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Generate Metadata
|
||||
|
||||
Create platform-specific metadata:
|
||||
- **Title**: Based on the brief's title and hook
|
||||
- **Description**: Summary of the content with relevant keywords
|
||||
- **Tags**: Derived from brief's key_points
|
||||
- **Chapters**: From script section timestamps
|
||||
|
||||
### Step 2: Thumbnail Concept
|
||||
|
||||
Describe or generate a thumbnail:
|
||||
- Extract a compelling frame from the footage (if frame_sampler available)
|
||||
- Add text overlay concept (title or key stat)
|
||||
|
||||
### Step 3: Package Export
|
||||
|
||||
Create the export directory:
|
||||
- Video file
|
||||
- Metadata JSON
|
||||
- Description text file
|
||||
- Chapter markers
|
||||
- Thumbnail concept
|
||||
|
||||
### Step 4: Build Publish Log
|
||||
|
||||
Document the publish event with platform, status (draft), and export path.
|
||||
|
||||
### Step 5: Self-Evaluate
|
||||
|
||||
| Criterion | Question |
|
||||
|-----------|----------|
|
||||
| **Metadata quality** | Is the title compelling and description informative? |
|
||||
| **Completeness** | Is the export package complete? |
|
||||
|
||||
### Step 6: Submit
|
||||
|
||||
Validate the publish_log against the schema and persist via checkpoint.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Scene Director — Talking Head Pipeline
|
||||
|
||||
## When to Use
|
||||
|
||||
You have a script (from transcription) and raw footage. Your job is to create a scene plan — mostly simple since talking-head footage is a single continuous shot, but you still need to plan where overlays, text cards, and B-roll might appear.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/scene_plan.schema.json` | Artifact validation |
|
||||
| Prior artifacts | Script, Brief | Section timing and context |
|
||||
| Tools | `frame_sampler` (optional) | Extract representative frames |
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Plan Base Scenes
|
||||
|
||||
For talking-head, the base is simple: one scene per script section, all type `talking_head`. The raw footage IS the scene.
|
||||
|
||||
### Step 2: Plan Enhancement Scenes
|
||||
|
||||
Based on script enhancement cues, plan overlay scenes:
|
||||
- Text cards for key terms or statistics
|
||||
- Lower thirds for speaker identification
|
||||
- B-roll suggestions for topic illustrations
|
||||
|
||||
### Step 3: Build Scene Plan
|
||||
|
||||
Create a scene per section with:
|
||||
- Type: `talking_head` (primary)
|
||||
- Timing from script sections
|
||||
- Required assets: subtitle file, any overlay images
|
||||
|
||||
### Step 4: Self-Evaluate
|
||||
|
||||
| Criterion | Question |
|
||||
|-----------|----------|
|
||||
| **Coverage** | Every script section has a scene? |
|
||||
| **Enhancement** | Are overlay opportunities identified? |
|
||||
| **Feasibility** | Can all required assets be generated? |
|
||||
|
||||
### Step 5: Submit
|
||||
|
||||
Validate the scene_plan against the schema and persist via checkpoint.
|
||||
@@ -0,0 +1,56 @@
|
||||
# Script Director — Talking Head Pipeline
|
||||
|
||||
## When to Use
|
||||
|
||||
You have a brief and raw talking-head footage. Your job is to transcribe the footage and structure it into a script artifact with timestamped sections.
|
||||
|
||||
Unlike the explainer pipeline (which writes a script from scratch), you're extracting and structuring existing speech.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/script.schema.json` | Artifact validation |
|
||||
| Prior artifacts | `state.artifacts["idea"]["brief"]` | Content context |
|
||||
| Tools | `transcriber` (WhisperX) | Speech-to-text with timestamps |
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Transcribe
|
||||
|
||||
Use the transcriber tool to get word-level timestamps:
|
||||
- Model: `large-v3` for best quality, `base` for speed
|
||||
- Enable word-level alignment for precise timing
|
||||
- Note language detection result
|
||||
|
||||
### Step 2: Segment into Sections
|
||||
|
||||
Group the transcript into logical sections:
|
||||
- Detect topic changes by content
|
||||
- Respect natural pauses (> 1.5s silence = potential section break)
|
||||
- Each section gets: id, text, start_seconds, end_seconds
|
||||
|
||||
### Step 3: Enhance Section Metadata
|
||||
|
||||
For each section, add:
|
||||
- Enhancement cues (where overlays, b-roll, or text cards could go)
|
||||
- Speaker notes (emphasis, pace changes detected in audio)
|
||||
|
||||
### Step 4: Build Script Artifact
|
||||
|
||||
Assemble the structured script with:
|
||||
- Total duration (from transcript)
|
||||
- All sections with timestamps
|
||||
- Enhancement cues per section
|
||||
|
||||
### Step 5: Self-Evaluate
|
||||
|
||||
| Criterion | Question |
|
||||
|-----------|----------|
|
||||
| **Transcription accuracy** | Are the words correct? (Spot-check a few sections) |
|
||||
| **Timestamp accuracy** | Do section boundaries align with actual speech? |
|
||||
| **Coverage** | Does the script span the full footage duration? |
|
||||
|
||||
### Step 6: Submit
|
||||
|
||||
Validate the script against the schema and persist via checkpoint.
|
||||
Reference in New Issue
Block a user