One-key formula: AI images, TTS narration, auto music, subtitles, and self-review pipeline
Prove that adding one API key (OPENAI_API_KEY) to the zero-key foundation produces dramatically better videos for ~$0.69 each. Two compositions built: The Abyss (deep ocean visual essay) and VOID (neural interface product ad). New tools: - audio_probe: ffprobe wrapper with probe_duration() helper - composition_validator: pre-render checks (asset existence, audio-video sync) - pixabay_music: royalty-free music scraper (no API key needed) - freesound_music: Freesound API search + download Remotion upgrades: - BackgroundImageLayer: AI images behind data scenes with ken-burns + dark overlay - Gradient support: all 9 components changed from backgroundColor to background CSS - CaptionOverlay: word spacing fix, WhisperX word-level subtitles - HeroTitle: reduced overlay opacity so background images show through Process codified in agent skills: - compose-director: audio acquisition flow (present user with voice/music/subtitle options), mandatory pre-render validation, post-render self-review (extract frames + transcribe + inspect + present findings to user) - scene-director: narration duration budgeting (word budget from video duration) - remotion skill: pre-render validation section - TTS tool: now returns audio_duration_seconds in result README updated with VOID product ad video embed.
This commit is contained in:
calesthio
@@ -88,6 +88,29 @@ across bar/pie/line scenes for visual unity.
|
||||
**Reference compositions:** See `remotion-composer/public/demo-props/climate-dashboard.json`
|
||||
as the gold standard, and other demo files for additional patterns.
|
||||
|
||||
### Pre-Render Validation (mandatory)
|
||||
|
||||
**Always run `composition_validator` before rendering.** It catches:
|
||||
- Missing asset files (images, audio) that would cause render failures
|
||||
- Narration audio longer than video duration (audio gets cut off)
|
||||
- Music shorter than video (silence at end)
|
||||
- Invalid cut timings (out ≤ in)
|
||||
|
||||
```python
|
||||
from tools.analysis.composition_validator import CompositionValidator
|
||||
result = CompositionValidator().execute({
|
||||
"composition_path": "path/to/composition.json",
|
||||
"assets_root": "remotion-composer/public",
|
||||
})
|
||||
# result.data["valid"] must be True before rendering
|
||||
```
|
||||
|
||||
**Audio duration alignment:**
|
||||
- After generating TTS narration, the tool returns `audio_duration_seconds`.
|
||||
- If narration exceeds video duration: shorten script and regenerate, OR extend the last scene.
|
||||
- Use `tools.analysis.audio_probe.probe_duration(path)` to check any audio file's duration.
|
||||
- Music should be ≥ video duration; the player handles fade-out via `fadeOutSeconds`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
|
||||
@@ -36,7 +36,84 @@ Based on the edit decisions, pick the rendering approach:
|
||||
|
||||
You can combine both: Remotion for animated segments, FFmpeg for final assembly.
|
||||
|
||||
### Step 2: Prepare Render Inputs
|
||||
### Step 2: Audio Acquisition (Narration, Music, Subtitles)
|
||||
|
||||
Before rendering, present the user with audio options and get their preferences.
|
||||
|
||||
**Present to the user:**
|
||||
|
||||
> **Audio setup for this video:**
|
||||
>
|
||||
> **Narration:** I can generate TTS narration using OpenAI TTS (`gpt-4o-mini-tts` — $0.015/min, 6 voices, voice direction). Which voice and tone would you like? I'll propose a voice based on the video topic, or you can choose:
|
||||
> - `onyx` — deep, authoritative (documentaries, tech)
|
||||
> - `echo` — resonant, futuristic (product ads, sci-fi)
|
||||
> - `nova` — bright, energetic (upbeat, explainers)
|
||||
> - `fable` — warm, storytelling (narratives, education)
|
||||
> - `shimmer` — expressive, warm (organic, lifestyle)
|
||||
> - `alloy` — neutral, balanced (general purpose)
|
||||
>
|
||||
> **Music:** I can automatically find royalty-free background music from Pixabay (no key needed). If you have a `FREESOUND_API_KEY`, I can also search Freesound as a backup.
|
||||
>
|
||||
> **Subtitles:** I'll generate word-level subtitles using WhisperX transcription of the final narration, burned into the video via Remotion captions.
|
||||
>
|
||||
> Want me to proceed with my recommendations, or adjust anything?
|
||||
|
||||
**After user confirms:**
|
||||
|
||||
1. **Write narration script with duration budget** (see scene-director Step 4b):
|
||||
- Calculate video duration from cuts
|
||||
- Budget at 85-90% of video duration
|
||||
- Use 2.0-2.5 words/sec for documentary, 2.5-3.0 for energetic
|
||||
- Verify word count before generating TTS
|
||||
|
||||
2. **Generate TTS narration:**
|
||||
```python
|
||||
from tools.audio.openai_tts import OpenAITTS
|
||||
result = OpenAITTS().execute({
|
||||
'text': narration_script,
|
||||
'voice': '<user-chosen or agent-recommended>',
|
||||
'instructions': '<voice direction matching video tone>',
|
||||
'output_path': 'path/to/narration.mp3',
|
||||
})
|
||||
# CRITICAL: Check result.data['audio_duration_seconds'] vs video duration
|
||||
# If narration exceeds video by >1s: shorten script and regenerate
|
||||
```
|
||||
|
||||
3. **Download background music:**
|
||||
```python
|
||||
from tools.audio.pixabay_music import PixabayMusic
|
||||
result = PixabayMusic().execute({
|
||||
'query': '<mood/genre matching video topic>',
|
||||
'min_duration': video_duration_seconds,
|
||||
'max_duration': 300,
|
||||
'output_path': 'path/to/music.mp3',
|
||||
})
|
||||
```
|
||||
|
||||
4. **Generate subtitles via WhisperX:**
|
||||
```python
|
||||
from tools.analysis.transcriber import Transcriber
|
||||
result = Transcriber().execute({
|
||||
'input_path': 'path/to/narration.mp3',
|
||||
'model_size': 'base',
|
||||
'language': 'en',
|
||||
})
|
||||
# Convert word_timestamps to Remotion caption format:
|
||||
# [{ "word": "Hello", "startMs": 0, "endMs": 340 }, ...]
|
||||
```
|
||||
|
||||
5. **Assemble composition JSON** with audio config:
|
||||
```json
|
||||
{
|
||||
"audio": {
|
||||
"narration": { "src": "path/to/narration.mp3", "volume": 1 },
|
||||
"music": { "src": "path/to/music.mp3", "volume": 0.1, "fadeInSeconds": 2, "fadeOutSeconds": 3 }
|
||||
},
|
||||
"captions": [ ... word-level captions from WhisperX ... ]
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Prepare Render Inputs
|
||||
|
||||
For each cut in the edit decisions:
|
||||
1. Verify the source asset exists at its declared path
|
||||
@@ -44,8 +121,8 @@ For each cut in the edit decisions:
|
||||
3. Prepare transform parameters (scale, position, crop)
|
||||
|
||||
For audio:
|
||||
1. Verify all narration segments exist
|
||||
2. Verify music track exists
|
||||
1. Verify narration duration fits within video duration (use `audio_probe`)
|
||||
2. Verify music duration covers video duration
|
||||
3. Prepare ducking parameters from edit decisions
|
||||
|
||||
### Step 3: Determine Output Profile
|
||||
@@ -152,7 +229,85 @@ Subtitles are mandatory for all explainer content. Generate them from the narrat
|
||||
|
||||
**The final deliverable is the subtitled version**, not the pre-subtitle render.
|
||||
|
||||
### Step 6: Verify Output
|
||||
### Step 5c: Pre-Render Validation (Mandatory)
|
||||
|
||||
**Always run the composition validator before rendering.** This catches problems that waste render time.
|
||||
|
||||
```python
|
||||
from tools.analysis.composition_validator import CompositionValidator
|
||||
result = CompositionValidator().execute({
|
||||
'composition_path': 'path/to/composition.json',
|
||||
'assets_root': 'remotion-composer/public',
|
||||
})
|
||||
# result.data['valid'] MUST be True before proceeding to render
|
||||
# If False: fix the reported errors first (missing assets, audio-video mismatch, etc.)
|
||||
```
|
||||
|
||||
Common catches:
|
||||
- Narration audio longer than video (would be cut off)
|
||||
- Missing image/audio files (render would fail)
|
||||
- Music shorter than video (silence at end)
|
||||
|
||||
**Do not skip this step.** If validation fails, fix the issue and re-validate before rendering.
|
||||
|
||||
### Step 6: Post-Render Self-Review (Mandatory)
|
||||
|
||||
After rendering, the agent **must review its own output** before presenting to the user. This catches issues the validator can't see (visual quality, audio sync, subtitle readability).
|
||||
|
||||
**6a. Extract review frames:**
|
||||
```python
|
||||
from tools.analysis.frame_sampler import FrameSampler
|
||||
# Extract one frame per scene at the midpoint
|
||||
midpoints = [(cut['in_seconds'] + cut['out_seconds']) / 2 for cut in cuts]
|
||||
FrameSampler().execute({
|
||||
'input_path': 'path/to/rendered_video.mp4',
|
||||
'strategy': 'timestamps',
|
||||
'timestamps': midpoints,
|
||||
'output_dir': 'path/to/review-frames',
|
||||
'format': 'png',
|
||||
})
|
||||
```
|
||||
|
||||
**6b. Transcribe rendered audio:**
|
||||
```python
|
||||
from tools.analysis.transcriber import Transcriber
|
||||
Transcriber().execute({
|
||||
'input_path': 'path/to/rendered_video.mp4',
|
||||
'model_size': 'base',
|
||||
'language': 'en',
|
||||
'output_dir': 'path/to/review-frames',
|
||||
})
|
||||
# Verify all narration words are present and not cut off
|
||||
```
|
||||
|
||||
**6c. Visual inspection — review each frame:**
|
||||
- Does the background color/gradient match intent? (watch for white backgrounds on dark-themed videos)
|
||||
- Are images rendering correctly? (not blank, not stretched)
|
||||
- Are subtitles visible and properly spaced?
|
||||
- Are overlays (section titles, stat reveals) positioned correctly?
|
||||
- Is the opening scene visually strong? (important for social media thumbnails)
|
||||
|
||||
**6d. Audio inspection — check transcript:**
|
||||
- Is the full narration captured? (compare last transcribed word to last scripted word)
|
||||
- Any words cut off at the end? (narration exceeding video duration)
|
||||
- Timing alignment — do narration segments roughly match their intended scenes?
|
||||
|
||||
**6e. Compile and present review to user:**
|
||||
|
||||
> **Post-render review for "[Video Title]":**
|
||||
>
|
||||
> **Audio:** [Complete/Cut off at Xs] — all N words captured / last sentence missing
|
||||
> **Visuals:** [N scenes inspected] — [issues or "all scenes rendering correctly"]
|
||||
> **Subtitles:** [Present/Missing] — [spacing ok / words running together]
|
||||
> **Issues found:** [list any issues with severity]
|
||||
>
|
||||
> **Recommendations:** [what to fix, if anything]
|
||||
>
|
||||
> Want me to fix these issues and re-render, or is this good to go?
|
||||
|
||||
**Only after user approves (or agent finds zero issues) should the video be considered final.**
|
||||
|
||||
### Step 6-old: File and Content Verification
|
||||
|
||||
**File verification:**
|
||||
- [ ] Output file exists at declared path
|
||||
@@ -165,9 +320,9 @@ Subtitles are mandatory for all explainer content. Generate them from the narrat
|
||||
- [ ] Audio channels present (stereo)
|
||||
- [ ] No audio clipping or silence gaps > 1s
|
||||
|
||||
**Quality check:**
|
||||
- [ ] Visual: scrub through at 25%, 50%, 75% marks — images display correctly
|
||||
- [ ] Audio: narration is audible and clear throughout
|
||||
**Quality check (covered by self-review above):**
|
||||
- [ ] Visual: all scene frames inspected
|
||||
- [ ] Audio: full transcription verified
|
||||
- [ ] Subtitles: visible and correctly timed
|
||||
|
||||
### Step 7: Build Render Report
|
||||
|
||||
@@ -131,6 +131,31 @@ Show code with syntax highlighting. Highlight specific lines as the narrator exp
|
||||
- Tools: `code_snippet` tool + Remotion
|
||||
- Example: "Python code: `results = collection.query(embedding, n_results=5)`. Highlight `embedding` parameter when narrator says 'vector'."
|
||||
|
||||
### Step 4b: Write Narration with Duration Budget
|
||||
|
||||
If the video includes narration, the script **must** be written to fit the video duration.
|
||||
|
||||
**Duration budgeting formula:**
|
||||
1. Calculate total video duration from scene timings (last cut's `out_seconds`).
|
||||
2. Target narration at **85-90%** of video duration to leave breathing room at intro/outro.
|
||||
3. Budget words: **2.0-2.5 words/second** for documentary style with natural pauses; **2.5-3.0 words/second** for energetic/fast-paced delivery.
|
||||
4. Example: 53s video → target 45-48s of narration → 90-120 words max (documentary) or 112-144 words (energetic).
|
||||
|
||||
**Per-scene word budgets:**
|
||||
- Allocate words proportionally to each scene's duration.
|
||||
- A 5s scene gets ~10-12 words. A 6s scene gets ~12-15 words.
|
||||
- Leave 0.5-1s of silence between scene transitions for visual breathing room.
|
||||
|
||||
**Validation (mandatory before TTS generation):**
|
||||
- [ ] Total word count is within budget for the target duration
|
||||
- [ ] No single scene's narration exceeds its time slot
|
||||
- [ ] Opening and closing scenes have brief narration (let visuals breathe)
|
||||
|
||||
**After TTS generation:**
|
||||
- The TTS tool returns `audio_duration_seconds` — compare it against video duration.
|
||||
- If narration exceeds video by >1s, either trim the script and regenerate, or extend the video's closing scene.
|
||||
- Always run `composition_validator` before rendering to catch mismatches automatically.
|
||||
|
||||
### Step 5: Validate Against Playbook
|
||||
|
||||
The style playbook constrains your visual choices:
|
||||
|
||||
Reference in New Issue
Block a user