Add documentary-montage pipeline for retrieval-first motion-clip montage
New end-to-end pipeline for building thematic documentary montages from a locally-indexed corpus of free stock footage (Pexels, Archive.org, NASA). The agent builds a project-local corpus, CLIP-ranks candidates per scene slot, edits with motion-aware arc logic, and composes via ffmpeg. No paid APIs required for the full path. Pipeline definition and director skills: - pipeline_defs/documentary-montage.yaml: 5-stage manifest (idea -> scene_plan -> assets -> edit -> compose) - skills/pipelines/documentary-montage/: 6 director skills (executive-producer + idea/scene/asset/edit/compose directors) Corpus and retrieval infrastructure: - tools/video/corpus_builder.py: multi-source stock fan-out with resumable append-only corpus index - tools/video/clip_search.py: CLIP ViT-B/32 retrieval — rank_for_slot, find_similar_set, diversify, stats - tools/video/stock_sources/: base + pexels + archive_org + nasa adapters with a pluggable BaseStockSource contract - lib/clip_embedder.py: CLIP wrapper - lib/corpus.py: corpus schema, jsonl append/read, motion-score caching video_compose fix rolled in because any concat-based pipeline depends on it: - Replace ambiguous -to with -t duration (was double-trimming cuts) - Force re-encode + normalize to 1920x1080 @ 30fps (was keyframe- snapping with -c copy and breaking concat on mixed-source corpora) - Add silent-audio anullsrc fallback for clips without an audio stream README: add Documentary Montage row to the pipeline table and bump the pipeline count from 11 to 12.
This commit is contained in:
calesthio
@@ -0,0 +1,365 @@
|
||||
# Asset Director - Documentary Montage Pipeline
|
||||
|
||||
## When To Use
|
||||
|
||||
The shot list exists. You now have to actually go out and find the
|
||||
clips that fill each slot. This is a two-step operation:
|
||||
|
||||
1. **Build the corpus** — fan the scene director's queries out across
|
||||
Pexels / Archive.org / NASA and download/embed the candidates.
|
||||
2. **Pick per slot** — run CLIP retrieval against the corpus with each
|
||||
slot description and choose one winner per slot.
|
||||
|
||||
The output is an `asset_manifest` mapping every slot to exactly one
|
||||
clip with full provenance.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/asset_manifest.schema.json` | Artifact validation |
|
||||
| Prior artifact | `state.artifacts["scene_plan"]["scene_plan"]` | Slot descriptions + queries + preferred_sources |
|
||||
| Prior artifact | `state.artifacts["idea"]["brief"]` | `era_mix`, `sources_allowed`, `music_plan` |
|
||||
| Tool | `corpus_builder` | Populates the retrieval index |
|
||||
| Tool | `clip_search` | Ranks clips against slot descriptions |
|
||||
| Tool (optional) | `music_gen`, user's `music_library/` | Score bed |
|
||||
|
||||
## Mental Model
|
||||
|
||||
The corpus is NOT a stock library. It is a search index the agent
|
||||
builds on demand. You do not scroll through it — you query it.
|
||||
|
||||
Three rules that follow from that:
|
||||
|
||||
1. **Build before picking.** Never call `clip_search.rank_for_slot`
|
||||
on a corpus that doesn't contain candidates for that slot's query
|
||||
family. The ranking will return junk and you'll waste the slot.
|
||||
2. **Grow, don't replace.** The corpus is append-only. If a slot's
|
||||
retrieval is weak, add more queries and rebuild — don't start over.
|
||||
3. **Pick per slot, not per clip.** Every clip only belongs to one
|
||||
slot in the final edit. Use `exclude_ids` to prevent double-use.
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Resolve The Corpus Directory
|
||||
|
||||
Decide where the corpus lives. Convention:
|
||||
|
||||
```
|
||||
projects/<project-name>/corpus/
|
||||
```
|
||||
|
||||
The same `corpus_dir` is passed to every `corpus_builder` and
|
||||
`clip_search` call. The corpus is reusable across re-runs — if the
|
||||
scene director adds slots later, you can grow the same corpus instead
|
||||
of rebuilding from scratch.
|
||||
|
||||
### 2. Fan Out The Queries Into `corpus_builder`
|
||||
|
||||
Read `scene_plan.metadata.slots[]`. Collect every `queries[]` array
|
||||
across every slot. De-duplicate. Group by `preferred_sources`.
|
||||
|
||||
Call `corpus_builder.execute(...)` with one fan-out per source set:
|
||||
|
||||
```python
|
||||
# Example shape. The agent constructs this from the shot list.
|
||||
corpus_builder.execute({
|
||||
"corpus_dir": "projects/<name>/corpus",
|
||||
"queries": [
|
||||
{"query": "raindrop on asphalt slow motion", "kind": "video", "per_source": 8},
|
||||
{"query": "wet city street night neon", "kind": "video", "per_source": 8},
|
||||
{"query": "taxi heavy rain yellow", "kind": "video", "per_source": 6},
|
||||
# ... one entry per unique slot query
|
||||
],
|
||||
"sources": ["pexels", "archive_org"], # from preferred_sources union
|
||||
"filters": {
|
||||
"min_duration": 3,
|
||||
"max_duration": 40,
|
||||
"orientation": "landscape",
|
||||
"min_width": 1280,
|
||||
},
|
||||
"max_new_clips": 150, # enlarge the search space
|
||||
"thumbs_per_video": 5,
|
||||
})
|
||||
```
|
||||
|
||||
**Rules for the fan-out:**
|
||||
|
||||
- Budget the corpus for 8-12x the slot count. A 15-slot montage wants
|
||||
~150 candidates so retrieval has real choices.
|
||||
- `per_source` of 4-8 per query is usually enough. Pushing to 20+
|
||||
mostly adds noise.
|
||||
- If `era_mix = "vintage"`, run a separate fan-out restricted to
|
||||
`["archive_org"]` with period-appropriate queries. Prelinger search
|
||||
is slow — don't interleave it with the modern Pexels batch.
|
||||
- If any slot has `nasa` in `preferred_sources`, run ONE small
|
||||
`nasa`-only batch. NASA is slow and its results are niche.
|
||||
|
||||
### 3. Sanity-Check The Corpus Before Retrieval
|
||||
|
||||
Before spending tokens on slot picks, call `clip_search` with
|
||||
`operation=stats`:
|
||||
|
||||
```python
|
||||
clip_search.execute({
|
||||
"operation": "stats",
|
||||
"corpus_dir": "projects/<name>/corpus",
|
||||
})
|
||||
```
|
||||
|
||||
Look at `rows`, `per_source`, `per_kind`, `mean_motion_score`. You're
|
||||
checking for three failure modes:
|
||||
|
||||
- `rows < 50` — corpus is too small. Grow it.
|
||||
- `per_source` heavily skewed (e.g. 98% pexels, 2% archive_org) on a
|
||||
vintage brief — run a targeted archive_org fan-out.
|
||||
- `mean_motion_score < 1.0` — corpus is full of static clips and will
|
||||
make for a slideshow. Rerun with different queries, or apply
|
||||
`motion_min` at rank time.
|
||||
|
||||
### 4. Rank Candidates Per Slot
|
||||
|
||||
For each slot in `scene_plan.metadata.slots[]`, call `clip_search`
|
||||
with `operation=rank_for_slot`:
|
||||
|
||||
```python
|
||||
clip_search.execute({
|
||||
"operation": "rank_for_slot",
|
||||
"corpus_dir": "projects/<name>/corpus",
|
||||
"query_text": slot["description"], # NOT slot["queries"] — the description is richer
|
||||
"k": 30 if slot.get("hero") else 12,
|
||||
"tag_weight": 0.3,
|
||||
"motion_min": 1.5,
|
||||
"kind": "video",
|
||||
"exclude_ids": already_picked_ids, # global accumulator
|
||||
})
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- Use the slot **description**, not the queries. The description is
|
||||
the rich noun-and-adjective string the scene director wrote. CLIP
|
||||
ranks it better than short search phrases.
|
||||
- `tag_weight=0.3` blends visual embedding (70%) with source-tag
|
||||
embedding (30%). Raise to 0.5 when Pexels URL tags are strong and
|
||||
the visual channel is noisy. Lower to 0.15 for Prelinger where tags
|
||||
are long prose.
|
||||
- Always pass `exclude_ids` with every clip already locked to a slot,
|
||||
so the same key-in-door clip doesn't win two slots.
|
||||
|
||||
### 5. Pick With Judgement, Not By Score
|
||||
|
||||
The top result is not always the right pick. Look at the top 3-5 and
|
||||
judge each one against:
|
||||
|
||||
- **Era fit.** Does a 2022 4K Pexels shot belong in an elegiac list
|
||||
montage about home? Maybe. Maybe not.
|
||||
- **Motion fit.** The tone table from the scene director tells you
|
||||
how long the hold will be. If the clip has a 4.0s hold target and
|
||||
the clip is 2s long with a fast whip pan, it won't stretch.
|
||||
- **Compositional carry.** Will this clip work NEXT to the clips
|
||||
picked for the adjacent slots? You don't know yet — but if slot_02
|
||||
is a wide rooftop-in-rain and the top hit for slot_03 is also a
|
||||
wide rooftop-in-rain, pick the #2 instead.
|
||||
- **Emotional register.** CLIP will happily match "empty city
|
||||
sidewalk at night" to a bright neon Vegas cutaway. The neon shot is
|
||||
WRONG for an elegiac brief. Score 0.42 does not override tone.
|
||||
|
||||
**Acceptable-score rules of thumb (CLIP ViT-B/32 cosine):**
|
||||
|
||||
- `>= 0.30` — strong match, usually usable.
|
||||
- `0.22-0.30` — plausible, needs human judgement.
|
||||
- `< 0.22` — the corpus doesn't contain what you need. Grow it,
|
||||
don't force a pick.
|
||||
|
||||
### 6. Grow The Corpus When Retrieval Is Weak
|
||||
|
||||
If a slot's top score is below 0.22, do NOT pick the best-of-a-bad-
|
||||
bunch. Instead:
|
||||
|
||||
1. Rewrite the slot's queries — maybe too abstract, maybe wrong
|
||||
vocabulary for the era.
|
||||
2. Run another `corpus_builder.execute(...)` pass with just the new
|
||||
queries for that one slot. The builder skips clips already in the
|
||||
index, so this is cheap.
|
||||
3. Re-rank.
|
||||
|
||||
Two growth passes per slot is plenty. If three passes can't find a
|
||||
score above 0.22, tell the idea director the slot is unfilmable from
|
||||
open corpora and recommend either dropping the slot or letting the
|
||||
user supply the footage.
|
||||
|
||||
### 7. Diversify Adjacent Picks
|
||||
|
||||
Once you have one candidate per slot, you have a list of clip_ids in
|
||||
timeline order. Visually-redundant adjacent shots kill the edit. Run
|
||||
`clip_search.diversify` across the list:
|
||||
|
||||
```python
|
||||
clip_search.execute({
|
||||
"operation": "diversify",
|
||||
"corpus_dir": "projects/<name>/corpus",
|
||||
"candidate_ids": picked_ids_in_timeline_order,
|
||||
"n": len(picked_ids_in_timeline_order),
|
||||
"diversity": 0.5,
|
||||
})
|
||||
```
|
||||
|
||||
If `diversify` drops a clip, it's telling you two of your picks are
|
||||
visually identical. Re-rank the slot whose clip got dropped with
|
||||
`exclude_ids` including the surviving twin.
|
||||
|
||||
### 8. Handle The Music Plan
|
||||
|
||||
Read `brief.music_plan`. Execute exactly the plan the idea director
|
||||
recorded — do not invent a new source here:
|
||||
|
||||
- **`source=library`**: Verify the file at `music_plan.path` exists.
|
||||
Record it in the asset manifest as `type=music`, `subtype=library`.
|
||||
- **`source=user`**: Same, with `subtype=provided`.
|
||||
- **`source=generated`**: Call the named music tool with the seed
|
||||
prompt from the brief. Sample first, batch only after confirming
|
||||
mood. Record provider and cost.
|
||||
- **`source=none`**: Do not generate silence. Do not swap in a track
|
||||
because the edit feels thin. If the user approved "no music", run
|
||||
with no music.
|
||||
|
||||
**Never switch music source at this stage.** That's a Decision
|
||||
Communication Contract violation — changing music mode is a major
|
||||
production change and needs user approval at proposal time.
|
||||
|
||||
### 9. Record The Asset Manifest
|
||||
|
||||
Emit one asset per slot using the canonical schema. Documentary-
|
||||
montage-specific fields live in `metadata`:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"assets": [
|
||||
{
|
||||
"id": "asset_slot_01",
|
||||
"type": "video",
|
||||
"path": "projects/<name>/corpus/clips/pexels_12345/video.mp4",
|
||||
"source_tool": "corpus_builder",
|
||||
"scene_id": "slot_01",
|
||||
"duration_seconds": 7.2,
|
||||
"resolution": "1920x1080",
|
||||
"format": "mp4",
|
||||
"provider": "pexels",
|
||||
"license": "Pexels License (free, no attribution required)",
|
||||
"original_url": "https://www.pexels.com/video/12345",
|
||||
"subtype": "stock",
|
||||
"generation_summary": "Retrieved via CLIP rank for slot 'raindrop on asphalt slow motion...'. Score 0.38."
|
||||
},
|
||||
{
|
||||
"id": "asset_music_bed",
|
||||
"type": "music",
|
||||
"path": "music_library/dawn_04.mp3",
|
||||
"source_tool": "music_library",
|
||||
"scene_id": "global",
|
||||
"subtype": "library",
|
||||
"license": "user-provided"
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"pipeline": "documentary-montage",
|
||||
"corpus_dir": "projects/<name>/corpus",
|
||||
"corpus_stats": { "rows": 157, "per_source": {"pexels": 98, "archive_org": 52, "nasa": 7} },
|
||||
"rejected_picks": [
|
||||
{
|
||||
"slot_id": "slot_03",
|
||||
"clip_id": "pexels_99921",
|
||||
"score": 0.41,
|
||||
"reason": "wrong era — 2022 4K kitchen, brief is vintage"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `rejected_picks` log matters. The edit director reads it when a
|
||||
pick feels wrong and needs to reach for the #2 option.
|
||||
|
||||
### 10. Quality Gate
|
||||
|
||||
- Every slot in the scene plan has exactly one asset mapped to it.
|
||||
- Every picked clip has `score >= 0.22` in the rejected-picks log
|
||||
(or a logged "user-approved override" note).
|
||||
- No clip_id appears as the primary pick for two slots.
|
||||
- `diversify` ran clean on the final list (no dropped picks, or all
|
||||
dropped picks were re-filled).
|
||||
- `corpus_stats` shows at least 8x the slot count in rows.
|
||||
- Music asset exists OR `music_plan.source = "none"` with explicit
|
||||
acknowledgement.
|
||||
- For vintage briefs, at least 60% of picks come from `archive_org`.
|
||||
- All file paths resolve.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Running `clip_search.rank_for_slot` against an empty corpus.**
|
||||
You will get an empty `results` list or a cryptic shape error.
|
||||
Always call `stats` after a build, before ranking.
|
||||
- **Picking by score alone.** Score is an input to judgement, not the
|
||||
judgement. An elegiac piece full of top-scored Pexels HD sunshine
|
||||
will feel wrong regardless of scores.
|
||||
- **Forgetting `exclude_ids`.** Without it, the same amazing clip
|
||||
wins every slot and the montage becomes a slideshow of one image.
|
||||
- **Quiet music substitution.** User said "none", agent generated
|
||||
anyway because "the edit felt thin". This is a major change and
|
||||
needs approval — see `skills/pipelines/documentary-montage/executive-producer.md`
|
||||
cross-stage rules.
|
||||
- **Growing the corpus unboundedly.** Two growth passes per weak slot
|
||||
is the limit. Beyond that, the footage probably doesn't exist in
|
||||
the open corpora and the slot needs to change.
|
||||
- **Using slot queries as the rank text.** Queries are search phrases
|
||||
for stock APIs; descriptions are semantic text for CLIP. They are
|
||||
different. Rank on descriptions.
|
||||
- **Losing provenance.** Every clip must carry `provider`,
|
||||
`original_url`, and `license` in the manifest. These are the
|
||||
non-negotiables for any downstream publishing step.
|
||||
|
||||
## Retrieval Recipes
|
||||
|
||||
A few retrieval moves that come up often:
|
||||
|
||||
### "Find N variants of this one clip I love"
|
||||
|
||||
```python
|
||||
clip_search.execute({
|
||||
"operation": "find_similar_set",
|
||||
"corpus_dir": "projects/<name>/corpus",
|
||||
"seed_clip_id": "pexels_12345",
|
||||
"n": 5,
|
||||
"diversity": 0.4,
|
||||
"candidate_pool": 40,
|
||||
})
|
||||
```
|
||||
|
||||
Used when a slot wants "five more shots like this one" — e.g. a
|
||||
catalogue of doorways all filmed in the same register.
|
||||
|
||||
### "I have 20 candidates, trim to 8 non-redundant picks"
|
||||
|
||||
```python
|
||||
clip_search.execute({
|
||||
"operation": "diversify",
|
||||
"corpus_dir": "projects/<name>/corpus",
|
||||
"candidate_ids": [...],
|
||||
"n": 8,
|
||||
"diversity": 0.5,
|
||||
})
|
||||
```
|
||||
|
||||
### "Look up one clip's full metadata"
|
||||
|
||||
```python
|
||||
clip_search.execute({
|
||||
"operation": "get",
|
||||
"corpus_dir": "projects/<name>/corpus",
|
||||
"clip_id": "archive_org_Prelinger_HomeMovies_0042",
|
||||
})
|
||||
```
|
||||
|
||||
Used when the edit director wants to confirm the provider/URL before
|
||||
locking the cut.
|
||||
@@ -0,0 +1,277 @@
|
||||
# Compose Director - Documentary Montage Pipeline
|
||||
|
||||
## When To Use
|
||||
|
||||
The timeline exists. Every cut has an in/out, transitions are
|
||||
chosen, the music bed is locked. You now have to render the piece
|
||||
and apply the register-smoothing pass (uniform crop + LUT + audio
|
||||
mix) that makes a mixed-era corpus feel like one film.
|
||||
|
||||
The output is a single mp4 plus a `render_report` artifact.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/render_report.schema.json` | Artifact validation |
|
||||
| Prior artifact | `state.artifacts["edit"]["edit_decisions"]` | Cuts, transitions, music, metadata hints |
|
||||
| Prior artifact | `state.artifacts["assets"]["asset_manifest"]` | File paths, durations, providers |
|
||||
| Tool | `video_compose` (FFmpeg + Remotion) | Primary render engine |
|
||||
| Tool | `audio_mixer` | Music fade, silence window, L-cuts |
|
||||
| Tool (optional) | `color_grade` | Uniform LUT across mixed-era clips |
|
||||
| Tool (optional) | `video_trimmer`, `video_stitch` | Lower-level helpers if needed |
|
||||
|
||||
## Mental Model
|
||||
|
||||
Most pipelines treat compose as a boring export step. For
|
||||
documentary montage it is a creative step: the last pass where grade
|
||||
and mix reconcile footage from radically different sources into one
|
||||
piece.
|
||||
|
||||
Three things must happen here that cannot happen earlier:
|
||||
|
||||
1. **Uniform aspect and letterbox.** Pexels 1920x1080, Prelinger
|
||||
640x480 4:3, NASA 1280x720 all need to land on one canvas.
|
||||
2. **Uniform color grade.** A single LUT across the whole timeline
|
||||
is what makes the 1962 home movie sit next to the 2023 kitchen
|
||||
without jumping out.
|
||||
3. **Audio mix.** Music level, silence window, L-cut ambient
|
||||
carries, final fade — done in one pass with the timeline in hand.
|
||||
|
||||
## Process
|
||||
|
||||
### 0. Hard Requirement Check
|
||||
|
||||
Read `brief` and `edit_decisions.metadata` for any hard requirements.
|
||||
If the brief said "no narration" and a narration track somehow
|
||||
appeared in the edit, STOP and ask. Do not render over a contract
|
||||
violation.
|
||||
|
||||
Also confirm the render engine you intend to use is actually
|
||||
available — `video_compose` in FFmpeg-only mode is fine for this
|
||||
pipeline (the whole piece is footage-led, no Remotion scenes needed
|
||||
unless the user asked for title cards). FFmpeg alone can render this
|
||||
pipeline end to end.
|
||||
|
||||
### 1. Resolve The Canvas
|
||||
|
||||
Read `brief.target_platform`:
|
||||
|
||||
| Target | Canvas | Letterbox |
|
||||
|--------|--------|-----------|
|
||||
| `social_short` (Instagram/TikTok) | 1080x1920 (9:16) | Top/bottom crop; center-anchor each clip |
|
||||
| `youtube` / `generic` | 1920x1080 (16:9) | None; optionally 2.35:1 top/bottom bars for cinematic feel |
|
||||
| `linkedin` | 1920x1080 (16:9) | None |
|
||||
|
||||
Every clip in the timeline must be scaled/cropped to this canvas.
|
||||
For `social_short`, this usually means center-cropping 16:9 footage.
|
||||
For `youtube` with the cinematic 2.35:1 bar treatment, pad 140px
|
||||
black top and bottom on a 1920x1080 canvas.
|
||||
|
||||
Commit this in `render_report.metadata.canvas` and
|
||||
`render_report.metadata.letterbox`.
|
||||
|
||||
### 2. Build The Concat Plan For `video_compose`
|
||||
|
||||
The edit artifact gives you a list of cuts with in/out, transitions,
|
||||
and source asset_ids. Walk the asset_manifest to resolve each
|
||||
asset_id to a real file path. Then build the render plan.
|
||||
|
||||
For a pipeline this simple, the cleanest path is:
|
||||
|
||||
```python
|
||||
video_compose.execute({
|
||||
"operation": "render",
|
||||
"output_path": "projects/<name>/renders/final.mp4",
|
||||
"canvas": { "width": 1920, "height": 1080, "fps": 24 },
|
||||
"cuts": [
|
||||
{
|
||||
"source": "<resolved file path>",
|
||||
"in": 1.2,
|
||||
"out": 5.2,
|
||||
"scale": "fit_canvas_center_crop",
|
||||
"transition_in": "fade",
|
||||
"transition_in_duration": 0.8,
|
||||
},
|
||||
# ... more cuts
|
||||
],
|
||||
"audio": {
|
||||
"music_path": "<resolved music path>",
|
||||
"music_volume": 0.7,
|
||||
"music_fade_in": 1.0,
|
||||
"music_fade_out": 4.0,
|
||||
"silence_windows": [{"start": 54.0, "end": 56.0}],
|
||||
"sfx_layers": [
|
||||
{"source": "<rain carry clip>", "start": 20.8, "duration": 1.2, "volume": 0.6}
|
||||
],
|
||||
},
|
||||
"frame_treatment": {
|
||||
"lut_path": "styles/luts/warm_film_100.cube",
|
||||
"letterbox": "2.35:1"
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
The exact field names come from the live `video_compose` schema at
|
||||
render time — consult the tool's `agent_skills` if available before
|
||||
writing the call. Do not invent parameters.
|
||||
|
||||
### 3. Apply Grade Via LUT, Not Per Clip
|
||||
|
||||
Read `edit_decisions.metadata.grade_profile`. Map it to a LUT file:
|
||||
|
||||
| Profile | LUT | Suits |
|
||||
|---------|-----|-------|
|
||||
| `warm_film_100` | vintage film warmth, slight lift | elegiac, dreamlike |
|
||||
| `cool_archive_60` | cool highlights, crushed blacks | urgent, wry |
|
||||
| `neutral_doc_20` | barely-there neutral balance | reverent |
|
||||
| `bleach_bypass_80` | desaturated, high contrast | wry, documentary-harsh |
|
||||
|
||||
If the profile isn't in the styles library, use `neutral_doc_20` and
|
||||
note it in `warnings`. Do not try to auto-grade — the LUT is the
|
||||
whole point of the register-smoothing pass.
|
||||
|
||||
Apply the LUT at the composition level, not per clip. One LUT, one
|
||||
timeline, one consistent look. This is what makes a 1962 Prelinger
|
||||
clip and a 2023 Pexels clip feel like the same film.
|
||||
|
||||
### 4. Mix The Audio Once, In Compose
|
||||
|
||||
The edit artifact already decided volumes, fades, silence windows,
|
||||
and L-cut sfx layers. Your job is to execute them faithfully:
|
||||
|
||||
- Music bed at `edit_decisions.audio.music.volume` (default 0.7).
|
||||
- Fade in per `fade_in_seconds`, fade out per `fade_out_seconds`.
|
||||
- Silence window = ducked to 0.0 for the window's duration, ramp
|
||||
back up with a 0.2s hold-off.
|
||||
- L-cut SFX layers = mix at 0.5-0.7 volume, under music.
|
||||
- No narration unless explicitly present in `edit_decisions.audio.narration`.
|
||||
|
||||
If the brief says "no music" and the edit correctly has no music
|
||||
entry, render silent. Do NOT add ambient noise "to fill the gap".
|
||||
|
||||
### 5. Render At Documentary Spec
|
||||
|
||||
Recommended encoder settings for doc montage:
|
||||
|
||||
| Field | Value | Why |
|
||||
|-------|-------|-----|
|
||||
| Codec | `libx264` (H.264) | Universal, small |
|
||||
| Pixel format | `yuv420p` | Universal compatibility |
|
||||
| CRF | `18` | Visually lossless for final deliverables |
|
||||
| FPS | `24` | Cinematic. Do NOT upconvert 24->30. |
|
||||
| Audio codec | `aac` | Universal |
|
||||
| Audio bitrate | `192k` | Music-bed friendly |
|
||||
|
||||
If the source clips are 30fps and the canvas is 24fps, let FFmpeg
|
||||
drop frames evenly — don't blend. Motion interpolation on
|
||||
mixed-source footage looks awful.
|
||||
|
||||
### 6. Post-Render Verification
|
||||
|
||||
After the render succeeds, actually probe the output file and check:
|
||||
|
||||
- **Duration.** Should match `sum(out - in for cut in cuts) + fade
|
||||
in/out` within ±0.5s.
|
||||
- **Resolution.** Should match the canvas.
|
||||
- **Audio presence.** If music was in the plan, the output must
|
||||
have an audio stream. If silence was planned, confirm.
|
||||
- **First and last frame.** Open the file, seek to 0s and to
|
||||
duration-0.1s. The first frame should be a fade-in. The last
|
||||
frame should be (or be fading to) black.
|
||||
- **Silence window.** Seek to the silence_window start. Audio level
|
||||
should drop visibly in the waveform.
|
||||
|
||||
Record verifications in `render_report.verification_notes`.
|
||||
|
||||
### 7. Emit The Render Report
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"outputs": [
|
||||
{
|
||||
"path": "projects/<name>/renders/final.mp4",
|
||||
"format": "mp4",
|
||||
"codec": "h264",
|
||||
"audio_codec": "aac",
|
||||
"resolution": "1920x1080",
|
||||
"fps": 24,
|
||||
"duration_seconds": 89.8,
|
||||
"file_size_bytes": 18234112,
|
||||
"platform_target": "youtube"
|
||||
}
|
||||
],
|
||||
"render_time_seconds": 42.3,
|
||||
"warnings": [],
|
||||
"verification_notes": [
|
||||
"Duration within +0.2s of planned",
|
||||
"First frame is black fade-in as specified",
|
||||
"Silence window 54-56s confirmed (music -60dB)",
|
||||
"Last frame fades to black at 89.0s"
|
||||
],
|
||||
"render_grammar": "cinematic-trailer",
|
||||
"metadata": {
|
||||
"pipeline": "documentary-montage",
|
||||
"canvas": { "width": 1920, "height": 1080 },
|
||||
"letterbox": "2.35:1",
|
||||
"lut": "warm_film_100",
|
||||
"music_present": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 8. Quality Gate
|
||||
|
||||
- Output file exists and plays.
|
||||
- Duration within ±1s of `brief.duration_seconds`.
|
||||
- Resolution matches `target_platform` canvas.
|
||||
- LUT was applied (or a warning logged).
|
||||
- Music is present iff the brief planned for it.
|
||||
- First and last frames verified.
|
||||
- Silence window (if any) verified in the waveform.
|
||||
- No narration unless brief-approved.
|
||||
- `render_report.warnings` lists every substitution.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Letting mixed-era clips render un-graded.** The piece will look
|
||||
like a PowerPoint slideshow of internet clips. The LUT is
|
||||
non-negotiable.
|
||||
- **Upscaling to match the canvas instead of letterboxing.**
|
||||
Prelinger 640x480 upscaled to 1920x1080 looks pixelated and wrong.
|
||||
Center it with letterbox bars, or embrace the squared crop as a
|
||||
design choice.
|
||||
- **Narration or ambient SFX added "to fill the gap".** Major
|
||||
change, needs user approval.
|
||||
- **Per-clip color grading.** One LUT across the whole piece. Do
|
||||
not try to balance each clip individually — it takes 10x the time
|
||||
and makes the register LESS consistent, not more.
|
||||
- **Quiet render engine swap.** If `video_compose` routes through
|
||||
Remotion for some reason and the aesthetic changes, stop and
|
||||
surface. This pipeline is FFmpeg-friendly and shouldn't need
|
||||
Remotion unless the user asked for title cards.
|
||||
- **Overriding edit decisions at render time.** If you find yourself
|
||||
adjusting volumes, fades, or trims in the render call, you're
|
||||
editing during compose. Go back to the edit stage, fix the
|
||||
decisions, re-emit the artifact, then re-render.
|
||||
- **Skipping verification.** A render that "succeeded" but is
|
||||
actually silent, or fades wrong, or clips the last hero frame, is
|
||||
worse than a failure. Open the file.
|
||||
|
||||
## When The Render Fails
|
||||
|
||||
If `video_compose` returns an error:
|
||||
|
||||
1. Check the error category per the Decision Communication Contract
|
||||
(auth / provider / tool bug / plan quality).
|
||||
2. If it's a path error, validate every asset_id → path resolution
|
||||
in the asset manifest. A single missing file fails the whole render.
|
||||
3. If it's a codec error, the input clips may have exotic containers
|
||||
(Archive.org sometimes serves Matroska). Try running each input
|
||||
through `video_trimmer` first to normalize to mp4/h264.
|
||||
4. If it's a memory or timeout error, split the render into halves
|
||||
with `video_stitch` at the end.
|
||||
5. Surface to the user before swapping to a lower-fidelity path.
|
||||
This pipeline is footage-led; there is no generated-stills
|
||||
fallback.
|
||||
@@ -0,0 +1,335 @@
|
||||
# Edit Director - Documentary Montage Pipeline
|
||||
|
||||
## When To Use
|
||||
|
||||
Every slot has a clip. You now have to turn a pile of clips into a
|
||||
piece. This stage decides in-points, out-points, transitions, music
|
||||
sync, and the order the clips actually run. The output is an
|
||||
`edit_decisions` artifact with a concrete timeline.
|
||||
|
||||
This is where documentary technique lives. If the asset director did
|
||||
its job, you have the raw material. The edit is the thinking.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/edit_decisions.schema.json` | Artifact validation |
|
||||
| Prior artifact | `state.artifacts["assets"]["asset_manifest"]` | Picked clips + music bed |
|
||||
| Prior artifact | `state.artifacts["scene_plan"]["scene_plan"]` | Slot order, hero flags, target holds |
|
||||
| Prior artifact | `state.artifacts["idea"]["brief"]` | Tone register, duration, shape |
|
||||
| Tool (optional) | `video_analyzer` | Probe a clip's motion if you need to re-check |
|
||||
|
||||
## Mental Model
|
||||
|
||||
Documentary montage lives in four dimensions you have to balance:
|
||||
|
||||
1. **Rhythm** — how long each hold lasts and how the holds relate.
|
||||
2. **Juxtaposition** — which image follows which, and what it means.
|
||||
3. **Music sync** — cuts landing on beats, dropouts earning weight.
|
||||
4. **Continuity of register** — the grain, color, and era don't swing
|
||||
wildly unless the swing is the point.
|
||||
|
||||
The enemy is "slideshow" — a sequence of clips played back-to-back
|
||||
with the same hold length and no sound design. If it feels like a
|
||||
slideshow, the edit has failed, regardless of how good the clips are.
|
||||
|
||||
## Process
|
||||
|
||||
### 0. Guardrails — No Silent Major Changes
|
||||
|
||||
Before touching the timeline, re-read the brief. If any of these are
|
||||
true, STOP and surface to the user per the Decision Communication
|
||||
Contract:
|
||||
|
||||
- The brief approved "no narration" but the edit feels like it needs
|
||||
voice-over. Narration is a MAJOR change.
|
||||
- The brief approved a music track that the edit director now wants
|
||||
to replace. Music swap is a MAJOR change.
|
||||
- The brief approved a 90s duration but the natural cut wants 2m30s.
|
||||
Duration stretch is a MAJOR change.
|
||||
|
||||
Fix the edit, don't paper over it. If the edit genuinely needs
|
||||
one of these, ask.
|
||||
|
||||
### 1. Set The Rhythm Grid
|
||||
|
||||
Read `brief.tone` and `brief.duration_seconds`. Compute the hold
|
||||
table from the scene director's tone chart:
|
||||
|
||||
| Tone | Base hold | Min hold | Max hold |
|
||||
|------|-----------|----------|----------|
|
||||
| elegiac | 4.0s | 2.5s | 7.0s |
|
||||
| reverent | 3.5s | 2.0s | 6.0s |
|
||||
| dreamlike | 3.0s | 1.5s | 5.5s |
|
||||
| wry | 2.0s | 1.0s | 4.0s |
|
||||
| urgent | 1.2s | 0.5s | 2.5s |
|
||||
|
||||
**Hero slots get max hold.** Mid-sequence cutaways get base. Quick
|
||||
transitions get min.
|
||||
|
||||
Total hold time must sum to within ±10% of `brief.duration_seconds`.
|
||||
If you overshoot, compress non-hero holds first — never cut heroes
|
||||
short to fit duration.
|
||||
|
||||
### 2. Arrange By Narrative Beat, Not By Score
|
||||
|
||||
The scene director gave you a slot order. That order is the intent.
|
||||
Don't rearrange it by CLIP score, motion score, or resolution.
|
||||
|
||||
You MAY reorder slots when:
|
||||
|
||||
- The music bed has a downbeat at a known timestamp and reordering
|
||||
two slots lands a hero on the beat (see step 4).
|
||||
- Two adjacent slots are visually identical and swapping one breaks
|
||||
the monotony (but see step 7 — diversify should have caught this
|
||||
already).
|
||||
- The final image isn't landing. The last 5-10s carries
|
||||
disproportionate weight; if the scene director's choice dies, move
|
||||
a stronger candidate to the tail.
|
||||
|
||||
Always log the reorder in `edit_decisions.metadata.reorder_notes`
|
||||
with the reason.
|
||||
|
||||
### 3. Trim Each Clip To Its Beat
|
||||
|
||||
For every picked clip, decide `in_seconds` and `out_seconds`. Three
|
||||
rules:
|
||||
|
||||
- **Find the best sub-window, not the whole clip.** A 12-second Pexels
|
||||
clip usually contains one 3-second moment that earns the hold and
|
||||
9 seconds of setup/settle. Find the moment.
|
||||
- **Cut BEFORE the action's natural end.** End on a look, not on a
|
||||
move-off. The cut feels intentional instead of exhausted.
|
||||
- **Leave a handle at both ends.** 4-6 frames of headroom so the
|
||||
composer can apply a fade or dissolve without clipping the moment.
|
||||
|
||||
If a clip is too short to fill its target hold, either:
|
||||
|
||||
- slow it down (speed 0.5-0.75, fine on static-ish footage, bad on
|
||||
anything with sync motion or faces talking),
|
||||
- let it cut early and borrow the remaining duration from the next
|
||||
slot's hold,
|
||||
- or swap to the #2 candidate from the rejected-picks log.
|
||||
|
||||
Do NOT hold on the last frozen frame. A freeze-frame in a doc montage
|
||||
reads as a technical mistake.
|
||||
|
||||
### 4. Sync To The Music Bed
|
||||
|
||||
Read `asset_manifest` for the music asset and load its duration.
|
||||
Documentary montages earn their emotional weight from cuts landing
|
||||
on musical events. Three sync moves:
|
||||
|
||||
- **Downbeat cuts.** If you have bars and beats metadata (from a
|
||||
provided track) or can hear them, place hero cuts on downbeats.
|
||||
If not, evenly-spaced cuts on 4s intervals for a 60bpm bed are a
|
||||
safe default.
|
||||
- **One held silence.** Drop the music out for ~2s at the piece's
|
||||
emotional center. Silence is a tool. Use it once. Use it hard.
|
||||
- **Tail fade.** Music fades under the last 3-5s so the final image
|
||||
can breathe without a musical resolution fighting it.
|
||||
|
||||
Record the music config in `edit_decisions.audio.music` with:
|
||||
|
||||
```json
|
||||
{
|
||||
"asset_id": "asset_music_bed",
|
||||
"volume": 0.7,
|
||||
"fade_in_seconds": 1.0,
|
||||
"fade_out_seconds": 4.0,
|
||||
"ducking": false
|
||||
}
|
||||
```
|
||||
|
||||
`ducking: false` is the default for this pipeline — there's no
|
||||
narration to duck under. If the user approved a narration track, set
|
||||
ducking to true and let it dip during segments.
|
||||
|
||||
### 5. Choose Transitions From A Small Vocabulary
|
||||
|
||||
Documentary montage uses maybe four transitions total across the
|
||||
entire piece:
|
||||
|
||||
| Transition | Use |
|
||||
|------------|-----|
|
||||
| `cut` (hard) | Default. Most cuts are hard cuts. |
|
||||
| `dissolve` (0.5-1.0s) | Emotional sibling clips, time passage |
|
||||
| `fade_to_black` (0.5s, then back up) | Act breaks in 3-act shape, or once near the end |
|
||||
| `fade_in` (first shot) / `fade_out` (last shot) | 0.5-1.0s bookends |
|
||||
|
||||
**Do not use:**
|
||||
|
||||
- wipes,
|
||||
- push/slide transitions,
|
||||
- zoom blurs,
|
||||
- RGB splits,
|
||||
- light leaks,
|
||||
- glitch effects.
|
||||
|
||||
These read as social-media edit language and will break the
|
||||
documentary register. If the piece is getting boring, fix the clip
|
||||
choices or the pacing, don't add transition flash.
|
||||
|
||||
Record each cut's `transition_in` / `transition_out` per the schema.
|
||||
Default `transition_in: "cut"` on most cuts.
|
||||
|
||||
### 6. Apply Register Continuity
|
||||
|
||||
Mixed-era corpora look wildly different. Pexels 2023 is clean, sharp,
|
||||
color-graded. Prelinger 1962 is grainy, warm, squared-off aspect.
|
||||
NASA archival is often low-res with text overlays. If you mash them
|
||||
together raw, the piece looks like a Wikipedia article.
|
||||
|
||||
You have two tools to smooth this:
|
||||
|
||||
1. **Crop to a uniform aspect ratio.** Pick one: 16:9 cinematic
|
||||
(`2.35:1` letterbox on top/bottom) for hero pieces, 9:16 for
|
||||
social. Enforce in the `transform.crop` field of each cut.
|
||||
2. **Flag the piece for a uniform color grade at compose time.** Put
|
||||
a `grade_profile` hint in `edit_decisions.metadata`. The compose
|
||||
director will apply a LUT across the whole timeline.
|
||||
|
||||
Don't try to color-grade individual clips here. That's the compose
|
||||
stage. Your job is to flag the need.
|
||||
|
||||
### 7. Enforce Adjacent Diversity One More Time
|
||||
|
||||
Walk the timeline in pairs. For each consecutive (cut_n, cut_n+1):
|
||||
|
||||
- Are they the same subject at the same scale? If yes, you have a
|
||||
slideshow moment. Swap one for a clip at a different scale (wide
|
||||
vs close).
|
||||
- Are they the same color palette (two night-blue clips back to
|
||||
back)? If yes, break the pattern at least every 4 cuts.
|
||||
- Are they the same motion direction (two left-to-right pans)? If
|
||||
yes, flip the second's horizontal axis or reorder.
|
||||
|
||||
Log any swaps you made in `metadata.diversity_swaps`.
|
||||
|
||||
### 8. The L-Cut Move (Optional But Powerful)
|
||||
|
||||
For any transition between two clips where the outgoing clip has
|
||||
strong ambient audio (rain, footsteps, traffic), carry the audio
|
||||
under the incoming clip for 0.5-1.5s. This is an L-cut and it
|
||||
welds two shots together more tightly than any visual transition.
|
||||
|
||||
Implement via the schema by using a short `dissolve` transition OR
|
||||
by layering the outgoing clip's audio as an SFX entry in
|
||||
`edit_decisions.audio.sfx` with a delayed end.
|
||||
|
||||
Documentary montages with L-cuts feel 50% more coherent than ones
|
||||
without. Use them on the 3-4 hardest transitions in the piece.
|
||||
|
||||
### 9. Emit The Edit Decisions
|
||||
|
||||
Canonical shape for this pipeline:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"cuts": [
|
||||
{
|
||||
"id": "cut_01",
|
||||
"source": "asset_slot_01",
|
||||
"in_seconds": 1.2,
|
||||
"out_seconds": 5.2,
|
||||
"layer": "primary",
|
||||
"transform": { "scale": 1.0, "position": "center" },
|
||||
"transition_in": "fade_in",
|
||||
"transition_out": "cut",
|
||||
"transition_duration": 0.8,
|
||||
"reason": "opening hero — raindrop on asphalt, 4s hold, slow-motion streetlamp glow"
|
||||
},
|
||||
{
|
||||
"id": "cut_02",
|
||||
"source": "asset_slot_02",
|
||||
"in_seconds": 2.0,
|
||||
"out_seconds": 5.5,
|
||||
"layer": "primary",
|
||||
"transition_in": "cut",
|
||||
"transition_out": "cut",
|
||||
"reason": "umbrella opening in doorway, hard cut from raindrop → street"
|
||||
}
|
||||
],
|
||||
"audio": {
|
||||
"music": {
|
||||
"asset_id": "asset_music_bed",
|
||||
"volume": 0.7,
|
||||
"fade_in_seconds": 1.0,
|
||||
"fade_out_seconds": 4.0,
|
||||
"ducking": false
|
||||
}
|
||||
},
|
||||
"metadata": {
|
||||
"pipeline": "documentary-montage",
|
||||
"tone": "elegiac",
|
||||
"shape": "list",
|
||||
"total_duration_seconds": 90.0,
|
||||
"hold_table_used": { "base": 4.0, "min": 2.5, "max": 7.0 },
|
||||
"grade_profile": "warm_film_100",
|
||||
"reorder_notes": [],
|
||||
"diversity_swaps": [
|
||||
{ "at": "cut_07-cut_08", "reason": "two wide rooftops-in-rain adjacent, swapped 08 for #2 pick" }
|
||||
],
|
||||
"silence_window": { "start_seconds": 54.0, "end_seconds": 56.0 },
|
||||
"l_cuts": [
|
||||
{ "from_cut": "cut_05", "to_cut": "cut_06", "carry_seconds": 1.2, "channel": "ambient_rain" }
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 10. Quality Gate
|
||||
|
||||
- `sum(out - in for cut in cuts)` is within ±10% of
|
||||
`brief.duration_seconds`.
|
||||
- Hero slots have the longest holds.
|
||||
- No two adjacent cuts share subject AND scale.
|
||||
- The transition vocabulary is at most 4 distinct values.
|
||||
- Music config exists (or brief explicitly says no music).
|
||||
- At least one `silence_window` entry for pieces >= 60s.
|
||||
- Every cut has a one-line `reason` — if you can't write one, the
|
||||
cut is arbitrary and should be reconsidered.
|
||||
- `metadata.total_duration_seconds` matches the sum of cut durations.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Cutting by information density instead of rhythm.** A doc
|
||||
montage is not a Wikipedia article. "But I need to show this" is
|
||||
not a reason — if the image doesn't sustain a hold, it doesn't
|
||||
belong.
|
||||
- **Over-using dissolves.** A dissolve on every cut says "I couldn't
|
||||
commit". Commit.
|
||||
- **Ignoring the music bed until the end.** Music is not a sweetener
|
||||
you add at compose time. It is a timing grid you cut TO.
|
||||
- **Letting the final image be a weak one.** The last frame is
|
||||
disproportionately remembered. If it's weak, swap it — the scene
|
||||
director's slot ordering is a strong suggestion, not a contract.
|
||||
- **Freeze-frame endings.** Reads as technical error. End on a
|
||||
fade-to-black instead.
|
||||
- **Silently adding a narration because the edit feels thin.** Major
|
||||
change. Ask.
|
||||
- **Hiding clip provider in the cuts.** Every `cut.source` must be
|
||||
an `asset_manifest` asset_id so provenance survives.
|
||||
- **Three different transition types in the first 15 seconds.**
|
||||
Readers will feel the edit working. Restraint is the brand.
|
||||
|
||||
## Worked Pacing Example — "A Minute in the Rain"
|
||||
|
||||
90 seconds, elegiac, list shape, 15 hero-flagged slots.
|
||||
|
||||
- Base hold 4.0s × 15 = 60s. Short by 30s.
|
||||
- Add 30s across 3 hero slots (1, 11, 15) at +10s each:
|
||||
hero_1 = 5.5s, hero_11 = 6.0s, hero_15 = 7.0s.
|
||||
- Tighten slots 4, 7, 13 to 3.0s each (small cutaways).
|
||||
- Insert silence_window 54.0-56.0s (right before hero_11).
|
||||
- L-cut slot_10 (boot in puddle) → slot_11 (lit window across
|
||||
street), carry rain-on-glass ambient 1.2s.
|
||||
- First cut `fade_in` 1.0s, last cut `fade_out` 1.5s.
|
||||
- All other cuts hard.
|
||||
- Music fades in 1.0s, fades out 4.0s under hero_15 + black.
|
||||
|
||||
This gives a 90s piece with 3 breathing points (fade_in, silence,
|
||||
fade_out), a clear hero arc (slots 1 → 11 → 15), and no adjacent
|
||||
scale collisions.
|
||||
@@ -0,0 +1,89 @@
|
||||
# Executive Producer - Documentary Montage Pipeline
|
||||
|
||||
## When To Use
|
||||
|
||||
The user wants a short (30-180s) non-narrative piece built from existing
|
||||
footage — a thematic collage, essay film, or Adam-Curtis-style tone
|
||||
poem. The piece is NOT a narrated explainer, NOT a talking head, NOT a
|
||||
single extended scene. It is an arranged sequence of real-world clips
|
||||
whose meaning emerges from juxtaposition (Kuleshov effect,
|
||||
Eisenstein's intellectual montage).
|
||||
|
||||
This is the right pipeline when the brief includes phrases like:
|
||||
|
||||
- "a montage about...",
|
||||
- "show me the feeling of...",
|
||||
- "like a tone poem",
|
||||
- "documentary-style collage",
|
||||
- "everyone who has ever..." / "the life of..." / "a portrait of...",
|
||||
- "cut together from stock footage",
|
||||
- "Adam Curtis", "Errol Morris", "Chris Marker",
|
||||
- "no narration, just images".
|
||||
|
||||
If the user asks for an explainer, a trailer with generated clips, or
|
||||
a talking-head video, pick a different pipeline.
|
||||
|
||||
## Philosophy
|
||||
|
||||
Documentary montage is retrieval-first, not generation-first.
|
||||
The corpus is the raw material; the edit is the thinking. Your job
|
||||
across all stages is to:
|
||||
|
||||
1. **Enlarge the search space before committing**. Build a corpus
|
||||
bigger than you think you need so the edit has room to breathe.
|
||||
2. **Let juxtaposition do the talking**. Two mundane clips next to
|
||||
each other can mean something neither one means alone.
|
||||
3. **Trust the footage**. If a clip shows a thing plainly, don't
|
||||
explain it with text or voice-over.
|
||||
4. **Pace is the message**. Cut on beat. Hold on images that earn it.
|
||||
Short cuts = urgency, long holds = grief/weight/awe.
|
||||
|
||||
## Stages
|
||||
|
||||
| Stage | Director skill | Produces |
|
||||
|-------|----------------|----------|
|
||||
| `idea` | `idea-director.md` | brief (topic, tone, duration, shape) |
|
||||
| `scene` | `scene-director.md` | shot_list (slot descriptions + queries) |
|
||||
| `assets` | `asset-director.md` | asset_manifest (corpus built + per-slot picks) |
|
||||
| `edit` | `edit-director.md` | edit_decisions (timeline + transitions + music) |
|
||||
| `compose` | `compose-director.md` | render_report (final mp4) |
|
||||
|
||||
Each director skill has its own quality gate. Read the director skill
|
||||
before starting the stage.
|
||||
|
||||
## Core Tools
|
||||
|
||||
| Tool | Role |
|
||||
|------|------|
|
||||
| `corpus_builder` | Fans out across Pexels/Archive.org/NASA, downloads + embeds + indexes |
|
||||
| `clip_search` | Ranks clips for a slot, finds similar sets, diversifies selections |
|
||||
| `video_compose` / Remotion | Renders the final timeline |
|
||||
|
||||
The agent talks to the stock sources through `corpus_builder` — never
|
||||
call adapter classes directly from a skill or director.
|
||||
|
||||
## Cross-Stage Rules
|
||||
|
||||
- **No generated clips** unless the user explicitly asks. This pipeline
|
||||
is about REAL footage, real texture, real grain. Generated B-roll
|
||||
breaks the aesthetic.
|
||||
- **No narration** unless the user explicitly asks. The brief should
|
||||
default to image-only + music. Adding voice is a MAJOR change and
|
||||
requires user approval per the Decision Communication Contract.
|
||||
- **Build the corpus before picking clips**. Do not run clip_search
|
||||
against an empty or half-built corpus. If retrieval results are
|
||||
weak (all scores < 0.25), grow the corpus with new queries.
|
||||
- **Keep a decision log of rejected picks**. When you pass on a clip
|
||||
with a high score, note why (wrong era, overlit, wrong emotional
|
||||
register). This helps the review stage.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- Treating the corpus as a stock library to pick from sequentially
|
||||
instead of as a search index to query per slot.
|
||||
- Arranging clips by score rather than by narrative beat.
|
||||
- Letting visually-repetitive clips sit adjacent. Use
|
||||
`clip_search` with `operation=diversify` before locking the edit.
|
||||
- Over-cutting. Documentary montage lives in the hold, not the jump.
|
||||
- Quietly inserting a narration track because the edit feels "thin".
|
||||
Fix the edit; don't paper over it.
|
||||
@@ -0,0 +1,126 @@
|
||||
# Idea Director - Documentary Montage Pipeline
|
||||
|
||||
## When To Use
|
||||
|
||||
You are turning a user prompt into the brief artifact that every
|
||||
downstream stage will read. For this pipeline, the brief is the
|
||||
thematic core: what the montage is ABOUT, what it should feel like,
|
||||
and how long it should run.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/brief.schema.json` | Artifact validation |
|
||||
| User input | Conversation history | The raw ask |
|
||||
| Meta | `skills/meta/reviewer.md` | Self-review pass |
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Extract The Thematic Question
|
||||
|
||||
A documentary montage answers a question the user could not put into
|
||||
a sentence. Your job is to name that question in ONE line.
|
||||
|
||||
Good thematic questions:
|
||||
|
||||
- "What does it feel like to come home?"
|
||||
- "How did the 20th century think about the future?"
|
||||
- "What happens in a city at 4am?"
|
||||
- "What do all the footprints on Earth look like?"
|
||||
|
||||
Bad thematic questions (too abstract or too concrete):
|
||||
|
||||
- "A video about cities" (too abstract — no feeling)
|
||||
- "A montage with 8 specific shots of the moon" (too concrete — that's
|
||||
a shot list, not a theme)
|
||||
|
||||
### 2. Fix The Tone
|
||||
|
||||
Choose ONE emotional register. Write it down. Everything downstream
|
||||
keys off this.
|
||||
|
||||
Common registers for this pipeline:
|
||||
|
||||
- **elegiac** — long holds, muted color, slow cuts (loss, memory, home)
|
||||
- **urgent** — short cuts, hard sync, motion-heavy (crisis, cities, now)
|
||||
- **reverent** — stately, symmetrical, patient (nature, ritual, scale)
|
||||
- **wry** — ironic juxtaposition, cut on absurdity (consumer culture,
|
||||
politics, mid-century optimism)
|
||||
- **dreamlike** — slow dissolves, repeated motifs, non-linear (childhood,
|
||||
grief, memory)
|
||||
|
||||
### 3. Pick A Duration And A Shape
|
||||
|
||||
Duration matters because it caps the number of beats.
|
||||
|
||||
| Duration | Beats | Use |
|
||||
|----------|-------|-----|
|
||||
| 30-45s | 8-12 cuts | Social/Instagram/reel — one feeling, no arc |
|
||||
| 60-90s | 15-25 cuts | Standard short — mini arc with a turn |
|
||||
| 2-3 min | 30-50 cuts | Proper essay montage — 3-act arc possible |
|
||||
|
||||
Shape options:
|
||||
|
||||
- **single-image expansion** — one idea, held from many angles (good
|
||||
for elegiac pieces under 60s)
|
||||
- **before/after** — first half establishes, second half turns (good
|
||||
for wry or urgent registers)
|
||||
- **three-act** — setup → turn → release (the Adam Curtis move, needs
|
||||
>90s)
|
||||
- **list/catalogue** — "everyone who..." structure, no arc, just
|
||||
accumulation (good for reverent or elegiac)
|
||||
|
||||
### 4. Note Music Intent
|
||||
|
||||
Documentary montage is inseparable from its music bed. Decide now:
|
||||
|
||||
- user-provided track (put path in `music_plan.source_path`),
|
||||
- music library pick (list what's in `music_library/`),
|
||||
- generated (name the tool and prompt seed),
|
||||
- or none (silence).
|
||||
|
||||
**Warn the user if no music source is available.** Do not silently
|
||||
defer this — it becomes an expensive surprise at the asset stage.
|
||||
|
||||
### 5. Record The Brief
|
||||
|
||||
Minimum fields the brief must carry:
|
||||
|
||||
```json
|
||||
{
|
||||
"topic": "A minute in the rain",
|
||||
"thematic_question": "What does rain show you about a city?",
|
||||
"tone": "elegiac",
|
||||
"duration_seconds": 90,
|
||||
"shape": "list",
|
||||
"sources_allowed": ["pexels", "archive_org", "nasa"],
|
||||
"generated_clips_allowed": false,
|
||||
"narration": "none",
|
||||
"music_plan": { "source": "library", "path": "music_library/dawn_04.mp3" },
|
||||
"era_mix": "any",
|
||||
"target_platform": "social_short"
|
||||
}
|
||||
```
|
||||
|
||||
`era_mix` is a documentary-specific field: "modern" biases toward
|
||||
Pexels, "vintage" biases toward Archive.org Prelinger, "any" leaves it
|
||||
open for the scene director to decide per slot.
|
||||
|
||||
### 6. Quality Gate
|
||||
|
||||
- Thematic question is ONE sentence.
|
||||
- Tone is ONE register from the fixed list.
|
||||
- Duration and shape are concrete numbers / enum values.
|
||||
- Music source is named OR the brief explicitly says "no music".
|
||||
- Sources list is non-empty and at least one is `available` per the
|
||||
tool registry.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- Stating multiple themes ("it's about cities AND technology AND loss").
|
||||
Pick one. The others become downstream associations.
|
||||
- Jumping to shot lists. The brief is about MEANING. Shots come next.
|
||||
- Ignoring duration. A 45s piece with 50 cuts is nausea. A 3-minute
|
||||
piece with 12 cuts is a slideshow.
|
||||
- Forgetting to ask about music. The user usually has an opinion.
|
||||
@@ -0,0 +1,301 @@
|
||||
# Scene Director - Documentary Montage Pipeline
|
||||
|
||||
## When To Use
|
||||
|
||||
The brief exists. You now have to turn a thematic question into a
|
||||
concrete list of SLOTS the retrieval layer can fill. Each slot is an
|
||||
intention ("a silhouette at a doorway at dusk") plus the queries that
|
||||
will find it in the real world (Pexels/Archive.org/NASA).
|
||||
|
||||
This is the most creative stage in the pipeline. Retrieval is only as
|
||||
good as the slot descriptions you write.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Layer | Resource | Purpose |
|
||||
|-------|----------|---------|
|
||||
| Schema | `schemas/artifacts/scene_plan.schema.json` | Artifact validation |
|
||||
| Prior artifact | `state.artifacts["idea"]["brief"]` | Thematic question, tone, duration, shape |
|
||||
| Reference | `skills/pipelines/documentary-montage/executive-producer.md` | Cross-stage rules |
|
||||
| Tools | none yet — this stage is pure planning | — |
|
||||
|
||||
## Mental Model
|
||||
|
||||
The scene director's job is NOT "pick clips". It is "describe what the
|
||||
clips need to be plainly enough that CLIP can find them".
|
||||
|
||||
Think like a location scout, not a stock librarian.
|
||||
|
||||
- A stock librarian says: *"rain in the city montage, 15 clips"*.
|
||||
- A location scout says: *"rain streaking sideways across a bus
|
||||
window at blue hour, passengers' faces in soft focus, traffic
|
||||
lights bleeding red and green through the glass"*.
|
||||
|
||||
The second one is what CLIP can actually rank. The first one is a
|
||||
category label CLIP will match weakly and indiscriminately.
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Turn The Shape Into A Beat Count
|
||||
|
||||
Read the brief's `duration_seconds` and `shape`. Derive the number of
|
||||
slots. Use these defaults unless the tone says otherwise:
|
||||
|
||||
| Tone | Average hold | Slots per 60s |
|
||||
|------|--------------|---------------|
|
||||
| elegiac | 4.0s | ~15 |
|
||||
| reverent | 3.5s | ~17 |
|
||||
| dreamlike | 3.0s | ~20 |
|
||||
| wry | 2.0s | ~30 |
|
||||
| urgent | 1.2s | ~50 |
|
||||
|
||||
Then plan the arc according to shape:
|
||||
|
||||
- **list**: N uniform slots, no inflection.
|
||||
- **before/after**: N/2 before slots + 1 pivot slot + N/2 after slots.
|
||||
- **three-act**: setup (30%) → turn (40%) → release (30%).
|
||||
- **single-image expansion**: 1 anchor image + N variations around it.
|
||||
|
||||
Write the beat count down before writing any slot.
|
||||
|
||||
### 2. Decompose The Thematic Question Into Concrete Beats
|
||||
|
||||
Take the ONE thematic question from the brief and answer it in
|
||||
sensory language. Not themes — textures.
|
||||
|
||||
**Example — "What does rain show you about a city?"**
|
||||
|
||||
Bad decomposition (abstract, unsearchable):
|
||||
|
||||
- "establishing the mood of the city"
|
||||
- "the feeling of being caught in weather"
|
||||
- "the universality of rain"
|
||||
|
||||
Good decomposition (concrete, searchable):
|
||||
|
||||
- a single raindrop hitting dry asphalt in slow motion
|
||||
- an umbrella opening in a doorway, a hand visible
|
||||
- neon signs reflected upside-down in a puddle
|
||||
- rain streaking across a bus window, passengers soft
|
||||
- a taxi roof light pushing through heavy rain, long lens
|
||||
- a storm drain swallowing leaves and water, overhead
|
||||
- a street vendor pulling plastic over a produce cart
|
||||
- steam rising off wet cobblestones under tungsten streetlight
|
||||
- a child's rubber boot stamping into a puddle
|
||||
- a lit apartment window seen through sheets of rain
|
||||
|
||||
Each of those is a SHOT. Each is CLIP-rankable. Each is also *a
|
||||
different angle on the same idea*, which is what gives a list-shaped
|
||||
montage its weight.
|
||||
|
||||
### 3. Write The Slot Description
|
||||
|
||||
Every slot carries a `description` field. This is the text CLIP will
|
||||
embed and rank against. Write it like a good stock-footage tag string
|
||||
— nouns and adjectives, no verbs of intention, no emotion words.
|
||||
|
||||
**Template:**
|
||||
|
||||
```
|
||||
<subject>, <action/pose>, <environment>, <lighting>, <era/texture hint>
|
||||
```
|
||||
|
||||
**Good:**
|
||||
|
||||
- `"a single raindrop hitting dry asphalt, close up, slow motion,
|
||||
warm streetlamp glow"`
|
||||
- `"empty city sidewalk at night after rain, reflected neon,
|
||||
handheld, 1970s grain"`
|
||||
- `"an umbrella opening in a doorway, hand visible, diffused
|
||||
afternoon light, shallow focus"`
|
||||
|
||||
**Bad:**
|
||||
|
||||
- `"the feeling of arriving home"` — emotion word, no subject
|
||||
- `"a warm welcoming moment"` — adjective soup, no image
|
||||
- `"someone going through a door in a symbolic way"` — intent, no shot
|
||||
|
||||
Rule of thumb: if you can't imagine a specific photograph from the
|
||||
description, CLIP can't either.
|
||||
|
||||
### 4. Write 2-3 Queries Per Slot
|
||||
|
||||
The slot description is what CLIP ranks against. The queries are what
|
||||
the `corpus_builder` uses to populate the candidate pool. These are
|
||||
different jobs, so write them differently.
|
||||
|
||||
Give each slot a `queries` array with 2-3 entries:
|
||||
|
||||
1. **Literal query** — the most direct stock-search phrase. This is
|
||||
what a Pexels user would type. `"raindrop on asphalt slow motion"`.
|
||||
2. **Lateral query** — the same idea from a different angle or scale.
|
||||
`"wet pavement close up"`.
|
||||
3. **Association query** (optional, for hero slots) — an adjacent
|
||||
concept that might surface texture clips the literal query misses.
|
||||
`"first rain city street"`.
|
||||
|
||||
Short queries beat long queries for stock search engines. 2-5 words
|
||||
each. No filler words.
|
||||
|
||||
### 5. Target Sources Per Slot (Era-Aware)
|
||||
|
||||
Read `brief.era_mix`. Assign each slot one or more `preferred_sources`
|
||||
based on what footage lives where:
|
||||
|
||||
| Source | Strengths | Use when |
|
||||
|--------|-----------|----------|
|
||||
| `pexels` | Modern HD footage, clean shots, people, cities, nature | Default for modern/any era |
|
||||
| `archive_org` | Prelinger home movies, mid-century educational film, 1940s-1980s texture | Vintage, wry, dreamlike, anything nostalgic |
|
||||
| `nasa` | Earth-from-orbit, astronomy, flight, scale imagery | Reverent, anything about scale, space, planet, flight |
|
||||
|
||||
If `era_mix = "vintage"`, bias slots toward `archive_org` and write
|
||||
queries in period-appropriate vocabulary ("commuter", "housewife",
|
||||
"suburb" not "influencer", "wfh", "coworking").
|
||||
|
||||
If `era_mix = "any"`, mix sources per slot — the scene director
|
||||
decides which slot gets which source based on the beat's meaning.
|
||||
|
||||
### 6. Mark Hero Slots
|
||||
|
||||
Every montage has 2-3 slots the whole piece depends on: the opening
|
||||
image, the turn, the final image. Mark these with `hero: true` in the
|
||||
slot metadata.
|
||||
|
||||
Hero slots get:
|
||||
|
||||
- longer holds (2-4s instead of the tone's default),
|
||||
- bigger candidate pools at asset time (k=30 instead of k=10),
|
||||
- more queries (3 instead of 2).
|
||||
|
||||
### 7. Leave Headroom For The Asset Stage
|
||||
|
||||
Don't over-specify. The asset director's job is to rank candidates
|
||||
against your description. If you nail down the description AND the
|
||||
exact clip, you've done the asset director's job badly and pre-empted
|
||||
its creative choices.
|
||||
|
||||
Rule: describe the slot the way you would describe it to a research
|
||||
assistant over the phone — specific enough to recognise, loose enough
|
||||
to surprise you.
|
||||
|
||||
### 8. Record The Shot List
|
||||
|
||||
Use the `scene_plan.schema.json` artifact with one `scene` per slot.
|
||||
For this pipeline, put documentary-montage-specific fields inside
|
||||
`metadata` on each scene. The canonical shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"scenes": [
|
||||
{
|
||||
"id": "slot_01",
|
||||
"type": "broll",
|
||||
"description": "a single raindrop hitting dry asphalt, close up, slow motion, warm streetlamp glow",
|
||||
"start_seconds": 0.0,
|
||||
"end_seconds": 3.5,
|
||||
"narrative_role": "establish_context",
|
||||
"hero_moment": true,
|
||||
"texture_keywords": ["wet", "slow motion", "streetlamp"],
|
||||
"required_assets": [
|
||||
{ "type": "video", "description": "raindrop on asphalt", "source": "source" }
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"pipeline": "documentary-montage",
|
||||
"shape": "list",
|
||||
"tone": "elegiac",
|
||||
"thematic_question": "What does rain show you about a city?",
|
||||
"slots": [
|
||||
{
|
||||
"id": "slot_01",
|
||||
"description": "a single raindrop hitting dry asphalt, close up, slow motion, warm streetlamp glow",
|
||||
"hero": true,
|
||||
"preferred_sources": ["pexels", "archive_org"],
|
||||
"queries": [
|
||||
"raindrop on asphalt slow motion",
|
||||
"wet pavement close up",
|
||||
"first rain city street"
|
||||
],
|
||||
"min_duration": 3.0,
|
||||
"target_hold_seconds": 3.5,
|
||||
"era_hint": "any"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `scenes[]` array satisfies the schema. The `metadata.slots[]`
|
||||
array is what the asset director actually reads — it carries the
|
||||
retrieval-specific fields (`queries`, `preferred_sources`, `hero`,
|
||||
`era_hint`) that `scene_plan.schema.json` doesn't know about.
|
||||
|
||||
### 9. Quality Gate
|
||||
|
||||
- Slot count matches the beat-count math from step 1.
|
||||
- Every slot `description` follows the noun-and-adjective template —
|
||||
no emotion words, no verbs of intention.
|
||||
- Every slot has 2-3 short queries (5 words or fewer each).
|
||||
- At least 2 slots are marked `hero`.
|
||||
- Sum of `target_hold_seconds` is within ±10% of `brief.duration_seconds`.
|
||||
- If `era_mix = "vintage"`, at least 60% of slots list `archive_org`
|
||||
in `preferred_sources`.
|
||||
- `metadata.thematic_question` echoes the brief verbatim (sanity check
|
||||
that you didn't drift).
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Writing slot descriptions as intentions instead of images.** "A
|
||||
moment of hesitation before entering" is a screenplay direction, not
|
||||
a CLIP query. "A woman standing still on a porch, hand near the
|
||||
knob" is.
|
||||
- **Category queries.** `"home"` and `"family"` match everything and
|
||||
nothing. Push for concrete nouns: door, mat, key, hall, shoe.
|
||||
- **One-query slots.** The second query is cheap insurance — if the
|
||||
first query returns junk, the corpus still has something usable.
|
||||
- **Forgetting duration math.** 90 elegiac seconds is ~15 holds of
|
||||
~6s. If you wrote 40 slots, you've drafted an urgent piece by
|
||||
accident.
|
||||
- **Skipping `era_hint` on a vintage brief.** Pexels will flood the
|
||||
corpus with 2020s HD footage and bury the Prelinger material.
|
||||
- **Letting the thematic question drift.** If the brief says "coming
|
||||
home" and your slot list has three shots of airplanes, the piece
|
||||
will be about travel, not home. Re-read the brief after drafting.
|
||||
|
||||
## Worked Example — "A Minute in the Rain"
|
||||
|
||||
- Duration: 90s, elegiac tone → ~15 slots at ~6s each.
|
||||
- Shape: list (catalogue of weather + city).
|
||||
- Thematic question: "What does rain show you about a city?"
|
||||
|
||||
Sketch of slots (abbreviated):
|
||||
|
||||
1. **hero** single raindrop hitting dry asphalt, slow motion
|
||||
2. umbrella opening in a doorway, diffused afternoon light
|
||||
3. neon sign reflected upside-down in a puddle, handheld
|
||||
4. rain streaking across a bus window, passengers soft focus
|
||||
5. a taxi roof light pushing through heavy rain, long lens
|
||||
6. storm drain swallowing leaves and water, overhead
|
||||
7. a street vendor pulling plastic over a produce cart
|
||||
8. wet cobblestone alley, steam rising, tungsten streetlamp
|
||||
9. rooftop antennae against a grey sky, wide shot
|
||||
10. a child's rubber boot stamping a puddle, low angle
|
||||
11. **hero** a lit apartment window seen through sheets of rain
|
||||
12. windshield wipers at night, colored city lights beyond
|
||||
13. rain beading on a parked bicycle seat, macro
|
||||
14. footprints filling with water on a tiled station floor
|
||||
15. **hero** first patch of blue sky breaking through grey clouds
|
||||
|
||||
Each slot gets:
|
||||
|
||||
- `description` in the noun-and-adjective template,
|
||||
- 2-3 short queries (e.g. slot 5: `"taxi heavy rain", "yellow cab
|
||||
wet street night", "city traffic downpour"`),
|
||||
- `preferred_sources` (slots 1-6 → pexels+archive_org, slot 8 →
|
||||
archive_org for period texture, slot 11 → pexels),
|
||||
- `hero: true` on slots 1, 11, 15,
|
||||
- `target_hold_seconds` summing to ~90.
|
||||
|
||||
This is the artifact the asset director will run retrieval against.
|
||||
Reference in New Issue
Block a user