docs(prompting): adopt 5-aspect video specification across skills
Incorporate the structured taxonomy from Lin et al. "Building a Precise Video Language with Human-AI Oversight" (CMU/Harvard, arXiv 2604.21718v2). The paper proves prompts structured around five aspects (Subject / Subject Motion / Scene / Spatial Framing / Camera) unlock controllable cinematography in fine-tuned video generation models. Off-the-shelf VLMs already nail subject and scene; the gains live in motion, spatial, and camera, which prompts routinely omit. Universal layer (skills/creative/video-gen-prompting.md, +125 lines): - 5-aspect prompt skeleton replaces flat formula - Camera movements regrouped (translation / rotation / lens-only) with dolly!=zoom, pan!=truck, bird's-eye!=aerial disambiguations - New primitive tables: camera height, camera angle, POV, lens distortion (fisheye vs barrel), focus / DoF (rack / pull / tracking), playback speed (6 modes), subject transitions - Order-matters and self-contained-prompt rules - Identity anchoring rule for multi-shot - Strict static-shot rule, anti-subjective callout, overlays-not-depth - Per-model word-count guidance Per-model guides (sora, veo, hunyuan, ltx, seedance): - Add the primitives each model honors literally - Word-count sweet spots per model - Strengthen seedance verbatim-identity and subject-transition guidance Pipeline directors (cinematic / explainer / animation scene-director, cinematic / explainer asset-director): - 5-aspect scene-plan checklist (per-pipeline adapted) - Overlays-not-depth callout - Pre / critique / post self-review loop for generation prompts Reviewer (skills/meta/reviewer.md): - CHAI critique-quality rules: accurate / complete / constructive - Critical findings now require a proposed_fix Storytelling, cinematic, broll, video-reference-analyst: - Anti-subjective rule (replace mood adjectives with visual causes) - Camera-intent-per-beat for script writers - POV column in stock-footage query templates - 5-aspect structured output mandatory for reference-video analysis skills/INDEX.md: video-gen-prompting marked as canonical 5-aspect spec.
This commit is contained in:
calesthio
@@ -70,14 +70,18 @@ Fallback: AI-generated image of server racks
|
||||
|
||||
### Query Templates by Scene Type
|
||||
|
||||
| Scene Type | Query Template | Example |
|
||||
Add a **POV keyword** to every query. Stock libraries (Pexels, Pixabay, Storyblocks, Artgrid) explicitly index POV terms — drone, aerial, OTS (over-the-shoulder), macro, top-down, dashcam, FPV, handheld, locked-off — and adding the POV often unlocks better matches than refining the subject. The CMU/Harvard CHAI taxonomy treats POV as a first-class Scene aspect for the same reason: it changes which library shelf you're searching.
|
||||
|
||||
| Scene Type | Query Template | Example with POV |
|
||||
|-----------|---------------|---------|
|
||||
| Establishing | `[place] [time of day]` | "tokyo skyline night" |
|
||||
| Activity | `[person] [action]` | "scientist microscope" |
|
||||
| Object | `[object] [style]` | "circuit board closeup" |
|
||||
| Nature | `[element] [quality]` | "ocean waves aerial" |
|
||||
| Abstract motion | `[movement] [style]` | "light trails timelapse" |
|
||||
| Workplace | `[setting] [activity]` | "modern office meeting" |
|
||||
| Establishing | `[place] [time of day] [POV]` | "tokyo skyline night drone" |
|
||||
| Activity | `[person] [action] [POV]` | "scientist microscope OTS" |
|
||||
| Object | `[object] [style] [POV]` | "circuit board macro top-down" |
|
||||
| Nature | `[element] [quality] [POV]` | "ocean waves aerial drone" |
|
||||
| Abstract motion | `[movement] [style] [POV]` | "light trails timelapse locked-off" |
|
||||
| Workplace | `[setting] [activity] [POV]` | "modern office meeting handheld" |
|
||||
|
||||
If the scene description doesn't already imply a POV, ask the script/scene director — don't default to "no POV." A wrong-POV match (handheld when the scene needs drone) is harder to fix than a wrong color grade.
|
||||
|
||||
## Evaluating Stock Footage Quality
|
||||
|
||||
@@ -89,6 +93,7 @@ When the stock tool returns results, evaluate before using:
|
||||
- **Style compatibility:** Doesn't clash with the playbook's visual style
|
||||
- **No watermarks:** Pexels/Pixabay are license-free, but verify
|
||||
- **Composition:** Subject is well-framed, not cut off awkwardly
|
||||
- **POV match:** Does the footage's actual POV (drone, OTS, macro, handheld, locked-off, etc.) match what the scene needs? A wrong POV — e.g., handheld when the scene wants drone — is **more costly to fix than a wrong color grade**. Reject and re-query rather than try to crop your way out of it.
|
||||
|
||||
### Video Criteria (all image criteria plus)
|
||||
- **Duration:** At least as long as the scene needs (can trim, can't extend)
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
> Sources: No Film School editorial guides, StudioBinder filmmaking resources, Film Riot
|
||||
> production tutorials, CinematographyDB shot databases, Walter Murch "In the Blink of an Eye"
|
||||
|
||||
> For the universal cinematography vocabulary (camera, lens, motion, focus primitives, plus the 5-aspect Subject / Subject Motion / Scene / Spatial Framing / Camera spec), see `skills/creative/video-gen-prompting.md`. This file layers cinematic-specific conventions on top of those primitives — it does not redefine them.
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```
|
||||
@@ -16,6 +18,19 @@ MUSIC: 60-90 BPM, orchestral or ambient, dynamic (not loop-based)
|
||||
TARGET LUFS: -14 LUFS integrated, -24 LUFS for quiet moments
|
||||
```
|
||||
|
||||
## Replace Mood Adjectives with Visual Causes
|
||||
|
||||
> **"Cinematic" and "epic" don't constrain pixels.** The cinematic look comes from concrete choices: aspect ratio, lens, lighting key, color grade, shot duration, and audio layer count. State those — the rest is decoration.
|
||||
>
|
||||
> The CMU/Harvard CHAI study showed subjective phrasing varies wildly across annotators and model interpretations, which means a beat tagged "moody" routes to a different visual every render. Replace the adjective with the lighting + grade + shot-duration combination that produces moodiness. See `skills/creative/storytelling.md` "Anti-Subjective Rule" for the script-side equivalent.
|
||||
>
|
||||
> | Mood adjective | Cinematic translation |
|
||||
> |---|---|
|
||||
> | "epic" | 2.39:1 letterbox, 24fps, 8s+ shot duration, orchestral score with crescendo |
|
||||
> | "moody" | `moody_dark` grade at 0.6, key light at 1/8 fill, 6s+ contemplative shots, ambient bed at -28dB |
|
||||
> | "intimate" | 1.85:1, 40-50mm equivalent, shallow DoF, 2 audio layers (dialogue + room tone), no music under dialogue |
|
||||
> | "cinematic" | (this word is banned — pick one of the above or describe the actual choices) |
|
||||
|
||||
## Aspect Ratios
|
||||
|
||||
| Ratio | Resolution (in 1080p frame) | Feel | When to Use |
|
||||
@@ -113,6 +128,7 @@ Add a subtle ambient layer to fill silence and create depth:
|
||||
- Highlights should be slightly rolled off (never pure white)
|
||||
- Skin tones must stay on the vectorscope skin tone line
|
||||
- Consistency across all clips — one LUT/profile for the entire video
|
||||
- **If a beat is described as "moody," rewrite as the lighting + grade + shot-duration combination that produces moodiness.** Don't pass mood adjectives to the asset/edit stages.
|
||||
|
||||
## Applying to OpenMontage
|
||||
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
> Source: [Tencent Prompt Handbook](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5/blob/main/assets/HunyuanVideo_1_5_Prompt_Handbook_EN.md)
|
||||
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
|
||||
|
||||
**Word count:** Hunyuan 1.5 reads well at 80–200 words; doesn't reward 400-word essays.
|
||||
|
||||
## HunyuanVideo Prompt Formula
|
||||
|
||||
### Text-to-Video
|
||||
@@ -34,15 +36,28 @@ Describe lighting with multiple dimensions:
|
||||
|
||||
| Movement | Type | HunyuanVideo Prompt |
|
||||
|----------|------|-------------------|
|
||||
| Crane / Pedestal | Vertical | "camera rises vertically" |
|
||||
| Truck / Tracking | Horizontal | "camera tracks left alongside subject" |
|
||||
| Dolly In | Push | "camera pushes forward toward subject" |
|
||||
| Dolly Out | Pull | "camera pulls back from subject" |
|
||||
| Pan | Rotation | "camera pans right across the scene" |
|
||||
| Crane / Pedestal | Translation (vertical) | "camera rises vertically" |
|
||||
| Truck / Tracking | Translation (horizontal) | "camera tracks left alongside subject" |
|
||||
| Dolly In | Translation (push) | "camera pushes forward toward subject" |
|
||||
| Dolly Out | Translation (pull) | "camera pulls back from subject" |
|
||||
| Pan | Rotation (yaw) | "camera pans right across the scene" |
|
||||
| Tilt | Rotation (pitch) | "camera tilts upward to follow the rocket" |
|
||||
| Roll | Rotation (Z-axis / Dutch) | "camera rolls clockwise into a Dutch tilt" |
|
||||
| Orbit | Circular | "camera orbits around subject" |
|
||||
| Follow | Lock-on | "camera follows subject from behind" |
|
||||
| Zoom | Lens-only (focal length) | "camera slowly zooms in on the figure" |
|
||||
| Rack focus | Lens-only (focal plane, snap) | "rack focus from the foreground bottle to the figure in the background" |
|
||||
| Pull focus | Lens-only (focal plane, gradual) | "camera shifts focus from foreground X to background Y" |
|
||||
| Static | Fixed | "static camera, no movement" |
|
||||
|
||||
### Focal-plane labels at start AND end of dynamic-DoF shots
|
||||
|
||||
Hunyuan benefits when both endpoints of focus-changing shots are stated. State where focus starts AND where it lands — don't leave one implicit.
|
||||
|
||||
Example: "shallow DoF; focus on the foreground bottle at start; focus pulls to the figure in the background by end."
|
||||
|
||||
Without both endpoints, Hunyuan often defaults to deep focus or holds on the wrong plane.
|
||||
|
||||
### Style Keywords
|
||||
|
||||
**Photorealistic / Cinematic**:
|
||||
@@ -65,6 +80,10 @@ When using image-to-video, the input image defines appearance. Your prompt shoul
|
||||
|
||||
**Bad I2V prompt**: "A beautiful woman in a red dress standing in a forest" — this repeats what the image already shows.
|
||||
|
||||
### Order motions temporally
|
||||
|
||||
Describe motion in temporal order; if multiple movements occur, separate them ("first the camera pans right, then tilts upward"). Hunyuan executes motion in the order it appears in the prompt — bundling two movements into one clause causes one of them to be dropped or blended.
|
||||
|
||||
## Example (T2V)
|
||||
|
||||
```
|
||||
|
||||
@@ -11,9 +11,13 @@ LTX-2 uses a clean, focused prompt structure:
|
||||
2. **Set the scene** — lighting, color palette, textures, atmosphere
|
||||
3. **Describe the action** — natural sequence flowing from beginning to end
|
||||
4. **Define the character(s)** — physical cues (age, hair, clothes), not abstract labels
|
||||
5. **Camera movement(s)** — specify how and when; describe what appears AFTER the movement
|
||||
5. **Camera movement(s)** — specify how and when; describe what appears AFTER the movement. (LTX honors the translation/rotation/lens distinction: `dolly` ≠ `zoom`, `pan` ≠ `truck`. Pick the right family — translation moves the rig, rotation pivots it, lens-only changes focal length or focal plane without moving the camera.)
|
||||
6. **Describe the audio** — ambient sound, music, speech, or singing
|
||||
|
||||
### Strict-Static-Shot rule
|
||||
|
||||
If you write "static camera," the shot must have NO movement, NO focus change, NO zoom. LTX takes "static" literally — adding any motion verb later in the prompt will either be ignored or will produce a glitch where the camera contradicts itself. Pick one: static, OR a single named movement.
|
||||
|
||||
## LTX-Specific Tips
|
||||
|
||||
### Post-Movement Description
|
||||
@@ -51,6 +55,7 @@ LTX organizes styles into three families:
|
||||
| Overloaded scenes | Many characters/actions reduces coherence |
|
||||
| Conflicting lighting descriptions | Pick one setup, commit to it |
|
||||
| Starting complex | Build up: simple prompt first, add layers |
|
||||
| Prompts over ~80 words | LTX-2 degrades past that. Pick the most important 5–6 elements. |
|
||||
|
||||
## LTX Technical Notes
|
||||
|
||||
|
||||
@@ -44,6 +44,8 @@ Seedance is unusually literal about camera language, multi-shot cuts, and quoted
|
||||
|
||||
## Multi-shot pattern
|
||||
|
||||
> **Repeat identity verbatim across every shot.** "the same character" / pronouns / "Aang again" do not work. Repeat the 3–6 disambiguating visual attributes verbatim in every shot block. Seedance treats each shot as if you said it cold.
|
||||
|
||||
Seedance honors explicit shot lists:
|
||||
|
||||
```
|
||||
@@ -63,6 +65,22 @@ Style: anamorphic lens, teal-orange cinematic grade, 35mm film grain.
|
||||
Audio: rising orchestral swell with low taiko pulse, wind, distant wingbeats.
|
||||
```
|
||||
|
||||
### Subject transition primitives in multi-shot
|
||||
|
||||
Seedance handles four distinct ways a subject can enter or exit a shot. Naming the primitive explicitly helps the model build the right transition between shots.
|
||||
|
||||
- **Subject revealing** (by camera move OR subject move) — the subject becomes visible mid-shot.
|
||||
Example: `Shot 2 (slow truck right): empty corridor at first; the camera trucks right to reveal Aang — bald, blue arrow tattoo, orange robes — pressed flat against the wall.`
|
||||
|
||||
- **Subject disappearing** — the subject leaves frame, by motion or occlusion.
|
||||
Example: `Shot 4 (static wide): Aang — bald, blue arrow tattoo, orange robes — sprints into the temple doorway and is swallowed by shadow; camera holds on the empty threshold.`
|
||||
|
||||
- **Subject switching** (rack focus / camera move) — focus or framing transfers from one subject to another.
|
||||
Example: `Shot 5 (close-up, rack focus): rack focus from Aang's glowing arrow tattoo in foreground to Sokka — dark hair, blue tunic, boomerang on back — emerging from the mist behind.`
|
||||
|
||||
- **Complex alternating focus** — focus oscillates between two subjects within one shot.
|
||||
Example: `Shot 7 (medium two-shot, alternating rack focus): focus on Aang — bald, blue arrow tattoo, orange robes — as he speaks, then pulls to Katara — long brown hair, blue water-tribe parka — as she answers, then back to Aang on the final beat.`
|
||||
|
||||
## Lip-sync pattern
|
||||
|
||||
```
|
||||
@@ -84,6 +102,7 @@ Sokka, half a step behind, replies: "Then we fight."
|
||||
| `generate_audio` | Keep `true` — sync audio is the moat. Strip in compose if unused. |
|
||||
| `model_variant` | `standard` for hero + multi-shot + camera-heavy. `fast` for b-roll, previews, latency-capped jobs. |
|
||||
| `seed` | Lock once a shot composition reads; iterate variants with the same seed. |
|
||||
| `prompt length` | 200–400 words for hero shots; 80–150 for inserts. Seedance is one of the few models that rewards long, structured 5-aspect prompts. |
|
||||
|
||||
## Iteration strategy
|
||||
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
> Source: [OpenAI Sora 2 Cookbook](https://developers.openai.com/cookbook/examples/sora/sora2_prompting_guide)
|
||||
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
|
||||
|
||||
**Word count:** Sora 2 plateaus around 100–250 words. Past 250, additional detail rarely improves output.
|
||||
|
||||
## Sora-Specific Prompt Template
|
||||
|
||||
Sora responds best to a structured format with prose + cinematography block + action beats:
|
||||
@@ -40,6 +42,9 @@ Sora uniquely responds to these production-level details that most models ignore
|
||||
| **Wardrobe** | "navy coat, sleeves rolled, suspenders loose" |
|
||||
| **Finishing** | "fine-grain overlay, mild halation, gate weave, soft vignette" |
|
||||
| **Shutter** | "180° shutter angle" |
|
||||
| **Playback speed** | "speed ramp from 1x to 0.25x mid-shot", "stop-motion staccato", "time-reversed exhale" |
|
||||
| **Lens distortion** | "fisheye barrel distortion at the edges", "subtle barrel curvature on straight lines" |
|
||||
| **Focus mode** | "rack focus from foreground bottle to background figure", "deep focus, FG to BG sharp" |
|
||||
|
||||
## What Sora Does Differently
|
||||
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
> Source: [Vertex AI Video Gen Prompt Guide](https://cloud.google.com/vertex-ai/generative-ai/docs/video/video-gen-prompt-guide)
|
||||
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
|
||||
|
||||
**Word count:** VEO 3.1 sweet spot is 100–250 words; longer prompts stop helping.
|
||||
|
||||
## VEO-Specific 14-Component Structure
|
||||
|
||||
VEO responds to the most comprehensive prompt structure of any model:
|
||||
@@ -29,16 +31,30 @@ VEO responds to the most comprehensive prompt structure of any model:
|
||||
- **Negative prompts**: Explicitly supported — "no text overlays, no watermarks, no lens flare"
|
||||
- **Editing vocabulary**: Understands "match cut", "jump cut", "montage", "split diopter" as prompt terms.
|
||||
|
||||
### Camera vocabulary VEO honors literally
|
||||
|
||||
VEO 3.1 distinguishes the three camera-motion families and treats their tokens as separate primitives. Mixing them up (e.g. asking for a "zoom" when you mean a "dolly") will produce the wrong move.
|
||||
|
||||
- **Translation (rig physically moves):** `dolly` (in/out along the lens axis), `truck` (left/right laterally), `pedestal` (up/down vertically)
|
||||
- **Rotation (rig stays put, camera rotates):** `pan` (yaw, left/right), `tilt` (pitch, up/down), `roll` (Dutch / Z-axis)
|
||||
- **Lens-only (rig and body don't move):** `zoom` (focal length change), `rack focus` / `pull focus` / `focus tracking` (focal-plane change)
|
||||
|
||||
dolly ≠ zoom; pan ≠ truck. VEO follows whichever token leads.
|
||||
|
||||
## VEO Lens Effects (Unique)
|
||||
|
||||
VEO specifically responds to optical effects most models ignore:
|
||||
|
||||
| Effect | Prompt Language |
|
||||
|--------|----------------|
|
||||
| **Rack focus** | "rack focus from foreground flower to background figure" |
|
||||
| **Rack focus** | "rack focus from foreground flower to background figure" (snap shift) |
|
||||
| **Pull focus** | "slow pull focus from the candle in the foreground to the doorway behind" (gradual, slower than rack) |
|
||||
| **Focus tracking** | "focus tracks the runner as she crosses frame; background stays soft" (focus follows a moving subject) |
|
||||
| **Dolly zoom (vertigo)** | "vertigo effect as character realizes the truth" |
|
||||
| **Fisheye** | "fisheye lens distortion, skatepark POV" |
|
||||
| **Lens flare** | "anamorphic lens flare from setting sun" |
|
||||
| **Anamorphic lens flare** | "anamorphic lens flare streaking horizontally from setting sun" |
|
||||
|
||||
These three focus modes (rack, pull, tracking) are different — VEO 3.1 honors the distinction per the paper.
|
||||
|
||||
## VEO Art Movement References
|
||||
|
||||
|
||||
@@ -57,6 +57,32 @@ For a **3-minute explainer video** (scale proportionally for other lengths):
|
||||
| 3 min | 3-5 | 8s | 22s | 100s | 30s | 15s |
|
||||
| 5 min | 5-8 | 10s | 30s | 180s | 50s | 20s |
|
||||
|
||||
## Anti-Subjective Rule
|
||||
|
||||
> Hooks, beats, and section descriptions in OpenMontage scripts must describe the **visual cause** of the emotion, not the emotion itself. The CMU/Harvard CHAI study showed that subjective phrasing varies wildly across annotators and across model interpretations — so it does not constrain pixels and it doesn't reliably guide downstream generation tools.
|
||||
>
|
||||
> | Avoid | Use instead |
|
||||
> |---|---|
|
||||
> | "epic reveal" | "wide aerial pull-back; subject silhouetted against rising sun" |
|
||||
> | "inspiring moment" | "low angle on the subject's face; light catches the edge of a tear" |
|
||||
> | "moody atmosphere" | "low-key key light, lifted shadows by 2 stops, fog volumetrics" |
|
||||
> | "powerful music swell" | "music drops out at 0:42, holds 1.5s of silence, returns with low taiko at half tempo" |
|
||||
>
|
||||
> The rule applies to script narration AND to the metadata fields scene-director consumes. For the universal vocabulary that names these visual primitives, see `skills/creative/video-gen-prompting.md`.
|
||||
|
||||
## Subject Transitions in the Script
|
||||
|
||||
When a script beat introduces a new subject, kills one off, or hands focus from one subject to another, **name the transition explicitly** so the scene-director doesn't have to infer it. The CMU/Harvard taxonomy uses four labels:
|
||||
|
||||
| Label | What it means |
|
||||
|---|---|
|
||||
| **revealing** | A new subject enters frame or is uncovered (door opens, camera pans to find them, fog clears). |
|
||||
| **disappearing** | An existing subject leaves frame or is removed (walks out, fades, eclipsed). |
|
||||
| **switching** | Focus jumps from subject A to subject B (cut, rack focus, camera whip). |
|
||||
| **complex-alternating** | Multiple subjects trade focus repeatedly within a beat (debate cross-cutting, ensemble action). |
|
||||
|
||||
Add 1-2 sentences in the beat describing the mechanism (cut, pan, reveal-by-light, etc.). This propagates into the scene_plan as a transition primitive.
|
||||
|
||||
## Hook Types
|
||||
|
||||
| Type | Pattern | Best For |
|
||||
@@ -114,6 +140,20 @@ Don't explain the answer. **Reconstruct the reasoning path** so the viewer feels
|
||||
**Progressive Revelation:** Never show the full picture at once. Build visuals layer by layer.
|
||||
Each layer arrives exactly when the narration references it.
|
||||
|
||||
## Camera Intent Per Beat
|
||||
|
||||
When writing a beat, attach one line of camera intent so the scene-director doesn't have to invent it from a blank slate. Use the universal vocabulary in `skills/creative/video-gen-prompting.md` (Subject / Subject Motion / Scene / Spatial Framing / Camera). One line is enough — the scene-director will expand it.
|
||||
|
||||
Example beat:
|
||||
|
||||
```
|
||||
[0:30] Concept 1 — atoms aren't tiny planets
|
||||
Narration: "We grew up imagining electrons as tiny planets orbiting the nucleus..."
|
||||
Camera intent: medium shot of stylized atom; slow rotation; deep focus.
|
||||
```
|
||||
|
||||
The camera-intent line is consumed verbatim by the scene-director's 5-aspect spec — keep it concrete, no mood adjectives.
|
||||
|
||||
## Pacing Rules
|
||||
|
||||
| Rule | Value | Source |
|
||||
|
||||
@@ -24,18 +24,50 @@ For model-specific tips, see the linked guides below.
|
||||
| **Kling 2.6** | [Kling Prompt Guide](https://fal.ai/learn/devs/kling-2-6-pro-prompt-guide) | 4-part structure. Supports `++emphasis++` syntax for key elements. |
|
||||
| **Wan 2.1 / CogVideoX** | Use this generic guide | No official prompt guide. Standard cinematographic vocabulary works well. |
|
||||
|
||||
## Order Matters
|
||||
|
||||
When listing multiple subjects or events:
|
||||
|
||||
- **Temporal order** when events unfold over time ("First X enters, then Y reacts").
|
||||
- **Prominence order** when temporal isn't relevant — humans before objects, largest/most-centered first, then secondary subjects.
|
||||
|
||||
## Self-Contained Prompt
|
||||
|
||||
> Write the prompt so that someone who has never seen the intended video could picture the subjects, scene, motion, and camera work from your text alone. If a reader could not picture it, a generation model will not render it.
|
||||
|
||||
## Universal Prompt Formula
|
||||
|
||||
All video generation models respond to this structure. Include what's relevant, omit what's not.
|
||||
Prior work (CMU/Harvard, "Building a Precise Video Language with Human-AI Oversight") shows VLMs reliably describe subject + scene but fail on motion, spatial, and camera. **Forcing prompts to fill all five slots is the highest-leverage change.**
|
||||
|
||||
The OpenMontage canonical 5-aspect skeleton:
|
||||
|
||||
```
|
||||
[Shot type/framing] + [Camera movement] + [Subject description] +
|
||||
[Action/motion in beats] + [Setting/environment] + [Lighting] +
|
||||
[Style/aesthetic] + [Audio/atmosphere]
|
||||
[Subject] type + key visual attributes + how to disambiguate when multiple
|
||||
[Subject Motion] actions in temporal order; subject↔object and subject↔subject interactions; group action
|
||||
[Scene] overlays (separately!) + POV + setting + time of day + scene dynamics
|
||||
[Spatial] shot size + position-in-frame + depth (FG/MG/BG) + camera-height-relative
|
||||
— and how those CHANGE during the clip
|
||||
[Camera] playback speed → lens distortion → height → angle → focus/DoF → steadiness → movement
|
||||
```
|
||||
|
||||
**Shorter prompts = more creative freedom. Longer prompts = more control.**
|
||||
|
||||
### Prompt Length by Model
|
||||
|
||||
Empirical sweet spots from the paper's Section 6 findings — different models reward different prompt densities:
|
||||
|
||||
| Model | Sweet Spot | Notes |
|
||||
|---|---|---|
|
||||
| Seedance 2.0 | 200–400 words for hero shots, 80–150 for inserts | Reward long, structured 5-aspect prompts |
|
||||
| Wan 2.2 | 200–400 words | Fine-tuned on long captions |
|
||||
| Sora 2 / VEO 3.1 | 100–250 words | Plateau past ~250 |
|
||||
| LTX-2 | ≤ 80 words | Degrades past that, keep tight |
|
||||
| Runway Gen-4 | ≤ 60 words | "Focus on motion, not appearance" |
|
||||
|
||||
### Overlays Are Not Scene Depth
|
||||
|
||||
> Overlays (titles, HUD, subtitles, watermarks, framing graphics) are NOT part of the scene's foreground/midground/background depth axis. List them separately with content and placement. Never say "overlay in the foreground."
|
||||
|
||||
---
|
||||
|
||||
## Camera Shot Types
|
||||
@@ -58,23 +90,56 @@ All video generation models respond to this structure. Include what's relevant,
|
||||
|
||||
## Camera Movements
|
||||
|
||||
| Movement | What It Does | Best For |
|
||||
|----------|-------------|----------|
|
||||
| **Static / fixed** | No movement | Dialogue, contemplation, stability |
|
||||
| **Pan** (left/right) | Rotates horizontally | Revealing a scene, following action |
|
||||
| **Tilt** (up/down) | Rotates vertically | Revealing height, slow reveal |
|
||||
| **Dolly in / out** | Physically moves toward/away | Building tension, emphasis |
|
||||
| **Truck** (left/right) | Moves sideways | Parallels subject movement |
|
||||
| **Pedestal** (up/down) | Moves vertically | Smooth elevation changes |
|
||||
| **Crane shot** | Sweeping vertical arcs | Epic reveals, transitions |
|
||||
| **Tracking / follow** | Follows subject | Action sequences, walk-and-talk |
|
||||
| **Arc shot** | Circles around subject | Dramatic emphasis, 360° reveal |
|
||||
| **Zoom** (in/out) | Lens focal length change | Quick emphasis (cheaper than dolly) |
|
||||
| **Whip pan** | Extremely fast pan (blurs) | Transitions, energy, surprise |
|
||||
| **Handheld / shaky cam** | Unstable, human feel | Documentary, urgency, realism |
|
||||
| **Aerial / drone** | High altitude, smooth | Landscapes, establishing shots |
|
||||
| **Slow push-in** | Gradual forward movement | Building intimacy or tension |
|
||||
| **Dolly zoom (vertigo)** | Dolly one way, zoom opposite | Disorientation, revelation |
|
||||
The paper shows current models confuse translation, rotation, and lens-only changes — group your prompts so the model can't conflate them:
|
||||
|
||||
| Group | Primitives | Rule |
|
||||
|---|---|---|
|
||||
| **Translation** (camera physically moves) | dolly in/out, truck left/right, pedestal up/down | "dolly forward toward subject" |
|
||||
| **Rotation** (camera pivots in place) | pan left/right, tilt up/down, roll CW/CCW | "pan right across the room" |
|
||||
| **Lens-only** (no camera move) | zoom in/out, rack focus, pull focus, focus tracking | "zoom in" ≠ "dolly in" |
|
||||
| **Hybrid / signature** | dolly zoom (vertigo), arc/orbit, crane, whip pan, tracking/follow, handheld | "vertigo" only at moments of revelation |
|
||||
| **Stillness states** | static (NO movement at all — strict), micro-shake, locked-off | "static" requires zero movement, focus change, or zoom |
|
||||
|
||||
> **dolly ≠ zoom.** dolly is camera translation; zoom is focal-length change. Models follow whichever token dominates. **pan ≠ truck.** pan rotates, truck translates laterally.
|
||||
|
||||
> **Static shot is strict.** A static shot has zero movement, zero focus change, zero zoom. If any of those occur, do NOT write "static camera" — pick the right movement primitive.
|
||||
|
||||
## Camera Height (relative to ground)
|
||||
|
||||
| Primitive | Example |
|
||||
|---|---|
|
||||
| Aerial-level | "drone-altitude wide of the city" |
|
||||
| Overhead-level | "rooftop height looking across the street" |
|
||||
| Eye-level | "framed at eye level" |
|
||||
| Hip-level | "hip-height tracking shot" |
|
||||
| Ground-level | "low to the ground, ankle height" |
|
||||
| Water-level | "skimming the water surface" |
|
||||
| Underwater | "submerged below the surface" |
|
||||
|
||||
## Camera Angle (relative to subject)
|
||||
|
||||
| Primitive | Definition |
|
||||
|---|---|
|
||||
| **Bird's-eye** | strict top-down. Not the same as aerial. |
|
||||
| High angle | looking down on subject |
|
||||
| Level angle | camera and subject at same height |
|
||||
| Low angle | looking up at subject |
|
||||
| Worm's-eye | looking straight up |
|
||||
| **Dutch angle (fixed)** | tilted horizon held steady |
|
||||
| **Dutch angle (rolling)** | horizon tilt changes during shot |
|
||||
|
||||
> **bird's-eye = strict top-down. aerial = altitude.** A drone shot at 45° looking down is a high angle from aerial height, NOT bird's-eye.
|
||||
|
||||
## Point of View (POV)
|
||||
|
||||
| POV | Example |
|
||||
|---|---|
|
||||
| First-person | "the camera follows the character's viewpoint as they walk" |
|
||||
| Drone | "aerial drone footage of city skyline" |
|
||||
| Over-the-shoulder | "OTS framing of the laptop screen" |
|
||||
| Top-down oblique | "top-down view of the chess board, tilted slightly" |
|
||||
| Dashcam | "vehicle dashcam framing of the road" |
|
||||
| Objective / Neutral | (default — use when no specific POV) |
|
||||
|
||||
## Lighting Vocabulary
|
||||
|
||||
@@ -100,14 +165,51 @@ All video generation models respond to this structure. Include what's relevant,
|
||||
|
||||
| Effect | Result |
|
||||
|--------|--------|
|
||||
| **Shallow depth of field** | Subject sharp, background bokeh |
|
||||
| **Deep focus** | Everything sharp, foreground to background |
|
||||
| **Wide-angle lens** (24-35mm) | Broader view, exaggerated perspective |
|
||||
| **Telephoto** (85mm+) | Compressed perspective, subject isolation |
|
||||
| **Anamorphic** | Stretched aspect, signature lens flares |
|
||||
| **Lens flare** | Streaks from bright light hitting lens |
|
||||
| **Rack focus** | Shift focus between subjects in-shot |
|
||||
| **Fisheye** | Ultra-wide, barrel distortion |
|
||||
|
||||
### Lens Distortion
|
||||
|
||||
The paper distinguishes two primitives that models honor as separate effects — they are NOT interchangeable:
|
||||
|
||||
| Primitive | Effect |
|
||||
|---|---|
|
||||
| **Fisheye** | extreme curvature, edges bent strongly outward |
|
||||
| **Barrel** | mild distortion, straight lines bow slightly outward |
|
||||
|
||||
### Focus / Depth of Field
|
||||
|
||||
| Primitive | Definition |
|
||||
|---|---|
|
||||
| Deep focus | everything sharp, FG to BG |
|
||||
| Shallow DoF | subject sharp, background bokeh |
|
||||
| Extremely shallow DoF | razor-thin focal plane |
|
||||
| Rack focus | shifts focus between two subjects mid-shot |
|
||||
| Pull focus | gradual focus shift (slower than rack) |
|
||||
| Focus tracking | focus follows a moving subject |
|
||||
|
||||
When DoF changes during a shot, label start AND end focal plane (FG/MG/BG/out-of-focus).
|
||||
|
||||
## Subject Transitions
|
||||
|
||||
When subjects enter, leave, or hand off focus, name the transition explicitly:
|
||||
|
||||
| Primitive | When |
|
||||
|---|---|
|
||||
| **Subject revealing** | a new subject enters frame (by subject movement OR camera movement) |
|
||||
| **Subject disappearing** | a subject exits frame |
|
||||
| **Subject switching** | focus shifts from one subject to another (often via rack focus or camera move) |
|
||||
| **Complex alternating** | subjects alternate focus multiple times |
|
||||
|
||||
Always name the cause: "by subject movement" or "by camera movement". This unlocks reveal-style camerawork in multi-shot prompts.
|
||||
|
||||
## Identity Anchoring for Multi-Shot Prompts
|
||||
|
||||
> Models lose character identity across cuts unless you re-state it. In every shot of a multi-shot prompt, repeat the same 3–6 disambiguating visual attributes for each named subject verbatim. Pronouns and "the same character" do not work.
|
||||
>
|
||||
> Example: "Aang — bald, blue arrow tattoo on forehead, orange-and-yellow robes — plants his staff. … Aang — bald, blue arrow tattoo on forehead, orange-and-yellow robes — turns to camera."
|
||||
|
||||
## Style & Aesthetic References
|
||||
|
||||
@@ -137,10 +239,23 @@ All video generation models respond to this structure. Include what's relevant,
|
||||
|
||||
## Temporal Effects
|
||||
|
||||
### Playback Speed
|
||||
|
||||
The paper defines six explicit playback-speed primitives. Use the right one — they're not synonymous:
|
||||
|
||||
| Primitive | Definition |
|
||||
|---|---|
|
||||
| Time-lapse | events significantly faster than real time (clouds racing) |
|
||||
| Fast-motion | slightly faster than real (1x–3x) |
|
||||
| Slow-motion | slower than real |
|
||||
| Stop-motion | frame-by-frame discrete movements |
|
||||
| Speed-ramp | mix of fast and slow within the same shot |
|
||||
| Time-reversed | plays in reverse |
|
||||
|
||||
### Other Temporal Devices
|
||||
|
||||
| Effect | Use |
|
||||
|--------|-----|
|
||||
| **Slow motion** | Emphasis, beauty, impact |
|
||||
| **Time-lapse** | Passage of time, processes |
|
||||
| **Freeze-frame** | Dramatic pause |
|
||||
| **Rapid cuts** | Energy, urgency |
|
||||
| **Continuous / long take** | Immersion, tension |
|
||||
@@ -160,6 +275,15 @@ Put dialogue in quotation marks: `Character says: "Hello world."`
|
||||
|
||||
## What to Avoid
|
||||
|
||||
> **Replace emotional adjectives with the visual cause of the emotion.**
|
||||
> - "sad character" → "tears on cheek, shoulders slumped, staring at empty chair"
|
||||
> - "cinematic mood" → "low-key Rembrandt key + 35mm anamorphic + crushed shadows, lifted-by-2-stops shadow detail"
|
||||
> - "epic" → "low-angle, 24mm wide, sun directly behind subject, lens flare on the rim"
|
||||
>
|
||||
> "Inspiring," "powerful," "moody," "epic" do not constrain pixels.
|
||||
|
||||
> **Static shot is strict.** A static shot has zero movement, zero focus change, zero zoom. If any of those occur, do NOT write "static camera" — pick the right movement primitive.
|
||||
|
||||
| Don't | Why | Do Instead |
|
||||
|-------|-----|-----------|
|
||||
| "Beautiful scene" | Too vague, no visual info | "Wet cobblestone street, warm streetlamp glow reflecting in puddles" |
|
||||
|
||||
Reference in New Issue
Block a user