freedak/OpenMontage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(prompting): adopt 5-aspect video specification across skills

Browse Source 
Incorporate the structured taxonomy from Lin et al. "Building a Precise
Video Language with Human-AI Oversight" (CMU/Harvard, arXiv 2604.21718v2).
The paper proves prompts structured around five aspects (Subject /
Subject Motion / Scene / Spatial Framing / Camera) unlock controllable
cinematography in fine-tuned video generation models. Off-the-shelf VLMs
already nail subject and scene; the gains live in motion, spatial, and
camera, which prompts routinely omit.

Universal layer (skills/creative/video-gen-prompting.md, +125 lines):
- 5-aspect prompt skeleton replaces flat formula
- Camera movements regrouped (translation / rotation / lens-only) with
  dolly!=zoom, pan!=truck, bird's-eye!=aerial disambiguations
- New primitive tables: camera height, camera angle, POV, lens
  distortion (fisheye vs barrel), focus / DoF (rack / pull / tracking),
  playback speed (6 modes), subject transitions
- Order-matters and self-contained-prompt rules
- Identity anchoring rule for multi-shot
- Strict static-shot rule, anti-subjective callout, overlays-not-depth
- Per-model word-count guidance

Per-model guides (sora, veo, hunyuan, ltx, seedance):
- Add the primitives each model honors literally
- Word-count sweet spots per model
- Strengthen seedance verbatim-identity and subject-transition guidance

Pipeline directors (cinematic / explainer / animation scene-director,
cinematic / explainer asset-director):
- 5-aspect scene-plan checklist (per-pipeline adapted)
- Overlays-not-depth callout
- Pre / critique / post self-review loop for generation prompts

Reviewer (skills/meta/reviewer.md):
- CHAI critique-quality rules: accurate / complete / constructive
- Critical findings now require a proposed_fix

Storytelling, cinematic, broll, video-reference-analyst:
- Anti-subjective rule (replace mood adjectives with visual causes)
- Camera-intent-per-beat for script writers
- POV column in stock-footage query templates
- 5-aspect structured output mandatory for reference-video analysis

skills/INDEX.md: video-gen-prompting marked as canonical 5-aspect spec.
This commit is contained in:
calesthio
2026-04-28 08:11:31 -07:00
parent 2b0801030c
commit fdd6457fed
17 changed files with 418 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/creative/prompting/hunyuan-prompting.md
+24 -5
View File
@@ -3,6 +3,8 @@
> Source: [Tencent Prompt Handbook](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5/blob/main/assets/HunyuanVideo_1_5_Prompt_Handbook_EN.md)
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
**Word count:** Hunyuan 1.5 reads well at 80–200 words; doesn't reward 400-word essays.
## HunyuanVideo Prompt Formula
### Text-to-Video
@@ -34,15 +36,28 @@ Describe lighting with multiple dimensions:
| Movement | Type | HunyuanVideo Prompt |
|----------|------|-------------------|
| Crane / Pedestal | Vertical | "camera rises vertically" |
| Truck / Tracking | Horizontal | "camera tracks left alongside subject" |
| Dolly In | Push | "camera pushes forward toward subject" |
| Dolly Out | Pull | "camera pulls back from subject" |
| Pan | Rotation | "camera pans right across the scene" |
| Crane / Pedestal | Translation (vertical) | "camera rises vertically" |
| Truck / Tracking | Translation (horizontal) | "camera tracks left alongside subject" |
| Dolly In | Translation (push) | "camera pushes forward toward subject" |
| Dolly Out | Translation (pull) | "camera pulls back from subject" |
| Pan | Rotation (yaw) | "camera pans right across the scene" |
| Tilt | Rotation (pitch) | "camera tilts upward to follow the rocket" |
| Roll | Rotation (Z-axis / Dutch) | "camera rolls clockwise into a Dutch tilt" |
| Orbit | Circular | "camera orbits around subject" |
| Follow | Lock-on | "camera follows subject from behind" |
| Zoom | Lens-only (focal length) | "camera slowly zooms in on the figure" |
| Rack focus | Lens-only (focal plane, snap) | "rack focus from the foreground bottle to the figure in the background" |
| Pull focus | Lens-only (focal plane, gradual) | "camera shifts focus from foreground X to background Y" |
| Static | Fixed | "static camera, no movement" |
### Focal-plane labels at start AND end of dynamic-DoF shots
Hunyuan benefits when both endpoints of focus-changing shots are stated. State where focus starts AND where it lands — don't leave one implicit.
Example: "shallow DoF; focus on the foreground bottle at start; focus pulls to the figure in the background by end."
Without both endpoints, Hunyuan often defaults to deep focus or holds on the wrong plane.
### Style Keywords
**Photorealistic / Cinematic**:
@@ -65,6 +80,10 @@ When using image-to-video, the input image defines appearance. Your prompt shoul
**Bad I2V prompt**: "A beautiful woman in a red dress standing in a forest" — this repeats what the image already shows.
### Order motions temporally
Describe motion in temporal order; if multiple movements occur, separate them ("first the camera pans right, then tilts upward"). Hunyuan executes motion in the order it appears in the prompt — bundling two movements into one clause causes one of them to be dropped or blended.
## Example (T2V)
```
skills/creative/prompting/ltx-prompting.md
+6 -1
View File
@@ -11,9 +11,13 @@ LTX-2 uses a clean, focused prompt structure:
2. **Set the scene** — lighting, color palette, textures, atmosphere
3. **Describe the action** — natural sequence flowing from beginning to end
4. **Define the character(s)** — physical cues (age, hair, clothes), not abstract labels
5. **Camera movement(s)** — specify how and when; describe what appears AFTER the movement
5. **Camera movement(s)** — specify how and when; describe what appears AFTER the movement. (LTX honors the translation/rotation/lens distinction: `dolly` ≠ `zoom`, `pan` ≠ `truck`. Pick the right family — translation moves the rig, rotation pivots it, lens-only changes focal length or focal plane without moving the camera.)
6. **Describe the audio** — ambient sound, music, speech, or singing
### Strict-Static-Shot rule
If you write "static camera," the shot must have NO movement, NO focus change, NO zoom. LTX takes "static" literally — adding any motion verb later in the prompt will either be ignored or will produce a glitch where the camera contradicts itself. Pick one: static, OR a single named movement.
## LTX-Specific Tips
### Post-Movement Description
@@ -51,6 +55,7 @@ LTX organizes styles into three families:
| Overloaded scenes | Many characters/actions reduces coherence |
| Conflicting lighting descriptions | Pick one setup, commit to it |
| Starting complex | Build up: simple prompt first, add layers |
| Prompts over ~80 words | LTX-2 degrades past that. Pick the most important 5–6 elements. |
## LTX Technical Notes
skills/creative/prompting/seedance-prompting.md
+19
View File
@@ -44,6 +44,8 @@ Seedance is unusually literal about camera language, multi-shot cuts, and quoted
## Multi-shot pattern
> **Repeat identity verbatim across every shot.** "the same character" / pronouns / "Aang again" do not work. Repeat the 3–6 disambiguating visual attributes verbatim in every shot block. Seedance treats each shot as if you said it cold.
Seedance honors explicit shot lists:
```
@@ -63,6 +65,22 @@ Style: anamorphic lens, teal-orange cinematic grade, 35mm film grain.
Audio: rising orchestral swell with low taiko pulse, wind, distant wingbeats.
```
### Subject transition primitives in multi-shot
Seedance handles four distinct ways a subject can enter or exit a shot. Naming the primitive explicitly helps the model build the right transition between shots.
- **Subject revealing** (by camera move OR subject move) — the subject becomes visible mid-shot.
Example: `Shot 2 (slow truck right): empty corridor at first; the camera trucks right to reveal Aang — bald, blue arrow tattoo, orange robes — pressed flat against the wall.`
- **Subject disappearing** — the subject leaves frame, by motion or occlusion.
Example: `Shot 4 (static wide): Aang — bald, blue arrow tattoo, orange robes — sprints into the temple doorway and is swallowed by shadow; camera holds on the empty threshold.`
- **Subject switching** (rack focus / camera move) — focus or framing transfers from one subject to another.
Example: `Shot 5 (close-up, rack focus): rack focus from Aang's glowing arrow tattoo in foreground to Sokka — dark hair, blue tunic, boomerang on back — emerging from the mist behind.`
- **Complex alternating focus** — focus oscillates between two subjects within one shot.
Example: `Shot 7 (medium two-shot, alternating rack focus): focus on Aang — bald, blue arrow tattoo, orange robes — as he speaks, then pulls to Katara — long brown hair, blue water-tribe parka — as she answers, then back to Aang on the final beat.`
## Lip-sync pattern
```
@@ -84,6 +102,7 @@ Sokka, half a step behind, replies: "Then we fight."
| `generate_audio` | Keep `true` — sync audio is the moat. Strip in compose if unused. |
| `model_variant` | `standard` for hero + multi-shot + camera-heavy. `fast` for b-roll, previews, latency-capped jobs. |
| `seed` | Lock once a shot composition reads; iterate variants with the same seed. |
| `prompt length` | 200–400 words for hero shots; 80–150 for inserts. Seedance is one of the few models that rewards long, structured 5-aspect prompts. |
## Iteration strategy
skills/creative/prompting/sora-prompting.md
+5
View File
@@ -3,6 +3,8 @@
> Source: [OpenAI Sora 2 Cookbook](https://developers.openai.com/cookbook/examples/sora/sora2_prompting_guide)
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
**Word count:** Sora 2 plateaus around 100–250 words. Past 250, additional detail rarely improves output.
## Sora-Specific Prompt Template
Sora responds best to a structured format with prose + cinematography block + action beats:
@@ -40,6 +42,9 @@ Sora uniquely responds to these production-level details that most models ignore
| **Wardrobe** | "navy coat, sleeves rolled, suspenders loose" |
| **Finishing** | "fine-grain overlay, mild halation, gate weave, soft vignette" |
| **Shutter** | "180° shutter angle" |
| **Playback speed** | "speed ramp from 1x to 0.25x mid-shot", "stop-motion staccato", "time-reversed exhale" |
| **Lens distortion** | "fisheye barrel distortion at the edges", "subtle barrel curvature on straight lines" |
| **Focus mode** | "rack focus from foreground bottle to background figure", "deep focus, FG to BG sharp" |
## What Sora Does Differently
skills/creative/prompting/veo-prompting.md
+18 -2
View File
@@ -3,6 +3,8 @@
> Source: [Vertex AI Video Gen Prompt Guide](https://cloud.google.com/vertex-ai/generative-ai/docs/video/video-gen-prompt-guide)
> For universal vocabulary, see: `skills/creative/video-gen-prompting.md`
**Word count:** VEO 3.1 sweet spot is 100–250 words; longer prompts stop helping.
## VEO-Specific 14-Component Structure
VEO responds to the most comprehensive prompt structure of any model:
@@ -29,16 +31,30 @@ VEO responds to the most comprehensive prompt structure of any model:
- **Negative prompts**: Explicitly supported — "no text overlays, no watermarks, no lens flare"
- **Editing vocabulary**: Understands "match cut", "jump cut", "montage", "split diopter" as prompt terms.
### Camera vocabulary VEO honors literally
VEO 3.1 distinguishes the three camera-motion families and treats their tokens as separate primitives. Mixing them up (e.g. asking for a "zoom" when you mean a "dolly") will produce the wrong move.
- **Translation (rig physically moves):** `dolly` (in/out along the lens axis), `truck` (left/right laterally), `pedestal` (up/down vertically)
- **Rotation (rig stays put, camera rotates):** `pan` (yaw, left/right), `tilt` (pitch, up/down), `roll` (Dutch / Z-axis)
- **Lens-only (rig and body don't move):** `zoom` (focal length change), `rack focus` / `pull focus` / `focus tracking` (focal-plane change)
dolly ≠ zoom; pan ≠ truck. VEO follows whichever token leads.
## VEO Lens Effects (Unique)
VEO specifically responds to optical effects most models ignore:
| Effect | Prompt Language |
|--------|----------------|
| **Rack focus** | "rack focus from foreground flower to background figure" |
| **Rack focus** | "rack focus from foreground flower to background figure" (snap shift) |
| **Pull focus** | "slow pull focus from the candle in the foreground to the doorway behind" (gradual, slower than rack) |
| **Focus tracking** | "focus tracks the runner as she crosses frame; background stays soft" (focus follows a moving subject) |
| **Dolly zoom (vertigo)** | "vertigo effect as character realizes the truth" |
| **Fisheye** | "fisheye lens distortion, skatepark POV" |
| **Lens flare** | "anamorphic lens flare from setting sun" |
| **Anamorphic lens flare** | "anamorphic lens flare streaking horizontally from setting sun" |
These three focus modes (rack, pull, tracking) are different — VEO 3.1 honors the distinction per the paper.
## VEO Art Movement References