Animation pipeline: AnimeScene engine, Ghibli-style compositions, audio energy tool, and README showcase
Add anime_scene rendering engine (AnimeScene + ParticleOverlay components) with multi-image crossfade, 9 camera motion types, 5 particle systems, and cinematic lighting overlays. Fix critical Remotion durationInFrames footgun by passing sceneDurationSeconds from parent. Add audio offset/loop support in Explainer for skipping quiet music intros. New tools: audio_energy.py analyzes per-second loudness via ebur128 to find optimal music offset and detect when looping is needed. Update all 6 animation pipeline skills (proposal, scene, asset, compose, executive-producer, remotion.md) with battle-tested image_animation workflow including tool availability scan, FLUX multi-image generation, composition JSON format, pre-render validation, and post-render self-review. Add 3 demo compositions (Candyland, Mori no Seishin, Deep Ocean) and anime-ghibli style playbook. Update README with 3 anime video showcases and animation prompts. Add Animation Pipeline section to PROMPT_GALLERY.md.
This commit is contained in:
calesthio
+47
-9
@@ -46,9 +46,26 @@ The Explainer composition supports the following cut types:
|
||||
| `pie_chart` | `chartData` [{label, value}], optional `donut`, `centerLabel` | Proportions, breakdowns |
|
||||
| `kpi_grid` | `chartData` [{label, value, prefix, suffix, change, icon}] | Dashboards, traction metrics |
|
||||
| `progress_bar` | `progress` (0-100), optional `progressSegments` | Journey viz, completion, stacked metrics |
|
||||
| `anime_scene` | `images` (1-4 paths), optional `animation`, `particles`, `particleColor`, `particleCount`, `particleIntensity`, `vignette`, `lightingFrom`, `lightingTo` | Anime/Ghibli-style scenes with multi-image crossfade, camera motion, particle overlays |
|
||||
|
||||
**Chart animations:** `grow-up`, `slide-in`, `pop` (bar), `draw`, `fade-in` (line), `spin`, `expand`, `sequential` (pie), `count-up`, `pop`, `cascade` (kpi)
|
||||
|
||||
### Anime Scene — Multi-Image Crossfade + Particles
|
||||
|
||||
The `anime_scene` type renders 1-4 images with smooth crossfade transitions, cinematic camera motion, and animated particle overlays. This creates the illusion of animation from still images.
|
||||
|
||||
**Camera motion types:** `zoom-in`, `zoom-out`, `pan-left`, `pan-right`, `ken-burns`, `drift-up`, `drift-down`, `parallax`, `static`
|
||||
|
||||
**Particle types:** `fireflies` (floating golden orbs), `petals` (falling cherry blossoms), `sparkles` (twinkling stars), `mist` (drifting fog layers), `light-rays` (crepuscular rays)
|
||||
|
||||
**Key prop:** `sceneDurationSeconds` is automatically passed by `SceneRenderer` — this fixes a critical Remotion pitfall where `useVideoConfig().durationInFrames` returns the full composition duration, not the scene's Sequence duration.
|
||||
|
||||
**Multi-image crossfade math:** Each image owns an equal time segment. Fade-out of image N and fade-in of image N+1 OVERLAP by `crossfadeDur` (~1.2s) so there's never a dead frame. Generate 2-3 images per scene with same style prefix + different seeds for subtle motion effect.
|
||||
|
||||
**Reference composition:** `remotion-composer/public/demo-props/mori-no-seishin.json` — 6 anime scenes, 30 seconds, with particles, lighting, overlays, and ambient music.
|
||||
|
||||
**Style playbook:** `styles/anime-ghibli.yaml` — Ghibli-inspired aesthetic with color palette, typography, motion parameters, and FLUX prompt prefix.
|
||||
|
||||
**Zero-key video strategy:** When no image or video generation is available, build
|
||||
entire videos from these component types. A well-composed sequence of hero_title →
|
||||
kpi_grid → bar_chart → comparison → stat_card → text_card produces a polished,
|
||||
@@ -153,19 +170,21 @@ remotion-composer/
|
||||
The orchestrator calls Remotion renders via CLI:
|
||||
|
||||
```bash
|
||||
# Standard render
|
||||
npx remotion render src/index.ts ExplainerVideo \
|
||||
--props='{"scenes": [...], "theme": "clean_professional"}' \
|
||||
--output=pipeline/<project>/output/final_output.mp4 \
|
||||
--codec=h264
|
||||
# Standard render (composition name is "Explainer", no entry point needed)
|
||||
npx remotion render Explainer \
|
||||
--props="public/demo-props/my-video.json" \
|
||||
--output=output/final.mp4 \
|
||||
--codec=h264 --crf=18
|
||||
|
||||
# With specific media profile
|
||||
npx remotion render src/index.ts ExplainerVideo \
|
||||
npx remotion render Explainer \
|
||||
--width=1080 --height=1920 --fps=30 \
|
||||
--props=props.json \
|
||||
--props="public/demo-props/my-video.json" \
|
||||
--output=output.mp4
|
||||
```
|
||||
|
||||
**Note:** Do NOT specify `src/index.ts` as entry point — Remotion auto-discovers compositions. The composition name is `Explainer` (not `ExplainerVideo`).
|
||||
|
||||
In Python, invoke via `subprocess` from `video_compose.py` when `backend="remotion"`.
|
||||
|
||||
### Media Profile Mapping
|
||||
@@ -249,12 +268,30 @@ const cleanProfessional = {
|
||||
|
||||
### Audio Layering
|
||||
|
||||
Narration + background music + SFX as parallel `<Audio>` components:
|
||||
Narration + background music + SFX as parallel `<Audio>` components.
|
||||
|
||||
**Music offset and looping:** The `audio.music` config supports:
|
||||
- `offsetSeconds` — skip quiet intros, start from the energetic part of the track. Use `tools/analysis/audio_energy.py` to find the optimal offset automatically.
|
||||
- `loop` — loop the music if it's shorter than the video. Remotion handles this natively.
|
||||
- `fadeInSeconds` / `fadeOutSeconds` — smooth volume ramps at start/end.
|
||||
|
||||
```json
|
||||
"audio": {
|
||||
"music": {
|
||||
"src": "project/music.mp3",
|
||||
"volume": 0.15,
|
||||
"offsetSeconds": 55,
|
||||
"loop": false,
|
||||
"fadeInSeconds": 2,
|
||||
"fadeOutSeconds": 3
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```tsx
|
||||
<AbsoluteFill>
|
||||
<Audio src={narrationUrl} />
|
||||
<Audio src={musicUrl} volume={0.06} />
|
||||
<Audio src={musicUrl} volume={0.06} startFrom={offsetFrames} loop />
|
||||
{sfxCues.map(cue => (
|
||||
<Sequence key={cue.id} from={secondsToFrames(cue.time)}>
|
||||
<Audio src={cue.url} volume={cue.volume} />
|
||||
@@ -276,6 +313,7 @@ Remotion renders are CPU-intensive but $0 API cost. Track via cost_tracker:
|
||||
- **No CSS animations or transitions** — they don't render correctly. Use `useCurrentFrame()` + `interpolate()` for all motion.
|
||||
- **No Tailwind animation classes** — `animate-*` classes break frame-based rendering. Static Tailwind utilities are fine.
|
||||
- **Always clamp interpolate()** — use `extrapolateLeft: 'clamp', extrapolateRight: 'clamp'` to prevent values shooting past endpoints.
|
||||
- **`useVideoConfig().durationInFrames` returns COMPOSITION duration, not Sequence duration** — This is the #1 Remotion footgun. If your composition is 31s (930 frames) and a scene's `<Sequence>` is 5s (150 frames), `durationInFrames` still returns 930 inside that scene. Any crossfade, camera motion, or timing logic that uses `durationInFrames` directly will be wildly wrong. **Fix:** Pass `sceneDurationSeconds` as a prop from the parent and compute `effectiveDuration = Math.round(sceneDurationSeconds * fps)` inside the component. The `AnimeScene` component implements this pattern.
|
||||
- **Node.js 18+ required** — listed as optional in minimum system, required in recommended.
|
||||
- **Render in series, not parallel** — unless the machine has enough RAM. Each render spawns a Chromium instance.
|
||||
|
||||
|
||||
@@ -33,6 +33,38 @@ Before batch-generating assets, produce one sample of each expensive type and sh
|
||||
|
||||
If rejected, adjust parameters and retry (max 3 iterations). Do not batch until approved.
|
||||
|
||||
### 1c. Multi-Image Generation for Image-Based Animation (Approach A)
|
||||
|
||||
When `animation_mode == "image_animation"`, each scene needs **2-3 images** for crossfade animation. This is what makes stills look like movement.
|
||||
|
||||
**Image generation workflow:**
|
||||
|
||||
1. **Define a STYLE_PREFIX** — a consistent prompt prefix used across ALL images in the project. This ensures visual coherence. Store it as a reusable asset.
|
||||
```
|
||||
Example: "Studio Ghibli anime style, hand-painted watercolor aesthetic,
|
||||
soft diffused lighting, lush natural environment, warm color palette,
|
||||
painterly brushstrokes visible, high detail..."
|
||||
```
|
||||
|
||||
2. **Use seed management** — for each scene, use nearby seed values (e.g., seed 100 and 101) for the A/B variants. Same prompt + different seed = same composition with subtle differences = natural crossfade motion.
|
||||
|
||||
3. **Generate one test image first** — render a single scene to verify the style prefix produces good results at 1920×1080 before batch generating all images.
|
||||
|
||||
4. **Batch generation** — generate all scene images. Skip any that already exist on disk (idempotent).
|
||||
|
||||
5. **Composition JSON** — each scene gets `type: "anime_scene"` with `images: ["path/a.png", "path/b.png"]` plus camera motion, particle type, and lighting config.
|
||||
|
||||
**Cost estimation:** 2-3 images per scene × $0.03-0.13/image depending on provider.
|
||||
|
||||
**Reference:** See `projects/mori-no-seishin/generate_images.py` for the proven batch generation pattern.
|
||||
|
||||
6. **Copy to Remotion public directory** — After generating all images, copy them to `remotion-composer/public/<project-name>/` so Remotion can access them via `staticFile()`. Image paths in the composition JSON are relative to this directory:
|
||||
```
|
||||
remotion-composer/public/<project-name>/scene1-a.png ← Remotion reads from here
|
||||
remotion-composer/public/<project-name>/ambient-music.mp3 ← Music too
|
||||
```
|
||||
**If you skip this step, the render will fail with missing file errors.** This is the #1 cause of render failures for new projects.
|
||||
|
||||
### 2. Build Reusable Systems
|
||||
|
||||
Create once:
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## When To Use
|
||||
|
||||
Render the animation with an emphasis on text sharpness, timing integrity, and consistent output cadence.
|
||||
Render the animation with an emphasis on text sharpness, timing integrity, and consistent output cadence. For `image_animation` approach, this stage also includes building the composition JSON, sourcing music, running pre-render validation, and performing post-render self-review.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -11,27 +11,196 @@ Render the animation with an emphasis on text sharpness, timing integrity, and c
|
||||
| Schema | `schemas/artifacts/render_report.schema.json` | Artifact validation |
|
||||
| Prior artifacts | `state.artifacts["edit"]["edit_decisions"]`, `state.artifacts["assets"]["asset_manifest"]` | Timing plan and asset files |
|
||||
| Tools | `video_compose`, `audio_mixer`, `video_stitch` | Final assembly |
|
||||
| Tools | `composition_validator` | Pre-render validation (MANDATORY) |
|
||||
| Tools | `audio_probe` | Music duration check |
|
||||
| Playbook | Active style playbook | Render consistency |
|
||||
| Reference | `remotion-composer/public/demo-props/mori-no-seishin.json` | Composition JSON format reference |
|
||||
| Reference | `skills/core/remotion.md` | Remotion patterns, anime_scene type, critical constraints |
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Preserve Motion Timing
|
||||
### 1. Ensure Assets Are in Remotion's Public Directory
|
||||
|
||||
**CRITICAL:** Remotion can only access files via `staticFile()`, which resolves from `remotion-composer/public/`. Generated images and music files MUST be copied or symlinked into this directory before rendering.
|
||||
|
||||
```
|
||||
Project structure:
|
||||
projects/<name>/assets/images/*.png ← where images were generated
|
||||
remotion-composer/public/<name>/*.png ← where Remotion reads them
|
||||
|
||||
Required: Copy or symlink images AND music into public/<project-name>/
|
||||
```
|
||||
|
||||
Image paths in the composition JSON are relative to `remotion-composer/public/`:
|
||||
```json
|
||||
"images": ["deep-ocean/scene1-a.png", "deep-ocean/scene1-b.png"]
|
||||
"src": "deep-ocean/ambient-music.mp3"
|
||||
```
|
||||
|
||||
**If you skip this step, the render will fail with missing file errors or produce black frames.**
|
||||
|
||||
### 2. Build the Composition JSON (image_animation approach)
|
||||
|
||||
For `anime_scene` compositions, build a JSON file at `remotion-composer/public/demo-props/<name>.json`.
|
||||
|
||||
**Required structure:**
|
||||
|
||||
```json
|
||||
{
|
||||
"cuts": [
|
||||
{
|
||||
"id": "scene-1-name",
|
||||
"source": "",
|
||||
"in_seconds": 0,
|
||||
"out_seconds": 5,
|
||||
"type": "anime_scene",
|
||||
"images": ["<project>/<image-a>.png", "<project>/<image-b>.png"],
|
||||
"animation": "<camera-motion>",
|
||||
"particles": "<particle-type>",
|
||||
"particleColor": "#HEXCOLOR",
|
||||
"particleCount": 20,
|
||||
"particleIntensity": 0.5,
|
||||
"backgroundColor": "#0A0A1A",
|
||||
"vignette": true,
|
||||
"lightingFrom": "rgba(r,g,b,a)",
|
||||
"lightingTo": "transparent"
|
||||
}
|
||||
],
|
||||
"overlays": [...],
|
||||
"audio": { "music": { "src": "<project>/music.mp3", "volume": 0.15, "fadeInSeconds": 2, "fadeOutSeconds": 3 } }
|
||||
}
|
||||
```
|
||||
|
||||
**Prop name reference (JSON field → AnimeScene prop):**
|
||||
|
||||
| JSON Field | Type | Values | Required |
|
||||
|------------|------|--------|----------|
|
||||
| `type` | string | `"anime_scene"` | YES |
|
||||
| `images` | string[] | 1-4 image paths relative to `public/` | YES |
|
||||
| `animation` | string | `zoom-in`, `zoom-out`, `pan-left`, `pan-right`, `ken-burns`, `drift-up`, `drift-down`, `parallax`, `static` | No (default: `ken-burns`) |
|
||||
| `particles` | string | `fireflies`, `petals`, `sparkles`, `mist`, `light-rays` | No |
|
||||
| `particleColor` | string | Hex color | No (default: `#FFE082`) |
|
||||
| `particleCount` | number | 1-50 | No (default: 20) |
|
||||
| `particleIntensity` | number | 0-1 | No (default: 0.6) |
|
||||
| `backgroundColor` | string | Hex color for scene background | No (default: `#0A0A1A`) |
|
||||
| `vignette` | boolean | Cinematic vignette overlay | No (default: true) |
|
||||
| `lightingFrom` | string | Starting gradient color (`rgba(...)` or `transparent`) | No |
|
||||
| `lightingTo` | string | Ending gradient color | No |
|
||||
|
||||
**References:** See `mori-no-seishin.json` (Ghibli forest) and `deep-ocean.json` (underwater bioluminescence) for complete working examples.
|
||||
|
||||
### 3. Source Music and Find Optimal Offset
|
||||
|
||||
Use `tools/audio/pixabay_music.py` to find royalty-free ambient music matching the mood.
|
||||
|
||||
**After downloading, run audio energy analysis (MANDATORY):**
|
||||
|
||||
```python
|
||||
from tools.analysis.audio_energy import AudioEnergy
|
||||
result = AudioEnergy().execute({
|
||||
"input_path": "path/to/music.mp3",
|
||||
"video_duration_seconds": 30, # your video duration
|
||||
})
|
||||
data = result.data
|
||||
print(f"Recommended offset: {data['recommended_offset_seconds']}s")
|
||||
print(f"Reason: {data['offset_reason']}")
|
||||
print(f"Needs loop: {data['needs_loop']}")
|
||||
```
|
||||
|
||||
This tool:
|
||||
1. **Finds the best section** — analyzes per-second loudness and finds the N-second window with highest average energy. Ambient music tracks often have quiet intros (10-30s) before the main melody kicks in.
|
||||
2. **Recommends loop** — if the music from the offset is shorter than the video, it tells you to enable looping.
|
||||
|
||||
**Apply the offset in the composition JSON:**
|
||||
|
||||
```json
|
||||
"audio": {
|
||||
"music": {
|
||||
"src": "project/music.mp3",
|
||||
"volume": 0.15,
|
||||
"fadeInSeconds": 2,
|
||||
"fadeOutSeconds": 3,
|
||||
"offsetSeconds": 55,
|
||||
"loop": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `offsetSeconds` — start playback from this point in the track (skips quiet intro)
|
||||
- `loop` — set to `true` if the remaining music is shorter than the video
|
||||
|
||||
**If the tool says `needs_loop: true`:** set `"loop": true` in the composition JSON. Remotion will loop the audio seamlessly with the volume fade resetting per loop.
|
||||
|
||||
### 4. Pre-Render Validation (MANDATORY — NO EXCEPTIONS)
|
||||
|
||||
Run `composition_validator` before every render:
|
||||
|
||||
```python
|
||||
from tools.analysis.composition_validator import CompositionValidator
|
||||
result = CompositionValidator().execute({
|
||||
"composition_path": "remotion-composer/public/demo-props/<name>.json",
|
||||
"assets_root": "remotion-composer/public",
|
||||
})
|
||||
# result.data["valid"] MUST be True before proceeding
|
||||
```
|
||||
|
||||
This catches:
|
||||
- Missing image/audio files that would cause black frames or render errors
|
||||
- Invalid cut timings (out ≤ in)
|
||||
- Audio longer than video duration
|
||||
|
||||
**If validation fails, fix the issue BEFORE rendering. Do not render an invalid composition.**
|
||||
|
||||
### 5. Preserve Motion Timing
|
||||
|
||||
Do not let export settings or careless composition change the perceived timing of holds, stagger, or scene transitions.
|
||||
|
||||
### 2. Protect Text And Diagram Sharpness
|
||||
### 6. Protect Text And Diagram Sharpness
|
||||
|
||||
Animation often fails on export through soft text, muddy thin lines, or cramped mobile framing.
|
||||
|
||||
### 3. Verify The First And Last Frames
|
||||
### 7. Render
|
||||
|
||||
Ensure:
|
||||
```bash
|
||||
cd remotion-composer
|
||||
npx remotion render Explainer \
|
||||
--props="public/demo-props/<name>.json" \
|
||||
--output="<output-path>/final.mp4" \
|
||||
--codec=h264 --crf=18
|
||||
```
|
||||
|
||||
- the opening frame reads immediately,
|
||||
- the final frame lands cleanly,
|
||||
- nothing important is clipped by safe zones.
|
||||
**Note:** The composition name is `Explainer` (not `ExplainerVideo`). Do NOT specify `src/index.ts` as entry point — Remotion auto-discovers it.
|
||||
|
||||
### 4. Use Render Metadata
|
||||
### 8. Post-Render Self-Review (MANDATORY)
|
||||
|
||||
After rendering, extract mid-scene frames and visually inspect:
|
||||
|
||||
```bash
|
||||
# Extract one frame from the middle of each scene
|
||||
ffmpeg -y -i final.mp4 \
|
||||
-vf "select='eq(n\,75)+eq(n\,225)+eq(n\,375)+eq(n\,525)+eq(n\,675)+eq(n\,825)'" \
|
||||
-vsync vfr frames/scene_%02d.png
|
||||
```
|
||||
|
||||
**Check each frame for:**
|
||||
- [ ] Images are visible (not black/dark frames)
|
||||
- [ ] Particles are rendering (sparkles, fireflies, etc. visible)
|
||||
- [ ] Camera motion is evident (framing differs from static)
|
||||
- [ ] Overlays display at correct moments with clean text
|
||||
- [ ] Color palette is consistent across scenes
|
||||
- [ ] Vignette creates cinematic depth
|
||||
|
||||
**Also verify the output file:**
|
||||
```bash
|
||||
ffprobe -v quiet -print_format json -show_format -show_streams final.mp4
|
||||
```
|
||||
- Duration within ±5% of target?
|
||||
- Resolution matches 1920×1080?
|
||||
- Audio stream present?
|
||||
|
||||
**If issues are found:** identify the cause (missing images, wrong timing, rendering glitch) and fix before presenting to user.
|
||||
|
||||
### 9. Use Render Metadata
|
||||
|
||||
Recommended metadata keys:
|
||||
|
||||
@@ -42,6 +211,10 @@ Recommended metadata keys:
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Forgetting to copy assets to `remotion-composer/public/`** — the #1 cause of render failures. Images generate to `projects/<name>/assets/` but Remotion reads from `public/`.
|
||||
- Soft or aliased text after rendering.
|
||||
- Compression choices that damage diagrams.
|
||||
- Scene cadence changing between preview and final.
|
||||
- **Skipping `composition_validator`** — catches missing files, bad timings, audio mismatches before you waste render time.
|
||||
- **Not extracting frames for self-review** — a rendered video is not "done" until frames are visually inspected. Black frames, missing particles, or invisible images are not always obvious from file size alone.
|
||||
- **Using `durationInFrames` from `useVideoConfig()` for scene-level timing** — this returns the FULL composition duration, not the scene's Sequence duration. See `skills/core/remotion.md` Critical Constraints.
|
||||
|
||||
@@ -43,7 +43,14 @@ EP_STATE:
|
||||
budget_remaining_usd: <budget_total>
|
||||
|
||||
# Animation-specific state
|
||||
animation_mode: <manim | remotion | ai_video | diagram_stills | mixed>
|
||||
# Approaches:
|
||||
# image_animation — Multi-image crossfade via Remotion (anime/Ghibli/illustration style)
|
||||
# clip_video — AI-generated video clips composited as a story
|
||||
# manim — Programmatic math/physics animation via ManimCE
|
||||
# remotion_dataviz — Data visualization with Remotion components (zero-key capable)
|
||||
# diagram_stills — Diagram + image stills with Ken Burns
|
||||
# mixed — Combination of multiple approaches per-scene
|
||||
animation_mode: <image_animation | clip_video | manim | remotion_dataviz | diagram_stills | mixed>
|
||||
reuse_strategy:
|
||||
recurring_motifs: []
|
||||
layout_system: null
|
||||
@@ -218,12 +225,15 @@ CHECK: Approval gate (CRITICAL)
|
||||
- If "approved_with_changes": apply modifications before proceeding
|
||||
- Extract: animation_mode, reuse_strategy, target_duration, playbook, budget, tool selections
|
||||
|
||||
CHECK: Animation mode feasibility
|
||||
- Does the selected animation mode's required tools exist in the registry?
|
||||
- If Manim mode selected: is math_animate available?
|
||||
- If Remotion mode selected: is video_compose (Remotion) available?
|
||||
- If AI video mode selected: are video generation providers available?
|
||||
- If any required tool is unavailable: alert user, offer alternatives
|
||||
CHECK: Animation approach feasibility
|
||||
- Does the selected animation approach's required tools exist in the registry?
|
||||
- If image_animation selected: is image_selector available? Which providers? Is Remotion available?
|
||||
- If clip_video selected: is video_selector available? Which providers?
|
||||
- If manim selected: is math_animate (ManimCE) available?
|
||||
- If remotion_dataviz selected: is video_compose (Remotion) available?
|
||||
- If diagram_stills selected: is diagram_gen + image_selector available?
|
||||
- If any required tool is unavailable: alert user, offer alternatives with specific setup instructions
|
||||
- NEVER silently downgrade — if an approach needs a key the user doesn't have, STOP and tell them
|
||||
|
||||
CHECK: Reuse strategy validity
|
||||
- Does the reuse strategy define recurring motifs?
|
||||
|
||||
@@ -22,9 +22,13 @@ Animation proposals have a unique dimension: **animation mode selection**. Unlik
|
||||
|
||||
## Process
|
||||
|
||||
### Step 1: Absorb the Research
|
||||
### Step 1: Absorb the Research (or Direct Brief)
|
||||
|
||||
Read the `research_brief` thoroughly. Extract:
|
||||
**If a `research_brief` artifact exists:** Read it thoroughly. Extract:
|
||||
|
||||
**If no research_brief exists (direct user brief):** The user has given you a creative brief directly. This is common for short videos (30-60s) where formal research is overkill. Use the user's brief as your input and proceed to Step 2. Note the missing research as a limitation — you won't have data_points, technique references, or audience_insights to draw from, so concept design relies on your knowledge and the user's direction.
|
||||
|
||||
**When a research_brief IS available,** extract:
|
||||
|
||||
- **`research_summary`** — read first. Contains both the key insight and the most promising animation approach.
|
||||
- **`angles_discovered`** — raw concept candidates, each with an `animation_fit` field.
|
||||
@@ -60,25 +64,105 @@ python -c "from tools.tool_registry import registry; import json; registry.disco
|
||||
|
||||
Record all findings. **Do not propose an animation mode that requires tools you don't have.**
|
||||
|
||||
### Step 3: Animation Mode Decision Matrix
|
||||
### Step 3: Animation Approach Selection
|
||||
|
||||
This is the key differentiator from the explainer proposal. For each viable animation mode, evaluate:
|
||||
This is the key differentiator from the explainer proposal. **Present the user with concrete animation approaches, explain what each looks like, what tools/keys they need, and what's already available.**
|
||||
|
||||
| Mode | Best For | Tool Required | Visual Quality | Cost | Iteration Speed |
|
||||
|------|----------|---------------|----------------|------|-----------------|
|
||||
| **Manim (ManimCE)** | Math, physics, geometry, algorithms | `math_animate` | Precise, programmatic | Free (local) | Fast (code-driven) |
|
||||
| **Remotion** | Data viz, charts, React components, kinetic type | `video_compose` (Remotion mode) | Smooth, web-native | Free (local) | Fast (code-driven) |
|
||||
| **AI Video Generation** | Abstract concepts, metaphors, transitions | `video_selector` providers | Variable, cinematic | $0.05-0.50/clip | Slow (generation time) |
|
||||
| **Diagram + Image Stills** | Process flows, architecture, comparisons | `diagram_gen` + `image_selector` | Clean, reliable | $0-0.05/image | Fast |
|
||||
| **Mixed Mode** | Complex topics needing multiple techniques | Multiple tools | Varied | Varies | Moderate |
|
||||
#### Step 3a: Tool Availability Scan
|
||||
|
||||
**Mode selection rules:**
|
||||
- If the topic involves math/formulas/geometry → prefer Manim
|
||||
- If the topic involves data/statistics/charts → prefer Remotion or diagram_gen
|
||||
- If the topic is abstract/conceptual → consider AI video for key moments
|
||||
- If the topic is process/workflow → prefer diagram builds
|
||||
- Always check tool availability before committing to a mode
|
||||
- Mixed mode is valid when different sections need different approaches
|
||||
Before designing concepts, scan what's available and present it honestly:
|
||||
|
||||
```
|
||||
TOOL AVAILABILITY SCAN
|
||||
──────────────────────
|
||||
Image generation:
|
||||
✅ FLUX (fal.ai) — FAL_KEY detected — $0.03-0.05/image
|
||||
❌ gpt-image-1 — OPENAI_API_KEY missing — $0.13/image
|
||||
❌ Stable Diffusion — Not installed locally — Free
|
||||
❌ FLUX (local) — Not installed locally — Free
|
||||
|
||||
Video generation:
|
||||
❌ Runway Gen-3 — No API key — $0.50/clip
|
||||
❌ Kling — No API key — $0.10-0.30/clip
|
||||
❌ CogVideoX (local) — Not installed — Free
|
||||
|
||||
Composition:
|
||||
✅ Remotion — Installed — Free (local CPU)
|
||||
✅ FFmpeg — Installed — Free
|
||||
|
||||
Audio:
|
||||
✅ Pixabay Music — No key needed — Free
|
||||
❌ OpenAI TTS — OPENAI_API_KEY missing — $0.015/min
|
||||
✅ Local TTS (piper) — Not checked — Free
|
||||
|
||||
Math/Diagram:
|
||||
❌ ManimCE — Not installed — Free
|
||||
✅ diagram_gen — Available — Free
|
||||
```
|
||||
|
||||
**Present this scan to the user.** Say: "Here's what I can see right now. Based on this, here are your animation approach options."
|
||||
|
||||
#### Step 3b: Animation Approach Decision Matrix
|
||||
|
||||
Present the approaches as clear options:
|
||||
|
||||
| Approach | What It Looks Like | Tools Required | Cost Range | Proven? |
|
||||
|----------|-------------------|----------------|------------|---------|
|
||||
| **A: Image-Based Animation (Remotion)** | AI-generated keyframes with crossfade, camera motion, particles. Looks like moving anime/illustration. | `image_selector` (any provider) + Remotion | $0.03-0.13/image × 2-3/scene | ✅ Proven (mori-no-seishin) |
|
||||
| **B: Clip-Based Video** | AI-generated video clips assembled as a story. Most cinematic but least consistent. | `video_selector` (Runway/Kling/etc.) | $0.10-0.50/clip × scenes | ❌ Not yet proven |
|
||||
| **C: Programmatic Animation (Manim)** | Code-driven math/geometry animation. Precise, clean, 3Blue1Brown style. | `math_animate` (ManimCE) | Free (local) | ❌ Not yet proven |
|
||||
| **D: Data Visualization (Remotion)** | Animated charts, KPIs, kinetic typography. Data-driven storytelling. | Remotion (built-in components) | Free (local) | ✅ Proven (zero-key formula) |
|
||||
| **E: Diagram + Image Stills** | Process flows and architecture diagrams with Ken Burns. | `diagram_gen` + `image_selector` | $0-0.05/image | ✅ Proven |
|
||||
| **F: Mixed Mode** | Combine any of the above per-scene. Most flexible. | Multiple tools | Varies | Partial |
|
||||
|
||||
**For each viable approach, present to the user:**
|
||||
|
||||
```
|
||||
APPROACH A: Image-Based Animation (Remotion)
|
||||
─────────────────────────────────────────────
|
||||
What it looks like: Multiple AI-generated images per scene, crossfaded with
|
||||
camera motion (zoom, pan, ken-burns) and particle overlays (fireflies, mist,
|
||||
sparkles). Creates the illusion of movement from still frames.
|
||||
|
||||
You need: An image generation API key.
|
||||
→ You already have: FAL_KEY (FLUX at $0.05/image)
|
||||
→ Alternative: Install Stable Diffusion locally (free, slower)
|
||||
→ Alternative: Add OPENAI_API_KEY for gpt-image-1 ($0.13/image)
|
||||
|
||||
Estimated cost for 30s video: ~$0.65 (13 images)
|
||||
Estimated cost for 5min video: ~$6.00 (120 images)
|
||||
|
||||
Style options: anime-ghibli, painterly, photorealistic, watercolor
|
||||
Reference: remotion-composer/public/demo-props/mori-no-seishin.json
|
||||
|
||||
APPROACH B: Clip-Based Video
|
||||
─────────────────────────────
|
||||
What it looks like: AI-generated 3-5 second video clips assembled as a story.
|
||||
Most cinematic output but hardest to maintain visual consistency across clips.
|
||||
|
||||
You need: A video generation API key.
|
||||
→ Currently available: None detected
|
||||
→ To enable: Add RUNWAY_API_KEY, KLING_API_KEY, or install CogVideoX locally
|
||||
|
||||
Estimated cost for 30s video: $3-15 depending on provider
|
||||
Estimated cost for 5min video: $30-150
|
||||
|
||||
Note: This approach is not yet proven in the OpenMontage pipeline.
|
||||
Consistency across clips is the #1 challenge.
|
||||
```
|
||||
|
||||
**Critical principle: Surface capabilities, don't hide limitations.** The user should know exactly what's possible right now vs. what needs setup.
|
||||
|
||||
#### Step 3c: Mode Selection Rules
|
||||
|
||||
- If the topic is visual/artistic (anime, illustration, fantasy) → **Approach A** (image-based)
|
||||
- If the topic involves data/statistics/business → **Approach D** (data viz) or **Approach A** with data overlays
|
||||
- If the topic involves math/physics → **Approach C** (Manim) if available, else **Approach E**
|
||||
- If the topic is abstract/conceptual and budget allows → **Approach B** (clip-based) for key moments
|
||||
- If no paid APIs available → **Approach D** (zero-key Remotion) or **Approach E** (diagrams)
|
||||
- If the user wants maximum quality and has video gen keys → **Approach F** (mixed: video clips for hero shots + Remotion for data)
|
||||
- **Always offer at least one free/local option** alongside paid approaches
|
||||
- **Never silently downgrade** — if the best approach needs a key the user doesn't have, say so explicitly
|
||||
|
||||
### Step 4: Design Concept Options
|
||||
|
||||
@@ -103,13 +187,15 @@ For each concept, specify:
|
||||
- Hook must promise a VISUAL experience, not just information
|
||||
- Hook must be grounded in a specific research finding
|
||||
|
||||
#### 4b: Animation Mode and Approach
|
||||
#### 4b: Animation Approach and Approach
|
||||
|
||||
For each concept, specify:
|
||||
- **Primary animation mode**: manim / remotion / ai_video / diagram_stills / mixed
|
||||
- **Why this mode**: grounded in technique research from the brief
|
||||
- **Animation approach**: `image_animation` / `clip_video` / `manim` / `remotion_dataviz` / `diagram_stills` / `mixed`
|
||||
- **Why this approach**: grounded in technique research AND tool availability from Step 3
|
||||
- **Image/video generation provider**: which specific provider from the preflight scan (e.g., "FLUX via fal.ai", "gpt-image-1 via OpenAI", "Stable Diffusion local")
|
||||
- **Reuse strategy**: What's the visual system? (recurring motifs, layout grid, color scheme, transition family)
|
||||
- **Complexity estimate**: How many unique scene types vs. reusable templates?
|
||||
- **Style playbook**: which playbook from `styles/*.yaml` (e.g., `anime-ghibli`, `clean-professional`)
|
||||
|
||||
#### 4c: Narrative Structure
|
||||
|
||||
@@ -130,11 +216,12 @@ Choose from: `myth_busting`, `problem_solution`, `data_narrative`, `comparison`,
|
||||
|
||||
#### 4e: Concept Diversity Check
|
||||
|
||||
- [ ] No two concepts use the same animation mode
|
||||
- [ ] No two concepts use the same animation approach
|
||||
- [ ] No two concepts use the same narrative structure
|
||||
- [ ] At least one concept is achievable with free/local tools only
|
||||
- [ ] At least one concept is achievable with free/local tools only (zero-key or local image gen)
|
||||
- [ ] At least one concept leverages the most surprising data point
|
||||
- [ ] Each concept's animation mode is grounded in technique research
|
||||
- [ ] Each concept's approach is grounded in tool availability AND technique research
|
||||
- [ ] Each concept states which API keys/tools it requires (and flags any the user doesn't have)
|
||||
|
||||
### Step 5: Present Concepts and Get Selection
|
||||
|
||||
@@ -255,9 +342,12 @@ Validate the `proposal_packet` artifact against `schemas/artifacts/proposal_pack
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Ignoring animation mode feasibility**: If Manim isn't installed, don't propose a Manim-based concept. Design around constraints.
|
||||
- **Three versions of the same concept with different titles**: Structural diversity means different animation modes, different narrative structures, different hooks.
|
||||
- **Not leveraging free tools**: Animation has a huge cost advantage — Manim, Remotion, and diagram_gen are free. If proposing expensive AI video, justify why free alternatives won't work.
|
||||
- **Not showing the Tool Availability Scan**: The user must know what's available BEFORE seeing concepts. Don't hide missing keys or tools.
|
||||
- **Ignoring animation approach feasibility**: If FLUX isn't available, don't propose image_animation without saying "you need to add FAL_KEY first." Design around constraints OR explicitly state what's needed.
|
||||
- **Three versions of the same concept with different titles**: Structural diversity means different animation approaches, different narrative structures, different hooks.
|
||||
- **Not leveraging free tools**: Animation has a huge cost advantage — Manim, Remotion data-viz, and diagram_gen are free. If proposing expensive AI video, justify why free alternatives won't work.
|
||||
- **Over-promising visual complexity**: 20 unique hand-crafted scenes is not realistic. Design reuse strategies that look varied but share underlying templates.
|
||||
- **Skipping the approval gate**: This is the whole point of pre-production. No shortcuts.
|
||||
- **Ignoring mathematical accuracy**: If the research brief flagged technical accuracy constraints, the concept MUST respect them. A beautiful but wrong animation is a failure.
|
||||
- **Not distinguishing image_animation from clip_video**: These are fundamentally different. Image-based animation (Approach A) generates still images and uses Remotion for motion/crossfade. Clip-based video (Approach B) generates actual video clips with an AI video model. The user should understand this distinction clearly.
|
||||
- **Silent downgrades**: If the user picked image_animation but image generation fails, STOP and tell them. Never silently fall back to text cards or diagram stills.
|
||||
|
||||
@@ -41,6 +41,40 @@ Use:
|
||||
- `text_card` for clean high-impact copy moments,
|
||||
- `generated` only where needed.
|
||||
|
||||
**For `image_animation` approach (anime/illustration style):**
|
||||
|
||||
Use `anime_scene` type for each scene. Plan:
|
||||
|
||||
- **Images per scene**: 2-3 images with consistent style prefix and nearby seeds for crossfade effect
|
||||
- **Camera motion**: choose from `zoom-in`, `zoom-out`, `pan-left`, `pan-right`, `ken-burns`, `drift-up`, `drift-down`, `parallax`, `static` — vary per scene to prevent monotony
|
||||
- **Particle type**: choose from `fireflies`, `petals`, `sparkles`, `mist`, `light-rays` — match to scene mood
|
||||
- **Lighting**: optional `lightingFrom`/`lightingTo` gradient for atmospheric shifts within the scene
|
||||
- **Vignette**: `true` for cinematic framing (default), `false` for bright/open scenes
|
||||
- **Scene duration**: 4-7 seconds per scene. Longer scenes need more images for crossfade variety.
|
||||
|
||||
**Scene variety rules for image_animation:**
|
||||
- Don't use the same camera motion for consecutive scenes
|
||||
- Alternate between warm and cool particle types
|
||||
- Mix close-up and wide establishing shots
|
||||
- Use overlays (`hero_title`, `section_title`) to add narrative structure
|
||||
|
||||
**JSON prop name mapping** (use these exact field names in the composition JSON):
|
||||
|
||||
| Concept | JSON Field | Example Values |
|
||||
|---------|-----------|----------------|
|
||||
| Camera motion | `animation` | `"zoom-in"`, `"pan-right"`, `"ken-burns"` |
|
||||
| Particle effect | `particles` | `"fireflies"`, `"sparkles"`, `"mist"` |
|
||||
| Particle color | `particleColor` | `"#FFE082"` |
|
||||
| Particle density | `particleCount` | `20` (range: 1-50) |
|
||||
| Particle brightness | `particleIntensity` | `0.5` (range: 0-1) |
|
||||
| Lighting start | `lightingFrom` | `"rgba(255,200,100,0.15)"` or `"transparent"` |
|
||||
| Lighting end | `lightingTo` | `"rgba(255,107,157,0.08)"` or `"transparent"` |
|
||||
| Cinematic edge darken | `vignette` | `true` / `false` |
|
||||
| Scene background | `backgroundColor` | `"#0A0A1A"` |
|
||||
|
||||
Reference: `remotion-composer/public/demo-props/mori-no-seishin.json` — 6 scenes using this pattern.
|
||||
Reference: `remotion-composer/public/demo-props/deep-ocean.json` — 6 underwater scenes with different palette.
|
||||
|
||||
### 4. Use Metadata For Timing Rules
|
||||
|
||||
Recommended metadata keys:
|
||||
|
||||
Reference in New Issue
Block a user