Zero-key video formula: fix rendering bugs, add 8 demo compositions, update skills
Fix ComparisonCard and CalloutBox dark-theme passthrough in SceneRenderer. Add 8 demo-props JSON files (3 fixed originals + 5 new compositions). Update scene-director and compose-director skills with all 10 Remotion scene types and zero-key rendering guidance. Document the proven all-dark-background formula in remotion.md. Add cinematic renderer, onboarding skill, prompt gallery, and demo render script.
This commit is contained in:
calesthio
@@ -271,6 +271,7 @@ Cross-cutting skills that apply to all pipelines:
|
||||
|
||||
| Skill | File | Purpose |
|
||||
|-------|------|---------|
|
||||
| Onboarding | `meta/onboarding.md` | First-interaction greeting, capability discovery, starter prompts |
|
||||
| Reviewer | `meta/reviewer.md` | Self-review protocol after every stage |
|
||||
| Checkpoint Protocol | `meta/checkpoint-protocol.md` | When/how to checkpoint and request human approval |
|
||||
| Skill Creator | `meta/skill-creator.md` | Dynamically create new skills during pipeline runs |
|
||||
|
||||
+60
-2
@@ -27,8 +27,66 @@ stages, how artifacts flow in, and how renders are triggered.
|
||||
| Talking-head (video-only cuts) | FFmpeg | No images/animations needed |
|
||||
|
||||
**Note:** The `render` operation auto-routes — if any cut contains images,
|
||||
animations, transitions, or component types (text_card, stat_card, etc.),
|
||||
it delegates to Remotion automatically. No need to manually select backend.
|
||||
animations, transitions, or component types, it delegates to Remotion
|
||||
automatically. No need to manually select backend.
|
||||
|
||||
## Supported Scene Types (Cut Types)
|
||||
|
||||
The Explainer composition supports the following cut types:
|
||||
|
||||
| Type | Props Required | Best For |
|
||||
|------|---------------|----------|
|
||||
| `text_card` | `text` | Statements, titles, closing messages |
|
||||
| `stat_card` | `stat`, optional `subtitle`, `accentColor` | Big numbers, impactful metrics |
|
||||
| `hero_title` | `text`, optional `heroSubtitle` | Opening titles, dramatic reveals |
|
||||
| `callout` | `text`, optional `title`, `callout_type` (info/warning/tip/quote) | Tips, quotes, important notes |
|
||||
| `comparison` | `leftLabel`, `rightLabel`, `leftValue`, `rightValue` | Before/after, A/B, versus |
|
||||
| `bar_chart` | `chartData` [{label, value}], optional `title`, `chartAnimation` | Category comparisons, rankings |
|
||||
| `line_chart` | `chartSeries` [{label, data: [{x,y}]}], optional `title` | Trends, time series, growth |
|
||||
| `pie_chart` | `chartData` [{label, value}], optional `donut`, `centerLabel` | Proportions, breakdowns |
|
||||
| `kpi_grid` | `chartData` [{label, value, prefix, suffix, change, icon}] | Dashboards, traction metrics |
|
||||
| `progress_bar` | `progress` (0-100), optional `progressSegments` | Journey viz, completion, stacked metrics |
|
||||
|
||||
**Chart animations:** `grow-up`, `slide-in`, `pop` (bar), `draw`, `fade-in` (line), `spin`, `expand`, `sequential` (pie), `count-up`, `pop`, `cascade` (kpi)
|
||||
|
||||
**Zero-key video strategy:** When no image or video generation is available, build
|
||||
entire videos from these component types. A well-composed sequence of hero_title →
|
||||
kpi_grid → bar_chart → comparison → stat_card → text_card produces a polished,
|
||||
professional video with zero external dependencies.
|
||||
|
||||
### The Proven Formula for Zero-Key Videos
|
||||
|
||||
These rules were discovered through systematic render testing and produce cinematic results:
|
||||
|
||||
**1. ALL-DARK BACKGROUNDS (mandatory).** Set `backgroundColor: "#0F172A"` on EVERY scene.
|
||||
This prevents jarring white↔dark flash transitions and makes chart colors pop dramatically.
|
||||
Dark backgrounds transform basic charts into cinematic data visualization.
|
||||
|
||||
**2. Flat props format.** All scene properties go at the TOP LEVEL of the cut object
|
||||
(e.g., `cut.text`, `cut.chartData`), NOT nested under a `props` key.
|
||||
|
||||
**3. KPI Grid data rules:**
|
||||
- `value` must be a small, human-readable number. The component auto-formats ≥1M→"XM", ≥1K→"XK".
|
||||
For "8.1 Billion" use `value: 8.1, suffix: " Billion"`. Never use raw huge numbers with a suffix.
|
||||
- `change` must be a NUMBER (e.g., `3.2`), not a string (e.g., NOT `"+3.2%"`).
|
||||
|
||||
**4. Comparison and Callout theming:**
|
||||
- `comparison` accepts `backgroundColor` and `color` (text color) for dark themes.
|
||||
- `callout` accepts `backgroundColor` which sets both the container and card background.
|
||||
|
||||
**5. Overlays add polish.**
|
||||
- `section_title` overlays group scenes narratively ("THE CRISIS", "THE DATA").
|
||||
- `stat_reveal` overlays float dramatic numbers over chart scenes (e.g., "10x" in corner).
|
||||
|
||||
**6. Scene pacing:** 4-6 seconds per scene, 8-10 scenes for a 45-50s video. Give chart
|
||||
animations at least 4 seconds to complete. Hero title needs only 4 seconds.
|
||||
|
||||
**7. Color palette cohesion.** Pick 4-5 accent colors that relate to the topic and use
|
||||
them consistently across charts, overlays, and accents. Use the same chartColors array
|
||||
across bar/pie/line scenes for visual unity.
|
||||
|
||||
**Reference compositions:** See `remotion-composer/public/demo-props/climate-dashboard.json`
|
||||
as the gold standard, and other demo files for additional patterns.
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
# Onboarding — Meta Skill
|
||||
|
||||
## When to Use
|
||||
|
||||
On the **very first interaction** with a user in a new session when the user has not yet specified a concrete production request — or when their request is vague ("make me a video", "what can you do?", "help me create something").
|
||||
|
||||
Skip this skill when the user arrives with a specific, actionable request like "Make a 60-second explainer about black holes." In that case, go directly to Rule Zero (identify pipeline → preflight → execute). The user already knows what they want.
|
||||
|
||||
**This skill transforms the agent from a passive executor into a creative partner.** Most users don't know what's possible. Your job is to show them — fast, clearly, and with copy-paste prompts they can try right now.
|
||||
|
||||
## Protocol
|
||||
|
||||
### Step 1: Run Preflight Discovery
|
||||
|
||||
Before saying anything creative, know what you're working with:
|
||||
|
||||
```bash
|
||||
python -c "
|
||||
from tools.tool_registry import registry
|
||||
import json
|
||||
registry.discover()
|
||||
envelope = registry.support_envelope()
|
||||
menu = registry.provider_menu()
|
||||
print('=== ENVELOPE ===')
|
||||
print(json.dumps(envelope, indent=2))
|
||||
print('=== MENU ===')
|
||||
print(json.dumps(menu, indent=2))
|
||||
"
|
||||
```
|
||||
|
||||
Parse the output into three buckets:
|
||||
|
||||
1. **Available** — tools with `status: AVAILABLE`
|
||||
2. **Quick unlocks** — tools with `status: UNAVAILABLE` whose `install_instructions` reference an env var (1-minute fixes)
|
||||
3. **Hardware unlocks** — tools requiring GPU or local model downloads
|
||||
|
||||
### Step 2: Determine the User's Setup Tier
|
||||
|
||||
Based on discovery, classify the setup:
|
||||
|
||||
| Tier | What's Available | Best Pipelines |
|
||||
|------|-----------------|----------------|
|
||||
| **Zero-key** | Piper TTS + Pexels/Pixabay stock (if keys added) + Remotion + FFmpeg | Animated Explainer (stock visuals + free narration) |
|
||||
| **Starter** | One image gen provider (e.g., FLUX via FAL_KEY) + free TTS + Remotion | Animated Explainer, Animation (AI-generated visuals) |
|
||||
| **Standard** | Image gen + TTS + music gen | Animated Explainer, Animation, Screen Demo, Hybrid |
|
||||
| **Full** | Video gen + image gen + premium TTS + music | All pipelines including Cinematic, Avatar, Talking Head |
|
||||
| **Full + GPU** | Cloud APIs + local video gen models | All pipelines with free local fallbacks |
|
||||
|
||||
### Step 3: Greet and Orient
|
||||
|
||||
Present a **short, friendly capability summary**. Do NOT dump the raw provider menu. Instead, translate it into plain language.
|
||||
|
||||
**Template (adapt to actual discovery results):**
|
||||
|
||||
---
|
||||
|
||||
**Welcome to OpenMontage!** I'm your video production agent. Here's what I can do with your current setup:
|
||||
|
||||
**Ready to go:**
|
||||
- [List 2-4 key capabilities in plain language, e.g., "Generate narration with free offline TTS (Piper)", "Create animated videos with spring transitions, captions, and charts (Remotion)", "Stock footage and images from Pexels"]
|
||||
|
||||
**Available pipelines:** [List the pipelines that work with their setup, with one-line descriptions]
|
||||
|
||||
**Quick upgrades:** [If applicable — "Add `FAL_KEY` to your `.env` to unlock AI-generated images (FLUX) and video (Veo, Kling, MiniMax) — one key, five tools."]
|
||||
|
||||
---
|
||||
|
||||
**Rules for this presentation:**
|
||||
- Lead with what WORKS, not what's missing. The user should feel empowered, not inadequate.
|
||||
- Keep it to 8-12 lines max. Don't overwhelm.
|
||||
- Mention at most 2 quick-unlock suggestions. Don't nag about every missing key.
|
||||
- Read actual `install_instructions` from the registry — do not hardcode provider names or key names.
|
||||
|
||||
### Step 4: Offer Starter Prompts
|
||||
|
||||
Based on the user's tier, present **3 ready-to-use prompts** they can copy right now. These should be prompts that will work well with their specific setup and produce impressive results.
|
||||
|
||||
**Zero-key prompts:**
|
||||
|
||||
> **Try this now:** "Make a 45-second animated explainer about why the sky is blue"
|
||||
>
|
||||
> This will research the topic, write a script, find stock visuals, generate narration with Piper, and compose an animated video with transitions and captions — all free.
|
||||
|
||||
> **Also try:** "Create a 60-second data-driven video about coffee consumption around the world"
|
||||
|
||||
> **Or:** "Make a short explainer about how the internet works, with narration and animated captions"
|
||||
|
||||
**Starter-tier prompts (image gen available):**
|
||||
|
||||
> **Try this:** "Create an animated explainer about how CRISPR gene editing works, with AI-generated visuals"
|
||||
>
|
||||
> I'll use FLUX to generate custom images for each scene — much more visually striking than stock.
|
||||
|
||||
> **Also try:** "Make a product launch teaser for a fictional smart water bottle called AquaPulse"
|
||||
|
||||
> **Or:** "Build a 90-second explainer about the psychology of color in marketing"
|
||||
|
||||
**Full-tier prompts (video gen available):**
|
||||
|
||||
> **Try this:** "Create a cinematic 30-second trailer for a sci-fi concept: humanity receives a warning from 1000 years in the future"
|
||||
>
|
||||
> I'll generate actual motion video clips, compose a soundtrack, and deliver a finished cinematic trailer.
|
||||
|
||||
> **Also try:** "Make a 60-second avatar spokesperson video announcing a company rebrand"
|
||||
|
||||
> **Or:** "Create a 90-second animated explainer about quantum computing for middle school students, with a fun narrator voice and custom soundtrack"
|
||||
|
||||
**Rules for prompt suggestions:**
|
||||
- Present exactly 3 prompts.
|
||||
- The first prompt should be the most impressive thing their setup can produce.
|
||||
- Each prompt should target a different pipeline or style.
|
||||
- Include a brief note explaining what makes this prompt a good fit for their setup.
|
||||
- Use blockquote formatting so prompts are visually distinct and easy to copy.
|
||||
|
||||
### Step 5: Explain the Workflow (Briefly)
|
||||
|
||||
After prompts, give a 2-3 sentence summary of what happens when they start:
|
||||
|
||||
"When you give me a prompt, I'll first research the topic with live web searches, then present you with concept options and cost estimates. You pick your favorite, and I'll produce the video stage by stage — asking for your approval at each creative decision. The final video lands in `projects/<name>/renders/`."
|
||||
|
||||
Do NOT explain the full architecture, three-layer knowledge system, or pipeline internals here. That's for the curious — point them to `AGENT_GUIDE.md` if they want to go deeper.
|
||||
|
||||
### Step 6: Handle Follow-Up Questions
|
||||
|
||||
Common questions and how to respond:
|
||||
|
||||
**"What does it cost?"**
|
||||
- Zero-key path: $0
|
||||
- With FAL_KEY: typically $0.30–$1.50 per video depending on image count
|
||||
- Full setup: $1–$3 for most videos
|
||||
- Always: "I'll show you exact cost estimates before spending anything."
|
||||
|
||||
**"Can you make [specific type]?"**
|
||||
- Match to a pipeline. If it fits, say which pipeline and what tools you'd use.
|
||||
- If it doesn't fit any pipeline, be honest — suggest the closest match and explain what would be different.
|
||||
|
||||
**"How long does it take?"**
|
||||
- Explainer (zero-key): 5-15 minutes
|
||||
- Explainer (with image gen): 10-20 minutes
|
||||
- Cinematic (with video gen): 20-40 minutes
|
||||
- "Most of the time is asset generation. The research and scripting stages are fast."
|
||||
|
||||
**"I just want to test it quickly"**
|
||||
- Suggest the shortest zero-key prompt: "Try: 'Make a 30-second explainer about why leaves change color.' It'll use free tools and finish in about 5 minutes."
|
||||
|
||||
**"Show me what you can do"**
|
||||
- Point to the demo video in the README, then offer the starter prompts from Step 4.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- **Don't dump the raw JSON** from `support_envelope()` or `provider_menu()` on the user. Translate it into plain language.
|
||||
- **Don't list every tool.** Group by capability ("I can generate images with FLUX" not "I have flux_image, google_imagen, openai_image, recraft_image...").
|
||||
- **Don't explain the architecture** unless asked. "Agent-first, instruction-driven" is interesting to developers, but the user came to make a video, not study the codebase.
|
||||
- **Don't apologize for missing capabilities.** Frame as "here's what you have" and optionally "here's a quick upgrade." Never "unfortunately you don't have..."
|
||||
- **Don't skip straight to production** if the user seems uncertain or exploratory. Take 30 seconds to orient them — it saves 10 minutes of confusion later.
|
||||
- **Don't suggest prompts that require tools the user doesn't have.** Every prompt must be achievable with their current setup. Mark any that need specific keys clearly.
|
||||
@@ -84,6 +84,13 @@ If using Remotion for animated segments:
|
||||
2. Call `video_compose` with `operation: "remotion_render"` for animated segments
|
||||
3. Assemble Remotion outputs with remaining segments via FFmpeg
|
||||
|
||||
**Zero-key Remotion render (component-only videos):**
|
||||
When all scenes are Remotion component types (hero_title, stat_card, bar_chart, line_chart,
|
||||
pie_chart, kpi_grid, comparison, callout, progress_bar, text_card), render the entire video
|
||||
as a single Remotion composition using the Explainer entry point. No FFmpeg assembly needed.
|
||||
The edit_decisions cuts array maps directly to Remotion props. See `skills/core/remotion.md`
|
||||
for the proven formula — especially the all-dark-background rule for visual consistency.
|
||||
|
||||
### Step 5: Audio Post-Processing
|
||||
|
||||
Call the `audio_mixer` tool to:
|
||||
|
||||
@@ -67,15 +67,25 @@ Transform each script section into 1-3 visual scenes. Each scene is a distinct v
|
||||
|
||||
| Type | Best For | Available Tools | Duration Guidance |
|
||||
|------|----------|-----------------|-------------------|
|
||||
| `animation` | Concepts needing motion (data flow, transformations, math) | Remotion, Manim | 4-10s |
|
||||
| `diagram` | Processes, architecture, relationships | `diagram_gen` (Mermaid), `image_selector` (stylized) | 4-8s |
|
||||
| `text_card` | Key terms, definitions, statistics, quotes | Remotion TextCard component | 3-5s |
|
||||
| `hero_title` | Opening titles, dramatic reveals | Remotion HeroTitle (cyan first word, animated underline) | 3-5s |
|
||||
| `stat_card` | Big dramatic numbers, impactful metrics | Remotion StatCard (large stat + subtitle) | 4-6s |
|
||||
| `bar_chart` | Category comparisons, rankings | Remotion BarChart (animated grow-up/slide-in/pop) | 5-7s |
|
||||
| `line_chart` | Trends, time series, growth curves | Remotion LineChart (draw/fade animation, multi-series) | 5-7s |
|
||||
| `pie_chart` | Proportions, breakdowns, distributions | Remotion PieChart (donut mode, center label, spin/expand) | 5-7s |
|
||||
| `kpi_grid` | Dashboards, traction metrics, at-a-glance data | Remotion KPIGrid (2-4 columns, count-up/pop/cascade) | 5-7s |
|
||||
| `comparison` | Before/after, A/B, versus comparisons | Remotion ComparisonCard (dual-value with divider) | 4-6s |
|
||||
| `callout` | Expert quotes, tips, warnings, important notes | Remotion CalloutBox (info/warning/tip/quote types) | 4-6s |
|
||||
| `progress_bar` | Journey visualization, completion, stacked metrics | Remotion ProgressBar (fill/pulse/step animations) | 4-6s |
|
||||
| `text_card` | Statements, closing messages, key terms | Remotion TextCard (centered, spring animation) | 3-5s |
|
||||
| `animation` | Concepts needing motion (data flow, math) | Remotion, Manim | 4-10s |
|
||||
| `diagram` | Processes, architecture, relationships | `diagram_gen` (Mermaid), `image_selector` | 4-8s |
|
||||
| `generated` | Illustrations, metaphors, real-world imagery | `image_selector` (FLUX/DALL-E) | 3-6s |
|
||||
| `talking_head` | AI avatar speaking (if HeyGen available) | HeyGen tools | 5-15s |
|
||||
| `broll` | Context, real-world examples | Stock or generated footage | 3-6s |
|
||||
| `transition` | Dedicated transition moment between topics | Remotion transition | 1-2s |
|
||||
| `screen_recording` | Code demos, UI walkthroughs | Recorded or simulated | 5-15s |
|
||||
|
||||
**Zero-key scene selection:** When no image/video generation is available, prefer `hero_title`, `stat_card`, `bar_chart`, `line_chart`, `pie_chart`, `kpi_grid`, `comparison`, `callout`, `progress_bar`, and `text_card`. These render entirely from Remotion components with zero external dependencies and produce professional, animated results. See `skills/core/remotion.md` for the proven formula (all-dark backgrounds, KPI data formatting rules, overlay techniques).
|
||||
|
||||
### Step 4: Apply the Visual Technique Library
|
||||
|
||||
These are proven patterns for explainer visuals. Reference them by name in scene descriptions:
|
||||
@@ -91,14 +101,20 @@ Show the abstract concept alongside its real-world analogy. Split screen or side
|
||||
- Example: "Left: actual vector space with dots. Right: a library with books sorted by topic."
|
||||
|
||||
**Stat Card Punch**
|
||||
Full-screen number or comparison. Appears with impact animation (scale up, slight bounce). Hold for 2-3 seconds.
|
||||
- Tools: Remotion TextCard component
|
||||
- Example: "1ms" in large text, then smaller text below: "vs 500ms with traditional search"
|
||||
Full-screen number with impact animation (scale up, slight bounce). Use `stat_card` type with a dark background and bold accent color. Hold for 4-5 seconds.
|
||||
- Tools: Remotion StatCard component
|
||||
- Example: stat="1ms", subtitle="vs 500ms with traditional search", accentColor="#22D3EE"
|
||||
|
||||
**Data Dashboard Sequence**
|
||||
A series of data visualization scenes that tell a story through numbers. Start with a KPI overview, then drill into specific charts. Use section_title overlays to group related data. This pattern works with zero external tools.
|
||||
- Tools: Remotion chart components (bar_chart, line_chart, pie_chart, kpi_grid)
|
||||
- Example: kpi_grid (4 key stats) → bar_chart (breakdown) → line_chart (trend) → pie_chart (distribution)
|
||||
- Always use dark backgrounds (`backgroundColor: "#0F172A"`) for cinematic feel.
|
||||
|
||||
**Before/After Split**
|
||||
Show the problem, then the solution. Can be sequential (problem → transition → solution) or split-screen.
|
||||
- Tools: `image_selector` for both states
|
||||
- Example: "Before: SQL query scanning millions of rows (slow). After: vector search finding nearest neighbors (fast)."
|
||||
Show the problem, then the solution using `comparison` type. The comparison card shows dual values side-by-side with animated entrance.
|
||||
- Tools: Remotion ComparisonCard component
|
||||
- Example: leftLabel="Before", leftValue="500ms", rightLabel="After", rightValue="1ms"
|
||||
|
||||
**Timeline Progression**
|
||||
Left-to-right or top-to-bottom sequence showing evolution or process steps. Each step appears as narrator describes it.
|
||||
|
||||
Reference in New Issue
Block a user