From a36ce99793cd5005e10be11a4704093ff7999fce Mon Sep 17 00:00:00 2001 From: calesthio Date: Thu, 16 Apr 2026 19:22:46 -0700 Subject: [PATCH] screen-demo: add synthetic-terminal mode via Remotion TerminalScene MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make synthetic screen recording a first-class, discoverable capability alongside real OS capture. For CLI / terminal / install-flow demos where commands and output are predictable, author a `terminal_scene` cut instead of driving a real screen recorder — deterministic, privacy-safe, pixel- perfect, and frame-accurate to narration cues. Components: - TerminalScene.tsx: window chrome, char-by-char typing, blinking cursor, scrolling output, non-blocking floating pills, spring-based reveals - ProviderChip.tsx: rotating badge overlay that cycles through provider names (used in AI-generated-motion scenes) - BackgroundVideoLayer + source_in_seconds + backgroundVideo props on every scene type — supports video-behind-component composition Discovery chain (six layers, so the next agent finds this without reading source code): 1. pipeline_defs/screen-demo.yaml — bump to 2.1, declare production_modes (real_capture, synthetic_terminal) with required_tools, scene_type, and agent_skills pointers 2. skills/pipelines/screen-demo/idea-director.md — mode-selection table at brief time; brief.metadata.production_mode contract 3. skills/pipelines/screen-demo/asset-director.md — reads production_mode and branches asset production (capture+overlays vs steps+narration+ pacing check) 4. .agents/skills/synthetic-screen-recording/SKILL.md — Layer 3 skill with step kinds (cmd/out/pause/pill), pacing rule, and the frozen-terminal failure mode captured from the showcase v3 retune 5. AGENT_GUIDE.md — TerminalScene added to Remotion routing; links SCENE_TYPES.md as the authoritative cut-type registry 6. remotion-composer/SCENE_TYPES.md — new cheat sheet of every cut.type and overlay.type with required fields, plus a "how to add a new scene type" section (candidates: ChatTranscript, EditorScene, PrReview, SlackThread, TicketBoard) Guardrail: - lib/verify_scene_pacing.py — reusable trace() and assert_alignment() helpers that mimic the TerminalScene frame math exactly. Fail loudly before render if steps burn through too fast or leave the scene frozen. --- .../synthetic-screen-recording/SKILL.md | 176 ++++++++++++ AGENT_GUIDE.md | 6 + lib/verify_scene_pacing.py | 134 +++++++++ pipeline_defs/screen-demo.yaml | 41 ++- remotion-composer/SCENE_TYPES.md | 64 +++++ remotion-composer/src/Explainer.tsx | 121 ++++++-- .../src/components/ProviderChip.tsx | 101 +++++++ .../src/components/TerminalScene.tsx | 266 ++++++++++++++++++ remotion-composer/src/components/index.ts | 3 + .../pipelines/screen-demo/asset-director.md | 17 ++ skills/pipelines/screen-demo/idea-director.md | 15 +- 11 files changed, 918 insertions(+), 26 deletions(-) create mode 100644 .agents/skills/synthetic-screen-recording/SKILL.md create mode 100644 lib/verify_scene_pacing.py create mode 100644 remotion-composer/SCENE_TYPES.md create mode 100644 remotion-composer/src/components/ProviderChip.tsx create mode 100644 remotion-composer/src/components/TerminalScene.tsx diff --git a/.agents/skills/synthetic-screen-recording/SKILL.md b/.agents/skills/synthetic-screen-recording/SKILL.md new file mode 100644 index 0000000..2109e88 --- /dev/null +++ b/.agents/skills/synthetic-screen-recording/SKILL.md @@ -0,0 +1,176 @@ +# Synthetic Screen Recording (Remotion TerminalScene) + +**Decision this skill answers:** When the user wants a screen-recording-looking demo of a terminal, CLI tool, or coding workflow — do I **capture the real desktop** (OS screen recording via `screen_recorder`, Windows-MCP, Cap, or Playwright), or do I **synthesize it in Remotion** with the `TerminalScene` component? + +> **Heuristic:** If the agent can author the exact command/output sequence in advance, synthesize. Only capture live when the real behavior is unpredictable, needs a real app UI, or the user explicitly asked for a real recording. + +## Why this exists + +v3 of the OpenMontage showcase tried to use Windows-MCP + `screen_recorder` to drive a Git-Bash window for the install walkthrough. It stalled on window positioning, focus races, and taskbar privacy concerns. We pivoted to **pure Remotion rendering** — a React component named `TerminalScene` that draws a fake terminal and types commands character-by-character. The output is visually indistinguishable from a real screen recording (same traffic-light window chrome, blinking cursor, scrolling output) but deterministic, privacy-safe, pixel-perfect at 1080p, and pace-controllable to the frame. + +That component + pattern is the capability this skill makes discoverable. + +## When to use synthetic (TerminalScene) + +**YES, synthesize when:** +- The demo is a **terminal / CLI / coding session** where commands and outputs are predictable +- The user wants a polished tutorial feel (clean typography, floating pills, cursor blink) +- Install walkthroughs, setup demos, API key config, `make` targets, `git clone` flows +- You need tight sync with narration — every command must land on a specific beat +- You want the result reproducible (re-render gets identical pixels) +- The user's actual desktop has private apps/windows visible you'd otherwise have to crop + +**NO, capture a real screen when:** +- The demo is **a real app UI** that can't be faked (Figma, Photoshop, a web app with live state, a browser flow) +- The user explicitly asked for a recording of *their* actual screen +- The behavior depends on timing you can't script (streaming LLM output, real network latency) +- There's a visual quirk (a cursor effect, a plugin pop-up) that only appears in the live environment + +**For a browser demo** → `playwright-recording` skill, not this one. +**For a real desktop** → `screen_recorder` tool or Cap via `cap_recorder`. + +## The component — `TerminalScene` + +Located at: `remotion-composer/src/components/TerminalScene.tsx` +Exported from: `remotion-composer/src/components/index.ts` +Wired in dispatch: `remotion-composer/src/Explainer.tsx` (`if (cut.type === "terminal_scene")`) + +**Props:** +```ts +interface TerminalSceneProps { + title?: string; // shown in the window title bar + steps: TerminalStep[]; // the timeline + prompt?: string; // "$", ">", etc. + accentColor?: string; // pill + prompt glow + backgroundColor?: string; +} +``` + +**Step kinds:** +```ts +{ kind: "cmd", text: string, typeSpeed?: number, holdSeconds?: number } +{ kind: "out", text: string, holdSeconds?: number } +{ kind: "pause", seconds: number } +{ kind: "pill", text: string, color?: string, durationSeconds?: number } +``` + +- `cmd` — prints the prompt, types the text character-by-character (`typeSpeed` is seconds per character, default 0.035), then holds for `holdSeconds` (default 0.3) +- `out` — a line of program output, reveals instantly with a short fade-in +- `pause` — dead time. Terminal holds on last visible state. USE THIS TO SYNC WITH NARRATION. +- `pill` — non-blocking floating badge (top-right). Spring-in, hold, spring-out. Does NOT advance the cursor — the next step runs in parallel. + +## Authoring pattern + +Author a new scene by adding a cut to `build_composition.py` (or your equivalent props builder): + +```python +install_steps = [ + {"kind": "pause", "seconds": 7.0}, # wait for intro narration + {"kind": "cmd", "text": "git clone https://github.com/calesthio/OpenMontage.git", + "typeSpeed": 0.045, "holdSeconds": 0.3}, + {"kind": "out", "text": "Cloning into 'OpenMontage'..."}, + {"kind": "out", "text": "remote: Enumerating objects: 2847, done."}, + {"kind": "pill", "text": "repo cloned", "color": "#34D399", "durationSeconds": 2.6}, + {"kind": "pause", "seconds": 3.8}, # bridge to next narration cue + # ... +] + +cuts.append({ + "id": "install-terminal", + "type": "terminal_scene", + "terminalTitle": "bash — OpenMontage setup", + "prompt": "$", + "accentColor": "#22D3EE", + "steps": install_steps, + "in_seconds": 50.0, + "out_seconds": 110.0, +}) +``` + +## THE RULE: pace with narration, never ahead + +**The #1 failure mode:** steps run continuously and burn through all content in the first 40% of the scene, leaving the terminal frozen for the remaining 60%. This is what killed the v3 first pass — the capability menu rendered at t=80s but narration didn't announce it until t=92s. + +**Do this instead:** + +1. **Know your narration cues** — for each scene, write down the exact video-time each narration segment starts. +2. **Start with a pause** that reaches the first narration cue before any command types. +3. **Time each command to land with its narration line** — `cmd` should start typing the moment narration says its line, not before. +4. **Put pauses between command groups** that bridge to the next narration cue. +5. **End with a closer hold** — a pause long enough that the final state is readable after narration ends. + +**Sanity-check your steps before rendering** — every minute of Remotion render is precious. Sum the step durations and verify they equal scene duration: + +```python +import math +def trace(steps, scene_start, fps=30): + t = 0.0 + for s in steps: + k = s["kind"] + if k == "cmd": + tf = math.ceil(len(s["text"]) * s.get("typeSpeed", 0.035) * fps) + t += tf / fps + s.get("holdSeconds", 0.3) + elif k == "out": + t += max(2, math.ceil(0.08 * fps)) / fps + s.get("holdSeconds", 0.15) + elif k == "pause": + t += s["seconds"] + # "pill" is non-blocking — does NOT advance cursor + print(f" {t + scene_start:6.2f}s {k}: {s.get('text', '')[:40]}") +trace(install_steps, 50) +``` + +Look at the output column. Each narration cue's video-time must appear adjacent to the command/output it announces. If a command lands 10s before or after its cue, adjust pauses. + +See `lib/verify_scene_pacing.py` for a reusable version of this script. + +## Design rules (inherited from the v3 retune) + +- **Intro pause** — every terminal scene opens with at least 2s of empty-terminal-with-blinking-cursor before anything types. The viewer needs to register the window. +- **Pill timing** — a pill should fire at the exact moment its named event completes on screen (e.g., `repo cloned` immediately after the last `Receiving objects` line). Pills are your substitute for real-world UI notifications. +- **Command hold after typing** — keep `holdSeconds` ≥ 0.3 on every `cmd` so viewers register the completed command before the first output scrolls in. +- **Output cadence** — space `holdSeconds` on output lines between 0.4 and 1.0. Output that flies too fast feels like a bug; output that crawls feels boring. +- **Auto-scroll works** — the terminal holds the most recent 18 lines. Don't worry about off-screen content. +- **Cursor blinks only on the latest command line while typing + a ~0.2s tail** after typing completes. + +## `ProviderChip` (companion component) + +The `.agents/skills/synthetic-screen-recording` pattern also owns `ProviderChip` — a rotating badge overlay that cycles through a list of provider names at a fixed cadence. Used in the v3 showcase to cycle through all 11 AI video-gen providers during the "generated motion" section. + +```python +overlays.append({ + "type": "provider_chip", + "providers": ["Veo 3.1", "Seedance 2.0", "Kling 2.5", ...], + "cycleSeconds": 2.5, + "position": "bottom-right", + "accentColor": "#22D3EE", + "label": "generated with", + "in_seconds": 195.0, + "out_seconds": 222.5, +}) +``` + +Wired in dispatch at: `remotion-composer/src/Explainer.tsx` overlay renderer (`overlay.type === "provider_chip"`). + +## Adding new synthetic-UI components + +The pattern generalizes. When you need to fake another UI surface (Claude Code chat bubbles, a Jira ticket view, a GitHub PR diff, a Slack message, a VS Code status bar): + +1. Copy `TerminalScene.tsx` as a template. +2. Define a `steps` interface for the relevant timeline primitives. +3. Render each step by interpolating `frame` against cumulative start/end times. +4. Wire it into `Explainer.tsx`'s `SceneRenderer` dispatch with a new `cut.type`. +5. Add the type to the `Cut` interface in `Explainer.tsx` and to `components/index.ts`. +6. Add a section to this skill documenting it. +7. Update `remotion-composer/SCENE_TYPES.md` with the new cut type. + +## Related skills + +- `.agents/skills/remotion` — general Remotion authoring (hooks, springs, sequences) +- `.agents/skills/playwright-recording` — real browser-flow capture for web apps +- `tools/capture/screen_recorder` — ffmpeg-based desktop capture +- `tools/capture/cap_recorder` — Cap.so polished desktop capture +- `skills/pipelines/screen-demo/asset-director.md` — chooses between synthetic and real for a screen-demo project + +## Provenance + +Introduced: OpenMontage showcase v3 render (2026-04-16). Original motivation: the v3 setup walkthrough section needed a 60-second install demo where every command aligned to Chirp 3 HD narration cues, and Windows-MCP-driven real capture was too flaky in practice. See `projects/openmontage-showcase/build_composition.py` for the reference implementation. diff --git a/AGENT_GUIDE.md b/AGENT_GUIDE.md index 3508302..8b61c3e 100644 --- a/AGENT_GUIDE.md +++ b/AGENT_GUIDE.md @@ -356,6 +356,12 @@ For these requests: - Explainer videos with `flat-motion-graphics` playbook -> Remotion animated scenes, not Ken Burns - Data-driven videos -> Remotion stat cards and charts, not static image screenshots - Any pipeline using still images -> Remotion spring animations, not FFmpeg pan-and-zoom +- **Screen demos of a CLI/terminal/install flow -> `TerminalScene` (synthetic screen recording), not OS-level capture.** See `.agents/skills/synthetic-screen-recording/SKILL.md`. Faster, deterministic, privacy-safe. Use real capture (`screen_recorder`, `cap_recorder`, `playwright-recording`) only when the demo is a real app UI or requires unpredictable live behavior. + +### Remotion scene types available in `remotion-composer/` + +See `remotion-composer/SCENE_TYPES.md` for the authoritative list and their cut schemas. Current scene types usable via `cut.type`: +`text_card`, `stat_card`, `callout`, `comparison`, `hero_title`, `terminal_scene`, `anime_scene`, `bar_chart`, `line_chart`, `pie_chart`, `kpi_grid`, `progress_bar`. Overlay types include `section_title`, `stat_reveal`, `hero_title`, `provider_chip`. **When Remotion is NOT available**, `video_compose` falls back to FFmpeg Ken Burns motion on still images. This still works but produces less engaging visuals. Mention this tradeoff in the proposal. diff --git a/lib/verify_scene_pacing.py b/lib/verify_scene_pacing.py new file mode 100644 index 0000000..caf9319 --- /dev/null +++ b/lib/verify_scene_pacing.py @@ -0,0 +1,134 @@ +"""Verify that a TerminalScene `steps` list paces with narration cues. + +Use from any build_composition.py or synthetic-UI builder: + + from lib.verify_scene_pacing import trace, assert_alignment + + trace(install_steps, scene_start=50.0) + assert_alignment( + install_steps, + scene_start=50.0, + scene_end=110.0, + narration_cues=[ + (57.0, "seg07 Clone the repo"), + (65.5, "seg08 Run make setup"), + (83.0, "seg09 Open the folder"), + (92.0, "seg10 agent reads guide"), + ], + tolerance=1.0, + ) + +The tracer mimics the frame math inside TerminalScene.tsx so video-time +estimates are exact to 1/fps. Fails loudly if any narration cue has no +matching command/output within `tolerance` seconds. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass +from typing import Any + + +def step_duration(step: dict[str, Any], fps: int = 30) -> float: + """Return the cursor-advancement for a single step (frame-accurate). + + Pills DO NOT advance the cursor — they're non-blocking overlays. + """ + k = step["kind"] + if k == "cmd": + type_frames = math.ceil(len(step["text"]) * step.get("typeSpeed", 0.035) * fps) + return type_frames / fps + step.get("holdSeconds", 0.3) + if k == "out": + reveal_frames = max(2, math.ceil(0.08 * fps)) + return reveal_frames / fps + step.get("holdSeconds", 0.15) + if k == "pause": + return float(step["seconds"]) + if k == "pill": + return 0.0 + raise ValueError(f"Unknown step kind: {k!r}") + + +@dataclass +class Landmark: + video_time: float + kind: str + text: str + + +def trace(steps: list[dict[str, Any]], scene_start: float = 0.0, fps: int = 30, *, quiet: bool = False) -> list[Landmark]: + """Walk the step list and print a video-time landmark for each visible event. + + Returns the list of landmarks (useful for alignment checks). + """ + cursor = 0.0 + out: list[Landmark] = [] + for s in steps: + k = s["kind"] + vt = round(cursor + scene_start, 2) + if k in ("cmd", "out", "pill"): + text = s.get("text", "") + out.append(Landmark(video_time=vt, kind=k.upper(), text=text)) + if not quiet: + prefix = {"CMD": "CMD ", "OUT": "OUT ", "PILL": "PILL "}[k.upper()] + print(f" {vt:7.2f}s {prefix}{text[:60]}") + cursor += step_duration(s, fps) + + end_vt = round(cursor + scene_start, 2) + if not quiet: + print(f" {end_vt:7.2f}s -- steps end --") + return out + + +def assert_alignment( + steps: list[dict[str, Any]], + scene_start: float, + scene_end: float, + narration_cues: list[tuple[float, str]], + *, + tolerance: float = 1.0, + fps: int = 30, +) -> None: + """Validate that every narration cue has a visual landmark within tolerance. + + Also checks that total step duration does not overflow scene_end. + Raises AssertionError on any mismatch. + """ + landmarks = trace(steps, scene_start, fps, quiet=True) + errors: list[str] = [] + + for cue_time, cue_desc in narration_cues: + # Find closest landmark by video-time + if not landmarks: + errors.append(f"cue {cue_time:.2f}s ({cue_desc}): no landmarks at all") + continue + closest = min(landmarks, key=lambda lm: abs(lm.video_time - cue_time)) + delta = closest.video_time - cue_time + if abs(delta) > tolerance: + errors.append( + f"cue {cue_time:.2f}s ({cue_desc}) has no visual within ±{tolerance:.1f}s — " + f"closest is {closest.kind} at {closest.video_time:.2f}s ({delta:+.2f}s off): {closest.text[:40]}" + ) + + # Overflow check + cursor = sum(step_duration(s, fps) for s in steps) + end_vt = scene_start + cursor + scene_duration = scene_end - scene_start + if cursor > scene_duration + 0.5: + errors.append( + f"steps overflow scene: cursor ends at {end_vt:.2f}s but scene_end is {scene_end:.2f}s " + f"(overflow {cursor - scene_duration:.2f}s)" + ) + if cursor < scene_duration - 5.0: + errors.append( + f"steps underfill scene by {scene_duration - cursor:.2f}s — last visible step holds " + f"frozen from {end_vt:.2f}s to {scene_end:.2f}s. Add a closer pause." + ) + + if errors: + raise AssertionError( + "Scene pacing check failed:\n - " + "\n - ".join(errors) + ) + + +__all__ = ["step_duration", "trace", "assert_alignment", "Landmark"] diff --git a/pipeline_defs/screen-demo.yaml b/pipeline_defs/screen-demo.yaml index 75487e7..7af4aeb 100644 --- a/pipeline_defs/screen-demo.yaml +++ b/pipeline_defs/screen-demo.yaml @@ -1,14 +1,45 @@ name: screen-demo -version: "2.0" +version: "2.1" description: > - Screen recording pipeline. Takes raw screen capture (app demo, browser walkthrough, - coding tutorial) and produces a polished demo video with callouts, zoom crops, - subtitles, and cleaned audio. EP orchestration adds quality gates for legibility, - audio clarity, and pacing. + Screen recording pipeline. Two production modes: + 1. REAL CAPTURE — takes raw screen capture (app demo, browser walkthrough, + coding tutorial) via screen_recorder, cap_recorder, or playwright-recording + and produces a polished demo video with callouts, zoom crops, subtitles, + and cleaned audio. + 2. SYNTHETIC (Remotion TerminalScene) — for CLI/terminal/install-flow demos, + renders a deterministic terminal animation with typed commands, scrolling + output, floating command pills, and blinking cursor. See + .agents/skills/synthetic-screen-recording/SKILL.md. Preferred when the + demo content is predictable (install walkthroughs, make targets, git clone, + API key config) because it's faster to iterate, privacy-safe, pixel-perfect, + and frame-accurate to narration cues. + The idea-director picks the mode at brief time based on whether the demo surface + is a real app UI (real capture) or a scriptable terminal session (synthetic). category: screen_recording stability: production default_checkpoint_policy: guided +# Production modes this pipeline supports +production_modes: + - name: real_capture + description: "OS screen recording of a real desktop/browser session" + required_tools: [screen_recorder] + optional_tools: [cap_recorder] + best_for: + - Real app UIs (Figma, Photoshop, web apps with live state) + - Demos that depend on unpredictable live behavior + - User explicitly asked for a recording of their own screen + - name: synthetic_terminal + description: "Remotion TerminalScene — synthesized terminal animation" + required_tools: [video_compose] # Remotion path + scene_type: terminal_scene + agent_skills: [synthetic-screen-recording] + best_for: + - CLI / terminal / coding session demos + - Install walkthroughs, setup demos, config flows + - Tight narration-synced pacing where every command lands on a beat + - Reproducible renders (same input = identical pixels) + orchestration: mode: executive-producer skill: pipelines/screen-demo/executive-producer diff --git a/remotion-composer/SCENE_TYPES.md b/remotion-composer/SCENE_TYPES.md new file mode 100644 index 0000000..51b3563 --- /dev/null +++ b/remotion-composer/SCENE_TYPES.md @@ -0,0 +1,64 @@ +# Remotion Composer — Scene & Overlay Cheat Sheet + +Authoritative list of `cut.type` and `overlay.type` values the `Explainer` composition accepts. Each row maps to a dispatch case in `src/Explainer.tsx`. + +When you add a new component, append it here and in `src/components/index.ts`. + +--- + +## Cut types (`cut.type`) + +| `type` | Component | Required fields | Common fields | Purpose | +|---|---|---|---|---| +| *(none — video)* | `OffthreadVideo` | `source` (path to mp4) | `source_in_seconds`, `animation` (zoom-in, ken-burns), `in_seconds`, `out_seconds` | Play an MP4 clip directly | +| *(none — image)* | `Img` | `source` (path to png/jpg) | `animation`, `in_seconds`, `out_seconds` | Play a still with Ken Burns | +| `text_card` | `TextCard` | `text` | `fontSize`, `backgroundVideo`, `backgroundOverlay`, `color` | Large-typography beat | +| `hero_title` | `HeroTitle` | `text` | `heroSubtitle`, `backgroundVideo`, `backgroundOverlay` | Title/end card | +| `stat_card` | `StatCard` | `stat` | `subtitle`, `accentColor`, `backgroundVideo` | A single big number | +| `callout` | `CalloutBox` | `text` | `callout_type` (info/warning/tip/quote), `title`, `backgroundVideo` | Boxed message with bullets | +| `comparison` | `ComparisonCard` | `leftLabel`, `leftValue`, `rightLabel`, `rightValue` | `title`, `backgroundColor` | Side-by-side compare | +| `bar_chart` | `BarChart` | `chartData` | `chartAnimation`, `showValues`, `showGrid`, `backgroundVideo` | Animated bars | +| `line_chart` | `LineChart` | `chartSeries` | `chartAnimation`, `xLabel`, `yLabel`, `showMarkers` | Animated line | +| `pie_chart` | `PieChart` | `chartData` | `donut`, `centerLabel`, `centerValue`, `showLegend` | Pie / donut | +| `kpi_grid` | `KPIGrid` | `chartData` | `title`, `columns`, `chartAnimation` | 2–4 column KPI grid | +| `progress_bar` | `ProgressBar` | `progress` | `progressLabel`, `progressColor`, `progressSegments` | Animated progress | +| `anime_scene` | `AnimeScene` | `images` (list) | `particles`, `lightingFrom`, `lightingTo`, `vignette` | Still-image anime scene with particles + camera motion | +| **`terminal_scene`** | **`TerminalScene`** | **`steps`** (list of cmd/out/pause/pill) | **`terminalTitle`, `prompt`, `accentColor`** | **Synthetic terminal animation — NO real capture needed. See [`.agents/skills/synthetic-screen-recording/SKILL.md`](../.agents/skills/synthetic-screen-recording/SKILL.md)** | + +--- + +## Overlay types (`overlay.type`) + +| `type` | Component | Required fields | Common fields | Purpose | +|---|---|---|---|---| +| `section_title` | `SectionTitle` | `text` | `accentColor`, `position` (top-left, etc.) | Tiny section label | +| `stat_reveal` | `StatReveal` | `text` | `subtitle`, `accentColor`, `position` | Corner stat badge | +| `hero_title` | `HeroTitle` (as overlay) | `text` | `subtitle` | Full-frame title overlay | +| **`provider_chip`** | **`ProviderChip`** | **`providers`** (list of strings) | **`cycleSeconds`, `position`, `accentColor`, `label`** | **Rotating badge that cycles through provider names — used in AI-generated-motion scenes to show which model produced the clip** | + +--- + +## Adding a new scene type + +1. Create the React component in `src/components/MyScene.tsx`. Use `interpolate(frame, [inFrame, outFrame], [from, to])` and `spring(...)` for motion. Read `useCurrentFrame()` and `useVideoConfig()`. +2. Export it in `src/components/index.ts`. +3. Add the `type` to the `Cut` interface in `src/Explainer.tsx` (and any new prop fields). +4. Add a dispatch case in `SceneRenderer`: + ```tsx + if (cut.type === "my_scene" && cut.mySceneData) { + return maybeWrapWithBg(); + } + ``` +5. Document it in this file. That's what makes it discoverable to the next agent. + +## Existing synthetic-UI components + +Currently only `TerminalScene` exists. The pattern generalizes — likely candidates to add next, if a pipeline needs them: + +- `ChatTranscript` — Claude/Cursor/GPT chat-bubble timeline with typing animation +- `EditorScene` — VS Code-style code editor with syntax highlight + cursor motion +- `PrReview` — GitHub PR diff view with inline-comment reveals +- `SlackThread` — Slack thread with avatars + reaction pops +- `TicketBoard` — Jira / Linear card moving across columns + +Pattern: follow `TerminalScene.tsx` — a `steps` list of timeline primitives, cursor-advancing durations, spring-based reveals, optional non-blocking pills/badges. diff --git a/remotion-composer/src/Explainer.tsx b/remotion-composer/src/Explainer.tsx index 1da4be7..d4e6532 100644 --- a/remotion-composer/src/Explainer.tsx +++ b/remotion-composer/src/Explainer.tsx @@ -41,6 +41,9 @@ import { StatReveal } from "./components/StatReveal"; import { HeroTitle } from "./components/HeroTitle"; import { AnimeScene } from "./components/AnimeScene"; import type { CameraMotion } from "./components/AnimeScene"; +import { TerminalScene } from "./components/TerminalScene"; +import type { TerminalStep } from "./components/TerminalScene"; +import { ProviderChip } from "./components/ProviderChip"; import type { ParticleType } from "./components/ParticleOverlay"; import { resolveTheme, type ThemeConfig, DEFAULT_THEME } from "./Root"; @@ -197,6 +200,9 @@ interface Cut { subtitle?: string; callout_type?: "info" | "warning" | "tip" | "quote"; title?: string; + // Video source trim — seek to this point in the source before playback. + // Defaults to 0 (play from beginning). Use this instead of in_seconds for source trimming. + source_in_seconds?: number; // Comparison props leftLabel?: string; rightLabel?: string; @@ -228,7 +234,9 @@ interface Cut { // Styling overrides backgroundColor?: string; backgroundImage?: string; // AI-generated or stock image rendered behind the component - backgroundOverlay?: number; // Opacity of dark overlay on backgroundImage (0-1, default 0.55) + backgroundVideo?: string; // Video clip rendered behind the component (takes priority over backgroundImage) + backgroundVideoStart?: number; // Seek position in seconds for background video (default 0) + backgroundOverlay?: number; // Opacity of dark overlay on backgroundImage/backgroundVideo (0-1, default 0.55) color?: string; accentColor?: string; fontSize?: number; @@ -250,16 +258,24 @@ interface Cut { vignette?: boolean; lightingFrom?: string; lightingTo?: string; + // Terminal scene props (type: "terminal_scene") + steps?: TerminalStep[]; + terminalTitle?: string; + prompt?: string; } interface Overlay { - type: "section_title" | "stat_reveal" | "hero_title"; + type: "section_title" | "stat_reveal" | "hero_title" | "provider_chip"; in_seconds: number; out_seconds: number; - text: string; + text?: string; subtitle?: string; accentColor?: string; position?: string; + // provider_chip + providers?: string[]; + cycleSeconds?: number; + label?: string; } interface AudioLayer { @@ -472,9 +488,54 @@ const BackgroundImageLayer: React.FC<{ ); }; +// Background video layer — plays a looping video behind component content with dark overlay +const BackgroundVideoLayer: React.FC<{ + src: string; + startFrom?: number; + overlayOpacity?: number; + children: React.ReactNode; +}> = ({ src, startFrom = 0, overlayOpacity = 0.55, children }) => { + const { fps } = useVideoConfig(); + + return ( + + {/* Background video */} + + {/* Dark overlay for readability */} + + {/* Component content on top */} + {children} + + ); +}; + const SceneRenderer: React.FC<{ cut: Cut; theme: ThemeConfig }> = ({ cut, theme }) => { - // Wrap component with background image if specified - const maybeWrapWithBgImage = (element: React.ReactElement) => { + // Wrap component with background video or image if specified + const maybeWrapWithBg = (element: React.ReactElement) => { + if (cut.backgroundVideo) { + return ( + + {element} + + ); + } if (cut.backgroundImage) { return ( = ({ cut, theme // Resolve the scene element based on cut type, then wrap with backgroundImage if set // Use transparent bg so the animated gradient background shows through // When no explicit backgroundColor on the cut, inherit from theme - const rawBg = cut.backgroundImage ? "transparent" : (cut.backgroundColor || theme.surfaceColor); + const rawBg = (cut.backgroundImage || cut.backgroundVideo) ? "transparent" : (cut.backgroundColor || theme.surfaceColor); const bgColor = (rawBg === theme.backgroundColor || rawBg === "#0F172A" || rawBg === "#0f172a") ? "transparent" : rawBg; const textColor = cut.color || theme.textColor; const accent = cut.accentColor || theme.accentColor; // Explicit component types — use theme-derived defaults for colors if (cut.type === "text_card" && cut.text) { - return maybeWrapWithBgImage( + return maybeWrapWithBg( ); } if (cut.type === "stat_card" && cut.stat) { - return maybeWrapWithBgImage( + return maybeWrapWithBg( ); } if (cut.type === "callout" && cut.text) { - return maybeWrapWithBgImage( + return maybeWrapWithBg( = ({ cut, theme ); } if (cut.type === "comparison" && cut.leftLabel && cut.rightLabel && cut.leftValue && cut.rightValue) { - return maybeWrapWithBgImage( + return maybeWrapWithBg( = ({ cut, theme ); } if (cut.type === "hero_title" && cut.text) { - return maybeWrapWithBgImage( + return maybeWrapWithBg( ); } + if (cut.type === "terminal_scene" && cut.steps) { + return maybeWrapWithBg( + + ); + } // --- Chart types — use theme.chartColors as default palette --- if (cut.type === "bar_chart" && cut.chartData) { - return maybeWrapWithBgImage( + return maybeWrapWithBg( = ({ cut, theme ); } if (cut.type === "line_chart" && cut.chartSeries) { - return maybeWrapWithBgImage( + return maybeWrapWithBg( = ({ cut, theme ); } if (cut.type === "pie_chart" && cut.chartData) { - return maybeWrapWithBgImage( + return maybeWrapWithBg( = ({ cut, theme ); } if (cut.type === "kpi_grid" && cut.chartData) { - return maybeWrapWithBgImage( + return maybeWrapWithBg( = ({ cut, theme ); } if (cut.type === "progress_bar" && cut.progress !== undefined) { - return maybeWrapWithBgImage( + return maybeWrapWithBg( = ({ cut, theme const animation = cut.animation || cut.transform?.animation; if (cut.source && isImage(cut.source)) { - return ; + return maybeWrapWithBg(); } if (cut.source && isVideo(cut.source)) { - return ; + return maybeWrapWithBg(); } // Final fallback — try as image if source exists, otherwise show text_card if (cut.source) { - return ; + return maybeWrapWithBg(); } // No source, no type — render as text card with cut id as fallback @@ -664,6 +736,17 @@ const OverlayRenderer: React.FC<{ overlay: Overlay }> = ({ overlay }) => { if (overlay.type === "hero_title") { return ; } + if (overlay.type === "provider_chip" && overlay.providers) { + return ( + + ); + } return null; }; diff --git a/remotion-composer/src/components/ProviderChip.tsx b/remotion-composer/src/components/ProviderChip.tsx new file mode 100644 index 0000000..fdc6937 --- /dev/null +++ b/remotion-composer/src/components/ProviderChip.tsx @@ -0,0 +1,101 @@ +import { AbsoluteFill, interpolate, spring, useCurrentFrame, useVideoConfig } from "remotion"; + +/** + * ProviderChip — rotating pill of AI video provider names that cycle through. + * Positioned in a corner over background video. + */ + +interface ProviderChipProps { + providers: string[]; + cycleSeconds?: number; + position?: "top-left" | "top-right" | "bottom-left" | "bottom-right"; + accentColor?: string; + label?: string; +} + +const POS_STYLES: Record = { + "top-left": { top: 48, left: 48 }, + "top-right": { top: 48, right: 48 }, + "bottom-left": { bottom: 96, left: 48 }, // avoid caption zone + "bottom-right": { bottom: 96, right: 48 }, +}; + +export const ProviderChip: React.FC = ({ + providers, + cycleSeconds = 2.5, + position = "bottom-right", + accentColor = "#22D3EE", + label = "generated with", +}) => { + const frame = useCurrentFrame(); + const { fps } = useVideoConfig(); + + const cycleFrames = Math.max(1, Math.round(cycleSeconds * fps)); + const idx = Math.floor(frame / cycleFrames) % providers.length; + const current = providers[idx]; + const framesIntoCycle = frame % cycleFrames; + + // Spring in on cycle start + const springIn = spring({ + frame: framesIntoCycle, + fps, + config: { damping: 14, stiffness: 200 }, + durationInFrames: Math.ceil(fps * 0.35), + }); + + // Fade out before cycle end + const fadeOut = + framesIntoCycle > cycleFrames - fps * 0.25 + ? interpolate(framesIntoCycle, [cycleFrames - fps * 0.25, cycleFrames], [1, 0], { extrapolateRight: "clamp" }) + : 1; + + const alpha = Math.min(springIn, fadeOut); + const translateY = interpolate(springIn, [0, 1], [12, 0]); + + return ( + +
+
+ {label} +
+
+ {current} +
+
+ + ); +}; diff --git a/remotion-composer/src/components/TerminalScene.tsx b/remotion-composer/src/components/TerminalScene.tsx new file mode 100644 index 0000000..96c8626 --- /dev/null +++ b/remotion-composer/src/components/TerminalScene.tsx @@ -0,0 +1,266 @@ +import { AbsoluteFill, interpolate, spring, useCurrentFrame, useVideoConfig } from "remotion"; + +/** + * TerminalScene — animated terminal with typed commands and scrolling output. + * + * Each "step" is either: + * { kind: "cmd", text: "git clone ...", typeSpeed?: number } — typed char-by-char with prompt + * { kind: "out", text: "cloning into 'OpenMontage'..." } — reveals instantly + * { kind: "pause", seconds: number } — silent dwell + * { kind: "pill", text: "Piper TTS installed", color?: string } — floating badge + * + * Steps execute in order at the specified durations. Terminal auto-scrolls when + * it fills up. + */ + +export type TerminalStep = + | { kind: "cmd"; text: string; typeSpeed?: number; holdSeconds?: number } + | { kind: "out"; text: string; holdSeconds?: number } + | { kind: "pause"; seconds: number } + | { kind: "pill"; text: string; color?: string; durationSeconds?: number }; + +interface TerminalSceneProps { + title?: string; + steps: TerminalStep[]; + prompt?: string; + accentColor?: string; + backgroundColor?: string; +} + +interface RenderedLine { + text: string; + isCmd: boolean; + startFrame: number; + endFrame: number; // frame at which typing completes +} + +interface RenderedPill { + text: string; + color: string; + startFrame: number; + endFrame: number; +} + +export const TerminalScene: React.FC = ({ + title = "Terminal", + steps, + prompt = "$", + accentColor = "#22D3EE", + backgroundColor = "#0B0F1A", +}) => { + const frame = useCurrentFrame(); + const { fps } = useVideoConfig(); + + // Lay out timing in frames + const lines: RenderedLine[] = []; + const pills: RenderedPill[] = []; + let cursorFrame = 0; + + for (const step of steps) { + if (step.kind === "cmd") { + const speed = step.typeSpeed ?? 0.035; // seconds per char + const typeFrames = Math.ceil(step.text.length * speed * fps); + const hold = Math.ceil((step.holdSeconds ?? 0.3) * fps); + lines.push({ + text: step.text, + isCmd: true, + startFrame: cursorFrame, + endFrame: cursorFrame + typeFrames, + }); + cursorFrame += typeFrames + hold; + } else if (step.kind === "out") { + const revealFrames = Math.max(2, Math.ceil(0.08 * fps)); + const hold = Math.ceil((step.holdSeconds ?? 0.15) * fps); + lines.push({ + text: step.text, + isCmd: false, + startFrame: cursorFrame, + endFrame: cursorFrame + revealFrames, + }); + cursorFrame += revealFrames + hold; + } else if (step.kind === "pause") { + cursorFrame += Math.ceil(step.seconds * fps); + } else if (step.kind === "pill") { + const dur = Math.ceil((step.durationSeconds ?? 2.2) * fps); + pills.push({ + text: step.text, + color: step.color ?? accentColor, + startFrame: cursorFrame, + endFrame: cursorFrame + dur, + }); + // pill is non-blocking — don't advance cursor + } + } + + // Only render lines that have started + const visibleLines = lines.filter(l => frame >= l.startFrame); + + // Auto-scroll: keep last N lines in view + const MAX_VISIBLE = 18; + const scrollStart = Math.max(0, visibleLines.length - MAX_VISIBLE); + const renderedLines = visibleLines.slice(scrollStart); + + // Cursor blinks on most recent command + const blinkPhase = Math.floor(frame / (fps * 0.55)) % 2 === 0; + + // Terminal window frame fade-in + const windowOpacity = spring({ frame, fps, config: { damping: 25, stiffness: 100 } }); + + return ( + +
+ {/* Title bar */} +
+
+
+
+
+ {title} +
+
+ + {/* Terminal body */} +
+ {renderedLines.map((line, idx) => { + if (line.isCmd) { + // Char-by-char typed command + const progress = interpolate( + frame, + [line.startFrame, line.endFrame], + [0, line.text.length], + { extrapolateRight: "clamp" } + ); + const typed = line.text.slice(0, Math.floor(progress)); + const isLatest = idx === renderedLines.length - 1; + const isActive = frame <= line.endFrame + fps * 0.2; + return ( +
+ {prompt} + {typed} + {isLatest && isActive && blinkPhase && ( + + )} +
+ ); + } else { + // Instant-reveal output line with fade-in + const alpha = interpolate( + frame, + [line.startFrame, line.endFrame], + [0, 1], + { extrapolateRight: "clamp" } + ); + return ( +
+ {line.text} +
+ ); + } + })} +
+ + {/* Floating command pills */} + {pills + .filter(p => frame >= p.startFrame && frame <= p.endFrame) + .map((pill, idx) => { + const lifeProgress = (frame - pill.startFrame) / Math.max(1, pill.endFrame - pill.startFrame); + // spring in (0 → 1), hold, spring out (0.8 → 1.0) + const inAlpha = spring({ + frame: frame - pill.startFrame, + fps, + config: { damping: 14, stiffness: 180 }, + durationInFrames: Math.ceil(fps * 0.35), + }); + const outAlpha = + lifeProgress > 0.82 + ? interpolate(lifeProgress, [0.82, 1], [1, 0], { extrapolateRight: "clamp" }) + : 1; + const alpha = Math.min(inAlpha, outAlpha); + const translateY = interpolate(inAlpha, [0, 1], [14, 0]); + return ( +
+ {pill.text} +
+ ); + })} +
+ + ); +}; diff --git a/remotion-composer/src/components/index.ts b/remotion-composer/src/components/index.ts index f42e070..86567a8 100644 --- a/remotion-composer/src/components/index.ts +++ b/remotion-composer/src/components/index.ts @@ -10,5 +10,8 @@ export { StatReveal } from "./StatReveal"; export { HeroTitle } from "./HeroTitle"; export { ParticleOverlay } from "./ParticleOverlay"; export { AnimeScene } from "./AnimeScene"; +export { TerminalScene } from "./TerminalScene"; +export { ProviderChip } from "./ProviderChip"; export type { ParticleType } from "./ParticleOverlay"; export type { CameraMotion, AnimeSceneProps } from "./AnimeScene"; +export type { TerminalStep } from "./TerminalScene"; diff --git a/skills/pipelines/screen-demo/asset-director.md b/skills/pipelines/screen-demo/asset-director.md index 28c4dc0..4f3df50 100644 --- a/skills/pipelines/screen-demo/asset-director.md +++ b/skills/pipelines/screen-demo/asset-director.md @@ -4,6 +4,23 @@ This stage produces the minimal but high-leverage assets that make a screen demo easier to follow: subtitles, audio cleanup, reusable overlays, masks, and optional light-weight support cards. +## Two production modes — pick before generating assets + +**Read the brief's `production_mode` field.** If `idea` didn't set one, decide here: + +| Mode | When | Asset production looks like | +|---|---|---| +| **`real_capture`** | Real app UI (browser, design tool, IDE with plugins); live behavior; user asked for their own screen recorded | Clean audio + subtitles + callout overlays (arrows, highlight masks) applied on top of the captured MP4 | +| **`synthetic_terminal`** | CLI, terminal, install flow, make targets, git/npm commands, `.env` config — anything scriptable | **No capture at all.** Author a `terminal_scene` cut for `video_compose` (Remotion). Commands type char-by-char, output scrolls, pills announce completions. See `.agents/skills/synthetic-screen-recording/SKILL.md`. | + +**Mode selection heuristic:** *"Can I predict every command and its output before shooting?"* If yes → synthetic. If no → real capture. + +For synthetic mode, the asset stage produces: +- **narration (tts_selector)** aligned to the exact video-time each command should type +- **a `steps` list** (cmd/out/pause/pill primitives) that paces with narration cues +- **a pacing verification** via `lib.verify_scene_pacing.assert_alignment(...)` — must pass before render +- **no** screen-recorder footage, no callout arrows, no zoom-crop regions (those are real-capture concepts) + ## Prerequisites | Layer | Resource | Purpose | diff --git a/skills/pipelines/screen-demo/idea-director.md b/skills/pipelines/screen-demo/idea-director.md index a251463..36ea929 100644 --- a/skills/pipelines/screen-demo/idea-director.md +++ b/skills/pipelines/screen-demo/idea-director.md @@ -2,9 +2,20 @@ ## When To Use -Use this pipeline when the source is already a screen recording: product walkthrough, software tutorial, coding demo, browser flow, or troubleshooting capture. +Use this pipeline whenever the deliverable is a screen-recording-style demo. There are **two production modes** — pick one in the brief: -Your job is to turn raw capture into a clear procedural video. The main deliverable at this stage is a schema-valid `brief`, with pipeline-specific detail stored in `brief.metadata`. +| Mode | Source material | Pick when | +|---|---|---| +| **`real_capture`** | An actual screen recording (MP4) captured via `screen_recorder`, `cap_recorder`, or `playwright-recording` | Real app UI, live behavior, browser flows, IDE plugins, user asked for their own screen | +| **`synthetic_terminal`** | None — nothing is captured. You author a `terminal_scene` cut for Remotion | CLI / terminal / install flow / make targets / git clone / API key config — anything scriptable where every command and output is predictable | + +**Decision question:** *"Can I predict every command and its output before shooting?"* If yes → synthetic. If no → real capture. + +**Record the mode in `brief.metadata.production_mode`.** The asset-director reads this field to choose between capture+overlay assets vs a `steps` list paced with narration. + +For `synthetic_terminal`, also read `.agents/skills/synthetic-screen-recording/SKILL.md` before proceeding — it encodes the pacing rule that killed an earlier showcase render (commands burned through in 40% of scene time, then terminal froze for the remaining 60%). + +Your job at this stage is to turn the user's request into a clear procedural video plan. The main deliverable is a schema-valid `brief`, with pipeline-specific detail stored in `brief.metadata`. ## Operating Principles