screen-demo: add synthetic-terminal mode via Remotion TerminalScene
Make synthetic screen recording a first-class, discoverable capability alongside real OS capture. For CLI / terminal / install-flow demos where commands and output are predictable, author a `terminal_scene` cut instead of driving a real screen recorder — deterministic, privacy-safe, pixel- perfect, and frame-accurate to narration cues. Components: - TerminalScene.tsx: window chrome, char-by-char typing, blinking cursor, scrolling output, non-blocking floating pills, spring-based reveals - ProviderChip.tsx: rotating badge overlay that cycles through provider names (used in AI-generated-motion scenes) - BackgroundVideoLayer + source_in_seconds + backgroundVideo props on every scene type — supports video-behind-component composition Discovery chain (six layers, so the next agent finds this without reading source code): 1. pipeline_defs/screen-demo.yaml — bump to 2.1, declare production_modes (real_capture, synthetic_terminal) with required_tools, scene_type, and agent_skills pointers 2. skills/pipelines/screen-demo/idea-director.md — mode-selection table at brief time; brief.metadata.production_mode contract 3. skills/pipelines/screen-demo/asset-director.md — reads production_mode and branches asset production (capture+overlays vs steps+narration+ pacing check) 4. .agents/skills/synthetic-screen-recording/SKILL.md — Layer 3 skill with step kinds (cmd/out/pause/pill), pacing rule, and the frozen-terminal failure mode captured from the showcase v3 retune 5. AGENT_GUIDE.md — TerminalScene added to Remotion routing; links SCENE_TYPES.md as the authoritative cut-type registry 6. remotion-composer/SCENE_TYPES.md — new cheat sheet of every cut.type and overlay.type with required fields, plus a "how to add a new scene type" section (candidates: ChatTranscript, EditorScene, PrReview, SlackThread, TicketBoard) Guardrail: - lib/verify_scene_pacing.py — reusable trace() and assert_alignment() helpers that mimic the TerminalScene frame math exactly. Fail loudly before render if steps burn through too fast or leave the scene frozen.
This commit is contained in:
calesthio
@@ -0,0 +1,176 @@
|
||||
# Synthetic Screen Recording (Remotion TerminalScene)
|
||||
|
||||
**Decision this skill answers:** When the user wants a screen-recording-looking demo of a terminal, CLI tool, or coding workflow — do I **capture the real desktop** (OS screen recording via `screen_recorder`, Windows-MCP, Cap, or Playwright), or do I **synthesize it in Remotion** with the `TerminalScene` component?
|
||||
|
||||
> **Heuristic:** If the agent can author the exact command/output sequence in advance, synthesize. Only capture live when the real behavior is unpredictable, needs a real app UI, or the user explicitly asked for a real recording.
|
||||
|
||||
## Why this exists
|
||||
|
||||
v3 of the OpenMontage showcase tried to use Windows-MCP + `screen_recorder` to drive a Git-Bash window for the install walkthrough. It stalled on window positioning, focus races, and taskbar privacy concerns. We pivoted to **pure Remotion rendering** — a React component named `TerminalScene` that draws a fake terminal and types commands character-by-character. The output is visually indistinguishable from a real screen recording (same traffic-light window chrome, blinking cursor, scrolling output) but deterministic, privacy-safe, pixel-perfect at 1080p, and pace-controllable to the frame.
|
||||
|
||||
That component + pattern is the capability this skill makes discoverable.
|
||||
|
||||
## When to use synthetic (TerminalScene)
|
||||
|
||||
**YES, synthesize when:**
|
||||
- The demo is a **terminal / CLI / coding session** where commands and outputs are predictable
|
||||
- The user wants a polished tutorial feel (clean typography, floating pills, cursor blink)
|
||||
- Install walkthroughs, setup demos, API key config, `make` targets, `git clone` flows
|
||||
- You need tight sync with narration — every command must land on a specific beat
|
||||
- You want the result reproducible (re-render gets identical pixels)
|
||||
- The user's actual desktop has private apps/windows visible you'd otherwise have to crop
|
||||
|
||||
**NO, capture a real screen when:**
|
||||
- The demo is **a real app UI** that can't be faked (Figma, Photoshop, a web app with live state, a browser flow)
|
||||
- The user explicitly asked for a recording of *their* actual screen
|
||||
- The behavior depends on timing you can't script (streaming LLM output, real network latency)
|
||||
- There's a visual quirk (a cursor effect, a plugin pop-up) that only appears in the live environment
|
||||
|
||||
**For a browser demo** → `playwright-recording` skill, not this one.
|
||||
**For a real desktop** → `screen_recorder` tool or Cap via `cap_recorder`.
|
||||
|
||||
## The component — `TerminalScene`
|
||||
|
||||
Located at: `remotion-composer/src/components/TerminalScene.tsx`
|
||||
Exported from: `remotion-composer/src/components/index.ts`
|
||||
Wired in dispatch: `remotion-composer/src/Explainer.tsx` (`if (cut.type === "terminal_scene")`)
|
||||
|
||||
**Props:**
|
||||
```ts
|
||||
interface TerminalSceneProps {
|
||||
title?: string; // shown in the window title bar
|
||||
steps: TerminalStep[]; // the timeline
|
||||
prompt?: string; // "$", ">", etc.
|
||||
accentColor?: string; // pill + prompt glow
|
||||
backgroundColor?: string;
|
||||
}
|
||||
```
|
||||
|
||||
**Step kinds:**
|
||||
```ts
|
||||
{ kind: "cmd", text: string, typeSpeed?: number, holdSeconds?: number }
|
||||
{ kind: "out", text: string, holdSeconds?: number }
|
||||
{ kind: "pause", seconds: number }
|
||||
{ kind: "pill", text: string, color?: string, durationSeconds?: number }
|
||||
```
|
||||
|
||||
- `cmd` — prints the prompt, types the text character-by-character (`typeSpeed` is seconds per character, default 0.035), then holds for `holdSeconds` (default 0.3)
|
||||
- `out` — a line of program output, reveals instantly with a short fade-in
|
||||
- `pause` — dead time. Terminal holds on last visible state. USE THIS TO SYNC WITH NARRATION.
|
||||
- `pill` — non-blocking floating badge (top-right). Spring-in, hold, spring-out. Does NOT advance the cursor — the next step runs in parallel.
|
||||
|
||||
## Authoring pattern
|
||||
|
||||
Author a new scene by adding a cut to `build_composition.py` (or your equivalent props builder):
|
||||
|
||||
```python
|
||||
install_steps = [
|
||||
{"kind": "pause", "seconds": 7.0}, # wait for intro narration
|
||||
{"kind": "cmd", "text": "git clone https://github.com/calesthio/OpenMontage.git",
|
||||
"typeSpeed": 0.045, "holdSeconds": 0.3},
|
||||
{"kind": "out", "text": "Cloning into 'OpenMontage'..."},
|
||||
{"kind": "out", "text": "remote: Enumerating objects: 2847, done."},
|
||||
{"kind": "pill", "text": "repo cloned", "color": "#34D399", "durationSeconds": 2.6},
|
||||
{"kind": "pause", "seconds": 3.8}, # bridge to next narration cue
|
||||
# ...
|
||||
]
|
||||
|
||||
cuts.append({
|
||||
"id": "install-terminal",
|
||||
"type": "terminal_scene",
|
||||
"terminalTitle": "bash — OpenMontage setup",
|
||||
"prompt": "$",
|
||||
"accentColor": "#22D3EE",
|
||||
"steps": install_steps,
|
||||
"in_seconds": 50.0,
|
||||
"out_seconds": 110.0,
|
||||
})
|
||||
```
|
||||
|
||||
## THE RULE: pace with narration, never ahead
|
||||
|
||||
**The #1 failure mode:** steps run continuously and burn through all content in the first 40% of the scene, leaving the terminal frozen for the remaining 60%. This is what killed the v3 first pass — the capability menu rendered at t=80s but narration didn't announce it until t=92s.
|
||||
|
||||
**Do this instead:**
|
||||
|
||||
1. **Know your narration cues** — for each scene, write down the exact video-time each narration segment starts.
|
||||
2. **Start with a pause** that reaches the first narration cue before any command types.
|
||||
3. **Time each command to land with its narration line** — `cmd` should start typing the moment narration says its line, not before.
|
||||
4. **Put pauses between command groups** that bridge to the next narration cue.
|
||||
5. **End with a closer hold** — a pause long enough that the final state is readable after narration ends.
|
||||
|
||||
**Sanity-check your steps before rendering** — every minute of Remotion render is precious. Sum the step durations and verify they equal scene duration:
|
||||
|
||||
```python
|
||||
import math
|
||||
def trace(steps, scene_start, fps=30):
|
||||
t = 0.0
|
||||
for s in steps:
|
||||
k = s["kind"]
|
||||
if k == "cmd":
|
||||
tf = math.ceil(len(s["text"]) * s.get("typeSpeed", 0.035) * fps)
|
||||
t += tf / fps + s.get("holdSeconds", 0.3)
|
||||
elif k == "out":
|
||||
t += max(2, math.ceil(0.08 * fps)) / fps + s.get("holdSeconds", 0.15)
|
||||
elif k == "pause":
|
||||
t += s["seconds"]
|
||||
# "pill" is non-blocking — does NOT advance cursor
|
||||
print(f" {t + scene_start:6.2f}s {k}: {s.get('text', '')[:40]}")
|
||||
trace(install_steps, 50)
|
||||
```
|
||||
|
||||
Look at the output column. Each narration cue's video-time must appear adjacent to the command/output it announces. If a command lands 10s before or after its cue, adjust pauses.
|
||||
|
||||
See `lib/verify_scene_pacing.py` for a reusable version of this script.
|
||||
|
||||
## Design rules (inherited from the v3 retune)
|
||||
|
||||
- **Intro pause** — every terminal scene opens with at least 2s of empty-terminal-with-blinking-cursor before anything types. The viewer needs to register the window.
|
||||
- **Pill timing** — a pill should fire at the exact moment its named event completes on screen (e.g., `repo cloned` immediately after the last `Receiving objects` line). Pills are your substitute for real-world UI notifications.
|
||||
- **Command hold after typing** — keep `holdSeconds` ≥ 0.3 on every `cmd` so viewers register the completed command before the first output scrolls in.
|
||||
- **Output cadence** — space `holdSeconds` on output lines between 0.4 and 1.0. Output that flies too fast feels like a bug; output that crawls feels boring.
|
||||
- **Auto-scroll works** — the terminal holds the most recent 18 lines. Don't worry about off-screen content.
|
||||
- **Cursor blinks only on the latest command line while typing + a ~0.2s tail** after typing completes.
|
||||
|
||||
## `ProviderChip` (companion component)
|
||||
|
||||
The `.agents/skills/synthetic-screen-recording` pattern also owns `ProviderChip` — a rotating badge overlay that cycles through a list of provider names at a fixed cadence. Used in the v3 showcase to cycle through all 11 AI video-gen providers during the "generated motion" section.
|
||||
|
||||
```python
|
||||
overlays.append({
|
||||
"type": "provider_chip",
|
||||
"providers": ["Veo 3.1", "Seedance 2.0", "Kling 2.5", ...],
|
||||
"cycleSeconds": 2.5,
|
||||
"position": "bottom-right",
|
||||
"accentColor": "#22D3EE",
|
||||
"label": "generated with",
|
||||
"in_seconds": 195.0,
|
||||
"out_seconds": 222.5,
|
||||
})
|
||||
```
|
||||
|
||||
Wired in dispatch at: `remotion-composer/src/Explainer.tsx` overlay renderer (`overlay.type === "provider_chip"`).
|
||||
|
||||
## Adding new synthetic-UI components
|
||||
|
||||
The pattern generalizes. When you need to fake another UI surface (Claude Code chat bubbles, a Jira ticket view, a GitHub PR diff, a Slack message, a VS Code status bar):
|
||||
|
||||
1. Copy `TerminalScene.tsx` as a template.
|
||||
2. Define a `steps` interface for the relevant timeline primitives.
|
||||
3. Render each step by interpolating `frame` against cumulative start/end times.
|
||||
4. Wire it into `Explainer.tsx`'s `SceneRenderer` dispatch with a new `cut.type`.
|
||||
5. Add the type to the `Cut` interface in `Explainer.tsx` and to `components/index.ts`.
|
||||
6. Add a section to this skill documenting it.
|
||||
7. Update `remotion-composer/SCENE_TYPES.md` with the new cut type.
|
||||
|
||||
## Related skills
|
||||
|
||||
- `.agents/skills/remotion` — general Remotion authoring (hooks, springs, sequences)
|
||||
- `.agents/skills/playwright-recording` — real browser-flow capture for web apps
|
||||
- `tools/capture/screen_recorder` — ffmpeg-based desktop capture
|
||||
- `tools/capture/cap_recorder` — Cap.so polished desktop capture
|
||||
- `skills/pipelines/screen-demo/asset-director.md` — chooses between synthetic and real for a screen-demo project
|
||||
|
||||
## Provenance
|
||||
|
||||
Introduced: OpenMontage showcase v3 render (2026-04-16). Original motivation: the v3 setup walkthrough section needed a 60-second install demo where every command aligned to Chirp 3 HD narration cues, and Windows-MCP-driven real capture was too flaky in practice. See `projects/openmontage-showcase/build_composition.py` for the reference implementation.
|
||||
Reference in New Issue
Block a user