Add reference video input analysis workflow
This commit is contained in:
calesthio
@@ -115,6 +115,40 @@ If a checkpoint exists with status `"awaiting_human"`:
|
||||
2. Present the checkpoint data for review
|
||||
3. Wait for approval before proceeding
|
||||
|
||||
### Sample Checkpoint (Reference-Driven Productions)
|
||||
|
||||
When a production is reference-driven (VideoAnalysisBrief exists), there is an
|
||||
additional checkpoint between proposal approval and full production:
|
||||
|
||||
| Stage | checkpoint_required | human_approval_default | Notes |
|
||||
|-------|--------------------|-----------------------|-------|
|
||||
| `sample` | true | true | Always requires human approval |
|
||||
|
||||
The sample checkpoint:
|
||||
1. Presents: rendered sample clip (10-15 seconds)
|
||||
2. Cost: sample cost vs. projected full-video cost
|
||||
3. Action: approve (→ proceed to script), revise (→ re-generate sample), abort
|
||||
|
||||
The sample checkpoint is NOT a pipeline stage — it's a sub-checkpoint within the
|
||||
proposal stage. It does not produce a canonical artifact. It produces a rendered
|
||||
preview clip stored at `projects/<name>/assets/sample/sample_v{N}.mp4`.
|
||||
|
||||
**Presentation format:**
|
||||
```
|
||||
## Sample Preview Ready
|
||||
|
||||
**Sample clip:** [path to sample_v1.mp4]
|
||||
- Duration: [X] seconds (hook + 1 middle scene)
|
||||
- Voice: [TTS provider + voice name]
|
||||
- Visuals: [description — AI images, Remotion animations, etc.]
|
||||
- Music: [source]
|
||||
|
||||
**Sample cost:** $[X.XX]
|
||||
**Projected full video cost:** $[X.XX]
|
||||
|
||||
Does this feel right? I can adjust: voice, visual style, pacing, music, colors.
|
||||
```
|
||||
|
||||
## Key Principles
|
||||
|
||||
1. **Always checkpoint completed work.** Even if `checkpoint_required: false`, consider checkpointing anyway if the stage took significant time or cost. Losing work is worse than an extra file on disk.
|
||||
|
||||
@@ -55,6 +55,26 @@ The intake_brief is passed as context to the research-director, not as a
|
||||
formal artifact. It exists to prevent the research stage from inventing
|
||||
intent that the user never expressed.
|
||||
|
||||
## Handling Reference Video Input
|
||||
|
||||
When the user provides a video URL or file as their starting point:
|
||||
|
||||
1. **Read the video-reference-analyst skill** (`skills/meta/video-reference-analyst.md`)
|
||||
and follow its protocol. Do not proceed with standard creative intake.
|
||||
|
||||
2. The VideoAnalysisBrief replaces the need for most intake questions — it provides
|
||||
tone, structure, pacing, audience signals, and style information directly from the
|
||||
reference.
|
||||
|
||||
3. The remaining intake questions are:
|
||||
- What topic/subject for YOUR version? (if different from reference)
|
||||
- How long?
|
||||
- Narration yes/no?
|
||||
- Budget ceiling?
|
||||
|
||||
4. Do NOT ask "what should it feel like?" — the reference video IS the answer to that
|
||||
question. Extract tone from the VideoAnalysisBrief instead.
|
||||
|
||||
## What NOT To Do
|
||||
|
||||
- Do not present a numbered survey. This is a conversation, not a form.
|
||||
|
||||
@@ -105,12 +105,23 @@ Based on the user's tier, present **3 ready-to-use prompts** they can copy right
|
||||
|
||||
> **Or:** "I recorded a founder update on my webcam — make it feel polished, confident, and premium without looking fake" *(Talking Head pipeline)*
|
||||
|
||||
**Reference-based prompts (all tiers):**
|
||||
|
||||
> **Have a video you love?** Paste a YouTube link and say "make me something like this"
|
||||
> — I'll analyze the style, pacing, and structure, then propose 2-3 creative variants
|
||||
> you can choose from. Works with YouTube, Shorts, Instagram Reels, and TikTok.
|
||||
> All analysis runs locally and free — no API keys needed.
|
||||
|
||||
> **Got your own footage?** Drop in a video file and say "I want to make a video using
|
||||
> this footage" — I'll transcribe it, detect scenes, and propose an edit plan.
|
||||
|
||||
**Rules for prompt suggestions:**
|
||||
- Present exactly 3 prompts.
|
||||
- The first prompt should be the most impressive thing their setup can produce.
|
||||
- Each prompt should target a different pipeline or style.
|
||||
- Include a brief note explaining what makes this prompt a good fit for their setup.
|
||||
- Use blockquote formatting so prompts are visually distinct and easy to copy.
|
||||
- Always include the reference-based prompts above — they work at every tier.
|
||||
|
||||
### Step 5: Explain the Workflow (Briefly)
|
||||
|
||||
|
||||
@@ -117,6 +117,39 @@ Structure your review as:
|
||||
| compose | Playability, duration accuracy, audio quality, pre-compose validation pass |
|
||||
| publish | SEO quality, metadata completeness, export packaging |
|
||||
|
||||
## Reference Alignment Review
|
||||
|
||||
Run at **every stage** when a VideoAnalysisBrief exists (reference-driven production).
|
||||
|
||||
### Checks:
|
||||
|
||||
1. **Grounding check:** Does the output reference specific findings from the
|
||||
VideoAnalysisBrief, or is it making things up about the reference?
|
||||
- Proposal mentions "fast pacing" but reference pacing_style is "slow_contemplative" → **CRITICAL**
|
||||
- Script claims reference has narration but VideoAnalysisBrief shows no narration → **CRITICAL**
|
||||
|
||||
2. **Differentiation check:** Does each concept/scene have a clear creative
|
||||
difference from the reference, or is it a copy?
|
||||
- Proposal is a carbon copy of the reference (same topic, same structure, same treatment) → **CRITICAL**
|
||||
- At least one element per concept MUST differ from the reference → **SUGGESTION** if weak
|
||||
- Creative differentiation seeds from the brief should be reflected in proposals
|
||||
|
||||
3. **Promise preservation:** Are the elements the user said they loved about the
|
||||
reference still present in the output?
|
||||
- User said "I love the pacing" but scene_plan has 2x longer scenes → **SUGGESTION**
|
||||
- User said "keep the hook style" but script uses a different hook → **SUGGESTION**
|
||||
|
||||
4. **Cost alignment:** Is the cost estimate still accurate, or has scope crept?
|
||||
- If actual spend exceeds estimate by >30% without user re-approval → **CRITICAL**
|
||||
- If new assets were added beyond the approved proposal → **SUGGESTION**
|
||||
|
||||
### Severity:
|
||||
- Factual errors about the reference video: **CRITICAL**
|
||||
- Carbon copy with no differentiation: **CRITICAL**
|
||||
- Weak differentiation (surface-level changes only): **SUGGESTION**
|
||||
- User preference not honored: **SUGGESTION**
|
||||
- Cost drift >30%: **CRITICAL**
|
||||
|
||||
## Slideshow Risk Review
|
||||
|
||||
Run at **scene_plan** and **edit** stages. Use `lib/slideshow_risk.py` to compute the score.
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
# Video Reference Analyst — Meta Skill
|
||||
|
||||
## When to Use
|
||||
|
||||
When the user provides a video URL (YouTube, Shorts, Instagram, TikTok, or any URL)
|
||||
or a local video file as a REFERENCE — meaning "make me something like this," not
|
||||
"edit this footage."
|
||||
|
||||
If the user says "edit this video" or "cut this into clips," route to the appropriate
|
||||
footage-led pipeline (clip-factory, talking-head, hybrid) instead. This skill is for
|
||||
REFERENCE-based production.
|
||||
|
||||
## Detection Signals
|
||||
|
||||
Trigger this skill when:
|
||||
- User pastes a YouTube/Shorts/Instagram/TikTok URL
|
||||
- User says "something like this," "inspired by," "in this style," "similar to"
|
||||
- User uploads a video and says "I want one like this"
|
||||
- User says "I saw this video and want to make something like it"
|
||||
|
||||
Do NOT trigger when:
|
||||
- User provides footage and says "edit this" or "cut this" → use source_media_review
|
||||
- User provides audio and says "make a video for this" → standard pipeline
|
||||
- User just wants a transcript → use TranscriptFetcher directly
|
||||
|
||||
## Protocol
|
||||
|
||||
### Step 1: Analyze the Reference
|
||||
|
||||
Run VideoAnalyzer with `analysis_depth: "standard"`:
|
||||
|
||||
```python
|
||||
video_analyzer.execute({
|
||||
"source": "<url or path>",
|
||||
"analysis_depth": "standard",
|
||||
"max_keyframes": 20
|
||||
})
|
||||
```
|
||||
|
||||
Read the resulting VideoAnalysisBrief. Before proceeding, present a summary to the
|
||||
user. This is NOT a raw dump. It's a conversational interpretation:
|
||||
|
||||
```
|
||||
"I've watched the video. Here's what I see:
|
||||
|
||||
**Content:** [2-sentence summary of what the video is about]
|
||||
**Style:** [1 sentence — pacing, visual treatment, energy]
|
||||
**Structure:** [X scenes over Y seconds, pacing style]
|
||||
**What makes it work:** [2-3 specific things — the hook technique, the pacing,
|
||||
the visual transitions, the narration style]
|
||||
|
||||
Now let me check what I can do with your current setup..."
|
||||
```
|
||||
|
||||
**Vision analysis:** After presenting the structural data, examine the extracted
|
||||
keyframes yourself. You ARE a multimodal model — look at the keyframe images and
|
||||
enrich the VideoAnalysisBrief with:
|
||||
- Per-frame descriptions (subjects, text, composition, color)
|
||||
- Cross-frame visual continuity and style consistency
|
||||
- Genre classification and production quality assessment
|
||||
- Color palette extraction (dominant colors across keyframes)
|
||||
- Typography style if on-screen text is present
|
||||
- Transition patterns visible between sequential keyframes
|
||||
|
||||
Update the brief's `content_analysis`, `style_profile`, and `replication_guidance`
|
||||
fields with your visual observations. This is where the analysis becomes truly
|
||||
comprehensive — the tools provide structure; your vision provides understanding.
|
||||
|
||||
### Step 2: Capability Audit
|
||||
|
||||
Run standard preflight:
|
||||
|
||||
```bash
|
||||
python -c "from tools.tool_registry import registry; import json; registry.discover(); print(json.dumps(registry.support_envelope(), indent=2))"
|
||||
python -c "from tools.tool_registry import registry; import json; registry.discover(); print(json.dumps(registry.provider_menu(), indent=2))"
|
||||
python -c "from tools.tool_registry import registry; import json; registry.discover(); print(json.dumps(registry.capability_catalog(), indent=2))"
|
||||
```
|
||||
|
||||
Map the reference video's requirements against available capabilities:
|
||||
|
||||
```
|
||||
REFERENCE NEEDS YOUR CAPABILITIES GAP
|
||||
───────────────────── ───────────────────── ──────────
|
||||
Video clips (sci-fi) Video gen: 0/12 configured BLOCKED without key
|
||||
Narration (deep male) TTS: ElevenLabs available READY
|
||||
Background music Music: MusicGen available READY
|
||||
Text animations Remotion: available READY
|
||||
Fast-cut editing FFmpeg: available READY
|
||||
```
|
||||
|
||||
Be honest about gaps. If video generation is needed but unavailable, say so clearly:
|
||||
|
||||
```
|
||||
"This reference uses generated sci-fi footage. Right now you don't have any video
|
||||
generation providers configured. Here are your options:
|
||||
|
||||
• Add FAL_KEY to .env → unlocks Kling 3.0, MiniMax, Wan (best for cinematic/sci-fi)
|
||||
• Add REPLICATE_API_TOKEN → unlocks LTX Video (good for short clips)
|
||||
• Proceed without video gen → I'll use stock footage + Remotion animations instead
|
||||
(different feel, but still works)
|
||||
|
||||
Which would you prefer?"
|
||||
```
|
||||
|
||||
Read install_instructions from the registry for each unavailable tool — do NOT
|
||||
hardcode key names or setup URLs.
|
||||
|
||||
### Step 3: Ask Critical Questions
|
||||
|
||||
Before proposing, gather what the VideoAnalysisBrief doesn't tell you:
|
||||
|
||||
1. "Do you want narration in your version, or visuals-only with music?"
|
||||
2. "How long should your video be? The reference is [X] seconds."
|
||||
3. "Is there a specific topic/subject you want, or should I riff on the
|
||||
same theme as the reference?"
|
||||
4. "Any elements from the reference you specifically love or hate?"
|
||||
|
||||
Do NOT ask all at once. Lead with the most important gap. If the user's initial
|
||||
message already answers some of these, skip those.
|
||||
|
||||
### Step 4: Creative Proposals (2-3 variants)
|
||||
|
||||
MANDATORY: The agent must NEVER propose a carbon copy. The reference is inspiration,
|
||||
not a template. Each proposal must have clear creative differentiation.
|
||||
|
||||
Use this structure for each variant:
|
||||
|
||||
```
|
||||
## Option [A/B/C]: "[Title]"
|
||||
|
||||
**Inspired by:** [what it keeps from the reference — pacing, structure, tone]
|
||||
**Creative twist:** [what it changes — angle, subject, visual treatment, hook]
|
||||
|
||||
**Visual plan:**
|
||||
- Playbook: [closest match + customizations]
|
||||
- Visual treatment: [how visuals will be created — which tools, which providers]
|
||||
- Motion: [Remotion animations / video gen clips / stock + Ken Burns / etc.]
|
||||
|
||||
**Audio plan:**
|
||||
- Narration: [yes/no, which TTS provider, voice style]
|
||||
- Music: [library track / generated / none]
|
||||
- Sound design: [any special audio needs]
|
||||
|
||||
**Duration:** [X seconds]
|
||||
**Estimated cost:** $[X.XX] breakdown:
|
||||
- Image generation: $X.XX (N images × $X.XX each via [provider])
|
||||
- Video generation: $X.XX (N clips × $X.XX each via [provider])
|
||||
- TTS narration: $X.XX (N words via [provider])
|
||||
- Music: $X.XX ([source])
|
||||
- Total: $X.XX
|
||||
|
||||
**Honest assessment:** [What this will look like realistically — don't oversell]
|
||||
```
|
||||
|
||||
**Differentiation patterns:**
|
||||
|
||||
| Pattern | Example |
|
||||
|---------|---------|
|
||||
| **Same structure, different subject** | Reference: "How black holes work" → Ours: "How neutron stars work" with same pacing |
|
||||
| **Same subject, different angle** | Reference: "Kubernetes explained" → Ours: "Kubernetes from a security engineer's POV" |
|
||||
| **Same tone, different visual treatment** | Reference: stock footage + voiceover → Ours: animated motion graphics + voiceover |
|
||||
| **Same content, different platform** | Reference: 10-min YouTube → Ours: 60-sec Shorts version with faster pacing |
|
||||
| **Counter-take** | Reference: "Why AI will replace jobs" → Ours: "Why AI won't replace YOUR job" |
|
||||
|
||||
**Cost transparency is mandatory.** Each concept must include:
|
||||
- Itemized cost estimate at the user's requested duration
|
||||
- Cost broken down by: image gen, video gen, TTS, music, total
|
||||
- Provider names for each cost line
|
||||
- Honest note about what the budget buys vs. doesn't buy
|
||||
|
||||
**Recommendation:** Always recommend one option with a brief reason why. Don't leave
|
||||
the user paralyzed with equal choices.
|
||||
|
||||
### Step 5: Sample-First Production (MANDATORY)
|
||||
|
||||
After the user picks a variant, ALWAYS say:
|
||||
|
||||
```
|
||||
"Great choice. Before I commit to the full [X]-second video, I'll produce a
|
||||
10-15 second sample first — the opening hook + one middle scene. This lets you
|
||||
hear the voice, see the visual style, and feel the pacing before we go all-in.
|
||||
|
||||
Estimated sample cost: $[X.XX]
|
||||
Shall I proceed with the sample?"
|
||||
```
|
||||
|
||||
The sample is NOT optional. Even if the user says "just do the whole thing," push
|
||||
back gently:
|
||||
|
||||
```
|
||||
"I'd really recommend the sample first — it's a tiny fraction of the cost and
|
||||
lets us catch any style mismatches early. If you love it, I'll proceed to the
|
||||
full video immediately."
|
||||
```
|
||||
|
||||
Only skip the sample if the user insists after being advised.
|
||||
|
||||
**Sample contents:**
|
||||
- 1-2 representative scenes (the hook + one middle scene)
|
||||
- Actual TTS narration with chosen voice
|
||||
- Actual generated/stock visuals
|
||||
- Music bed snippet
|
||||
- Subtitle style preview
|
||||
|
||||
**Sample checkpoint:**
|
||||
Present the sample with: "Here's a preview. Does this feel right? Things I can
|
||||
adjust: voice, visual style, pacing, music, colors."
|
||||
|
||||
Iterate on sample feedback until approved. Store samples at:
|
||||
`projects/<name>/assets/sample/sample_v{N}.mp4`
|
||||
|
||||
### Step 6: Enter Pipeline
|
||||
|
||||
After sample approval, enter the appropriate pipeline with:
|
||||
- VideoAnalysisBrief as grounding context in the research/proposal stage
|
||||
- User's chosen variant as the approved direction
|
||||
- Sample feedback incorporated into the brief
|
||||
- All creative differentiation decisions recorded in the decision_log
|
||||
|
||||
The pipeline takes over from here. The VideoAnalysisBrief travels alongside the
|
||||
standard artifacts, providing reference grounding at every stage.
|
||||
|
||||
## Multiple Reference Videos
|
||||
|
||||
When the user provides multiple reference URLs:
|
||||
|
||||
1. Analyze each video separately (run VideoAnalyzer on each)
|
||||
2. Present a comparative summary: "Video A does X well, Video B does Y well"
|
||||
3. In proposals, note which elements are inspired by which reference
|
||||
4. The VideoAnalysisBrief for the primary reference travels with the pipeline;
|
||||
secondary references are noted in the research_brief
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Failure | Action |
|
||||
|---------|--------|
|
||||
| URL download fails | Report error, suggest: try another URL, provide local file, or proceed without reference |
|
||||
| No captions available | Download video, transcribe with Whisper locally |
|
||||
| Scene detection fails | Fall back to uniform frame sampling |
|
||||
| All analysis fails | Ask user to describe the reference video verbally, proceed with standard creative intake |
|
||||
|
||||
Never silently skip analysis steps. If something fails, tell the user what happened
|
||||
and what the impact is on the analysis quality.
|
||||
Reference in New Issue
Block a user