Add expressive TTS governance
This commit is contained in:
calesthio
@@ -70,7 +70,7 @@ Before generating anything:
|
||||
|
||||
Before batch-generating assets, produce one sample of each expensive asset type and present them to the user for approval:
|
||||
|
||||
1. **TTS sample**: Generate narration for the first script section only. Play it for the user. Confirm voice, pace, and tone are acceptable before generating the rest.
|
||||
1. **TTS sample**: Generate narration for `script.voice_performance.sample_section_id` when present; otherwise pick the section with the most demanding delivery. Play it for the user. Confirm voice, pace, pauses, emphasis, and tone are acceptable before generating the rest.
|
||||
2. **Image sample**: Generate one image for the most representative scene. Show it to the user. Confirm the style, quality, and prompt approach before batch-generating all images.
|
||||
3. **Music sample** (if using `music_gen`): Generate one short clip. Confirm mood and energy before committing.
|
||||
|
||||
@@ -85,13 +85,24 @@ This step typically costs $0.03–0.08 total and prevents $1–3 of wasted gener
|
||||
|
||||
For each script section:
|
||||
1. Extract the narration text
|
||||
2. Apply speaker directions from the script (pace, emphasis, emotion)
|
||||
3. Apply the playbook's `audio.voice_style`
|
||||
4. Generate using `tts_selector` — it auto-routes to the best available TTS provider based on user preference and availability. Check the registry's `best_for` fields to understand each provider's strengths.
|
||||
5. Verify the audio file exists and duration matches expected timing (±15%)
|
||||
2. Read `script.voice_performance` and section `delivery_cues`
|
||||
3. Use `delivery_cues.provider_text` when present; otherwise transform the section text with purposeful punctuation and break tags only when the selected provider supports them
|
||||
4. Apply speaker directions from the script (pace, emphasis, emotion)
|
||||
5. Apply the playbook's `audio.voice_style`
|
||||
6. Map cues to provider parameters:
|
||||
- OpenAI: `instructions` only with `model: "gpt-4o-mini-tts"`; use `response_format` for output format
|
||||
- Google TTS: `input_type: "ssml"` when using `<break>` tags, plus `speaking_rate` in `0.25..2.0` and `pitch` in `-20..20`
|
||||
- ElevenLabs: `stability`, `similarity_boost`, `style`, `speed`, and `use_speaker_boost`
|
||||
7. Generate using `tts_selector` — it auto-routes to the best available TTS provider based on user preference and availability. Check the registry's `best_for` fields to understand each provider's strengths.
|
||||
8. Record the applied `voice_performance` metadata on each narration asset
|
||||
9. Verify the audio file exists and duration matches expected timing (±15%)
|
||||
|
||||
**Pronunciation guide**: If the script contains technical terms, jargon, or names with non-obvious pronunciation, include a pronunciation map in the TTS request.
|
||||
|
||||
**Flat voice failure:** If the approved voice sounds monotone, robotic, rushed,
|
||||
or ignores intended pauses, do not batch the remaining sections. Revise the
|
||||
`voice_performance` plan or provider parameters and regenerate the sample.
|
||||
|
||||
### Step 4: Generate Visual Assets
|
||||
|
||||
Process asset tasks grouped by tool for efficiency:
|
||||
@@ -204,6 +215,8 @@ Assemble all generated assets into the manifest:
|
||||
|
||||
**Quality check:**
|
||||
- [ ] Narration durations within ±15% of expected timing
|
||||
- [ ] Narration assets record `voice_performance.delivery_cues_applied`
|
||||
- [ ] Approved TTS sample uses the same provider, voice, and expressive settings as the batch
|
||||
- [ ] Images match the playbook's style (review consistency anchors)
|
||||
- [ ] Diagrams are legible and complete
|
||||
- [ ] Total cost within budget
|
||||
@@ -242,6 +255,7 @@ the AI model's training data — it may be wrong or outdated.
|
||||
- **Generating before checking budget**: Always estimate total cost first. A 60-second video with 15 images can burn $3+ quickly.
|
||||
- **Inconsistent image style**: Each image_selector call is independent. Use consistent anchors, but adapt them per scene. If you paste the same style prefix into every prompt, the video will feel machine-made and repetitive.
|
||||
- **Ignoring narration timing**: If TTS produces 12s of audio for a 10s section, the edit phase will struggle. Check durations.
|
||||
- **Ignoring delivery cues**: Generating raw script text when `provider_text` or `delivery_cues` exist will flatten the read. Apply the voice-performance contract first.
|
||||
- **Missing pronunciation guide**: "PostgreSQL" or "Kubernetes" will be mispronounced without explicit guidance.
|
||||
- **One retry then give up**: If an image doesn't match, refine the prompt specifically — don't just retry the same prompt.
|
||||
- **AI-generating images with exact text (CTA, business names, contact info)**: AI image models frequently hallucinate wrong text — wrong business name, wrong phone number, misspelled words. **Never use AI image generation for scenes where text must be verbatim.** Use Remotion `text_card` type instead. This applies to: CTA screens, title cards with business names, contact info overlays, legal disclaimers. If a scene's `type` is `text_card` in the scene plan, do NOT generate an image for it — skip it and let the compose stage render it natively in Remotion.
|
||||
|
||||
@@ -14,6 +14,7 @@ The script is the backbone of the video. Every visual, every scene, every audio
|
||||
| Prior artifact | `proposal_packet` | Selected concept with title, hook, key_points, core_message, tone, narrative_structure, duration |
|
||||
| Prior artifact | `research_brief` (optional but high-value) | Data points, audience insights, expert quotes — ground the script in real facts |
|
||||
| Playbook | Active style playbook from `proposal_packet.selected_concept.suggested_playbook` | Voice style, pacing rules |
|
||||
| Meta skill | `skills/meta/voice-performance-director.md` | Structured TTS delivery cues for natural, expressive narration |
|
||||
| Layer 3 | TTS provider skills (check `agent_skills` on the selected TTS tool) | TTS capabilities for speaker directions |
|
||||
|
||||
## Process
|
||||
@@ -77,6 +78,11 @@ Map each of the brief's `key_points` to a specific section in the BUILD phase.
|
||||
|
||||
### Step 4: Write the Script
|
||||
|
||||
Before writing sections, create a top-level `voice_performance` plan using
|
||||
`skills/meta/voice-performance-director.md`. The plan must describe the vocal
|
||||
intent, pacing profile, energy curve, pause policy, and which section should be
|
||||
used for TTS sample approval. Do not leave this as a vague "natural voice" note.
|
||||
|
||||
Write each section with these fields:
|
||||
|
||||
```json
|
||||
@@ -87,6 +93,14 @@ Write each section with these fields:
|
||||
"start_seconds": 0,
|
||||
"end_seconds": 5,
|
||||
"speaker_directions": "Emphasize 'every single row' with measured pacing. Brief pause before the question.",
|
||||
"delivery_cues": {
|
||||
"pace": "measured",
|
||||
"energy": "curious",
|
||||
"emphasis_words": ["every", "single"],
|
||||
"pause_after_seconds": 0.6,
|
||||
"delivery_note": "Let the repetition feel intentional, then soften into the question.",
|
||||
"provider_text": "Your database searches every single row. Every. Single. One. <break time=\"0.6s\"/> What if it didn't have to?"
|
||||
},
|
||||
"enhancement_cues": [
|
||||
{
|
||||
"type": "animation",
|
||||
@@ -117,7 +131,8 @@ Count your words. If you're 20%+ over budget, the TTS will either rush or exceed
|
||||
|
||||
#### Speaker Directions
|
||||
|
||||
Write directions that TTS can actually implement. Reference ElevenLabs capabilities:
|
||||
Write directions that TTS can actually implement. Prefer structured
|
||||
`delivery_cues` over prose-only `speaker_directions`:
|
||||
|
||||
| Direction | TTS Implementation |
|
||||
|-----------|-------------------|
|
||||
@@ -129,6 +144,12 @@ Write directions that TTS can actually implement. Reference ElevenLabs capabilit
|
||||
|
||||
Avoid directions TTS can't do: "smile while speaking", "gesture toward screen", "look at camera."
|
||||
|
||||
**Expressive narration rule:** every narration-led section must include at
|
||||
least two concrete cues among `pace`, `energy`, `emphasis_words`,
|
||||
`pause_before_seconds`, `pause_after_seconds`, `delivery_note`, or
|
||||
`provider_text`. Use `provider_text` when punctuation or SSML break tags are
|
||||
needed to make the read sound human.
|
||||
|
||||
#### Enhancement Cues
|
||||
|
||||
Every section should have at least one enhancement cue. These tell the Scene Planner and Asset Generator what visuals to create.
|
||||
@@ -162,6 +183,7 @@ Read the active style playbook and verify:
|
||||
|----------------|---------------|
|
||||
| `identity.pace` | Match word density. `contemplative` = fewer words, longer pauses |
|
||||
| `audio.voice_style` | Shape tone of speaker directions |
|
||||
| `voice_performance` | Confirm pacing, pauses, and energy curve are explicit enough for TTS |
|
||||
| `motion.pacing_rules` | E.g., "hold establishing shots for 2s minimum" affects section timing |
|
||||
| `identity.mood` | Word choice: `warm` uses casual language; `professional` uses precise language |
|
||||
|
||||
@@ -175,6 +197,7 @@ Score your script (1-5):
|
||||
| **Word count accuracy** | Within ±10% of target for the duration? |
|
||||
| **Narrative flow** | Does each section build on the last? "Therefore/but" not "and then"? |
|
||||
| **Enhancement density** | At least one cue every 8-10 seconds? |
|
||||
| **Voice performance** | Are pauses, emphasis, pace, and sample section explicit? |
|
||||
| **Jargon management** | Technical terms explained or have pronunciation guides? |
|
||||
| **Climax payoff** | Does the aha moment deliver on the hook's promise? |
|
||||
| **CTA relevance** | Is the call to action specific and actionable? |
|
||||
|
||||
Reference in New Issue
Block a user