Compare commits

...

10 Commits

Author SHA1 Message Date
Calesthio 299dd4f5fd Merge pull request #236 from calesthio/fix/skill-blob-hardlink-divergence
CI / Validate Python (push) Has been cancelled
fix: reconcile hardlinked .claude/skills ↔ .agents/skills blobs (clean dirty tree)
2026-06-29 23:05:46 -07:00
Calesthio 46ed550883 Merge pull request #229 from calesthio/codex/expressive-tts-governance
[codex] Add expressive TTS governance
2026-06-29 23:04:56 -07:00
calesthio 1918be6526 fix: reconcile .claude/skills blobs with hardlinked .agents/skills
The skill SKILL.md files are hardlinked across .agents/skills/ and
.claude/skills/ (same inode, two paths), but git had committed divergent
blobs: the .agents/ copies carried an "Extended reference: AGENTS.md"
note that the .claude/ copies lacked. Because one physical file cannot
match two different blobs, the working tree was permanently dirty on the
three .claude/skills paths.

Stage the .claude copies so both paths commit identical content (with the
AGENTS.md reference note, the fuller intended version). Restores a clean
git status and keeps it clean across checkouts.

Affected skills: flux-best-practices, vercel-composition-patterns,
vercel-react-best-practices.
2026-06-29 22:58:56 -07:00
Calesthio 5ae4439105 Merge pull request #230 from Alcb4/comfyui-bridge
Add native ComfyUI provider (rebase of #29, conflicts resolved)
2026-06-29 14:55:03 -07:00
Alastair Beal 85cf047c93 comfyui: route custom workflows through image/video selectors
The selector path previously hid the custom-workflow feature: video_selector
filtered tools on per-operation readiness (bundled WAN models) and both
selectors only chose ToolStatus.AVAILABLE providers, so comfyui_image/
comfyui_video — DEGRADED when bundled model metadata is missing — were
dropped even when the ComfyUI server was up and the caller supplied a full
workflow_json/workflow_path plus output_node.

- Add a custom-workflow readiness path to both selectors: when a custom
  workflow is supplied, eligibility is based on server availability (status
  != UNAVAILABLE) for any provider advertising supports.custom_workflow,
  not on bundled-model readiness. A custom workflow also restricts routing
  to custom-workflow-capable providers, since the graph JSON is ComfyUI
  specific.
- Expose workflow_json, workflow_path, output_node, workflow_name,
  workflow_model, and workflow_model_stack in both selector schemas so
  agents can discover the feature without bypassing the selectors.
- image_selector only forwards the workflow inputs to providers that
  declare them.
- Add contract tests for the new eligibility path and schema exposure.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 22:38:18 +01:00
calesthio 5e4943a41b Add expressive TTS governance 2026-06-29 12:16:07 -07:00
calesthio 7c4bb08890 comfyui: satisfy provider contract review items 2026-06-29 19:09:22 +01:00
martimramos 4c62186c95 comfyui: drop music tool — ACE-Step node interface not standardized
Removed comfyui_music and its workflow. The ACE-Step model runs in
ComfyUI but the node class names differ across custom node packs
(AceStepModelLoader vs native TextEncodeAceStepAudio, etc.), so a
bundled workflow would break for most users.

Documented the reasoning in the plan doc and listed it as an open
question for future work. Users with ACE-Step working can still use
the workflow_json override on any tool.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-29 19:09:22 +01:00
martimramos e3947e1f11 comfyui: add model discovery and actionable error messages
- Client queries ComfyUI /object_info to discover installed models
  (checkpoints, diffusion models, VAE, CLIP, LoRAs)
- Each tool declares its required models and checks them on execute()
- get_status() returns DEGRADED when server is up but models are missing
- Clear error messages tell the user exactly which models to download
- When COMFYUI_SERVER_URL is not set, error message tells the user to
  configure it in .env instead of silently failing on localhost:8188
- 8 new tests covering URL config, error messages, and model requirements

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-29 19:08:49 +01:00
martimramos 6ec2bbb090 comfyui: add native ComfyUI provider for image, video, and music generation
Adds three new BaseTool providers that delegate GPU work to a running
ComfyUI server via its REST API.  This avoids the need to install
PyTorch/diffusers directly, which is critical on hardware where the
ecosystem hasn't caught up (e.g. NVIDIA Blackwell / DGX Spark, aarch64
+ CUDA 13.0).

New files:
- tools/_comfyui/client.py — shared REST client (submit/poll/download)
- tools/_comfyui/workflows/ — 4 bundled workflow templates
- tools/graphics/comfyui_image.py — FLUX 2 Dev NVFP4 text-to-image
- tools/video/comfyui_video.py — WAN 2.2 14B t2v + i2v (4-step LightX2V)
- tools/audio/comfyui_music.py — ACE-Step 3.5B music generation
- tests/contracts/test_comfyui_tools.py — 41 contract tests
- docs/comfyui-adapter-plan.md — design document

Zero changes to existing tools, selectors, registry, or pipelines.
Tools are auto-discovered and selectors pick them up via capability match.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-29 19:08:49 +01:00
36 changed files with 3577 additions and 33 deletions
+55
View File
@@ -0,0 +1,55 @@
---
name: comfyui
description: Use when working with ComfyUI workflows in OpenMontage, including comfyui_image/comfyui_video, custom workflow_json/workflow_path inputs, output_node selection, missing model setup, LoRAs, low-VRAM workflow choices, and community workflow imports.
---
# ComfyUI Workflows in OpenMontage
Use this skill before calling `comfyui_image` or `comfyui_video`, and when converting a community ComfyUI workflow into an OpenMontage tool call.
## Server Contract
- ComfyUI must be running before the tool can generate. The default server is `http://localhost:8188`; override it with `COMFYUI_SERVER_URL`.
- Health and hardware status come from `GET /system_stats`.
- Jobs are submitted to `POST /prompt`, completed outputs are read from `GET /history/{prompt_id}`, and artifact bytes are downloaded with `GET /view`.
- Export workflows with ComfyUI's API-format JSON, not the UI layout format. If a downloaded workflow will not submit, re-export it from ComfyUI with API format enabled.
## Choosing a Workflow
- Use bundled workflows when the requested operation matches and the local machine has the required models and VRAM.
- Use a custom `workflow_json` or `workflow_path` when the user needs a community recipe, a lower-VRAM model, a different style family, or custom nodes.
- For 8GB-12GB GPUs, prefer lower-footprint workflows such as Wan 2.1 1.3B, LTXV FP8 or quantized workflows, or Wan 2.2 GGUF/quantized community workflows. The bundled Wan 2.2 14B FP8 video workflows are a 16GB-class path, not a provider-wide floor.
- Do not promise that arbitrary custom workflows will fit a machine. The workflow, quantization, resolution, frame count, and offload settings determine the real resource envelope.
## Output Node Contract
- Custom workflows must pass `output_node`.
- Pick the node that writes the artifact, usually `SaveImage`, `SaveVideo`, `VHS_VideoCombine`, or another terminal saver node.
- Pass the node ID as a string, for example `"108"`. Do not pass the class name.
- If a workflow has multiple savers, choose the final deliverable node, not previews or intermediates.
## Templated vs Fixed Nodes
- Identify templated nodes before execution: prompt text, seed, dimensions, frame count, source image, sampler settings, and output filename prefix.
- Fixed nodes are model loaders, VAEs, text encoders, LoRA loaders, schedulers, and graph wiring. Do not mutate those unless the workflow author intended that customization.
- For community workflows, inspect each loader node and note every required model or custom node before running. Missing models should be handled through the tool's structured `missing_models` payload when available.
## Model and LoRA Setup
- Use ComfyUI Manager or the workflow author's model links when available, and respect model licenses.
- Place models in the folders expected by the loader nodes: diffusion models under `ComfyUI/models/diffusion_models/`, text encoders under `ComfyUI/models/text_encoders/`, VAEs under `ComfyUI/models/vae/`, and LoRAs under `ComfyUI/models/loras/`.
- For LoRA stacks, use `LoraLoader` or `LoraLoaderModelOnly` chains in the workflow. Record each LoRA name plus `strength_model` and `strength_clip` when applicable.
- The current ComfyUI tools do not inject LoRAs into arbitrary graphs. To use LoRAs, provide a workflow that already contains the LoRA loader chain and pass model-stack provenance.
## Provenance
- For custom workflows, provide `workflow_name` and `workflow_model` when known.
- Provide `workflow_model_stack` for reproducibility when the workflow is not bundled. Include base checkpoint or diffusion model, quantization, text encoder, VAE, LoRAs and strengths, sampler or scheduler, steps, and guidance if the workflow exposes them.
- The tools record the final workflow hash. Treat that hash plus the model stack, seed, dimensions, and prompt as the reproducibility contract.
## Failure Handling
- If the server is unavailable, surface the structured setup offer. Starting ComfyUI or setting `COMFYUI_SERVER_URL` is the first fix.
- If models are missing, read `data.missing_models[]`; each item should include the file name, role, destination hint, and download URL when OpenMontage knows it.
- If custom nodes are missing, ask the user to install them through ComfyUI Manager or the workflow author's documented install path, then restart ComfyUI.
- If a long render times out locally, check ComfyUI history before retrying from scratch; the server may still have completed the prompt.
@@ -65,6 +65,27 @@ node skills/hyperframes-media/scripts/heygen-tts.mjs --list # public starfish
| Offline, no API key, fast iteration | **Kokoro** |
| Non-English multilingual with deterministic phonemization | **Kokoro** (`ef_dora`, `jf_alpha`, `zf_xiaobei`, …) |
## Expressive narration contract
Before generating narration, write a compact voice-performance plan:
- `performance_intent` - who the narrator is and how they should feel
- `pacing_profile` - contemplative, conversational, energetic, technical, or custom
- `energy_curve` - how the read changes across the piece
- `pause_policy` - where silence should happen and why
- section-level cues - `pace`, `energy`, `emphasis_words`, `pause_before_seconds`,
`pause_after_seconds`, and optional provider-ready text
Do not rely on a vague instruction like "make it natural." Put the direction in
the text or provider settings:
- Use short sentences and purposeful punctuation.
- Use `<break time="0.4s"/>` to `<break time="1.0s"/>` for important pauses when
the chosen provider supports SSML-style break tags.
- Generate a sample from the most performance-sensitive section before batching.
- If the sample sounds monotone, rushed, or ignores pauses, revise the plan or
provider settings before generating the rest.
## ffmpeg requirement
HeyGen + ElevenLabs return mp3. The CLI transcodes to wav when `--output` ends in `.wav` (the default and what downstream `ffprobe` + Whisper expect). If you'd rather skip the transcode, pass `-o file.mp3`. Without `ffmpeg` on PATH, `.wav` output from the cloud providers fails — install ffmpeg or use `.mp3`.
+19
View File
@@ -356,6 +356,25 @@ Rules:
- Must have spaces before and after the tag
- Self-closing tag format
## Expressive Voice Direction
For narration, create a short voice-performance plan before generating audio:
- narrator persona and emotional intent
- pacing profile
- energy curve across the script
- where pauses should land
- words or phrases that need emphasis
Use concrete cues, not generic instructions. "Warm but decisive; pause before
the contrast; slow down on the final sentence" is useful. "Sound natural" is
not.
When the selected voice supports pauses, put the most important pauses directly
in the text with break tags. Generate a sample from the most performance-heavy
section first, and do not batch-generate the rest if the sample sounds flat,
rushed, or ignores the intended breaks.
## Best Practices
1. **Use `GET /v1/audio/voices`** to find compatible voices — not all voices from `GET /v2/voices` support Starfish TTS
@@ -11,6 +11,8 @@ metadata:
Use this skill when generating prompts for any BFL FLUX model to ensure optimal image quality and accurate prompt interpretation.
> **Extended reference:** [`AGENTS.md`](AGENTS.md) in this directory is the long-form upstream FLUX guide (vendored from Black Forest Labs). It is supplementary reference material scoped to this skill only — `SKILL.md` is the loadable entry point and the authority. It does not override or extend the repository-root `AGENTS.md` / `AGENT_GUIDE.md`.
## When to Use
- Creating prompts for FLUX.2 or FLUX.1 models
+19
View File
@@ -356,6 +356,25 @@ Rules:
- Must have spaces before and after the tag
- Self-closing tag format
## Expressive Voice Direction
For narration, create a short voice-performance plan before generating audio:
- narrator persona and emotional intent
- pacing profile
- energy curve across the script
- where pauses should land
- words or phrases that need emphasis
Use concrete cues, not generic instructions. "Warm but decisive; pause before
the contrast; slow down on the final sentence" is useful. "Sound natural" is
not.
When the selected voice supports pauses, put the most important pauses directly
in the text with break tags. Generate a sample from the most performance-heavy
section first, and do not batch-generate the rest if the sample sounds flat,
rushed, or ignores the intended breaks.
## Best Practices
1. **Use `GET /v1/audio/voices`** to find compatible voices — not all voices from `GET /v2/voices` support Starfish TTS
@@ -19,6 +19,8 @@ boolean prop proliferation by using compound components, lifting state, and
composing internals. These patterns make codebases easier for both humans and AI
agents to work with as they scale.
> **Extended reference:** [`AGENTS.md`](AGENTS.md) in this directory is the long-form upstream guide (vendored from Vercel). It is supplementary reference material scoped to this skill only — `SKILL.md` is the loadable entry point and the authority. It does not override or extend the repository-root `AGENTS.md` / `AGENT_GUIDE.md`.
## When to Apply
Reference these guidelines when:
@@ -11,6 +11,8 @@ metadata:
Comprehensive performance optimization guide for React and Next.js applications, maintained by Vercel. Contains 65 rules across 8 categories, prioritized by impact to guide automated refactoring and code generation.
> **Extended reference:** [`AGENTS.md`](AGENTS.md) in this directory is the long-form upstream guide (vendored from Vercel). It is supplementary reference material scoped to this skill only — `SKILL.md` is the loadable entry point and the authority. It does not override or extend the repository-root `AGENTS.md` / `AGENT_GUIDE.md`.
## When to Apply
Reference these guidelines when:
+2 -1
View File
@@ -92,4 +92,5 @@ remotion-composer/public/demo-props/test-*
remotion-composer/public/demo-props/talking-head-*
remotion-composer/public/demo-props/caption-burn-*
venv/
venv/
.venv/
+473
View File
@@ -0,0 +1,473 @@
# ComfyUI Provider Adapter for OpenMontage
**RFC: Native ComfyUI backend for image and video generation**
---
## Motivation
OpenMontage's local GPU tools (`wan_video`, `hunyuan_video`, `cogvideo_video`,
`local_diffusion`) use HuggingFace `diffusers` directly. This works on x86 +
consumer GPUs but breaks on newer hardware where the PyTorch ecosystem hasn't
caught up:
| Issue | Detail |
|-------|--------|
| **NVIDIA Blackwell (sm_121)** | No stable PyTorch wheels for aarch64 + CUDA 13.0. Requires NGC containers or nightly builds. |
| **Flash Attention** | Does not support sm_121. Must be replaced with SageAttention v3 or native SDPA. |
| **Unified Memory (GB10/DGX Spark)** | `nvidia-smi` cannot report VRAM. Diffusers' memory estimation breaks. |
| **Model format mismatch** | Diffusers expects HF repos. Production deployments use `.safetensors` checkpoints with quantized variants (NVFP4, FP8) that diffusers doesn't natively load. |
ComfyUI already solves all of these. NVIDIA ships official ComfyUI containers
for DGX Spark. The community has optimized workflows for Blackwell (SageAttention,
NVFP4 quantization, LightX2V 4-step LoRAs). Models like WAN 2.2, FLUX 2,
and ACE-Step run reliably through ComfyUI on hardware where diffusers cannot.
A ComfyUI adapter gives OpenMontage access to any model ComfyUI supports,
on any hardware ComfyUI runs on, without shipping or maintaining PyTorch builds.
---
## Design
### Architecture
```
OpenMontage Agent
|
v
video_selector / image_selector
|
v
comfyui_video comfyui_image (new tools)
| |
v v
ComfyUI REST API (POST /prompt, GET /history, GET /view)
|
v
GPU (any hardware ComfyUI supports)
```
### Integration model
Two new `BaseTool` subclasses plus one shared client library:
```
tools/
_comfyui/
__init__.py
client.py # Shared ComfyUI REST client
workflows/ # Bundled workflow templates
flux2-txt2img.json
wan22-t2v-4step.json
wan22-i2v-4step.json
graphics/
comfyui_image.py # capability="image_generation", provider="comfyui"
video/
comfyui_video.py # capability="video_generation", provider="comfyui"
```
### Registry and selector integration
The tools declare `capability` and `provider` as class attributes.
`tool_registry.discover()` picks them up automatically via `pkgutil.walk_packages`.
`video_selector` and `image_selector` find them via `registry.get_by_capability()`.
The only selector change is operation-specific filtering in `video_selector` so
ComfyUI is not selected for `image_to_video` when only the text-to-video bundled
models are installed, or vice versa.
---
## Shared Client: `tools/_comfyui/client.py`
Encapsulates the ComfyUI REST API pattern proven in production (used by the
Bard project's Airflow DAGs for thousands of generations):
The endpoint contract was checked against current ComfyUI server documentation
and the April 2026 third-party developer guide:
- Official routes: `POST /prompt`, `GET /history/{prompt_id}`, `GET /view`,
`POST /upload/image`, `GET /object_info/{node_class}`, `GET /models/{folder}`,
`GET /system_stats`, and `WS /ws` are documented server routes.
- `/prompt` accepts the workflow in API format under the `prompt` key and
returns `prompt_id`, `number`, and `node_errors` on validation.
- `/history/{prompt_id}` returns completed node outputs; artifact records include
`filename`, `subfolder`, and `type`. The client passes all three through to
`/view` instead of assuming `type=output`.
- Workflows must be exported in ComfyUI API format, not the regular visual
canvas workflow format.
References:
- https://docs.comfy.org/development/comfyui-server/comms_routes
- https://www.runflow.io/blog/comfyui-api-developer-guide
```python
class ComfyUIClient:
"""Thin client for the ComfyUI REST API."""
def __init__(self, server_url: str | None = None):
self.server_url = server_url or os.environ.get(
"COMFYUI_SERVER_URL", "http://localhost:8188"
)
def is_available(self) -> bool:
"""Health check -- can we reach the server?"""
def submit(self, workflow: dict) -> str:
"""POST /prompt. Returns prompt_id. Raises on node_errors."""
def poll(self, prompt_id: str, timeout: int = 600, interval: int = 5) -> dict:
"""GET /history/{prompt_id} until complete. Returns outputs dict."""
def download(self, filename: str, subfolder: str, dest: Path) -> Path:
"""GET /view?filename=...&type=output. Writes bytes to dest."""
def upload_image(self, local_path: Path, name: str) -> str:
"""POST /upload/image. Returns server-side filename for LoadImage nodes."""
def generate(self, workflow: dict, output_node: str, dest: Path,
timeout: int = 600) -> Path:
"""Full cycle: submit -> poll -> download. Returns artifact path."""
```
**Why a shared client?** The submit/poll/download cycle is identical across
image and video generation. The only differences are: which workflow template,
which nodes to customize, and which output node to read from.
---
## Tool Specifications
### `comfyui_image` -- Image Generation
| Field | Value |
|-------|-------|
| capability | `image_generation` |
| provider | `comfyui` |
| runtime | `LOCAL_GPU` |
| tier | `GENERATE` |
| stability | `EXPERIMENTAL` |
| capabilities | `text_to_image`, `image_to_image` |
| dependencies | (runtime: ComfyUI server reachable) |
| fallback_tools | `flux_image`, `local_diffusion`, `openai_image` |
| cost | `$0.00` (local compute) |
**Bundled workflow:** `flux2-txt2img.json`
Loads FLUX 2 Dev (NVFP4) with Mistral text encoder. Templated nodes:
| Node | Class | Templated field |
|------|-------|-----------------|
| 4 | CLIPTextEncode | `text` (prompt) |
| 6 | EmptyFlux2LatentImage | `width`, `height` |
| 7 | RandomNoise | `noise_seed` |
| 10 | Flux2Scheduler | `steps` |
| 13 | SaveImage | `filename_prefix` |
**Input schema:**
```yaml
prompt: string # required
width: integer # default 1024
height: integer # default 1024
steps: integer # default 20
seed: integer # optional (random if omitted)
guidance: number # default 3.5
output_path: string # where to save the image
workflow_json: string # optional custom workflow; requires output_node
workflow_path: string # optional path to workflow JSON; requires output_node
output_node: string # required for custom workflows
workflow_name: string # optional custom workflow provenance label
workflow_model: string # optional custom model/provenance label
workflow_model_stack: [] # optional custom dependency provenance
```
**get_status():** Pings ComfyUI server and checks bundled FLUX model names via
`/object_info`. Returns `AVAILABLE` when the server and bundled model set are
ready, `DEGRADED` when the server is reachable but bundled models are missing,
and `UNAVAILABLE` when the server cannot be reached.
**execute() flow:**
1. Deep-copy workflow template
2. Inject prompt, seed, dimensions, steps into templated nodes
3. `client.generate(workflow, output_node="13", dest=output_path)`
4. Return `ToolResult` with artifact path, seed, model info
For custom workflows, the caller must provide `workflow_json` or `workflow_path`
plus `output_node`. The tool does not assume bundled node IDs for custom
workflows, and provenance is reported as user-supplied unless the caller provides
`workflow_model`. Results also include the final workflow SHA-256 hash and, for
bundled workflows, the known model stack.
---
### `comfyui_video` -- Video Generation
| Field | Value |
|-------|-------|
| capability | `video_generation` |
| provider | `comfyui` |
| runtime | `LOCAL_GPU` |
| tier | `GENERATE` |
| stability | `EXPERIMENTAL` |
| capabilities | `text_to_video`, `image_to_video` |
| dependencies | (runtime: ComfyUI server reachable) |
| fallback_tools | `wan_video`, `hunyuan_video`, `ltx_video_local` |
| cost | `$0.00` (local compute) |
**Bundled workflows:**
1. **`wan22-i2v-4step.json`** -- Image-to-video (WAN 2.2 14B, fp8, 4-step LightX2V LoRA)
2. **`wan22-t2v-4step.json`** -- Text-to-video (WAN 2.2 14B, fp8, 4-step LightX2V LoRA)
These bundled WAN 2.2 14B FP8 workflows are the high-quality profile and
recommend roughly 16GB VRAM. That is not a ComfyUI-wide requirement. The
`comfyui_video` tool's top-level `resource_profile` is an 8GB provider floor so
preflight does not imply ComfyUI itself requires 16GB. Low-VRAM users should use
custom workflows such as Wan 2.1 1.3B, LTX-Video/LTXV FP8 or quantized graphs,
or Wan 2.2 GGUF/quantized community workflows, with shorter frame counts and
lower resolutions as needed.
**I2V workflow -- templated nodes:**
| Node | Class | Templated field |
|------|-------|-----------------|
| 93 | CLIPTextEncode | `text` (positive prompt) |
| 97 | LoadImage | `image` (server filename from upload) |
| 98 | WanImageToVideo | `width`, `height`, `length` |
| 86 | KSamplerAdvanced | `noise_seed` |
| 108 | SaveVideo | `filename_prefix` |
**Input schema:**
```yaml
prompt: string # required
operation: string # "text_to_video" | "image_to_video" (default: t2v)
reference_image_path: string # local path (for i2v)
reference_image_url: string # URL (for i2v, downloaded first)
width: integer # default 640
height: integer # default 640
num_frames: integer # default 81 (5s at 16fps)
seed: integer # optional
output_path: string # where to save the video
workflow_json: string # optional custom workflow; requires output_node
workflow_path: string # optional path to workflow JSON; requires output_node
output_node: string # required for custom workflows
workflow_name: string # optional custom workflow provenance label
workflow_model: string # optional custom model/provenance label
workflow_model_stack: [] # optional custom dependency provenance
```
**execute() flow (i2v):**
1. Upload reference image via `client.upload_image()`
2. Deep-copy i2v workflow template
3. Inject prompt, uploaded image name, seed, dimensions
4. `client.generate(workflow, output_node="108", dest=output_path, timeout=900)`
5. Return `ToolResult`
**execute() flow (t2v):**
1. Deep-copy t2v workflow template
2. Inject prompt, seed, dimensions
3. `client.generate(workflow, output_node="16", dest=output_path, timeout=900)`
4. Return `ToolResult`
`comfyui_video` publishes `operation_statuses` in `get_info()` and implements
`is_operation_available(operation)` for selector routing. This keeps partial
ComfyUI installs useful for the installed mode without advertising unavailable
operation modes as ready. `video_selector` also applies this readiness check
when `operation="rank"` by using `target_operation`, so preflight rankings do
not promote ComfyUI for an operation whose bundled models are missing.
---
### `comfyui_music` -- Music Generation (not shipped)
We explored adding a `comfyui_music` tool using the ACE-Step 3.5B model.
The model runs well in ComfyUI, but the ComfyUI node interface for
ACE-Step is not standardized -- there are multiple custom node packs with
different class names (`AceStepModelLoader` vs native `TextEncodeAceStepAudio`,
etc.). Shipping a workflow that only works with one specific custom node
pack would break for most users.
**Future path:** ACE-Step support should be revisited once OpenMontage decides
the music-generation routing shape and a portable ComfyUI audio workflow
contract. Current image/video workflow overrides are intentionally scoped to
image and video artifacts, not arbitrary audio workflows.
---
## Workflow Override Mechanism
The image and video tools accept either `workflow_json` or `workflow_path`.
When provided, the custom workflow replaces the bundled template entirely and
the caller must also provide `output_node`. This stricter contract is required
because community workflows use arbitrary node IDs.
- Using newer model checkpoints without code changes
- Custom sampling strategies (different schedulers, step counts, LoRAs)
- Community workflows dropped in as-is
- A/B testing different generation approaches
The agent can also read workflow files from `tools/_comfyui/workflows/` and
modify them programmatically before passing to `execute()`.
Custom workflow result metadata reports `workflow_provenance.source` as
`user_supplied` and uses `workflow_model`, `model`, or `workflow_name` as the
model label when provided. If no custom label is supplied, the model is reported
as `custom-comfyui-workflow` instead of one of the bundled model names. The
provenance payload also records `workflow_hash_sha256`. For user-supplied
workflows, callers should provide `workflow_model_stack` with base model, text
encoder, VAE, LoRAs and strengths, scheduler, steps, and guidance when known.
---
## Agent Skill and Setup Contract
Both ComfyUI tools advertise the Layer 3 `comfyui` skill. Agents must read
`.agents/skills/comfyui/SKILL.md` before calling either tool so they know how to
load community workflows, identify output nodes, handle LoRA loader chains, and
record custom workflow provenance.
Unavailable ComfyUI tools expose a structured `setup_offer` in `get_info()`,
`provider_menu()`, and `provider_menu_summary().setup_offers[]`:
```yaml
kind: local_server
env_var: COMFYUI_SERVER_URL
default_url: http://localhost:8188
health_check: GET /system_stats
```
When bundled models are missing, the tool returns a machine-readable
`data.missing_models[]` list with filename, role, destination hint, and download
URL when OpenMontage knows the canonical source. Agents should surface that
payload rather than parsing prose error text.
---
## Configuration
**Environment variables:**
```bash
# .env
COMFYUI_SERVER_URL=http://localhost:8188 # ComfyUI API endpoint
COMFYUI_POLL_INTERVAL=5 # seconds between status checks
COMFYUI_POLL_TIMEOUT=600 # max wait for image gen
COMFYUI_VIDEO_TIMEOUT=900 # max wait for video gen
```
**For Docker Compose setups** (ComfyUI in a container):
```bash
COMFYUI_SERVER_URL=http://host.docker.internal:8188
# or
COMFYUI_SERVER_URL=http://comfyui:8188 # if on same docker network
```
---
## Provider Selection Behavior
When the adapter is available, selectors will rank it alongside other providers
using OpenMontage's 7-dimension scoring:
| Dimension | ComfyUI score | Rationale |
|-----------|---------------|-----------|
| Task fit | High | Supports t2i, i2v, t2v |
| Quality | High | Latest models (FLUX 2, WAN 2.2 14B) |
| Control | Highest | Full workflow customization |
| Reliability | High | Proven in production |
| Cost | $0 | Local compute |
| Latency | Medium | GPU-bound, no network round-trip |
| Continuity | High | Deterministic with seeds |
When ComfyUI is unavailable (server down), selectors fall through to other
available providers. When only one video operation is configured, `video_selector`
uses the tool's operation-specific readiness to avoid selecting ComfyUI for the
missing mode.
---
## What This Unlocks
### Immediate (with existing models)
- **FLUX 2 Dev NVFP4** image generation -- Blackwell-optimized, ~60s per image
- **WAN 2.2 14B FP8 high-quality profile** i2v with 4-step acceleration -- ~3.5 min per 5s clip, about 16GB VRAM recommended
- **WAN 2.2 14B FP8 high-quality profile** t2v (models downloaded, workflow included), about 16GB VRAM recommended
### Low-VRAM profile
ComfyUI can still be useful on 8GB-12GB GPUs when the user supplies an
appropriate `workflow_json` or `workflow_path`. Good candidates include:
- Wan 2.1 1.3B workflows for lower-memory text-to-video.
- LTX-Video/LTXV FP8 or quantized workflows for fast short clips.
- Wan 2.2 GGUF/quantized community workflows at lower resolution and frame count.
OpenMontage should treat those as custom workflow profiles until a blessed
low-VRAM workflow is bundled. For custom workflows, resource requirements are
workflow-supplied rather than inferred from the bundled WAN 2.2 14B profile.
### Future (add models to ComfyUI, no code changes to OpenMontage)
- Newer checkpoints (WAN 3.x, FLUX 3, etc.) -- just update workflow JSON
- ControlNet, IP-Adapter, AnimateDiff -- supported via ComfyUI custom nodes
- Upscaling, inpainting, outpainting -- ComfyUI nodes exist
- Any model the ComfyUI ecosystem supports
### Hardware portability
The same adapter works on:
- NVIDIA DGX Spark (GB10, aarch64, CUDA 13.0)
- Consumer GPUs (RTX 3090/4090, x86)
- Cloud instances (A100, H100)
- Multi-GPU setups (ComfyUI handles device placement)
No PyTorch version pinning, no architecture-specific wheels, no CUDA
compatibility matrices. ComfyUI is the abstraction layer.
---
## Implementation Scope
| Component | Files | Estimated size |
|-----------|-------|----------------|
| Shared client | `tools/_comfyui/client.py` | ~180 lines |
| Shared metadata | `tools/_comfyui/metadata.py` | setup, model stack, provenance helpers |
| Image tool | `tools/graphics/comfyui_image.py` | ~140 lines |
| Video tool | `tools/video/comfyui_video.py` | ~190 lines |
| Layer 3 skill | `.agents/skills/comfyui/SKILL.md` | usage contract |
| Registry summary | `tools/tool_registry.py` | setup offer surfacing |
| Selector readiness filter | `tools/video/video_selector.py` | small operation-readiness check |
| Workflow templates | `tools/_comfyui/workflows/*.json` | 3 files |
| Tests | `tests/contracts/test_comfyui_tools.py` | ~200 lines |
| Docs | `docs/comfyui-adapter-plan.md` | This file |
**Total:** ~500 lines of Python + 3 workflow JSONs.
No changes to: `base_tool.py`, existing non-ComfyUI generation providers, any
pipeline definition, or any schema.
---
## Open Questions
1. **Workflow versioning:** Should workflow JSONs live in the repo or be
user-provided via a config directory? Bundling gives reproducibility;
external gives flexibility.
2. **Async generation:** ComfyUI supports websocket connections for real-time
progress. Worth implementing for long video generations, or is polling
sufficient?
3. **Multi-server:** Should the adapter support multiple ComfyUI instances
(e.g., one for images, one for video) via per-capability URLs?
4. **Music generation:** ACE-Step works in ComfyUI but OpenMontage needs a
dedicated music-generation routing contract before adding `comfyui_music`.
The follow-up should decide selector integration, audio artifact schemas, and
a portable workflow/output-node contract rather than treating music as a
hidden image/video workflow override.
+3
View File
@@ -40,6 +40,7 @@ required_skills:
- meta/checkpoint-protocol
- meta/skill-creator
- meta/animation-runtime-selector
- meta/voice-performance-director
orchestration:
mode: executive-producer
@@ -130,6 +131,7 @@ stages:
- "Enhancement cue density: 1 per 8-10 seconds"
- "Narrative arc: hook -> setup -> build -> climax -> landing"
- Speaker directions present for TTS
- Voice performance plan includes concrete pacing, pause, emphasis, and sample cues
- Script incorporates research data points and cited facts
success_criteria:
- Schema-valid script with enhancement_cues per section
@@ -186,6 +188,7 @@ stages:
review_focus:
- All asset files exist on disk
- Narration covers all script sections
- Narration assets apply the approved voice performance plan and sample settings
- Total cost within approved budget from proposal_packet
- Image style consistency across all generated visuals
success_criteria:
+3
View File
@@ -40,6 +40,7 @@ required_skills:
- meta/reviewer
- meta/checkpoint-protocol
- meta/animation-runtime-selector
- meta/voice-performance-director
orchestration:
mode: executive-producer
@@ -137,6 +138,7 @@ stages:
- Script beats are concise and animation-friendly
- On-screen text remains readable and restrained
- Timing structure leaves room for visual holds
- Narration-led scripts include concrete voice performance cues
- Word count within +/-10% of duration target from selected concept
- Script incorporates research data points where relevant
success_criteria:
@@ -196,6 +198,7 @@ stages:
- Asset production path is explicit per scene
- Reusable motifs and templates are prepared and referenced
- Missing tool paths are surfaced honestly
- Narration assets apply the approved voice performance plan and sample settings
- Total cost within approved budget from proposal_packet
- "Layer 3 skills read for EVERY generation tool before writing prompts (check agent_skills field)"
- "Clip duration maximized (prefer 10s over 5s) to reduce API calls and cost"
+18 -1
View File
@@ -33,7 +33,24 @@
"generation_summary": { "type": "string", "description": "Brief summary of how the asset was generated or sourced" },
"provider": { "type": "string", "description": "Provider name (e.g. pixabay, google_imagen)" },
"license": { "type": "string", "description": "License type (e.g. Pixabay License, CC0)" },
"original_url": { "type": "string", "description": "Source URL if downloaded from a stock service" }
"original_url": { "type": "string", "description": "Source URL if downloaded from a stock service" },
"voice_performance": {
"type": "object",
"description": "Applied voice-performance contract for narration assets.",
"properties": {
"source_section_id": { "type": "string" },
"delivery_cues_applied": { "type": "boolean" },
"provider_text_used": { "type": "boolean" },
"provider_settings": {
"type": "object",
"additionalProperties": true
},
"sample_approved": { "type": "boolean" },
"sample_path": { "type": "string" },
"review_notes": { "type": "string" }
},
"additionalProperties": false
}
},
"additionalProperties": false
}
+13 -1
View File
@@ -183,7 +183,19 @@
"provider": { "type": "string" },
"voice_id": { "type": "string" },
"rationale": { "type": "string" },
"estimated_cost_usd": { "type": "number", "minimum": 0 }
"estimated_cost_usd": { "type": "number", "minimum": 0 },
"delivery_style": {
"type": "string",
"description": "Plain-language voice direction, e.g. warm expert, expressive narrator, crisp tutorial."
},
"pacing_policy": {
"type": "string",
"description": "How the voice should use pauses, speed, and emphasis across the piece."
},
"sample_approval_required": {
"type": "boolean",
"description": "True when a TTS sample must be approved before batch narration generation."
}
},
"additionalProperties": false
},
+45
View File
@@ -9,6 +9,28 @@
"version": { "type": "string", "const": "1.0" },
"title": { "type": "string" },
"total_duration_seconds": { "type": "number", "minimum": 1 },
"voice_performance": {
"type": "object",
"description": "Narration delivery contract used by TTS asset generation.",
"properties": {
"performance_intent": { "type": "string" },
"pacing_profile": {
"type": "string",
"enum": ["contemplative", "conversational", "energetic", "technical", "cinematic", "custom"]
},
"energy_curve": { "type": "string" },
"pause_policy": { "type": "string" },
"sample_section_id": {
"type": "string",
"description": "Most performance-sensitive section to use for TTS sample approval."
},
"provider_notes": {
"type": "object",
"additionalProperties": { "type": "string" }
}
},
"additionalProperties": false
},
"sections": {
"type": "array",
"items": {
@@ -21,6 +43,29 @@
"start_seconds": { "type": "number", "minimum": 0 },
"end_seconds": { "type": "number", "minimum": 0 },
"speaker_directions": { "type": "string" },
"delivery_cues": {
"type": "object",
"description": "Structured voice-performance cues that TTS generation must apply.",
"properties": {
"pace": {
"type": "string",
"enum": ["slow", "measured", "conversational", "brisk", "fast", "custom"]
},
"energy": { "type": "string" },
"emphasis_words": {
"type": "array",
"items": { "type": "string" }
},
"pause_before_seconds": { "type": "number", "minimum": 0 },
"pause_after_seconds": { "type": "number", "minimum": 0 },
"delivery_note": { "type": "string" },
"provider_text": {
"type": "string",
"description": "Provider-ready narration text, including purposeful punctuation or SSML break tags when supported."
}
},
"additionalProperties": false
},
"enhancement_cues": {
"type": "array",
"items": {
+93
View File
@@ -0,0 +1,93 @@
# Voice Performance Director
Use this meta skill whenever a pipeline will generate narration with TTS.
The goal is to make generated narration sound directed, not merely read. Do not
leave expressiveness as "read naturally" in a prompt. Carry a concrete voice
performance plan from script to asset generation, then verify it with a sample.
## Required Contract
Every narration-led script should include a top-level `voice_performance`
object and section-level `delivery_cues` where the schema allows it.
Top-level voice performance:
```json
{
"performance_intent": "Warm, decisive product narrator with human pauses.",
"pacing_profile": "conversational",
"energy_curve": "measured hook, warmer middle, more deliberate close",
"pause_policy": "Use short pauses after setup lines and longer pauses before reversals or important claims.",
"provider_notes": {
"openai": "Use instructions for emotional arc and emphasis.",
"google_tts": "Use SSML input with break tags when the selected voice supports it.",
"elevenlabs": "Use lower stability and moderate style for expressive narration."
}
}
```
Section-level delivery cues:
```json
{
"pace": "measured",
"energy": "curious",
"emphasis_words": ["not", "process"],
"pause_before_seconds": 0.2,
"pause_after_seconds": 0.7,
"delivery_note": "Set up the contrast, then slow down on the final phrase.",
"provider_text": "This is not just another tool. <break time=\"0.6s\"/> It is a process."
}
```
## Writing Rules
- Write spoken language, not essay language. Prefer short sentences, light
contractions, and clear punctuation.
- Use silence as structure. Add a pause before reversals, after surprising
claims, and before the final takeaway.
- Keep pause tags purposeful. Too many breaks sound theatrical and slow.
- Avoid generic directions such as "natural", "engaging", or "expressive"
unless they are paired with exact pace, emphasis, pause, or energy cues.
- Prefer one delivery idea per section. If a section needs three emotional
turns, split it.
## Provider Mapping
- OpenAI TTS: use `model: "gpt-4o-mini-tts"` when sending `instructions`.
Put the emotional arc, pacing, emphasis, and role in `instructions`; keep the
input text clean but punctuated. Do not send `instructions` to `tts-1` or
`tts-1-hd`.
- Google TTS: use `input_type: "ssml"` only when adding break tags or other
SSML. The tool maps this to Google `input.ssml` and wraps the utterance in
`<speak>...</speak>` when needed. Keep `speaking_rate` in Google's supported
`0.25..2.0` range and pitch in `-20..20`.
- ElevenLabs: use lower `stability` for more variation, moderate `style` for
expressiveness, `speed` in the provider's `0.7..1.2` range, and keep
`similarity_boost` high enough to preserve the voice.
- Offline/basic voices: rely on punctuation, shorter sentences, and explicit
segment splitting because provider-level emotion controls may be unavailable.
## Sample Gate
Before batch narration generation:
1. Generate a sample from the most performance-sensitive section, not
automatically the first section.
2. Verify voice, pace, pauses, emphasis, and emotional arc.
3. If the sample is flat, adjust the `voice_performance` plan or provider
settings before generating the rest.
4. Record the approved sample path and provider settings in the asset manifest.
## Failure Conditions
Treat these as quality failures:
- A narration-led script has no `voice_performance` plan.
- Section directions only say "read naturally" or "expressive" with no concrete
pause, emphasis, pace, or energy cue.
- TTS provider, voice, speed, or model changes after sample approval without a
new sample.
- Final narration is generated from raw script text while structured
`provider_text` or `delivery_cues` were present.
+12 -3
View File
@@ -49,7 +49,7 @@ Prefer the lowest-variance useful path:
Before batch-generating assets, produce one sample of each expensive type and show the user:
1. **TTS sample** (if narration-led): Generate one section. Confirm voice and tone before batching.
1. **TTS sample** (if narration-led): Generate `script.voice_performance.sample_section_id` when present; otherwise choose the section with the strongest emotional or pacing change. Confirm voice, pace, pauses, emphasis, and tone before batching.
2. **Visual sample**: Generate one representative scene visual (diagram, illustration, or motion background). Confirm style and quality before batching the rest.
If rejected, adjust parameters and retry (max 3 iterations). Do not batch until approved.
@@ -100,7 +100,12 @@ Create once:
### 3. Narration Is Optional, But The Plan Must Be Explicit
If the project is narration-led, produce or source narration. If it is text-led or music-led, say so clearly in metadata.
If the project is narration-led, produce or source narration. Read
`skills/meta/voice-performance-director.md`, then apply `script.voice_performance`
and each section's `delivery_cues` when building TTS requests. Use
`provider_text` when present, map cues to provider controls, and record the
applied settings on each narration asset. If it is text-led or music-led, say so
clearly in metadata.
### 4. Use Metadata For Feasibility Truth
@@ -109,6 +114,7 @@ Recommended metadata keys:
- `tool_path_map`
- `reusable_assets`
- `narration_assets`
- `voice_performance`: sample approval path, provider settings, and whether delivery cues were applied
- `scene_asset_index`
- `blocked_assets`
@@ -117,7 +123,8 @@ Recommended metadata keys:
- the asset path is explicit per scene,
- reusable assets are actually reused,
- missing capabilities are surfaced honestly,
- every referenced file exists.
- every referenced file exists,
- narration-led assets apply the approved voice-performance settings.
### Mid-Production Fact Verification
@@ -135,6 +142,8 @@ the AI model's training data — it may be wrong or outdated.
- Using high-variance generation when a deterministic asset would work better.
- Rebuilding the same title or label system repeatedly.
- Hiding failed asset paths instead of reporting them.
- Treating TTS as raw text-to-audio. Narration-led animation needs pauses,
emphasis, and pace cues carried from the script into the generated audio.
- Treating "consistency" as "same prompt every time." Good animation keeps a recognizable world while still letting each beat feel fresh.
@@ -11,6 +11,7 @@ This stage turns the approved proposal into animation-ready beats. The script mu
| Schema | `schemas/artifacts/script.schema.json` | Artifact validation |
| Prior artifact | `proposal_packet` from Proposal Director | Selected concept, animation mode, target duration, reuse strategy |
| Optional artifact | `research_brief` from Research Director | Data points, audience insights, accuracy constraints |
| Meta skill | `skills/meta/voice-performance-director.md` | Structured TTS delivery cues for natural, expressive narration |
| Tools | `transcriber` | Optional source transcript support |
## Process
@@ -41,6 +42,11 @@ Each section should express ONE clear visual idea:
- **Comparison** — show two things side by side (split screen or sequential)
- **Conclusion** — land the insight (hold + emphasis)
If the piece is narration-led, also create `voice_performance` and section
`delivery_cues` using `skills/meta/voice-performance-director.md`. Motion-heavy
videos need vocal breathing room: mark pauses where the viewer must absorb an
animation, and mark emphasis where a reveal, transform, or comparison lands.
**Animation mode affects writing style:**
| Mode | Writing Style |
@@ -82,6 +88,7 @@ Recommended metadata keys per section:
- `narration_plan`: how narration relates to visual (describes / complements / silent)
- `visual_priority`: what the viewer should focus on (the animation, the text, the data)
- `hold_time_seconds`: minimum visual hold time after this section's content
- `delivery_cues`: pace, energy, emphasis, pause timing, and provider-ready narration text for TTS
- `data_source`: if this section uses a research data point, reference it
### 6. Research Integration
@@ -101,6 +108,7 @@ Before submitting the script, verify:
- [ ] Every section supports ONE strong visual idea
- [ ] On-screen text is concise (phrases, not paragraphs)
- [ ] Timing is animation-friendly (holds budgeted)
- [ ] Narration-led sections include concrete delivery cues and a voice-performance sample section
- [ ] Word count is within ±10% of target duration
- [ ] Animation mode is respected in writing style
- [ ] Research data points are integrated (if research_brief available)
+19 -5
View File
@@ -70,7 +70,7 @@ Before generating anything:
Before batch-generating assets, produce one sample of each expensive asset type and present them to the user for approval:
1. **TTS sample**: Generate narration for the first script section only. Play it for the user. Confirm voice, pace, and tone are acceptable before generating the rest.
1. **TTS sample**: Generate narration for `script.voice_performance.sample_section_id` when present; otherwise pick the section with the most demanding delivery. Play it for the user. Confirm voice, pace, pauses, emphasis, and tone are acceptable before generating the rest.
2. **Image sample**: Generate one image for the most representative scene. Show it to the user. Confirm the style, quality, and prompt approach before batch-generating all images.
3. **Music sample** (if using `music_gen`): Generate one short clip. Confirm mood and energy before committing.
@@ -85,13 +85,24 @@ This step typically costs $0.030.08 total and prevents $13 of wasted gener
For each script section:
1. Extract the narration text
2. Apply speaker directions from the script (pace, emphasis, emotion)
3. Apply the playbook's `audio.voice_style`
4. Generate using `tts_selector` — it auto-routes to the best available TTS provider based on user preference and availability. Check the registry's `best_for` fields to understand each provider's strengths.
5. Verify the audio file exists and duration matches expected timing (±15%)
2. Read `script.voice_performance` and section `delivery_cues`
3. Use `delivery_cues.provider_text` when present; otherwise transform the section text with purposeful punctuation and break tags only when the selected provider supports them
4. Apply speaker directions from the script (pace, emphasis, emotion)
5. Apply the playbook's `audio.voice_style`
6. Map cues to provider parameters:
- OpenAI: `instructions` only with `model: "gpt-4o-mini-tts"`; use `response_format` for output format
- Google TTS: `input_type: "ssml"` when using `<break>` tags, plus `speaking_rate` in `0.25..2.0` and `pitch` in `-20..20`
- ElevenLabs: `stability`, `similarity_boost`, `style`, `speed`, and `use_speaker_boost`
7. Generate using `tts_selector` — it auto-routes to the best available TTS provider based on user preference and availability. Check the registry's `best_for` fields to understand each provider's strengths.
8. Record the applied `voice_performance` metadata on each narration asset
9. Verify the audio file exists and duration matches expected timing (±15%)
**Pronunciation guide**: If the script contains technical terms, jargon, or names with non-obvious pronunciation, include a pronunciation map in the TTS request.
**Flat voice failure:** If the approved voice sounds monotone, robotic, rushed,
or ignores intended pauses, do not batch the remaining sections. Revise the
`voice_performance` plan or provider parameters and regenerate the sample.
### Step 4: Generate Visual Assets
Process asset tasks grouped by tool for efficiency:
@@ -204,6 +215,8 @@ Assemble all generated assets into the manifest:
**Quality check:**
- [ ] Narration durations within ±15% of expected timing
- [ ] Narration assets record `voice_performance.delivery_cues_applied`
- [ ] Approved TTS sample uses the same provider, voice, and expressive settings as the batch
- [ ] Images match the playbook's style (review consistency anchors)
- [ ] Diagrams are legible and complete
- [ ] Total cost within budget
@@ -242,6 +255,7 @@ the AI model's training data — it may be wrong or outdated.
- **Generating before checking budget**: Always estimate total cost first. A 60-second video with 15 images can burn $3+ quickly.
- **Inconsistent image style**: Each image_selector call is independent. Use consistent anchors, but adapt them per scene. If you paste the same style prefix into every prompt, the video will feel machine-made and repetitive.
- **Ignoring narration timing**: If TTS produces 12s of audio for a 10s section, the edit phase will struggle. Check durations.
- **Ignoring delivery cues**: Generating raw script text when `provider_text` or `delivery_cues` exist will flatten the read. Apply the voice-performance contract first.
- **Missing pronunciation guide**: "PostgreSQL" or "Kubernetes" will be mispronounced without explicit guidance.
- **One retry then give up**: If an image doesn't match, refine the prompt specifically — don't just retry the same prompt.
- **AI-generating images with exact text (CTA, business names, contact info)**: AI image models frequently hallucinate wrong text — wrong business name, wrong phone number, misspelled words. **Never use AI image generation for scenes where text must be verbatim.** Use Remotion `text_card` type instead. This applies to: CTA screens, title cards with business names, contact info overlays, legal disclaimers. If a scene's `type` is `text_card` in the scene plan, do NOT generate an image for it — skip it and let the compose stage render it natively in Remotion.
+24 -1
View File
@@ -14,6 +14,7 @@ The script is the backbone of the video. Every visual, every scene, every audio
| Prior artifact | `proposal_packet` | Selected concept with title, hook, key_points, core_message, tone, narrative_structure, duration |
| Prior artifact | `research_brief` (optional but high-value) | Data points, audience insights, expert quotes — ground the script in real facts |
| Playbook | Active style playbook from `proposal_packet.selected_concept.suggested_playbook` | Voice style, pacing rules |
| Meta skill | `skills/meta/voice-performance-director.md` | Structured TTS delivery cues for natural, expressive narration |
| Layer 3 | TTS provider skills (check `agent_skills` on the selected TTS tool) | TTS capabilities for speaker directions |
## Process
@@ -77,6 +78,11 @@ Map each of the brief's `key_points` to a specific section in the BUILD phase.
### Step 4: Write the Script
Before writing sections, create a top-level `voice_performance` plan using
`skills/meta/voice-performance-director.md`. The plan must describe the vocal
intent, pacing profile, energy curve, pause policy, and which section should be
used for TTS sample approval. Do not leave this as a vague "natural voice" note.
Write each section with these fields:
```json
@@ -87,6 +93,14 @@ Write each section with these fields:
"start_seconds": 0,
"end_seconds": 5,
"speaker_directions": "Emphasize 'every single row' with measured pacing. Brief pause before the question.",
"delivery_cues": {
"pace": "measured",
"energy": "curious",
"emphasis_words": ["every", "single"],
"pause_after_seconds": 0.6,
"delivery_note": "Let the repetition feel intentional, then soften into the question.",
"provider_text": "Your database searches every single row. Every. Single. One. <break time=\"0.6s\"/> What if it didn't have to?"
},
"enhancement_cues": [
{
"type": "animation",
@@ -117,7 +131,8 @@ Count your words. If you're 20%+ over budget, the TTS will either rush or exceed
#### Speaker Directions
Write directions that TTS can actually implement. Reference ElevenLabs capabilities:
Write directions that TTS can actually implement. Prefer structured
`delivery_cues` over prose-only `speaker_directions`:
| Direction | TTS Implementation |
|-----------|-------------------|
@@ -129,6 +144,12 @@ Write directions that TTS can actually implement. Reference ElevenLabs capabilit
Avoid directions TTS can't do: "smile while speaking", "gesture toward screen", "look at camera."
**Expressive narration rule:** every narration-led section must include at
least two concrete cues among `pace`, `energy`, `emphasis_words`,
`pause_before_seconds`, `pause_after_seconds`, `delivery_note`, or
`provider_text`. Use `provider_text` when punctuation or SSML break tags are
needed to make the read sound human.
#### Enhancement Cues
Every section should have at least one enhancement cue. These tell the Scene Planner and Asset Generator what visuals to create.
@@ -162,6 +183,7 @@ Read the active style playbook and verify:
|----------------|---------------|
| `identity.pace` | Match word density. `contemplative` = fewer words, longer pauses |
| `audio.voice_style` | Shape tone of speaker directions |
| `voice_performance` | Confirm pacing, pauses, and energy curve are explicit enough for TTS |
| `motion.pacing_rules` | E.g., "hold establishing shots for 2s minimum" affects section timing |
| `identity.mood` | Word choice: `warm` uses casual language; `professional` uses precise language |
@@ -175,6 +197,7 @@ Score your script (1-5):
| **Word count accuracy** | Within ±10% of target for the duration? |
| **Narrative flow** | Does each section build on the last? "Therefore/but" not "and then"? |
| **Enhancement density** | At least one cue every 8-10 seconds? |
| **Voice performance** | Are pauses, emphasis, pace, and sample section explicit? |
| **Jargon management** | Technical terms explained or have pronunciation guides? |
| **Climax payoff** | Does the aha moment deliver on the hook's promise? |
| **CTA relevance** | Is the call to action specific and actionable? |
+652
View File
@@ -0,0 +1,652 @@
"""Contract tests for ComfyUI provider tools.
These tests verify that the tools satisfy the BaseTool contract without
requiring a running ComfyUI server. They check class attributes,
schemas, status reporting, and cost estimates.
"""
import json
from pathlib import Path
import pytest
from tools.base_tool import (
BaseTool,
ToolRuntime,
ToolStability,
ToolStatus,
ToolTier,
)
from tools.graphics.comfyui_image import ComfyUIImage
from tools.graphics.image_selector import ImageSelector
from tools.tool_registry import ToolRegistry
from tools.video.video_selector import VideoSelector
from tools.video.comfyui_video import ComfyUIVideo
TOOLS = [ComfyUIImage, ComfyUIVideo]
WORKFLOW_DIR = Path(__file__).resolve().parent.parent.parent / "tools" / "_comfyui" / "workflows"
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
# ------------------------------------------------------------------
# Contract compliance
# ------------------------------------------------------------------
@pytest.mark.parametrize("cls", TOOLS, ids=lambda c: c.name)
class TestContract:
def test_inherits_base_tool(self, cls):
assert issubclass(cls, BaseTool)
def test_has_required_identity(self, cls):
tool = cls()
assert tool.name
assert tool.version
assert tool.capability
assert tool.provider == "comfyui"
assert tool.tier == ToolTier.GENERATE
assert tool.stability == ToolStability.EXPERIMENTAL
assert tool.runtime == ToolRuntime.LOCAL_GPU
def test_has_input_schema(self, cls):
tool = cls()
schema = tool.input_schema
assert schema.get("type") == "object"
assert "prompt" in schema.get("properties", {})
assert "prompt" in schema.get("required", [])
def test_has_capabilities(self, cls):
tool = cls()
assert len(tool.capabilities) > 0
def test_has_agent_skills(self, cls):
tool = cls()
assert tool.agent_skills
assert "comfyui" in tool.agent_skills
def test_comfyui_layer3_skill_exists(self, cls):
skill_path = PROJECT_ROOT / ".agents" / "skills" / "comfyui" / "SKILL.md"
assert skill_path.exists()
assert "output_node" in skill_path.read_text(encoding="utf-8")
def test_has_fallbacks(self, cls):
tool = cls()
assert tool.fallback or tool.fallback_tools
def test_cost_is_zero(self, cls):
tool = cls()
assert tool.estimate_cost({"prompt": "test"}) == 0.0
def test_runtime_estimate_positive(self, cls):
tool = cls()
assert tool.estimate_runtime({"prompt": "test"}) > 0
def test_get_info_returns_dict(self, cls):
tool = cls()
info = tool.get_info()
assert isinstance(info, dict)
assert info["name"] == tool.name
assert info["provider"] == "comfyui"
assert info["runtime"] == "local_gpu"
assert info["setup_offer"]["env_var"] == "COMFYUI_SERVER_URL"
def test_video_resource_profile_does_not_mandate_16gb(self, cls):
if cls is not ComfyUIVideo:
return
tool = ComfyUIVideo()
info = tool.get_info()
assert info["resource_profile"]["vram_mb"] == 8000
assert info["resource_profiles"]["provider_floor"]["vram_mb"] == 8000
assert info["resource_profiles"]["bundled_wan22_14b_fp8"]["vram_mb"] == 16000
assert "not a ComfyUI provider-wide requirement" in (
info["resource_profiles"]["bundled_wan22_14b_fp8"]["applies_to"]
)
def test_status_unavailable_without_server(self, cls):
"""Without a running server, status should be UNAVAILABLE."""
tool = cls()
# Point to a port that's almost certainly not running ComfyUI
tool._client.server_url = "http://127.0.0.1:19999"
assert tool.get_status() == ToolStatus.UNAVAILABLE
def test_idempotency_key_fields(self, cls):
tool = cls()
assert len(tool.idempotency_key_fields) > 0
assert "prompt" in tool.idempotency_key_fields
def test_custom_workflow_schema_requires_output_node_contract(self, cls):
tool = cls()
props = tool.input_schema.get("properties", {})
assert "workflow_json" in props
assert "workflow_path" in props
assert "output_node" in props
def test_custom_workflow_requires_output_node(self, cls):
tool = cls()
result = tool.execute({"prompt": "test", "workflow_json": "{}"})
assert result.success is False
assert "output_node" in result.error
# ------------------------------------------------------------------
# Workflow files
# ------------------------------------------------------------------
EXPECTED_WORKFLOWS = [
"flux2-txt2img.json",
"wan22-i2v-4step.json",
"wan22-t2v-4step.json",
]
@pytest.mark.parametrize("filename", EXPECTED_WORKFLOWS)
def test_workflow_exists_and_valid_json(filename):
path = WORKFLOW_DIR / filename
assert path.exists(), f"Missing workflow: {path}"
with open(path) as f:
data = json.load(f)
assert isinstance(data, dict)
assert len(data) > 0
def test_flux2_workflow_has_templated_nodes():
with open(WORKFLOW_DIR / "flux2-txt2img.json") as f:
w = json.load(f)
assert "4" in w # CLIPTextEncode (prompt)
assert "7" in w # RandomNoise (seed)
assert "13" in w # SaveImage (output)
def test_i2v_workflow_has_templated_nodes():
with open(WORKFLOW_DIR / "wan22-i2v-4step.json") as f:
w = json.load(f)
assert "93" in w # CLIPTextEncode (prompt)
assert "97" in w # LoadImage (reference)
assert "86" in w # KSamplerAdvanced (seed)
assert "108" in w # SaveVideo (output)
def test_t2v_workflow_has_templated_nodes():
with open(WORKFLOW_DIR / "wan22-t2v-4step.json") as f:
w = json.load(f)
assert "2" in w # CLIPTextEncode (prompt)
assert "12" in w # KSamplerAdvanced (seed)
assert "16" in w # SaveVideo (output)
# ------------------------------------------------------------------
# Client unit tests
# ------------------------------------------------------------------
class TestClientHelpers:
def test_load_workflow(self):
from tools._comfyui.client import ComfyUIClient
w = ComfyUIClient.load_workflow(WORKFLOW_DIR / "flux2-txt2img.json")
assert isinstance(w, dict)
assert "1" in w
def test_patch_workflow(self):
from tools._comfyui.client import ComfyUIClient
w = ComfyUIClient.load_workflow(WORKFLOW_DIR / "flux2-txt2img.json")
patched = ComfyUIClient.patch_workflow(w, {
"4": {"text": "hello world"},
"7": {"noise_seed": 123},
})
assert patched["4"]["inputs"]["text"] == "hello world"
assert patched["7"]["inputs"]["noise_seed"] == 123
# Original unchanged
assert w["4"]["inputs"]["text"] == ""
def test_patch_workflow_bad_node(self):
from tools._comfyui.client import ComfyUIClient, ComfyUIError
w = {"1": {"inputs": {"x": 1}}}
with pytest.raises(ComfyUIError, match="not found"):
ComfyUIClient.patch_workflow(w, {"99": {"x": 2}})
def test_submit_surfaces_node_errors_before_http_error(self, monkeypatch):
from tools._comfyui.client import ComfyUIClient, ComfyUIError
class FakeResponse:
status_code = 400
def json(self):
return {
"error": {"message": "Prompt outputs failed validation"},
"node_errors": {"4": {"class_type": "MissingNode"}},
}
def raise_for_status(self):
raise AssertionError("HTTPError should not hide node_errors")
monkeypatch.setattr(
"tools._comfyui.client.requests.post",
lambda *args, **kwargs: FakeResponse(),
)
with pytest.raises(ComfyUIError, match="Node errors"):
ComfyUIClient("http://comfy.test").submit({})
def test_random_seed_range(self):
from tools._comfyui.client import ComfyUIClient
for _ in range(100):
s = ComfyUIClient.random_seed()
assert 0 <= s < 2**32
def test_generate_passes_history_item_type_to_view(self, monkeypatch, tmp_path):
from tools._comfyui.client import ComfyUIClient
client = ComfyUIClient("http://comfy.test")
seen = {}
monkeypatch.setattr(client, "submit", lambda workflow: "prompt-1")
monkeypatch.setattr(client, "poll", lambda prompt_id, **kwargs: {
"outputs": {
"9": {
"images": [{
"filename": "preview.png",
"subfolder": "previews",
"type": "temp",
}]
}
}
})
def fake_download(filename, subfolder, dest, folder_type="output"):
seen["filename"] = filename
seen["subfolder"] = subfolder
seen["folder_type"] = folder_type
return Path(dest)
monkeypatch.setattr(client, "download", fake_download)
client.generate({"9": {"inputs": {}}}, "9", tmp_path / "preview.png")
assert seen == {
"filename": "preview.png",
"subfolder": "previews",
"folder_type": "temp",
}
def test_is_default_url_when_env_not_set(self, monkeypatch):
from tools._comfyui.client import ComfyUIClient
monkeypatch.delenv("COMFYUI_SERVER_URL", raising=False)
client = ComfyUIClient()
assert client.is_default_url is True
def test_is_not_default_url_when_env_set(self, monkeypatch):
from tools._comfyui.client import ComfyUIClient
monkeypatch.setenv("COMFYUI_SERVER_URL", "http://myhost:9999")
client = ComfyUIClient()
assert client.is_default_url is False
def test_unavailable_reason_default_url(self, monkeypatch):
from tools._comfyui.client import ComfyUIClient
monkeypatch.delenv("COMFYUI_SERVER_URL", raising=False)
client = ComfyUIClient()
msg = client.unavailable_reason()
assert "COMFYUI_SERVER_URL" in msg
assert ".env" in msg
def test_unavailable_reason_custom_url(self, monkeypatch):
from tools._comfyui.client import ComfyUIClient
monkeypatch.setenv("COMFYUI_SERVER_URL", "http://myhost:9999")
client = ComfyUIClient()
msg = client.unavailable_reason()
assert "myhost:9999" in msg
assert "COMFYUI_SERVER_URL" not in msg
# ------------------------------------------------------------------
# Model discovery (offline, no server needed)
# ------------------------------------------------------------------
class TestModelRequirements:
def test_image_tool_has_required_models(self):
from tools.graphics.comfyui_image import _REQUIRED_MODELS
assert len(_REQUIRED_MODELS) > 0
assert any("flux" in m.lower() for m in _REQUIRED_MODELS)
def test_video_tool_has_required_models_i2v(self):
from tools.video.comfyui_video import _REQUIRED_MODELS_I2V
assert len(_REQUIRED_MODELS_I2V) > 0
assert any("i2v" in m.lower() for m in _REQUIRED_MODELS_I2V)
def test_video_tool_has_required_models_t2v(self):
from tools.video.comfyui_video import _REQUIRED_MODELS_T2V
assert len(_REQUIRED_MODELS_T2V) > 0
assert any("t2v" in m.lower() for m in _REQUIRED_MODELS_T2V)
# ------------------------------------------------------------------
# Custom workflow contract and provenance
# ------------------------------------------------------------------
class TestCustomWorkflowContract:
def test_image_custom_workflow_uses_caller_output_node_and_provenance(self, tmp_path):
tool = ComfyUIImage()
tool._client.is_available = lambda: True
seen = {}
def fake_generate(workflow, output_node, dest, **kwargs):
seen["workflow"] = workflow
seen["output_node"] = output_node
return [Path(dest)]
tool._client.generate = fake_generate
result = tool.execute({
"prompt": "test",
"workflow_json": json.dumps({"99": {"inputs": {}}}),
"output_node": "99",
"workflow_model": "custom-flux",
"output_path": str(tmp_path / "image.png"),
})
assert result.success is True
assert seen["output_node"] == "99"
assert result.model == "custom-flux"
assert result.data["model"] == "custom-flux"
assert result.data["workflow_provenance"]["source"] == "user_supplied"
assert result.data["workflow_provenance"]["output_node"] == "99"
assert result.data["workflow_provenance"]["workflow_hash_sha256"]
assert result.data["workflow_provenance"]["model_stack_source"] == (
"unknown_custom_workflow"
)
def test_video_custom_workflow_uses_caller_output_node_and_provenance(self, tmp_path):
tool = ComfyUIVideo()
tool._client.is_available = lambda: True
seen = {}
def fake_generate(workflow, output_node, dest, **kwargs):
seen["workflow"] = workflow
seen["output_node"] = output_node
return [Path(dest)]
tool._client.generate = fake_generate
result = tool.execute({
"prompt": "test",
"workflow_json": json.dumps({"42": {"inputs": {}}}),
"output_node": "42",
"workflow_model": "custom-wan",
"output_path": str(tmp_path / "video.mp4"),
})
assert result.success is True
assert seen["output_node"] == "42"
assert result.model == "custom-wan"
assert result.data["model"] == "custom-wan"
assert result.data["workflow_provenance"]["source"] == "user_supplied"
assert result.data["workflow_provenance"]["output_node"] == "42"
assert result.data["workflow_provenance"]["workflow_hash_sha256"]
assert result.data["workflow_provenance"]["model_stack_source"] == (
"unknown_custom_workflow"
)
def test_custom_workflow_accepts_model_stack_provenance(self, tmp_path):
tool = ComfyUIVideo()
tool._client.is_available = lambda: True
tool._client.generate = lambda workflow, output_node, dest, **kwargs: [Path(dest)]
result = tool.execute({
"prompt": "test",
"workflow_json": json.dumps({"42": {"inputs": {}}}),
"output_node": "42",
"workflow_model_stack": [{"role": "lora", "name": "style.safetensors"}],
"output_path": str(tmp_path / "video.mp4"),
})
provenance = result.data["workflow_provenance"]
assert provenance["model_stack"] == [{"role": "lora", "name": "style.safetensors"}]
assert provenance["model_stack_source"] == "caller_supplied"
def test_image_missing_models_are_structured(self):
tool = ComfyUIImage()
tool._client.is_available = lambda: True
tool._client.check_models = lambda required: (
[],
["flux2-vae.safetensors"],
)
result = tool.execute({"prompt": "test"})
assert result.success is False
assert result.data["provider"] == "comfyui"
assert result.data["missing_models"][0]["name"] == "flux2-vae.safetensors"
assert result.data["missing_models"][0]["destination_hint"] == "ComfyUI/models/vae/"
assert result.data["missing_models"][0]["download_url"]
def test_video_missing_models_are_structured(self):
tool = ComfyUIVideo()
tool._client.is_available = lambda: True
tool._client.check_models = lambda required: (
[],
["wan2.2_t2v_high_noise_14B_fp8_scaled.safetensors"],
)
result = tool.execute({"prompt": "test", "operation": "text_to_video"})
assert result.success is False
assert result.data["operation"] == "text_to_video"
assert result.data["missing_models"][0]["role"] == "diffusion_model_high_noise"
assert result.data["missing_models"][0]["download_url"]
def test_bundled_workflow_provenance_records_hash_and_stack(self, tmp_path):
tool = ComfyUIImage()
tool._client.is_available = lambda: True
tool._client.check_models = lambda required: (list(required), [])
tool._client.generate = lambda workflow, output_node, dest, **kwargs: [Path(dest)]
result = tool.execute({
"prompt": "test",
"output_path": str(tmp_path / "image.png"),
})
provenance = result.data["workflow_provenance"]
assert provenance["source"] == "bundled"
assert provenance["workflow_hash_sha256"]
assert any(item["role"] == "vae" for item in provenance["model_stack"])
class TestComfyUISetupOffer:
def test_provider_menu_summary_includes_structured_setup_offer(self):
registry = ToolRegistry()
tool = ComfyUIImage()
tool._client.is_available = lambda: False
registry.register(tool)
registry._discovered_packages.add("tools")
summary = registry.provider_menu_summary()
offer = summary["setup_offers"][0]
assert offer["tool"] == "comfyui_image"
assert offer["env_var"] == "COMFYUI_SERVER_URL"
assert offer["default_url"] == "http://localhost:8188"
assert offer["health_check"] == "GET /system_stats"
# ------------------------------------------------------------------
# Operation-specific video readiness
# ------------------------------------------------------------------
class TestVideoOperationReadiness:
def test_video_tool_reports_partial_operation_readiness(self):
from tools.video.comfyui_video import _REQUIRED_MODELS_I2V, _REQUIRED_MODELS_T2V
tool = ComfyUIVideo()
tool._client.is_available = lambda: True
def fake_check_models(required):
if required == _REQUIRED_MODELS_T2V:
return list(required), []
if required == _REQUIRED_MODELS_I2V:
return [], list(required)
return [], list(required)
tool._client.check_models = fake_check_models
assert tool.get_status() == ToolStatus.AVAILABLE
assert tool.is_operation_available("text_to_video") is True
assert tool.is_operation_available("image_to_video") is False
assert tool.operation_statuses() == {
"text_to_video": "available",
"image_to_video": "degraded",
}
def test_video_selector_filters_operation_unready_tools(self):
class PartialVideoTool(BaseTool):
name = "partial_video"
capability = "video_generation"
provider = "partial"
supports = {"image_to_video": True}
input_schema = {"type": "object", "properties": {}}
def is_operation_available(self, operation):
return operation == "text_to_video"
def execute(self, inputs):
raise AssertionError("not used")
selector = VideoSelector()
candidates = [PartialVideoTool()]
assert selector._filter_candidates(
{"operation": "image_to_video"}, candidates
) == []
def test_video_selector_rank_uses_target_operation_for_readiness(self):
class PartialVideoTool(BaseTool):
name = "partial_video"
capability = "video_generation"
provider = "partial"
supports = {"image_to_video": True}
input_schema = {"type": "object", "properties": {}}
def is_operation_available(self, operation):
return operation == "text_to_video"
def execute(self, inputs):
raise AssertionError("not used")
selector = VideoSelector()
candidates = [PartialVideoTool()]
rank_inputs = selector._rank_inputs({
"operation": "rank",
"target_operation": "image_to_video",
})
assert rank_inputs["operation"] == "image_to_video"
assert selector._filter_candidates(rank_inputs, candidates) == []
# ------------------------------------------------------------------
# Custom-workflow selector eligibility
# ------------------------------------------------------------------
class _DegradedComfyVideo(BaseTool):
"""Server reachable, but bundled WAN models missing -> DEGRADED, no
operation ready. Stands in for comfyui_video on a low-VRAM box."""
name = "comfyui_video"
capability = "video_generation"
provider = "comfyui"
supports = {"custom_workflow": True, "image_to_video": True}
input_schema = {"type": "object", "properties": {"workflow_json": {"type": "string"}}}
def get_status(self):
return ToolStatus.DEGRADED
def is_operation_available(self, operation):
return False
def execute(self, inputs):
raise AssertionError("not used")
class _DegradedComfyImage(BaseTool):
name = "comfyui_image"
capability = "image_generation"
provider = "comfyui"
supports = {"custom_workflow": True}
input_schema = {"type": "object", "properties": {"workflow_json": {"type": "string"}}}
def get_status(self):
return ToolStatus.DEGRADED
def execute(self, inputs):
raise AssertionError("not used")
class TestCustomWorkflowSelectorEligibility:
def test_video_selector_passes_degraded_tool_for_custom_workflow(self):
selector = VideoSelector()
candidates = [_DegradedComfyVideo()]
inputs = {
"prompt": "x",
"operation": "text_to_video",
"workflow_json": "{}",
"output_node": "14",
}
# Without the custom-workflow path this DEGRADED, operation-unready tool
# would be filtered out; with it, it is eligible and selectable.
filtered = selector._filter_candidates(inputs, candidates)
assert [t.name for t in filtered] == ["comfyui_video"]
assert selector._tool_selectable(candidates[0], inputs) is True
def test_video_selector_custom_workflow_requires_output_node(self):
selector = VideoSelector()
candidates = [_DegradedComfyVideo()]
inputs = {"prompt": "x", "operation": "text_to_video", "workflow_json": "{}"}
# output_node missing -> not eligible -> filtered out.
assert selector._filter_candidates(inputs, candidates) == []
assert selector._tool_selectable(candidates[0], inputs) is False
def test_video_selector_custom_workflow_needs_server(self):
class _OfflineComfyVideo(_DegradedComfyVideo):
def get_status(self):
return ToolStatus.UNAVAILABLE
selector = VideoSelector()
candidates = [_OfflineComfyVideo()]
inputs = {
"prompt": "x",
"operation": "text_to_video",
"workflow_json": "{}",
"output_node": "14",
}
assert selector._filter_candidates(inputs, candidates) == []
def test_image_selector_passes_degraded_tool_for_custom_workflow(self):
selector = ImageSelector()
candidates = [_DegradedComfyImage()]
inputs = {"prompt": "x", "workflow_json": "{}", "output_node": "13"}
filtered = selector._filter_candidates(inputs, candidates)
assert [t.name for t in filtered] == ["comfyui_image"]
assert selector._tool_selectable(candidates[0], inputs) is True
def test_image_selector_custom_workflow_requires_output_node(self):
selector = ImageSelector()
candidates = [_DegradedComfyImage()]
inputs = {"prompt": "x", "workflow_json": "{}"}
assert selector._filter_candidates(inputs, candidates) == []
assert selector._tool_selectable(candidates[0], inputs) is False
def test_selector_schemas_expose_custom_workflow_inputs(self):
for selector in (VideoSelector(), ImageSelector()):
props = selector.input_schema["properties"]
for field in (
"workflow_json",
"workflow_path",
"output_node",
"workflow_name",
"workflow_model",
"workflow_model_stack",
):
assert field in props, f"{selector.name} missing {field}"
+63
View File
@@ -1062,6 +1062,69 @@ def test_proposal_packet_schema_accepts_render_runtime():
assert props["render_runtime"]["enum"] == ["remotion", "hyperframes", "ffmpeg"]
def test_schemas_accept_voice_performance_contract():
root = Path(__file__).resolve().parent.parent.parent
script_schema = json.loads(
(root / "schemas" / "artifacts" / "script.schema.json").read_text(
encoding="utf-8"
)
)
assert "voice_performance" in script_schema["properties"]
section_props = script_schema["properties"]["sections"]["items"]["properties"]
assert "delivery_cues" in section_props
assert "provider_text" in section_props["delivery_cues"]["properties"]
proposal_schema = json.loads(
(root / "schemas" / "artifacts" / "proposal_packet.schema.json").read_text(
encoding="utf-8"
)
)
voice_selection = proposal_schema["properties"]["production_plan"]["properties"][
"voice_selection"
]["properties"]
assert "delivery_style" in voice_selection
assert "pacing_policy" in voice_selection
assert "sample_approval_required" in voice_selection
asset_schema = json.loads(
(root / "schemas" / "artifacts" / "asset_manifest.schema.json").read_text(
encoding="utf-8"
)
)
asset_props = asset_schema["properties"]["assets"]["items"]["properties"]
assert "voice_performance" in asset_props
assert "provider_settings" in asset_props["voice_performance"]["properties"]
def test_tts_provider_contracts_match_supported_fields():
from tools.audio.elevenlabs_tts import ElevenLabsTTS
from tools.audio.google_tts import GoogleTTS
from tools.audio.openai_tts import OpenAITTS
google_props = GoogleTTS.input_schema["properties"]
assert google_props["input_type"]["enum"] == ["text", "ssml"]
assert google_props["speaking_rate"]["maximum"] == 2.0
assert google_props["pitch"]["minimum"] == -20.0
assert google_props["pitch"]["maximum"] == 20.0
openai_props = OpenAITTS.input_schema["properties"]
assert "response_format" in openai_props
assert {"mp3", "opus", "aac", "flac", "wav", "pcm"}.issubset(
set(openai_props["response_format"]["enum"])
)
assert OpenAITTS._supports_instructions("gpt-4o-mini-tts")
assert not OpenAITTS._supports_instructions("tts-1")
assert not OpenAITTS._supports_instructions("tts-1-hd")
eleven_props = ElevenLabsTTS.input_schema["properties"]
assert {"stability", "similarity_boost", "style", "speed", "use_speaker_boost"}.issubset(
set(eleven_props)
)
assert eleven_props["speed"]["minimum"] == 0.7
assert eleven_props["speed"]["maximum"] == 1.2
def test_edit_decisions_schema_accepts_render_runtime():
schema_path = (
Path(__file__).resolve().parent.parent.parent
+1
View File
@@ -0,0 +1 @@
"""ComfyUI integration — shared client and bundled workflow templates."""
+296
View File
@@ -0,0 +1,296 @@
"""Thin REST client for a running ComfyUI server.
Handles the full generation cycle: submit workflow, poll for completion,
download artifacts. Used by comfyui_image, comfyui_video, and comfyui_music.
"""
from __future__ import annotations
import copy
import json
import os
import random
import time
from pathlib import Path
from typing import Any
import requests
class ComfyUIError(Exception):
"""Raised when ComfyUI returns an error or times out."""
class ComfyUIClient:
"""Client for the ComfyUI REST API.
The protocol is simple and battle-tested:
1. POST /prompt → queue a workflow, get a prompt_id
2. GET /history/{id} → poll until outputs appear
3. GET /view?filename=… → download the generated artifact
4. POST /upload/image → stage a local image for I2V workflows
"""
def __init__(self, server_url: str | None = None) -> None:
self.server_url = (
server_url
or os.environ.get("COMFYUI_SERVER_URL", "http://localhost:8188")
).rstrip("/")
# ------------------------------------------------------------------
# Health
# ------------------------------------------------------------------
@property
def is_default_url(self) -> bool:
"""True if using the fallback URL (user didn't set COMFYUI_SERVER_URL)."""
return not os.environ.get("COMFYUI_SERVER_URL")
def is_available(self) -> bool:
"""Return True if the ComfyUI server is reachable."""
try:
resp = requests.get(
f"{self.server_url}/system_stats", timeout=5
)
return resp.status_code == 200
except Exception:
return False
def unavailable_reason(self) -> str:
"""Human-readable explanation of why the server can't be reached."""
if self.is_default_url:
return (
f"No ComfyUI server found at {self.server_url} "
f"(default — no COMFYUI_SERVER_URL configured).\n"
f"Set COMFYUI_SERVER_URL in your .env file to the address of "
f"your ComfyUI server (e.g. http://localhost:8188)."
)
return (
f"ComfyUI server not reachable at {self.server_url}.\n"
f"Check that ComfyUI is running and the URL is correct."
)
# ------------------------------------------------------------------
# Model discovery
# ------------------------------------------------------------------
def list_models(self) -> dict[str, list[str]]:
"""Query ComfyUI for available models, grouped by type.
Returns a dict like::
{
"checkpoints": ["sd_xl_base.safetensors", ...],
"diffusion_models": ["flux2-dev-nvfp4.safetensors", ...],
"vae": ["ae.safetensors", ...],
"clip": ["clip_l.safetensors", ...],
"loras": ["my_lora.safetensors", ...],
}
"""
node_to_key = {
"CheckpointLoaderSimple": ("ckpt_name", "checkpoints"),
"UNETLoader": ("unet_name", "diffusion_models"),
"VAELoader": ("vae_name", "vae"),
"CLIPLoader": ("clip_name", "clip"),
"LoraLoaderModelOnly": ("lora_name", "loras"),
}
result: dict[str, list[str]] = {}
for node_class, (field, group) in node_to_key.items():
try:
resp = requests.get(
f"{self.server_url}/object_info/{node_class}", timeout=10
)
resp.raise_for_status()
data = resp.json()
options = (
data.get(node_class, {})
.get("input", {})
.get("required", {})
.get(field, [[]])[0]
)
if isinstance(options, list):
result[group] = options
except Exception:
result[group] = []
return result
def check_models(
self, required: list[str]
) -> tuple[list[str], list[str]]:
"""Check which of *required* model filenames are available.
Returns ``(found, missing)`` — two lists of filenames.
"""
all_models: set[str] = set()
for names in self.list_models().values():
all_models.update(names)
found = [m for m in required if m in all_models]
missing = [m for m in required if m not in all_models]
return found, missing
# ------------------------------------------------------------------
# Core cycle
# ------------------------------------------------------------------
def submit(self, workflow: dict) -> str:
"""Queue a workflow for execution. Returns the ``prompt_id``."""
resp = requests.post(
f"{self.server_url}/prompt",
json={"prompt": workflow},
timeout=30,
)
try:
data = resp.json()
except ValueError:
data = {}
if data.get("node_errors"):
raise ComfyUIError(f"Node errors: {json.dumps(data['node_errors'])}")
if data.get("error"):
raise ComfyUIError(f"Prompt error: {json.dumps(data['error'])}")
resp.raise_for_status()
prompt_id = data.get("prompt_id")
if not prompt_id:
raise ComfyUIError(f"No prompt_id in response: {data}")
return prompt_id
def poll(
self,
prompt_id: str,
*,
timeout: int = 600,
interval: int = 5,
) -> dict:
"""Block until *prompt_id* finishes. Returns the history entry."""
deadline = time.time() + timeout
while time.time() < deadline:
resp = requests.get(
f"{self.server_url}/history/{prompt_id}", timeout=10
)
resp.raise_for_status()
history = resp.json()
if prompt_id in history:
entry = history[prompt_id]
status = entry.get("status", {})
if status.get("status_str") == "error":
msgs = status.get("messages", [])
raise ComfyUIError(f"Execution error: {msgs}")
return entry
time.sleep(interval)
raise ComfyUIError(
f"Prompt {prompt_id} did not complete within {timeout}s"
)
def download(
self,
filename: str,
subfolder: str,
dest: Path,
folder_type: str = "output",
) -> Path:
"""Download an output artifact from the ComfyUI server."""
resp = requests.get(
f"{self.server_url}/view",
params={
"filename": filename,
"subfolder": subfolder,
"type": folder_type,
},
timeout=120,
)
resp.raise_for_status()
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_bytes(resp.content)
return dest
def upload_image(self, local_path: Path, name: str) -> str:
"""Upload a local image so it can be referenced by LoadImage nodes.
Returns the server-side filename.
"""
with open(local_path, "rb") as f:
resp = requests.post(
f"{self.server_url}/upload/image",
files={"image": (name, f, "image/png")},
timeout=30,
)
resp.raise_for_status()
return resp.json()["name"]
# ------------------------------------------------------------------
# High-level helper
# ------------------------------------------------------------------
def generate(
self,
workflow: dict,
output_node: str,
dest: Path,
*,
timeout: int = 600,
interval: int = 5,
) -> list[Path]:
"""Submit → poll → download. Returns list of artifact paths."""
prompt_id = self.submit(workflow)
entry = self.poll(prompt_id, timeout=timeout, interval=interval)
outputs = entry.get("outputs", {})
node_output = outputs.get(output_node, {})
# ComfyUI stores images and videos under the "images" key
items = node_output.get("images", []) or node_output.get("gifs", [])
if not items:
raise ComfyUIError(
f"No output artifacts on node {output_node}. "
f"Available nodes: {list(outputs.keys())}"
)
paths: list[Path] = []
for i, item in enumerate(items):
suffix = Path(item["filename"]).suffix
if len(items) == 1:
target = dest
else:
target = dest.with_stem(f"{dest.stem}_{i:03d}").with_suffix(suffix)
self.download(
item["filename"],
item.get("subfolder", ""),
target,
item.get("type", "output"),
)
paths.append(target)
return paths
# ------------------------------------------------------------------
# Workflow helpers
# ------------------------------------------------------------------
@staticmethod
def load_workflow(path: Path) -> dict:
"""Load a workflow JSON template from disk."""
with open(path) as f:
return json.load(f)
@staticmethod
def patch_workflow(
workflow: dict, patches: dict[str, dict[str, Any]]
) -> dict:
"""Deep-copy *workflow* and apply *patches*.
*patches* maps ``node_id`` → ``{input_name: value, ...}``.
"""
w = copy.deepcopy(workflow)
for node_id, values in patches.items():
if node_id not in w:
raise ComfyUIError(
f"Node {node_id!r} not found in workflow. "
f"Available: {list(w.keys())}"
)
for key, val in values.items():
w[node_id]["inputs"][key] = val
return w
@staticmethod
def random_seed() -> int:
"""Return a random seed suitable for ComfyUI noise nodes."""
return random.randint(0, 2**32 - 1)
+222
View File
@@ -0,0 +1,222 @@
"""Shared metadata helpers for ComfyUI provider tools."""
from __future__ import annotations
import hashlib
import json
from typing import Any
COMFYUI_SETUP_OFFER: dict[str, Any] = {
"kind": "local_server",
"fix_complexity": "1-minute env-var if ComfyUI is already running; otherwise local install",
"env_var": "COMFYUI_SERVER_URL",
"default_url": "http://localhost:8188",
"health_check": "GET /system_stats",
"what_it_unlocks": [
"free local image generation through ComfyUI workflows",
"free local video generation through ComfyUI workflows",
"community workflow_json/workflow_path execution",
],
}
BUNDLED_MODEL_STACKS: dict[str, list[dict[str, Any]]] = {
"flux2-txt2img": [
{
"role": "diffusion_model",
"name": "flux2-dev-nvfp4.safetensors",
"quantization": "NVFP4",
"destination_hint": "ComfyUI/models/diffusion_models/",
"download_url": (
"https://huggingface.co/black-forest-labs/FLUX.2-dev-NVFP4"
),
},
{
"role": "text_encoder",
"name": "mistral_3_small_flux2_fp4_mixed.safetensors",
"quantization": "FP4 mixed",
"destination_hint": "ComfyUI/models/text_encoders/",
"download_url": (
"https://huggingface.co/Comfy-Org/flux2-dev/tree/main/"
"split_files/text_encoders"
),
},
{
"role": "vae",
"name": "flux2-vae.safetensors",
"destination_hint": "ComfyUI/models/vae/",
"download_url": (
"https://huggingface.co/Comfy-Org/flux2-dev/blob/main/"
"split_files/vae/flux2-vae.safetensors"
),
},
],
"wan22-t2v-4step": [
{
"role": "text_encoder",
"name": "umt5_xxl_fp8_e4m3fn_scaled.safetensors",
"quantization": "FP8",
"destination_hint": "ComfyUI/models/text_encoders/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/"
"tree/main/split_files/text_encoders"
),
},
{
"role": "diffusion_model_high_noise",
"name": "wan2.2_t2v_high_noise_14B_fp8_scaled.safetensors",
"quantization": "FP8",
"destination_hint": "ComfyUI/models/diffusion_models/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/"
"blob/main/split_files/diffusion_models/"
"wan2.2_t2v_high_noise_14B_fp8_scaled.safetensors"
),
},
{
"role": "diffusion_model_low_noise",
"name": "wan2.2_t2v_low_noise_14B_fp8_scaled.safetensors",
"quantization": "FP8",
"destination_hint": "ComfyUI/models/diffusion_models/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/"
"tree/main/split_files/diffusion_models"
),
},
{
"role": "vae",
"name": "wan2.2_vae.safetensors",
"destination_hint": "ComfyUI/models/vae/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/"
"tree/main/split_files/vae"
),
},
{
"role": "lora",
"name": "wan2.2_t2v_lightx2v_4steps_lora_v1.1_high_noise.safetensors",
"strength_model": 1.0,
"destination_hint": "ComfyUI/models/loras/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/"
"tree/main/split_files/loras"
),
},
{
"role": "lora",
"name": "wan2.2_t2v_lightx2v_4steps_lora_v1.1_low_noise.safetensors",
"strength_model": 1.0,
"destination_hint": "ComfyUI/models/loras/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/"
"tree/main/split_files/loras"
),
},
],
"wan22-i2v-4step": [
{
"role": "text_encoder",
"name": "umt5_xxl_fp8_e4m3fn_scaled.safetensors",
"quantization": "FP8",
"destination_hint": "ComfyUI/models/text_encoders/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/"
"tree/main/split_files/text_encoders"
),
},
{
"role": "diffusion_model_high_noise",
"name": "wan2.2_i2v_high_noise_14B_fp8_scaled.safetensors",
"quantization": "FP8",
"destination_hint": "ComfyUI/models/diffusion_models/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/"
"blob/main/split_files/diffusion_models/"
"wan2.2_i2v_high_noise_14B_fp8_scaled.safetensors"
),
},
{
"role": "diffusion_model_low_noise",
"name": "wan2.2_i2v_low_noise_14B_fp8_scaled.safetensors",
"quantization": "FP8",
"destination_hint": "ComfyUI/models/diffusion_models/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/"
"tree/main/split_files/diffusion_models"
),
},
{
"role": "vae",
"name": "wan_2.1_vae.safetensors",
"destination_hint": "ComfyUI/models/vae/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/"
"tree/main/split_files/vae"
),
},
{
"role": "lora",
"name": "wan2.2_i2v_lightx2v_4steps_lora_v1_high_noise.safetensors",
"strength_model": 1.0,
"destination_hint": "ComfyUI/models/loras/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/"
"tree/main/split_files/loras"
),
},
{
"role": "lora",
"name": "wan2.2_i2v_lightx2v_4steps_lora_v1_low_noise.safetensors",
"strength_model": 1.0,
"destination_hint": "ComfyUI/models/loras/",
"download_url": (
"https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/"
"tree/main/split_files/loras"
),
},
],
}
def workflow_hash(workflow: dict[str, Any]) -> str:
"""Return a stable hash of the final workflow JSON submitted to ComfyUI."""
payload = json.dumps(workflow, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
def model_stack(workflow_key: str | None, inputs: dict[str, Any]) -> list[dict[str, Any]]:
"""Return bundled or caller-supplied model stack metadata."""
if workflow_key:
return [dict(item) for item in BUNDLED_MODEL_STACKS[workflow_key]]
stack = inputs.get("workflow_model_stack")
return stack if isinstance(stack, list) else []
def missing_models_payload(
missing: list[str],
*,
workflow_key: str,
workflow_name: str,
operation: str | None = None,
) -> dict[str, Any]:
"""Build a machine-readable missing-model error payload."""
stack_by_name = {
item["name"]: item for item in BUNDLED_MODEL_STACKS.get(workflow_key, [])
}
items = []
for name in missing:
meta = dict(stack_by_name.get(name, {}))
meta.setdefault("name", name)
meta.setdefault("role", "unknown")
meta.setdefault("destination_hint", "ComfyUI/models/ matching the workflow node")
meta.setdefault("download_url", None)
items.append(meta)
return {
"provider": "comfyui",
"workflow": workflow_name,
"operation": operation,
"missing_models": items,
"setup_offer": COMFYUI_SETUP_OFFER,
}
@@ -0,0 +1,96 @@
{
"1": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "flux2-dev-nvfp4.safetensors",
"weight_dtype": "default"
}
},
"2": {
"class_type": "CLIPLoader",
"inputs": {
"clip_name": "mistral_3_small_flux2_fp4_mixed.safetensors",
"type": "flux2",
"device": "cpu"
}
},
"3": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "flux2-vae.safetensors"
}
},
"4": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": ["2", 0],
"text": ""
}
},
"5": {
"class_type": "FluxGuidance",
"inputs": {
"conditioning": ["4", 0],
"guidance": 3.5
}
},
"6": {
"class_type": "EmptyFlux2LatentImage",
"inputs": {
"width": 1024,
"height": 1024,
"batch_size": 1
}
},
"7": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 42
}
},
"8": {
"class_type": "BasicGuider",
"inputs": {
"model": ["1", 0],
"conditioning": ["5", 0]
}
},
"9": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "euler"
}
},
"10": {
"class_type": "Flux2Scheduler",
"inputs": {
"steps": 20,
"width": 1024,
"height": 1024
}
},
"11": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": ["7", 0],
"guider": ["8", 0],
"sampler": ["9", 0],
"sigmas": ["10", 0],
"latent_image": ["6", 0]
}
},
"12": {
"class_type": "VAEDecode",
"inputs": {
"samples": ["11", 0],
"vae": ["3", 0]
}
},
"13": {
"class_type": "SaveImage",
"inputs": {
"images": ["12", 0],
"filename_prefix": "openmontage"
}
}
}
@@ -0,0 +1,154 @@
{
"84": {
"class_type": "CLIPLoader",
"inputs": {
"clip_name": "umt5_xxl_fp8_e4m3fn_scaled.safetensors",
"type": "wan",
"device": "default"
}
},
"89": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": ["84", 0],
"text": "oversaturated, overexposed, static, blurry details, subtitles, style, artwork, painting, still frame, gray overall, worst quality, low quality, JPEG artifacts, ugly, deformed, extra fingers, poorly drawn hands, poorly drawn face, deformed limbs, fused fingers, static frame, cluttered background, three legs, many people in background, walking backwards"
}
},
"90": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "wan_2.1_vae.safetensors"
}
},
"93": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": ["84", 0],
"text": ""
}
},
"95": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "wan2.2_i2v_high_noise_14B_fp8_scaled.safetensors",
"weight_dtype": "default"
}
},
"96": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "wan2.2_i2v_low_noise_14B_fp8_scaled.safetensors",
"weight_dtype": "default"
}
},
"97": {
"class_type": "LoadImage",
"inputs": {
"image": ""
}
},
"98": {
"class_type": "WanImageToVideo",
"inputs": {
"width": 640,
"height": 640,
"length": 81,
"batch_size": 1,
"positive": ["93", 0],
"negative": ["89", 0],
"vae": ["90", 0],
"start_image": ["97", 0]
}
},
"101": {
"class_type": "LoraLoaderModelOnly",
"inputs": {
"model": ["95", 0],
"lora_name": "wan2.2_i2v_lightx2v_4steps_lora_v1_high_noise.safetensors",
"strength_model": 1.0
}
},
"102": {
"class_type": "LoraLoaderModelOnly",
"inputs": {
"model": ["96", 0],
"lora_name": "wan2.2_i2v_lightx2v_4steps_lora_v1_low_noise.safetensors",
"strength_model": 1.0
}
},
"103": {
"class_type": "ModelSamplingSD3",
"inputs": {
"model": ["102", 0],
"shift": 5.0
}
},
"104": {
"class_type": "ModelSamplingSD3",
"inputs": {
"model": ["101", 0],
"shift": 5.0
}
},
"86": {
"class_type": "KSamplerAdvanced",
"inputs": {
"model": ["104", 0],
"positive": ["98", 0],
"negative": ["98", 1],
"latent_image": ["98", 2],
"add_noise": "enable",
"noise_seed": 42,
"control_after_generate": "randomize",
"steps": 4,
"cfg": 1.0,
"sampler_name": "euler",
"scheduler": "simple",
"start_at_step": 0,
"end_at_step": 2,
"return_with_leftover_noise": "enable"
}
},
"85": {
"class_type": "KSamplerAdvanced",
"inputs": {
"model": ["103", 0],
"positive": ["98", 0],
"negative": ["98", 1],
"latent_image": ["86", 0],
"add_noise": "disable",
"noise_seed": 0,
"control_after_generate": "fixed",
"steps": 4,
"cfg": 1.0,
"sampler_name": "euler",
"scheduler": "simple",
"start_at_step": 2,
"end_at_step": 4,
"return_with_leftover_noise": "disable"
}
},
"87": {
"class_type": "VAEDecode",
"inputs": {
"samples": ["85", 0],
"vae": ["90", 0]
}
},
"94": {
"class_type": "CreateVideo",
"inputs": {
"images": ["87", 0],
"fps": 16
}
},
"108": {
"class_type": "SaveVideo",
"inputs": {
"video": ["94", 0],
"filename_prefix": "openmontage_i2v",
"format": "auto",
"codec": "auto"
}
}
}
@@ -0,0 +1,143 @@
{
"1": {
"class_type": "CLIPLoader",
"inputs": {
"clip_name": "umt5_xxl_fp8_e4m3fn_scaled.safetensors",
"type": "wan",
"device": "default"
}
},
"2": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": ["1", 0],
"text": ""
}
},
"3": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": ["1", 0],
"text": "oversaturated, overexposed, static, blurry details, subtitles, style, artwork, painting, still frame, gray overall, worst quality, low quality, JPEG artifacts, ugly, deformed, extra fingers, poorly drawn hands, poorly drawn face, deformed limbs, fused fingers, static frame, cluttered background, three legs, many people in background, walking backwards"
}
},
"4": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "wan2.2_vae.safetensors"
}
},
"5": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "wan2.2_t2v_high_noise_14B_fp8_scaled.safetensors",
"weight_dtype": "default"
}
},
"6": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "wan2.2_t2v_low_noise_14B_fp8_scaled.safetensors",
"weight_dtype": "default"
}
},
"7": {
"class_type": "LoraLoaderModelOnly",
"inputs": {
"model": ["5", 0],
"lora_name": "wan2.2_t2v_lightx2v_4steps_lora_v1.1_high_noise.safetensors",
"strength_model": 1.0
}
},
"8": {
"class_type": "LoraLoaderModelOnly",
"inputs": {
"model": ["6", 0],
"lora_name": "wan2.2_t2v_lightx2v_4steps_lora_v1.1_low_noise.safetensors",
"strength_model": 1.0
}
},
"9": {
"class_type": "ModelSamplingSD3",
"inputs": {
"model": ["7", 0],
"shift": 5.0
}
},
"10": {
"class_type": "ModelSamplingSD3",
"inputs": {
"model": ["8", 0],
"shift": 5.0
}
},
"11": {
"class_type": "EmptyLatentImage",
"inputs": {
"width": 832,
"height": 480,
"batch_size": 81
}
},
"12": {
"class_type": "KSamplerAdvanced",
"inputs": {
"model": ["9", 0],
"positive": ["2", 0],
"negative": ["3", 0],
"latent_image": ["11", 0],
"add_noise": "enable",
"noise_seed": 42,
"control_after_generate": "randomize",
"steps": 4,
"cfg": 1.0,
"sampler_name": "euler",
"scheduler": "simple",
"start_at_step": 0,
"end_at_step": 2,
"return_with_leftover_noise": "enable"
}
},
"13": {
"class_type": "KSamplerAdvanced",
"inputs": {
"model": ["10", 0],
"positive": ["2", 0],
"negative": ["3", 0],
"latent_image": ["12", 0],
"add_noise": "disable",
"noise_seed": 0,
"control_after_generate": "fixed",
"steps": 4,
"cfg": 1.0,
"sampler_name": "euler",
"scheduler": "simple",
"start_at_step": 2,
"end_at_step": 4,
"return_with_leftover_noise": "disable"
}
},
"14": {
"class_type": "VAEDecode",
"inputs": {
"samples": ["13", 0],
"vae": ["4", 0]
}
},
"15": {
"class_type": "CreateVideo",
"inputs": {
"images": ["14", 0],
"fps": 16
}
},
"16": {
"class_type": "SaveVideo",
"inputs": {
"video": ["15", 0],
"filename_prefix": "openmontage_t2v",
"format": "auto",
"codec": "auto"
}
}
}
+29 -6
View File
@@ -96,6 +96,16 @@ class ElevenLabsTTS(BaseTool):
"minimum": 0,
"maximum": 1,
},
"speed": {
"type": "number",
"default": 1.0,
"minimum": 0.7,
"maximum": 1.2,
},
"use_speaker_boost": {
"type": "boolean",
"default": True,
},
"output_path": {"type": "string"},
"output_format": {
"type": "string",
@@ -109,7 +119,16 @@ class ElevenLabsTTS(BaseTool):
cpu_cores=1, ram_mb=256, vram_mb=0, disk_mb=50, network_required=True
)
retry_policy = RetryPolicy(max_retries=2, retryable_errors=["rate_limit", "timeout"])
idempotency_key_fields = ["text", "voice_id", "model_id"]
idempotency_key_fields = [
"text",
"voice_id",
"model_id",
"stability",
"similarity_boost",
"style",
"speed",
"use_speaker_boost",
]
side_effects = ["writes audio file to output_path", "calls ElevenLabs API"]
user_visible_verification = ["Listen to generated audio for natural speech quality"]
@@ -145,6 +164,13 @@ class ElevenLabsTTS(BaseTool):
voice_id = inputs.get("voice_id", self.DEFAULT_VOICE_ID)
model_id = inputs.get("model_id", "eleven_multilingual_v2")
output_format = inputs.get("output_format", "mp3_44100_128")
voice_settings = {
"stability": inputs.get("stability", 0.5),
"similarity_boost": inputs.get("similarity_boost", 0.75),
"style": inputs.get("style", 0.0),
"speed": inputs.get("speed", 1.0),
"use_speaker_boost": inputs.get("use_speaker_boost", True),
}
response = requests.post(
f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}",
@@ -156,11 +182,7 @@ class ElevenLabsTTS(BaseTool):
json={
"text": text,
"model_id": model_id,
"voice_settings": {
"stability": inputs.get("stability", 0.5),
"similarity_boost": inputs.get("similarity_boost", 0.75),
"style": inputs.get("style", 0.0),
},
"voice_settings": voice_settings,
},
params={"output_format": output_format},
timeout=120,
@@ -178,6 +200,7 @@ class ElevenLabsTTS(BaseTool):
"provider": self.provider,
"model": model_id,
"voice_id": voice_id,
"voice_settings": voice_settings,
"text_length": len(text),
"output": str(output_path),
"format": output_format,
+29 -3
View File
@@ -78,6 +78,12 @@ class GoogleTTS(BaseTool):
"required": ["text"],
"properties": {
"text": {"type": "string", "description": "Text to convert to speech"},
"input_type": {
"type": "string",
"default": "text",
"enum": ["text", "ssml"],
"description": "Set to 'ssml' when text contains SSML tags such as <speak> or <break>.",
},
"voice": {
"type": "string",
"default": "en-US-Chirp3-HD-Orus",
@@ -92,7 +98,7 @@ class GoogleTTS(BaseTool):
"type": "number",
"default": 1.0,
"minimum": 0.25,
"maximum": 4.0,
"maximum": 2.0,
"description": "Speaking speed. 1.0 = normal, 0.5 = half speed, 2.0 = double speed",
},
"pitch": {
@@ -116,7 +122,7 @@ class GoogleTTS(BaseTool):
cpu_cores=1, ram_mb=256, vram_mb=0, disk_mb=50, network_required=True
)
retry_policy = RetryPolicy(max_retries=2, retryable_errors=["rate_limit", "timeout"])
idempotency_key_fields = ["text", "voice", "language_code", "speaking_rate", "pitch"]
idempotency_key_fields = ["text", "input_type", "voice", "language_code", "speaking_rate", "pitch"]
side_effects = ["writes audio file to output_path", "calls Google Cloud TTS API"]
user_visible_verification = ["Listen to generated audio for natural speech quality"]
@@ -200,14 +206,33 @@ class GoogleTTS(BaseTool):
import requests
text = inputs["text"]
input_type = inputs.get("input_type", "text")
voice_name = inputs.get("voice", "en-US-Chirp3-HD-Orus")
language_code = inputs.get("language_code", "en-US")
speaking_rate = inputs.get("speaking_rate", 1.0)
pitch = inputs.get("pitch", 0.0)
audio_encoding = inputs.get("audio_encoding", "MP3")
if not 0.25 <= speaking_rate <= 2.0:
return ToolResult(
success=False,
error="Google TTS speaking_rate must be between 0.25 and 2.0.",
)
if not -20.0 <= pitch <= 20.0:
return ToolResult(
success=False,
error="Google TTS pitch must be between -20.0 and 20.0 semitones.",
)
if input_type == "ssml":
stripped = text.strip()
ssml = stripped if stripped.startswith("<speak") else f"<speak>{stripped}</speak>"
synthesis_input = {"ssml": ssml}
else:
synthesis_input = {"text": text}
payload = {
"input": {"text": text},
"input": synthesis_input,
"voice": {
"languageCode": language_code,
"name": voice_name,
@@ -253,6 +278,7 @@ class GoogleTTS(BaseTool):
"voice": voice_name,
"language_code": language_code,
"text_length": len(text),
"input_type": input_type,
"output": str(output_path),
"format": audio_encoding,
"speaking_rate": speaking_rate,
+34 -4
View File
@@ -80,10 +80,24 @@ class OpenAITTS(BaseTool):
"type": "string",
"default": "mp3",
"enum": ["mp3", "wav", "pcm"],
"description": "Backward-compatible alias for response_format.",
},
"response_format": {
"type": "string",
"default": "mp3",
"enum": ["mp3", "opus", "aac", "flac", "wav", "pcm"],
"description": "OpenAI speech response_format.",
},
"instructions": {
"type": "string",
"description": "Optional delivery instructions for the voice",
"description": "Optional delivery instructions. Supported by gpt-4o-mini-tts.",
},
"speed": {
"type": "number",
"default": 1.0,
"minimum": 0.25,
"maximum": 4.0,
"description": "OpenAI speech speed multiplier.",
},
"output_path": {"type": "string"},
},
@@ -93,7 +107,7 @@ class OpenAITTS(BaseTool):
cpu_cores=1, ram_mb=256, vram_mb=0, disk_mb=50, network_required=True
)
retry_policy = RetryPolicy(max_retries=2, retryable_errors=["rate_limit", "timeout"])
idempotency_key_fields = ["text", "voice", "model", "format"]
idempotency_key_fields = ["text", "voice", "model", "format", "response_format", "instructions", "speed"]
side_effects = ["writes audio file to output_path", "calls OpenAI API"]
user_visible_verification = ["Listen to generated audio for intelligibility and tone"]
@@ -105,6 +119,10 @@ class OpenAITTS(BaseTool):
def estimate_cost(self, inputs: dict[str, Any]) -> float:
return round(len(inputs.get("text", "")) * 0.000015, 4)
@staticmethod
def _supports_instructions(model: str) -> bool:
return model.startswith("gpt-4o-mini-tts")
def execute(self, inputs: dict[str, Any]) -> ToolResult:
if not os.environ.get("OPENAI_API_KEY"):
return ToolResult(success=False, error="No OpenAI API key. " + self.install_instructions)
@@ -124,11 +142,20 @@ class OpenAITTS(BaseTool):
from tools.analysis.audio_probe import probe_duration
client = OpenAI()
text = inputs["text"]
model = inputs.get("model", "gpt-4o-mini-tts")
voice = inputs.get("voice", "alloy")
fmt = inputs.get("format", "mp3")
fmt = inputs.get("response_format") or inputs.get("format", "mp3")
if inputs.get("instructions") and not self._supports_instructions(model):
return ToolResult(
success=False,
error=(
"OpenAI TTS instructions are only supported by "
"gpt-4o-mini-tts. Use that model or omit instructions."
),
)
client = OpenAI()
output_path = Path(inputs.get("output_path", f"openai_tts.{fmt}"))
output_path.parent.mkdir(parents=True, exist_ok=True)
@@ -155,6 +182,9 @@ class OpenAITTS(BaseTool):
"model": model,
"voice": voice,
"format": fmt,
"response_format": fmt,
"instructions": inputs.get("instructions"),
"speed": inputs.get("speed", 1.0),
"text_length": len(text),
"audio_duration_seconds": round(audio_duration, 2) if audio_duration else None,
"output": str(output_path),
+37
View File
@@ -61,6 +61,43 @@ class TTSSelector(BaseTool):
"type": "number", "minimum": 0, "maximum": 1,
"description": "Style exaggeration (ElevenLabs). Higher = more expressive.",
},
"instructions": {
"type": "string",
"description": "Provider-level delivery instructions for expressive narration when supported.",
},
"speaking_rate": {
"type": "number",
"minimum": 0.25,
"maximum": 2.0,
"description": "Google-style speakingRate control. Use speed for OpenAI/ElevenLabs-style controls.",
},
"speed": {
"type": "number",
"minimum": 0.25,
"maximum": 4.0,
"description": "Alias for speaking speed used by some providers.",
},
"pitch": {
"type": "number",
"minimum": -50,
"maximum": 50,
"description": "Provider-specific pitch control. Google TTS accepts -20..20; HeyGen-style providers may accept wider ranges.",
},
"input_type": {
"type": "string",
"enum": ["text", "ssml"],
"default": "text",
"description": "Use 'ssml' only when the selected provider supports tags such as <break>.",
},
"voice_performance": {
"type": "object",
"description": "Structured voice-performance plan or section delivery cues from the script artifact.",
},
"sample_mode": {
"type": "boolean",
"default": False,
"description": "True when generating an approval sample before batch narration.",
},
"output_format": {
"type": "string",
"description": "Audio output format (e.g. mp3_44100_128). Passed through to provider.",
+295
View File
@@ -0,0 +1,295 @@
"""ComfyUI image generation via a local or remote ComfyUI server.
Default workflow: FLUX 2 Dev (NVFP4) with Mistral text encoder.
Supports custom workflows via the ``workflow_json`` input.
"""
from __future__ import annotations
import json
import time
from pathlib import Path
from typing import Any
from tools.base_tool import (
BaseTool,
Determinism,
ExecutionMode,
ResourceProfile,
RetryPolicy,
ToolResult,
ToolRuntime,
ToolStability,
ToolStatus,
ToolTier,
)
from tools._comfyui.client import ComfyUIClient, ComfyUIError
from tools._comfyui.metadata import (
BUNDLED_MODEL_STACKS,
COMFYUI_SETUP_OFFER,
missing_models_payload,
model_stack,
workflow_hash,
)
_WORKFLOWS = Path(__file__).resolve().parent.parent / "_comfyui" / "workflows"
# Models required by the bundled flux2-txt2img workflow
_REQUIRED_MODELS = [
"flux2-dev-nvfp4.safetensors",
"mistral_3_small_flux2_fp4_mixed.safetensors",
"flux2-vae.safetensors",
]
class ComfyUIImage(BaseTool):
name = "comfyui_image"
version = "0.1.0"
tier = ToolTier.GENERATE
capability = "image_generation"
provider = "comfyui"
stability = ToolStability.EXPERIMENTAL
execution_mode = ExecutionMode.SYNC
determinism = Determinism.SEEDED
runtime = ToolRuntime.LOCAL_GPU
dependencies = [] # checked at runtime via server health
setup_offer = COMFYUI_SETUP_OFFER
install_instructions = (
"Start a ComfyUI server and set COMFYUI_SERVER_URL "
"(default http://localhost:8188).\n"
"See https://github.com/comfyanonymous/ComfyUI for setup."
)
agent_skills = ["comfyui", "flux-best-practices"]
capabilities = ["text_to_image"]
supports = {
"seed": True,
"custom_size": True,
"custom_workflow": True,
"custom_output_node": True,
"offline": True,
}
best_for = [
"local GPU generation without API costs",
"Blackwell / DGX Spark hardware where diffusers is unsupported",
"full control over sampling via custom ComfyUI workflows",
]
not_good_for = [
"setups without a running ComfyUI server",
"CPU-only machines",
]
fallback = "flux_image"
fallback_tools = ["flux_image", "local_diffusion", "openai_image"]
input_schema = {
"type": "object",
"required": ["prompt"],
"properties": {
"prompt": {"type": "string", "description": "Text prompt for image generation"},
"width": {"type": "integer", "default": 1024},
"height": {"type": "integer", "default": 1024},
"steps": {"type": "integer", "default": 20},
"guidance": {"type": "number", "default": 3.5},
"seed": {"type": "integer", "description": "Random if omitted"},
"output_path": {"type": "string", "description": "Where to save the image"},
"workflow_json": {
"type": "string",
"description": "Optional full ComfyUI workflow JSON. Requires output_node.",
},
"workflow_path": {
"type": "string",
"description": "Optional path to a ComfyUI workflow JSON file. Requires output_node.",
},
"output_node": {
"type": "string",
"description": "ComfyUI output node ID for custom workflow_json/workflow_path.",
},
"workflow_name": {
"type": "string",
"description": "Optional human-readable provenance label for a custom workflow.",
},
"workflow_model": {
"type": "string",
"description": "Optional model/provenance label for a custom workflow.",
},
"workflow_model_stack": {
"type": "array",
"description": (
"Optional provenance metadata for custom workflow dependencies. "
"Items should include name, role, quantization, and LoRA strengths when known."
),
"items": {"type": "object"},
},
},
}
resource_profile = ResourceProfile(
cpu_cores=2, ram_mb=8000, vram_mb=8000, disk_mb=500, network_required=False,
)
retry_policy = RetryPolicy(max_retries=1, retryable_errors=["timeout"])
idempotency_key_fields = ["prompt", "width", "height", "steps", "seed"]
side_effects = ["writes image file to output_path"]
user_visible_verification = ["Inspect generated image for quality and prompt adherence"]
def __init__(self) -> None:
self._client = ComfyUIClient()
def get_status(self) -> ToolStatus:
if not self._client.is_available():
return ToolStatus.UNAVAILABLE
_, missing = self._client.check_models(_REQUIRED_MODELS)
if missing:
return ToolStatus.DEGRADED
return ToolStatus.AVAILABLE
def estimate_cost(self, inputs: dict[str, Any]) -> float:
return 0.0
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
return float(inputs.get("steps", 20)) * 1.5
def get_info(self) -> dict[str, Any]:
info = super().get_info()
info["setup_offer"] = self.setup_offer
info["bundled_model_stack"] = BUNDLED_MODEL_STACKS["flux2-txt2img"]
return info
def execute(self, inputs: dict[str, Any]) -> ToolResult:
custom_workflow = bool(inputs.get("workflow_json") or inputs.get("workflow_path"))
if custom_workflow and not inputs.get("output_node"):
return ToolResult(
success=False,
error=(
"Custom ComfyUI workflows require output_node so OpenMontage "
"knows which ComfyUI node to download artifacts from."
),
)
if not self._client.is_available():
return ToolResult(
success=False,
error=self._client.unavailable_reason(),
)
if not custom_workflow:
_, missing = self._client.check_models(_REQUIRED_MODELS)
if missing:
return ToolResult(
success=False,
data=missing_models_payload(
missing,
workflow_key="flux2-txt2img",
workflow_name="flux2-txt2img.json",
),
error=(
f"ComfyUI server is running but missing required models: "
f"{', '.join(missing)}.\n"
f"See data.missing_models for destination hints and download URLs."
),
)
start = time.time()
seed = inputs.get("seed") or ComfyUIClient.random_seed()
width = inputs.get("width", 1024)
height = inputs.get("height", 1024)
steps = inputs.get("steps", 20)
guidance = inputs.get("guidance", 3.5)
output_path = Path(inputs.get("output_path", f"comfyui_image_{seed}.png"))
try:
if custom_workflow:
workflow = self._load_custom_workflow(inputs)
output_node = str(inputs["output_node"])
else:
workflow = ComfyUIClient.load_workflow(_WORKFLOWS / "flux2-txt2img.json")
workflow = ComfyUIClient.patch_workflow(workflow, {
"4": {"text": inputs["prompt"]},
"5": {"guidance": guidance},
"6": {"width": width, "height": height, "batch_size": 1},
"7": {"noise_seed": seed},
"10": {"steps": steps, "width": width, "height": height},
"13": {"filename_prefix": output_path.stem},
})
output_node = "13"
provenance = self._workflow_provenance(
inputs, custom_workflow, output_node, workflow
)
paths = self._client.generate(
workflow, output_node=output_node, dest=output_path, timeout=600,
)
except ComfyUIError as exc:
return ToolResult(success=False, error=str(exc))
except Exception as exc:
return ToolResult(success=False, error=f"ComfyUI image generation failed: {exc}")
model_name = self._model_name(inputs, custom_workflow)
return ToolResult(
success=True,
data={
"provider": "comfyui",
"model": model_name,
"prompt": inputs["prompt"],
"width": width,
"height": height,
"steps": steps,
"guidance": guidance,
"output": str(paths[0]),
"format": "png",
"workflow_provenance": provenance,
},
artifacts=[str(p) for p in paths],
cost_usd=0.0,
duration_seconds=round(time.time() - start, 2),
seed=seed,
model=model_name,
)
@staticmethod
def _load_custom_workflow(inputs: dict[str, Any]) -> dict:
if inputs.get("workflow_json"):
return json.loads(inputs["workflow_json"])
return ComfyUIClient.load_workflow(Path(inputs["workflow_path"]))
@staticmethod
def _model_name(inputs: dict[str, Any], custom_workflow: bool) -> str:
if not custom_workflow:
return "flux2-dev-nvfp4"
return (
inputs.get("workflow_model")
or inputs.get("model")
or inputs.get("workflow_name")
or "custom-comfyui-workflow"
)
@staticmethod
def _workflow_provenance(
inputs: dict[str, Any],
custom_workflow: bool,
output_node: str,
workflow: dict[str, Any],
) -> dict[str, Any]:
if not custom_workflow:
return {
"source": "bundled",
"workflow": "flux2-txt2img.json",
"workflow_hash_sha256": workflow_hash(workflow),
"model_stack": model_stack("flux2-txt2img", inputs),
"output_node": output_node,
}
return {
"source": "user_supplied",
"workflow_name": inputs.get("workflow_name"),
"workflow_path": inputs.get("workflow_path"),
"model": inputs.get("workflow_model") or inputs.get("model"),
"workflow_hash_sha256": workflow_hash(workflow),
"model_stack": model_stack(None, inputs),
"model_stack_source": (
"caller_supplied"
if inputs.get("workflow_model_stack")
else "unknown_custom_workflow"
),
"output_node": output_node,
}
+73 -1
View File
@@ -94,6 +94,38 @@ class ImageSelector(BaseTool):
"default": "generate",
"description": "Operation mode. 'rank' returns scored provider rankings without generating.",
},
"workflow_json": {
"type": "string",
"description": (
"Optional full ComfyUI workflow JSON. Routes to a custom-workflow-capable "
"provider (e.g. comfyui_image) based on server availability, not bundled "
"model readiness. Requires output_node."
),
},
"workflow_path": {
"type": "string",
"description": (
"Optional path to a ComfyUI workflow JSON file. Routes to a custom-workflow-"
"capable provider based on server availability. Requires output_node."
),
},
"output_node": {
"type": "string",
"description": "ComfyUI output node ID for a custom workflow_json/workflow_path.",
},
"workflow_name": {
"type": "string",
"description": "Optional human-readable provenance label for a custom workflow.",
},
"workflow_model": {
"type": "string",
"description": "Optional model/provenance label for a custom workflow.",
},
"workflow_model_stack": {
"type": "array",
"items": {"type": "object"},
"description": "Optional provenance metadata for custom workflow dependencies.",
},
"output_path": {"type": "string"},
},
}
@@ -184,6 +216,12 @@ class ImageSelector(BaseTool):
"image_path",
"image_urls",
"image_paths",
"workflow_json",
"workflow_path",
"output_node",
"workflow_name",
"workflow_model",
"workflow_model_stack",
):
if passthrough_key in adapted and passthrough_key not in props:
stripped.append(f"{passthrough_key}={adapted.pop(passthrough_key)}")
@@ -226,7 +264,7 @@ class ImageSelector(BaseTool):
tool_by_provider: dict[str, BaseTool] = {}
for tool in candidates:
if tool.provider not in tool_by_provider and tool.get_status() == ToolStatus.AVAILABLE:
if tool.provider not in tool_by_provider and self._tool_selectable(tool, inputs):
tool_by_provider[tool.provider] = tool
if preferred != "auto":
@@ -277,6 +315,12 @@ class ImageSelector(BaseTool):
return serialized
def _filter_candidates(self, inputs: dict[str, Any], candidates: list[BaseTool]) -> list[BaseTool]:
# A caller-supplied custom workflow is provider-specific (ComfyUI graph
# JSON). Route it only to custom-workflow-capable providers whose server
# is reachable — bundled-model readiness is irrelevant in that case.
if self._has_custom_workflow(inputs):
return [t for t in candidates if self._custom_workflow_eligible(t, inputs)]
wants_edit = (
inputs.get("generation_mode") == "edit"
or inputs.get("image_url")
@@ -296,3 +340,31 @@ class ImageSelector(BaseTool):
):
filtered.append(tool)
return filtered or candidates
@staticmethod
def _has_custom_workflow(inputs: dict[str, Any]) -> bool:
return bool(inputs.get("workflow_json") or inputs.get("workflow_path"))
def _custom_workflow_eligible(self, tool: BaseTool, inputs: dict[str, Any]) -> bool:
"""Whether a tool can run the caller-supplied custom workflow.
Eligibility is based on server availability, not bundled-model readiness:
a provider qualifies when it advertises ``custom_workflow`` support, an
``output_node`` is supplied, and its backend is reachable (status is not
UNAVAILABLE).
"""
if not self._has_custom_workflow(inputs):
return False
if not inputs.get("output_node"):
return False
supports = getattr(tool, "supports", {})
if not supports.get("custom_workflow"):
return False
return tool.get_status() != ToolStatus.UNAVAILABLE
def _tool_selectable(self, tool: BaseTool, inputs: dict[str, Any]) -> bool:
"""A provider is selectable if it is AVAILABLE, or if it can serve a
caller-supplied custom workflow even while bundled models report DEGRADED."""
if tool.get_status() == ToolStatus.AVAILABLE:
return True
return self._custom_workflow_eligible(tool, inputs)
+46
View File
@@ -282,6 +282,7 @@ class ToolRegistry:
"provider": tool.provider,
"runtime": tool.runtime.value,
"best_for": tool.best_for,
"dependencies": info.get("dependencies", []),
"install_instructions": tool.install_instructions,
"status": status.value,
}
@@ -291,6 +292,10 @@ class ToolRegistry:
"render_engines",
"remotion_note",
"provider_matrix",
"setup_offer",
"operation_statuses",
"resource_profiles",
"resource_profile_note",
):
if extra_key in info:
entry[extra_key] = info[extra_key]
@@ -398,6 +403,40 @@ class ToolRegistry:
setup_offers: list[dict[str, Any]] = []
for cap, bucket in menu.items():
for entry in bucket.get("unavailable", []):
offer = entry.get("setup_offer")
if offer:
setup_offers.append(
{
"capability": cap,
"tool": entry.get("name"),
"provider": entry.get("provider"),
"runtime": entry.get("runtime"),
"install_instructions": entry.get("install_instructions") or "",
**offer,
}
)
continue
env_vars = [
dep[4:]
for dep in entry.get("dependencies", [])
if isinstance(dep, str) and dep.startswith("env:")
]
if env_vars:
setup_offers.append(
{
"capability": cap,
"tool": entry.get("name"),
"provider": entry.get("provider"),
"runtime": entry.get("runtime"),
"kind": "env_var",
"fix_complexity": "1-minute env-var",
"env_vars": env_vars,
"install_instructions": entry.get("install_instructions") or "",
}
)
continue
hint = entry.get("install_instructions") or ""
# Heuristic: 1-minute fixes mention an env var or API key.
if any(k in hint.lower() for k in ["api key", "env", "_key=", "_api"]):
@@ -406,10 +445,17 @@ class ToolRegistry:
"capability": cap,
"tool": entry.get("name"),
"provider": entry.get("provider"),
"runtime": entry.get("runtime"),
"install_instructions": hint,
}
)
for entry in bucket.get("available", []) + bucket.get("unavailable", []):
if entry.get("resource_profile_note"):
runtime_warnings.append(
f"{entry.get('name')}: {entry.get('resource_profile_note')}"
)
result = {
"composition_runtimes": comp_runtimes,
"capabilities": capabilities,
+473
View File
@@ -0,0 +1,473 @@
"""ComfyUI video generation via a local or remote ComfyUI server.
Supports text-to-video and image-to-video using WAN 2.2 14B with
4-step LightX2V LoRA acceleration. Custom workflows are accepted
via the ``workflow_json`` input.
"""
from __future__ import annotations
import json
import time
from pathlib import Path
from typing import Any
import requests
from tools.base_tool import (
BaseTool,
Determinism,
ExecutionMode,
ResourceProfile,
RetryPolicy,
ToolResult,
ToolRuntime,
ToolStability,
ToolStatus,
ToolTier,
)
from tools._comfyui.client import ComfyUIClient, ComfyUIError
from tools._comfyui.metadata import (
BUNDLED_MODEL_STACKS,
COMFYUI_SETUP_OFFER,
missing_models_payload,
model_stack,
workflow_hash,
)
_WORKFLOWS = Path(__file__).resolve().parent.parent / "_comfyui" / "workflows"
# Output node IDs in the bundled workflows
_T2V_OUTPUT_NODE = "16"
_I2V_OUTPUT_NODE = "108"
# Models required by the bundled WAN 2.2 workflows
_REQUIRED_MODELS_COMMON = [
"umt5_xxl_fp8_e4m3fn_scaled.safetensors",
]
_REQUIRED_MODELS_I2V = [
*_REQUIRED_MODELS_COMMON,
"wan2.2_i2v_high_noise_14B_fp8_scaled.safetensors",
"wan2.2_i2v_low_noise_14B_fp8_scaled.safetensors",
"wan_2.1_vae.safetensors",
"wan2.2_i2v_lightx2v_4steps_lora_v1_high_noise.safetensors",
"wan2.2_i2v_lightx2v_4steps_lora_v1_low_noise.safetensors",
]
_REQUIRED_MODELS_T2V = [
*_REQUIRED_MODELS_COMMON,
"wan2.2_t2v_high_noise_14B_fp8_scaled.safetensors",
"wan2.2_t2v_low_noise_14B_fp8_scaled.safetensors",
"wan2.2_vae.safetensors",
"wan2.2_t2v_lightx2v_4steps_lora_v1.1_high_noise.safetensors",
"wan2.2_t2v_lightx2v_4steps_lora_v1.1_low_noise.safetensors",
]
_RESOURCE_PROFILES = {
"provider_floor": {
"vram_mb": 8000,
"ram_mb": 16000,
"applies_to": (
"ComfyUI provider availability and low-VRAM custom workflows. "
"Actual requirements depend on workflow_json/workflow_path."
),
},
"bundled_wan22_14b_fp8": {
"vram_mb": 16000,
"ram_mb": 32000,
"applies_to": (
"Bundled WAN 2.2 14B FP8 T2V/I2V workflows. This is not a "
"ComfyUI provider-wide requirement."
),
},
"low_vram_custom_workflows": {
"vram_mb": "8000-12000",
"ram_mb": "16000-32000",
"examples": [
"Wan 2.1 1.3B",
"LTX-Video / LTXV FP8 or quantized workflows",
"Wan 2.2 GGUF / quantized community workflows",
],
},
}
class ComfyUIVideo(BaseTool):
name = "comfyui_video"
version = "0.1.0"
tier = ToolTier.GENERATE
capability = "video_generation"
provider = "comfyui"
stability = ToolStability.EXPERIMENTAL
execution_mode = ExecutionMode.SYNC
determinism = Determinism.SEEDED
runtime = ToolRuntime.LOCAL_GPU
dependencies = []
setup_offer = COMFYUI_SETUP_OFFER
install_instructions = (
"Start a ComfyUI server and set COMFYUI_SERVER_URL "
"(default http://localhost:8188).\n"
"Requires WAN 2.2 models and LightX2V LoRAs in ComfyUI's model directory."
)
agent_skills = ["comfyui", "ai-video-gen", "ltx2"]
capabilities = ["text_to_video", "image_to_video"]
supports = {
"seed": True,
"reference_image": True,
"custom_workflow": True,
"custom_output_node": True,
"offline": True,
}
best_for = [
"local GPU video generation without API costs",
"Blackwell / DGX Spark hardware where diffusers is unsupported",
"image-to-video with WAN 2.2 14B (4-step accelerated)",
"text-to-video with WAN 2.2 14B (4-step accelerated)",
"custom low-VRAM ComfyUI workflows on 8GB-12GB GPUs",
]
not_good_for = [
"setups without a running ComfyUI server",
"CPU-only machines",
"running the bundled WAN 2.2 14B FP8 workflows on GPUs below 16GB VRAM",
]
fallback = "wan_video"
fallback_tools = ["wan_video", "hunyuan_video", "ltx_video_local", "kling_video"]
input_schema = {
"type": "object",
"required": ["prompt"],
"properties": {
"prompt": {"type": "string", "description": "Text prompt for video generation"},
"operation": {
"type": "string",
"enum": ["text_to_video", "image_to_video"],
"default": "text_to_video",
},
"reference_image_path": {
"type": "string",
"description": "Local path to reference image (for image_to_video)",
},
"reference_image_url": {
"type": "string",
"description": "URL of reference image (for image_to_video, downloaded first)",
},
"width": {"type": "integer", "default": 832, "description": "T2V default 832, I2V default 640"},
"height": {"type": "integer", "default": 480, "description": "T2V default 480, I2V default 640"},
"num_frames": {"type": "integer", "default": 81, "description": "81 frames = 5s at 16fps"},
"seed": {"type": "integer", "description": "Random if omitted"},
"output_path": {"type": "string", "description": "Where to save the video"},
"workflow_json": {
"type": "string",
"description": "Optional full ComfyUI workflow JSON. Requires output_node.",
},
"workflow_path": {
"type": "string",
"description": "Optional path to a ComfyUI workflow JSON file. Requires output_node.",
},
"output_node": {
"type": "string",
"description": "ComfyUI output node ID for custom workflow_json/workflow_path.",
},
"workflow_name": {
"type": "string",
"description": "Optional human-readable provenance label for a custom workflow.",
},
"workflow_model": {
"type": "string",
"description": "Optional model/provenance label for a custom workflow.",
},
"workflow_model_stack": {
"type": "array",
"description": (
"Optional provenance metadata for custom workflow dependencies. "
"Items should include name, role, quantization, scheduler, "
"and LoRA strengths when known."
),
"items": {"type": "object"},
},
},
}
resource_profile = ResourceProfile(
cpu_cores=2, ram_mb=16000, vram_mb=8000, disk_mb=2000, network_required=False,
)
retry_policy = RetryPolicy(max_retries=1, retryable_errors=["timeout"])
idempotency_key_fields = ["prompt", "operation", "width", "height", "num_frames", "seed"]
side_effects = ["writes video file to output_path"]
user_visible_verification = ["Watch generated clip for motion coherence and artifacts"]
def __init__(self) -> None:
self._client = ComfyUIClient()
def get_status(self) -> ToolStatus:
if not self._client.is_available():
return ToolStatus.UNAVAILABLE
statuses = self.operation_statuses()
if any(status == "available" for status in statuses.values()):
return ToolStatus.AVAILABLE
if statuses:
return ToolStatus.DEGRADED
return ToolStatus.UNAVAILABLE
def operation_statuses(self) -> dict[str, str]:
"""Return per-operation readiness for selector routing and preflight."""
if not self._client.is_available():
return {
"text_to_video": "unavailable",
"image_to_video": "unavailable",
}
_, missing_t2v = self._client.check_models(_REQUIRED_MODELS_T2V)
_, missing_i2v = self._client.check_models(_REQUIRED_MODELS_I2V)
return {
"text_to_video": "available" if not missing_t2v else "degraded",
"image_to_video": "available" if not missing_i2v else "degraded",
}
def is_operation_available(self, operation: str) -> bool:
if operation not in {"text_to_video", "image_to_video"}:
return False
return self.operation_statuses().get(operation) == "available"
def get_info(self) -> dict[str, Any]:
info = super().get_info()
info["operation_statuses"] = self.operation_statuses()
info["resource_profiles"] = _RESOURCE_PROFILES
info["setup_offer"] = self.setup_offer
info["bundled_model_stacks"] = {
"text_to_video": BUNDLED_MODEL_STACKS["wan22-t2v-4step"],
"image_to_video": BUNDLED_MODEL_STACKS["wan22-i2v-4step"],
}
info["resource_profile_note"] = (
"The top-level resource_profile is a ComfyUI provider floor, not a "
"promise that every workflow fits 8GB VRAM. Bundled WAN 2.2 14B FP8 "
"workflows recommend 16GB VRAM; custom low-VRAM workflows can target "
"8GB-12GB depending on model, quantization, resolution, and frame count."
)
return info
def estimate_cost(self, inputs: dict[str, Any]) -> float:
return 0.0
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
operation = inputs.get("operation", "text_to_video")
if operation == "image_to_video":
return 210.0 # ~3.5 min
return 240.0 # ~4 min
def execute(self, inputs: dict[str, Any]) -> ToolResult:
custom_workflow = bool(inputs.get("workflow_json") or inputs.get("workflow_path"))
if custom_workflow and not inputs.get("output_node"):
return ToolResult(
success=False,
error=(
"Custom ComfyUI workflows require output_node so OpenMontage "
"knows which ComfyUI node to download artifacts from."
),
)
if not self._client.is_available():
return ToolResult(
success=False,
error=self._client.unavailable_reason(),
)
operation = inputs.get("operation", "text_to_video")
if not custom_workflow:
required = _REQUIRED_MODELS_I2V if operation == "image_to_video" else _REQUIRED_MODELS_T2V
_, missing = self._client.check_models(required)
if missing:
workflow_key = (
"wan22-i2v-4step"
if operation == "image_to_video"
else "wan22-t2v-4step"
)
return ToolResult(
success=False,
data=missing_models_payload(
missing,
workflow_key=workflow_key,
workflow_name=f"{workflow_key}.json",
operation=operation,
),
error=(
f"ComfyUI server is running but missing models for {operation}: "
f"{', '.join(missing)}.\n"
f"See data.missing_models for destination hints and download URLs."
),
)
start = time.time()
seed = inputs.get("seed") or ComfyUIClient.random_seed()
output_path = Path(
inputs.get("output_path", f"comfyui_video_{operation}_{seed}.mp4")
)
try:
if custom_workflow:
workflow = self._load_custom_workflow(inputs)
output_node = str(inputs["output_node"])
elif operation == "image_to_video":
workflow, output_node = self._build_i2v(inputs, seed, output_path)
else:
workflow, output_node = self._build_t2v(inputs, seed, output_path)
provenance = self._workflow_provenance(
inputs, custom_workflow, output_node, operation, workflow
)
paths = self._client.generate(
workflow,
output_node=output_node,
dest=output_path,
timeout=900,
interval=10,
)
except ComfyUIError as exc:
return ToolResult(success=False, error=str(exc))
except Exception as exc:
return ToolResult(success=False, error=f"ComfyUI video generation failed: {exc}")
width = inputs.get("width", 832 if operation == "text_to_video" else 640)
height = inputs.get("height", 480 if operation == "text_to_video" else 640)
num_frames = inputs.get("num_frames", 81)
model_name = self._model_name(inputs, custom_workflow)
return ToolResult(
success=True,
data={
"provider": "comfyui",
"model": model_name,
"prompt": inputs["prompt"],
"operation": operation,
"width": width,
"height": height,
"num_frames": num_frames,
"fps": 16,
"duration_seconds": round(num_frames / 16, 2),
"output": str(paths[0]),
"format": "mp4",
"workflow_provenance": provenance,
},
artifacts=[str(p) for p in paths],
cost_usd=0.0,
duration_seconds=round(time.time() - start, 2),
seed=seed,
model=model_name,
)
# ------------------------------------------------------------------
# Workflow builders
# ------------------------------------------------------------------
def _build_t2v(
self, inputs: dict[str, Any], seed: int, output_path: Path
) -> tuple[dict, str]:
width = inputs.get("width", 832)
height = inputs.get("height", 480)
num_frames = inputs.get("num_frames", 81)
workflow = ComfyUIClient.load_workflow(_WORKFLOWS / "wan22-t2v-4step.json")
workflow = ComfyUIClient.patch_workflow(workflow, {
"2": {"text": inputs["prompt"]},
"11": {"width": width, "height": height, "batch_size": num_frames},
"12": {"noise_seed": seed},
"16": {"filename_prefix": output_path.stem},
})
return workflow, _T2V_OUTPUT_NODE
def _build_i2v(
self, inputs: dict[str, Any], seed: int, output_path: Path
) -> tuple[dict, str]:
width = inputs.get("width", 640)
height = inputs.get("height", 640)
num_frames = inputs.get("num_frames", 81)
# Resolve reference image
ref_path = inputs.get("reference_image_path")
ref_url = inputs.get("reference_image_url")
if ref_url and not ref_path:
# Download to a temp location
resp = requests.get(ref_url, timeout=60)
resp.raise_for_status()
ref_path = str(output_path.with_suffix(".ref.png"))
Path(ref_path).parent.mkdir(parents=True, exist_ok=True)
Path(ref_path).write_bytes(resp.content)
if not ref_path:
raise ComfyUIError(
"image_to_video requires reference_image_path or reference_image_url"
)
# Upload to ComfyUI
upload_name = f"om_{output_path.stem}.png"
server_name = self._client.upload_image(Path(ref_path), upload_name)
workflow = ComfyUIClient.load_workflow(_WORKFLOWS / "wan22-i2v-4step.json")
workflow = ComfyUIClient.patch_workflow(workflow, {
"93": {"text": inputs["prompt"]},
"97": {"image": server_name},
"98": {"width": width, "height": height, "length": num_frames},
"86": {"noise_seed": seed},
"108": {"filename_prefix": output_path.stem},
})
return workflow, _I2V_OUTPUT_NODE
@staticmethod
def _load_custom_workflow(inputs: dict[str, Any]) -> dict:
if inputs.get("workflow_json"):
return json.loads(inputs["workflow_json"])
return ComfyUIClient.load_workflow(Path(inputs["workflow_path"]))
@staticmethod
def _model_name(inputs: dict[str, Any], custom_workflow: bool) -> str:
if not custom_workflow:
return "wan2.2-14b-fp8-4step"
return (
inputs.get("workflow_model")
or inputs.get("model")
or inputs.get("workflow_name")
or "custom-comfyui-workflow"
)
@staticmethod
def _workflow_provenance(
inputs: dict[str, Any],
custom_workflow: bool,
output_node: str,
operation: str,
workflow: dict[str, Any],
) -> dict[str, Any]:
if not custom_workflow:
workflow_key = (
"wan22-i2v-4step"
if operation == "image_to_video"
else "wan22-t2v-4step"
)
return {
"source": "bundled",
"workflow": (
"wan22-i2v-4step.json"
if operation == "image_to_video"
else "wan22-t2v-4step.json"
),
"workflow_hash_sha256": workflow_hash(workflow),
"model_stack": model_stack(workflow_key, inputs),
"output_node": output_node,
}
return {
"source": "user_supplied",
"workflow_name": inputs.get("workflow_name"),
"workflow_path": inputs.get("workflow_path"),
"model": inputs.get("workflow_model") or inputs.get("model"),
"workflow_hash_sha256": workflow_hash(workflow),
"model_stack": model_stack(None, inputs),
"model_stack_source": (
"caller_supplied"
if inputs.get("workflow_model_stack")
else "unknown_custom_workflow"
),
"output_node": output_node,
}
+101 -7
View File
@@ -54,6 +54,12 @@ class VideoSelector(BaseTool):
"enum": ["text_to_video", "image_to_video", "reference_to_video", "rank"],
"default": "text_to_video",
},
"target_operation": {
"type": "string",
"enum": ["text_to_video", "image_to_video", "reference_to_video"],
"description": "Operation to score when operation='rank'.",
"default": "text_to_video",
},
"aspect_ratio": {
"type": "string",
"enum": ["16:9", "9:16", "1:1"],
@@ -90,6 +96,38 @@ class VideoSelector(BaseTool):
"type": "string",
"description": "Resolution hint for providers that support named output resolutions.",
},
"workflow_json": {
"type": "string",
"description": (
"Optional full ComfyUI workflow JSON. Routes to a custom-workflow-capable "
"provider (e.g. comfyui_video) based on server availability, not bundled "
"model readiness. Requires output_node."
),
},
"workflow_path": {
"type": "string",
"description": (
"Optional path to a ComfyUI workflow JSON file. Routes to a custom-workflow-"
"capable provider based on server availability. Requires output_node."
),
},
"output_node": {
"type": "string",
"description": "ComfyUI output node ID for a custom workflow_json/workflow_path.",
},
"workflow_name": {
"type": "string",
"description": "Optional human-readable provenance label for a custom workflow.",
},
"workflow_model": {
"type": "string",
"description": "Optional model/provenance label for a custom workflow.",
},
"workflow_model_stack": {
"type": "array",
"items": {"type": "object"},
"description": "Optional provenance metadata for custom workflow dependencies.",
},
"output_path": {"type": "string"},
},
}
@@ -137,11 +175,13 @@ class VideoSelector(BaseTool):
def execute(self, inputs: dict[str, object]) -> ToolResult:
from lib.scoring import rank_providers
task_context = self._prepare_task_context(inputs)
candidates = self._providers()
# Rank mode — return scored provider rankings without generating
if inputs.get("operation") == "rank":
rank_inputs = self._rank_inputs(inputs)
task_context = self._prepare_task_context(rank_inputs)
candidates = self._filter_candidates(rank_inputs, candidates)
rankings = rank_providers(candidates, task_context)
return ToolResult(
success=True,
@@ -153,6 +193,7 @@ class VideoSelector(BaseTool):
)
# Normal generation — use scored selection
task_context = self._prepare_task_context(inputs)
tool, score = self._select_best_tool(inputs, candidates, task_context)
if tool is None:
return ToolResult(success=False, error="No video generation provider available.")
@@ -222,10 +263,10 @@ class VideoSelector(BaseTool):
rankings = rank_providers(candidates, task_context)
# Build tool lookup: provider → tool (first available per provider)
# Build tool lookup: provider → tool (first selectable per provider)
tool_by_provider: dict[str, BaseTool] = {}
for tool in candidates:
if tool.provider not in tool_by_provider and tool.get_status() == ToolStatus.AVAILABLE:
if tool.provider not in tool_by_provider and self._tool_selectable(tool, inputs):
tool_by_provider[tool.provider] = tool
# If a preferred provider is explicitly requested and available,
@@ -252,6 +293,12 @@ class VideoSelector(BaseTool):
operation=str(inputs.get("operation", "text_to_video")),
)
@staticmethod
def _rank_inputs(inputs: dict[str, object]) -> dict[str, object]:
rank_inputs = dict(inputs)
rank_inputs["operation"] = inputs.get("target_operation", "text_to_video")
return rank_inputs
@staticmethod
def _tool_context_payload(tool: BaseTool) -> dict[str, object]:
info = tool.get_info()
@@ -283,25 +330,72 @@ class VideoSelector(BaseTool):
inputs: dict[str, object],
candidates: list[BaseTool],
) -> list[BaseTool]:
# A caller-supplied custom workflow is provider-specific (ComfyUI graph
# JSON). Route it only to custom-workflow-capable providers whose server
# is reachable — bundled-model readiness is irrelevant in that case.
if self._has_custom_workflow(inputs):
return [t for t in candidates if self._custom_workflow_eligible(t, inputs)]
operation = inputs.get("operation", "text_to_video")
if operation == "rank":
return candidates
operation = inputs.get("target_operation", "text_to_video")
filtered: list[BaseTool] = []
matched_operation = False
for tool in candidates:
supports = getattr(tool, "supports", {})
props = getattr(tool, "input_schema", {}).get("properties", {})
if operation == "image_to_video":
if supports.get("image_to_video") or "image_url" in props or "reference_image_url" in props:
filtered.append(tool)
matched_operation = True
if self._operation_ready(tool, "image_to_video"):
filtered.append(tool)
continue
if operation == "reference_to_video":
if supports.get("reference_to_video") or "reference_image_urls" in props:
matched_operation = True
filtered.append(tool)
continue
filtered.append(tool)
matched_operation = True
if self._operation_ready(tool, str(operation)):
filtered.append(tool)
return filtered or candidates
return filtered if matched_operation else candidates
@staticmethod
def _operation_ready(tool: BaseTool, operation: str) -> bool:
checker = getattr(tool, "is_operation_available", None)
if not callable(checker):
return True
return bool(checker(operation))
@staticmethod
def _has_custom_workflow(inputs: dict[str, object]) -> bool:
return bool(inputs.get("workflow_json") or inputs.get("workflow_path"))
def _custom_workflow_eligible(self, tool: BaseTool, inputs: dict[str, object]) -> bool:
"""Whether a tool can run the caller-supplied custom workflow.
Eligibility is based on server availability, not bundled-model readiness:
a provider qualifies when it advertises ``custom_workflow`` support, an
``output_node`` is supplied, and its backend is reachable (status is not
UNAVAILABLE).
"""
if not self._has_custom_workflow(inputs):
return False
if not inputs.get("output_node"):
return False
supports = getattr(tool, "supports", {})
if not supports.get("custom_workflow"):
return False
return tool.get_status() != ToolStatus.UNAVAILABLE
def _tool_selectable(self, tool: BaseTool, inputs: dict[str, object]) -> bool:
"""A provider is selectable if it is AVAILABLE, or if it can serve a
caller-supplied custom workflow even while bundled models report DEGRADED."""
if tool.get_status() == ToolStatus.AVAILABLE:
return True
return self._custom_workflow_eligible(tool, inputs)