Initial release — OpenMontage: the first open-source agentic video production system
11 production pipelines, 47 tools, 124 agent skills. Supports cloud APIs (fal.ai, OpenAI, ElevenLabs, Suno, HeyGen, Runway) and free local providers (diffusers, Piper TTS, WAN 2.1, Hunyuan, CogVideo). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
calesthio
@@ -0,0 +1,274 @@
|
||||
---
|
||||
name: speech-to-text
|
||||
description: Transcribe audio to text using ElevenLabs Scribe v2. Use when converting audio/video to text, generating subtitles, transcribing meetings, or processing spoken content.
|
||||
license: MIT
|
||||
compatibility: Requires internet access and an ElevenLabs API key (ELEVENLABS_API_KEY).
|
||||
metadata: {"openclaw": {"requires": {"env": ["ELEVENLABS_API_KEY"]}, "primaryEnv": "ELEVENLABS_API_KEY"}}
|
||||
---
|
||||
|
||||
# ElevenLabs Speech-to-Text
|
||||
|
||||
Transcribe audio to text with Scribe v2 - supports 90+ languages, speaker diarization, and word-level timestamps.
|
||||
|
||||
> **Setup:** See [Installation Guide](references/installation.md). For JavaScript, use `@elevenlabs/*` packages only.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Python
|
||||
|
||||
```python
|
||||
from elevenlabs import ElevenLabs
|
||||
|
||||
client = ElevenLabs()
|
||||
|
||||
with open("audio.mp3", "rb") as audio_file:
|
||||
result = client.speech_to_text.convert(file=audio_file, model_id="scribe_v2")
|
||||
|
||||
print(result.text)
|
||||
```
|
||||
|
||||
### JavaScript
|
||||
|
||||
```javascript
|
||||
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
|
||||
import { createReadStream } from "fs";
|
||||
|
||||
const client = new ElevenLabsClient();
|
||||
const result = await client.speechToText.convert({
|
||||
file: createReadStream("audio.mp3"),
|
||||
modelId: "scribe_v2",
|
||||
});
|
||||
console.log(result.text);
|
||||
```
|
||||
|
||||
### cURL
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.elevenlabs.io/v1/speech-to-text" \
|
||||
-H "xi-api-key: $ELEVENLABS_API_KEY" -F "file=@audio.mp3" -F "model_id=scribe_v2"
|
||||
```
|
||||
|
||||
## Models
|
||||
|
||||
| Model ID | Description | Best For |
|
||||
|----------|-------------|----------|
|
||||
| `scribe_v2` | State-of-the-art accuracy, 90+ languages | Batch transcription, subtitles, long-form audio |
|
||||
| `scribe_v2_realtime` | Low latency (~150ms) | Live transcription, voice agents |
|
||||
|
||||
## Transcription with Timestamps
|
||||
|
||||
Word-level timestamps include type classification and speaker identification:
|
||||
|
||||
```python
|
||||
result = client.speech_to_text.convert(
|
||||
file=audio_file, model_id="scribe_v2", timestamps_granularity="word"
|
||||
)
|
||||
|
||||
for word in result.words:
|
||||
print(f"{word.text}: {word.start}s - {word.end}s (type: {word.type})")
|
||||
|
||||
```
|
||||
|
||||
## Speaker Diarization
|
||||
|
||||
Identify WHO said WHAT - the model labels each word with a speaker ID, useful for meetings, interviews, or any multi-speaker audio:
|
||||
|
||||
```python
|
||||
result = client.speech_to_text.convert(
|
||||
file=audio_file,
|
||||
model_id="scribe_v2",
|
||||
diarize=True
|
||||
)
|
||||
|
||||
for word in result.words:
|
||||
print(f"[{word.speaker_id}] {word.text}")
|
||||
```
|
||||
|
||||
## Keyterm Prompting
|
||||
|
||||
Help the model recognize specific words it might otherwise mishear - product names, technical jargon, or unusual spellings (up to 100 terms):
|
||||
|
||||
```python
|
||||
result = client.speech_to_text.convert(
|
||||
file=audio_file,
|
||||
model_id="scribe_v2",
|
||||
keyterms=["ElevenLabs", "Scribe", "API"]
|
||||
)
|
||||
```
|
||||
|
||||
## Language Detection
|
||||
|
||||
Automatic detection with optional language hint:
|
||||
|
||||
```python
|
||||
result = client.speech_to_text.convert(
|
||||
file=audio_file,
|
||||
model_id="scribe_v2",
|
||||
language_code="eng" # ISO 639-1 or ISO 639-3 code
|
||||
)
|
||||
|
||||
print(f"Detected: {result.language_code} ({result.language_probability:.0%})")
|
||||
```
|
||||
|
||||
## Supported Formats
|
||||
|
||||
**Audio:** MP3, WAV, M4A, FLAC, OGG, WebM, AAC, AIFF, Opus
|
||||
**Video:** MP4, AVI, MKV, MOV, WMV, FLV, WebM, MPEG, 3GPP
|
||||
|
||||
**Limits:** Up to 3GB file size, 10 hours duration
|
||||
|
||||
## Response Format
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "The full transcription text",
|
||||
"language_code": "eng",
|
||||
"language_probability": 0.98,
|
||||
"words": [
|
||||
{"text": "The", "start": 0.0, "end": 0.15, "type": "word", "speaker_id": "speaker_0"},
|
||||
{"text": " ", "start": 0.15, "end": 0.16, "type": "spacing", "speaker_id": "speaker_0"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Word types:**
|
||||
- `word` - An actual spoken word
|
||||
- `spacing` - Whitespace between words (useful for precise timing)
|
||||
- `audio_event` - Non-speech sounds the model detected (laughter, applause, music, etc.)
|
||||
|
||||
## Error Handling
|
||||
|
||||
```python
|
||||
try:
|
||||
result = client.speech_to_text.convert(file=audio_file, model_id="scribe_v2")
|
||||
except Exception as e:
|
||||
print(f"Transcription failed: {e}")
|
||||
```
|
||||
|
||||
Common errors:
|
||||
- **401**: Invalid API key
|
||||
- **422**: Invalid parameters
|
||||
- **429**: Rate limit exceeded
|
||||
|
||||
## Tracking Costs
|
||||
|
||||
Monitor usage via `request-id` response header:
|
||||
|
||||
```python
|
||||
response = client.speech_to_text.convert.with_raw_response(file=audio_file, model_id="scribe_v2")
|
||||
result = response.parse()
|
||||
print(f"Request ID: {response.headers.get('request-id')}")
|
||||
```
|
||||
|
||||
## Real-Time Streaming
|
||||
|
||||
For live transcription with ultra-low latency (~150ms), use the real-time API. The real-time API produces two types of transcripts:
|
||||
|
||||
- **Partial transcripts**: Interim results that update frequently as audio is processed - use these for live feedback (e.g., showing text as the user speaks)
|
||||
- **Committed transcripts**: Final, stable results after you "commit" - use these as the source of truth for your application
|
||||
|
||||
A "commit" tells the model to finalize the current segment. You can commit manually (e.g., when the user pauses) or use Voice Activity Detection (VAD) to auto-commit on silence.
|
||||
|
||||
### Python (Server-Side)
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from elevenlabs import ElevenLabs
|
||||
|
||||
client = ElevenLabs()
|
||||
|
||||
async def transcribe_realtime():
|
||||
async with client.speech_to_text.realtime.connect(
|
||||
model_id="scribe_v2_realtime",
|
||||
include_timestamps=True,
|
||||
) as connection:
|
||||
await connection.stream_url("https://example.com/audio.mp3")
|
||||
|
||||
async for event in connection:
|
||||
if event.type == "partial_transcript":
|
||||
print(f"Partial: {event.text}")
|
||||
elif event.type == "committed_transcript":
|
||||
print(f"Final: {event.text}")
|
||||
|
||||
asyncio.run(transcribe_realtime())
|
||||
```
|
||||
|
||||
### JavaScript (Client-Side with React)
|
||||
|
||||
```typescript
|
||||
import { useScribe, CommitStrategy } from "@elevenlabs/react";
|
||||
|
||||
function TranscriptionComponent() {
|
||||
const [transcript, setTranscript] = useState("");
|
||||
|
||||
const scribe = useScribe({
|
||||
modelId: "scribe_v2_realtime",
|
||||
commitStrategy: CommitStrategy.VAD, // Auto-commit on silence for mic input
|
||||
onPartialTranscript: (data) => console.log("Partial:", data.text),
|
||||
onCommittedTranscript: (data) => setTranscript((prev) => prev + data.text),
|
||||
});
|
||||
|
||||
const start = async () => {
|
||||
// Get token from your backend (never expose API key to client)
|
||||
const { token } = await fetch("/scribe-token").then((r) => r.json());
|
||||
|
||||
await scribe.connect({
|
||||
token,
|
||||
microphone: { echoCancellation: true, noiseSuppression: true },
|
||||
});
|
||||
};
|
||||
|
||||
return <button onClick={start}>Start Recording</button>;
|
||||
}
|
||||
```
|
||||
|
||||
### Commit Strategies
|
||||
|
||||
| Strategy | Description |
|
||||
|----------|-------------|
|
||||
| **Manual** | You call `commit()` when ready - use for file processing or when you control the audio segments |
|
||||
| **VAD** | Voice Activity Detection auto-commits when silence is detected - use for live microphone input |
|
||||
|
||||
```typescript
|
||||
// React: set commitStrategy on the hook (recommended for mic input)
|
||||
import { useScribe, CommitStrategy } from "@elevenlabs/react";
|
||||
|
||||
const scribe = useScribe({
|
||||
modelId: "scribe_v2_realtime",
|
||||
commitStrategy: CommitStrategy.VAD,
|
||||
// Optional VAD tuning:
|
||||
vadSilenceThresholdSecs: 1.5,
|
||||
vadThreshold: 0.4,
|
||||
});
|
||||
```
|
||||
|
||||
```javascript
|
||||
// JavaScript client: pass vad config on connect
|
||||
const connection = await client.speechToText.realtime.connect({
|
||||
modelId: "scribe_v2_realtime",
|
||||
vad: {
|
||||
silenceThresholdSecs: 1.5,
|
||||
threshold: 0.4,
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### Event Types
|
||||
|
||||
| Event | Description |
|
||||
|-------|-------------|
|
||||
| `partial_transcript` | Live interim results |
|
||||
| `committed_transcript` | Final results after commit |
|
||||
| `committed_transcript_with_timestamps` | Final with word timing |
|
||||
| `error` | Error occurred |
|
||||
|
||||
See real-time references for complete documentation.
|
||||
|
||||
## References
|
||||
|
||||
- [Installation Guide](references/installation.md)
|
||||
- [Transcription Options](references/transcription-options.md)
|
||||
- [Real-Time Client-Side Streaming](references/realtime-client-side.md)
|
||||
- [Real-Time Server-Side Streaming](references/realtime-server-side.md)
|
||||
- [Commit Strategies](references/realtime-commit-strategies.md)
|
||||
- [Real-Time Event Reference](references/realtime-events.md)
|
||||
@@ -0,0 +1,92 @@
|
||||
# Installation
|
||||
|
||||
## JavaScript / TypeScript
|
||||
|
||||
```bash
|
||||
npm install @elevenlabs/elevenlabs-js
|
||||
```
|
||||
|
||||
> **Important:** Always use `@elevenlabs/elevenlabs-js`. The old `elevenlabs` npm package (v1.x) is deprecated and should not be used.
|
||||
|
||||
```javascript
|
||||
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
|
||||
|
||||
// Option 1: Environment variable (recommended)
|
||||
// Set ELEVENLABS_API_KEY in your environment
|
||||
const client = new ElevenLabsClient();
|
||||
|
||||
// Option 2: Pass directly
|
||||
const client = new ElevenLabsClient({ apiKey: "your-api-key" });
|
||||
```
|
||||
|
||||
### Migrating from deprecated packages
|
||||
|
||||
If you have old packages installed, remove them:
|
||||
|
||||
```bash
|
||||
# Remove deprecated packages
|
||||
npm uninstall elevenlabs
|
||||
|
||||
# Install the current packages
|
||||
npm install @elevenlabs/elevenlabs-js
|
||||
|
||||
# For client-side/browser usage, also install:
|
||||
npm install @elevenlabs/client # Browser client
|
||||
npm install @elevenlabs/react # React hooks
|
||||
```
|
||||
|
||||
**Import changes:**
|
||||
```javascript
|
||||
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
|
||||
import { Scribe } from "@elevenlabs/client";
|
||||
import { useScribe } from "@elevenlabs/react";
|
||||
```
|
||||
|
||||
## Python
|
||||
|
||||
```bash
|
||||
pip install elevenlabs
|
||||
```
|
||||
|
||||
```python
|
||||
from elevenlabs import ElevenLabs
|
||||
|
||||
# Option 1: Environment variable (recommended)
|
||||
# Set ELEVENLABS_API_KEY in your environment
|
||||
client = ElevenLabs()
|
||||
|
||||
# Option 2: Pass directly
|
||||
client = ElevenLabs(api_key="your-api-key")
|
||||
```
|
||||
|
||||
## cURL / REST API
|
||||
|
||||
Set your API key as an environment variable:
|
||||
|
||||
```bash
|
||||
export ELEVENLABS_API_KEY="your-api-key"
|
||||
```
|
||||
|
||||
Include in requests via the `xi-api-key` header:
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.elevenlabs.io/v1/speech-to-text" \
|
||||
-H "xi-api-key: $ELEVENLABS_API_KEY" \
|
||||
-F "file=@audio.mp3" \
|
||||
-F "model_id=scribe_v2"
|
||||
```
|
||||
|
||||
## Getting an API Key
|
||||
|
||||
1. Sign up at [elevenlabs.io](https://elevenlabs.io)
|
||||
2. Go to [API Keys](https://elevenlabs.io/app/settings/api-keys)
|
||||
3. Click **Create API Key**
|
||||
4. Copy and store securely
|
||||
|
||||
Or use the `setup-api-key` skill for guided setup.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `ELEVENLABS_API_KEY` | Your ElevenLabs API key (required) |
|
||||
@@ -0,0 +1,192 @@
|
||||
# Client-Side Real-Time Streaming
|
||||
|
||||
Stream audio from the browser directly to ElevenLabs for real-time transcription.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# React
|
||||
npm install @elevenlabs/react @elevenlabs/elevenlabs-js
|
||||
|
||||
# JavaScript
|
||||
npm install @elevenlabs/client @elevenlabs/elevenlabs-js
|
||||
```
|
||||
|
||||
> **Warning:** Always use the `@elevenlabs/*` namespace for client-side packages.
|
||||
|
||||
## Token Generation
|
||||
|
||||
Client-side streaming requires a single-use token to protect your API key. Generate tokens on your backend:
|
||||
|
||||
```typescript
|
||||
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
|
||||
|
||||
const elevenlabs = new ElevenLabsClient({
|
||||
apiKey: process.env.ELEVENLABS_API_KEY,
|
||||
});
|
||||
|
||||
app.get("/scribe-token", yourAuthMiddleware, async (req, res) => {
|
||||
const token = await elevenlabs.tokens.singleUse.create("realtime_scribe");
|
||||
res.json(token);
|
||||
});
|
||||
```
|
||||
|
||||
**Note:** Single-use tokens expire after 15 minutes.
|
||||
|
||||
## React Implementation
|
||||
|
||||
```typescript
|
||||
import { useScribe, CommitStrategy } from "@elevenlabs/react";
|
||||
|
||||
function TranscriptionComponent() {
|
||||
const [transcript, setTranscript] = useState("");
|
||||
|
||||
const scribe = useScribe({
|
||||
modelId: "scribe_v2_realtime",
|
||||
commitStrategy: CommitStrategy.VAD, // Auto-commit on silence for mic input
|
||||
onPartialTranscript: (data) => {
|
||||
// Show live feedback as user speaks
|
||||
console.log("Partial:", data.text);
|
||||
},
|
||||
onCommittedTranscript: (data) => {
|
||||
// Final transcript for this segment
|
||||
setTranscript((prev) => prev + data.text);
|
||||
},
|
||||
});
|
||||
|
||||
const startRecording = async () => {
|
||||
const tokenResponse = await fetch("/scribe-token");
|
||||
const { token } = await tokenResponse.json();
|
||||
|
||||
await scribe.connect({
|
||||
token,
|
||||
microphone: {
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
autoGainControl: true,
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
const stopRecording = () => {
|
||||
scribe.disconnect();
|
||||
};
|
||||
|
||||
return (
|
||||
<div>
|
||||
<div>Status: {scribe.status}</div>
|
||||
<button onClick={startRecording}>Start</button>
|
||||
<button onClick={stopRecording}>Stop</button>
|
||||
<p>{transcript}</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
> **Important:** The default commit strategy is `CommitStrategy.MANUAL`, which requires you to call `scribe.commit()` explicitly. For microphone input, always set `CommitStrategy.VAD` so the server auto-commits when silence is detected. Without this, committed transcripts will never fire and the connection may drop.
|
||||
|
||||
### `scribe.status` Values
|
||||
|
||||
| Status | Meaning |
|
||||
|--------|---------|
|
||||
| `"disconnected"` | No active connection |
|
||||
| `"connecting"` | Connection is being established |
|
||||
| `"connected"` | Connected and ready to receive audio |
|
||||
| `"transcribing"` | Actively processing speech (transitions from `"connected"` when audio is detected or VAD commits) |
|
||||
| `"error"` | An error occurred |
|
||||
|
||||
> **Important:** When checking if the session is active, always check for both `"connected"` and `"transcribing"`. The status transitions to `"transcribing"` during speech processing, so checking only `"connected"` will cause UI elements (buttons, waveforms, indicators) to incorrectly reset mid-session.
|
||||
|
||||
```typescript
|
||||
// Correct - handles both active states
|
||||
const isListening = scribe.status === "connected" || scribe.status === "transcribing";
|
||||
|
||||
// Wrong - will flicker/reset when VAD commits
|
||||
const isListening = scribe.status === "connected";
|
||||
```
|
||||
|
||||
## JavaScript Implementation
|
||||
|
||||
```typescript
|
||||
import { Scribe, RealtimeEvents } from "@elevenlabs/client";
|
||||
|
||||
async function startTranscription() {
|
||||
const tokenResponse = await fetch("/scribe-token");
|
||||
const { token } = await tokenResponse.json();
|
||||
|
||||
const connection = Scribe.connect({
|
||||
token,
|
||||
modelId: "scribe_v2_realtime",
|
||||
includeTimestamps: true,
|
||||
microphone: {
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
autoGainControl: true,
|
||||
},
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.OPEN, () => {
|
||||
console.log("Connected");
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.PARTIAL_TRANSCRIPT, (data) => {
|
||||
console.log("Partial:", data.text);
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.COMMITTED_TRANSCRIPT, (data) => {
|
||||
console.log("Committed:", data.text);
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.COMMITTED_TRANSCRIPT_WITH_TIMESTAMPS, (data) => {
|
||||
for (const word of data.words) {
|
||||
console.log(`${word.text}: ${word.start}s - ${word.end}s`);
|
||||
}
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.ERROR, (error) => {
|
||||
console.error("Error:", error);
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.CLOSE, () => {
|
||||
console.log("Disconnected");
|
||||
});
|
||||
|
||||
return connection;
|
||||
}
|
||||
```
|
||||
|
||||
## Manual Audio Chunking
|
||||
|
||||
For file uploads or custom audio sources, encode to PCM-16 and send in chunks:
|
||||
|
||||
```typescript
|
||||
const chunkSize = 4096;
|
||||
|
||||
for (let offset = 0; offset < pcmData.length; offset += chunkSize) {
|
||||
const chunk = pcmData.slice(offset, offset + chunkSize);
|
||||
const bytes = new Uint8Array(chunk.buffer);
|
||||
const base64 = btoa(String.fromCharCode(...bytes));
|
||||
|
||||
scribe.sendAudio(base64);
|
||||
|
||||
// Simulate real-time streaming
|
||||
await new Promise((resolve) => setTimeout(resolve, 50));
|
||||
}
|
||||
|
||||
// Finalize transcription
|
||||
scribe.commit();
|
||||
```
|
||||
|
||||
## Microphone Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `echoCancellation` | Remove echo from speakers |
|
||||
| `noiseSuppression` | Filter background noise |
|
||||
| `autoGainControl` | Normalize volume levels |
|
||||
|
||||
## Security
|
||||
|
||||
- Never expose your API key to the client
|
||||
- Always generate single-use tokens on your backend
|
||||
- Use authentication middleware to protect token endpoints
|
||||
@@ -0,0 +1,144 @@
|
||||
# Transcripts and Commit Strategies
|
||||
|
||||
Control when and how transcripts are finalized in real-time streaming.
|
||||
|
||||
## Why Commits Matter
|
||||
|
||||
In real-time transcription, the model continuously refines its understanding as more audio arrives. A word that sounds like "their" might become "there" or "they're" once more context is heard. The **commit** mechanism lets you decide when to "lock in" the transcript.
|
||||
|
||||
## Transcript Types
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| **Partial** | Interim "best guess" results that update frequently as audio is processed. Use for live feedback (showing text as the user speaks), but don't save these - they may change. |
|
||||
| **Committed** | Final, stable results after a commit occurs. Use these as the source of truth for your application - they won't change. |
|
||||
| **Committed with Timestamps** | Same as committed, but includes word-level timing data for subtitles, karaoke, or lip-sync. |
|
||||
|
||||
## Manual Commit (Default)
|
||||
|
||||
You explicitly control when transcript segments finalize.
|
||||
|
||||
### Python
|
||||
|
||||
```python
|
||||
async with client.speech_to_text.realtime.connect(
|
||||
model_id="scribe_v2_realtime",
|
||||
) as connection:
|
||||
# Send audio
|
||||
await connection.send({
|
||||
"audio_base_64": audio_base_64,
|
||||
"sample_rate": 16000,
|
||||
})
|
||||
|
||||
# Commit when ready (e.g., pause in speech, end of sentence)
|
||||
await connection.commit()
|
||||
```
|
||||
|
||||
### JavaScript
|
||||
|
||||
```javascript
|
||||
const connection = await client.speechToText.realtime.connect({
|
||||
modelId: "scribe_v2_realtime",
|
||||
});
|
||||
|
||||
// Send audio
|
||||
connection.send({
|
||||
audioBase64: audioBase64,
|
||||
sampleRate: 16000,
|
||||
});
|
||||
|
||||
// Commit when ready
|
||||
connection.commit();
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
- **Commit every 20-30 seconds** for optimal performance
|
||||
- **Commit during silence** or logical breaks (end of sentence, speaker change)
|
||||
- **Auto-commit at 90 seconds** if no manual commit is sent
|
||||
|
||||
### Providing Context
|
||||
|
||||
Send previous text with the first audio chunk to help the model:
|
||||
|
||||
```python
|
||||
await connection.send({
|
||||
"audio_base_64": first_chunk,
|
||||
"sample_rate": 16000,
|
||||
"previous_text": "So as I was saying," # Keep under 50 characters
|
||||
})
|
||||
```
|
||||
|
||||
This helps with:
|
||||
- Continuing conversations after reconnection
|
||||
- Providing context for better accuracy
|
||||
- Handling sentence fragments
|
||||
|
||||
## Voice Activity Detection (VAD)
|
||||
|
||||
VAD listens for silence and automatically commits when the speaker pauses. This creates natural transcript segments that match how people actually speak - pausing between sentences and thoughts. Recommended for live microphone input.
|
||||
|
||||
### Configuration
|
||||
|
||||
#### React (`useScribe`)
|
||||
|
||||
```typescript
|
||||
import { useScribe, CommitStrategy } from "@elevenlabs/react";
|
||||
|
||||
const scribe = useScribe({
|
||||
modelId: "scribe_v2_realtime",
|
||||
commitStrategy: CommitStrategy.VAD,
|
||||
// Optional VAD tuning:
|
||||
vadSilenceThresholdSecs: 1.5, // Silence duration before commit
|
||||
vadThreshold: 0.4, // Speech detection sensitivity (0-1)
|
||||
minSpeechDurationMs: 100, // Minimum speech length required
|
||||
minSilenceDurationMs: 100, // Minimum silence length required
|
||||
});
|
||||
```
|
||||
|
||||
> **Important:** The default is `CommitStrategy.MANUAL`. For microphone input, always set `CommitStrategy.VAD` — without it, committed transcripts will never fire and the connection may drop.
|
||||
|
||||
#### JavaScript client
|
||||
|
||||
```javascript
|
||||
const connection = await client.speechToText.realtime.connect({
|
||||
modelId: "scribe_v2_realtime",
|
||||
vad: {
|
||||
silenceThresholdSecs: 1.5, // Silence duration before commit
|
||||
threshold: 0.4, // Speech detection sensitivity (0-1)
|
||||
minSpeechDurationMs: 100, // Minimum speech length required
|
||||
minSilenceDurationMs: 100, // Minimum silence length required
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `silenceThresholdSecs` | Seconds of silence before auto-commit | 1.5 |
|
||||
| `threshold` | Speech detection sensitivity (lower = more sensitive) | 0.4 |
|
||||
| `minSpeechDurationMs` | Ignore speech shorter than this | 100 |
|
||||
| `minSilenceDurationMs` | Ignore silence shorter than this | 100 |
|
||||
|
||||
### When to Use VAD
|
||||
|
||||
- Live microphone input
|
||||
- Conversational applications
|
||||
- When natural speech boundaries are preferred
|
||||
- Client-side implementations
|
||||
|
||||
### When to Use Manual Commit
|
||||
|
||||
- Processing audio files
|
||||
- Known segment boundaries
|
||||
- Maximum control over timing
|
||||
- Server-side batch processing
|
||||
|
||||
## Supported Audio Formats
|
||||
|
||||
| Format | Sample Rate | Notes |
|
||||
|--------|-------------|-------|
|
||||
| PCM 16-bit | 16kHz | Recommended, best balance |
|
||||
| PCM 16-bit | 8kHz - 48kHz | Supported range |
|
||||
| μ-law 8-bit | 8kHz | Telephony compatibility |
|
||||
@@ -0,0 +1,209 @@
|
||||
# Real-Time Event Reference
|
||||
|
||||
Complete reference for events in real-time speech-to-text streaming.
|
||||
|
||||
## Sent Events (Client → Server)
|
||||
|
||||
### input_audio_chunk
|
||||
|
||||
Send audio data for transcription.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "input_audio_chunk",
|
||||
"audio_base_64": "<base64-encoded-pcm-audio>",
|
||||
"commit": false,
|
||||
"sample_rate": 16000
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `message_type` | string | Yes | Always `"input_audio_chunk"` |
|
||||
| `audio_base_64` | string | Yes | Base64-encoded PCM audio data |
|
||||
| `commit` | boolean | Yes | Whether to commit after this chunk |
|
||||
| `sample_rate` | number | No | Sample rate in Hz (8000-48000) |
|
||||
| `previous_text` | string | No | Context from prior transcript (first chunk only, max 50 chars) |
|
||||
|
||||
### commit
|
||||
|
||||
Finalize the current transcript segment.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "commit"
|
||||
}
|
||||
```
|
||||
|
||||
## Received Events (Server → Client)
|
||||
|
||||
All received events use `message_type` as the discriminator field.
|
||||
|
||||
### session_started
|
||||
|
||||
Connection established successfully.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "session_started",
|
||||
"session_id": "0b0a72b57fd743ebbed6555d44836cf2",
|
||||
"config": {
|
||||
"sample_rate": 16000,
|
||||
"audio_format": "pcm_16000",
|
||||
"language_code": "en",
|
||||
"model_id": "scribe_v2_realtime",
|
||||
"commit_strategy": "manual",
|
||||
"include_timestamps": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### partial_transcript
|
||||
|
||||
Interim transcription results, updates frequently as audio is processed.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "partial_transcript",
|
||||
"text": "Hello, how are"
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `message_type` | string | `"partial_transcript"` |
|
||||
| `text` | string | Current partial transcription |
|
||||
|
||||
### committed_transcript
|
||||
|
||||
Final transcription after commit.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "committed_transcript",
|
||||
"text": "Hello, how are you today?"
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `message_type` | string | `"committed_transcript"` |
|
||||
| `text` | string | Finalized transcription |
|
||||
|
||||
### committed_transcript_with_timestamps
|
||||
|
||||
Final transcription with word-level timing. Sent after `committed_transcript` when `include_timestamps=true`.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "committed_transcript_with_timestamps",
|
||||
"text": "Hello, how are you today?",
|
||||
"language_code": "en",
|
||||
"words": [
|
||||
{"text": "Hello", "start": 0.0, "end": 0.32, "type": "word"},
|
||||
{"text": " ", "start": 0.32, "end": 0.35, "type": "spacing"},
|
||||
{"text": "how", "start": 0.40, "end": 0.55, "type": "word"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `message_type` | string | `"committed_transcript_with_timestamps"` |
|
||||
| `text` | string | Full transcription text |
|
||||
| `language_code` | string | Detected language code |
|
||||
| `words` | array | Word-level timing data |
|
||||
| `words[].text` | string | The word or token |
|
||||
| `words[].start` | number | Start time in seconds |
|
||||
| `words[].end` | number | End time in seconds |
|
||||
| `words[].type` | string | `"word"`, `"spacing"`, or `"audio_event"` |
|
||||
| `words[].speaker_id` | string | Speaker identifier (if diarization enabled) |
|
||||
|
||||
## Error Events
|
||||
|
||||
### error
|
||||
|
||||
Sent when an error occurs.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "error",
|
||||
"error": "input_error"
|
||||
}
|
||||
```
|
||||
|
||||
### Error Codes
|
||||
|
||||
| Code | Description |
|
||||
|------|-------------|
|
||||
| `auth_error` | Invalid API key or token |
|
||||
| `quota_exceeded` | Usage limit reached |
|
||||
| `input_error` | Unsupported audio format or invalid input |
|
||||
| `rate_limited` | Too many requests |
|
||||
| `commit_throttled` | Commits sent too frequently |
|
||||
| `session_time_limit_exceeded` | Session exceeded max duration |
|
||||
| `unaccepted_terms` | Terms not accepted in dashboard |
|
||||
| `resource_exhausted` | Server capacity reached |
|
||||
| `queue_overflow` | Server queue capacity reached |
|
||||
| `chunk_size_exceeded` | Audio chunk too large |
|
||||
| `insufficient_audio_activity` | Not enough speech detected |
|
||||
| `transcriber_error` | Internal processing error |
|
||||
|
||||
## Connection Events
|
||||
|
||||
### open
|
||||
|
||||
WebSocket connection established (standard WebSocket event, not a JSON message).
|
||||
|
||||
### close
|
||||
|
||||
WebSocket connection closed (standard WebSocket close frame with code and reason).
|
||||
|
||||
## Event Handling Examples
|
||||
|
||||
### Python
|
||||
|
||||
The Python SDK abstracts the wire protocol. You can use `event.type` (not `message_type`) when using the SDK's event objects:
|
||||
|
||||
```python
|
||||
async for event in connection:
|
||||
if event.type == "session_started":
|
||||
print(f"Session: {event.session_id}")
|
||||
elif event.type == "partial_transcript":
|
||||
print(f"Partial: {event.text}")
|
||||
elif event.type == "committed_transcript":
|
||||
print(f"Final: {event.text}")
|
||||
elif event.type == "committed_transcript_with_timestamps":
|
||||
for word in event.words:
|
||||
print(f" {word.text}: {word.start}s - {word.end}s")
|
||||
elif event.type == "error":
|
||||
print(f"Error: {event.error}")
|
||||
```
|
||||
|
||||
### JavaScript
|
||||
|
||||
The JavaScript SDK uses event names matching the `message_type` values:
|
||||
|
||||
```javascript
|
||||
connection.on("session_started", (data) => {
|
||||
console.log("Session:", data.sessionId);
|
||||
});
|
||||
|
||||
connection.on("partial_transcript", (data) => {
|
||||
console.log("Partial:", data.text);
|
||||
});
|
||||
|
||||
connection.on("committed_transcript", (data) => {
|
||||
console.log("Final:", data.text);
|
||||
});
|
||||
|
||||
connection.on("committed_transcript_with_timestamps", (data) => {
|
||||
for (const word of data.words) {
|
||||
console.log(` ${word.text}: ${word.start}s - ${word.end}s`);
|
||||
}
|
||||
});
|
||||
|
||||
connection.on("error", (error) => {
|
||||
console.error("Error:", error);
|
||||
});
|
||||
```
|
||||
@@ -0,0 +1,317 @@
|
||||
# Server-Side Real-Time Streaming
|
||||
|
||||
Transcribe audio streams in real-time from your server with ultra-low latency.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Python
|
||||
pip install elevenlabs python-dotenv pydub
|
||||
|
||||
# JavaScript
|
||||
npm install @elevenlabs/elevenlabs-js dotenv
|
||||
```
|
||||
|
||||
> **Warning:** Do not use `npm install elevenlabs` - that's an outdated v1.x package. Always use `@elevenlabs/elevenlabs-js`.
|
||||
|
||||
## Configuration
|
||||
|
||||
Store your API key in a `.env` file:
|
||||
|
||||
```
|
||||
ELEVENLABS_API_KEY=<your_api_key_here>
|
||||
```
|
||||
|
||||
## Stream from URL
|
||||
|
||||
### Python
|
||||
|
||||
```python
|
||||
from dotenv import load_dotenv
|
||||
import os
|
||||
import asyncio
|
||||
from elevenlabs import ElevenLabs
|
||||
from elevenlabs import RealtimeEvents, RealtimeUrlOptions
|
||||
|
||||
load_dotenv()
|
||||
|
||||
async def main():
|
||||
elevenlabs = ElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY"))
|
||||
stop_event = asyncio.Event()
|
||||
|
||||
connection = await elevenlabs.speech_to_text.realtime.connect(RealtimeUrlOptions(
|
||||
model_id="scribe_v2_realtime",
|
||||
url="https://npr-ice.streamguys1.com/live.mp3",
|
||||
include_timestamps=True,
|
||||
))
|
||||
|
||||
def on_partial_transcript(data):
|
||||
print(f"Partial: {data.get('text', '')}")
|
||||
|
||||
def on_committed_transcript(data):
|
||||
print(f"Committed: {data.get('text', '')}")
|
||||
|
||||
def on_error(error):
|
||||
print(f"Error: {error}")
|
||||
stop_event.set()
|
||||
|
||||
def on_close():
|
||||
print("Connection closed")
|
||||
|
||||
connection.on(RealtimeEvents.PARTIAL_TRANSCRIPT, on_partial_transcript)
|
||||
connection.on(RealtimeEvents.COMMITTED_TRANSCRIPT, on_committed_transcript)
|
||||
connection.on(RealtimeEvents.ERROR, on_error)
|
||||
connection.on(RealtimeEvents.CLOSE, on_close)
|
||||
|
||||
try:
|
||||
await stop_event.wait()
|
||||
except KeyboardInterrupt:
|
||||
print("\nStopping transcription...")
|
||||
finally:
|
||||
await connection.close()
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### JavaScript
|
||||
|
||||
```javascript
|
||||
import "dotenv/config";
|
||||
import { ElevenLabsClient, RealtimeEvents } from "@elevenlabs/elevenlabs-js";
|
||||
|
||||
const elevenlabs = new ElevenLabsClient();
|
||||
|
||||
const connection = await elevenlabs.speechToText.realtime.connect({
|
||||
modelId: "scribe_v2_realtime",
|
||||
url: "https://npr-ice.streamguys1.com/live.mp3",
|
||||
includeTimestamps: true,
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.PARTIAL_TRANSCRIPT, (transcript) => {
|
||||
console.log("Partial transcript", transcript);
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.COMMITTED_TRANSCRIPT, (transcript) => {
|
||||
console.log("Committed transcript", transcript);
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.ERROR, (error) => {
|
||||
console.log("Error", error);
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.CLOSE, () => {
|
||||
console.log("Connection closed");
|
||||
});
|
||||
```
|
||||
|
||||
## Manual Audio Chunking
|
||||
|
||||
For local files or custom audio streams, convert to PCM format and send in chunks.
|
||||
|
||||
### Python
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import base64
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
from pathlib import Path
|
||||
from elevenlabs import ElevenLabs
|
||||
from elevenlabs import AudioFormat, CommitStrategy, RealtimeEvents, RealtimeAudioOptions
|
||||
from pydub import AudioSegment
|
||||
|
||||
load_dotenv()
|
||||
|
||||
def load_and_convert_audio(audio_path: str | Path, target_sample_rate: int = 16000) -> bytes:
|
||||
if str(audio_path).lower().endswith('.pcm'):
|
||||
with open(audio_path, 'rb') as f:
|
||||
return f.read()
|
||||
|
||||
audio = AudioSegment.from_file(audio_path)
|
||||
if audio.channels > 1:
|
||||
audio = audio.set_channels(1)
|
||||
if audio.frame_rate != target_sample_rate:
|
||||
audio = audio.set_frame_rate(target_sample_rate)
|
||||
audio = audio.set_sample_width(2)
|
||||
return audio.raw_data
|
||||
|
||||
async def main():
|
||||
elevenlabs = ElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY"))
|
||||
transcription_complete = asyncio.Event()
|
||||
|
||||
connection = await elevenlabs.speech_to_text.realtime.connect(RealtimeAudioOptions(
|
||||
model_id="scribe_v2_realtime",
|
||||
audio_format=AudioFormat.PCM_16000,
|
||||
sample_rate=16000,
|
||||
commit_strategy=CommitStrategy.MANUAL,
|
||||
include_timestamps=True,
|
||||
))
|
||||
|
||||
def on_session_started(data):
|
||||
print(f"Session started: {data}")
|
||||
asyncio.create_task(send_audio())
|
||||
|
||||
def on_partial_transcript(data):
|
||||
transcript = data.get('text', '')
|
||||
if transcript:
|
||||
print(f"Partial: {transcript}")
|
||||
|
||||
def on_committed_transcript(data):
|
||||
transcript = data.get('text', '')
|
||||
print(f"\nCommitted transcript: {transcript}")
|
||||
|
||||
def on_committed_transcript_with_timestamps(data):
|
||||
print(f"Timestamps: {data.get('words', '')}")
|
||||
transcription_complete.set()
|
||||
|
||||
def on_error(error):
|
||||
print(f"Error: {error}")
|
||||
transcription_complete.set()
|
||||
|
||||
def on_close():
|
||||
print("Connection closed")
|
||||
transcription_complete.set()
|
||||
|
||||
connection.on(RealtimeEvents.SESSION_STARTED, on_session_started)
|
||||
connection.on(RealtimeEvents.PARTIAL_TRANSCRIPT, on_partial_transcript)
|
||||
connection.on(RealtimeEvents.COMMITTED_TRANSCRIPT, on_committed_transcript)
|
||||
connection.on(RealtimeEvents.COMMITTED_TRANSCRIPT_WITH_TIMESTAMPS, on_committed_transcript_with_timestamps)
|
||||
connection.on(RealtimeEvents.ERROR, on_error)
|
||||
connection.on(RealtimeEvents.CLOSE, on_close)
|
||||
|
||||
async def send_audio():
|
||||
audio_file_path = Path("audio.mp3")
|
||||
audio_data = load_and_convert_audio(audio_file_path)
|
||||
chunk_size = 32000 # 1 second of audio at 16kHz
|
||||
chunks = [audio_data[i:i + chunk_size] for i in range(0, len(audio_data), chunk_size)]
|
||||
|
||||
for i, chunk in enumerate(chunks):
|
||||
chunk_base64 = base64.b64encode(chunk).decode('utf-8')
|
||||
await connection.send({"audio_base_64": chunk_base64, "sample_rate": 16000})
|
||||
|
||||
if i < len(chunks) - 1:
|
||||
await asyncio.sleep(1)
|
||||
|
||||
await asyncio.sleep(0.5)
|
||||
await connection.commit()
|
||||
|
||||
try:
|
||||
await transcription_complete.wait()
|
||||
except KeyboardInterrupt:
|
||||
print("\nStopping...")
|
||||
finally:
|
||||
await connection.close()
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### JavaScript
|
||||
|
||||
```javascript
|
||||
import "dotenv/config";
|
||||
import * as fs from "node:fs";
|
||||
import { ElevenLabsClient, RealtimeEvents, AudioFormat } from "@elevenlabs/elevenlabs-js";
|
||||
|
||||
const elevenlabs = new ElevenLabsClient();
|
||||
|
||||
const connection = await elevenlabs.speechToText.realtime.connect({
|
||||
modelId: "scribe_v2_realtime",
|
||||
audioFormat: AudioFormat.PCM_16000,
|
||||
sampleRate: 16000,
|
||||
includeTimestamps: true,
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.SESSION_STARTED, (data) => {
|
||||
console.log("Session started", data);
|
||||
sendAudio();
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.PARTIAL_TRANSCRIPT, (transcript) => {
|
||||
console.log("Partial transcript", transcript);
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.COMMITTED_TRANSCRIPT, (transcript) => {
|
||||
console.log("Committed transcript", transcript);
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.COMMITTED_TRANSCRIPT_WITH_TIMESTAMPS, (transcript) => {
|
||||
console.log("Committed with timestamps", transcript);
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.ERROR, (error) => {
|
||||
console.log("Error", error);
|
||||
});
|
||||
|
||||
connection.on(RealtimeEvents.CLOSE, () => {
|
||||
console.log("Connection closed");
|
||||
});
|
||||
|
||||
async function sendAudio() {
|
||||
const pcmFilePath = "audio.pcm";
|
||||
const chunkSize = 32000;
|
||||
const audioBuffer = fs.readFileSync(pcmFilePath);
|
||||
const chunks: Buffer[] = [];
|
||||
|
||||
for (let i = 0; i < audioBuffer.length; i += chunkSize) {
|
||||
const chunk = audioBuffer.subarray(i, i + chunkSize);
|
||||
chunks.push(chunk);
|
||||
}
|
||||
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
const chunk = chunks[i];
|
||||
const chunkBase64 = chunk.toString("base64");
|
||||
|
||||
connection.send({
|
||||
audioBase64: chunkBase64,
|
||||
sampleRate: 16000,
|
||||
});
|
||||
|
||||
if (i < chunks.length - 1) {
|
||||
await new Promise((resolve) => setTimeout(resolve, 1000));
|
||||
}
|
||||
}
|
||||
|
||||
await new Promise((resolve) => setTimeout(resolve, 500));
|
||||
connection.commit();
|
||||
}
|
||||
```
|
||||
|
||||
## Direct WebSocket Connection
|
||||
|
||||
For cases where the SDK cannot be used:
|
||||
|
||||
```
|
||||
wss://api.elevenlabs.io/v1/speech-to-text/realtime?model_id=scribe_v2_realtime
|
||||
```
|
||||
|
||||
### Message Format
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "input_audio_chunk",
|
||||
"audio_base_64": "<base64-encoded-audio>",
|
||||
"commit": false,
|
||||
"sample_rate": 16000
|
||||
}
|
||||
```
|
||||
|
||||
### Commit Message
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "commit"
|
||||
}
|
||||
```
|
||||
|
||||
## Audio Requirements
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Format | PCM 16-bit |
|
||||
| Sample Rate | 16000 Hz (recommended) |
|
||||
| Channels | Mono |
|
||||
| Chunk Size | 32,000 bytes = 1 second |
|
||||
|
||||
Supported sample rates: 8kHz to 48kHz
|
||||
@@ -0,0 +1,184 @@
|
||||
# Transcription Options
|
||||
|
||||
## Request Parameters
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `file` | file | Yes | Audio or video file to transcribe |
|
||||
| `model_id` | string | Yes | `scribe_v2` (or legacy `scribe_v1`) for batch transcription |
|
||||
| `language_code` | string | No | Language hint (ISO 639-1 or ISO 639-3, e.g., `en` or `eng`) |
|
||||
| `timestamps_granularity` | string | No | `none`, `word`, or `character` (default: `word`) |
|
||||
| `diarize` | boolean | No | Enable speaker diarization (default: `false`; up to 32 speakers) |
|
||||
| `num_speakers` | integer | No | Maximum speakers to detect (up to 32 for batch) |
|
||||
| `diarization_threshold` | number | No | Tune diarization sensitivity (default: ~0.22; only when `diarize=true` and `num_speakers` is not set) |
|
||||
| `keyterms` | array | No | Terms to bias transcription (up to 100 terms; each ≤50 chars, ≤5 words) |
|
||||
| `tag_audio_events` | boolean | No | Detect non-speech sounds like laughter, applause (default: `true`) |
|
||||
| `entity_detection` | string or array | No | Detect entities (e.g., `pii`, `phi`, `pci`, `offensive_language`) |
|
||||
| `no_verbatim` | boolean | No | If `true`, removes filler words, false starts, and non-speech sounds (supported with `scribe_v2`) |
|
||||
| `use_multi_channel` | boolean | No | Split multichannel audio into separate transcripts (default: `false`; max 5 channels, max 1 hour) |
|
||||
| `cloud_storage_url` | string | No | HTTPS URL to transcribe instead of uploading a file (max 2GB) |
|
||||
| `webhook` | boolean | No | Process async and send result to webhook (default: `false`) |
|
||||
| `webhook_id` | string | No | Target specific webhook (only when `webhook=true`) |
|
||||
| `webhook_metadata` | string or object | No | Custom metadata included in webhook responses (max 16KB) |
|
||||
| `temperature` | double | No | Output randomness (0.0-2.0); defaults vary by model |
|
||||
| `seed` | integer | No | Deterministic output (0-2147483647); same seed = same result |
|
||||
| `additional_formats` | array | No | Export transcript as `docx`, `html`, `pdf`, `srt`, `txt`, or `segmented_json` |
|
||||
| `file_format` | string | No | `pcm_s16le_16` (for lower latency) or `other` (default) |
|
||||
| `enable_logging` | boolean | No | Set `false` for zero retention mode (enterprise only; default: `true`) |
|
||||
|
||||
## Python Example
|
||||
|
||||
```python
|
||||
from elevenlabs import ElevenLabs
|
||||
|
||||
client = ElevenLabs()
|
||||
|
||||
with open("audio.mp3", "rb") as audio_file:
|
||||
result = client.speech_to_text.convert(
|
||||
file=audio_file,
|
||||
model_id="scribe_v2",
|
||||
language_code="eng",
|
||||
timestamps_granularity="word",
|
||||
diarize=True,
|
||||
keyterms=["ElevenLabs", "Scribe"]
|
||||
)
|
||||
```
|
||||
|
||||
## JavaScript Example
|
||||
|
||||
```javascript
|
||||
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
|
||||
import { createReadStream } from "fs";
|
||||
|
||||
const client = new ElevenLabsClient();
|
||||
|
||||
const result = await client.speechToText.convert({
|
||||
file: createReadStream("audio.mp3"),
|
||||
modelId: "scribe_v2",
|
||||
languageCode: "eng",
|
||||
timestampsGranularity: "word",
|
||||
diarize: true,
|
||||
keyterms: ["ElevenLabs", "Scribe"],
|
||||
});
|
||||
```
|
||||
|
||||
## cURL Example
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.elevenlabs.io/v1/speech-to-text" \
|
||||
-H "xi-api-key: $ELEVENLABS_API_KEY" \
|
||||
-F "file=@audio.mp3" \
|
||||
-F "model_id=scribe_v2" \
|
||||
-F "language_code=eng" \
|
||||
-F "timestamps_granularity=word" \
|
||||
-F "diarize=true"
|
||||
```
|
||||
|
||||
## Response Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "The complete transcribed text from the audio file.",
|
||||
"language_code": "eng",
|
||||
"language_probability": 0.98,
|
||||
"words": [
|
||||
{
|
||||
"text": "The",
|
||||
"start": 0.0,
|
||||
"end": 0.15,
|
||||
"type": "word",
|
||||
"speaker_id": "speaker_0"
|
||||
},
|
||||
{
|
||||
"text": " ",
|
||||
"start": 0.15,
|
||||
"end": 0.16,
|
||||
"type": "spacing",
|
||||
"speaker_id": "speaker_0"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Response Fields
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `text` | string | Full transcription text |
|
||||
| `language_code` | string | Detected language (ISO 639-1 or ISO 639-3) |
|
||||
| `language_probability` | float | Confidence in detection (0-1) |
|
||||
| `words` | array | Word-level timestamps (if requested) |
|
||||
| `words[].text` | string | The transcribed word or spacing |
|
||||
| `words[].start` | float | Start time in seconds |
|
||||
| `words[].end` | float | End time in seconds |
|
||||
| `words[].type` | string | `word`, `spacing`, or `audio_event` |
|
||||
| `words[].speaker_id` | string | Speaker identifier (if diarization enabled) |
|
||||
| `transcription_id` | string | Unique identifier for this transcription |
|
||||
| `additional_formats` | array | Exported transcript formats (if requested) |
|
||||
| `entities` | array | Detected entities with text, type, and character offsets (if entity_detection enabled) |
|
||||
|
||||
## Supported Languages (90+)
|
||||
|
||||
Common languages (ISO 639-3 codes):
|
||||
|
||||
| Code | Language | Code | Language |
|
||||
|------|----------|------|----------|
|
||||
| `eng` | English | `jpn` | Japanese |
|
||||
| `spa` | Spanish | `kor` | Korean |
|
||||
| `fra` | French | `zho` | Mandarin |
|
||||
| `deu` | German | `ara` | Arabic |
|
||||
| `ita` | Italian | `hin` | Hindi |
|
||||
| `por` | Portuguese | `tur` | Turkish |
|
||||
| `nld` | Dutch | `swe` | Swedish |
|
||||
| `pol` | Polish | `dan` | Danish |
|
||||
| `rus` | Russian | `fin` | Finnish |
|
||||
|
||||
Full list: Afrikaans, Amharic, Armenian, Azerbaijani, Belarusian, Bengali, Bosnian, Bulgarian, Burmese, Cantonese, Catalan, Cebuano, Croatian, Czech, Estonian, Filipino, Georgian, Greek, Gujarati, Hausa, Hebrew, Hungarian, Icelandic, Indonesian, Irish, Javanese, Kannada, Kazakh, Khmer, Kyrgyz, Lao, Latvian, Lithuanian, Luxembourgish, Macedonian, Malay, Malayalam, Maltese, Māori, Marathi, Mongolian, Nepali, Norwegian, Odia, Pashto, Persian, Punjabi, Romanian, Serbian, Shona, Sindhi, Slovak, Slovenian, Somali, Swahili, Tamil, Tajik, Telugu, Thai, Ukrainian, Urdu, Uzbek, Vietnamese, Welsh, Wolof, Xhosa, Yoruba, Zulu.
|
||||
|
||||
## Format Requirements
|
||||
|
||||
**Audio:** MP3, WAV, M4A, FLAC, OGG, WebM, AAC, AIFF, Opus
|
||||
**Video:** MP4, AVI, MKV, MOV, WMV, FLV, WebM, MPEG, 3GPP
|
||||
|
||||
**Limits:**
|
||||
- Maximum file size: 3GB (file upload) or 2GB (cloud storage URL)
|
||||
- Maximum duration: 10 hours (standard) or 1 hour (multichannel mode)
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Subtitle Generation with Speakers
|
||||
|
||||
```python
|
||||
result = client.speech_to_text.convert(
|
||||
file=audio_file,
|
||||
model_id="scribe_v2",
|
||||
timestamps_granularity="word",
|
||||
diarize=True
|
||||
)
|
||||
|
||||
# Generate SRT with speaker labels
|
||||
for i, word in enumerate(result.words, 1):
|
||||
if word.type == "word":
|
||||
print(f"[{word.speaker_id}] {word.text} ({word.start:.2f}s)")
|
||||
```
|
||||
|
||||
### Meeting Transcription with Custom Terms
|
||||
|
||||
```python
|
||||
with open("meeting.mp3", "rb") as f:
|
||||
result = client.speech_to_text.convert(
|
||||
file=f,
|
||||
model_id="scribe_v2",
|
||||
diarize=True,
|
||||
keyterms=["Q4 forecast", "revenue target", "ACME Corp"]
|
||||
)
|
||||
|
||||
# Group by speaker
|
||||
current_speaker = None
|
||||
for word in result.words:
|
||||
if word.type == "word":
|
||||
if word.speaker_id != current_speaker:
|
||||
current_speaker = word.speaker_id
|
||||
print(f"\n[{current_speaker}]:", end=" ")
|
||||
print(word.text, end="")
|
||||
```
|
||||
Reference in New Issue
Block a user