Initial commit: FunASR Speech Recognition Toolkit

Add complete FunASR codebase including models, runtime, and documentation.

runtime/llama.cpp/BENCHMARKS.md

@@ -0,0 +1,104 @@
+# FunASR (llama.cpp / GGUF) vs whisper.cpp — CPU benchmark
+
+How does the FunASR llama.cpp runtime compare with [whisper.cpp](https://github.com/ggml-org/whisper.cpp),
+the de-facto on-device ASR runtime, on **Chinese** speech? This page reports a
+head-to-head on identical hardware and audio.
+
+**TL;DR — for Chinese ASR on CPU, FunASR is ~2.7× more accurate than whisper.cpp at
+every model tier, and faster.**
+
+## Results
+
+Dataset: **184 real Mandarin clips with human references** (the standard FunASR
+benchmark set). Metric: **micro-CER** with `normalize_zh` (lower is better). Speed:
+real-time factor on **CPU, 8 threads** (model-load excluded). whisper forced to
+Chinese (`-l zh`).
+
+| system | **CER** (micro, normalize_zh) ↓ | speed ↑ | size |
+|---|---|---|---|
+| **FunASR Fun-ASR-Nano** | **8.06** (fp32 ref) / **8.42** (Q8 runtime) | LLM decode¹ | enc + Qwen3-0.6B GGUF |
+| **FunASR SenseVoiceSmall** | **7.81** (fp32 ref) / **8.17** (Q8 runtime) | **~20× real-time** | 449 MB (f16) |
+| **FunASR Paraformer** | **10.18** (fp32 ref) / **9.89** (Q8 runtime) | **~21× real-time** | 401 MB (f16) |
+| whisper.cpp base | 31.33 | 9.9× | 142 MB |
+| whisper.cpp small | 22.12 | 4.6× | 466 MB |
+| whisper.cpp large-v3-turbo | 23.15 | 3.2× | 1.6 GB |
+
+**Each FunASR row shows two numbers:** the published **fp32 reference** (PyTorch,
+the number on funasr.com / the model cards) and the **Q8 llama.cpp CPU runtime**
+measured here. The ~0.3 % gap is normal int8 quantization + VAD segment boundaries;
+Q8 is the real CPU/edge deployment config. Either way, **FunASR ~8–10 % vs
+whisper.cpp 22–31 % — a 2.7×+ accuracy gap that holds at every tier.**
+
+¹ Fun-ASR-Nano runs an autoregressive 0.6B LLM decoder (slower than the encoder-only
+SenseVoice/Paraformer; it is the accuracy leader). A clean RTF lands once the CLI
+separates model-load from compute.
+
+### Transparency / segmentation (read this before quoting numbers)
+
+- **Segmentation differs by system, each using its natural strategy:** FunASR uses an
+  `fsmn-vad` front end (segments → ASR → concatenate); whisper.cpp uses its built-in
+  30 s windowing. This is a fair system-level comparison.
+- **Engine-internal VAD is now implemented** — a native ggml FSMN-VAD built into the
+  binaries (`--vad fsmn-vad.gguf`). The **bare binary, with no Python front end**, now
+  reaches the reference end-to-end: SenseVoiceSmall **8.01 %**, Paraformer **9.85 %**,
+  Fun-ASR-Nano **8.30 %** (micro, normalize_zh, full 184). The built-in C++ VAD matches
+  the PyTorch `fsmn-vad` front end (segment boundaries within ~10 ms, slightly better
+  CER), so the runtime is now fully self-contained.
+- For full disclosure, **bare binary with no VAD at all (whole-clip)** is higher —
+  SenseVoiceSmall 9.99 %, Paraformer 12.82 % — because long clips decoded as one segment
+  are out-of-distribution; that is exactly what the built-in VAD fixes.
+
+## Why FunASR wins on Chinese
+
+1. **Training data.** SenseVoice / Paraformer / Fun-ASR-Nano are trained primarily on
+   large-scale Mandarin; Whisper is a general multilingual model where Chinese is a
+   small slice. On Chinese homophones Whisper makes substitution errors the FunASR
+   models do not (example below).
+2. **Architecture → speed.** Paraformer is non-autoregressive (CIF predictor + one
+   decoder pass) and SenseVoiceSmall is encoder + CTC (one forward pass); Whisper is
+   autoregressive (one step per output token).
+
+## Qualitative example (clip 002)
+
+| system | output (excerpt) |
+|---|---|
+| ground truth | 我想问，我在**滨海新区**有房…所以我必须拿到**抚养权** |
+| FunASR (Nano / SenseVoice / Paraformer) | …我在**滨海新区**有房…拿到**抚养权**… ✓ |
+| whisper base | …我在**冰海心区**有房…我想要**扶养权**…上学**方面**… ✗ |
+| whisper small | …我在**冰海新区**有房…我想要**抚养全**… ✗ |
+| whisper large-v3-turbo | …滨海新区…上学**方面**… ✗ |
+
+## Methodology
+
+- **Data:** the standard 184-clip Mandarin benchmark set (`benchmark/testset.json`),
+  ~44–60 s each, with human references.
+- **Metric (canonical):** **micro-average CER** (`Σ edits / Σ ref chars`) after
+  **`normalize_zh`**: `re.sub(r'[^\w一-鿿]', '', text).upper()` (strip punctuation/
+  whitespace, keep word chars + CJK, upper-case; SenseVoice `<|...|>` tags stripped).
+  This is the canonical FunASR口径 — the same one behind the published fp32 numbers.
+  (A macro-average / simplified-normalize variant gives different, non-canonical
+  numbers; it is not used here.)
+- **FunASR fp32 reference:** PyTorch, micro + normalize_zh, 184 set — SenseVoice 7.81,
+  Paraformer 10.18, Fun-ASR-Nano 8.06 (matches funasr.com / READMEs / model cards).
+- **FunASR Q8 runtime:** this llama.cpp runtime (Q8 LLM / f16 encoder) + `fsmn-vad`
+  front end (`max_single_segment_time=30000`), full 184. SenseVoice uses `use_itn=True`
+  to match the reference.
+- **whisper.cpp:** ggml `base` / `small` / `large-v3-turbo`, `-l zh`, internal 30 s
+  windowing, full 184.
+- **Speed (RTF):** `Σ compute_time / Σ audio_duration`, model-load excluded, **8 threads
+  for all systems**.
+
+## Caveats (fair use)
+
+- This is a **Chinese** benchmark — FunASR's home turf. Whisper is a *general
+  multilingual* model (translation, 99 languages, timestamps); for English / other
+  languages it is the stronger general choice. The takeaway is specifically:
+  **for Chinese ASR on CPU, FunASR is the accuracy + speed leader.**
+- SenseVoiceSmall also outputs language ID / emotion / audio-event; Paraformer is
+  Mandarin-specialised; Fun-ASR-Nano is the most accurate (LLM decoder). Pick per use case.
+
+## Reproduce
+
+See [`benchmarks/`](benchmarks/) — `compute_cer.py` (micro-CER + normalize_zh + RTF)
+and the per-system run commands. Produce hypotheses with each tool, then compute CER
+against the references and RTF against clip durations.