Initial commit: FunASR Speech Recognition Toolkit

Add complete FunASR codebase including models, runtime, and documentation.

runtime/llama.cpp/fun-asr-nano/README.md

@@ -0,0 +1,144 @@
+# Fun-ASR-Nano on llama.cpp / GGUF
+
+Run **Fun-ASR-Nano** entirely on the [llama.cpp](https://github.com/ggml-org/llama.cpp)
+/ ggml stack — **CPU, edge, a single binary, no Python at runtime**. This is to
+Fun-ASR what [whisper.cpp](https://github.com/ggml-org/whisper.cpp) is to Whisper.
+
+## Why this exists
+
+Fun-ASR-Nano normally runs on PyTorch / vLLM (GPU). That is great for a server
+serving many requests, but it cannot run where there is no GPU and no Python.
+This runtime ports the model to **ggml + GGUF**, so Fun-ASR-Nano can run:
+
+- on a laptop / phone / Raspberry Pi / edge box, offline, CPU-only;
+- embedded directly into a C/C++ application (one static binary);
+- with quantized weights (Q8 / Q4), shrinking the model to ~1.3 GB total.
+
+| | vLLM (existing) | this runtime (llama.cpp) |
+|---|---|---|
+| target | GPU server, high QPS | CPU / edge / embedded |
+| deps | Python + CUDA + PyTorch | none (C/C++ single binary) |
+| weights | HF fp16/bf16 | GGUF, quantized |
+| best for | online service, batch | offline, on-device |
+
+## Architecture
+
+Fun-ASR-Nano = **SenseVoice SAN-M encoder (70 layers) + adaptor + Qwen3-0.6B LLM**.
+The whole pipeline runs in C++:
+
+```
+ audio.wav (16k mono)
+      │  kaldi 80-mel fbank + LFR            (C++)
+      ▼
+   features [T, 560]
+      │  SAN-M encoder + adaptor             (ggml)   ── funasr-encoder.gguf
+      ▼
+   audio embeds [T', 1024]
+      │  keep first fake_token_len frames (low-frame-rate)
+      ▼
+ [ prefix tokens | audio embeds | suffix tokens ]
+      │  Qwen3-0.6B, embeds injected via llama_decode (llava/mtmd style)  ── qwen3-0.6b.gguf
+      ▼
+   transcription
+```
+
+The audio embeddings are fed into the LLM through `llama_decode`'s embedding-input
+path — exactly how llava/mtmd inject vision embeddings.
+
+## Quickstart
+
+**1. Build** (drop the examples into a llama.cpp checkout):
+```bash
+git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
+cp -r /path/to/runtime/llama.cpp/funasr-cli examples/
+echo 'add_subdirectory(funasr-cli)' >> examples/CMakeLists.txt
+cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF
+cmake --build build -j --target llama-funasr-cli
+```
+
+**2. Convert weights to GGUF** (one-time; needs the checkpoint, e.g.
+`FunAudioLLM/Fun-ASR-Nano-2512`):
+```bash
+# LLM half — Qwen3-0.6B is natively supported by llama.cpp
+python llama.cpp/convert_hf_to_gguf.py <model>/Qwen3-0.6B-vllm \
+    --outfile qwen3-0.6b-f32.gguf --outtype f32
+build/bin/llama-quantize qwen3-0.6b-f32.gguf qwen3-0.6b-q8_0.gguf Q8_0   # smaller, recommended
+
+# audio half — SenseVoice encoder + adaptor
+python runtime/llama.cpp/export_encoder_gguf.py \
+    --model_pt <model>/model.pt --out funasr-encoder.gguf              # f32, 935 MB
+python runtime/llama.cpp/export_encoder_gguf.py \
+    --model_pt <model>/model.pt --out funasr-encoder-f16.gguf --wtype f16   # 469 MB
+```
+
+**3. Transcribe:**
+```bash
+build/bin/llama-funasr-cli \
+    --enc funasr-encoder.gguf -m qwen3-0.6b-q8_0.gguf \
+    -a audio.wav --chunk 15
+```
+Expected output (one of the benchmark clips):
+```
+我想问我在滨海新区有房我一直没有照顾孩子但是我想要抚养权...你觉得这是正常的想法吗
+[done] 7.40s ; chunk=15s
+```
+
+## Models & sizes
+
+| file | dtype | size |
+|---|---|---|
+| funasr-encoder.gguf | f32 | 935 MB |
+| funasr-encoder-f16.gguf | f16 (matmul weights) | 469 MB |
+| qwen3-0.6b-f32.gguf | f32 | 3.0 GB |
+| qwen3-0.6b-q8_0.gguf | Q8_0 | 805 MB |
+| qwen3-0.6b-q4km.gguf | Q4_K_M | 484 MB |
+
+Fully-quantized config (f16 encoder + Q8 LLM) ≈ **1.3 GB**, edge-friendly.
+
+## Accuracy & validation
+
+Validated against the PyTorch reference on the 184-file benchmark:
+
+- **Encoder + adaptor (ggml) vs PyTorch:** cosine **1.000000**, max_abs_diff **5e-3** (f32).
+- **kaldi fbank (C++) vs torchaudio:** cosine **1.000000**.
+- **End-to-end CER, identical conditions (f32 LLM, 15 s chunking):**
+  C++ macro 17.41% / micro 11.68%  vs  PyTorch macro 17.42% / micro 11.70%
+  → aggregate CER matches to **0.02%**; the port is faithful.
+- Best practical config (Q8 LLM + 15 s chunking): **micro-CER 9.51%** (production
+  VAD-segmented reference is ~8.2%; the gap is fixed-window vs VAD, not the port).
+
+## Tips & gotchas
+
+- **Use `--chunk 15`** for long audio. Decoding a whole 60 s clip in one segment is
+  out-of-distribution and makes greedy decoding loop; 15 s windows fix it
+  (micro-CER 29% → 9.5%).
+- **Low-frame-rate truncation** is required: only the first `fake_token_len`
+  adaptor frames are real audio tokens. The CLI does this automatically; feeding
+  all frames makes the LLM repeat.
+- **Use bf16/fp32, avoid fp16 for the audio path** — the adaptor output has large
+  magnitude (std ≈ 28, |max| ≈ 1187); fp16 can overflow. The GGUFs here are f32/f16
+  weights with f32 activations, which is safe.
+- **WAV input** currently assumes 16 kHz mono PCM16. Resample first if needed.
+- Q8 quantization slightly *helps* greedy stability (quant noise regularizes away
+  from repetition loops), so Q8 is a good default.
+
+## Implementation notes
+
+- FSMN depthwise memory is an exact f32 shift-accumulate (avoids the F16-only,
+  upstream-flagged `ggml_conv_1d_dw`).
+- LayerNorm eps = 1e-5; sinusoidal position encoding depth = input feature dim (560),
+  positions start at 1; encoder input pre-scaled by sqrt(512).
+- Prompt is fed as tokens via `llama_tokenize(parse_special=true)` (prefix = 18
+  tokens, matching the HF tokenizer), so no Python embedding table is needed.
+
+## Files
+```
+funasr-cli/        integrated binary: WAV → transcription
+funasr-encoder/    encoder+adaptor only (ggml) — validation/debugging
+funasr-embd/       LLM decode from precomputed embeds — validation/debugging
+export_encoder_gguf.py   export the audio encoder + adaptor to GGUF
+```
+
+## Roadmap
+- True FSMN-VAD segmentation (replace fixed windows; closes the last ~1.3% CER).
+- Arbitrary WAV formats / resampling; encoder Q8 quantization; single packaged GGUF.

runtime/llama.cpp/fun-asr-nano/export_encoder_gguf.py

@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+"""Export Fun-ASR-Nano audio encoder + adaptor weights to a GGUF file.
+
+Packs all `audio_encoder.*` and `audio_adaptor.*` tensors (bf16 -> f32) plus
+architecture metadata into funasr-encoder.gguf, for the ggml C++ forward pass.
+Tensor names are kept verbatim (e.g. audio_encoder.encoders.3.norm1.weight) so
+the C++ side can look them up directly.
+"""
+import argparse, os
+import numpy as np
+import torch
+import gguf
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--model_pt", required=True)
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--wtype", default="f32", choices=["f32", "f16", "q8_0"],
+                    help="dtype for 2D Linear (matmul) weights; norms/bias/fsmn stay f32")
+    args = ap.parse_args()
+
+    sd = torch.load(args.model_pt, map_location="cpu")
+    sd = sd.get("state_dict", sd)
+
+    w = gguf.GGUFWriter(args.out, "funasr-sensevoice-encoder")
+
+    # --- architecture metadata (from config.yaml) ---
+    w.add_uint32("funasr.enc.input_size", 560)      # lfr_m(7) * n_mels(80)
+    w.add_uint32("funasr.enc.output_size", 512)
+    w.add_uint32("funasr.enc.attention_heads", 4)
+    w.add_uint32("funasr.enc.linear_units", 2048)
+    w.add_uint32("funasr.enc.num_blocks", 50)       # encoders0(1) + encoders(49)
+    w.add_uint32("funasr.enc.tp_blocks", 20)
+    w.add_uint32("funasr.enc.kernel_size", 11)
+    w.add_uint32("funasr.enc.sanm_shfit", 0)
+    w.add_uint32("funasr.adp.llm_dim", 1024)
+    w.add_uint32("funasr.adp.encoder_dim", 512)
+    w.add_uint32("funasr.adp.ffn_dim", 2048)
+    w.add_uint32("funasr.adp.n_layer", 2)
+    w.add_uint32("funasr.adp.attention_heads", 8)
+    w.add_uint32("funasr.adp.downsample_rate", 1)
+    w.add_uint32("funasr.frontend.n_mels", 80)
+    w.add_uint32("funasr.frontend.lfr_m", 7)
+    w.add_uint32("funasr.frontend.lfr_n", 6)
+
+    n = 0
+    for k, v in sd.items():
+        if not (k.startswith("audio_encoder.") or k.startswith("audio_adaptor.")):
+            continue
+        arr = v.detach().to(torch.float32).contiguous().numpy()
+        # FSMN depthwise kernel: store as (K, D) so the C++ side can slice a
+        # contiguous per-tap [D] vector and do an exact f32 shift-accumulate
+        # (avoids the F16-only ggml_conv_1d_dw path).
+        if k.endswith("fsmn_block.weight"):       # (D, 1, K) -> (K, D)
+            arr = np.ascontiguousarray(arr[:, 0, :].T)
+        # matmul (Linear) weights -> optional f16; norms/biases/fsmn stay f32
+        elif args.wtype == "f16" and arr.ndim == 2 and "norm" not in k:
+            arr = arr.astype(np.float16)
+        if args.wtype == "q8_0" and arr.ndim == 2 and "norm" not in k and "fsmn_block" not in k and arr.shape[1] % 32 == 0:
+            from gguf import quants as _q, GGMLQuantizationType as _QT
+            w.add_tensor(k, _q.quantize(arr, _QT.Q8_0), raw_dtype=_QT.Q8_0)
+        else:
+            w.add_tensor(k, arr)
+        n += 1
+    print(f"writing {n} tensors to {args.out}")
+
+    w.write_header_to_file()
+    w.write_kv_data_to_file()
+    w.write_tensors_to_file()
+    w.close()
+    print(f"done: {args.out} ({os.path.getsize(args.out)/1e6:.1f} MB)")
+
+
+if __name__ == "__main__":
+    main()

runtime/llama.cpp/fun-asr-nano/funasr-cli/CMakeLists.txt

@@ -0,0 +1,6 @@
+set(TARGET llama-funasr-cli)
+add_executable(${TARGET} funasr-cli.cpp)
+install(TARGETS ${TARGET} RUNTIME)
+target_include_directories(${TARGET} PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/../funasr-common)
+target_link_libraries(${TARGET} PRIVATE llama ggml ${CMAKE_THREAD_LIBS_INIT})
+target_compile_features(${TARGET} PRIVATE cxx_std_17)

runtime/llama.cpp/fun-asr-nano/funasr-cli/funasr-cli.cpp

@@ -0,0 +1,251 @@
+// funasr-cli: end-to-end Fun-ASR-Nano in C++ on the llama.cpp / ggml stack.
+//
+//   wav(16k mono) -> kaldi fbank -> SAN-M encoder + adaptor (ggml) ->
+//   low-frame-rate truncation -> [prefix tokens | audio embeds | suffix tokens]
+//   -> Qwen3 LLM (llama.cpp) -> transcription.
+//
+// This is the whisper.cpp-style single-binary path: no Python at runtime.
+//
+//   funasr-cli --enc funasr-encoder.gguf -m qwen3-0.6b.gguf -a audio.wav
+
+#include "ggml.h"
+#include "ggml-cpu.h"
+#include "ggml-alloc.h"
+#include "ggml-backend.h"
+#include "gguf.h"
+#include "llama.h"
+
+#include <cmath>
+#include <cstdint>
+#include <cstdio>
+#include <cstring>
+#include <map>
+#include <string>
+#include <vector>
+
+// any audio (wav/mp3/flac, any rate/channels) -> 16 kHz mono f32, via miniaudio
+#define FUNASR_AUDIO_IMPLEMENTATION
+#include "funasr_audio.h"
+// built-in FSMN-VAD front end (single-binary --vad segmentation)
+#include "funasr_vad.h"
+#include <utility>
+
+// ======================= kaldi fbank + LFR =======================
+static const int FS=16000, WINLEN=400, SHIFT=160, NFFT=512, NMEL=80, LFR_M=7, LFR_N=6;
+static const float PREEMPH=0.97f, LOWF=20.0f, HIGHF=8000.0f;
+static inline float mel(float f){ return 1127.0f*logf(1.0f+f/700.0f); }
+static void fft(std::vector<float>&re,std::vector<float>&im,int n){
+    for(int i=1,j=0;i<n;i++){int b=n>>1;for(;j&b;b>>=1)j^=b;j^=b;if(i<j){std::swap(re[i],re[j]);std::swap(im[i],im[j]);}}
+    for(int len=2;len<=n;len<<=1){double a=-2.0*M_PI/len;float wr=cosf(a),wi=sinf(a);
+        for(int i=0;i<n;i+=len){float cr=1,ci=0;for(int k=0;k<len/2;k++){
+            float ur=re[i+k],ui=im[i+k];float vr=re[i+k+len/2]*cr-im[i+k+len/2]*ci,vi=re[i+k+len/2]*ci+im[i+k+len/2]*cr;
+            re[i+k]=ur+vr;im[i+k]=ui+vi;re[i+k+len/2]=ur-vr;im[i+k+len/2]=ui-vi;
+            float n2=cr*wr-ci*wi;ci=cr*wi+ci*wr;cr=n2;}}}
+}
+// returns [T x 560] row-major, sets T
+static std::vector<float> compute_fbank(std::vector<float> wav, int & T_out) {
+    for (auto & v : wav) v *= 32768.0f;
+    std::vector<float> win(WINLEN);
+    for (int i=0;i<WINLEN;i++) win[i]=0.54f-0.46f*cosf(2.0f*M_PI*i/(WINLEN-1));
+    const int NBIN=NFFT/2+1; float bw=(float)FS/NFFT, ml=mel(LOWF), mh=mel(HIGHF), dm=(mh-ml)/(NMEL+1);
+    std::vector<std::vector<float>> fb(NMEL, std::vector<float>(NBIN,0.0f));
+    for(int m=0;m<NMEL;m++){float L=ml+m*dm,C=ml+(m+1)*dm,R=ml+(m+2)*dm;
+        for(int k=0;k<NBIN;k++){float mf=mel(bw*k); if(mf>L&&mf<R) fb[m][k]=mf<=C?(mf-L)/(C-L):(R-mf)/(R-C);}}
+    int N=wav.size(); int T=(N-WINLEN)/SHIFT+1;
+    std::vector<std::vector<float>> feat(T, std::vector<float>(NMEL));
+    std::vector<float> re(NFFT),im(NFFT),fr(WINLEN);
+    const float fl=1.1920929e-07f;
+    for(int t=0;t<T;t++){const float*s=wav.data()+t*SHIFT;
+        double mn=0;for(int i=0;i<WINLEN;i++)mn+=s[i];mn/=WINLEN;
+        for(int i=0;i<WINLEN;i++)fr[i]=s[i]-(float)mn;
+        for(int i=WINLEN-1;i>0;i--)fr[i]-=PREEMPH*fr[i-1];fr[0]-=PREEMPH*fr[0];
+        for(int i=0;i<NFFT;i++){re[i]=i<WINLEN?fr[i]*win[i]:0.0f;im[i]=0.0f;}
+        fft(re,im,NFFT);
+        for(int m=0;m<NMEL;m++){float e=0;for(int k=0;k<NBIN;k++)if(fb[m][k]>0)e+=fb[m][k]*(re[k]*re[k]+im[k]*im[k]);
+            feat[t][m]=logf(e>fl?e:fl);}}
+    // LFR
+    const int pad=(LFR_M-1)/2; int T_lfr=(T+LFR_N-1)/LFR_N;
+    std::vector<std::vector<float>> pd; pd.reserve(T+pad+LFR_M);
+    for(int i=0;i<pad;i++)pd.push_back(feat[0]);
+    for(int t=0;t<T;t++)pd.push_back(feat[t]);
+    while((int)pd.size()<(T_lfr-1)*LFR_N+LFR_M)pd.push_back(feat[T-1]);
+    int D=LFR_M*NMEL; std::vector<float> out((size_t)T_lfr*D);
+    for(int i=0;i<T_lfr;i++)for(int j=0;j<LFR_M;j++)
+        memcpy(&out[(size_t)i*D+j*NMEL],pd[i*LFR_N+j].data(),NMEL*sizeof(float));
+    T_out=T_lfr; return out;
+}
+
+// ======================= ggml SAN-M encoder + adaptor =======================
+struct cfg { int d_model=512,n_head=4,num_blocks=50,tp_blocks=20,kernel=11,adp_llm=1024,adp_layers=2,adp_head=8; };
+struct enc_model { cfg c; ggml_context*ctx_w=nullptr; std::map<std::string,ggml_tensor*> t;
+    ggml_tensor* g(const std::string&n){auto it=t.find(n);if(it==t.end()){fprintf(stderr,"missing %s\n",n.c_str());exit(1);}return it->second;} };
+static const float LN_EPS=1e-5f;
+static bool load_enc(const char*p, enc_model&m){
+    gguf_init_params gp={false,&m.ctx_w}; gguf_context*g=gguf_init_from_file(p,gp); if(!g)return false;
+    auto rd=[&](const char*k,int d){int i=gguf_find_key(g,k);return i<0?d:(int)gguf_get_val_u32(g,i);};
+    m.c.d_model=rd("funasr.enc.output_size",512); m.c.n_head=rd("funasr.enc.attention_heads",4);
+    m.c.num_blocks=rd("funasr.enc.num_blocks",50); m.c.tp_blocks=rd("funasr.enc.tp_blocks",20);
+    m.c.kernel=rd("funasr.enc.kernel_size",11); m.c.adp_llm=rd("funasr.adp.llm_dim",1024);
+    m.c.adp_layers=rd("funasr.adp.n_layer",2); m.c.adp_head=rd("funasr.adp.attention_heads",8);
+    int n=gguf_get_n_tensors(g); for(int i=0;i<n;i++){const char*nm=gguf_get_tensor_name(g,i);m.t[nm]=ggml_get_tensor(m.ctx_w,nm);}
+    gguf_free(g); return true;
+}
+static ggml_tensor* lin(ggml_context*c,ggml_tensor*w,ggml_tensor*b,ggml_tensor*x){auto y=ggml_mul_mat(c,w,x);return b?ggml_add(c,y,b):y;}
+static ggml_tensor* lnorm(ggml_context*c,ggml_tensor*x,ggml_tensor*g,ggml_tensor*b){return ggml_add(c,ggml_mul(c,ggml_norm(c,x,LN_EPS),g),b);}
+static ggml_tensor* sanm_attn(ggml_context*c,enc_model&m,const std::string&p,ggml_tensor*x,int T){
+    const int D=m.c.d_model,H=m.c.n_head,dk=D/H,K=m.c.kernel;
+    ggml_tensor*qkv=lin(c,m.g(p+"linear_q_k_v.weight"),m.g(p+"linear_q_k_v.bias"),x); size_t nb1=qkv->nb[1];
+    ggml_tensor*q=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,0));
+    ggml_tensor*k=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)D*sizeof(float)));
+    ggml_tensor*v=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)2*D*sizeof(float)));
+    const int pad=(K-1)/2; ggml_tensor*fk=m.g(p+"fsmn_block.weight");
+    ggml_tensor*vp=ggml_pad_ext(c,v,0,0,pad,pad,0,0,0,0); ggml_tensor*fsmn=v;
+    for(int j=0;j<K;j++){auto sl=ggml_view_2d(c,vp,D,T,vp->nb[1],(size_t)j*vp->nb[1]);
+        auto wj=ggml_view_1d(c,fk,D,(size_t)j*fk->nb[1]); fsmn=ggml_add(c,fsmn,ggml_mul(c,ggml_cont(c,sl),wj));}
+    q=ggml_permute(c,ggml_reshape_3d(c,q,dk,H,T),0,2,1,3); k=ggml_permute(c,ggml_reshape_3d(c,k,dk,H,T),0,2,1,3);
+    ggml_tensor*vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,v,dk,H,T),1,2,0,3));
+    ggml_tensor*kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk)));
+    ggml_tensor*o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,T);
+    return ggml_add(c,lin(c,m.g(p+"linear_out.weight"),m.g(p+"linear_out.bias"),o),fsmn);
+}
+static ggml_tensor* sanm_layer(ggml_context*c,enc_model&m,const std::string&p,ggml_tensor*x,int T,bool res){
+    auto r=x; auto h=lnorm(c,x,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));
+    auto sa=sanm_attn(c,m,p+"self_attn.",h,T); x=res?ggml_add(c,r,sa):sa; r=x;
+    h=lnorm(c,x,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));
+    h=lin(c,m.g(p+"feed_forward.w_1.weight"),m.g(p+"feed_forward.w_1.bias"),h); h=ggml_relu(c,h);
+    h=lin(c,m.g(p+"feed_forward.w_2.weight"),m.g(p+"feed_forward.w_2.bias"),h); return ggml_add(c,r,h);
+}
+static ggml_tensor* adp_layer(ggml_context*c,enc_model&m,const std::string&p,ggml_tensor*x,int T){
+    const int D=m.c.adp_llm,H=m.c.adp_head,dk=D/H; auto r=x;
+    auto h=lnorm(c,x,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));
+    auto q=ggml_permute(c,ggml_reshape_3d(c,lin(c,m.g(p+"self_attn.linear_q.weight"),m.g(p+"self_attn.linear_q.bias"),h),dk,H,T),0,2,1,3);
+    auto k=ggml_permute(c,ggml_reshape_3d(c,lin(c,m.g(p+"self_attn.linear_k.weight"),m.g(p+"self_attn.linear_k.bias"),h),dk,H,T),0,2,1,3);
+    auto vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,lin(c,m.g(p+"self_attn.linear_v.weight"),m.g(p+"self_attn.linear_v.bias"),h),dk,H,T),1,2,0,3));
+    auto kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk)));
+    auto o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,T);
+    x=ggml_add(c,r,lin(c,m.g(p+"self_attn.linear_out.weight"),m.g(p+"self_attn.linear_out.bias"),o)); r=x;
+    h=lnorm(c,x,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));
+    h=lin(c,m.g(p+"feed_forward.w_1.weight"),m.g(p+"feed_forward.w_1.bias"),h); h=ggml_relu(c,h);
+    h=lin(c,m.g(p+"feed_forward.w_2.weight"),m.g(p+"feed_forward.w_2.bias"),h); return ggml_add(c,r,h);
+}
+static void add_posenc(std::vector<float>&x,int T,int depth){
+    double inc=log(10000.0)/(depth/2.0-1.0);
+    for(int t=0;t<T;t++){double pos=t+1;for(int i=0;i<depth/2;i++){double its=exp(i*-inc),st=pos*its;
+        x[(size_t)t*depth+i]+=(float)sin(st);x[(size_t)t*depth+depth/2+i]+=(float)cos(st);}}
+}
+// fbank [T x F] -> adaptor out [T x adp_llm] row-major
+static std::vector<float> run_encoder(enc_model&m,std::vector<float> fbank,int T,int F,int&Dout){
+    float sc=sqrtf((float)m.c.d_model); for(auto&v:fbank)v*=sc; add_posenc(fbank,T,F);
+    ggml_backend_t be=ggml_backend_cpu_init();
+    ggml_init_params cp={(size_t)1024*1024*1024,nullptr,true}; ggml_context*c=ggml_init(cp);
+    ggml_tensor*inp=ggml_new_tensor_2d(c,GGML_TYPE_F32,F,T); ggml_set_input(inp);
+    ggml_tensor*x=sanm_layer(c,m,"audio_encoder.encoders0.0.",inp,T,false);
+    for(int i=0;i<m.c.num_blocks-1;i++) x=sanm_layer(c,m,"audio_encoder.encoders."+std::to_string(i)+".",x,T,true);
+    x=lnorm(c,x,m.g("audio_encoder.after_norm.weight"),m.g("audio_encoder.after_norm.bias"));
+    for(int i=0;i<m.c.tp_blocks;i++) x=sanm_layer(c,m,"audio_encoder.tp_encoders."+std::to_string(i)+".",x,T,true);
+    x=lnorm(c,x,m.g("audio_encoder.tp_norm.weight"),m.g("audio_encoder.tp_norm.bias"));
+    x=lin(c,m.g("audio_adaptor.linear1.weight"),m.g("audio_adaptor.linear1.bias"),x); x=ggml_relu(c,x);
+    x=lin(c,m.g("audio_adaptor.linear2.weight"),m.g("audio_adaptor.linear2.bias"),x);
+    for(int i=0;i<m.c.adp_layers;i++) x=adp_layer(c,m,"audio_adaptor.blocks."+std::to_string(i)+".",x,T);
+    ggml_set_output(x);
+    ggml_cgraph*gf=ggml_new_graph_custom(c,32768,false); ggml_build_forward_expand(gf,x);
+    ggml_gallocr_t ga=ggml_gallocr_new(ggml_backend_cpu_buffer_type()); ggml_gallocr_alloc_graph(ga,gf);
+    ggml_backend_tensor_set(inp,fbank.data(),0,ggml_nbytes(inp));
+    ggml_backend_cpu_set_n_threads(be,8); ggml_backend_graph_compute(be,gf);
+    Dout=(int)x->ne[0]; std::vector<float> out((size_t)Dout*T); ggml_backend_tensor_get(x,out.data(),0,ggml_nbytes(x));
+    ggml_gallocr_free(ga); ggml_free(c); ggml_backend_free(be); return out;
+}
+
+// ======================= LLM (llama.cpp) =======================
+static int decode_batch(llama_context*ctx,int n,llama_token*tok,float*embd,int n_embd,int&n_past,bool last_logits){
+    std::vector<llama_pos> pos(n); std::vector<int32_t> nsid(n,1);
+    std::vector<llama_seq_id> s0(1,0); std::vector<llama_seq_id*> sid(n); std::vector<int8_t> lg(n,0);
+    for(int i=0;i<n;i++){pos[i]=n_past+i;sid[i]=s0.data();}
+    if(last_logits) lg[n-1]=1;
+    llama_batch b={n,tok,embd,pos.data(),nsid.data(),sid.data(),lg.data()};
+    int r=llama_decode(ctx,b); n_past+=n; return r;
+}
+
+int main(int argc,char**argv){
+    std::string enc_path,llm_path,wav_path,vad_path; int npred=512; double chunk_sec=0; float rep=1.0f;
+    int vad_maxseg=30000;
+    for(int i=1;i<argc;i++){
+        if(!strcmp(argv[i],"--enc")&&i+1<argc)enc_path=argv[++i];
+        else if(!strcmp(argv[i],"-m")&&i+1<argc)llm_path=argv[++i];
+        else if(!strcmp(argv[i],"-a")&&i+1<argc)wav_path=argv[++i];
+        else if(!strcmp(argv[i],"-n")&&i+1<argc)npred=atoi(argv[++i]);
+        else if(!strcmp(argv[i],"--chunk")&&i+1<argc)chunk_sec=atof(argv[++i]);
+        else if(!strcmp(argv[i],"--vad")&&i+1<argc)vad_path=argv[++i];
+        else if(!strcmp(argv[i],"--vad-maxseg")&&i+1<argc)vad_maxseg=atoi(argv[++i]);
+        else if(!strcmp(argv[i],"--rep")&&i+1<argc)rep=atof(argv[++i]);
+        else {fprintf(stderr,"usage: %s --enc enc.gguf -m llm.gguf -a audio.wav [-n npred] [--chunk sec] [--vad fsmn-vad.gguf [--vad-maxseg ms]]\n",argv[0]);return 1;}
+    }
+    if(enc_path.empty()||llm_path.empty()||wav_path.empty()){fprintf(stderr,"missing args\n");return 1;}
+
+    std::vector<float> wav;
+    if(!funasr_load_audio_16k_mono(wav_path.c_str(),wav)){fprintf(stderr,"failed to read audio\n");return 1;}
+    int64_t t0=ggml_time_us();
+
+    enc_model em; if(!load_enc(enc_path.c_str(),em))return 1;
+    ggml_backend_load_all();
+    llama_model_params mp=llama_model_default_params(); mp.n_gpu_layers=0;
+    llama_model*model=llama_model_load_from_file(llm_path.c_str(),mp); if(!model)return 1;
+    const llama_vocab*vocab=llama_model_get_vocab(model);
+    llama_context_params cp=llama_context_default_params();
+    cp.n_ctx=2048; cp.n_batch=2048; cp.n_ubatch=2048;
+    llama_context*ctx=llama_init_from_model(model,cp);
+    if(!ctx){fprintf(stderr,"failed to create llama context\n");llama_model_free(model);return 1;}
+    auto sp=llama_sampler_chain_default_params(); llama_sampler*smpl=llama_sampler_chain_init(sp);
+    if(rep!=1.0f) llama_sampler_chain_add(smpl,llama_sampler_init_penalties(256,rep,0.0f,0.0f));
+    llama_sampler_chain_add(smpl,llama_sampler_init_greedy());
+
+    const char*prefix="<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n语音转写：";
+    const char*suffix="<|im_end|>\n<|im_start|>assistant\n";
+    auto tokenize=[&](const char*s){int n=-llama_tokenize(vocab,s,strlen(s),nullptr,0,false,true);
+        std::vector<llama_token> v(n); llama_tokenize(vocab,s,strlen(s),v.data(),n,false,true); return v;};
+    auto pre=tokenize(prefix); auto suf=tokenize(suffix);
+
+    // Build the list of [offset,len] windows to transcribe (in samples).
+    //   --vad : FSMN-VAD speech segments (single-binary front end, replaces fixed chunking)
+    //   --chunk sec : fixed-size chunks ; otherwise the whole file in one window
+    std::vector<std::pair<int,int>> wins;   // {sample offset, sample len}
+    if(!vad_path.empty()){
+        std::vector<std::pair<int,int>> segs; // ms
+        if(!funasr_vad_segments(vad_path,wav,vad_maxseg,segs)){fprintf(stderr,"vad failed\n");return 1;}
+        for(auto&s:segs){ int off=(int)((int64_t)s.first*16000/1000), end=(int)((int64_t)s.second*16000/1000);
+            if(end>(int)wav.size())end=wav.size(); if(end-off>0) wins.push_back({off,end-off}); }
+        fprintf(stderr,"[vad] %zu segments\n",wins.size());
+    } else {
+        int chunk_n = chunk_sec > 0 ? std::max(1, (int)(chunk_sec*16000)) : (int)wav.size();
+        for(size_t off=0; off<wav.size(); off+=chunk_n) wins.push_back({(int)off,(int)std::min((size_t)chunk_n,wav.size()-off)});
+    }
+    std::string full;
+    for (auto& w : wins) {
+        int off = w.first, len = w.second;
+        if (len < WINLEN) continue;                    // too short for one frame
+        std::vector<float> seg(wav.begin()+off, wav.begin()+off+len);
+        int T=0; auto fbank=compute_fbank(seg,T);
+        int D=0; auto adp=run_encoder(em,fbank,T,560,D);
+        int ol=1+(T-3+2)/2; ol=1+(ol-3+2)/2; int n_aud=(ol-1)/2+1;
+
+        llama_memory_clear(llama_get_memory(ctx), true);  // fresh context per chunk
+        int n_past=0;
+        decode_batch(ctx,pre.size(),pre.data(),nullptr,0,n_past,false);
+        decode_batch(ctx,n_aud,nullptr,adp.data(),D,n_past,false);
+        decode_batch(ctx,suf.size(),suf.data(),nullptr,0,n_past,true);
+        llama_token tk=llama_sampler_sample(smpl,ctx,-1);
+        for(int i=0;i<npred;i++){
+            if(llama_vocab_is_eog(vocab,tk))break;
+            char buf[256]; int k=llama_token_to_piece(vocab,tk,buf,sizeof(buf),0,true);
+            if(k>0) full.append(buf,k);
+            decode_batch(ctx,1,&tk,nullptr,0,n_past,true);
+            tk=llama_sampler_sample(smpl,ctx,-1);
+        }
+    }
+    printf("%s\n", full.c_str());
+    int64_t t2=ggml_time_us();
+    fprintf(stderr,"[done] %.2fs ; chunk=%.0fs\n",(t2-t0)/1e6, chunk_sec);
+    llama_sampler_free(smpl); llama_free(ctx); llama_model_free(model);
+    if(em.ctx_w) ggml_free(em.ctx_w);
+    return 0;
+}

runtime/llama.cpp/fun-asr-nano/funasr-embd/CMakeLists.txt

@@ -0,0 +1,5 @@
+set(TARGET llama-funasr-embd)
+add_executable(${TARGET} funasr-embd.cpp)
+install(TARGETS ${TARGET} RUNTIME)
+target_link_libraries(${TARGET} PRIVATE llama ${CMAKE_THREAD_LIBS_INIT})
+target_compile_features(${TARGET} PRIVATE cxx_std_17)

runtime/llama.cpp/fun-asr-nano/funasr-embd/funasr-embd.cpp

@@ -0,0 +1,141 @@
+// funasr-embd: decode Fun-ASR-Nano audio embeddings through a Qwen3 GGUF.
+//
+// Reads an inputs_embeds matrix (produced by the FunASR audio encoder+adaptor,
+// concatenated with the text prompt embeddings) and feeds it directly to the
+// LLM via llama_decode's embedding input path -- the same mechanism llava/mtmd
+// use to inject vision embeddings. This bridges FunASR's audio frontend to the
+// llama.cpp / GGUF ecosystem.
+//
+// embeds.bin format: int32 n_tokens, int32 n_embd, then n_tokens*n_embd float32
+// (row-major). n_embd must equal the model's input embedding dim.
+
+#include "llama.h"
+
+#include <cstdint>
+#include <cstdio>
+#include <cstring>
+#include <string>
+#include <vector>
+
+static void print_usage(char ** argv) {
+    printf("\nusage: %s -m model.gguf -e embeds.bin [-n n_predict] [-ngl n_gpu_layers]\n\n", argv[0]);
+}
+
+// read embeds.bin -> (n_tokens, n_embd, data)
+static bool read_embeds(const std::string & path, int & n_tokens, int & n_embd, std::vector<float> & data) {
+    FILE * f = fopen(path.c_str(), "rb");
+    if (!f) { fprintf(stderr, "error: cannot open %s\n", path.c_str()); return false; }
+    int32_t hdr[2];
+    if (fread(hdr, sizeof(int32_t), 2, f) != 2) { fclose(f); return false; }
+    n_tokens = hdr[0];
+    n_embd   = hdr[1];
+    if (n_tokens <= 0 || n_embd <= 0) { fclose(f); return false; }
+    data.resize((size_t) n_tokens * n_embd);
+    size_t got = fread(data.data(), sizeof(float), data.size(), f);
+    fclose(f);
+    if (got != data.size()) {
+        fprintf(stderr, "error: short read (%zu/%zu floats)\n", got, data.size());
+        return false;
+    }
+    return true;
+}
+
+int main(int argc, char ** argv) {
+    std::string model_path, embeds_path;
+    int n_predict = 512;
+    int ngl = 0; // CPU by default; the whole point is CPU/edge
+
+    for (int i = 1; i < argc; i++) {
+        if      (!strcmp(argv[i], "-m")   && i + 1 < argc) model_path  = argv[++i];
+        else if (!strcmp(argv[i], "-e")   && i + 1 < argc) embeds_path = argv[++i];
+        else if (!strcmp(argv[i], "-n")   && i + 1 < argc) n_predict   = std::stoi(argv[++i]);
+        else if (!strcmp(argv[i], "-ngl") && i + 1 < argc) ngl         = std::stoi(argv[++i]);
+        else { print_usage(argv); return 1; }
+    }
+    if (model_path.empty() || embeds_path.empty()) { print_usage(argv); return 1; }
+
+    int n_tokens = 0, n_embd_in = 0;
+    std::vector<float> embd;
+    if (!read_embeds(embeds_path, n_tokens, n_embd_in, embd)) return 1;
+    fprintf(stderr, "loaded embeds: n_tokens=%d n_embd=%d\n", n_tokens, n_embd_in);
+
+    ggml_backend_load_all();
+
+    llama_model_params mparams = llama_model_default_params();
+    mparams.n_gpu_layers = ngl;
+    llama_model * model = llama_model_load_from_file(model_path.c_str(), mparams);
+    if (!model) { fprintf(stderr, "error: unable to load model\n"); return 1; }
+
+    const llama_vocab * vocab = llama_model_get_vocab(model);
+    const int n_embd_model = llama_model_n_embd_inp(model);
+    if (n_embd_in != n_embd_model) {
+        fprintf(stderr, "error: embd dim %d != model input embd dim %d\n", n_embd_in, n_embd_model);
+        llama_model_free(model);
+        return 1;
+    }
+
+    llama_context_params cparams = llama_context_default_params();
+    cparams.n_ctx   = n_tokens + n_predict + 8;
+    cparams.n_batch = n_tokens + 8;   // process the whole embd prompt in one ubatch
+    cparams.n_ubatch = n_tokens + 8;
+    cparams.no_perf = false;
+    llama_context * ctx = llama_init_from_model(model, cparams);
+    if (!ctx) { fprintf(stderr, "error: failed to create context\n"); llama_model_free(model); return 1; }
+
+    auto sparams = llama_sampler_chain_default_params();
+    llama_sampler * smpl = llama_sampler_chain_init(sparams);
+    llama_sampler_chain_add(smpl, llama_sampler_init_greedy());
+
+    // --- decode the embedding prompt (causal, single sequence, positions 0..n-1) ---
+    std::vector<llama_pos>      pos(n_tokens);
+    std::vector<int32_t>        n_seq_id(n_tokens, 1);
+    std::vector<llama_seq_id>   seq_id_0(1, 0);
+    std::vector<llama_seq_id *> seq_id(n_tokens);
+    std::vector<int8_t>         logits(n_tokens, 0);
+    for (int i = 0; i < n_tokens; i++) { pos[i] = i; seq_id[i] = seq_id_0.data(); }
+    logits[n_tokens - 1] = 1; // only need logits for the last position
+
+    llama_batch batch = {
+        /*n_tokens =*/ n_tokens,
+        /*token    =*/ nullptr,
+        /*embd     =*/ embd.data(),
+        /*pos      =*/ pos.data(),
+        /*n_seq_id =*/ n_seq_id.data(),
+        /*seq_id   =*/ seq_id.data(),
+        /*logits   =*/ logits.data(),
+    };
+
+    const int64_t t_start = ggml_time_us();
+    if (llama_decode(ctx, batch) != 0) {
+        fprintf(stderr, "error: llama_decode failed on embd prompt\n");
+        return 1;
+    }
+
+    // --- generation loop ---
+    std::string out;
+    int n_decode = 0;
+    llama_token tok = llama_sampler_sample(smpl, ctx, -1);
+    for (int n_pos = n_tokens; n_pos < n_tokens + n_predict; ) {
+        if (llama_vocab_is_eog(vocab, tok)) break;
+        char buf[256];
+        int n = llama_token_to_piece(vocab, tok, buf, sizeof(buf), 0, true);
+        if (n > 0) { out.append(buf, n); }
+        printf("%.*s", n > 0 ? n : 0, buf);
+        fflush(stdout);
+
+        llama_batch tb = llama_batch_get_one(&tok, 1);
+        if (llama_decode(ctx, tb) != 0) { fprintf(stderr, "error: decode failed\n"); return 1; }
+        n_pos += 1;
+        n_decode += 1;
+        tok = llama_sampler_sample(smpl, ctx, -1);
+    }
+    printf("\n");
+    const int64_t t_end = ggml_time_us();
+    fprintf(stderr, "\n[funasr-embd] generated %d tokens in %.2f s (%.1f tok/s)\n",
+            n_decode, (t_end - t_start) / 1e6, n_decode / ((t_end - t_start) / 1e6));
+
+    llama_sampler_free(smpl);
+    llama_free(ctx);
+    llama_model_free(model);
+    return 0;
+}

runtime/llama.cpp/fun-asr-nano/funasr-encoder/CMakeLists.txt

@@ -0,0 +1,5 @@
+set(TARGET llama-funasr-encoder)
+add_executable(${TARGET} funasr-encoder.cpp)
+install(TARGETS ${TARGET} RUNTIME)
+target_link_libraries(${TARGET} PRIVATE ggml ${CMAKE_THREAD_LIBS_INIT})
+target_compile_features(${TARGET} PRIVATE cxx_std_17)

runtime/llama.cpp/fun-asr-nano/funasr-encoder/funasr-encoder.cpp

@@ -0,0 +1,275 @@
+// funasr-encoder: ggml C++ forward pass for the Fun-ASR-Nano audio encoder
+// (SenseVoice SAN-M, 50+20 layers) + Transformer adaptor.
+//
+// Input : fbank.bin (T x 560 f32, the encoder input features)
+// Output: out.bin   (T' x 1024 f32, audio embeddings for the LLM)
+// Weights: funasr-encoder.gguf (exported by export_encoder_gguf.py)
+//
+// Validated layer-by-layer against PyTorch golden dumps. fbank is currently
+// produced in Python; porting the fbank frontend to C++ is the remaining piece.
+
+#include "ggml.h"
+#include "ggml-cpu.h"
+#include "ggml-alloc.h"
+#include "ggml-backend.h"
+#include "gguf.h"
+
+#include <cmath>
+#include <cstdint>
+#include <cstdio>
+#include <cstring>
+#include <map>
+#include <string>
+#include <vector>
+
+struct cfg {
+    int input_size = 560, d_model = 512, n_head = 4, ffn = 2048;
+    int num_blocks = 50, tp_blocks = 20, kernel = 11;
+    int adp_llm = 1024, adp_ffn = 2048, adp_layers = 2, adp_head = 8;
+};
+
+static const float LN_EPS = 1e-5f;
+
+struct funasr_model {
+    cfg c;
+    struct ggml_context * ctx_w = nullptr;   // weights (CPU malloc, data set)
+    std::map<std::string, struct ggml_tensor *> t;
+    struct ggml_tensor * get(const std::string & n) {
+        auto it = t.find(n);
+        if (it == t.end()) { fprintf(stderr, "missing tensor: %s\n", n.c_str()); exit(1); }
+        return it->second;
+    }
+};
+
+static bool load_model(const char * path, funasr_model & m) {
+    struct gguf_init_params p = { /*no_alloc=*/false, /*ctx=*/&m.ctx_w };
+    struct gguf_context * gguf = gguf_init_from_file(path, p);
+    if (!gguf) { fprintf(stderr, "failed to load gguf %s\n", path); return false; }
+    auto rd = [&](const char * k, int def) {
+        int i = gguf_find_key(gguf, k); return i < 0 ? def : (int) gguf_get_val_u32(gguf, i);
+    };
+    m.c.input_size = rd("funasr.enc.input_size", 560);
+    m.c.d_model    = rd("funasr.enc.output_size", 512);
+    m.c.n_head     = rd("funasr.enc.attention_heads", 4);
+    m.c.ffn        = rd("funasr.enc.linear_units", 2048);
+    m.c.num_blocks = rd("funasr.enc.num_blocks", 50);
+    m.c.tp_blocks  = rd("funasr.enc.tp_blocks", 20);
+    m.c.kernel     = rd("funasr.enc.kernel_size", 11);
+    m.c.adp_llm    = rd("funasr.adp.llm_dim", 1024);
+    m.c.adp_ffn    = rd("funasr.adp.ffn_dim", 2048);
+    m.c.adp_layers = rd("funasr.adp.n_layer", 2);
+    m.c.adp_head   = rd("funasr.adp.attention_heads", 8);
+    int n = gguf_get_n_tensors(gguf);
+    for (int i = 0; i < n; i++) {
+        const char * name = gguf_get_tensor_name(gguf, i);
+        m.t[name] = ggml_get_tensor(m.ctx_w, name);
+    }
+    fprintf(stderr, "loaded %d tensors; cfg: d_model=%d heads=%d blocks=%d tp=%d kernel=%d adp_llm=%d\n",
+            n, m.c.d_model, m.c.n_head, m.c.num_blocks, m.c.tp_blocks, m.c.kernel, m.c.adp_llm);
+    gguf_free(gguf);
+    return true;
+}
+
+// helpers ---------------------------------------------------------------
+static struct ggml_tensor * linear(ggml_context * ctx, ggml_tensor * w, ggml_tensor * b, ggml_tensor * x) {
+    struct ggml_tensor * y = ggml_mul_mat(ctx, w, x);   // [out, T]
+    if (b) y = ggml_add(ctx, y, b);
+    return y;
+}
+static struct ggml_tensor * layernorm(ggml_context * ctx, ggml_tensor * x, ggml_tensor * g, ggml_tensor * b) {
+    x = ggml_norm(ctx, x, LN_EPS);
+    x = ggml_mul(ctx, x, g);
+    x = ggml_add(ctx, x, b);
+    return x;
+}
+
+// SAN-M self-attention + FSMN. x:[D_in,T] -> [d_model,T]
+static struct ggml_tensor * sanm_attn(ggml_context * ctx, funasr_model & m, const std::string & pfx,
+                                      ggml_tensor * x, int T) {
+    const int D = m.c.d_model, H = m.c.n_head, dk = D / H, K = m.c.kernel;
+    struct ggml_tensor * qkv = linear(ctx, m.get(pfx + "linear_q_k_v.weight"),
+                                      m.get(pfx + "linear_q_k_v.bias"), x);   // [3D, T]
+    size_t nb1 = qkv->nb[1];
+    struct ggml_tensor * q = ggml_cont(ctx, ggml_view_2d(ctx, qkv, D, T, nb1, 0));
+    struct ggml_tensor * k = ggml_cont(ctx, ggml_view_2d(ctx, qkv, D, T, nb1, (size_t) D * sizeof(float)));
+    struct ggml_tensor * v = ggml_cont(ctx, ggml_view_2d(ctx, qkv, D, T, nb1, (size_t) 2 * D * sizeof(float)));
+
+    // FSMN: depthwise conv1d along time (per-channel kernel K, "same" padding),
+    // plus residual v. Implemented as an exact f32 shift-accumulate to avoid the
+    // F16-only ggml_conv_1d_dw path. fsmn kernel stored as [D, K] (ne0=D, ne1=K).
+    const int pad = (K - 1) / 2;
+    struct ggml_tensor * fk = m.get(pfx + "fsmn_block.weight");               // [D, K]
+    struct ggml_tensor * vpad = ggml_pad_ext(ctx, v, 0, 0, pad, pad, 0, 0, 0, 0); // [D, T+2*pad]
+    struct ggml_tensor * fsmn = v;                                            // residual
+    for (int j = 0; j < K; j++) {
+        struct ggml_tensor * sl = ggml_view_2d(ctx, vpad, D, T, vpad->nb[1], (size_t) j * vpad->nb[1]);
+        struct ggml_tensor * wj = ggml_view_1d(ctx, fk, D, (size_t) j * fk->nb[1]);
+        fsmn = ggml_add(ctx, fsmn, ggml_mul(ctx, ggml_cont(ctx, sl), wj));
+    }
+
+    // multi-head attention
+    q = ggml_reshape_3d(ctx, q, dk, H, T);
+    k = ggml_reshape_3d(ctx, k, dk, H, T);
+    struct ggml_tensor * vh = ggml_reshape_3d(ctx, v, dk, H, T);
+    q = ggml_permute(ctx, q, 0, 2, 1, 3);   // [dk, T, H]
+    k = ggml_permute(ctx, k, 0, 2, 1, 3);   // [dk, T, H]
+    vh = ggml_cont(ctx, ggml_permute(ctx, vh, 1, 2, 0, 3)); // [T, dk, H]
+    struct ggml_tensor * kq = ggml_mul_mat(ctx, k, q);      // [T, T, H]
+    kq = ggml_scale(ctx, kq, 1.0f / sqrtf((float) dk));
+    kq = ggml_soft_max(ctx, kq);
+    struct ggml_tensor * kqv = ggml_mul_mat(ctx, vh, kq);   // [dk, T, H]
+    kqv = ggml_permute(ctx, kqv, 0, 2, 1, 3);               // [dk, H, T]
+    kqv = ggml_cont_2d(ctx, kqv, D, T);                     // [D, T]
+    struct ggml_tensor * att = linear(ctx, m.get(pfx + "linear_out.weight"),
+                                      m.get(pfx + "linear_out.bias"), kqv);
+    return ggml_add(ctx, att, fsmn);
+}
+
+// one SAN-M encoder layer. in_size may differ from d_model (first layer)
+static struct ggml_tensor * sanm_layer(ggml_context * ctx, funasr_model & m, const std::string & pfx,
+                                       ggml_tensor * x, int T, bool residual_attn) {
+    struct ggml_tensor * res = x;
+    struct ggml_tensor * h = layernorm(ctx, x, m.get(pfx + "norm1.weight"), m.get(pfx + "norm1.bias"));
+    struct ggml_tensor * sa = sanm_attn(ctx, m, pfx + "self_attn.", h, T);
+    x = residual_attn ? ggml_add(ctx, res, sa) : sa;
+    res = x;
+    h = layernorm(ctx, x, m.get(pfx + "norm2.weight"), m.get(pfx + "norm2.bias"));
+    h = linear(ctx, m.get(pfx + "feed_forward.w_1.weight"), m.get(pfx + "feed_forward.w_1.bias"), h);
+    h = ggml_relu(ctx, h);
+    h = linear(ctx, m.get(pfx + "feed_forward.w_2.weight"), m.get(pfx + "feed_forward.w_2.bias"), h);
+    return ggml_add(ctx, res, h);
+}
+
+// standard transformer layer (adaptor). d=adp_llm
+static struct ggml_tensor * adp_layer(ggml_context * ctx, funasr_model & m, const std::string & pfx,
+                                      ggml_tensor * x, int T) {
+    const int D = m.c.adp_llm, H = m.c.adp_head, dk = D / H;
+    struct ggml_tensor * res = x;
+    struct ggml_tensor * h = layernorm(ctx, x, m.get(pfx + "norm1.weight"), m.get(pfx + "norm1.bias"));
+    struct ggml_tensor * q = linear(ctx, m.get(pfx + "self_attn.linear_q.weight"), m.get(pfx + "self_attn.linear_q.bias"), h);
+    struct ggml_tensor * k = linear(ctx, m.get(pfx + "self_attn.linear_k.weight"), m.get(pfx + "self_attn.linear_k.bias"), h);
+    struct ggml_tensor * v = linear(ctx, m.get(pfx + "self_attn.linear_v.weight"), m.get(pfx + "self_attn.linear_v.bias"), h);
+    q = ggml_permute(ctx, ggml_reshape_3d(ctx, q, dk, H, T), 0, 2, 1, 3);
+    k = ggml_permute(ctx, ggml_reshape_3d(ctx, k, dk, H, T), 0, 2, 1, 3);
+    struct ggml_tensor * vh = ggml_cont(ctx, ggml_permute(ctx, ggml_reshape_3d(ctx, v, dk, H, T), 1, 2, 0, 3));
+    struct ggml_tensor * kq = ggml_soft_max(ctx, ggml_scale(ctx, ggml_mul_mat(ctx, k, q), 1.0f / sqrtf((float) dk)));
+    struct ggml_tensor * kqv = ggml_cont_2d(ctx, ggml_permute(ctx, ggml_mul_mat(ctx, vh, kq), 0, 2, 1, 3), D, T);
+    struct ggml_tensor * att = linear(ctx, m.get(pfx + "self_attn.linear_out.weight"), m.get(pfx + "self_attn.linear_out.bias"), kqv);
+    x = ggml_add(ctx, res, att);
+    res = x;
+    h = layernorm(ctx, x, m.get(pfx + "norm2.weight"), m.get(pfx + "norm2.bias"));
+    h = linear(ctx, m.get(pfx + "feed_forward.w_1.weight"), m.get(pfx + "feed_forward.w_1.bias"), h);
+    h = ggml_relu(ctx, h);
+    h = linear(ctx, m.get(pfx + "feed_forward.w_2.weight"), m.get(pfx + "feed_forward.w_2.bias"), h);
+    return ggml_add(ctx, res, h);
+}
+
+// sinusoidal position encoding, depth = input feature dim, positions 1..T
+static void add_posenc(std::vector<float> & x, int T, int depth) {
+    double inc = log(10000.0) / (depth / 2.0 - 1.0);
+    for (int t = 0; t < T; t++) {
+        double pos = t + 1;  // positions start at 1
+        for (int i = 0; i < depth / 2; i++) {
+            double its = exp(i * -inc);
+            double st = pos * its;
+            x[(size_t) t * depth + i]             += (float) sin(st);
+            x[(size_t) t * depth + depth / 2 + i] += (float) cos(st);
+        }
+    }
+}
+
+int main(int argc, char ** argv) {
+    std::string gguf_path, fbank_path, out_path = "out.bin";
+    int limit = -1;          // -L: run only first N (encoders0+encoders) layers, dump running x
+    bool run_adaptor = true;
+    for (int i = 1; i < argc; i++) {
+        if      (!strcmp(argv[i], "-m") && i+1 < argc) gguf_path  = argv[++i];
+        else if (!strcmp(argv[i], "-f") && i+1 < argc) fbank_path = argv[++i];
+        else if (!strcmp(argv[i], "-o") && i+1 < argc) out_path   = argv[++i];
+        else if (!strcmp(argv[i], "-L") && i+1 < argc) { limit = atoi(argv[++i]); run_adaptor = false; }
+        else { fprintf(stderr, "usage: %s -m enc.gguf -f fbank.bin [-o out.bin] [-L nlayers]\n", argv[0]); return 1; }
+    }
+
+    funasr_model m;
+    if (!load_model(gguf_path.c_str(), m)) return 1;
+
+    // read fbank.bin (T x F)
+    FILE * f = fopen(fbank_path.c_str(), "rb");
+    if (!f) { fprintf(stderr, "cannot open %s\n", fbank_path.c_str()); return 1; }
+    int32_t T, F; if (fread(&T, 4, 1, f) != 1 || fread(&F, 4, 1, f) != 1) { fclose(f); return 1; }
+    std::vector<float> fbank((size_t) T * F);
+    if (fread(fbank.data(), sizeof(float), fbank.size(), f) != fbank.size()) { fclose(f); return 1; }
+    fclose(f);
+    fprintf(stderr, "fbank: T=%d F=%d\n", T, F);
+
+    // pre-scale (*sqrt(d_model)) and add position encoding on the host
+    float scale = sqrtf((float) m.c.d_model);
+    for (auto & v : fbank) v *= scale;
+    add_posenc(fbank, T, F);
+
+    // backend + compute context
+    ggml_backend_t backend = ggml_backend_cpu_init();
+    size_t ctx_size = (size_t) 1024*1024*1024;  // graph metadata
+    struct ggml_init_params cp = { ctx_size, nullptr, true };
+    struct ggml_context * ctx = ggml_init(cp);
+
+    struct ggml_tensor * inp = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, F, T);
+    ggml_set_name(inp, "inp");
+    ggml_set_input(inp);
+
+    struct ggml_tensor * x = inp;
+    int done = 0;
+    bool stop = false;
+    // encoders0 (1 layer, in_size=input_size != d_model -> no attn residual)
+    x = sanm_layer(ctx, m, "audio_encoder.encoders0.0.", x, T, /*residual_attn=*/false);
+    done++;
+    if (limit >= 0 && done >= limit) stop = true;
+    // encoders (num_blocks-1 layers)
+    for (int i = 0; i < m.c.num_blocks - 1 && !stop; i++) {
+        x = sanm_layer(ctx, m, "audio_encoder.encoders." + std::to_string(i) + ".", x, T, true);
+        done++;
+        if (limit >= 0 && done >= limit) stop = true;
+    }
+    if (!stop) {
+        x = layernorm(ctx, x, m.get("audio_encoder.after_norm.weight"), m.get("audio_encoder.after_norm.bias"));
+        for (int i = 0; i < m.c.tp_blocks; i++)
+            x = sanm_layer(ctx, m, "audio_encoder.tp_encoders." + std::to_string(i) + ".", x, T, true);
+        x = layernorm(ctx, x, m.get("audio_encoder.tp_norm.weight"), m.get("audio_encoder.tp_norm.bias"));
+        // adaptor: downsample_rate=1 -> linear1(relu)linear2 then blocks
+        if (run_adaptor) {
+            x = linear(ctx, m.get("audio_adaptor.linear1.weight"), m.get("audio_adaptor.linear1.bias"), x);
+            x = ggml_relu(ctx, x);
+            x = linear(ctx, m.get("audio_adaptor.linear2.weight"), m.get("audio_adaptor.linear2.bias"), x);
+            for (int i = 0; i < m.c.adp_layers; i++)
+                x = adp_layer(ctx, m, "audio_adaptor.blocks." + std::to_string(i) + ".", x, T);
+        }
+    }
+    ggml_set_output(x);
+
+    struct ggml_cgraph * gf = ggml_new_graph_custom(ctx, 32768, false);
+    ggml_build_forward_expand(gf, x);
+
+    ggml_gallocr_t galloc = ggml_gallocr_new(ggml_backend_cpu_buffer_type());
+    ggml_gallocr_alloc_graph(galloc, gf);
+    ggml_backend_tensor_set(inp, fbank.data(), 0, ggml_nbytes(inp));
+
+    ggml_backend_cpu_set_n_threads(backend, 8);
+    int64_t t0 = ggml_time_us();
+    if (ggml_backend_graph_compute(backend, gf) != GGML_STATUS_SUCCESS) {
+        fprintf(stderr, "compute failed\n"); return 1;
+    }
+    int64_t t1 = ggml_time_us();
+
+    int D = (int) x->ne[0];
+    std::vector<float> out((size_t) D * T);
+    ggml_backend_tensor_get(x, out.data(), 0, ggml_nbytes(x));
+    FILE * fo = fopen(out_path.c_str(), "wb");
+    if (!fo) { fprintf(stderr, "failed to open output file %s\n", out_path.c_str()); return 1; }
+    fwrite(&T, 4, 1, fo); fwrite(&D, 4, 1, fo); fwrite(out.data(), sizeof(float), out.size(), fo);
+    fclose(fo);
+    fprintf(stderr, "done: wrote %s [%d x %d] in %.2f s (layers run=%d, adaptor=%d)\n",
+            out_path.c_str(), T, D, (t1 - t0)/1e6, done, run_adaptor && !stop);
+    ggml_gallocr_free(galloc); ggml_free(ctx); ggml_backend_free(backend);
+    if (m.ctx_w) ggml_free(m.ctx_w);
+    return 0;
+}