Initial commit: FunASR Speech Recognition Toolkit

Add complete FunASR codebase including models, runtime, and documentation.

examples/industrial_data_pretraining/whisper/README.md

@@ -0,0 +1,65 @@
+# Whisper Fine-tuning with FunASR
+
+Fine-tune OpenAI Whisper models on your own data using FunASR's training framework.
+
+## Supported Models
+
+- whisper-tiny / whisper-tiny.en
+- whisper-base / whisper-base.en
+- whisper-small / whisper-small.en
+- whisper-medium / whisper-medium.en
+- whisper-large-v1 / whisper-large-v2 / whisper-large-v3 / whisper-large-v3-turbo
+
+## Data Preparation
+
+Prepare data in JSONL format:
+
+```json
+{"key": "utt001", "source": "/path/to/audio1.wav", "target": "the transcription text"}
+{"key": "utt002", "source": "/path/to/audio2.wav", "target": "another transcription"}
+```
+
+## Fine-tuning
+
+```bash
+bash finetune.sh
+```
+
+Or customize directly:
+
+```python
+from funasr import AutoModel
+
+model = AutoModel(model="Whisper-large-v3", model_conf={"hub": "openai"})
+
+# Training uses the forward() method which computes cross-entropy loss
+# on (mel-spectrogram, token_ids) pairs
+```
+
+## Key Parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| model | Whisper-large-v3 | Model size |
+| lr | 1e-5 | Learning rate (lower for larger models) |
+| max_epoch | 10 | Training epochs |
+| batch_size | 4 | Per-GPU batch size |
+| warmup_steps | 500 | LR warmup |
+
+## Tips
+
+- For Chinese fine-tuning, use `whisper-large-v3` (best multilingual base)
+- Freeze encoder for faster training: add `++train_conf.freeze_param="model.encoder"`
+- Use smaller learning rates (1e-5 ~ 5e-6) to avoid catastrophic forgetting
+- Recommended: 100+ hours of target-domain audio for meaningful improvement
+
+## After Fine-tuning
+
+```python
+from funasr import AutoModel
+
+# Load fine-tuned model
+model = AutoModel(model="/path/to/exp/whisper_finetune")
+result = model.generate(input="test.wav")
+print(result[0]["text"])
+```

examples/industrial_data_pretraining/whisper/demo.py

@@ -0,0 +1,30 @@
+#!/usr/bin/env python3
+# -*- encoding: utf-8 -*-
+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.
+#  MIT License  (https://opensource.org/licenses/MIT)
+
+# To install requirements: pip3 install -U openai-whisper
+
+from funasr import AutoModel
+
+model = AutoModel(
+    model="Whisper-large-v3-turbo",
+    vad_model="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch",
+    vad_kwargs={"max_single_segment_time": 30000},
+)
+
+DecodingOptions = {
+    "task": "transcribe",
+    "language": None,
+    "beam_size": None,
+    "fp16": True,
+    "without_timestamps": False,
+    "prompt": None,
+}
+res = model.generate(
+    DecodingOptions=DecodingOptions,
+    batch_size_s=0,
+    input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav",
+)
+
+print(res)

examples/industrial_data_pretraining/whisper/demo_from_openai.py

@@ -0,0 +1,34 @@
+#!/usr/bin/env python3
+# -*- encoding: utf-8 -*-
+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.
+#  MIT License  (https://opensource.org/licenses/MIT)
+
+# To install requirements: pip3 install -U openai-whisper
+
+from funasr import AutoModel
+
+# model = AutoModel(model="Whisper-small", hub="openai")
+# model = AutoModel(model="Whisper-medium", hub="openai")
+# model = AutoModel(model="Whisper-large-v2", hub="openai")
+model = AutoModel(
+    model="Whisper-large-v3-turbo",
+    vad_model="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch",
+    vad_kwargs={"max_single_segment_time": 30000},
+    hub="openai",
+)
+
+DecodingOptions = {
+    "task": "transcribe",
+    "language": None,
+    "beam_size": None,
+    "fp16": True,
+    "without_timestamps": False,
+    "prompt": None,
+}
+res = model.generate(
+    DecodingOptions=DecodingOptions,
+    batch_size_s=0,
+    input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav",
+)
+
+print(res)

examples/industrial_data_pretraining/whisper/finetune.sh

@@ -0,0 +1,31 @@
+#!/bin/bash
+# Whisper Fine-tuning with FunASR
+#
+# This script fine-tunes OpenAI Whisper models on custom data using FunASR's training framework.
+# Supports: whisper-tiny, whisper-base, whisper-small, whisper-medium, whisper-large-v3
+#
+# Data format: JSONL with "audio" and "text" fields
+# {"key": "utt1", "source": "/path/to/audio.wav", "target": "transcription text"}
+
+export CUDA_VISIBLE_DEVICES=0,1
+
+model_name="Whisper-large-v3"
+train_data="data/train.jsonl"
+val_data="data/val.jsonl"
+output_dir="exp/whisper_finetune"
+
+python -m funasr.bin.train \
+    ++model="${model_name}" \
+    ++model_conf.hub="openai" \
+    ++train_data_set_list="${train_data}" \
+    ++valid_data_set_list="${val_data}" \
+    ++dataset_conf.batch_size=4 \
+    ++dataset_conf.num_workers=4 \
+    ++train_conf.output_dir="${output_dir}" \
+    ++train_conf.max_epoch=10 \
+    ++train_conf.lr=1e-5 \
+    ++train_conf.warmup_steps=500 \
+    ++optim="adam" \
+    ++optim_conf.lr=1e-5 \
+    ++scheduler="warmuplr" \
+    ++scheduler_conf.warmup_steps=500

examples/industrial_data_pretraining/whisper/infer.sh

@@ -0,0 +1,22 @@
+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.
+#  MIT License  (https://opensource.org/licenses/MIT)
+
+# To install requirements: pip3 install -U openai-whisper
+
+# method1, inference from model hub
+
+# for more input type, please ref to readme.md
+input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav"
+
+output_dir="./outputs/debug"
+
+model="iic/speech_whisper-large_asr_multilingual"
+
+
+device="cuda:0" # "cuda:0" for gpu0, "cuda:1" for gpu1, "cpu"
+
+python -m funasr.bin.inference \
+++model=${model} \
+++input="${input}" \
+++output_dir="${output_dir}" \
+++device="${device}" \

examples/industrial_data_pretraining/whisper/infer_from_local.sh

@@ -0,0 +1,42 @@
+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.
+#  MIT License  (https://opensource.org/licenses/MIT)
+
+# To install requirements: pip3 install -U openai-whisper
+
+# method2, inference from local model
+
+# for more input type, please ref to readme.md
+input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav"
+
+output_dir="./outputs/debug"
+
+workspace=`pwd`
+
+# download model
+local_path_root=${workspace}/modelscope_models
+mkdir -p ${local_path_root}
+#Whisper-large-v2
+#local_path=${local_path_root}/speech_whisper-large_asr_multilingual
+#git clone https://www.modelscope.cn/iic/speech_whisper-large_asr_multilingual.git ${local_path}
+#init_param="${local_path}/large-v2.pt"
+#Whisper-large-v3
+local_path=${local_path_root}/Whisper-large-v3
+git clone https://www.modelscope.cn/iic/Whisper-large-v3.git ${local_path}
+init_param="${local_path}/large-v3.pt"
+
+device="cuda:0" # "cuda:0" for gpu0, "cuda:1" for gpu1, "cpu"
+
+config="config.yaml"
+
+
+python -m funasr.bin.inference \
+--config-path "${local_path}" \
+--config-name "${config}" \
+++init_param="${init_param}" \
+++input="${input}" \
+++output_dir="${output_dir}" \
+++device="${device}" \
+
+
+
+

examples/industrial_data_pretraining/whisper/infer_from_openai.sh

@@ -0,0 +1,26 @@
+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.
+#  MIT License  (https://opensource.org/licenses/MIT)
+
+# To install requirements: pip3 install -U openai-whisper
+
+# method1, inference from model hub
+
+# for more input type, please ref to readme.md
+input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav"
+
+output_dir="./outputs/debug"
+
+#model="Whisper-small"
+#model="Whisper-medium"
+model="Whisper-large-v2"
+#model="Whisper-large-v3"
+hub="openai"
+
+device="cuda:0" # "cuda:0" for gpu0, "cuda:1" for gpu1, "cpu"
+
+python -m funasr.bin.inference \
+++model=${model} \
+++hub=${hub} \
+++input="${input}" \
+++output_dir="${output_dir}" \
+++device="${device}" \