ASR benchmark runbook#

This page runs the ASR benchmarks of SGLang-Omni on one GPU and explains how to measure them accurately. Audio goes in and text comes out. Correctness is WER, or CER and cpCER for MOSS-Transcribe-Diarize. Performance is throughput, latency and RTF.

The workloads are the ones in the ASR CI (.github/workflows/test-asr-ci.yaml). Run every command in bash, in order. The only values you set yourself are in Variables.

What is Measured#

CI stage	Model	Data	Concurrency
1	MOSS-Transcribe-Diarize, multiple speakers	`movies800times` (800 samples, run once without and once with streaming), `aishell4_long` (20), `googletime` (25)	16
2	Qwen3-ASR-1.7B, Whisper-large-v3 or Fun-ASR-Nano	SeedTTS EN (1088) and ZH (2020)	32

CI stage 1 also has aishell4_long90, one 90 minute sample stitched together from three aishell4_long clips. It only runs through pytest (see Run the CI tests). CI stage 2 runs one of the three models per CI run, picked at random unless a run-fun-asr, run-qwen3-asr or run-whisper-asr label is set.

This page uses one server on one GPU, with the same datasets, concurrency and request settings as CI.

Variables#

Run the host block on the host before Step 1, and again in every new host shell. Change only GPU and HF_TOKEN:

export GPU=0
export HF_TOKEN=hf_your_token_here
export IMAGE=hongccc/sglang-omni@sha256:ebe4239e29a764ee3a2806385c061c5fd438a26f01458e503d3822dcba5790df
export CPUSET=$(cat /sys/bus/pci/devices/$(nvidia-smi -i $GPU --query-gpu=pci.bus_id --format=csv,noheader | tail -c 13 | tr A-F a-f)/local_cpulist)
echo $CPUSET

GPU is the GPU under test. No other job may use it. HF_TOKEN needs access to the three private MOSS-Transcribe-Diarize datasets. IMAGE is the CI image. CPUSET is the CPUs on the NUMA node of GPU, such as 0-55,112-167.

Run the container block inside the container after Step 1, and again after every docker exec. Change only COMMIT (the commit to measure) and NAME (a label for its results):

export COMMIT=origin/main
export NAME=main
export SEEDTTS_REV=27f4c1adee83b5b29b7c4b375f6b976324bda308
export HF_HUB=/root/.cache/huggingface/hub
export QWEN3_ASR=$HF_HUB/models--Qwen--Qwen3-ASR-1.7B/snapshots/7278e1e70fe206f11671096ffdd38061171dd6e5
export WHISPER=$HF_HUB/models--openai--whisper-large-v3/snapshots/06f233fe06e710322aca913c1bc4249a0d71fce1
export FUN_ASR=$HF_HUB/models--FunAudioLLM--Fun-ASR-Nano-2512-hf/snapshots/7ef6cafdc445b5beff0b730cbeb51952c793ca6e
export MOSS_TD=$HF_HUB/models--OpenMOSS-Team--MOSS-Transcribe-Diarize/snapshots/704aa4a9c304e8520be88901e0d1960158ef5b15

CI loads these models by repo id without a revision. The SHAs above are what the 2026-10-09 CI runners had cached. Each server here loads its model from these snapshot directories, so the revision cannot drift.

Step 1: Start the container (on the host)#

mkdir -p $HOME/asr-bench
docker run -d --name asr-bench \
    --gpus device=$GPU --cpuset-cpus $CPUSET \
    -v /dev/shm:/dev/shm \
    -v $HOME/.cache/huggingface:/root/.cache/huggingface \
    -v $HOME/asr-bench:/work \
    -e HF_TOKEN \
    $IMAGE sleep infinity
docker exec -it asr-bench bash

Every command after this runs inside the container. Run the container block of Variables. Then nvidia-smi -L must print exactly one GPU.

If the docker exec shell closes, the variables are lost and a running client stops. Run docker exec -it asr-bench bash on the host, the container block and the RUN line of the cut off run. Then run that run again from Step 4.

Step 2: Build the venv#

These lines follow .github/scripts/prepare_omni_venv.sh, without the extras and the CosyVoice checkout that other models need. So freeze.txt will not match a CI venv exactly.

[ -d /work/sglang-omni ] || git clone https://github.com/sgl-project/sglang-omni.git /work/sglang-omni
cd /work/sglang-omni
git fetch origin
git checkout --detach $COMMIT
rm -rf .venv
uv venv --system-site-packages .venv -p /usr/bin/python3.12
echo 'import site; site.addsitedir("/opt/sglang/lib/python3.12/site-packages")' > .venv/lib/python3.12/site-packages/sglang-image.pth
. .venv/bin/activate
python .github/scripts/omni_missing_dependencies.py pyproject.toml | xargs -r -d '\n' -n 1 python -m pip install
python .github/scripts/omni_missing_dependencies.py --overrides pyproject.toml | xargs -r -d '\n' python -m pip install --no-deps
uv pip install --no-deps -e .
python .github/scripts/verify_omni_installed_pins.py .venv/bin/python .

The last line must print Verified <N> exact dependency pins. Run this block again whenever COMMIT changes. For a pull request, run git fetch origin pull/<N>/head first and set COMMIT to its SHA.

Step 3: Download data and models#

Download the SeedTTS dataset and the four models at their pinned revisions:

cd /work/sglang-omni && . .venv/bin/activate
python -m benchmarks.dataset.prepare --dataset seedtts --revision $SEEDTTS_REV
python -c "from huggingface_hub import snapshot_download; snapshot_download('Qwen/Qwen3-ASR-1.7B', revision='7278e1e70fe206f11671096ffdd38061171dd6e5')"
python -c "from huggingface_hub import snapshot_download; snapshot_download('openai/whisper-large-v3', revision='06f233fe06e710322aca913c1bc4249a0d71fce1')"
python -c "from huggingface_hub import snapshot_download; snapshot_download('FunAudioLLM/Fun-ASR-Nano-2512-hf', revision='7ef6cafdc445b5beff0b730cbeb51952c793ca6e')"
python -c "from huggingface_hub import snapshot_download; snapshot_download('OpenMOSS-Team/MOSS-Transcribe-Diarize', revision='704aa4a9c304e8520be88901e0d1960158ef5b15')"
ls $QWEN3_ASR/config.json $WHISPER/config.json $FUN_ASR/config.json $MOSS_TD/config.json

The ls line must print four paths and no error. If it fails, run echo $HF_HOME. If that prints a path, set HF_HUB to $HF_HOME/hub and rerun the container block with that value.

The three MOSS-Transcribe-Diarize datasets are private. Download them once with HF_TOKEN set. An access error means the token cannot read them:

python -c "import datasets; datasets.load_dataset('zhaochenyang20/movies800time', split='validation')"
python -c "import datasets; datasets.load_dataset('zhaochenyang20/AISHELL4', split='validation')"
python -c "import datasets; datasets.load_dataset('zhaochenyang20/googletime', split='validation')"

The client cannot pin these datasets. Record the loaded dataset fingerprints and the remote repository heads right after the download, while still online. The remote head alone does not identify the cached rows. Compare both values between machines or dates before comparing numbers:

mkdir -p /work/results
python - > /work/results/moss_dataset_revisions.txt <<'EOF'
from huggingface_hub import HfApi
from datasets import load_dataset
for r in ("zhaochenyang20/movies800time", "zhaochenyang20/AISHELL4", "zhaochenyang20/googletime"):
    dataset = load_dataset(r, split="validation")
    print(r, "remote_head=" + HfApi().dataset_info(r).sha, "loaded_fingerprint=" + dataset._fingerprint)
EOF
cat /work/results/moss_dataset_revisions.txt

Every run sets HF_HUB_OFFLINE=1 (Step 4), so the datasets stay at the cached revisions. Check that every dataset loads offline. The last line must print 1088 2020 800 20 25:

HF_HUB_OFFLINE=1 python -c "
from benchmarks.dataset.seedtts import load_seedtts_samples as s, SEEDTTS_DATASET_ID as d
from benchmarks.tasks.transcribe_diarize import load_movies800_samples as m
a = dict(split='validation', audio_column='audio', expected_column='transcription', expected_sample_count=None)
print(len(s(d, split='en', revision='$SEEDTTS_REV')), len(s(d, split='zh', revision='$SEEDTTS_REV')), len(m('zhaochenyang20/movies800time', **a)), len(m('zhaochenyang20/AISHELL4', **a)), len(m('zhaochenyang20/googletime', **a)))"

Step 4: Start a run#

A run measures COMMIT once, with a fresh server for each model. Pick the run name first. warm is the unscored first run (see Measuring accurately), then 1, 2 and 3:

export RUN=warm

Then:

cd /work/sglang-omni && . .venv/bin/activate
export HF_HUB_OFFLINE=1 SGLANG_OMNI_STRICT_PORT=1 SGLANG_OMNI_TORCH_COMPILE_DEFAULT=0 SGLANG_OMNI_STARTUP_TIMEOUT=1800
export TORCHINDUCTOR_CACHE_DIR=/work/torchinductor
find /work/results -name server.pid -exec bash -c 'kill -9 -- -$(cat "$1") 2>/dev/null' _ {} \;
timeout 600 bash -c 'until [ $(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits) -lt 1024 ]; do sleep 5; done' && echo GPU_FREE
export OUT=/work/results/$NAME/run$RUN
rm -rf $OUT
mkdir -p $OUT
git rev-parse HEAD > $OUT/commit.txt
python -m pip freeze --exclude-editable > $OUT/freeze.txt

CI sets SGLANG_OMNI_TORCH_COMPILE_DEFAULT=0. Without it, every model that does not set compile itself (Qwen3-ASR, Whisper, Fun-ASR) runs torch.compile. MOSS-Transcribe-Diarize turns compile on itself. TORCHINDUCTOR_CACHE_DIR keeps its compile cache on the /work mount, so it survives a new container. Delete that directory to measure a cold start. SGLANG_OMNI_STARTUP_TIMEOUT=1800 raises the server’s own startup limit from 600 s. On a host with few CPU cores, a cold Whisper start can spend more than 600 s capturing CUDA graphs.

The find line stops every server left over from an earlier run. The block must print GPU_FREE. If it does not, run docker restart asr-bench on the host, reattach as in Step 1, and run this step again. The block deletes earlier results of the same run, so a run is always redone whole.

Step 5: Run SeedTTS ASR#

Each block starts a server for one model, runs EN and ZH, then stops the server. Run the blocks one at a time.

In every block, the curl line must print PORT_FREE, the timeout 1800 line SERVER_READY and the last line GPU_FREE. The timeout 1800 line stops waiting as soon as the server process exits, and then prints the end of server.log instead of SERVER_READY. If GPU_FREE does not print, some server processes survived the kill. Restart the container as in Step 4 and redo the run. The setsid line starts the server in its own process group, so kill stops all of it. Each client run prints a line ending with evaluated=1088/1088 (EN) or evaluated=2020/2020 (ZH). The client exits with code 0 even when it skipped samples, so check coverage after the run.

Qwen3-ASR-1.7B#

mkdir -p $OUT/qwen3
curl -sf localhost:8000/health > /dev/null || echo PORT_FREE
setsid bash -c 'echo $$ > "$1"; shift; exec "$@"' _ $OUT/qwen3/server.pid \
    .venv/bin/sgl-omni serve --model-path $QWEN3_ASR --model-name Qwen/Qwen3-ASR-1.7B --port 8000 \
    > $OUT/qwen3/server.log 2>&1 &
timeout 1800 bash -c 'until curl -sf localhost:8000/health > /dev/null; do sleep 5; grep -q "^State:[[:space:]]*[A-Y]" /proc/$(cat "$1")/status 2>/dev/null || exit 1; done' _ $OUT/qwen3/server.pid && echo SERVER_READY || tail -n 20 $OUT/qwen3/server.log
python -m benchmarks.eval.benchmark_asr_seedtts --port 8000 --model-path Qwen/Qwen3-ASR-1.7B --lang en \
    --concurrencies 32 --repeats 1 --warmup \
    --dataset-revision $SEEDTTS_REV --output $OUT/qwen3/asr_en.json
python -m benchmarks.eval.benchmark_asr_seedtts --port 8000 --model-path Qwen/Qwen3-ASR-1.7B --lang zh \
    --concurrencies 32 --repeats 1 --warmup \
    --dataset-revision $SEEDTTS_REV --output $OUT/qwen3/asr_zh.json
kill -- -$(cat $OUT/qwen3/server.pid)
timeout 600 bash -c 'until [ $(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits) -lt 1024 ]; do sleep 5; done' && echo GPU_FREE

Whisper-large-v3#

mkdir -p $OUT/whisper
curl -sf localhost:8000/health > /dev/null || echo PORT_FREE
setsid bash -c 'echo $$ > "$1"; shift; exec "$@"' _ $OUT/whisper/server.pid \
    .venv/bin/sgl-omni serve --model-path $WHISPER --model-name openai/whisper-large-v3 --port 8000 \
    > $OUT/whisper/server.log 2>&1 &
timeout 1800 bash -c 'until curl -sf localhost:8000/health > /dev/null; do sleep 5; grep -q "^State:[[:space:]]*[A-Y]" /proc/$(cat "$1")/status 2>/dev/null || exit 1; done' _ $OUT/whisper/server.pid && echo SERVER_READY || tail -n 20 $OUT/whisper/server.log
python -m benchmarks.eval.benchmark_asr_seedtts --port 8000 --model-path openai/whisper-large-v3 --lang en \
    --concurrencies 32 --repeats 1 --warmup \
    --dataset-revision $SEEDTTS_REV --output $OUT/whisper/asr_en.json
python -m benchmarks.eval.benchmark_asr_seedtts --port 8000 --model-path openai/whisper-large-v3 --lang zh \
    --concurrencies 32 --repeats 1 --warmup \
    --dataset-revision $SEEDTTS_REV --output $OUT/whisper/asr_zh.json
kill -- -$(cat $OUT/whisper/server.pid)
timeout 600 bash -c 'until [ $(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits) -lt 1024 ]; do sleep 5; done' && echo GPU_FREE

Fun-ASR-Nano#

mkdir -p $OUT/fun
curl -sf localhost:8000/health > /dev/null || echo PORT_FREE
setsid bash -c 'echo $$ > "$1"; shift; exec "$@"' _ $OUT/fun/server.pid \
    .venv/bin/sgl-omni serve --model-path $FUN_ASR --model-name FunAudioLLM/Fun-ASR-Nano-2512-hf --port 8000 \
    > $OUT/fun/server.log 2>&1 &
timeout 1800 bash -c 'until curl -sf localhost:8000/health > /dev/null; do sleep 5; grep -q "^State:[[:space:]]*[A-Y]" /proc/$(cat "$1")/status 2>/dev/null || exit 1; done' _ $OUT/fun/server.pid && echo SERVER_READY || tail -n 20 $OUT/fun/server.log
python -m benchmarks.eval.benchmark_asr_seedtts --port 8000 --model-path FunAudioLLM/Fun-ASR-Nano-2512-hf --lang en \
    --concurrencies 32 --repeats 1 --warmup \
    --dataset-revision $SEEDTTS_REV --output $OUT/fun/asr_en.json
python -m benchmarks.eval.benchmark_asr_seedtts --port 8000 --model-path FunAudioLLM/Fun-ASR-Nano-2512-hf --lang zh \
    --concurrencies 32 --repeats 1 --warmup \
    --dataset-revision $SEEDTTS_REV --output $OUT/fun/asr_zh.json
kill -- -$(cat $OUT/fun/server.pid)
timeout 600 bash -c 'until [ $(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits) -lt 1024 ]; do sleep 5; done' && echo GPU_FREE

Step 6: Run MOSS-Transcribe-Diarize#

The server arguments and request settings match the CI test (tests/test_model/test_asr_ci_multi_speaker.py). Like CI, one server serves all four datasets. The checks are the same as in Step 5. Each client line must also print CLIENT_OK. The client exits with code 0, and so prints CLIENT_OK, only when every request succeeded. Otherwise it prints an error or an Evaluation failed line.

For movies800times CI sends no max_new_tokens, so the server picks a budget scaled to the audio length, and the request timeout is 300 s. The client CLI always sends --max-new-tokens (default 65536) and has no flag to send none. So the two movies800times lines below call the client with that default set to None. The aishell4_long and googletime lines use the CLI defaults (65536 tokens, 1800 s), as CI does.

mkdir -p $OUT/moss_td
curl -sf localhost:8000/health > /dev/null || echo PORT_FREE
setsid bash -c 'echo $$ > "$1"; shift; exec "$@"' _ $OUT/moss_td/server.pid \
    .venv/bin/sgl-omni serve --model-path $MOSS_TD --model-name OpenMOSS-Team/MOSS-Transcribe-Diarize --port 8000 \
    --asr.engine.max_running_requests 16 --asr.engine.cuda_graph_max_bs 16 --mem-fraction-static 0.8 \
    > $OUT/moss_td/server.log 2>&1 &
timeout 1800 bash -c 'until curl -sf localhost:8000/health > /dev/null; do sleep 5; grep -q "^State:[[:space:]]*[A-Y]" /proc/$(cat "$1")/status 2>/dev/null || exit 1; done' _ $OUT/moss_td/server.pid && echo SERVER_READY || tail -n 20 $OUT/moss_td/server.log
python -c 'import sys, benchmarks.eval.benchmark_asr_transcribe_diarize as b; b.DEFAULT_MAX_NEW_TOKENS = None; sys.exit(b.main(sys.argv[1:]))' \
    --use-existing-server --port 8000 --dataset movies800times --max-concurrency 16 --request-timeout-s 300 \
    --output-dir $OUT/moss_td/movies800times && echo CLIENT_OK
python -c 'import sys, benchmarks.eval.benchmark_asr_transcribe_diarize as b; b.DEFAULT_MAX_NEW_TOKENS = None; sys.exit(b.main(sys.argv[1:]))' \
    --use-existing-server --port 8000 --dataset movies800times --stream --max-concurrency 16 --request-timeout-s 300 \
    --output-dir $OUT/moss_td/movies800times_stream && echo CLIENT_OK
python -m benchmarks.eval.benchmark_asr_transcribe_diarize --use-existing-server --port 8000 \
    --dataset aishell4_long --max-concurrency 16 --output-dir $OUT/moss_td/aishell4_long && echo CLIENT_OK
python -m benchmarks.eval.benchmark_asr_transcribe_diarize --use-existing-server --port 8000 \
    --dataset googletime --max-concurrency 16 --output-dir $OUT/moss_td/googletime && echo CLIENT_OK
kill -- -$(cat $OUT/moss_td/server.pid)
timeout 600 bash -c 'until [ $(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits) -lt 1024 ]; do sleep 5; done' && echo GPU_FREE

The config.max_new_tokens stored in each transcribe_diarize_speed_results.json shows what was sent. It must be null for both movies800times directories.

Then go back to Step 4 with the next RUN (warm, 1, 2, 3). To measure only one model, run Step 4 and only that model’s block, the same way in every run.

Step 7: Summarize#

Save this script once. For one result directory it reads the three scored runs (run1, run2 and run3, never runwarm), checks coverage, and prints the mean, min and max of each metric:

cat > /work/results/summarize.py <<'EOF'
import glob, json, os, statistics, sys
SEED = ("corpus_wer", "throughput_samples_per_s", "latency_mean_s", "latency_p95_s", "rtf_mean")
MOSS = ("cer", "cp_cer", "cer_valid_samples", "cp_cer_valid_samples", "speaker_timestamp_der")
SPEED = ("throughput_qps", "latency_mean_s", "latency_p95_s", "rtf_mean")
EXPECTED = {"asr_en.json": 1088, "asr_zh.json": 2020, "movies800times": 800,
            "movies800times_stream": 800, "aishell4_long": 20, "googletime": 25}
runs = [f"{sys.argv[1]}/run{i}" for i in (1, 2, 3)]
expected_keys = {f"{model}/{name}" for model in ("qwen3", "whisper", "fun") for name in ("asr_en.json", "asr_zh.json")}
expected_keys |= {f"moss_td/{name}" for name in EXPECTED if not name.endswith(".json")}
rows = {}
for run in runs:
    for path in glob.glob(f"{run}/*/asr_*.json"):
        r = json.load(open(path))["results"][0]
        ok = r["evaluated"] == r["total"] == EXPECTED[os.path.basename(path)] and r["skipped"] == 0
        rows.setdefault(os.path.relpath(path, run), []).append((ok, {k: r[k]["mean"] for k in SEED}))
    for path in glob.glob(f"{run}/moss_td/*/transcribe_diarize_results.json"):
        p = json.load(open(path))
        s = json.load(open(path.replace("_results.json", "_speed_results.json")))["speed"]
        name = os.path.basename(os.path.dirname(path))
        ok = p["summary"]["evaluated"] == p["summary"]["total_samples"] == EXPECTED[name] and s.get("failed_requests") == 0
        d = p["diarization_metrics"]
        rows.setdefault("moss_td/" + name, []).append((ok, {k: d.get(k) for k in MOSS} | {k: s.get(k) for k in SPEED}))
print(f"scored runs: {[os.path.basename(r) for r in runs]}")
print(f"missing workloads: {sorted(expected_keys - rows.keys())}")
print(f"coverage complete: {rows.keys() == expected_keys and all(len(rs) == 3 and all(ok for ok, _ in rs) for rs in rows.values())}")
for key, rs in sorted(rows.items()):
    print(f"{key}  runs={len(rs)} complete={len(rs) == len(runs) and all(ok for ok, _ in rs)}")
    for k in rs[0][1]:
        v = [m[k] for _, m in rs]
        if None in v:
            print(f"  {k:26} missing in a run: {v}")
        else:
            print(f"  {k:26} mean {statistics.mean(v):.6f}  min {min(v):.6f}  max {max(v):.6f}")
EOF
python /work/results/summarize.py /work/results/$NAME

Report the mean with the min and max of the three scored runs. coverage complete: True requires all ten workloads in all three runs with the expected sample counts. If you intentionally measured one model, its workload lines should each show runs=3 complete=True, while coverage of the full suite remains incomplete. An incomplete workload can still print partial means, so report its missing coverage instead of presenting those means as three full runs.

Where each metric lives#

SeedTTS, in $OUT/<model>/asr_en.json and $OUT/<model>/asr_zh.json, where <model> is qwen3, whisper or fun. With --repeats 1 each mean is the single measured value.

Metric	Field	Unit
WER	`results[0].corpus_wer.mean`	fraction (0.0120 is 1.20%)
Throughput	`results[0].throughput_samples_per_s.mean`	samples per second
Latency mean, p95	`results[0].latency_mean_s.mean`, `results[0].latency_p95_s.mean`	seconds
RTF mean	`results[0].rtf_mean.mean`	ratio
Coverage	`results[0].evaluated`, `results[0].total`, `results[0].skipped`	sample counts

MOSS-Transcribe-Diarize, in $OUT/moss_td/<dataset>/. The first five rows are in transcribe_diarize_results.json, the rest in transcribe_diarize_speed_results.json:

Metric	Field	Unit
Coverage	`summary.evaluated`, `summary.total_samples`	sample counts
CER, cpCER	`diarization_metrics.cer`, `diarization_metrics.cp_cer`	fraction
Valid sample counts	`diarization_metrics.cer_valid_samples`, `diarization_metrics.cp_cer_valid_samples`	sample counts
Valid samples	`per_sample[].id` with `per_sample[].cer_valid` and `per_sample[].cp_cer_valid`	IDs
Speaker timestamp DER	`diarization_metrics.speaker_timestamp_der`, `diarization_metrics.speaker_timestamp_der_skipped_parse_error`	fraction, count
Failed requests	`speed.failed_requests`	request count
Throughput	`speed.throughput_qps`	requests per second
Latency mean, p95	`speed.latency_mean_s`, `speed.latency_p95_s`	seconds
RTF mean	`speed.rtf_mean`	ratio

The printed client output uses diarization_metrics_percent, the same values times 100. The speed values are stored rounded: throughput and latency to 3 decimals, RTF to 4. When every request of a run fails, speed has no throughput, latency or RTF fields.

Coverage#

A run is complete only when every workload you ran has all its samples and no failures:

SeedTTS: evaluated equals total (1088 for EN, 2020 for ZH) and skipped is 0.
MOSS-Transcribe-Diarize: summary.evaluated equals summary.total_samples (800, 800, 20 and 25) and speed.failed_requests is 0.

summarize.py prints complete=True only when this holds for every scored run. If a check line in Step 5 or Step 6 prints nothing, or a key shows complete=False, read that server.log and redo the run from Step 4.

Measuring accurately#

Fixed GPU and CPU set. Step 1 gives the container one GPU and the CPUs of its NUMA node. Keep the same GPU and CPUSET for every run you compare. No other job may use that GPU.
Fresh server per run. Every model block starts and stops its own server, so no state carries over from an earlier run.
Warmup. The SeedTTS client sends the full dataset once, unscored, before the timed pass (--warmup). This fills CUDA graphs, compile caches and allocator pools. The MOSS-Transcribe-Diarize client has no warmup pass (--warmup defaults to 0, and CI uses 0 too).
Compile and cold start. Step 4 turns compile off for the models that do not set it, as CI does. The first run after a new container, a new commit or an empty /work/torchinductor can be much slower (see Caveats). So the warm run is never scored.
Three runs. Score run1, run2 and run3, and report the mean with the min and max. One run is not enough to tell a change from noise.
Noise floor. To learn the spread of your machine, measure the same commit twice under two names and compare the two summaries.
Comparing two commits. Use the same container, GPU, CPUSET, models and datasets. Alternate the commits run by run, for example A, B, B, A, A, B, so slow drift hits both. Set COMMIT and NAME and rebuild the venv (Step 2) at each switch. Check that freeze.txt is the same for both, or the comparison includes a dependency change.
Not CI numbers. CI runs two H100s behind the Rust router (DP 2), pins CPUs through OMNI_CI_CPUSET, and warms SeedTTS with only 64 requests. Numbers from this page are comparable only with other runs of this page, never with the CI gates.

Reference results#

These are for orientation only.

One H100, concurrency 32 (docstring of benchmarks/eval/benchmark_asr_seedtts.py): Qwen3-ASR EN reaches 97.9 samples/s with WER 0.0122. Fun-ASR EN reaches about 127.5 samples/s with WER 0.0171. Fun-ASR ZH reaches 167.4 samples/s with WER 0.0135. These numbers come from an older commit, with unknown compile settings, and may not match current code.
Two H100s in CI (the second pytest attempt inside CI job 113922224972): MOSS-Transcribe-Diarize on movies800times gave CER 5.82%, cpCER 13.13%, 784 CER valid samples and 58.167 req/s.
Run to run variation in CI: the first and second pytest attempts inside CI job 113922224972 gave movies800times CER of 5.87% and 5.82%.

Run the CI tests#

The CI tests need two GPUs. Pick two GPUs other than $GPU (here 2 and 3), or run docker rm -f asr-bench first. On the host, start a second container with the same checkout:

docker run -d --name asr-ci \
    --gpus '"device=2,3"' \
    -v /dev/shm:/dev/shm \
    -v $HOME/.cache/huggingface:/root/.cache/huggingface \
    -v $HOME/asr-bench:/work \
    -e HF_TOKEN \
    $IMAGE sleep infinity
docker exec -it asr-ci bash

Inside it:

nvidia-smi -L
cd /work/sglang-omni && . .venv/bin/activate
export PYTHONPATH=$PWD
export OMNI_CI_HOME=$HOME/omni_ci
export SGLANG_OMNI_TORCH_COMPILE_DEFAULT=0
export TORCHINDUCTOR_CACHE_DIR=$OMNI_CI_HOME/.torchinductor
export SGLANG_OMNI_ROUTER_BIN=$(bash .github/scripts/prepare_rust_router.sh build)
ASR_CI_MODEL=qwen3 python -m pytest tests/test_model/test_asr_ci_seedtts.py -v -s -x
ASR_CI_MODEL=fun python -m pytest tests/test_model/test_asr_ci_seedtts.py -v -s -x
ASR_CI_MODEL=whisper python -m pytest tests/test_model/test_asr_ci_seedtts.py -v -s -x
python -m pytest tests/test_model/test_asr_ci_multi_speaker.py -v -s -x

nvidia-smi -L must print exactly two GPUs. prepare_rust_router.sh needs network access, curl and a C linker to install rustup and build the router under $HOME/rust-router. The CI tests load models by repo id at their latest revision, so they do not use the pins in Variables. Do not wrap local runs in run_flaky_pytest.sh. It reruns a failing test up to three times, so a green result can hide a failure. Remove the container with docker rm -f asr-ci when done.

Caveats#

Private datasets. movies800time, AISHELL4 and googletime are private and cannot be pinned by the client. Compare the loaded fingerprints and remote heads in moss_dataset_revisions.txt before comparing MOSS-Transcribe-Diarize numbers from different machines or dates.
Known flaky stage. On 2026-10-09, 5 sampled CI stage 1 jobs reached only 43.2 to 44.7 req/s on movies800times in their first pytest attempt. After the server restarted for the second attempt, they reached 57.0 to 59.1 req/s. The cause is not known. A compile or file cache on disk is the likely suspect. CI sets TORCHINDUCTOR_CACHE_DIR to $OMNI_CI_HOME/.torchinductor. Outside CI, SGLang-Omni uses ~/.cache/sglang-omni/torchinductor unless TORCHINDUCTOR_CACHE_DIR is set, as Step 4 does.
Token budget on movies800times. Sending a fixed 65536 token budget instead of none lets a clip that loops run far past the duration scaled budget. That changes throughput and latency a lot. Check config.max_new_tokens in transcribe_diarize_speed_results.json before comparing with CI.
Warmup sees the scored audio. --warmup sends the same audio as the timed pass. Any cache keyed on audio content, file path or sample ID would hit on every timed request and make throughput meaningless.
Stage 2 is one model per CI run. A CI run measures only the model it picked. Run all three blocks of Step 5 here to see all of them.

For a profile of where the time goes, see the model profiling skill in .claude/skills/model-profiling/.

ASR benchmark runbook

Contents