MiniCPM-o offline benchmark runbook#

This page measures MiniCPM-o 4.5 offline serving in SGLang-Omni on one GPU. It also shows how to run the MiniCPM-o CI stages on a GPU pair. Full duplex serving is covered in the Full-Duplex-Bench v1.5 runbook.

Many commands end with a check that prints a fixed word in capitals, such as READY or GPU_FREE. If the word does not print, the step did not finish and its numbers are not usable.

What is Measured#

Two workloads, both on one GPU at concurrency 16:

Workload	Input and output	Samples	Server parameters
Video-AMME Talker	Video plus a spoken question in, text plus speech out	The first 10 questions of `Video_AMME_ci`	As in CI stage 10
seed-tts voice clone	Text in, cloned speech out	The 1088 sample EN full set	As in CI stage 2

Every metric is a JSON field. Paths are under the run directory $OUT.

File	Fields
`videoamme/videoamme_results.json`	Accuracy `summary.accuracy` (0 to 1), failed requests `summary.failed`, prompt tokens `per_sample[].prompt_tokens` by `sample_id`, and the speed fields below under `speed.`
`seedtts/speed_results.json`	Failed requests `summary.failed_requests`, prompt tokens `per_request[].prompt_tokens` by `id`, output tokens `per_request[].completion_tokens`, and the speed fields below under `summary.`
Speed fields	Throughput `throughput_qps` (req/s), latency `latency_mean_s` and `latency_p95_s` (s), `rtf_mean`, audio duration mean `audio_duration_mean_s` (s), audio throughput `audio_throughput_s_per_s` (seconds of audio per second)
`videoamme/wer_summary.json`	The WER fields below, at top level
`seedtts/wer_results.json`	The WER fields below, under `summary.`
WER fields	WER `wer_corpus` (a fraction, 0.005 is 0.5 points), samples above 50% WER `n_above_50_pct_wer`, WER corpus without those samples `wer_below_50_corpus` (the CI metric), unscored samples `skipped`
`seedtts/similarity_results.json`	SIM `summary.speaker_similarity_mean` (cosine × 100), `summary.skipped`
`seedtts/utmos_results.json`	UTMOS `summary.utmos_mean` (1 to 5), `summary.skipped`

Higher is better for Accuracy, throughput, SIM and UTMOS. Lower is better for latency, RTF and WER. Audio throughput shows whether a req/s change comes with shorter or longer speech.

WER uses Qwen3-ASR-1.7B served by SGLang-Omni, as in CI. SIM and UTMOS run without a server.

Setup#

One container holds a GPU pair. Single GPU measurements use the first GPU, pinned to its CPU group. The CI stages use both GPUs. Never run the two at the same time. Downloads, caches and results live in $BENCH_DIR on the host.

The host needs python3, lscpu and nvidia-smi. The CI stages build the router on first use, which needs internet access to rustup and crates and takes several minutes.

Start the container#

On the host:

export BENCH_DIR=/data/omni-bench
export GPU_A=0 GPU_B=1
mkdir -p "$BENCH_DIR/hf" "$BENCH_DIR/omni-cache" "$BENCH_DIR/results" "$BENCH_DIR/sglang-omni"

Make the CPU groups. Each GPU gets 14 cores on its NUMA node with their SMT siblings, skipping cores 0 and 1. BENCH_CPUS is the group of GPU_A. CI_CPUS covers both GPUs. It must print CPUS_OK.

python3 - "$GPU_A" "$GPU_B" > "$BENCH_DIR/cpus.env" <<'PY'
import subprocess, sys
from collections import defaultdict
run = lambda cmd: subprocess.run(cmd, capture_output=True, text=True, check=True).stdout
siblings, core_node, free, groups = defaultdict(list), {}, defaultdict(list), {}
for line in run(["lscpu", "-p=CPU,CORE,NODE"]).splitlines():
    if not line.startswith("#"):
        cpu, core, node = line.split(",")
        siblings[int(core)].append(int(cpu))
        core_node[int(core)] = int(node or 0)
for core in sorted(core_node):
    if core not in (0, 1):
        free[core_node[core]].append(core)
for line in run(["nvidia-smi", "--query-gpu=index,pci.bus_id", "--format=csv,noheader"]).splitlines():
    index, bus = (x.strip() for x in line.split(","))
    node = max(int(open(f"/sys/bus/pci/devices/{bus[-12:].lower()}/numa_node").read()), 0)
    if len(free[node]) >= 14:
        group, free[node] = free[node][:14], free[node][14:]
        groups[index] = sorted(c for core in group for c in siblings[core])
a, b = groups[sys.argv[1]], groups[sys.argv[2]]
print("BENCH_CPUS=" + ",".join(map(str, a)))
print("CI_CPUS=" + ",".join(map(str, sorted(a + b))))
PY
cat "$BENCH_DIR/cpus.env" && . "$BENCH_DIR/cpus.env" && test -n "$BENCH_CPUS" && echo CPUS_OK

A KeyError means a GPU of the pair gets no group. Set GPU_A=2 GPU_B=3 (then 4 5, then 6 7) and run it again. Compare with lscpu -p=CPU,CORE,NODE and nvidia-smi topo -m if no pair works.

Start the container with the CI image:

docker run -d --init --ipc host --cap-add=SYS_PTRACE \
    --gpus "\"device=$GPU_A,$GPU_B\"" --cpuset-cpus "$CI_CPUS" --env-file "$BENCH_DIR/cpus.env" \
    -v "$BENCH_DIR/hf:/root/.cache/huggingface" \
    -v "$BENCH_DIR/omni-cache:/root/.cache/sglang-omni" \
    -v "$BENCH_DIR/results:/results" \
    -v "$BENCH_DIR/sglang-omni:/sglang-omni" \
    --name omni-bench \
    hongccc/sglang-omni@sha256:ebe4239e29a764ee3a2806385c061c5fd438a26f01458e503d3822dcba5790df \
    sleep infinity

Open a shell with this command whenever you need one. After a host reboot, run docker start omni-bench first. All later commands run inside the container.

docker exec -it omni-bench bash

Check out and build the venv#

Set COMMIT to the commit to measure. The default below measures the fetched main branch. Build the venv with the CI script, which ends with Fresh environment ready. The script deletes and rebuilds OMNI_CI_HOME.

git clone https://github.com/sgl-project/sglang-omni.git /sglang-omni
export COMMIT=origin/main
git -C /sglang-omni checkout --detach "$COMMIT"
cd /sglang-omni && OMNI_CI_HOME=/root/omni_ci bash .github/scripts/prepare_omni_venv.sh omni

After every checkout, this must print VENV_OK. If it does not, run prepare_omni_venv.sh again. Source changes need no rebuild, because the package is installed editable.

cd /sglang-omni && source .github/scripts/omni_ci_deps_hash.sh && test "$(omni_ci_deps_hash)" = "$(cat /root/omni_ci/.deps-hash)" && echo VENV_OK

Download models and data#

cd /sglang-omni && . omni/bin/activate
python -c "from huggingface_hub import snapshot_download as d; d('openbmb/MiniCPM-o-4_5', revision='503e754207c94da6bb26850b4469f367c9ea3582')"
python -c "from huggingface_hub import snapshot_download as d; d('Qwen/Qwen3-ASR-1.7B', revision='7278e1e70fe206f11671096ffdd38061171dd6e5')"
python -c "from huggingface_hub import snapshot_download as d; d('zhaochenyang20/Video_AMME_ci', repo_type='dataset', revision='7a37507f1b53416b9cb6641378e5a098ea535ab8')"
python -c "from huggingface_hub import snapshot_download as d; d('zhaochenyang20/Video_MME_ci', repo_type='dataset', revision='833bd815c628ff277911bea3b1563545b21d5e27')"
python -m benchmarks.dataset.prepare --dataset seedtts
python -m benchmarks.dataset.prepare --dataset seedtts-50
python -m benchmarks.dataset.prepare --dataset mmmu-ci-50
python -m benchmarks.dataset.prepare --dataset mmsu-ci-2000
python -m benchmarks.metrics.speaker_similarity_assets --warm-cache
python -m benchmarks.metrics.utmos --warm-cache

Video_MME_ci holds the videos that Video-AMME points to. seedtts-50, mmmu-ci-50 and mmsu-ci-2000 are only for the CI stages. A download by commit does not set refs/main, so set those references for offline loading:

H=/root/.cache/huggingface/hub
mkdir -p $H/models--openbmb--MiniCPM-o-4_5/refs $H/models--Qwen--Qwen3-ASR-1.7B/refs $H/datasets--zhaochenyang20--Video_AMME_ci/refs $H/datasets--zhaochenyang20--Video_MME_ci/refs
echo -n 503e754207c94da6bb26850b4469f367c9ea3582 > $H/models--openbmb--MiniCPM-o-4_5/refs/main
echo -n 7278e1e70fe206f11671096ffdd38061171dd6e5 > $H/models--Qwen--Qwen3-ASR-1.7B/refs/main
echo -n 7a37507f1b53416b9cb6641378e5a098ea535ab8 > $H/datasets--zhaochenyang20--Video_AMME_ci/refs/main
echo -n 833bd815c628ff277911bea3b1563545b21d5e27 > $H/datasets--zhaochenyang20--Video_MME_ci/refs/main

Verify that every pinned snapshot loads offline at its revision. It must print REVISIONS_OK. seed-tts loads at 27f4c1adee83b5b29b7c4b375f6b976324bda308, pinned as SEEDTTS_DATASET_REVISION in benchmarks/dataset/prepare.py.

cd /sglang-omni && . omni/bin/activate
HF_HUB_OFFLINE=1 python - <<'PY'
from huggingface_hub import constants, snapshot_download
from benchmarks.dataset.seedtts import load_seedtts_samples
assert constants.HF_HUB_CACHE == "/root/.cache/huggingface/hub", constants.HF_HUB_CACHE
pins = [("openbmb/MiniCPM-o-4_5", "model", "503e754207c94da6bb26850b4469f367c9ea3582"),
        ("Qwen/Qwen3-ASR-1.7B", "model", "7278e1e70fe206f11671096ffdd38061171dd6e5"),
        ("zhaochenyang20/Video_AMME_ci", "dataset", "7a37507f1b53416b9cb6641378e5a098ea535ab8"),
        ("zhaochenyang20/Video_MME_ci", "dataset", "833bd815c628ff277911bea3b1563545b21d5e27")]
for repo, kind, sha in pins:
    path = snapshot_download(repo, repo_type=kind)
    assert path.endswith(sha), (repo, path)
    print(repo, sha)
n = len(load_seedtts_samples("zhaochenyang20/seed-tts-eval-arrow", None, split="en"))
assert n == 1088, n
print("REVISIONS_OK")
PY

Run the benchmark#

The shell block#

Run it at the start of every shell used for single GPU measurement.

taskset -pc "$BENCH_CPUS" $$
cd /sglang-omni && . omni/bin/activate
unset SGLANG_OMNI_TORCH_COMPILE_DEFAULT PYTORCH_ALLOC_CONF
export CUDA_VISIBLE_DEVICES=0 HF_HUB_OFFLINE=1 SGLANG_OMNI_STRICT_PORT=1 SGLANG_OMNI_STARTUP_TIMEOUT=1800

Torch compile stays on (the default) and PYTORCH_ALLOC_CONF stays unset. A cold start with compile needs more than the default 600 second startup timeout. Stay in /sglang-omni, because benchmarks is imported from the working directory.

One run#

A run measures the checkout once on both workloads, each with a fresh server, then scores it. Start with run0, then repeat this section with run1, run2 and run3, as explained in Measuring accurately. For run0, stop after the seed-tts server is stopped.

export OUT=/results/$(git rev-parse --short HEAD)/run0

It must print RUN_READY. OUT must be new and port 8000 must be free.

test ! -e "$OUT" && ! curl -s -o /dev/null localhost:8000/health && mkdir -p "$OUT/videoamme" "$OUT/seedtts" && git rev-parse HEAD > "$OUT/commit.txt" && git status --porcelain > "$OUT/dirty.txt" && echo RUN_READY

Video-AMME Talker#

Start the server. It must print READY. /tmp/omni_server.pid holds the server’s process group, so the stop command stops all of it.

setsid bash -c 'echo $$ > "$0"; exec "$@"' /tmp/omni_server.pid \
    sgl-omni serve \
    --model-path openbmb/MiniCPM-o-4_5 --model-name minicpmo \
    --thinker.engine.mem_fraction_static 0.55 --talker.engine.mem_fraction_static 0.15 \
    --thinker.factory.max_seq_len 32768 \
    --port 8000 > "$OUT/videoamme.log" 2>&1 &
timeout 1800 bash -c 'until curl -sf localhost:8000/health > /dev/null; do sleep 5; done' && echo READY || echo NOT_READY

Run the client. As in CI, WER is computed later from the ASR server.

python - --model minicpmo --port 8000 --repo-id zhaochenyang20/Video_AMME_ci --max-samples 10 \
    --video-fps 2 --video-max-frames 128 --video-max-pixels 401408 --max-tokens 256 --temperature 0 \
    --timeout-s 500 --max-concurrency 16 --enable-audio --output-dir "$OUT/videoamme" <<'PY'
import argparse, asyncio
from benchmarks.eval import benchmark_omni_videoamme as m
parser = argparse.ArgumentParser()
m.add_video_eval_args(parser, repo_help="")
cfg = m.video_eval_config_from_args(parser.parse_args())
asyncio.run(m.run_videoamme_eval(cfg, compute_wer=False))
PY

Stop the server. It must print GPU_FREE (memory on the GPU below 1 GiB).

kill -- -"$(cat /tmp/omni_server.pid)"
timeout 600 bash -c 'until [ "$(nvidia-smi -i 0 --query-gpu=memory.used --format=csv,noheader,nounits)" -lt 1024 ]; do sleep 5; done' && echo GPU_FREE || echo GPU_BUSY

seed-tts voice clone#

Start the server. It must print READY.

setsid bash -c 'echo $$ > "$0"; exec "$@"' /tmp/omni_server.pid \
    sgl-omni serve \
    --model-path openbmb/MiniCPM-o-4_5 --model-name minicpmo \
    --thinker.engine.mem_fraction_static 0.55 --talker.engine.mem_fraction_static 0.15 \
    --thinker.factory.max_seq_len 8192 \
    --port 8000 > "$OUT/seedtts.log" 2>&1 &
timeout 1800 bash -c 'until curl -sf localhost:8000/health > /dev/null; do sleep 5; done' && echo READY || echo NOT_READY

Run the client:

python -m benchmarks.eval.benchmark_omni_seedtts --generate-only \
    --meta zhaochenyang20/seed-tts-eval-arrow --model minicpmo --port 8000 \
    --voice-clone --reference-audio-field audio.ref_audio --max-concurrency 16 \
    --output-dir "$OUT/seedtts"

Stop the server. It must print GPU_FREE.

kill -- -"$(cat /tmp/omni_server.pid)"
timeout 600 bash -c 'until [ "$(nvidia-smi -i 0 --query-gpu=memory.used --format=csv,noheader,nounits)" -lt 1024 ]; do sleep 5; done' && echo GPU_FREE || echo GPU_BUSY

Score#

Start Qwen3-ASR. It must print READY.

setsid bash -c 'echo $$ > "$0"; exec "$@"' /tmp/omni_server.pid \
    sgl-omni serve \
    --model-path Qwen/Qwen3-ASR-1.7B --model-name Qwen/Qwen3-ASR-1.7B \
    --port 8000 > "$OUT/asr.log" 2>&1 &
timeout 1800 bash -c 'until curl -sf localhost:8000/health > /dev/null; do sleep 5; done' && echo READY || echo NOT_READY

seed-tts WER:

python -m benchmarks.eval.benchmark_omni_seedtts --transcribe-only \
    --meta zhaochenyang20/seed-tts-eval-arrow --model minicpmo --port 8000 \
    --asr-concurrency 4 --output-dir "$OUT/seedtts"

Video-AMME Talker WER, with the function CI stage 10 calls:

python - "$OUT/videoamme" <<'PY'
import json, sys
from benchmarks.tasks.asr import compute_text_audio_consistency_from_records
d = sys.argv[1]
records = json.load(open(f"{d}/videoamme_results.json"))["per_sample"]
wer = compute_text_audio_consistency_from_records(records, "en", "cuda:0", asr_router_port=8000, asr_concurrency=4)
json.dump(wer["summary"], open(f"{d}/wer_summary.json", "w"), indent=2)
PY

Stop the ASR server. It must print GPU_FREE.

kill -- -"$(cat /tmp/omni_server.pid)"
timeout 600 bash -c 'until [ "$(nvidia-smi -i 0 --query-gpu=memory.used --format=csv,noheader,nounits)" -lt 1024 ]; do sleep 5; done' && echo GPU_FREE || echo GPU_BUSY

SIM and UTMOS:

python -m benchmarks.eval.benchmark_omni_seedtts --similarity-only \
    --meta zhaochenyang20/seed-tts-eval-arrow --model minicpmo --output-dir "$OUT/seedtts"
python -m benchmarks.eval.benchmark_omni_seedtts --utmos-only \
    --meta zhaochenyang20/seed-tts-eval-arrow --model minicpmo --output-dir "$OUT/seedtts"

Check coverage and collect the metrics#

This prints every field from What is Measured, checks the speech files and writes $OUT/metrics.json. The last line is RUN_COMPLETE when every request and every scoring stage covered all 10 Video-AMME and 1088 seed-tts samples, with unique IDs, one WAV per sample, 0 failed requests, 0 skipped samples and no empty or NaN audio. A missing JSON field raises an error.

python - "$OUT" <<'PY'
import glob, json, sys
import numpy as np
import soundfile as sf
d = sys.argv[1]
J = lambda p: json.load(open(f"{d}/{p}"))
v, vw, s = J("videoamme/videoamme_results.json"), J("videoamme/wer_summary.json"), J("seedtts/speed_results.json")
w, sim, u = (J(f"seedtts/{n}_results.json")["summary"] for n in ("wer", "similarity", "utmos"))
m = {"videoamme.accuracy": v["summary"]["accuracy"], "videoamme.failed": v["summary"]["failed"],
     "seedtts.failed": s["summary"]["failed_requests"], "seedtts.sim": sim["speaker_similarity_mean"],
     "seedtts.sim_skipped": sim["skipped"], "seedtts.utmos": u["utmos_mean"], "seedtts.utmos_skipped": u["skipped"],
     "seedtts.completion_tokens_sum": sum(r["completion_tokens"] or 0 for r in s["per_request"])}
for k in ["throughput_qps", "latency_mean_s", "latency_p95_s", "rtf_mean", "audio_duration_mean_s", "audio_throughput_s_per_s"]:
    m[f"videoamme.{k}"], m[f"seedtts.{k}"] = v["speed"][k], s["summary"][k]
for k in ["wer_corpus", "wer_below_50_corpus", "n_above_50_pct_wer", "skipped"]:
    m[f"videoamme.wer.{k}"], m[f"seedtts.wer.{k}"] = vw[k], w[k]
audio = [sf.read(p) for p in sorted(glob.glob(f"{d}/*/audio/*.wav"))]
rates = sorted({int(r) for _, r in audio})
m["audio.files"] = len(audio)
m["audio.empty_or_nan"] = sum(int(a.size == 0 or bool(np.isnan(a).any())) for a, _ in audio)
for k, val in m.items():
    print(k, val)
print("audio.rates", rates)
clean = len(audio) > 0 and all(m[k] == 0 for k in m if k.endswith(("failed", "skipped", "empty_or_nan")))
coverage = (len(v["per_sample"]) == v["summary"]["total_samples"] == vw["evaluated"] == 10
            and len({r["sample_id"] for r in v["per_sample"]}) == 10
            and len(s["per_request"]) == s["summary"]["completed_requests"] == w["evaluated"] == sim["evaluated"] == u["evaluated"] == 1088
            and len({r["id"] for r in s["per_request"]}) == 1088
            and len(glob.glob(f"{d}/videoamme/audio/*.wav")) == 10
            and len(glob.glob(f"{d}/seedtts/audio/*.wav")) == 1088)
print("coverage.complete", coverage)
json.dump({"metrics": m, "rates": rates, "complete": clean and coverage}, open(f"{d}/metrics.json", "w"))
print("RUN_COMPLETE" if clean and coverage else "RUN_INCOMPLETE")
PY

A run is incomplete when a step printed NOT_READY or GPU_BUSY, or this script does not print RUN_COMPLETE. Stop any server, move the run aside and repeat it:

kill -- -"$(cat /tmp/omni_server.pid)"
mv "$OUT" "$OUT-incomplete"

Measuring accurately#

Fixed GPU and CPUs. Measure on one GPU, with the shell pinned to that GPU’s CPU group by the shell block. The server and the client share those 28 logical CPUs. Do not run the CI stages or anything else on the pair at the same time.
Fresh server per workload. Every workload in every run starts its own server and waits for GPU memory to drop below 1 GiB after stopping it. Never reuse a server across runs.
Compile and cold start. Torch compile is on. The compile cache is in $BENCH_DIR/omni-cache/torchinductor. It is kept across containers and commits. The first server start with an empty cache is much slower than later starts. Run run0 first to fill the cache and discard it. Run run0 again after any change to model code. Delete that directory to measure a true cold start.
Client warmup. Before the timed window, both clients send 16 warmup requests (the concurrency), all with the first sample. A failed warmup request stops the client. Keep this default so the timed batch does not absorb cold work.
Three runs. Run run1, run2 and run3 after run0. Report the mean and the spread. This prints mean, sample standard deviation and the three values for each metric:

D=/results/$(git rev-parse --short HEAD)
python - "$D/run1" "$D/run2" "$D/run3" <<'PY'
import json, statistics, sys
runs = [json.load(open(f"{d}/metrics.json")) for d in sys.argv[1:]]
assert all(r["complete"] for r in runs), "a run is incomplete"
for k in runs[0]["metrics"]:
    vals = [r["metrics"][k] for r in runs]
    print(k, round(statistics.mean(vals), 6), round(statistics.stdev(vals), 6), vals)
PY

Same session. Keep the GPU, CPU allocation and container configuration fixed. Record any session or allocation boundary when comparing runs, because it can change measurement noise.
Not comparable with CI gates. CI runs these workloads on two GPUs behind a router with two workers, with compile off, on seedtts-50 instead of the full set, and retries failed stages. The thresholds in tests/test_model/omni_ci_config.py do not apply to single GPU numbers.

To find where the time goes, see the model-profiling skill in this repository.

Running the CI stages locally#

This runs the 10 MiniCPM-o stage files of Omni model CI (.github/workflows/test-qwen3-omni-ci.yaml) as they are, on both GPUs with compile off. Stage 11 runs only for Qwen3-Omni. Open a new shell without the shell block, and set the output directory:

export CI_OUT=/results/$(git -C /sglang-omni rev-parse --short HEAD)/ci/run1

The prepare block sets the variables of the CI workflow and the omni-setup action that change results. Hugging Face stays offline and cache paths stay at their defaults. A new shell is limited to the container CPU set, so OMNI_CI_CPUSET reads it back in the format the CI pin script expects. It must print CI_READY.

cd /sglang-omni && . omni/bin/activate
export OMNI_CI_MODEL=minicpmo SGLANG_OMNI_TORCH_COMPILE_DEFAULT=0 CUDA_VISIBLE_DEVICES=0,1
export HF_HUB_OFFLINE=1
export OMNI_CI_CPUSET="$(taskset -pc $$ | awk '{print $NF}')" OMNI_CI_HOME=/root/omni_ci PYTHONPATH="$PWD"
export PYTORCH_ALLOC_CONF=expandable_segments:True NCCL_NVLS_ENABLE=0
export TORCHINDUCTOR_CACHE_DIR="$OMNI_CI_HOME/.torchinductor"
export FLASHINFER_WORKSPACE_BASE=/root FLASHINFER_JIT_DEBUG=0
export SGLANG_OMNI_ROUTER_BIN="$(bash .github/scripts/prepare_rust_router.sh build)"
test -x "$SGLANG_OMNI_ROUTER_BIN" && test ! -e "$CI_OUT" && mkdir -p "$CI_OUT" && git rev-parse HEAD > "$CI_OUT/commit.txt" && echo CI_READY

The test block writes a log and an exit code per stage, and clears the GPUs after each stage so a crashed stage does not hold them for the next one:

python -m pytest tests/test_model/test_qwen3_omni_thinker_length.py -v -s -x > "$CI_OUT/thinker_length.log" 2>&1; echo $? > "$CI_OUT/thinker_length.rc"; bash .github/scripts/delete_gpu_process.sh --kill-orphans > /dev/null
python -m pytest tests/test_model/test_qwen3_omni_tts_ci.py -v -s -x > "$CI_OUT/tts_ci.log" 2>&1; echo $? > "$CI_OUT/tts_ci.rc"; bash .github/scripts/delete_gpu_process.sh --kill-orphans > /dev/null
python -m pytest tests/test_model/test_qwen3_omni_mmmu_ci.py -v -s -x > "$CI_OUT/mmmu_ci.log" 2>&1; echo $? > "$CI_OUT/mmmu_ci.rc"; bash .github/scripts/delete_gpu_process.sh --kill-orphans > /dev/null
python -m pytest tests/test_model/test_qwen3_omni_mmmu_talker_ci.py -v -s -x > "$CI_OUT/mmmu_talker_ci.log" 2>&1; echo $? > "$CI_OUT/mmmu_talker_ci.rc"; bash .github/scripts/delete_gpu_process.sh --kill-orphans > /dev/null
python -m pytest tests/test_model/test_qwen3_omni_mmsu_ci.py -v -s -x > "$CI_OUT/mmsu_ci.log" 2>&1; echo $? > "$CI_OUT/mmsu_ci.rc"; bash .github/scripts/delete_gpu_process.sh --kill-orphans > /dev/null
python -m pytest tests/test_model/test_qwen3_omni_mmsu_talker_ci.py -v -s -x > "$CI_OUT/mmsu_talker_ci.log" 2>&1; echo $? > "$CI_OUT/mmsu_talker_ci.rc"; bash .github/scripts/delete_gpu_process.sh --kill-orphans > /dev/null
python -m pytest tests/test_model/test_qwen3_omni_videomme_ci.py -v -s -x > "$CI_OUT/videomme_ci.log" 2>&1; echo $? > "$CI_OUT/videomme_ci.rc"; bash .github/scripts/delete_gpu_process.sh --kill-orphans > /dev/null
python -m pytest tests/test_model/test_qwen3_omni_videomme_talker_ci.py -v -s -x > "$CI_OUT/videomme_talker_ci.log" 2>&1; echo $? > "$CI_OUT/videomme_talker_ci.rc"; bash .github/scripts/delete_gpu_process.sh --kill-orphans > /dev/null
python -m pytest tests/test_model/test_qwen3_omni_videoamme_ci.py -v -s -x > "$CI_OUT/videoamme_ci.log" 2>&1; echo $? > "$CI_OUT/videoamme_ci.rc"; bash .github/scripts/delete_gpu_process.sh --kill-orphans > /dev/null
python -m pytest tests/test_model/test_qwen3_omni_videoamme_talker_tp2_ci.py -v -s -x > "$CI_OUT/videoamme_talker_tp2_ci.log" 2>&1; echo $? > "$CI_OUT/videoamme_talker_tp2_ci.rc"; bash .github/scripts/delete_gpu_process.sh --kill-orphans > /dev/null
grep -H . "$CI_OUT"/*.rc

An exit code of 0 is a pass. All stages except thinker_length and tts_ci print an Accuracy: or Overall accuracy: line in their log.

Reference results#

There are no published numbers yet for one GPU at concurrency 16 with the commands above. These come from other setups:

Workload	Source	Setup	Results
Video-AMME Talker	#2580, main `5ed8a8d`	One H200, concurrency 8, 50 questions, seed 0 (CI sends no seed), text accuracy only with no WER	Accuracy 0.68, 0.33 req/s
Video-AMME Talker	#2552, CI stage 10, 7 runs	Two H100s, DP 2, compile off	0.643 to 0.668 req/s, Accuracy 0.7 in every run, WER 0.004032 in 6 runs and 0.008065 in one excluded run (one word apart)
seed-tts voice clone	#2532, main plus that PR before merge	One H200, CI server parameters, concurrency 16, 1088 samples, MPS on	14.22 req/s, WER corpus without samples above 50% 1.47%, SIM 49.78

The #2532 run used MPS. The commands on this page do not.

Notes#

The first start is slow. A start with an empty compile cache takes much longer than a warm start. One run starts three servers.
Video-AMME Talker is one batch. Ten requests do not fill 16 slots, so its throughput is mostly the completion time of that one batch. Small changes in one request move it.
seed-tts output length varies. The client samples at temperature 0.7 without a seed, as CI does. One long output can move throughput at concurrency 16 a lot. Check seedtts.completion_tokens_sum before reading a throughput change. Passing --seed 0 controls sampling variation, but does not guarantee identical outputs under concurrent scheduling. Record the seed and output lengths. CI sends no seed, so that setting differs from CI.
CI differs from these runs. CI wraps each stage in .github/scripts/run_flaky_pytest.sh, which runs a failed stage up to 3 times. The local test block does not, so a stage that fails only sometimes shows up here. CI gives the GPU pair 28 logical CPUs. This page gives it 56. MiniCPM-o CI runs only when selected (run-minicpmo label or /tag-and-rerun-ci minicpmo).
Stages may fail on main. A stage can fail on main. Run it on an earlier commit to tell an existing failure from a new one.
Some datasets are CI subsets. Video_AMME_ci, Video_MME_ci, seed-tts-eval-50-arrow, mmmu-ci-50 and mmsu-ci-2000 are CI subsets hosted on Hugging Face. Their results are not full benchmark estimates.
Some CI subsets are not pinned. seed-tts-eval-50-arrow, mmmu-ci-50 and mmsu-ci-2000 load at whatever revision is cached. When this page was written, Hugging Face main was 351f38c8d5ea36deda18dd9218645cda8f5c91b3, ff72fd69cc7e0719e04a0ddb12d160de89c6fefe and 5ae6ed4343a89566be6dd90023d529f1bd97802d.

MiniCPM-o offline benchmark runbook

Contents