Qwen3-ASR High-Concurrency Benchmark and Bottleneck Profile

Qwen3-ASR High-Concurrency Benchmark and Bottleneck Profile#

This is the issue #1324 Q-PR2 deliverable: a reproducible current-main benchmark of Qwen3-ASR serving at concurrency 1–64, compared against a fixed serving baseline on identical pre-segmented audio, with enough internal telemetry to attribute where request time goes. The numbers below are the before-baseline for Q-PR3 (#1326, merged), Q-PR4, and Q-PR5.

Method#

  • Model: Qwen/Qwen3-ASR-1.7B, bf16, one 80 GB GPU per server (DP=1).

  • Data: full SeedTTS reference sets — EN 1088 clips, ZH 2020 clips — staged by benchmarks.dataset.seedtts; both systems receive the same files through the same client (benchmarks/eval/benchmark_asr_seedtts.py).

  • Sweep: concurrency 1, 8, 16, 32, 64, one discarded warmup pass plus three measured repeats per level; closed-loop client.

  • Current is SGLang-Omni at the commit under review, served with the default Qwen3-ASR pipeline (max_running_requests=32, request_build_max_workers=2, request_build_max_pending=16, async decode on). Baseline is a fixed OpenAI-compatible serving stack for the same checkpoint with its stock configuration, on an identical GPU of the same host. Neither side was tuned.

  • Telemetry: per-request profiler events (request build, admission queue, first forward, decode tail), host CPU and per-GPU utilization sampling, raw per-request JSONL, and an environment fingerprint are captured by the benchmark’s --profile-events --sample-util --fingerprint --save-raw-dir flags added in this PR.

Reproduce#

# current
sgl-omni serve --model-path Qwen/Qwen3-ASR-1.7B --port 8511

python -m benchmarks.eval.benchmark_asr_seedtts \
  --port 8511 --concurrencies 1,8,16,32,64 --repeats 3 --warmup \
  --profile-events --profile-event-dir /tmp/asr_profile \
  --sample-util --util-gpu-ids <gpu> --fingerprint \
  --save-raw-dir raw-current --output asr_en_current.json

# baseline: point the same client at the baseline server's port
python -m benchmarks.eval.benchmark_asr_seedtts \
  --port <baseline-port> --concurrencies 1,8,16,32,64 --repeats 3 --warmup \
  --sample-util --util-gpu-ids <gpu> --fingerprint \
  --output asr_en_baseline.json

Shared-host variance#

The measurement host is shared; co-tenant load shifts between runs and shows up directly in these host-bound workloads (see #907). Runs on a quiet host and a loaded host are both reported: the absolute peaks move, but the structural findings — the concurrency knee, its cause, and the stage decomposition — are identical in every run. Current-vs-baseline comparisons were taken back-to-back so both sides see similar external load.

Results — current (quiet host)#

SeedTTS EN, 1088 clips, mean over three repeats:

conc

req/s

RTFx

lat mean

lat p95

corpus WER

completed

1

11.6

54.8

0.088 s

0.129 s

0.0122

3264/3264

8

27.5

130.4

0.289 s

0.437 s

0.0122

3264/3264

16

53.0

251.2

0.304 s

0.458 s

0.0122

3264/3264

32

108.0

511.4

0.295 s

0.399 s

0.0122

3264/3264

64

103.0

486.8

0.610 s

0.741 s

0.0125

3165/3264

SeedTTS ZH, 2020 clips, mean over three repeats:

conc

req/s

RTFx

lat mean

lat p95

corpus CER

completed

1

14.3

66.9

0.070 s

0.088 s

0.0062

6060/6060

8

64.2

300.8

0.124 s

0.175 s

0.0064

6060/6060

16

96.5

451.9

0.165 s

0.231 s

0.0062

6060/6060

32

126.4

591.6

0.252 s

0.353 s

0.0062

6060/6060

64

129.2

604.3

0.489 s

0.613 s

0.0062

5993/6060

Reading:

  • Throughput saturates at concurrency 32 and stops scaling to 64 on both languages while mean latency roughly doubles.

  • At concurrency 64 both languages shed 1–4 % of requests with HTTP 500: a single worker admits at most request_build_max_pending=16 request builds and max_running_requests=32 running requests, so the 64-deep closed loop overflows the build backlog.

  • WER/CER stay in band at every level (the concurrency-64 uptick is the shed-request denominator, not transcription regressions).

  • Relative to the pre-#1326 reference (97.9 req/s at concurrency 32 on the same EN set and hardware class), async decode lifted saturation throughput by about 10 %.

Results — current vs baseline (back-to-back, same host window)#

SeedTTS EN, 1088 clips, mean over three repeats; the two sweeps ran back-to-back on identical neighboring GPUs so both saw the same co-tenant load. One baseline concurrency-1 repeat was hit by a co-tenant burst (7.3 req/s against 22.6/21.4); the mean below includes it.

conc

current req/s

baseline req/s

ratio

current lat mean/p95

baseline lat mean/p95

current completed

baseline completed

1

12.0

17.1

1.42Ă—

0.083 / 0.102 s

0.076 / 0.149 s

3264/3264

3264/3264

8

41.9

70.3

1.68Ă—

0.191 / 0.256 s

0.117 / 0.174 s

3264/3264

3264/3264

16

62.6

120.6

1.93Ă—

0.255 / 0.346 s

0.132 / 0.171 s

3264/3264

3264/3264

32

79.5

183.8

2.31Ă—

0.401 / 0.549 s

0.175 / 0.242 s

3264/3264

3264/3264

64

80.9

264.6

3.27Ă—

0.777 / 0.947 s

0.238 / 0.291 s

3208/3264

3264/3264

Corpus WER stayed 0.0122–0.0125 (current) and 0.0123–0.0124 (baseline) at every level. Reading:

  • The gap is a scaling gap, not a single-request model-forward gap: baseline keeps scaling past current’s concurrency-32 saturation and completes every request at 64 while current sheds 1–2 % there.

  • Excluding the contended repeat, baseline is ~1.8Ă— faster even at concurrency 1 (0.044 s vs 0.083 s mean), so part of the gap is fixed per-request serving overhead, and the rest grows with concurrency — consistent with the admission cap and host-dispatch costs quantified below.

  • This reproduces the external report’s direction with a wider magnitude on this hardware/window; the concurrency knee the roadmap describes is confirmed on current main.

SeedTTS ZH, 2020 clips, mean over three repeats, same back-to-back pairing (current concurrency-1 repeats were partially contended in this window):

conc

current req/s

baseline req/s

ratio

current lat mean/p95

baseline lat mean/p95

current completed

baseline completed

1

9.1

25.2

2.78Ă—

0.116 / 0.178 s

0.039 / 0.053 s

6060/6060

6060/6060

8

47.8

131.0

2.74Ă—

0.167 / 0.238 s

0.061 / 0.083 s

6060/6060

6060/6060

16

68.4

186.5

2.73Ă—

0.233 / 0.324 s

0.085 / 0.117 s

6060/6060

6060/6060

32

86.6

245.5

2.83Ă—

0.368 / 0.522 s

0.129 / 0.164 s

6060/6060

6060/6060

64

82.1

265.7

3.24Ă—

0.772 / 1.017 s

0.238 / 0.339 s

5961/6060

6060/6060

Corpus CER stayed 0.0061–0.0063 (current) and 0.0064–0.0066 (baseline) at every level.

Bottleneck decomposition (profiled passes, EN)#

Per-request stage averages from the request-event profiler; encoder work runs inside the first LM forward on this path, so “first forward” is encoder + prefill:

conc

build

build→queued

queued→scheduled

first forward

decode tail

total

1

5.6 ms

0.7 ms

1.1 ms

28.0 ms

55.3 ms

91 ms

8

6.0 ms

16.5 ms

2.0 ms

29.5 ms

131.8 ms

196 ms

16

6.1 ms

20.0 ms

1.9 ms

30.1 ms

185.6 ms

259 ms

32

6.5 ms

23.1 ms

5.9 ms

46.2 ms

357.7 ms

472 ms

64

6.3 ms

30.1 ms

412.8 ms

33.3 ms

392.6 ms

910 ms

GPU utilization stays between 33 % and 54 % (mean) at every level; the GPU is never the constraint.

Dominant costs, in order:

  1. Admission queueing at concurrency 64. The queued→scheduled wait jumps from ~6 ms to ~413 ms — the entire knee. Requests beyond max_running_requests=32 wait; requests beyond the build backlog are rejected. This is the Q-PR5 target.

  2. Decode-step host dispatch. The decode tail grows from 55 ms to ~360 ms per request as concurrency rises while GPU utilization falls — decode wall-clock is dominated by host-side per-step work shared across the running batch, consistent with the #907 causal profiling (GPU-idle-94 %, CPU-sensitivity-0.69). Q-PR5’s knob sweep and any further host-side batching improvements attack this.

  3. First-forward stalls. Encoder + prefill costs 28–46 ms per admitted request and executes on the scheduler thread and default stream, so every admission stalls the whole running decode batch for that long (and forces an async-decode drain). This is the Q-PR4 target: move the encoder ahead of LM admission onto its own thread/stream and batch it.

  4. Build-to-queued gap. 16–30 ms at concurrency ≥ 8 from the FIFO build-result drain and admission-lock path; small but visible, also in Q-PR5 scope.

Environment#

Fingerprints are embedded in each result JSON produced by --fingerprint: a client block (git SHA, dependency freeze hash, driver, GPU, cached model revision — describing the benchmark process) and a server block (what the target server reports about itself). In the runs above, client and server shared one host and checkout, so the client git state also identifies the server code. Raw per-request records and profiler event JSONL live beside the result files on the measurement host.

Summary for the runs above: NVIDIA H100 80GB HBM3, driver 580.126.20, torch 2.11.0+cu130, sglang 0.5.16, transformers 5.12.1, model snapshot 7278e1e70fe206f11671096ffdd38061171dd6e5, dependency freeze 49652ce3ea5a7720…. The current-side numbers were produced at this branch’s commit; note the fingerprint’s git.dirty flag also counts untracked run-output directories.