Whisper ASR

Whisper ASR#

Whisper ASR checkpoints can be started through the OpenAI-compatible /v1/audio/transcriptions endpoint. This path remains experimental in the current SGLang-Omni tree; validate checkpoint-specific accuracy and operational behavior before production deployment.

Prerequisites#

Install sglang-omni by following Installation or Installation - Ascend NPU, then download a Whisper checkpoint:

hf download openai/whisper-large-v3

Ascend NPU#

Follow NPU Installation and use the SGLang/CANN runtime specified in pyproject_npu.toml. Whisper runs on a single Ascend NPU in FP16. Select a device and start the service:

export ASCEND_RT_VISIBLE_DEVICES=0
export SGLANG_OMNI_PLATFORM_SPEC=sglang_omni.platforms.npu.NPUOmniPlatform
sgl-omni serve --model-path openai/whisper-large-v3 --port 8000

Pipeline startup defaults to mem_fraction_static=0.50 on NPU and 0.85 elsewhere. Set stages.asr.engine.mem_fraction_static in a pipeline YAML to override this budget; explicit values take precedence, including 0.85 on NPU. Direct Python calls to create_sglang_whisper_asr_executor bypass pipeline configuration and retain the original mem_fraction_static: float = 0.85 default; pass 0.50 explicitly when needed.

NPU decode graphs are disabled by default so Whisper can use the existing attention backend with ordinary decode. Prefill graph and compilation settings are preserved. Explicit graph overrides retain their precedence, but re-enabling NPU decode graphs is outside the validated scope. Ordinary decode can reduce throughput.

The encoder uses NPU SDPA; the encoder CUDA Graph and optional dense-prefill optimizations described below apply to CUDA only.

Server Configuration#

Whisper ASR runs a single ASR stage on one GPU/Ascend NPU.

sgl-omni serve \
  --model-path openai/whisper-large-v3 \
  --port 8000

For Intel XPU installation and launch, see the Whisper XPU recipe.

Encoder CUDA Graph#

The encoder CUDA Graph is enabled by default. With pre-LM encoding (the default), capture buckets follow pre_lm_max_batch_size (8), so batches 1/2/4/8 are captured. request_build_max_workers defaults to 8, matching Qwen3-ASR and Fun-ASR. When enable_pre_lm_encoder is false, buckets follow the atomic prefill budget (6144 // 1500 = 4). To use eager encoder execution, override the pipeline configuration:

config_cls: WhisperASRPipelineConfig
name: whisper
model_path: openai/whisper-large-v3-turbo

stages:
  asr:
    factory:
      enable_encoder_cuda_graph: false

The graph is captured after SGLang’s generation graphs. With pre-LM off, raise max_prefill_tokens before configuring larger LM-side buckets (12/16). Each request uses the smallest captured bucket that fits its batch. Requests larger than every captured bucket, with a different feature shape, or without a successful capture run eagerly. Startup and first-replay logs identify the captured and executed buckets.

Breakable Prefill CUDA Graph#

The decoder body uses SGLang’s breakable prefill CUDA Graph backend by default. Whisper encoder states and cross-attention K/V are prepared outside the captured decoder body. The default capture ladder stops at the largest aggregate decoder token count atomic admission can form. The cap considers every reachable request count because a batch of more, shorter prompts can contain more decoder tokens than a batch sized from the longest possible request. With the default 6,144 budget, 1,500 encoder placeholders, and at most 232 decoder tokens per request, the cap is 696. Startup logs report the capture cost and confirm the active buckets with prefill CUDA graphs attested.

The current benchmark results were collected with cuda_graph_max_bs_prefill=256. To reproduce that capture profile and limit startup time and GPU memory, set the prefill graph cap explicitly:

stages:
  asr:
    engine:
      cuda_graph_max_bs_prefill: 256

Whisper runs three independently configured graph planes, each bucketed on its own axis; cross-attention K/V is written once per request between the first two and only read afterwards:

plane

bucket axis

config

encoder forward

batch size

enable_encoder_cuda_graph, encoder graph buckets

decoder prefill body

aggregate prefill tokens

cuda_graph_backend_prefill, cuda_graph_bs_prefill

decoder decode

batch size

cuda_graph_bs, cuda_graph_max_bs

Disabling one plane leaves the other two active.

To disable only the prefill graph while keeping decode and encoder CUDA Graphs enabled, override the ASR stage:

config_cls: WhisperASRPipelineConfig
name: whisper
model_path: openai/whisper-large-v3

stages:
  asr:
    engine:
      cuda_graph_backend_prefill: disabled

Prefill Coalescing#

Whisper builds requests with eight worker threads by default, matching other pre-LM ASR pipelines. The coalescing gate targets two requests, while the default 6,144-token atomic budget lets the LM scheduler admit up to four 1,504-token Whisper requests together. A partial batch waits for at most 6 ms only while another request build is pending; a single request and a partial batch with no remaining build work are released immediately.

request_build_max_pending bounds submitted request-build futures, not the request backlog. When max_queued_requests is unset, requests beyond that pending-build limit remain queued for later construction. Setting max_queued_requests retains the configured finite-queue rejection behavior.

Use prefill_coalesce_requests and prefill_coalesce_wait_ms to tune the gate. Set prefill_coalesce_requests: 0 to disable only coalescing, or also set request_build_max_workers: 1 to restore the pre-optimization request-build path:

stages:
  asr:
    factory:
      request_build_max_workers: 1
      prefill_coalesce_requests: 0

Async Decode#

Whisper enables the shared one-step-lookahead decode path at every batch size, including a single request. It overlaps the current decode step’s GPU work with the previous step’s host-side result processing. The default running-request limit is 64. Disable async decode on the stage to compare against synchronous decode or diagnose a request lifecycle issue:

sgl-omni serve \
  --model-path openai/whisper-large-v3 \
  --asr.factory.enable_async_decode false \
  --port 8000

Speculative Decoding#

Whisper large-v3 can draft with wilsonzheng0327/Whisper-large-v3-distil-draft, distil-large-v3 fine-tuned on large-v3’s own transcripts in English, Chinese, Japanese and Cantonese. The draft shares the target’s encoder, so it reads the target’s encoder states through its own two-layer decoder. Each round it proposes speculative_num_steps tokens (7 by default) and the target checks them in one forward:

sgl-omni serve \
  --config examples/configs/whisper_large_v3_speculative.yaml \
  --port 8000

A request with a positive temperature samples under the draft: each draft token is accepted with probability min(1, p/q), the target’s probability over the draft’s, and the first rejected one is redrawn from the remainder, so the transcript follows the same distribution as without the draft. On the 1,000 English clips below at temperature 0.7, WER was 4.21% to 4.35% across three runs with the draft and 4.28% to 4.44% without it, the draft accepted 6.2 tokens per round, and throughput rose 13% at concurrency 1 and 12% at 8. Segment timestamps (srt, vtt) apply Whisper’s timestamp rules to every draft and verify token, and 299 of 300 srt files matched the server without the draft byte for byte. The draft was trained without timestamps and accepts about 4.7 tokens per round on them, so srt is 14% slower than without the draft at concurrency 1 and 4% faster at 8. It runs synchronously, so async decode is off. The first startup compiles the verify graphs for every decode batch size, about 14 minutes on an H200 against 9 without a draft; with a warm compile cache both start in about 6 minutes.

Send the form field return_spec_tokens_details=true to get sglext.spec_tokens_details (spec_accept_length, spec_accept_rate, the draft counts and the per-verify histogram) on json and verbose_json responses and on the streamed transcript.text.done event. Audio long enough to be transcribed in chunks returns none.

Send the form field return_spec_tokens_details=true to get sglext.spec_tokens_details (spec_accept_length, spec_accept_rate, the draft counts and the per-verify histogram) on json and verbose_json responses and on the streamed transcript.text.done event. Audio long enough to be transcribed in chunks returns none.

On one H200 with the default 64 running requests, 1,000 English clips of 1 to 16 s (pipecat-ai/stt-benchmark-data), one warmup and one measured pass per concurrency, against the same server without a draft. Fresh audio misses the encoder cache on every request; repeated audio is the warmup’s clips again, so every request reuses its cached encoder output:

Audio

Concurrency

No draft req/s

Draft req/s

Throughput gain

No draft p95 latency (s)

Draft p95 latency (s)

Fresh

1

8.76

11.14

27%

0.151

0.125

8

35.66

40.21

13%

0.322

0.299

32

64.25

69.41

8%

0.720

0.664

64

73.19

77.19

6%

1.331

1.214

Repeated

1

10.51

13.79

31%

0.132

0.106

8

43.90

53.34

22%

0.270

0.234

32

84.75

117.03

38%

0.556

0.424

64

114.63

139.59

22%

0.843

0.712

Corpus WER was 0.0365 in every run and no request failed. The draft accepted 7.0 tokens per round. 994 to 996 of the 1,000 transcripts matched the no-draft run, which matched itself on 995 to 1,000 across the two audio conditions. Fresh audio gains less because the encoder, which the draft does not shorten, is a larger share of each request. The draft’s other languages and long audio, repeated audio, measured on SGLang 0.5.22:

Set

Concurrency

No draft req/s

Draft req/s

Throughput gain

Error rate, no draft / draft

FLEURS ja test, 300 clips

1 / 8 / 32

6.35 / 31.68 / 62.34

9.13 / 38.52 / 83.87

44% / 22% / 35%

CER 0.0496 / 0.0495

FLEURS yue test, 300 clips

1 / 8 / 32

6.86 / 32.55 / 63.34

8.44 / 36.31 / 78.36

23% / 12% / 24%

CER 0.1074 / 0.1069

LongLibriHeavy 60 s, 200 clips

1 / 4

3.35 / 9.53

4.54 / 12.63

36% / 33%

WER 0.1021 / 0.1022

Chinese transcripts are shorter, so the gain is smaller: on 500 SeedTTS ZH clips with 16 running requests, 14% to 18% at concurrency 1 and none at 16. The draft was trained on transcription in these four languages only and has not been measured on translation or on other languages.

Transcribe Audio#

curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F model=openai/whisper-large-v3 \
  -F file=@tests/data/query_to_cars.wav \
  -F response_format=json

import requests

with open("tests/data/query_to_cars.wav", "rb") as f:
    resp = requests.post(
        "http://localhost:8000/v1/audio/transcriptions",
        data={
            "model": "openai/whisper-large-v3",
            "response_format": "json",
        },
        files={"file": ("query_to_cars.wav", f, "audio/wav")},
        timeout=300,
    )

resp.raise_for_status()
print(resp.json()["text"])

Translate Audio#

Whisper multilingual checkpoints can translate source speech to English via /v1/audio/translations. Use a multilingual, non-turbo checkpoint: *.en checkpoints have no translate task, and whisper-large-v3-turbo was distilled without it.

curl -X POST http://localhost:8000/v1/audio/translations \
  -F model=openai/whisper-large-v3 \
  -F file=@tests/data/query_to_cars.wav \
  -F language=fr \
  -F response_format=json

For this endpoint, language is an optional source-language hint and a SGLang-Omni extension. OpenAI’s official audio translations request schema does not include language; the translation target is English in both APIs. See the audio translation support matrix for response formats and other ASR models.

Request Parameters#

Parameter

Type

Default

Description

file

file

required

Audio file uploaded as multipart form data

model

string

server default

Model identifier

language

string

unset

Optional source-language hint; on translations this is a SGLang-Omni extension

prompt

string

unset

Optional text used as Whisper prev-context conditioning

response_format

string

json

json, verbose_json, raw text, srt, or vtt

temperature

float

0.0

Sampling temperature; defaults to greedy decoding. A negative value returns HTTP 422

The serving route selects the internal task from the endpoint (transcribe or translate); it is not a public form field. The route uses the ASR stage default unless the pipeline is configured another way. For smoke tests, keep the request minimal and use response_format=json.

For both transcription and translation, srt and vtt request Whisper’s model-derived segment timestamps. Non-streaming subtitle requests are supported; stream=true with either subtitle format returns HTTP 400.

Long Audio#

Whisper reads at most 30 seconds of audio in one request: the feature extractor works on a fixed 30-second mel window and drops everything past it. In SGLang-Omni, we transcribe longer uploads in chunks by splitting the audio at the quietest point near each 30-second boundary, running each chunk as its own engine request, and joining the transcripts back in order. By default, chunks decode independently and the caller’s prompt is sent to each chunk. Setting condition_on_previous_text to true makes chunks decode sequentially: the caller prompt conditions the first chunk, then each chunk uses the immediately preceding chunk’s decoded text as its prompt. If an enabled chunk contains a sustained repeated character cycle, it is retried once without previous context; the first result is discarded, and the retry result conditions the next chunk. The detector checks periods of 8–128 normalized letter, number, or combining-mark characters repeated at least three times. It does not depend on whitespace-delimited words, so it also covers languages without reliable word boundaries. A single Latin-script word repeated three times is excluded to preserve literal repetition. The three-copy threshold caught the observed pathological loops while producing zero retries on the submitted TED-LIUM evaluation; it remains a conservative recovery heuristic rather than a transcription guarantee.

This opt-in is not identical to OpenAI Whisper’s condition_on_previous_text=True: OpenAI Whisper carries decoded token history across internal windows and can reset that history during fallback, while SGLang-Omni currently passes the previous server chunk’s decoded text. The current TED-LIUM evaluation did not show an accuracy or throughput advantage from enabling this implementation, so it remains disabled by default. Changing the default or implementing token-level parity should be evaluated in a separate PR.

The behavior follows two kinds of values.

The scheduling policy is yours to tune, with dotted flags or the matching YAML keys:

Name

Default

Meaning

--audio_chunking.max_audio_clip_s

30

Longest clip we send to the engine in one request, and therefore the chunk length. Unlike Qwen3-ASR you can only lower it: 30s is the hard edge of the model’s mel window.

--audio_chunking.max_concurrent_chunks

8

Per-request concurrency cap used while chunks are independent. When previous-text conditioning is enabled, one request’s Whisper chunks decode in order while chunks from different requests can still batch together.

--audio_chunking.max_total_audio_s

3600

Upper limit on one upload; you get HTTP 400 above it. It bounds a single decoded waveform, not the total across uploads; that is the next knob’s job.

--audio_chunking.max_concurrent_long_audio_requests

max_running_requests // (2 × max_concurrent_chunks), at least 1; 4 with the stock defaults

How many long uploads the server admits at once. A long upload past the cap gets HTTP 503 instead of queueing; short uploads are never gated. The slot is taken before the upload is decoded and returned when its chunks are done.

The last knob is the aggregate guard: decoded waveforms held at once are at most max_concurrent_long_audio_requests × max_total_audio_s × 16000 × 4 bytes (decoding itself has transient peaks above that), and long audio holds at most max_concurrent_long_audio_requests × max_concurrent_chunks engine slots at once. The default keeps that product at half of --asr.engine.max_running_requests; an explicit value whose product reaches max_running_requests logs a warning at startup, since short requests would then queue behind long audio whenever it is saturated.

The model properties are ClassVars on WhisperASRPipelineConfig; no configuration path reaches them:

Name

Value

Meaning

allow_audio_chunking

true

Whisper transcribes an isolated chunk correctly, so chunking is on.

max_native_clip_s

30

The mel-window edge. Streaming cannot chunk, so stream=true takes audio up to 30s and gets HTTP 400 above that.

min_tail_s

1

Shortest final chunk worth transcribing; if the tail would be shorter, we move the previous cut earlier to absorb it, which keeps Whisper from hallucinating on very short clips.

condition_on_previous_text

false

Whether Whisper serializes chunks and conditions each chunk on the preceding decoded text. Disabled chunks remain independent and can use the per-request concurrency cap.

Benchmarking#

Use the shared SeedTTS benchmark for end-to-end concurrency, WER, latency, and throughput:

python -m benchmarks.eval.benchmark_asr_seedtts \
  --port 8000 --model-path openai/whisper-base \
  --max-samples 128 --concurrencies 1,2,4,8,16,32 \
  --repeats 5 --warmup --output whisper_concurrency.json

To reproduce the async-decode comparison below, resolve the pinned checkpoint and start each mode separately on the same GPU:

MODEL_REVISION=06f233fe06e710322aca913c1bc4249a0d71fce1
MODEL_PATH="$(
  hf download openai/whisper-large-v3 \
    --revision "$MODEL_REVISION" \
    --quiet
)"

CUDA_VISIBLE_DEVICES=0 sgl-omni serve \
  --model-path "$MODEL_PATH" \
  --mem-fraction-static 0.30 \
  --port 8000

# Replace the command above with this one for the synchronous baseline.
CUDA_VISIBLE_DEVICES=0 sgl-omni serve \
  --model-path "$MODEL_PATH" \
  --mem-fraction-static 0.30 \
  --asr.factory.enable_async_decode false \
  --port 8000

Run the same client command once per mode, changing only the output filename:

python -m benchmarks.eval.benchmark_asr_seedtts \
  --port 8000 \
  --model-path openai/whisper-large-v3 \
  --model-revision 06f233fe06e710322aca913c1bc4249a0d71fce1 \
  --dataset-revision 27f4c1adee83b5b29b7c4b375f6b976324bda308 \
  --max-samples 128 \
  --concurrencies 1,2,4,8,16,32,64 \
  --repeats 3 \
  --warmup \
  --dtype float16 \
  --cuda-graph \
  --torch-compile \
  --max-running-requests 64 \
  --mem-fraction-static 0.30 \
  --fingerprint \
  --output whisper_async.json

Benchmark Results#

The following W-PR1 results used the 20-sample SeedTTS EN subset on a single H200 with openai/whisper-base in FP16. Each mode ran one discarded warmup and three measured repeats per concurrency.

Concurrency

Eager req/s

CUDA Graph req/s

Throughput gain

Eager mean latency (s)

CUDA Graph mean latency (s)

Corpus WER

1

19.57

20.29

3.7%

0.051

0.049

0.0415

2

28.41

30.87

8.7%

0.070

0.065

0.0415

4

37.90

41.70

10.0%

0.104

0.094

0.0415

8

42.10

49.00

16.4%

0.185

0.158

0.0415

All 480 W-PR1 measured requests completed successfully. Corpus WER was unchanged across eager and CUDA Graph modes at every concurrency.

The following W-PR2 results were measured separately on the same H200 and 20-sample subset with five measured repeats plus one discarded warmup per concurrency. The baseline used one request-build worker with coalescing disabled; the attribution run used two workers with coalescing disabled; the optimized run used two workers, a batch target of two, and a pending-build-aware 6 ms deadline.

Concurrency

Baseline req/s

Two workers req/s

Coalesced req/s

Total gain

Gate gain

Baseline latency (s)

Coalesced latency (s)

Corpus WER

1

21.04

22.51

22.46

6.8%

-0.3%

0.047

0.044

0.0415

2

30.45

36.68

41.96

37.8%

14.4%

0.066

0.047

0.0415

4

40.24

55.62

62.83

56.2%

13.0%

0.097

0.063

0.0415

8

48.03

75.93

82.15

71.0%

8.2%

0.161

0.092

0.0415

All 1,200 measured requests completed successfully. Corpus WER remained 0.0415 in all three modes and at every concurrency. Logs from the optimized run showed Replaying Whisper encoder CUDA graph batch=2 request_batch=2 and prefill batches with two sequences and 3,008 new tokens.

The async-decode comparison used the 128-sample SeedTTS EN subset on the same H200 with openai/whisper-large-v3 in FP16, one discarded warmup, and three measured repeats per concurrency. The baseline disabled async decode; all other serving settings, including the 6,144-token prefill budget, were identical.

Concurrency

Sync req/s

Async req/s

Throughput change

Sync P95 (s)

Async P95 (s)

P95 change

Corpus WER

1

11.26

11.44

+1.6%

0.117

0.115

-1.7%

0.0084

2

18.45

19.53

+5.8%

0.140

0.133

-5.4%

0.0084

4

27.40

29.35

+7.1%

0.197

0.185

-6.2%

0.0084

8

38.77

40.88

+5.4%

0.285

0.268

-6.2%

0.0084

16

55.59

57.90

+4.2%

0.396

0.366

-7.6%

0.0084

32

66.47

69.91

+5.2%

0.691

0.639

-7.6%

0.0084

All 4,608 measured requests across both modes completed successfully, and all 2,304 paired transcripts matched exactly. Batch size 1 uses the synchronous fast path, so its 1.6% difference is run-to-run noise rather than async work. At concurrency 32, request-stage profiling measured 614.3 ms synchronous versus 585.5 ms asynchronous P95 from prefill completion to request completion. A separate async-only openai/whisper-base budget comparison showed why 6,144 is the default: relative to 4,096, scheduler queue P95 fell from 92.2 ms to 52.2 ms and throughput rose from 134.83 to 166.69 req/s.

Known Limitations#

  • Whisper ASR remains experimental. Validate checkpoint-specific accuracy and operational behavior before production deployment.

  • verbose_json returns a single segment spanning the audio duration; srt and vtt are not supported and return HTTP 400.

  • Encoder CUDA Graph is enabled by default and requires SGLang generation CUDA Graph. Validate the selected buckets before production use.

  • Audio encoding runs before LM admission by default (pre_lm_max_batch_size=8, request_build_max_workers=8). Set enable_pre_lm_encoder: false under stages.asr.factory to run the encoder inside prefill again.

  • The pre-LM encoder cache (pre_lm_cache_max_entries=1024) keeps its entries in page-locked (pinned) host memory so device-to-host and host-to-device copies run asynchronously on the DMA path instead of blocking a worker thread through a pageable bounce buffer. The whole budget (entries × 3.84 MB for large-v3, ≈3.9 GB) is locked at start-up and cannot be swapped; size container memory limits accordingly, or set pre_lm_cache_pin_host_memory: false to fall back to pageable memory.

  • Prefill budget defaults to 6,144 tokens (⌊6144/1500⌋=4) under atomic admission (chunked_prefill_size=0). This caps LM-side prefill batching independently of the pre-LM encoder batch limit.

  • Chunked prefill stays disabled because the Whisper encoder prefix must be admitted atomically. Requests that exceed the current prefill budget wait for the next batch instead of splitting the encoder prefix.

  • First startup can take several minutes.

  • The endpoint accepts one uploaded file per request.

  • Audio is resampled to 16 kHz before transcription.

  • prompt conditions decoding via Whisper prev-context tokens. Only the last 223 prompt tokens are kept (224 prev-context tokens including <|startofprev|>) — fewer when max_new_tokens is large, since prompt, task prefix, and output share Whisper’s 448-token decoder context. max_new_tokens is likewise clamped to that context. The prompt must not contain Whisper special tokens.