Nemotron Transcribe

Nemotron Transcribe 30B-A4B serves whole-file transcription through /v1/audio/transcriptions. A native PyTorch frontend, speech transformer, and Sortformer speaker-fusion encoder produce audio embeddings for SGLang’s NemotronH decoder. The serving path does not require NeMo.

This integration accepts consolidated NeMo SpeechLM checkpoints with the NeMoSpeechLMForConditionalGeneration architecture, a NemotronH llm_config, and perception and pe_encoder_config sections. The checkpoint directory must contain its tokenizer files and model weights. The loader creates a small configuration shim and reads the original weights without copying them.

Serve a checkpoint

Install SGLang-Omni following Installation, then download the checkpoint:

hf download nvidia/Nemotron-3.5-Transcribe-30B-A4B

The BF16 30B-A4B checkpoint was validated on one H100 80 GB. Leave memory for audio encoding and graph workspaces in addition to the weights and KV cache:

sgl-omni serve \
  --model-path nvidia/Nemotron-3.5-Transcribe-30B-A4B \
  --model-name nemotron-transcribe \
  --mem-fraction-static 0.85 \
  --port 8000

On a fresh install, the first launch also builds FlashInfer kernels, which can take longer than the default 600 second startup timeout. Set SGLANG_OMNI_STARTUP_TIMEOUT=1800 for that launch.

The pipeline runs one ASR stage. It accepts uploads up to one hour, resamples to mono 16 kHz, and keeps each upload as one decoder request. Measured transcripts are the short clips in Short-clip validation: 4.352 s and 5.855 s. A one-hour recording can repeat sentences until the 16384-token response budget. The HTTP status stays 200. json and verbose_json then include "finish_reason": "length". text sets X-Finish-Reason: length. A stream puts the same reason on transcript.text.done. The encoder processes the recording in checkpoint-defined windows with left and right context; the default checkpoint uses 40-second core windows.

Setting	Default	Purpose
--asr.engine.max_running_requests	64	Maximum concurrent decoder requests
--asr.factory.encoder_max_batch_size	8	Maximum waveforms per encoder invocation
--asr.factory.context_length	73728	Decoder context budget in tokens
--asr.factory.max_new_tokens	16384	Response token budget; length means the transcript stopped there
--asr.factory.chunked_prefill_size	4096	Decoder prefill chunk in tokens
--asr.factory.mm_embedding_cache_size_bytes	0	Audio embedding cache capacity
--asr.factory.compile_perception	false	Compile encoder transformer layers
--asr.factory.perception_cuda_graphs	false	Capture encoder graphs; requires compilation

Decoder CUDA graphs are enabled. Decoder compilation is unsupported in this integration; use encoder compilation to reduce launch overhead. Encoder batch size bounds activation memory separately from decoder concurrency. For longer audio or a smaller GPU, reduce the encoder batch size and choose a memory fraction that still fits the model weights.

Transcribe an upload

curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F model=nemotron-transcribe \
  -F file=@tests/data/query_to_cars.wav \
  -F response_format=json

The default instruction asks for a verbatim transcript in the spoken language. Supply prompt to replace that instruction:

curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F model=nemotron-transcribe \
  -F file=@recording.wav \
  -F 'prompt=Produce a verbatim transcript of the audio.' \
  -F response_format=text

Set stream=true to receive text deltas as Server-Sent Events. The server encodes the uploaded audio before generating text; this is not live PCM streaming. Word timestamps, speaker-attributed output, translation, speculative decoding, and /v1/realtime are outside this integration.

Compile the encoder

Enable compilation and encoder CUDA graphs independently of decoder graphs:

SGLANG_FLASHINFER_AUTOTUNE_EXTEND=1 \
SGLANG_FLASHINFER_AUTOTUNE_CACHE=0 \
sgl-omni serve \
  --model-path nvidia/Nemotron-3.5-Transcribe-30B-A4B \
  --model-name nemotron-transcribe \
  --mem-fraction-static 0.85 \
  --asr.engine.max_running_requests 16 \
  --asr.engine.cuda_graph_max_bs 16 \
  --asr.factory.encoder_max_batch_size 32 \
  --asr.factory.request_build_max_workers 16 \
  --asr.factory.request_build_max_pending 256 \
  --asr.factory.compile_perception true \
  --asr.factory.perception_compile_mode default \
  --asr.factory.perception_cuda_graphs true

The first requests compile kernels and capture encoder shapes. Exclude this warmup from throughput measurements. Single-request serving uses static encoder compilation; concurrent serving uses dynamic batch shapes. Explicit encoder graphs use private pools and cache at most four shapes per encoder. Additional shapes run without capturing more graphs. default is the pipeline default and skips Inductor’s kernel search. max-autotune-no-cudagraphs times candidates on the serving GPU and keeps graph ownership outside Inductor.

For c=1, set both decoder limits to 1. For long recordings, start with the default encoder batch cap rather than this profile’s cap of 32.

Validate the integration

The unit suite covers pipeline registration, checkpoint conversion, tensor mapping, placeholder counts, request validation, text streaming, mel features, rotary positions, encoder padding, and graph replay across changed inputs.

pytest -q tests/unit_test/nemotron_transcribe

CPU tests run without checkpoint downloads. CUDA tests skip when CUDA is unavailable.

RTFx is successful audio seconds divided by elapsed wall time, excluding warmup. Short-clip throughput and normalized transcript agreement do not establish WER on a representative evaluation corpus.

Benchmark on a corpus

Use the shared ASR benchmark against the server name from the launch command:

python -m benchmarks.dataset.prepare --dataset seedtts
python -m benchmarks.eval.benchmark_asr_seedtts \
  --port 8000 --model-path nemotron-transcribe \
  --concurrencies 1,16 --repeats 3 --warmup \
  --disable-resource-monitor

This reports corpus WER, RTFx, and request latency. The resource monitor is disabled in this example to keep GPU sampling out of timed runs. Corpus WER results for this checkpoint are not included in the short-clip validation.

Short-clip validation

On 2026-10-27, the default launch above ran on one H100 80 GB with BF16 and SGLang 0.5.22, keeping 64 decoder slots with encoder compilation and encoder graphs disabled.

Client concurrency	Median RTFx	RTFx range	Median p50	Median p90
1	23.40	23.31 to 23.41	0.220 s	0.229 s
16	102.17	100.22 to 104.91	0.716 s	0.999 s

Each concurrency had three rounds of 256 measured requests on two English clips lasting 4.352 s and 5.855 s, with warmup requests excluded. All 1,536 measured requests succeeded with matching transcripts. These measurements cover short clips and do not establish long-audio throughput or corpus WER.