Nemotron 3.5 ASR#

Nemotron 3.5 ASR Streaming 0.6B is a multilingual speech-recognition model with a FastConformer encoder and an RNN-T decoder. SGLang-Omni supports complete-file transcription through /v1/audio/transcriptions and native cache-aware PCM streaming through the shared WebSocket session API.

Prerequisites#

Follow Installation, then run the examples from the repository root. Use the repository’s pinned Transformers version; the Nemotron compatibility implementation is included.

For Intel GPUs, follow the Nemotron 3.5 ASR XPU recipe.

Server Configuration#

The default pipeline runs one ASR stage on one GPU in float32:

sgl-omni serve \
  --model-path nvidia/nemotron-3.5-asr-streaming-0.6b \
  --port 8000

Tune the ASR stage with --asr.factory.* flags:

Option	Default	Description
`dtype`	`float32`	Model dtype
`num_lookahead_tokens`	`3`	Encoder right context; the checkpoint supports `0`, `3`, `6`, and `13`
`max_batch_size`	`8`	Maximum number of requests in a scheduler batch
`max_batch_wait_ms`	`2.0`	Maximum wait to form a batch, in milliseconds
`enable_encoder_cuda_graph`	`true`	Capture steady-state streaming encoder batches at startup
`encoder_graph_max_batch_size`	`16`	Maximum captured batch size, capped by `max_batch_size` and pool capacity
`session_max_concurrency`	`max(4, max_batch_size)`	Concurrent hooks/ordinary requests per ASR replica
`max_open_sessions`	`64`	Open streams per ASR replica
`max_state_bytes`	`8 GiB`	Total reserved session state budget, excluding shared model weights
`max_pcm_bytes`	`2 MiB`	Buffered PCM per session; consumed audio is discarded except overlap
`max_history_tokens`	`16384`	Bounded token/duration history; excess fails the session
`max_text_bytes`	`2 MiB`	Combined raw/clean transcript limit

For example:

sgl-omni serve \
  --model-path nvidia/nemotron-3.5-asr-streaming-0.6b \
  --asr.factory.num_lookahead_tokens 3 \
  --asr.factory.max_batch_size 8 \
  --asr.factory.max_batch_wait_ms 2 \
  --port 8000

Within each scheduler batch, complete-file requests with the same max_new_tokens value share one model generate() call. Different token limits are processed in separate batches to preserve each request’s output limit.

Transcribe Audio#

Upload a complete audio file; the server converts it to mono 16 kHz audio:

curl http://localhost:8000/v1/audio/transcriptions \
  -F model=nvidia/nemotron-3.5-asr-streaming-0.6b \
  -F file=@tests/data/query_to_cars.wav \
  -F language=auto \
  -F response_format=verbose_json

Use response_format=text for the transcript alone, or json for a JSON response. verbose_json also includes duration and language information. Locale tags are removed from the transcript. With language=auto, the response reports a language when the model emits one unambiguous locale tag.

Request Parameters#

Parameter	Default	Description
`file`	required	Audio file uploaded as multipart form data
`model`	server default	Model identifier
`language`	`auto`	Checkpoint-defined locale or language code, matched case-insensitively; `auto` enables language detection
`response_format`	`json`	`json`, `verbose_json`, or `text`; streaming responses accept only `json` or `text`
`temperature`	`0`	Greedy RNN-T decoding only; non-zero values are rejected
`max_new_tokens`	model default	Optional positive output-token limit
`prompt`	unset	Text prompts are unsupported; non-empty values are rejected
`stream`	`false`	Stream the HTTP response after a complete file upload; see below

Supported language values come from the checkpoint’s prompt dictionary. Unsupported values fail before model inference. Nemotron supports transcription only; /v1/audio/translations returns HTTP 400. Segment timestamps, SRT/VTT output, and speaker diarization are not supported.

Native Streaming#

Enable realtime on the server with --enable-realtime, then connect to /v1/realtime?model=nvidia/nemotron-3.5-asr-streaming-0.6b. This uses the shared native path (default intent=conversation) for ASR text. It does not add support for intent=transcription, VAD, manual commit, or conversation generation.

The public configuration accepts 16 kHz mono PCM16 and text output. Language is auto and decoding uses the model default; the shared schema has no language or max_new_tokens fields. Instructions are unsupported. Ordinary file transcription parameters above are unchanged.

Run this example with a mono 16 kHz PCM16 WAV file:

import asyncio
import base64
import json
import wave

import websockets


async def transcribe(path: str) -> None:
    with wave.open(path, "rb") as audio:
        assert (audio.getnchannels(), audio.getsampwidth(), audio.getframerate()) == (1, 2, 16000)
        pcm = audio.readframes(audio.getnframes())
    url = "ws://localhost:8000/v1/realtime?model=nvidia/nemotron-3.5-asr-streaming-0.6b"
    async with websockets.connect(url) as websocket:
        assert json.loads(await websocket.recv())["type"] == "session.created"
        await websocket.send(json.dumps({
            "event_id": "configure", "type": "session.update",
            "session": {"output_modalities": ["text"]},
        }))
        assert json.loads(await websocket.recv())["type"] == "session.updated"

        async def receive() -> None:
            while True:
                event = json.loads(await websocket.recv())
                if event["type"] == "error":
                    raise RuntimeError(event)
                elif event["type"] == "response.output_text.delta":
                    print(event["delta"], end="", flush=True)
                elif event["type"] == "response.output_text.done":
                    print("\nFinal:", event["text"])
                elif event["type"] == "sglang.input_audio.drained":
                    return
                else:
                    pass

        reader = asyncio.create_task(receive())
        for sequence, offset in enumerate(range(0, len(pcm), 640)):
            await websocket.send(json.dumps({
                "event_id": f"audio-{sequence}", "type": "input_audio_buffer.append",
                "audio": base64.b64encode(pcm[offset:offset + 640]).decode(),
                "sglang": {"seq": sequence},
            }))
            await asyncio.sleep(0.02)
        await websocket.send(json.dumps({
            "event_id": "end", "type": "sglang.input_audio.end",
        }))
        await reader
        await websocket.send(json.dumps({
            "event_id": "close", "type": "session.close",
        }))
        while json.loads(await websocket.recv())["type"] != "session.closed":
            pass


asyncio.run(transcribe("tests/data/query_to_draw.wav"))

One session is one recognition stream. EOS flushes a residual window and emits exactly one final, including an empty final for an empty stream. Audio after EOS is rejected. Close/disconnect cancels and releases state; it does not flush. Clear only discards PCM still pending in the realtime runtime, preserving an active unit and model caches already submitted to the stage. Open a new session for another recognition stream.

The 20 ms native unit is an input operation, not a model window or a first text latency guarantee. Deployment callers can configure native_unit_ms on create_realtime_deployment. Short units complete without waiting for a future window; long appends drain all currently runnable windows.

Each ASR replica owns one engine and model thread. Different sessions share model batches while retaining separate attention, convolution and RNN-T caches. H concurrent hook threads plus one inbox bridge thread feed that engine; if H is smaller than max_batch_size, the public path cannot submit a full batch. Replicas do not share batches. Offline and streaming work alternate at batch boundaries when both are ready; a long offline forward cannot be preempted. max_batch_wait_ms is a collection deadline, not an end-to-end latency bound.

Model task, PCM, cache reservation, token history and text budgets are bounded. Usage reports actual persistent cache/history/PCM bytes; batch temporaries and shared weights are measured separately at process level. When an explicit internal-session token limit is reached, later PCM is validated and counted toward duration without further inference or buffering. Budget errors after stage acceptance terminate the stream; they are not a promise that retrying the same seq is safe. Public admission rejection follows the shared runtime’s sequence/retry contract.

Persistent encoder state pool#

Streaming preallocates a bounded pool for attention and convolution history. Each session leases one slot, and batches address state by slot ID, preserving stable device storage across audio windows.

Capacity is limited by max_open_sessions and max_state_bytes; a budget too small for one slot fails at startup. Completion, cancellation, failure and shutdown release slots, which are cleared before reuse. The pool is the sole streaming encoder state representation. Offline transcription is unchanged.

Streaming encoder CUDA graphs#

At startup, CUDA captures every exact subsequent-window batch size from min(encoder_graph_max_batch_size, max_batch_size, pool_capacity) down to 1, largest first, in one shared graph memory pool. The defaults capture B8 through B1. First windows and uncovered shapes use the pooled eager encoder. Capture failures fail startup; set enable_encoder_cuda_graph=false to run entirely eager.

Graphs update persistent attention, convolution and frame-count state by slot ID. History length therefore needs no additional buckets or Python cache packing. Graph workspaces and static inputs and outputs are outside max_state_bytes.

Setting stream=true on /v1/audio/transcriptions streams the response to a complete uploaded file. It does not select native PCM input.

Nemotron 3.5 ASR

Contents