Nemotron 3.5 ASR#

Nemotron 3.5 ASR Streaming 0.6B is a multilingual speech-recognition model with a FastConformer encoder and an RNN-T decoder. SGLang-Omni supports complete-file transcription through /v1/audio/transcriptions and native cache-aware PCM streaming through the shared WebSocket session API.

Prerequisites#

Follow Installation, then run the examples from the repository root. Use the repository’s pinned Transformers version; the Nemotron compatibility implementation is included.

Server Configuration#

The default pipeline runs one ASR stage on one GPU in float32:

sgl-omni serve \
  --model-path nvidia/nemotron-3.5-asr-streaming-0.6b \
  --port 8000

Tune the ASR stage with --asr.factory.* flags:

Option	Default	Description
`dtype`	`float32`	Model dtype
`num_lookahead_tokens`	`3`	Encoder right context; the checkpoint supports `0`, `3`, `6`, and `13`
`max_batch_size`	`8`	Maximum number of requests in a scheduler batch
`max_batch_wait_ms`	`2.0`	Maximum wait to form a batch, in milliseconds
`session_max_concurrency`	`max(4, max_batch_size)`	Concurrent hooks/ordinary requests per ASR replica
`max_open_sessions`	`64`	Open streams per ASR replica
`max_state_bytes`	`8 GiB`	Total reserved session state budget, excluding shared model weights
`max_pcm_bytes`	`2 MiB`	Buffered PCM per session; consumed audio is discarded except overlap
`max_history_tokens`	`16384`	Bounded token/duration history; excess fails the session
`max_text_bytes`	`2 MiB`	Combined raw/clean transcript limit

For example:

sgl-omni serve \
  --model-path nvidia/nemotron-3.5-asr-streaming-0.6b \
  --asr.factory.num_lookahead_tokens 3 \
  --asr.factory.max_batch_size 8 \
  --asr.factory.max_batch_wait_ms 2 \
  --port 8000

Within each scheduler batch, complete-file requests with the same max_new_tokens value share one model generate() call. Different token limits are processed in separate batches to preserve each request’s output limit.

Transcribe Audio#

Upload a complete audio file; the server converts it to mono 16 kHz audio:

curl http://localhost:8000/v1/audio/transcriptions \
  -F model=nvidia/nemotron-3.5-asr-streaming-0.6b \
  -F file=@tests/data/query_to_cars.wav \
  -F language=auto \
  -F response_format=verbose_json

Use response_format=text for the transcript alone, or json for a JSON response. verbose_json also includes duration and language information. Locale tags are removed from the transcript. With language=auto, the response reports a language when the model emits one unambiguous locale tag.

Request Parameters#

Parameter	Default	Description
`file`	required	Audio file uploaded as multipart form data
`model`	server default	Model identifier
`language`	`auto`	Checkpoint-defined locale or language code, matched case-insensitively; `auto` enables language detection
`response_format`	`json`	`json`, `verbose_json`, or `text`; streaming responses accept only `json` or `text`
`temperature`	`0`	Greedy RNN-T decoding only; non-zero values are rejected
`max_new_tokens`	model default	Optional positive output-token limit
`prompt`	unset	Text prompts are unsupported; non-empty values are rejected
`stream`	`false`	Stream the HTTP response after a complete file upload; see below

Supported language values come from the checkpoint’s prompt dictionary. Unsupported values fail before model inference. Nemotron supports transcription only; /v1/audio/translations returns HTTP 400. Segment timestamps, SRT/VTT output, and speaker diarization are not supported.

Native Streaming#

Enable realtime on the server with --enable-realtime, then connect to /v1/realtime?model=nvidia/nemotron-3.5-asr-streaming-0.6b. This uses the shared native path (default intent=conversation) for ASR text. It does not add support for intent=transcription, VAD, manual commit, or conversation generation.

The public configuration accepts 16 kHz mono PCM16 and text output. Language is auto and decoding uses the model default; the shared schema has no language or max_new_tokens fields. Instructions are unsupported. Ordinary file transcription parameters above are unchanged.

Run this example with a mono 16 kHz PCM16 WAV file:

import asyncio
import base64
import json
import wave

import websockets


async def transcribe(path: str) -> None:
    with wave.open(path, "rb") as audio:
        assert (audio.getnchannels(), audio.getsampwidth(), audio.getframerate()) == (1, 2, 16000)
        pcm = audio.readframes(audio.getnframes())
    url = "ws://localhost:8000/v1/realtime?model=nvidia/nemotron-3.5-asr-streaming-0.6b"
    async with websockets.connect(url) as websocket:
        assert json.loads(await websocket.recv())["type"] == "session.created"
        await websocket.send(json.dumps({
            "event_id": "configure", "type": "session.update",
            "session": {"output_modalities": ["text"]},
        }))
        assert json.loads(await websocket.recv())["type"] == "session.updated"

        async def receive() -> None:
            while True:
                event = json.loads(await websocket.recv())
                if event["type"] == "error":
                    raise RuntimeError(event)
                elif event["type"] == "response.output_text.delta":
                    print(event["delta"], end="", flush=True)
                elif event["type"] == "response.output_text.done":
                    print("\nFinal:", event["text"])
                elif event["type"] == "sglang.input_audio.drained":
                    return
                else:
                    pass

        reader = asyncio.create_task(receive())
        for sequence, offset in enumerate(range(0, len(pcm), 640)):
            await websocket.send(json.dumps({
                "event_id": f"audio-{sequence}", "type": "input_audio_buffer.append",
                "audio": base64.b64encode(pcm[offset:offset + 640]).decode(),
                "sglang": {"seq": sequence},
            }))
            await asyncio.sleep(0.02)
        await websocket.send(json.dumps({
            "event_id": "end", "type": "sglang.input_audio.end",
        }))
        await reader
        await websocket.send(json.dumps({
            "event_id": "close", "type": "session.close",
        }))
        while json.loads(await websocket.recv())["type"] != "session.closed":
            pass


asyncio.run(transcribe("tests/data/query_to_draw.wav"))

One session is one recognition stream. EOS flushes a residual window and emits exactly one final, including an empty final for an empty stream. Audio after EOS is rejected. Close/disconnect cancels and releases state; it does not flush. Clear only discards PCM still pending in the realtime runtime, preserving an active unit and model caches already submitted to the stage. Open a new session for another recognition stream.

The 20 ms native unit is an input operation, not a model window or a first text latency guarantee. Deployment callers can configure native_unit_ms on create_realtime_deployment. Short units complete without waiting for a future window; long appends drain all currently runnable windows.

Each ASR replica owns one engine and model thread. Different sessions share model batches while retaining separate attention, convolution and RNN-T caches. H concurrent hook threads plus one inbox bridge thread feed that engine; if H is smaller than max_batch_size, the public path cannot submit a full batch. Replicas do not share batches. Offline and streaming work alternate at batch boundaries when both are ready; a long offline forward cannot be preempted. max_batch_wait_ms is a collection deadline, not an end-to-end latency bound.

Model task, PCM, cache reservation, token history and text budgets are bounded. Usage reports actual persistent cache/history/PCM bytes; batch temporaries and shared weights are measured separately at process level. When an explicit internal-session token limit is reached, later PCM is validated and counted toward duration without further inference or buffering. Budget errors after stage acceptance terminate the stream; they are not a promise that retrying the same seq is safe. Public admission rejection follows the shared runtime’s sequence/retry contract.

Setting stream=true on /v1/audio/transcriptions streams the response to a complete uploaded file. It does not select native PCM input.

Nemotron 3.5 ASR

Contents