NemotronLabs VoiceChat#

NVIDIA-NemotronLabs-VoiceChat-11B is an 11B end-to-end full-duplex speech-to-speech model. It is frame-locked at 12.5 Hz: every 80 ms of caller audio advances a Fast Conformer perception encoder, a Nemotron-H thinker that emits one text token and one function token per frame, an EAR-TTS talker that turns each text token into 31 RVQ codes, and an RVQ-VAE codec that renders the codes to 22.05 kHz audio. The model decides when to speak; there is no external VAD.

SGLang-Omni supports offline recordings and live duplex conversations over /v1/realtime.

Prerequisites#

The checkpoint is a single 44 GB fp32 model.safetensors with a NeMo-style config.json. Download it with:

hf download nvidia/NVIDIA-NemotronLabs-VoiceChat-11B

The checkpoint ships neither an HF tokenizer nor the backbone config.json; both come from the sibling repo named in its config (model.stt.model.pretrained_llm, currently nvidia/NVIDIA-Nemotron-Nano-9B-v2), about 17 MB of files and no weights. In offline environments, download those files too:

hf download nvidia/NVIDIA-Nemotron-Nano-9B-v2 config.json tokenizer.json tokenizer_config.json special_tokens_map.json

If they are missing and the Hub is unreachable, start-up fails with an error naming the repo and this command.

The four stages share one GPU. Measured on one H200 (143.8 GB) with the offline example, the run peaks at about 129 GB: the talker engine reserves mem_fraction_static=0.35 (about 47 GB, most of it KV pool) and the thinker takes what remains (about 73 GB, 17.7 GB of it bf16 weights). Lower --talker.engine.mem_fraction_static if the GPU is smaller or shared; the perception encoder and codec need only a few GB each.

On CUDA the talker captures its decode step as a CUDA graph by default; the thinker, and the talker on other platforms, decode eagerly. On one H100 with the offline pipeline, the talker graph cuts end-to-end latency 1.66–1.76x on 8–44 s inputs and keeps the reply text identical. The reply audio stays within run-to-run variation but is not bit-identical to eager decode after about 7 s: past that context length the graph’s attention reduces in a different order, and the code sampler breaks near-ties differently. Set --talker.engine.disable_cuda_graph true to decode the talker eagerly.

--thinker.engine.disable_cuda_graph false captures the thinker as well, roughly halving its decode step and the time to first text. Greedy reply text can then change where two candidate tokens tie in bf16 (7 of 44 test clips), so it is off by default.

Running the offline example#

python examples/run_nemotron_voicechat.py \
  --model-path /path/to/NVIDIA-NemotronLabs-VoiceChat-11B \
  --audio /path/to/NVIDIA-NemotronLabs-VoiceChat-11B/turn_taking.wav \
  --out reply.wav

The example builds NemotronVoiceChatPipelineConfig, starts the four stages with MultiProcessPipelineRunner, sends one request and writes the reply as 16-bit PCM at 22.05 kHz. Pick the GPU with CUDA_VISIBLE_DEVICES. Start-up takes about a minute once the checkpoint is in the page cache; the 41 s turn_taking.wav sample renders in roughly real time.

Two input conventions matter:

The recording is resampled to 16 kHz and channel 0 is used (a two-party recording carries the agent on channel 1); it is zero-padded to a multiple of 1280 samples (80 ms).
Leave trailing silence after the caller’s last words. The model answers only while it hears silence, so a clip that ends immediately after the question yields a short or empty reply.

The reply text is the thinker’s spoken tokens only; frames where the model is listening carry a marker token and are dropped from the text.

For Intel GPUs, follow the Nemotron VoiceChat XPU recipe.

Streaming codec chunk sizes#

The offline pipeline enables the (1, 1, 1, 1, 2, 2, 4) chunk ramp by default to reduce decoder calls and audio transfers. To disable the ramp and decode one codec frame at a time, set the code2wav factory’s chunk_frames:

python examples/run_nemotron_voicechat.py \
  --model-path /path/to/NVIDIA-NemotronLabs-VoiceChat-11B \
  --audio /path/to/NVIDIA-NemotronLabs-VoiceChat-11B/turn_taking.wav \
  --out reply.wav \
  --code2wav.factory.chunk_frames '[1]'

The equivalent YAML setting, for commands that take --config, is:

config_cls: NemotronVoiceChatPipelineConfig
model_path: /path/to/NVIDIA-NemotronLabs-VoiceChat-11B
stages:
  code2wav:
    factory:
      chunk_frames: [1]

CLI settings override YAML settings. Preview the resolved schedule and its source:

sgl-omni config resolve --config voicechat.yaml \
  --code2wav.factory.chunk_frames '[1, 2, 4]'
sgl-omni config explain stages.code2wav.factory.chunk_frames \
  --config voicechat.yaml --code2wav.factory.chunk_frames '[1, 2, 4]'

Each entry is the number of new codec frames to collect for a decode; the last entry repeats for the rest of the request. The default ramp emits four small chunks before growing to two frames and then four (320 ms of audio per steady chunk). (1,) restores the previous single-frame cadence. The schedule restarts for each request, and end-of-stream flushes a partially filled chunk.

Larger chunks trade playback arrival frequency for fewer decoder calls. Measure both first-chunk latency and playback underruns with the intended client buffer when choosing a schedule. This setting does not control the native realtime duplex pipeline. Chunk sizes must be positive integers and leave enough history in the 16-frame decode window for the checkpoint’s causal decoder and held-back tail; empty or null schedules and non-positive or non-integer entries fail during configuration resolution. The decoder-dependent window limit is checked when the code2wav scheduler is created.

Realtime conversation#

VoiceChat also serves live full-duplex voice sessions on /v1/realtime with its session variant. Launching, settings and the protocol are in Full-Duplex Guide.

Offline request parameters#

Parameter	Effect
`temperature`, `top_p`, `top_k`	Ignored. The thinker always decodes greedily so that the tokens it streams to the talker are the tokens it committed to; a non-zero `temperature` logs a warning.
`max_new_tokens`	Ignored; the frame count is fixed by the input length (one token per 80 ms).

Audio randomness is governed by the checkpoint’s own settings, read from config.json (inference_noise_scale, inference_top_p_or_k): the talker samples its codes with Gumbel-max component selection and Gaussian noise inside the MoG head, so two runs on the same input produce different code sequences and slightly different audio even though the text is identical. There is no seed control yet.

Tool calling#

VoiceChat can call configured Python functions during a conversation and use their results in its spoken replies. Tool calling is optional; no tools are enabled by default.

To try the random-number example, run this command from the repository root:

PYTHONPATH=. sgl-omni serve --config examples/nemotron_voicechat/tool.yaml --enable-realtime --port 8000

Connect your realtime client and ask, for example, “Give me a random number between one and fifty.” The server executes the tool and returns its result to the model automatically; the client does not need to handle tool calls.

To use your own tools, copy examples/nemotron_voicechat/tool.yaml and update:

tools.<name>.function: the OpenAI function schema, whose name must match the registry key.
tools.<name>.implementation: the importable Python function path.

Each tool is declared once. Set tool_names to a list of registered names to offer a subset; otherwise all registered tools are offered.

Each function accepts an argument dictionary and returns a JSON-serializable value. See examples/nemotron_voicechat/tools.py for an example. Make your tool module importable in the server environment, then launch with --config pointing to your configuration file.

Keep the realtime audio stream running, including silence while waiting for a reply. Stopping audio input can prevent the model from finishing its response.

Known limitations#

The offline pipeline handles one request at a time (max_running_requests=1 for both engines), without a barge-in or interrupt API. Native duplex sessions have independent state but do not yet batch perception or codec work across conversations.
A conversation holds at most 6144 positions in the thinker and in the talker: the system prompt, then one per 80 ms frame, so about eight minutes; a duplex session that reaches it ends with context_exhausted. The checkpoint was trained on conversations of at most about two minutes, and context beyond that may not be retained reliably.
Single speaker (Aria, the checkpoint’s baked-in prompt latents).
Classifier-free guidance is off (guidance_scale=0) although the checkpoint config enables it at 0.2.
Talker output is not bit-identical to NeMo’s offline script even with deterministic sampling on both sides (about 82 % of quantizer cells and 63 of 150 frames agree on a 12 s fixture; transcripts and amplitude envelopes match). Known causes: the backbone and KV cache run in bfloat16 because SGLang’s attention backends and sgl_kernel’s Gemma RMSNorm have no fp32 path, while NeMo runs the whole talker in fp32; NeMo’s offline script enables classifier-free guidance by default; and NeMo’s offline recipe walks the system-prompt region before frame 0 (realtime convention here; see prompt_region_steps). The thinker is frame-exact with NeMo.
NeMo decodes the perception encoder’s flush row as a 151st frame; this pipeline computes it but does not decode it, so replies are one frame (80 ms) shorter.

Tests#

tests/unit_test/nemotron_voicechat/ runs on CPU without weights: request frame counts, streaming codec equivalence with whole-utterance decoding, and checkpoint-shim isolation.

NemotronLabs VoiceChat

Contents