NemotronLabs VoiceChat

NemotronLabs VoiceChat#

NVIDIA-NemotronLabs-VoiceChat-11B is an 11B end-to-end full-duplex speech-to-speech model. It is frame-locked at 12.5 Hz: every 80 ms of caller audio advances a Fast Conformer perception encoder, a Nemotron-H thinker that emits one text token and one function token per frame, an EAR-TTS talker that turns each text token into 31 RVQ codes, and an RVQ-VAE codec that renders the codes to 22.05 kHz audio. The model decides when to speak; there is no external VAD.

SGLang-Omni supports offline recordings and live duplex conversations over /v1/realtime.

Prerequisites#

The checkpoint is a single 44 GB fp32 model.safetensors with a NeMo-style config.json. Download it with:

hf download nvidia/NVIDIA-NemotronLabs-VoiceChat-11B

The checkpoint ships neither an HF tokenizer nor the backbone config.json; both come from the sibling repo named in its config (model.stt.model.pretrained_llm, currently nvidia/NVIDIA-Nemotron-Nano-9B-v2), about 17 MB of files and no weights. In offline environments, download those files too:

hf download nvidia/NVIDIA-Nemotron-Nano-9B-v2 config.json tokenizer.json tokenizer_config.json special_tokens_map.json

If they are missing and the Hub is unreachable, start-up fails with an error naming the repo and this command.

The four stages share one GPU. Measured on one H200 (143.8 GB) with the offline example, the run peaks at about 129 GB: the talker engine reserves mem_fraction_static=0.35 (about 47 GB, most of it KV pool) and the thinker takes what remains (about 73 GB, 17.7 GB of it bf16 weights). Lower --talker.engine.mem_fraction_static if the GPU is smaller or shared; the perception encoder and codec need only a few GB each.

Running the offline example#

python examples/run_nemotron_voicechat.py \
  --model-path /path/to/NVIDIA-NemotronLabs-VoiceChat-11B \
  --audio /path/to/NVIDIA-NemotronLabs-VoiceChat-11B/turn_taking.wav \
  --out reply.wav

The example builds NemotronVoiceChatPipelineConfig, starts the four stages with MultiProcessPipelineRunner, sends one request and writes the reply as 16-bit PCM at 22.05 kHz. Pick the GPU with CUDA_VISIBLE_DEVICES. Start-up takes about a minute once the checkpoint is in the page cache; the 41 s turn_taking.wav sample renders in roughly real time.

Two input conventions matter:

  • The recording is resampled to 16 kHz and channel 0 is used (a two-party recording carries the agent on channel 1); it is zero-padded to a multiple of 1280 samples (80 ms).

  • Leave trailing silence after the caller’s last words. The model answers only while it hears silence, so a clip that ends immediately after the question yields a short or empty reply.

The reply text is the thinker’s spoken tokens only; frames where the model is listening carry a marker token and are dropped from the text.

For Intel GPUs, follow the Nemotron VoiceChat XPU recipe.

Realtime conversation#

python examples/run_nemotron_voicechat_duplex.py \
  --model-path /path/to/NVIDIA-NemotronLabs-VoiceChat-11B \
  --serve --port 8097

Open http://localhost:8097, click Start, and allow microphone access. If the server runs on a remote GPU host, forward port 8097 to your local machine first. Headphones help avoid acoustic feedback. The example uses the shared realtime playground and supports one session on one GPU, with a four-minute browser limit. It uses the checkpoint’s default prompt and voice.

Offline request parameters#

Parameter

Effect

temperature, top_p, top_k

Ignored. The thinker always decodes greedily so that the tokens it streams to the talker are the tokens it committed to; a non-zero temperature logs a warning.

max_new_tokens

Ignored; the frame count is fixed by the input length (one token per 80 ms).

Audio randomness is governed by the checkpoint’s own settings, read from config.json (inference_noise_scale, inference_top_p_or_k): the talker samples its codes with Gumbel-max component selection and Gaussian noise inside the MoG head, so two runs on the same input produce different code sequences and slightly different audio even though the text is identical. There is no seed control yet.

Tool calling#

VoiceChat can call configured Python functions during a conversation and use their results in its spoken replies. Tool calling is optional; no tools are enabled by default.

To try the random-number example, run this command from the repository root:

PYTHONPATH=. sgl-omni serve --config examples/nemotron_voicechat/tool.yaml --enable-realtime --port 8000

Connect your realtime client and ask, for example, “Give me a random number between one and fifty.” The server executes the tool and returns its result to the model automatically; the client does not need to handle tool calls.

To use your own tools, copy examples/nemotron_voicechat/tool.yaml and update:

  • tools.<name>.function: the OpenAI function schema, whose name must match the registry key.

  • tools.<name>.implementation: the importable Python function path.

Each tool is declared once. Set tool_names to a list of registered names to offer a subset; otherwise all registered tools are offered.

Each function accepts an argument dictionary and returns a JSON-serializable value. See examples/nemotron_voicechat/tools.py for an example. Make your tool module importable in the server environment, then launch with --config pointing to your configuration file.

Keep the realtime audio stream running, including silence while waiting for a reply. Stopping audio input can prevent the model from finishing its response.

Known limitations#

  • The offline pipeline handles one request at a time (max_running_requests=1 for both engines), without a barge-in or interrupt API.

  • Single speaker (Aria, the checkpoint’s baked-in prompt latents).

  • Classifier-free guidance is off (guidance_scale=0) although the checkpoint config enables it at 0.2.

  • Talker output is not bit-identical to NeMo’s offline script even with deterministic sampling on both sides (about 82 % of quantizer cells and 63 of 150 frames agree on a 12 s fixture; transcripts and amplitude envelopes match). Known causes: the backbone and KV cache run in bfloat16 because SGLang’s attention backends and sgl_kernel’s Gemma RMSNorm have no fp32 path, while NeMo runs the whole talker in fp32; NeMo’s offline script enables classifier-free guidance by default; and NeMo’s offline recipe walks the system-prompt region before frame 0 (realtime convention here; see prompt_region_steps). The thinker is frame-exact with NeMo.

  • NeMo decodes the perception encoder’s flush row as a 151st frame; this pipeline computes it but does not decode it, so replies are one frame (80 ms) shorter.

Tests#

tests/unit_test/nemotron_voicechat/ runs on CPU without weights: request frame counts, streaming codec equivalence with whole-utterance decoding, and checkpoint-shim isolation.