Ming-Omni-TTS

Ming-Omni-TTS#

Ming-Omni-TTS-16.8B-A3B is a mixture-of-experts audio generation model from inclusionAI. The current SGLang-Omni serving path supports text-to-speech and zero-shot voice cloning through the OpenAI-compatible /v1/audio/speech endpoint and produces 44.1 kHz audio.

Ming-Omni-TTS model architecture

The serving pipeline keeps the SGLang autoregressive backbone and the Ming acoustic feedback loop in one generation stage:

preprocessing -> reference_encode -> tts_engine -> audio_decode
                                      |       ^
                                      +-------+
                                       latent feedback

reference_encode is a no-op for text-only requests. For voice cloning it extracts the speaker embedding and prompt latents before the autoregressive loop starts. tts_engine runs the SGLang backbone, FlowLoss/CFM acoustic tail, stop head, and feedback projection. audio_decode converts the generated latent sequence into the final waveform with the Ming AudioVAE.

Prerequisites#

Install sglang-omni by following Installation, then download the checkpoint:

hf download inclusionAI/Ming-omni-tts-16.8B-A3B

The provided configuration uses TP1 on GPU 0.

Server Configuration#

sgl-omni serve \
  --model-path inclusionAI/Ming-omni-tts-16.8B-A3B \
  --config examples/configs/ming_omni_tts.yaml \
  --port 8000

The provided configuration enables the AR and acoustic-tail CUDA graphs and a fixed-width CUDA graph for streaming AudioVAE transitions. Non-streaming full-sequence AudioVAE decode remains compact eager, and requests are non-streaming unless stream is set.

For non-streaming requests, audio_decode sends the complete generated latent sequence through one full-sequence AudioVAE decode. Streaming requests use the separate incremental AudioVAE path with request-local cache and overlap state. Older configs need three changes, all enforced during config loading: remove decode_mode from the audio_decode stage’s factory group, because non-streaming chunked decode is no longer supported; set tts_engine.stream_to to [audio_decode] to declare the latent stream edge; and set audio_decode.can_accept_stream_before_payload to true so the consumer accepts latents that arrive while generation is still running. The provided YAML already carries all three.

Cross-request non-streaming AudioVAE batching is not implemented yet. The only supported non-streaming batch configuration is max_batch_size: 1 with max_batch_wait_ms: 0, as shown in the provided YAML; other values are rejected before the server starts.

stream_slots is the maximum number of streaming requests that the AudioVAE decoder can keep active at the same time. Each active stream uses one slot to preserve its decoding progress between audio chunks. If all slots are occupied, additional streams wait until a slot is released. The provided configuration uses stream_slots: 8 to match its concurrency-8 workload. Increasing it supports more simultaneous streams but uses more GPU memory and fixed-graph work; reducing it lowers those costs but also lowers streaming concurrency. It does not change non-streaming batching.

Synthesizing Speech#

Text Only#

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ming-omni-tts",
    "input": "SGLang-Omni is a great project!",
    "response_format": "wav"
  }' \
  --output output.wav

Voice Cloning#

Ming-Omni-TTS currently accepts one local reference clip and requires its transcript. Start the server with access to the directory containing the clip:

sgl-omni serve \
  --model-path inclusionAI/Ming-omni-tts-16.8B-A3B \
  --config examples/configs/ming_omni_tts.yaml \
  --allowed-local-media-path /path/to/references \
  --port 8000

Then submit the reference as a file:// URL:

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ming-omni-tts",
    "input": "Get the trust fund to the bank early.",
    "references": [{
      "audio_path": "file:///path/to/references/prompt.wav",
      "text": "We asked over twenty different people, and they all said it was his."
    }],
    "response_format": "wav"
  }' \
  --output cloned.wav

ref_audio and ref_text are accepted as shorthand for the single references item.

Streaming#

Streaming returns headerless mono signed 16-bit little-endian PCM (s16le) at 44.1 kHz with Content-Type: audio/pcm. The X-Sample-Rate, X-Channels, and X-Bit-Depth headers report the sample rate, channel count, and bit depth; HTTP EOF ends the stream.

Ming AudioVAE uses separate initial and steady cadence settings. Its first non-terminal call buffers stages.audio_decode.factory.initial_chunk_patches latent patches and emits no PCM; a later call supplies the right-hand context needed to emit that initial group. Subsequent calls consume steady_chunk_patches at a time. The provided configuration uses two initial patches and four steady patches, and the terminal step flushes any remainder. Pipe the response to ffplay to play it during generation:

curl -sS --fail --no-buffer -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ming-omni-tts",
    "input": "SGLang-Omni supports streaming speech generation.",
    "stream": true,
    "response_format": "pcm"
  }' \
  | ffplay -nodisp -autoexit -f s16le -ar 44100 -ac 1 -

To save the raw stream instead, use --output output.pcm. The file has no WAV header; convert it with ffmpeg -f s16le -ar 44100 -ac 1 -i output.pcm output.wav.

Generation Parameters#

Parameter

Default

Notes

input

(required)

Non-empty text to synthesize

references

null

At most one local reference clip with non-empty text

ref_audio / ref_text

null

Shorthand for the reference clip and transcript

max_new_tokens

200 when omitted

Per-request upper bound on acoustic generation steps. The provided configuration accepts values from 1 to 256; generation may stop earlier

temperature

0.0 when omitted

Non-negative SDE temperature used by the FlowLoss sampler

response_format

wav

Use pcm when stream is enabled; wav is used by the reference benchmark

stream

false

Streams raw PCM audio when enabled

voice

default

Only the default voice selector is accepted

speed

1.0

Other speed values are not supported

Advanced FlowLoss controls can be passed through stage_params.tts_engine:

{
  "stage_params": {
    "tts_engine": {
      "cfg": 2.0,
      "sigma": 0.25,
      "temperature": 0.0
    }
  }
}

cfg must be at least 1e-5 and cannot equal 1.0; sigma and temperature must be non-negative.

Benchmarking#

The benchmark uses Seed-TTS-Eval with concurrency 8. Run each row below for both en and zh, replacing {lang} in the output directory:

Response mode

Input mode

Scenario flags

Output directory

Non-streaming

Reference

(none)

results/ming_tts/nonstream/reference/{lang}

Non-streaming

Text-only

--no-ref-audio

results/ming_tts/nonstream/text_only/{lang}

Streaming

Reference

--stream

results/ming_tts/stream/reference/{lang}

Streaming

Text-only

--stream --no-ref-audio

results/ming_tts/stream/text_only/{lang}

With the Ming-TTS server running on port 8000, generate each scenario by substituting its language, flags, and output directory in this command:

python -m benchmarks.eval.benchmark_tts_seedtts \
  --generate-only --use-existing-server \
  --base-url http://127.0.0.1:8000 \
  --model ming-omni-tts \
  --meta zhaochenyang20/seed-tts-eval-arrow \
  --output-dir <output-directory> \
  --lang <lang> --ref-format references \
  --max-new-tokens 256 --max-concurrency 8 --warmup 8 \
  <scenario-flags>

After generation finishes, stop the TTS server and start the ASR server in another terminal:

sgl-omni serve \
  --model-path Qwen/Qwen3-ASR-1.7B \
  --port 8100

Then transcribe each output directory with the same language and scenario flags used for generation:

python -m benchmarks.eval.benchmark_tts_seedtts \
  --transcribe-only --use-existing-server \
  --host 127.0.0.1 --port 8100 \
  --model ming-omni-tts \
  --meta zhaochenyang20/seed-tts-eval-arrow \
  --output-dir <output-directory> \
  --lang <lang> --ref-format references \
  --max-new-tokens 256 --max-concurrency 8 --asr-concurrency 1 \
  <scenario-flags>

Benchmark Results#

Known Limitations#

  • Serving optimizations. Prefix/radix cache and torch.compile are not supported and remain disabled in the provided configuration.

  • Reference inputs. The current request adapter accepts one local reference audio file with a non-empty transcript; remote URLs, data URLs, precomputed prompt latents, and speaker embeddings are not supported.

  • Generation controls. Request-local seed, logits sampling fields (top_p, top_k, repetition_penalty), named voices, explicit language selection, instructions, and duration control are not supported. initial_codec_chunk_frames is rejected because AudioVAE cadence is a pipeline-level setting.

  • Checkpoint coverage. The current serving implementation supports only the 16.8B-A3B MoE checkpoint; the dense 0.5B checkpoint is not supported.