SGLang-Omni#
SGLang-Omni is a high-performance serving framework for omni and multimodal models, built on top of SGLang. It is designed to orchestrate multi-stage pipelines with low latency and OpenAI-compatible APIs.
Modern omni models — such as speech-output LLMs and multimodal generation systems — decompose into heterogeneous stages with fundamentally different computational profiles: a compute-bound thinker, a memory-bound talker, a latency-sensitive codec. SGLang-Omni is built around a computation-centric design: each stage runs its own independent scheduler tuned to its bottleneck, communicates through a shared inbox/outbox abstraction, and transfers tensors via zero-copy shared memory. This prevents any single stage from degrading the others and allows new models to plug into the framework by declaring a pipeline topology rather than building an inference system from scratch.
About#
Core features:
Multi-Stage Pipeline: Flexible framework for orchestrating preprocessing, AR engine, codec, and vocoder stages across processes and GPUs.
Native SGLang Integration: Leverages SGLang’s RadixAttention, continuous batching, and CUDA Graph optimizations for the AR backbone.
OpenAI-Compatible Server: Drop-in
/v1/audio/speech,/v1/audio/transcriptions,/v1/audio/translations, and/v1/chat/completionsendpoints with real-time streaming support.Broad Model Support: TTS (Higgs, Fish S2-Pro, Voxtral, Qwen3-TTS, MOSS-TTS / Local, Ming-Omni-TTS, dots.tts, ZONOS2), Music (MiniMax Music 3), ASR (Qwen3-ASR, Fun-ASR, ARK-ASR, Whisper, MOSS-Transcribe-Diarize), Omni (Qwen3-Omni, Ming-Omni), and LLaDA2.0-Uni.
Supported Models#
Model |
Type |
Notes |
|---|---|---|
TTS |
Voice cloning, streaming, 100+ languages |
|
TTS |
Voice cloning, streaming |
|
TTS |
Named voices, streaming, 9 languages |
|
TTS |
Voice cloning, streaming, 10 languages, 0.6B / 1.7B |
|
TTS |
Delay-pattern MOSS-TTS; voice cloning, streaming, 31 languages |
|
TTS |
Local-transformer MOSS-TTS; 48 kHz stereo, streaming |
|
TTS |
Text-to-speech and zero-shot voice cloning |
|
TTS |
48 kHz continuous-latent TTS; also |
|
TTS |
MoE TTS, 9 DAC codebooks, voice cloning |
|
Music |
Text-to-music; lyrics + caption → 32 kHz stereo song |
|
ASR |
Multilingual transcription with 30 language hints |
|
ASR |
Multilingual Fun-ASR-Nano |
|
ASR |
Multilingual ARK-ASR |
|
ASR |
Multi-speaker transcription + diarization + timestamps |
|
ASR |
Experimental transcription and speech-to-English translation routes; see the audio translation support matrix |
|
Omni |
Text, image, audio, video → text + audio |
|
Omni |
Streaming TTS |
|
Multimodal |
Text + image understanding and generation |
Get Started
Cookbook
General Usage
Benchmarks