Native Chat Stage

Native Chat Stage#

Understanding models that native SGLang serves through its OpenAI chat API, such as the Cosmos3 Reasoner, run in Omni as a native chat stage. The stage starts a native SGLang Engine and its chat service in the stage process and sends every Omni request through them, so the chat template, multimodal processing, sampling and KV cache are those of native SGLang. --text-only selects the Cosmos3 Reasoner.

sgl-omni serve --model-path /path/to/Cosmos3-Nano --text-only --port 8000
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages": [{"role": "user", "content": [{"type": "text", "text": "What is in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/scene.png"}}]}], "max_tokens": 128}'

Requests#

  • Messages carry text, image_url, video_url and audio_url content parts. The top level images and videos lists are added to the last user message. Top level audios and the video processing options such as video_fps are refused with 400, since native SGLang reads media only from content parts.

  • Sampling fields that a request leaves out come from the checkpoint generation config, as with sglang serve. For Cosmos3 Nano that is temperature 0.7, top_p 0.8 and top_k 20.

  • Other native chat request fields go under the stage name in stage_params, for example {"stage_params": {"reasoner": {"ignore_eos": true}}}. Native SGLang validates them, and an invalid one returns 400.

  • The stage returns one text completion per request, so stage_params that ask for native tools, n above 1 or log probabilities on a stream are refused with 400. Run sglang serve on the same checkpoint when you need native tool calls, a reasoning parser, several choices or streamed log probabilities.

Media policy#

--allowed-local-media-path and --allowed-media-domain apply as on the other chat routes. File URLs need --allowed-local-media-path and must resolve inside it, and local paths are confined to it once it is set. While --allowed-media-domain is set, the stage fetches remote media itself, checking the domain on every redirect, keeps native SGLang’s media_url_max_file_size_mb limit, and hands native SGLang a temporary file that it removes when the request ends. A refused reference returns 400 before native SGLang loads anything. A fetch that fails returns the status the other chat routes return: 400 when the media host answers 404, 500 when it fails, times out or sends more than the limit.

Sizing and admission#

  • Native SGLang derives the context length from the checkpoint and the memory fraction from the GPU, as sglang serve does. stages.reasoner.gpu_memory_fraction turns the placement budget into the memory fraction, and --mem-fraction-static or any native server argument under stages.reasoner.engine, for example --reasoner.engine.context_length 32768, overrides native SGLang. An explicit memory fraction wins over the placement budget with a warning.

  • Admission is native SGLang’s by default: up to its max_running_requests run at once and further requests wait in its queue, as with sglang serve. Set stages.reasoner.engine.max_running_requests and stages.reasoner.engine.max_queued_requests to bound it. Omni then answers 503 to requests beyond their sum before any media is loaded.

  • Before the stage reports ready it sends the same warmup request as sglang serve. stages.reasoner.engine.skip_server_warmup turns it off.

Lifecycle#

  • A cancelled request or a closed stream aborts the native request at once.

  • Native SGLang processes exit with the stage process. If a native scheduler or detokenizer process fails, native SGLang stops the stage and the server stops.