Architecture#
SGLang-Omni is the multi-stage runtime for omni models: models that accept mixed text, image, audio, and video inputs and may emit text, audio, or other modalities.
System Overview#
HTTP API -> Client -> Coordinator -> Stage -> Scheduler -> ModelRunner -> model forward
Layer |
Duty |
|---|---|
OpenAI-compatible request and response schemas, SSE framing, HTTP errors |
|
|
|
Request lifecycle, entry-stage submission, terminal result collection, abort broadcast |
|
Control-plane IO, relay IO, fan-in, stream routing, scheduler inbox/outbox bridging |
|
Per-stage execution loop and failure propagation to stage outbox |
|
AR forward preparation, model forward dispatch, output extraction |
|
Control-plane messages and relay data transfer between stages |
|
Checklist and lifecycle rules for adding TTS model families |
Refer to the layer-specific document for specific design details.
Directory Layout#
sglang_omni/
|-- pipeline/ # Inter-stage orchestration, stages, coordinator, processes
|-- scheduling/ # Scheduler loops and inbox/outbox message types
|-- model_runner/ # Shared model runner abstractions for AR stages
|-- speculative/ # Adapters that run SGLang's draft workers on AR stages
|-- models/ # Model-specific configs, stages, request builders, modules
|-- config/ # PipelineConfig, StageConfig, config manager, topology
|-- relay/ # Data transfer backends
|-- serve/ # HTTP server and OpenAI-compatible API adapter
|-- client/ # Internal client used by API adapters
`-- proto/ # Request, payload, stage, and control-plane message types
Model Directory Convention#
Model-specific code should stay under sglang_omni/models/<model>/.
Recommended layout:
models/<model>/
|-- config.py # PipelineConfig subclass and StageConfig list
|-- stages.py # stage factories
|-- routing.py # optional data-driven routing helpers
|-- request_builders.py # inter-stage payload transforms
|-- payload_types.py # typed model-specific payload state
|-- callbacks.py # feedback callbacks or strategy, when needed
`-- components/ # model modules, processors, vocoders, adapters
Only model-local behavior belongs here. The framework-owned layers are still
Stage, Coordinator, schedulers, model-runner bases, relay, runtime prep, and
runners.
Naming and lint checks#
Use public names for classes, functions, methods, and attributes defined in
sglang_omni/ and tests/. The leading-underscore lint hook checks every Python
file under those directories, including new model packages and test suites. It
flags class and function
definitions, every attribute read or write, and string names passed to
getattr. It ignores vendor copies, dunder names such as __init__, a lone
_, and definitions inside functions.
A name inherited from an upstream object still fails the check. Keep that spelling, and mark the line:
session_request._omni_prompt_cache_key = adapter_request._omni_prompt_cache_key # noqa: leading-underscore
When a local definition must keep a name because an external interface
requires it, put # noqa: leading-underscore on that line and explain why:
class ExternalAdapter(ExternalBase):
# ExternalBase calls this hook by its exact name.
def _required_external_hook(self): # noqa: leading-underscore
return self.state
The exemption keeps the definition out of lint violations and automatic renames.
Its references retain the same name. A comment such as
# skip leading-underscore class and function names is not an exemption.
The pre-commit hook runs scripts/check_leading_underscore.py and only reports
violations. It does not rewrite files. To rename violating class and function
definitions and their references within the same file, run the fixer and review
the diff:
python scripts/check_leading_underscore.py --fix
The fixer never renames attributes, and it does not update callers in other
files, pytest fixture parameters, or string references such as
monkeypatch.setattr(module, "_name", ...). Rename those by hand, preserve
third-party API names, and update those references explicitly.
Every if under sglang_omni/ must have an else, or belong to an if/elif
chain that ends in else. Returning, raising, or a one-line body does not
exempt it. There is no # noqa exemption. Prefer an else branch that does
real work. At least write else: pass.
The pre-commit hook runs scripts/check_if_else.py and only reports
violations. It does not rewrite files. When the other branch truly does
nothing, insert else: pass yourself, or run the fixer:
python scripts/check_if_else.py --fix
Run pre-commit run --all-files before submitting a change.
Documentation checks#
The doc-style hook rejects emoji in docs/**/*.md, including inline code and URLs, while allowing text presentation. Only top-level fences indented by up to three spaces are exempt; fences nested in lists or blockquotes are not supported. Unclosed fences are reported at their opening line. The checker reports file/line diagnostics without modifying files (exit 0: clean, 1: violations).
Opening code fences must have a non-empty info string, such as python, bash, text, or mermaid. Any non-empty value is accepted; there is no language whitelist. This check uses the same top-level fence parsing as the emoji rule and reports missing labels at the opening line.
Markdown basenames in docs/**/*.md must match [a-z0-9_]+\.md, using lowercase ASCII letters, digits, and underscores. Only docs/README.md is exempt; docs/nested/README.md is rejected. Directory names are unrestricted. For example, guide.md, model_v2.md, and 123.md are valid, while Guide.md, model-v2.md, and names containing spaces or non-ASCII characters are rejected. The checker reports the file path and line 1 without renaming or rewriting files.
Markdown pages must appear in a toctree in docs/**/*.rst or a fenced MyST {toctree} directive in docs/**/*.md. Entries may use guide.md, extensionless guide, or a title such as Guide title <guide>. Paths beginning with / resolve from docs/; other paths resolve from the containing document’s directory. With :glob: (or glob: true in MyST YAML options), patterns match extensionless document names: * stays within one directory and ** can cross directories. External URLs, self, option values, and examples inside code blocks or comments do not count as page entries. Missing entries are reported at the Markdown file’s line 1 as missing-toctree. Editing a navigation file triggers the existing doc-style hook, alongside Markdown, checker, and test changes.
Only these pages are exempt from navigation inclusion; emoji, fence-label, and filename checks still apply:
README.md: repository overview, separate from the published manual.design/gpu_radix_hash.mdanddesign/refactor_rfc.md: internal notes outside the user manual.developer_reference/mlx.md: excluded redirect; the published page ismlx/index.md.get_started/apiserver_quickstart.md: legacy exemption; this page is now listed in the Get Started sidebar.basic_usage/python_router.md: retained until the Python router is removed separately.
python -m pip install -r scripts/requirements-doc-lint.txt
python scripts/check_doc_style.py
When upgrading emoji, recheck the supplements for standalone regional indicators
and [0-9#*] followed by VS16.