CI Workflows and Labels#
SGLang-Omni CI has two parts. A few fast checks run on GitHub-hosted runners for every pull request, though one that changes only the MLX path runs just Lint among them. Everything else runs on self-hosted runners (H100 machines for the GPU stages, an Intel Xeon host and an Intel GPU host), except Mac CI, which runs on GitHub’s macOS runners but also needs its label.
A pull request runs none of the self-hosted CI until a maintainer adds CI labels. Once labeled, every later push re-runs CI for as long as the labels stay. Draft pull requests are blocked from self-hosted CI even when they are labeled.
In this sense, two kinds of labels exist and they combine: a stage label decides whether a part of the self-hosted CI runs, and a model label names only the model. A model label never includes a stage name and never selects a stage by itself. run-lint-only is separate: it satisfies the gate and runs no self-hosted job. The full label rules come after the workflows. .github/scripts/resolve_ci_labels.py implements them, and every self-hosted workflow resolves its labels through that script.
Overview#
Workflow |
Runner |
What it runs |
When it runs |
|---|---|---|---|
Lint |
GitHub-hosted |
pre-commit |
Every pull request |
Test Layout |
GitHub-hosted |
Test file location check |
Every pull request except MLX-only ones |
Docs Check |
GitHub-hosted |
Sphinx documentation build and the documentation style checker’s tests |
Every pull request except MLX-only ones |
Playground Test |
GitHub-hosted |
The playground frontend tests |
Every pull request except MLX-only ones |
Rust Router |
GitHub-hosted |
Rust router format, lint, tests and release build |
Every pull request except MLX-only ones |
Release Docker Images Nightly (NPU) |
GitHub-hosted |
NPU configuration checks and an ARM64 image build |
Pull requests that change its Dockerfile, installer, pyproject files or workflows |
Omni CI: PR Test |
H100 |
The unit test suite twice: once with CUDA hidden, once for the tests that need a GPU |
Any stage label except |
Omni CI: ASR |
H100 |
Multi-speaker transcription and diarization with MOSS-Transcribe-Diarize, then SeedTTS English and Chinese word error rate and speed on Fun-ASR-Nano-2512, Qwen3-ASR-1.7B or Whisper-large-v3 |
|
Omni CI: TTS |
H100 |
Voice cloning on SeedTTS with Higgs-TTS, MOSS-TTS, Qwen3-TTS or Fun-CosyVoice3: non-streaming and streaming speed, word error rate, speaker similarity and UTMOS, streaming consistency, first-audio latency for Qwen3-TTS, and a serving stress test on Higgs-TTS |
|
Omni CI: MPS |
H100 |
External CUDA MPS DP2 non-streaming validation, native MPS lifecycle validation, and native CUDA IPC weight-sharing lifecycle validation |
|
Omni CI: Omni |
H100 |
Qwen3-Omni or MiniCPM-o 4.5: thinker length limits, SeedTTS voice cloning, accuracy and speed on MMMU, MMSU, Video-MME and Video-AMME with text and spoken answers, a tensor parallel thinker, and Qwen3-Omni process replicas |
|
Omni CI: Duplex |
H100 |
Full-Duplex-Bench v1.0 and v1.5 on three interaction strategies: native full duplex (MiniCPM-o 4.5), turn based (Qwen3-Omni with server VAD) and cascaded (Qwen3-ASR-1.7B, Qwen3.8-27B and Qwen3-TTS Base cloning one Vivian recording). Each gates turn taking, overlap behavior, latency and listening quality against its own baseline |
|
CPU CI |
Intel Xeon ( |
Job |
|
XPU CI |
Intel GPU ( |
Intel GPU device layer tests |
|
Mac CI |
GitHub macOS |
MLX native runtime build and protocol, the Voxt app’s Swift tests |
|
The Omni CI stages are the core of the project’s CI: they serve the main ASR, TTS and omni models end to end on H100 and gate their accuracy and speed.
Checks On Every Pull Request#
These workflows run on GitHub-hosted runners for every pull request, drafts included, with no label. An MLX-only pull request changes nothing outside sglang_omni_mlx/, Voxt/, .github/workflows/mac-ci.yaml and .github/workflows/voxt*.yaml. It runs only Lint among these checks and does not start Omni CI: Mac CI and Voxt CI test it, and Mac CI checks its labels.
Workflow |
Job |
What it checks |
|---|---|---|
Lint |
|
|
Test Layout |
|
Every Python file under |
Docs Check |
|
The documentation builds with |
Docs Check |
|
The tests of the documentation style checker in |
Playground Test |
|
|
Rust Router |
|
|
Release Docker Images Nightly (NPU) |
|
Only when the pull request changes |
The Omni CI workflow#
Omni CI runs PR Test and the seven H100 model stages. Its preflight job waits for lint, resolves the labels and draws the models once. A shared setup job then builds the virtualenv and downloads model weights. With every stage label, the stages run in this order:
PR Test → ASR, TTS, Omni and Duplex side by side → MPS
Only PR Test and the selected stages actually run; a stage that is not selected is skipped at once. ASR, TTS, Omni and Duplex start together as soon as PR Test finishes, whether it passed or not, and their jobs spread over whichever H100 runners are free. Jobs within a stage run side by side as well, except TTS stage 3, which waits for TTS stages 1 and 2 because it compares their results. MPS starts after the other selected stages finish: it tests the PR head commit instead of the merge commit, so it may rebuild the PR’s shared virtualenv, and that must not happen while another stage is using it. With run-tts-ci and run-mps-ci, the order is PR Test → TTS → MPS. A stage that fails does not stop any other stage. A failed setup job blocks every stage.
PR Test#
Job |
What it runs |
|---|---|
|
First runtime-balanced partition of |
|
Second runtime-balanced partition of the same CPU unit-test suite, with CUDA hidden |
|
Third runtime-balanced partition of the same CPU unit-test suite, with CUDA hidden |
|
|
The CPU partitions run as independent matrix jobs with fail-fast disabled. Files stay intact and are balanced using .github/unit_test_durations.json. Pytest discovers new tests automatically; files without a timing entry receive a one-second estimate, and new cases in an existing file stay in that file’s partition. Local pytest remains unpartitioned unless the CI plugin is explicitly loaded.
The weights are per-file p90 elapsed estimates from three successful H100 A/B runs: 41197670240, 41161315193 and 41140707601. Each file has three observations. The estimates aggregate timestamp intervals between completed pytest cases, excluding initial collection; they are not isolated setup/call/teardown measurements. These are PR runs with different revisions, not scheduled main-branch observations. The weights are a committed snapshot, not an automatically refreshed model.
run-unit-test-ci-only selects PR Test on H100 without selecting model or hardware stages. It does not run the Intel Xeon job intel-cpu-ci (unit_test/cpu); that job needs run-intel-cpu-ci and the CPU path filter. Other stage labels remain additive, so keep them absent for unit tests only. Shared setup still runs, including model-weight provisioning. PR Test also runs whenever any other stage label selects a stage, the Intel hardware stages included. run-mac-ci and run-lint-only do not run PR Test or setup.
ASR#
Job |
What it covers |
|---|---|
stage 1 - MOSS-TD multi-speaker |
Multi-speaker transcription and diarization with MOSS-Transcribe-Diarize, behind a two-worker router: movies800 in offline and streaming mode, AISHELL-4 long-form audio and googletime. It gates character error rate and speed. This job always uses MOSS-Transcribe-Diarize, whatever model the stage draws. |
stage 2 - SeedTTS ASR |
The selected model on the SeedTTS English and Chinese splits. It gates word error rate on both splits, and throughput, latency and real-time factor on the English split. |
The stage label is run-asr-ci. A model label does not name the stage. Combine it with run-asr-ci to pick the model for stage 2:
Model label |
Checkpoint |
In the random pool |
|---|---|---|
|
|
Yes |
|
|
Yes |
|
|
Yes |
TTS#
Job |
What it covers |
|---|---|
stage 1 - non-streaming |
Voice cloning on SeedTTS English at concurrency 16, behind a two-worker router: speed, word error rate, speaker similarity and UTMOS. |
stage 2 - streaming |
The same benchmark with streaming output: speed and word error rate. |
stage latency - streaming first audio |
Median time to first playable audio under Poisson arrivals, with one worker alone on the GPU. Runs only when the stage uses one of the two Qwen3-TTS checkpoints. |
stage 3 - consistency |
Checks the artifacts of stages 1 and 2: both cover all 1088 SeedTTS English samples, every streaming request finished, and streaming and non-streaming audio durations agree. |
stage 4 - serving stress |
A mixed production workload over the REST, streaming and WebSocket endpoints through a two-worker router. This job always uses Higgs-TTS, whatever model the stage draws. |
The stage label is run-tts-ci. A model label does not name the stage. Combine it with run-tts-ci to pick the model for stages 1, 2, 3 and the latency stage:
Model label |
Checkpoint |
In the random pool |
|---|---|---|
|
|
Yes |
|
|
Yes |
|
|
Yes |
|
|
No |
|
|
Yes |
Qwen3-TTS CustomVoice uses a named voice instead of a reference clip, and its thresholds are not yet calibrated on the CI host. It reports its metrics without gating on them, and only its model label selects it.
MPS#
run-mps-ci runs three sequential validations on H100:
External CUDA MPS DP2 non-streaming validation checks that two TTS router workers share one H100 through CUDA MPS and meet the TTS word error rate, speaker similarity, UTMOS and audio integrity assertions.
Native MPS lifecycle validation uses Higgs-TTS to check daemon sharing, recovery behavior and clean teardown.
Native CUDA IPC weight-sharing lifecycle validation uses MOSS-TTS Local to check shared-weight memory savings, process startup order and leader failure handling.
All three validations run under the existing run-mps-ci label. The label accepts run-higgs-tts-3-4b and run-moss-tts-local-transformer-v1.5 for the external CUDA MPS DP2 validation; those labels do not change the fixed models used by the native lifecycle validations. A model label does not run MPS. With run-tts-ci and run-mps-ci, one of these labels selects the external MPS model and the TTS model. With only run-tts-ci, MPS does not run. With no Higgs or MOSS label, the external MPS validation draws from its own pool.
Model label |
Checkpoint |
MPS config |
In the random pool |
|---|---|---|---|
|
|
|
Yes |
|
|
|
Yes |
Omni#
Every job runs the selected model unless noted otherwise. A job named “Talker” asks for text and audio output, and also gates the word error rate between the text answer and an ASR transcript of the spoken answer.
Job |
What it covers |
|---|---|
stage 1 - thinker length integration |
Overlong prompts and prompt plus |
stage 2 - TTS |
Voice cloning on SeedTTS: speed, word error rate, speaker similarity and UTMOS. |
stages 3 and 4 - MMMU, MMMU Talker |
Image and text question answering: accuracy and speed. |
stages 5 and 6 - MMSU, MMSU Talker |
Spoken audio understanding: accuracy and speed. |
stages 7 and 8 - Video-MME, Video-MME Talker |
Video understanding: accuracy and speed. |
stage 9 - Video-AMME |
Video with its audio track: accuracy and speed. |
stage 10 - Video-AMME Talker |
Video with audio and spoken answers, with the thinker sharded across two GPUs (tensor parallel size 2). |
stage 11 - process replicas |
A two-GPU deployment with one code2wav replica on each GPU; checks that both replicas start, register and receive requests. Runs only for Qwen3-Omni. |
The stage label is run-omni-ci. A model label does not name the stage. Combine it with run-omni-ci to pick the model:
Model label |
Checkpoint |
In the random pool |
|---|---|---|
|
|
Yes |
|
|
Yes |
Duplex#
Each job runs a fixed Full-Duplex-Bench selection on one interaction strategy, the first samples of every v1.5 category and v1.0 subset, then scores it with the runbook: Parakeet transcripts, the official timing, unified v1.0 metrics, listening quality, and the Qwen3.8-27B behavior and semantic judges. The three strategies behave differently by design, so each job gates only regressions from its own baseline, calibrated on H100. A run must first exercise the v1.5 overlaps: the model has to be speaking when the event begins in at least three of the five pairs of every category and fifteen of the twenty in all, or the overlap scores say nothing about the model. The seven behavior and v1.0 rates score five cases each and are gated together: the run fails when a rate scored fewer than five cases, when it has more than five fewer cases on target than the baseline in all, or when a rate whose baseline met three or more of its five cases meets none. A baseline rate is the mean count over repeated runs, so it may stand for a fraction of a case. The latency metrics, UTMOS and voice similarity each have their own bound. The listening omission rate and the semantic judge’s two rates are reported but not gated.
Stage label |
Job |
System |
|---|---|---|
|
Full-Duplex-Bench - full duplex |
MiniCPM-o 4.5, native full duplex, one server on each of two GPUs |
|
Full-Duplex-Bench - turn based |
Qwen3-Omni with server VAD turn detection ( |
|
Full-Duplex-Bench - cascaded |
Qwen3-ASR-1.7B, Qwen3.8-27B with thinking off and Qwen3-TTS Base cloning one Vivian recording, with server VAD, on two GPUs ( |
The three jobs share the pull request’s Full-Duplex-Bench assets, so they run one after another, and a failed job does not skip the next. Each stage has one system and no model label.
Hardware workflows#
Each hardware stage is its own workflow. Its gate checks live labels and skips draft blocking, label resolution and lint checks unless its stage label selects it. None of them has model labels. Each keeps a path filter, so a stage label alone does not force a run: a pull request that changes none of the stage’s paths skips its tests.
CPU CI#
CPU CI builds the Intel Xeon image from docker/cpu.Dockerfile and runs pytest tests/unit_test/cpu/ in the job intel-cpu-ci (unit_test/cpu). Its tests run when the pull request touches sglang_omni/ (except Markdown files), tests/unit_test/cpu/, pyproject_cpu.toml, scripts/cpu/, docker/cpu.Dockerfile or the workflow itself.
The stage label is run-intel-cpu-ci. run-unit-test-ci-only does not select this job. When the paths are untouched, or when the pull request selects other stages but not Intel CPU, the test job is skipped and the CPU CI Status check still reports success.
XPU CI#
XPU CI pulls or builds the Intel GPU image from docker/xpu.Dockerfile and runs the device layer tests in tests/unit_test/xpu/test_device_layer.py and the dots.tts flow head and tail tests. Its tests run when the pull request touches sglang_omni/, tests/unit_test/xpu/, tests/unit_test/dots_tts/, pyproject_xpu.toml, scripts/xpu/, docker/xpu.Dockerfile or the workflow itself.
The stage label is run-intel-xpu-ci.
Mac CI#
Mac CI has two jobs on GitHub’s macOS runners. The native runtime job builds the MLX runtime and a stub server, then tests the protocol every native server shares on the stub server, each server’s startup checks, and that the installed runtime links only the system and itself. The Swift test job runs the tests of the Voxt app’s helper (Voxt/helper). Model outputs are checked by hand per model PR with the tools in sglang_omni_mlx/native/ci. The workflow starts when the pull request touches sglang_omni_mlx/, Voxt/, .github/workflows/mac-ci.yaml or .github/workflows/voxt*.yaml; otherwise it does not appear on the pull request at all. Unlike the other hardware gates, its gate fails without a CI label, because Omni CI does not start on an MLX-only pull request. On a pull request, the native runtime job runs only when sglang_omni_mlx/ changes and the Swift test job only when Voxt/helper/ changes, not counting Markdown files; a change to the workflow itself runs both, and Voxt Nightly always runs both.
The stage label is run-mac-ci. Use it for any change on the MLX path: the SGLang Omni MLX runtime in sglang_omni_mlx/, such as a runtime optimization or a new native server, as well as the Voxt app. It runs only this workflow, which waits for lint to pass and then starts the Mac jobs. It does not run PR Test or any other H100 job, because the Mac jobs test the MLX runtime and the Voxt helper, not the sglang_omni Python package that PR Test covers.
Label rules#
A stage runs only when its stage label is on the pull request. There is no label that selects every stage; add the stage labels the change needs. If a fundamental change really needs all of CI, add
run-asr-ci,run-tts-ci,run-mps-ci,run-omni-ci,run-native-duplex,run-turn-based-duplex,run-cascaded-duplex,run-intel-cpu-ci,run-intel-xpu-ciandrun-mac-ci, or comment/tag-and-rerun-ci asr tts mps omni full-duplex turn-based cascaded intel-cpu intel-xpu mac.Self-hosted CI needs a stage label or
run-lint-only. Without one, the Omni CI gate fails withNo CI label found, and so does the Mac CI gate when Mac CI starts; CPU CI and XPU CI skip their tests without failing. An MLX-only pull request does not start Omni CI, so the Mac CI gate is the one that fails:run-mac-ciruns its Mac tests, andrun-lint-onlyruns none. The checks on GitHub-hosted runners do not depend on labels.run-lint-onlyruns no self-hosted job, so a green lint check leaves no self-hosted failure. It does not turn off a stage label that is also present.Any stage label except
run-mac-cialso runs PR Test. A pull request labeled onlyrun-mac-ciruns Mac CI and no H100 job.A model label names a model, not a stage. It picks that model for every selected stage that accepts it, and never selects a stage by itself. Higgs and MOSS labels are accepted by TTS and MPS. If no selected stage accepts the label, the gate prints a warning.
A selected stage without a model label draws its model at random from that stage’s pool. The draw is seeded by the workflow run ID, so re-running jobs in the same run keeps the same model, while a new push or a newly added label starts a new run and draws again.
Each stage takes at most one model label. Two model labels for the same stage fail the gate, even when that stage is not selected.
Stage labels combine.
run-tts-citogether withrun-mps-ciruns PR Test, TTS and MPS, and nothing else.
Stage label |
Selects |
|---|---|
|
No self-hosted job. The gates succeed. |
|
PR Test on H100. Not |
|
ASR in Omni CI |
|
TTS in Omni CI |
|
MPS in Omni CI |
|
Omni in Omni CI |
|
Duplex full duplex in Omni CI |
|
Duplex turn based in Omni CI |
|
Duplex cascaded in Omni CI |
|
Intel Xeon job |
|
XPU CI |
|
Mac CI. Not PR Test. |
Examples#
The table is for a pull request that is not MLX-only and lists only the self-hosted CI; the checks on GitHub-hosted runners run in every case. An MLX-only pull request starts Mac CI instead of Omni CI: with no label its gate fails, and run-mac-ci runs Mac CI alone.
Labels |
What runs |
|---|---|
none |
Nothing. The Omni CI gate, and the Mac CI gate when Mac CI starts, fail with |
|
Nothing self-hosted. The gates succeed. |
|
PR Test only, after shared setup. |
|
PR Test, then TTS on a random model from its pool. |
|
PR Test, then TTS on MOSS-TTS. MPS does not run. |
|
PR Test, then MPS on Higgs-TTS. TTS does not run. |
|
PR Test, TTS and MPS, each drawing its own model. |
|
PR Test, then TTS and MPS, both on Higgs-TTS. |
|
PR Test and TTS on H100, and Mac CI. |
|
Mac CI only, once lint passes. Omni CI passes its gate and skips setup and PR Test. |
|
PR Test and TTS. The Omni model label is ignored with a warning, because Omni is not selected. |
|
Nothing. The gate fails, because a model label does not select a stage. |
|
Nothing. The gate fails, because TTS and MPS accept two model labels. |
Label changes, re-runs and manual runs#
Adding a label manually, CI or not, starts a new run of Omni CI (unless the pull request is MLX-only), CPU CI, XPU CI and Mac CI (when its paths match) and cancels the run of the same workflow in progress, so the new label set takes effect at once. Models that no label pins are drawn again in the new run.
Removing a label cancels only the workflows that the remaining labels no longer select. Removing run-mac-ci while run-tts-ci stays cancels Mac CI and leaves Omni CI running. Removing run-tts-ci while run-mac-ci stays cancels Omni CI and leaves Mac CI running. Removing the last stage label cancels Omni CI, CPU CI, XPU CI and Mac CI. run-lint-only does not keep any of them. A run that is still selected keeps the stages and models it resolved when it started; push again or re-run all jobs to pick up the change.
Re-running jobs reuses the run ID, so every stage keeps the model it drew. The gate reads the live labels from the pull request, so re-running all jobs picks up labels added or removed since the run started.
A maintainer can start Omni CI by hand from the Actions tab or with gh workflow run. The ci_labels input takes space-separated labels and resolves them exactly like pull request labels. It is required:
gh workflow run omni-ci.yaml --ref <branch> \
-f ci_labels="run-tts-ci run-moss-tts-local-transformer-v1.5"
Manual runs of CPU CI, XPU CI and Mac CI ignore labels and always run their tests, and so does the run of Mac CI that Voxt Nightly starts each night.
Maintainers with CI permissions can use /tag-and-rerun-ci with one or more short stage or model targets to re-run completed CI workflows:
/tag-and-rerun-ci unit-test
/tag-and-rerun-ci tts asr
/tag-and-rerun-ci tts asr omni
/tag-and-rerun-ci asr qwen3-omni
/tag-and-rerun-ci qwen3-tts
/tag-and-rerun-ci qwen3-tts qwen3-asr
/tag-and-rerun-ci mps-higgs
Stage targets are lint, unit-test, asr, tts, mps, omni, full-duplex, turn-based, cascaded, intel-cpu, intel-xpu and mac. lint adds run-lint-only. Model targets are fun-asr, qwen3-asr, whisper-asr, higgs, moss, qwen3-tts, qwen3-tts-custom-voice, cosyvoice3, qwen3-omni, minicpmo, mps-higgs and mps-moss. higgs and moss add run-tts-ci plus run-higgs-tts-3-4b or run-moss-tts-local-transformer-v1.5. mps-higgs and mps-moss add run-mps-ci plus that same model label. qwen3-tts adds run-tts-ci plus run-qwen3-tts-12hz-1.7b-base. A short model target adds its stage label and its model label. The model label itself never includes the stage name.
higgs or mps-higgs selects Higgs for every selected stage that accepts it, TTS and MPS included. Model choices not specified by the command keep their existing labels or use the normal random selection. Bare /tag-and-rerun-ci and full GitHub label arguments, such as run-tts-ci, are not accepted. Manual dispatch still accepts full labels through its ci_labels input.
The command replaces a conflicting Higgs or MOSS label for both TTS and MPS, because those labels are shared. It replaces other model labels only in the requested stages, and keeps other labels. Existing stage labels still apply, so arguments do not narrow an already broader selection. It re-runs all jobs of the newest completed selected parent workflows on the current PR commit, unless GitHub requires workflow approval. If the newest run is active, the command leaves it running rather than retrying an older run. To apply changed labels after preflight has resolved the selection, run the command again after the active workflow finishes. Path filters and PR gates still apply.
/tag-run-ci-label accepts the same short targets but only updates labels; it also requires at least one target. Labels added by the bot do not start workflows; push a commit or use a tag-and-rerun command to start them. /rerun-failed-ci retries failed jobs without changing labels or stopping active runs. PR authors can use it on their own PRs.
Gate messages#
Message |
Meaning |
|---|---|
|
The pull request has none of those labels. Add one of them. |
|
Two model labels name the same stage. Remove one of them. |
|
A warning, not a failure. The model label has no effect until its stage is selected. |
|
Lint failed on this commit. Fix it and push; self-hosted CI starts once lint passes. |
|
Merge or rebase onto the current |
A branch created before these labels existed does not contain resolve_ci_labels.py. On such a branch, the Omni CI and Mac CI gates fail until the branch merges the current main.