MLX Path

MLX Path#

Rules for Apple Silicon work. They write down what the native runtime PRs under #2537 already do.

Layout#

Path

Contents

sglang_omni_mlx/native/

C++ runtime on MLX: one server binary per model, a parity CLI per model in tools/, shared code in src/layers, src/swift_port, src/asr_service.

sglang_omni_mlx/native/ci/

Model pins, the 392-clip corpus, golden outputs, server API tests.

sglang_omni_mlx/<model>/

Python MLX servers, e.g. the Qwen3-ASR reference server.

Voxt/

The macOS app that runs the native servers. See Voxt/backend/README.md.

Rule 1: MLX is a separate path#

MLX code is not abstracted together with the CUDA path in sglang_omni.

  • New MLX models, kernels and servers go under sglang_omni_mlx/.

  • Do not add MLX branches, workers or platform hooks to sglang_omni/.

  • No shared backend interface, base class or registry across the two paths. Share code freely inside the MLX path, not across paths.

  • The paths meet only at the OpenAI-style HTTP API (/v1/audio/transcriptions, /v1/realtime, /health).

Why:

  • Runtimes differ. sglang_omni is a multi-process CUDA pipeline on PyTorch; the MLX path is one process per model on unified memory, shipped as C++ binaries. An interface over both hides what makes each fast.

  • Dependencies differ. SGLang’s wheel pulls CUDA. Keeping MLX out lets Voxt run without CUDA, PyTorch or Python.

  • CI differs. Mac CI only runs on Voxt/** and sglang_omni_mlx/**, and GPU CI never runs on a Mac. A shared layer would break one side without CI noticing.

The legacy MLX code in sglang_omni (mlx_model_worker.py, MlxTpModelWorker, the per-model mlx/ packages) is removed in #2725; do not add it back. If a change really needs a cross-path abstraction, open an issue and agree on it with a maintainer first.

Rule 2: Contribution requirements#

Placement and reuse

  • A native model adds <model>_model, <model>_transcriber, a server on the shared asr_service, a CLI in tools/, and CMake targets.

  • Reuse layers, swift_port, the Sortformer front end and the Silero VAD port. If your code duplicates something on main, move it into the shared file in the same PR.

  • Voxt changes stay in Omni*.swift, VoxtTests/Omni* and Voxt/backend/, plus minimal routing edits.

Parity

  • Name the reference (Voxt’s Swift backend at a pinned mlx-audio-swift commit, or a Python MLX reference) and use the same weights and MLX version.

  • Keep the reference’s behavior. Behavior changes go in a separate PR.

  • Report English WER, Chinese CER and mixed MER on the 392-clip corpus, reference vs native, plus the count of identical clips. Explain where differing clips diverge.

Golden and pins

  • Pin every checkpoint file by revision and SHA-256 in ci/model_pins.json.

  • Add a golden under ci/model_golden/. Import the Mac CI runner’s outputs (golden-outputs artifact, --import) so the runner checks exactly.

  • Add a second golden for paths the corpus misses (long-audio chunking, VAD).

Tests

  • Server tests in ci/test_model_servers.py: JSON, SSE and realtime success paths; every invalid input returns 400; session-state errors, cancel on disconnect, and wrong model kind.

  • Assert structure, not one chip’s exact text; exact text lives in the golden.

  • New Voxt Omni tests go into the voxt-unit job in voxt-mac-ci.yaml.

Robustness

  • Check the checkpoint config at load, so bad checkpoints fail at load, not mid-request.

  • Validate requests before they reach the worker.

  • Streaming sessions use bounded memory: trim consumed audio, cap session length, clear the MLX cache.

Performance

  • Mac CI checks correctness only. Report total time, realtime factor, peak memory and load time against the reference, with the device stated.

CI and style

  • Target main (stacked PRs get no Mac CI), non-draft, with the run-ci label. Both Mac CI jobs must pass on a real run.

  • Run pre-commit and follow coding-style.md.

  • Update Voxt/backend/README.md. Follow the PR format of #2676 or #2689.