Same-GPU Data Parallelism with CUDA MPS#
TL;DR: Same-GPU DP with CUDA MPS can substantially increase throughput. In the pinned TTS tests below, saturated DP2 and DP3 configurations reached 1.4 to 2.1x the tuned single-replica throughput.
A common data-parallel deployment assigns one GPU to each replica. When a tuned replica still leaves substantial GPU headroom, colocating multiple replicas on the same GPU can improve per-GPU throughput.
Same-GPU data parallelism runs several complete serving replicas on one GPU and lets CUDA MPS share the GPU between them. This is a conditional and ongoing optimization. We are excited to share it and call for the community to join the exploration.
Native runtime support (--mps)#
The runtime can manage MPS itself for the processes of one pipeline. When a
pipeline colocates two or more single-GPU stage processes on one GPU (for
example a frontend process next to the generation process, or process-level
replicas), pass --mps auto:
sgl-omni serve --model-path <model> --mps auto
Modes (--mps on the CLI or mps: in the pipeline config; default off):
off: MPS is never touched.auto: MPS is enabled on every GPU that hosts two or more single-GPU, non-TP CUDA processes of this pipeline. GPUs with one process and TP groups run without MPS.on: a single eligible process is enough, and an MPS-incapable platform is a hard error instead of a warning. Useonfor same-GPU data parallelism: everyserve --mps onon one GPU joins the same daemon.
Both auto and on reject startup before acquiring MPS state when one process’s
resolved placement spans more than one physical GPU; use mps=off for that
placement. Factory CUDA devices use the narrowed worker’s local namespace:
cuda:0 is compatible with any single-GPU placement, while a nonzero cuda:N
is excluded because narrowing a worker to one UUID makes its only valid CUDA
ordinal cuda:0.
Pipeline-edge transport remains the responsibility of the existing router and
relay layers; it does not participate in MPS eligibility.
The daemon is shared per physical GPU (keyed by device UUID): MPS merges
kernels only for clients of one server, so the first serve creates the
daemon, later serves join it, and the last one to leave drains the clients
and quits it. Logical GPU ordinals are resolved once against the parent
process’s CUDA visibility and then grouped by physical UUID; auto counts the
combined client processes in each physical group. Pipeline or stage
environment defaults must not override CUDA_VISIBLE_DEVICES or
CUDA_DEVICE_ORDER while native MPS is enabled; set them on the parent command
instead, or use mps=off. Same-GPU DP is therefore just N serve commands:
sgl-omni serve --model-path <model> --mps on --mem-fraction-static 0.35 --port 8807
sgl-omni serve --model-path <model> --mps on --mem-fraction-static 0.35 --port 8808
Start replicas one after another and give each an explicit memory budget
(--mem-fraction-static or the stage-qualified
--<engine-stage>.engine.max_total_tokens), for the same KV-sizing
reasons described under the script recipe below. Route traffic with the
Omni Router.
Process-level replicas can size their pools in bytes instead, with
engine.kv_cache_bytes per stage and total_reserve_bytes for the replica’s
full footprint. The budgets are then checked against the card before any
replica is spawned: colocation requires a declared footprint, the summed
reserves and fractions must fit the card, and the summed KV pools alone must
too. engine.kv_cache_bytes and engine.max_total_tokens are mutually
exclusive on one stage, since the lower token cap would silently shrink the
byte-derived pool.
The runtime owns the full lifecycle. Every managed process is verified against the daemon’s client list before serving starts, because a process that misses the pipe directory silently falls back to time slicing. A watchdog fails the pipeline if daemon identity or control access is lost mid-serving. Shutdown re-evaluates the current client list, drains this serve’s clients, and quits the daemon only when no other serve still owns it.
If a managed worker does not exit before the shutdown timeout, the runtime
terminates that directly owned child process and reaps it before the launcher
exits, even when that directly owned worker is also an MPS client. It sends no
additional signal based on an MPS snapshot or client PID, and never
automatically signals the daemon, an unknown descendant, or a GPU-wide process
set. Process ownership and shared MPS state are handled independently: if
daemon identity, client ownership, or control state cannot be proved after the
workers are gone, the owner file is marked retained, its lock is released,
and the state directory is preserved. The current command then exits with a
detailed non-zero error instead of keeping a CLI owner alive.
Dirty state is never repaired automatically. A join requires the native
nvidia-cuda-mps-control.pid identity, a responsive control socket, and every
published owner lease to still be held. After a hard kill (SIGKILL, OOM kill,
node crash), even an idle daemon or one dead co-owner makes the next start
preserve the state and fail with owner/client details and safe cleanup guidance.
An unlocked or retained owner blocks every later start until an operator has
inspected and cleaned the state. Existing healthy co-owners keep serving, but
new owners cannot join and no process retries cleanup automatically. Clean up
and start again. A normal shutdown leaves nothing behind.
Operator notes: state lives under /tmp/sglang-omni-mps-<user>/<gpu-uuid>/
(SGLANG_OMNI_MPS_STATE_ROOT overrides it). Serves that are meant to share
one GPU must use the same state root, or they cannot discover each other’s
daemon and will run separate MPS servers that time-slice against each other.
The state root is created with mode 0700; an existing root must already be a
non-symlink directory owned by the current user with that mode. Native MPS
rejects CUDA_MPS_PIPE_DIRECTORY in the parent, pipeline, or stage environment
instead of overwriting or joining an external daemon. It likewise rejects
SGLANG_OMNI_WEIGHT_SHARE in those locations: inside one pipeline, CUDA IPC
weight sharing is requested with --weight-share on and the runtime assigns
replica roles itself, while the examples/mps_dp/launch.sh recipe below sets
that variable for its own separate serve processes.
Deploy#
The steps below are one continuous flow. We provide examples/mps_dp/launch.sh to manage the private MPS daemon and serving replicas for one run. It records replica processes, ports, and logs, starts replicas sequentially, verifies their KV capacity and MPS attachment, and tears down only the run it recorded. Detailed instructions are as follows:
Choose the GPU and NUMA node.
nvidia-smi --query-gpu=index,utilization.gpu,memory.used --format=csv,noheader
export GPU_ID=0
BUS=$(nvidia-smi --query-gpu=pci.bus_id --format=csv,noheader -i $GPU_ID)
BUS=${BUS,,}; BUS=${BUS:4}
NODE=$(cat /sys/bus/pci/devices/$BUS/numa_node) # if -1, set the node explicitly
numactl -H | grep "node $NODE cpus"
Pick a GPU that is idle, then find its NUMA node from the PCI bus id (drm card ordinals do not always match nvidia-smi ordinals). Choose non-overlapping physical CPU-core blocks from that node, one block per replica.
Launch the replicas.
CONFIG=examples/mps_dp/configs/higgs_h100_dp3.yaml N=3 CORE_BLOCKS="0-9 10-19 20-29" bash examples/mps_dp/launch.sh up
The command above is the validated H100 Higgs DP3 recipe. For the validated H200 Higgs DP8 recipe, use:
CONFIG=examples/mps_dp/configs/higgs_h200_dp8.yaml N=8 CORE_BLOCKS="0-3 4-7 8-11 12-15 16-19 20-23 24-27 28-31" bash examples/mps_dp/launch.sh up
The pipeline config supplies the model and per-replica runtime settings. The launcher environment supplies host-local placement, including the GPU, replica count, and CPU blocks. The launcher resolves the GPU’s NUMA node, assigns local ports automatically, starts a private MPS daemon, waits for each replica’s health check before starting the next, and verifies MPS attachment. The CPU blocks above are specific to the tested hosts; derive the correct non-overlapping blocks for your own CPU topology.
Both profiles set mem_fraction_static to 0.85; MF can override it. When CONFIG is unset, the existing MODEL and MAX_TOTAL_TOKENS interface remains available. Launching replicas sequentially avoids overlapping memory profiling and CUDA-graph capture during startup.
Identical --mem-fraction-static flags do not mean identical KV capacity. --mem-fraction-static budgets model weights and the KV pool against the GPU memory available when each replica starts. Roughly, the profiled KV memory is the requested fraction of free memory measured before model loading, minus model and fixed runtime allocations. It is a per-replica budget, not an additive share of the card. Because replicas start sequentially, earlier ones have already reserved memory, so later ones see a smaller free pool and allocate fewer KV tokens even when every flag is the same (in one run, three sequential mf=0.27 replicas received 97,503 / 53,149 / 20,961 KV tokens).
Memory profiling does not coordinate KV allocation across independent replica processes. For N > 1, every replica must resolve the same KV capacity, which can come from either sizing knob but never both:
engine.kv_cache_bytesin the pipeline config sizes every replica’s pool deterministically; the launcher verifies all replicas resolved the same capacity and rejects a simultaneousMAX_TOTAL_TOKENS, since a lower token cap would silently shrink the byte-derived pool.Without a byte budget, the launcher requires one common
max_total_tokensvalue, from the pipeline config or fromMAX_TOTAL_TOKENS, and rejects startup unless every replica resolves exactly that capacity.
Either cap applies independently to each replica; it is not divided across the pool. It is also independent of the request-level max_new_tokens limit and does not distribute requests between replicas.
The H100 Higgs DP3 profile uses 100000 tokens per replica. The H200 Higgs DP8 profile uses 30000 tokens per replica, preserving GPU memory headroom for non-KV runtime allocations, including the colocated audio encoder and vocoder. These values are specific to their configurations, not universal hardware defaults. Recalculate the cap after changing the model, GPU, runtime, replica count, memory settings, or CUDA-graph settings. If a replica cannot allocate the common cap, lower it or reduce the replica count.
The H200 profile’s 30000-token KV pool is smaller than the worst-case demand from 64 requests each generating up to 2048 new tokens, even before accounting for input tokens. Therefore, max_running_requests=64 is an admission ceiling rather than a guarantee that 64 long requests can decode concurrently. If the pool fills, SGLang retracts requests and returns them to the waiting queue until KV capacity becomes available.
Drive every replica to saturation.
The case study used one dedicated client per replica and drove all replicas in parallel. The measured goal is to keep every replica saturated. With equivalent replicas, random or round-robin routing can distribute a shared ingress across the pool; fill-one-then-next is another possible strategy, but the study did not compare them. Equal KV capacity makes the replicas comparable, but it does not by itself balance their queues. Validate the routing policy and per-replica saturation under your workload.
Verify MPS attachment.
MPS should be verified carefully. Four things are easy to conflate: environment variables set, daemon running, an MPS server exists, and the replica processes you launched are actually attached as clients. Only the last makes the comparison valid, and a replica that missed the pipe directory falls back to time-slicing without any error. The launcher verifies every replica against the MPS client list, writes the server-to-client PID mapping to mps_attach.txt, and fails startup if any replica is not attached.
Route traffic.
For easy deployment, you can register each replica endpoint with the Python Router. Keep the router’s --max-connections at least as large as the total offered concurrency. The case study did not benchmark router scheduling policies, so confirm that the selected policy keeps every replica driven and meets your workload’s latency and throughput requirements.
Tear down safely.
Stop new traffic, then run the teardown command printed by the launcher:
bash examples/mps_dp/launch.sh down <RUN_ID>
On a shared host, only touch processes you launched, and never treat “the GPU is empty” as the success condition. The launcher stops only the replica processes recorded for the selected run, waits for their MPS clients to detach, and then stops the private MPS daemon. It keeps the run state whenever cleanup cannot be confirmed.
Setting up and tearing down MPS is more involved than running a single replica, but in the pinned H100 Higgs tests the throughput gain was substantial. The table below shows the nominal completed-run ranges; the full accounting, including the failed and degraded runs, is in the case study.
Configuration |
Nominal throughput |
Relative to single |
|---|---|---|
Single c96 |
21.7 to 22.1 qps |
1.0x |
DP2 + MPS, 2 x c64 |
31.5 to 37.7 qps |
1.4 to 1.7x |
DP3 + MPS, 3 x c64 |
39.9 to 46.9 qps |
1.8 to 2.1x |
The throughput results in the table and H100 case study are from an 80 GB H100 with Higgs. The H200 DP8 profile was validated separately on the full SeedTTS English dataset at concurrency 64 per replica. Re-evaluate replica count, CPU allocation, token capacity, and saturation concurrency before applying either profile to different hardware or workloads.
How We Found This#
This recipe grew out of the serving profiling in #907. Our profiling found substantial unused GPU capacity across several omni serving workloads, with strong host-dispatch-bound evidence in the tested ASR setup. From there we ran same-GPU DP experiments on Higgs and Moss TTS models.
Experiment |
GPU signal |
Controlled observation |
Result |
Interpretation |
|---|---|---|---|---|
ASR single replica |
GPU timeline 94.3% idle |
throughput 0.90x at SM clock 0.455x; 0.31x at host CPU near 0.25x |
sensitive to CPU, not to GPU compute |
strong host-dispatch-bound causal evidence in this ASR setup |
Higgs tuned single |
SM Active about 29%, GPU idle about 71% |
throughput plateaued, worker fully driven |
1.00x normalized |
clear reclaimable GPU headroom, but not the full ASR causal closure |
Higgs DP2 without MPS |
SM Active about 37 to 38%, GPU idle about 62 to 63% |
added a second same-card server process |
about 1.24x normalized |
the second process reclaims part of the idle gap; host scheduling and long-tail batching can both contribute |
Higgs DP with MPS |
see the pinned case study in Evaluate |
each replica saturated, MPS attachment confirmed |
1.4 to 2.1x nominal, repeated |
MPS-enabled saturated runs produced the largest gains observed in the later pinned tests. |
ASR is the strongest host-bound evidence. Higgs started as a gray zone but clearly leaves GPU headroom at a tuned single replica. Running several replicas as separate processes changes host execution, scheduling, and long-tail behavior, and it is not the same as enlarging one replica’s batch. Without MPS the CUDA contexts mostly time-slice and recover only part of the idle; MPS lets kernels from different processes run concurrently when resources permit, and the later MPS-enabled saturated runs produced the largest gains observed in the pinned tests.
Common questions#
Throughput has plateaued, so why is the GPU still idle?
Serving throughput depends on more than the GPU’s peak compute. It also depends on how much parallel work a single replica exposes per step, how fast the host side handles scheduling and stage handoffs, and the request-length and batching distribution. A single Higgs replica can have a full request queue and still sit at about 29% SM Active; adding a second independent replica improves GPU idle and throughput together. So one process’s serving path is not keeping the card fed, but the cause is not a single CPU function: multiple host execution paths, batching behavior, and a latency-bounded decode shape can all contribute.
Replicating the weights costs VRAM. What does that buy?
Same-GPU DP does not save VRAM; it spends more of it. It copies the weights per replica and gives each replica its own, smaller KV pool. What it buys is the otherwise idle compute, reclaimed. That trade pays off only when a tuned single replica leaves the GPU idle (so there are idle SMs to fill) and the model is small enough that its weights are a modest slice of the card, so two or three full replicas still fit. On a compute-bound model, or one too large to hold several weight copies, extra replicas buy little. (Weight sharing over CUDA IPC relaxes the fit constraint — followers attach the leader’s copy instead of loading their own — but not the idle-compute precondition.)
Why does this pay for TTS models and not for general LLM serving?
Memory fit is the enabling condition, not the cause. The cause is idle that a single engine cannot reclaim, and TTS-style AR audio models produce it on two axes at once:
Latency-capped batch shapes. Streaming first-chunk latency pins the per-replica batch small, and a 0.6–4B talker at that batch runs low-occupancy kernels. The usual LLM remedy — batch deeper in one engine — spends the latency budget the product is built around.
Host-heavy serving path. Sampler pools, vocoder scheduling, chunk assembly, and HTTP streaming do per-step host work that rivals the GPU step time, so a single process idles the card temporally between launches. N processes overlap one replica’s dispatch bubble with another’s kernels; this is also why same-GPU DP scaling is sensitive to the CPU cores allotted per replica.
A large dense transformer inverts every part of this: its decode batch can grow until the GEMMs saturate the SM array (continuous batching in one engine already multiplexes requests over one weight copy), SM utilization is high at serving batch sizes so MPS has no idle to harvest — only contention to add — and at tens of GiB per weight copy, same-card replicas stop fitting at all. The scaling tools there are TP/PP/EP within one engine, not DP behind MPS. Rule of thumb: colocate replicas when a tuned single replica holds roughly ≤60% SM-active under its latency SLO and N× the footprint fits (weight sharing extends the fit); otherwise scale the batch, not the process count.
Reproduce the results#
We release our early results and the guidance to reproduce them below.
Prepare the baseline#
The single-replica baseline decides whether same-GPU DP is worth it, and an under-driven baseline makes DP look better than it is. Tune and measure one replica first, then treat its throughput, latency, and GPU utilization as the number every DP configuration has to beat.
Sweep concurrency to the plateau. Raise client concurrency until throughput stops climbing, and read the scheduler log lines (
#running-req,#queue-req) at each step rather than assuming a good operating point.Know the admission limit. Higgs serves with
max_running_requests=64andcuda_graph_max_bs=64by default; both can be raised viasgl-omni serve --max_running_requests N --cuda_graph_max_bs N(the CUDA-graph capture range must cover the admission limit, and raising it costs capture memory). Whether the default cap binds depends on the runtime, so check the queue, do not assume.Separate client from server. Client concurrency is not the active generation batch: requests beyond the admission limit wait in the scheduler queue, and requests also spend time in the other pipeline stages.
Prerequisites. NVIDIA CUDA MPS available with GPU compute mode
Default, so a per-user daemon needs no root; enough GPU memory for every replica’s common KV cap plus roughly fixed per-replica overhead (weights, codec, MPS context); non-overlapping CPU core blocks, one per replica, on the GPU’s NUMA node (on SMT machines logical CPUsNandN + ncoresare often the same physical core, so checklscpu -e=CPU,CORE,NODE); and enough offered concurrency to saturate each replica, not just the pool.
Evaluate#
Whether same-GPU DP helps is easy to measure incorrectly, so hold the comparison to the same discipline for every configuration:
Control |
Why it matters |
|---|---|
tune the single replica to its throughput plateau |
keeps the baseline from being artificially weak |
hold total GPU and CPU resources fixed |
separates replica splitting from simply adding resources |
give each replica dedicated CPU cores |
keeps replicas from contending for host dispatch |
saturate each replica separately |
keeps the DP pool from being under-fed |
pin software and runtime settings |
makes the comparison reproducible |
report latency and unsuccessful runs |
avoids showing only the best throughput |
Case Study on H100 with Higgs TTS Model#
One H100 80 GB (driver 580.126.20 / CUDA 13), sglang-omni a78de4cb, sglang 0.5.12.post1, bosonai/higgs-tts-3-4b (snapshot 7556c17e), /v1/audio/speech, seed-tts-eval EN, 300 samples per client, default max_running_requests=64 / cuda_graph_max_bs=64, 32 server cores of the GPU’s NUMA node split per replica, one client per replica on the SMT-sibling cores, fresh servers per run, interleaved on a shared host. Every attempted run is reported.
Configuration |
Nominal throughput |
Relative to single |
Run outcome |
|---|---|---|---|
Single c96 |
21.7 to 22.1 qps |
1.0x |
4/4 completed |
DP2 + MPS, 2 x c64 |
31.5 to 37.7 qps |
1.4 to 1.7x |
3 nominal of 5 attempts |
DP3 + MPS, 3 x c64 |
39.9 to 46.9 qps |
1.8 to 2.1x |
2 nominal and 1 degraded of 4 attempts |
The failures: one DP2 benchmark run hit cudaErrorMpsRpcFailure, and one DP2 and one DP3 replica failed to start, all coinciding with host-load spikes. One DP3 run completed every request but at 13.3 qps, so it is marked degraded rather than excluded. The core-pinned single stayed within a few percent across all runs, and DP3 was not clearly repeatably better than DP2.
Note: the MAX_TOTAL_TOKENS setting makes per-replica KV sizing more explicit and comparable. It is not a direct fix for cudaErrorMpsRpcFailure, and the launch and runtime failure rate has not been re-measured with it in place; the failures in the table reflect the runs as recorded.
The #907 profiling, this repeated case study, and the reviewer verification below are three separate measurement series. They ran on different dates and load, and in some cases different software, so they should not be compared by absolute QPS; the differences between roughly 61, 21, and 29.9 qps are not attributed to a single cause.
A separate reviewer verification on the same pinned software revision measured 29.9, 59.7, and 64.5 qps for single, DP2, and DP3. Absolute throughput differed between the two runtime environments, including different observed admission behavior, so the two series should not be combined. Both nevertheless showed a clear DP gain once every configuration was saturated.
To measure your own setup, check whether one tuned replica is below GPU saturation under your real workload before adopting DP:
nvidia-smi dmon -i $GPU_ID -s um -d 5 # coarse utilization
nsys profile --gpu-metrics-devices $GPU_ID --gpu-metrics-set gh100 \
-d 60 -o one_replica -f true sleep 63 # device-level SM-active
Low SM activity at the tuned single replica’s peak may indicate reclaimable headroom; confirm it with a controlled DP comparison before relying on it. If SM activity is already near the ceiling, stop here.
Limits and next steps#
Generality is not fully validated. Beyond the pinned H100 Higgs case study, we also ran related experiments on H200 and used SGLang to serve Qwen3-4B directly; both lines of work largely confirmed the same-GPU DP gains. Space and time limit how completely we can present those results here, and the measurements are not yet as polished as we would like. We believe same-GPU DP is a promising direction for smaller models on GPUs with ample memory and compute headroom, but the experimental coverage is still incomplete.
KV sizing is hardware- and workload-specific. The launcher enforces equal per-replica KV capacity through a common
MAX_TOTAL_TOKENS. A sizing procedure that generalizes across models, runtimes, and GPU configurations still requires further study.Router and scheduler still need a deeper dive. Both the router and the SGLang Omni scheduler need further optimization. On the router side, better routing strategies for a colocated pool are clearly required. On the scheduler side, a more ambitious question is whether we can borrow the spirit of LLM prefill–decode (PD) disaggregation: keep one large shared KV cache and let multiple replicas share it. That direction is extremely challenging, and we believe the potential payoff is correspondingly large.
Same-GPU DP with MPS can recover idle GPU time on host- or dispatch-bound serving today, but broader validation and the work above are still unfinished. If this direction interests you, or you have results from other models, GPUs, or workloads that confirm or challenge these findings, we would like to work with you.