Domino disaggregated performance findings

This page preserves the measured tuning results from the Qwen3-8B Domino online-disaggregated investigation on one 8×H200 node: one patched SGLang capture server, seven FSDP draft trainers, and an approximately 1.1B-parameter Domino/DFlash-family draft. Treat these numbers as historical evidence for that hardware, model, dataset, and software revision, not as a current-release performance claim. Rerun the benchmark before extrapolating to another setup.

All throughput values below are total samples divided by total elapsed time over at least 390 seconds. Short, approximately 20-second windows were not stable because supply was bursty and varied by roughly 1.8×.

Headline

Configuration-only tuning took the 1-server + DP7 setup from 28.8 to 50.1 samples/s (+74%). The capture server reported 98% GPU utilization and 76% SM utilization; trainer GPUs reported 77–100% GPU utilization and 76–85% SM utilization. The model and default behavior were unchanged; every setting below is opt-in.

Best measured settings

The canonical entry remains specforge train -c YAML. Repeat the same URL in the typed list to create multiple producer workers against one capture server; use distinct URLs to fan out across distinct servers:

export CLONE_ON_FETCH=0
export LOADER_PREFETCH=2
export MOONCAKE_PROTOCOL=rdma
export MOONCAKE_RDMA_DEVICES=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7

# Start the external capture server with --mem-fraction-static 0.5, then run:
specforge train -c examples/configs/qwen3-8b-domino-disaggregated.yaml \
  'deployment.disaggregated.server_urls=["http://127.0.0.1:30000","http://127.0.0.1:30000","http://127.0.0.1:30000","http://127.0.0.1:30000","http://127.0.0.1:30000","http://127.0.0.1:30000","http://127.0.0.1:30000","http://127.0.0.1:30000"]' \
  deployment.trainer.nproc_per_node=7 \
  training.batch_size=2 \
  training.accumulation_steps=8

GPU visibility and the external server’s memory fraction remain deployment settings. See the disaggregated training guide for the distinction between multiple capture servers and multiple workers driving one server.

Self-contained one-server + DP7 topology

The deleted self-contained shell’s process topology is retained as a typed, managed-local recipe. One command owns Mooncake, a capture server on GPU 0, and seven trainer ranks on GPUs 1–7:

specforge train -c \
  examples/configs/qwen3-8b-domino-1server-dp7-disaggregated.yaml

The recipe records batch size 2, accumulation 8, and capture-server memory fraction 0.5. It intentionally creates one producer worker for its one owned server. The 50.1 samples/s result above used eight workers against one external server, so use the repeated-URL command in Best measured settings when reproducing that exact historical peak. CLONE_ON_FETCH=0, LOADER_PREFETCH=2, and hardware-specific RDMA settings remain opt-in runtime environment controls.

What each setting does

Setting	Effect	Mechanism
Repeat one URL eight times in deployment.disaggregated.server_urls	Breaks the single-producer ceiling	Eight rollout workers take disjoint leases and issue blocking HTTP prefill calls concurrently to one server.
training.accumulation_steps=8	460 to approximately 280 ms/microstep	FSDP no_sync amortizes reduce-scatter across eight microsteps; communication fell to about 1 ms/microstep.
CLONE_ON_FETCH=0	Approximately 15 ms/batch lower	Mooncake get() already allocates a fresh tensor, so the defensive clone is redundant on this path.
LOADER_PREFETCH=2	Removes fetch latency from the measured train step	A background thread materializes up to two batches before the trainer consumes them.
External SGLang --mem-fraction-static 0.5	Server memory fell from 126 GB to about 78 GB	The default 0.85 reservation held KV-cache memory that this capture-only server did not need.

The server-side capture patch also kept hidden-state slices on GPU and used one torch.cat plus one D2H copy per request instead of a per-prefill-batch unpinned copy. In this run, capture D2H fell from 5–8 ms/sample to approximately 3.8 ms/sample.

The pipeline is a supply/demand seesaw

At 50 samples/s the system was supply-bound: one capture server only just fed seven trainers, loader waits were approximately 40 ms/batch, and producer in-flight depth stayed low. The GPU split was therefore a real trade-off:

Split	Throughput	Regime
1 server + 7 trainers	50.1/s	Supply-bound; best measured split
2 servers + 6 trainers, batch 2	39.0/s	Demand-bound; only six trainers
2 servers + 6 trainers, batch 8	39.5/s	Demand-bound; larger batch did not help

Each trainer GPU consumed approximately 7 samples/s, while one server supplied approximately 52–57 samples/s. Seven trainers, about 50 samples/s of demand, were almost exactly balanced by one server. Trading away a trainer for a second server lost more demand than it added supply on this eight-GPU allocation.

Bigger batches did not help

Batch	Accumulation	Throughput	Trainer memory	Trainer utilization
2	8	50.1/s	45 GB (about 31%)	77–100%
4	4	47.5/s	65 GB (about 45%)	64–99%
8	2	47.2/s	113 GB (about 79%)	100%

Memory climbed with batch size, but throughput fell. The larger batch made each fetch roughly four times heavier (get_ms rose from about 20 to 90) while per-sample compute stayed flat. Low trainer memory in the best configuration was efficient rather than evidence of a stall: the trainer was compute-bound at approximately 44% measured MFU.

Why per-sample trainer demand is low

The benchmark’s synchronized step measurements at batch size two were:

num_anchors	Forward	Backward	Optimizer	Data wait
256, recipe default	approximately 88 ms	approximately 150 ms	1.3 ms	approximately 40 ms (14%)
64	approximately 43 ms	approximately 58 ms	1.3 ms	approximately 145 ms (38%), supply-starved

Domino/DFlash training expands every sample to num_anchors × block_size: 256 × 16 = 4096 draft positions independent of input sequence length. Each position passes through the five-layer draft and a full 151,936-vocabulary head. A fit of these measurements gives approximately 53 ms fixed work plus 0.73 ms per anchor, so the anchor expansion accounts for roughly 78% of the 240 ms microstep at 256 anchors.

Reducing anchors is therefore not a free throughput optimization. At 256 anchors the run delivered approximately 2,130 anchor updates/s, versus about 1,280 at 64 anchors. Fewer anchors raised sequences/s but reduced training signal/s by roughly 40%, while putting more pressure on the already supply-bound capture server. Treat num_anchors as a quality and data-efficiency choice validated against acceptance length; the checked-in recipe keeps 256.

MFU and FLOPs

The isolated trainer microbenchmark used bench_domino_mfu.py, BF16, real Qwen3-8B Domino shapes, num_anchors=256, sequence length 768, CUDA event timing, and torch.utils.flop_counter:

Batch	Step	Per sample	FLOP/sample, forward + backward	Achieved	MFU	Peak memory
2	211 ms	105.6 ms	approximately 45 T	430 TFLOP/s	43.5%	25 GB
4	416 ms	103.9 ms	approximately 45 T	437 TFLOP/s	44.1%	46 GB
8	836 ms	104.6 ms	approximately 45 T	434 TFLOP/s	43.9%	87 GB

The approximately 1.1B draft was compute-bound, not stalled. Per-sample time was flat across batch sizes, and memory grew by roughly 10 GB/sample. One sample cost approximately 45 TFLOP for forward plus backward—about nine times a normal 1.1B forward—because of the 256-anchor expansion and two full-vocabulary paths.

Target prefill was estimated at a similar 43% MFU. At 50 samples/s the server logged approximately 27,000 tokens/s over 1,080 batches, with an average prompt of about 550 tokens. For the 8B target, 2 × parameters × tokens/s is about 430 TFLOP/s. This estimate excludes attention FLOPs and the five auxiliary capture layers.

Remaining ceilings

1. Supply at approximately 50 samples/s was prefill-compute-bound, not sink-bound. With at least eight producer workers, target prefill ran nearly continuously at about 27,000 tokens/s. The Mooncake sink sustained roughly 44–57 samples/s and was the next ceiling, not the measured one. Potential supply improvements include higher prefill MFU, larger prefill batches, and lighter auxiliary-state extraction/D2H; a second inference GPU was worse on this fixed eight-GPU split.

2. Trainer demand at approximately 50 samples/s was also compute-bound near 44% MFU. Raising trainer MFU would reduce trainer GPU cost, but could not improve end-to-end throughput until capture supply increased. Reducing num_anchors would change training signal and is not an equivalent systems optimization.

Profiling controls

Production runs expose the typed profiling configuration (enabled, start_step, num_steps, and record_shapes) for bounded PyTorch traces on trainer ranks. The component timers used to collect this benchmark were temporary measurement instrumentation and are not runtime environment APIs. FSDP_SHARDING=NO_SHARD remains available for controlled sharding comparisons.