DeepSeek-V4-Flash DSpark disaggregated training

Trains the DSpark drafter in configs/deepseek-v4-flash-dspark.json (a five-layer Qwen3-style GQA decoder at the target's width) for deepseek-ai/DeepSeek-V4-Flash-0731, from scratch on ShareGPT. The recipe fits one 8-GPU B200 node: two TP2 capture servers on GPUs 0-3 and a four-rank trainer on GPUs 4-7 (global batch 128; ~4.6 s/step measured). A two-node split only changes the endpoints and CUDA_VISIBLE_DEVICES.

The flow is the standard disaggregated setup (docs/basic_usage/disaggregated_training.md); what is DeepSeek-V4-specific:

• Capture hook: SGLang v0.5.18 patched with scripts/apply_sglang_spec_capture_patch.sh --target v0.5.18, launched with --spec-capture-method dspark. The generic DFlash capture is not equivalent for V4: the inter-layer residual is the widened mHC tensor, so capture must use the model's own set_dspark_layers_to_capture hook.
• Chat encoding: the checkpoint ships a Python encoder rather than tokenizer Jinja. The dedicated DeepSeekV4Parser uses a vendored copy of the official encoder, pinned to the model release revision, for the recipe's ShareGPT data.
• B200 MoE: --moe-runner-backend flashinfer_mxfp4 (the routed experts are fp4; the default MoE path cannot run them), with FLASHINFER_USE_CUDA_NORM=1 FLASHINFER_USE_CUDA_QUANT=1 exported to work around CuTe-DSL kernel miscompiles on some nvidia-cutlass-dsl pins.

Data

python scripts/prepare_data.py --dataset sharegpt writes cache/dataset/sharegpt_train.jsonl, which the recipe points at.

Capture servers (GPUs 0-3)

mooncake_master \
  --enable_http_metadata_server=true \
  --http_metadata_server_host=0.0.0.0 \
  --rpc_port=35551 \
  --http_metadata_server_port=35880 \
  --metrics_port=35903 \
  --default_kv_lease_ttl=5m

--default_kv_lease_ttl=5m and MC_TRANSFER_TIMEOUT=300 (on server and trainer) are required for multi-hundred-MB feature objects; the Kimi K3 runbook explains the failure modes. Then one server per GPU pair (repeat with CUDA_VISIBLE_DEVICES=2,3 --port 30001; for a two-node run replace 127.0.0.1 with the capture node's routable IP):

export MOONCAKE_MASTER_SERVER_ADDR=127.0.0.1:35551
export MOONCAKE_METADATA_SERVER=http://127.0.0.1:35880/metadata
export MOONCAKE_LOCAL_HOSTNAME=127.0.0.1
export MOONCAKE_PROTOCOL=tcp
export MC_TRANSFER_TIMEOUT=300
export MOONCAKE_GLOBAL_SEGMENT_SIZE=$((200 << 30))
export MOONCAKE_LOCAL_BUFFER_SIZE=$((1 << 30))
export FLASHINFER_USE_CUDA_NORM=1
export FLASHINFER_USE_CUDA_QUANT=1
CUDA_VISIBLE_DEVICES=0,1 python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 30000 \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code \
  --skip-tokenizer-init \
  --tp-size 2 \
  --mem-fraction-static 0.88 \
  --context-length 8704 \
  --max-running-requests 8 \
  --chunked-prefill-size -1 \
  --max-prefill-tokens 8704 \
  --moe-runner-backend flashinfer_mxfp4 \
  --enable-spec-capture \
  --spec-capture-method dspark \
  --spec-capture-aux-layer-ids 1 11 21 31 41

Trainer (GPUs 4-7)

CUDA_VISIBLE_DEVICES=4,5,6,7 specforge train \
  -c examples/configs/online/disaggregated/external/deepseek-v4-flash-dspark-disaggregated.yaml

Supply HF_TOKEN and WANDB_API_KEY through the environment, not YAML.

On AMD MI355X

Use deepseek-v4-flash-dspark-disaggregated-amd.yaml, this recipe with the capture-server fields changed for ROCm, inside the lmsysorg/sglang:v0.5.18-rocm720-mi35x container from the AMD ROCm tutorial. Everything above applies, with three changes to each capture-server command:

• drop FLASHINFER_USE_CUDA_NORM=1 and FLASHINFER_USE_CUDA_QUANT=1;
• add export AITER_BF16_FP8_MOE_BOUND=0, the setting SGLang's own AMD DeepSeek-V4 tests use; without it the server fails on the first MoE batch under 256 tokens;
• replace --moe-runner-backend flashinfer_mxfp4 with --attention-backend dsv4 --page-size 256 --swa-full-tokens-ratio 0.15 --disable-radix-cache.

Set HIP_VISIBLE_DEVICES alongside CUDA_VISIBLE_DEVICES. Measured on one MI355X node over the full two epochs (1,885 optimizer steps): 6.32 s per 128-sample step on average, 6.1-6.4 s at steady state (about 3.1 s waiting for capture and 3.0 s of trainer compute), 3 h 18 min end to end.

Fresh attempts

Delete the run's outputs/ directory and, whenever a capture server was restarted, use a fresh Mooncake namespace (restart mooncake_master, or override deployment.disaggregated.store_id=<unique-attempt-id>): the producer dedups against keys already registered in the master, and keys whose replicas lived in a dead server's segment poison the consumer with get_into failed errors.