Installation — NPU#

Prerequisites#

Both installation methods require Ascend host drivers and firmware compatible with your hardware and selected CANN release. Follow the HDK driver and firmware installation guide and the Ascend software compatibility guide to select and install matching versions. For Docker, match the CANN version in the image tag.

Verify the host installation before continuing:

npu-smi info

Option A: Docker (recommended)#

The ARM64 image includes SGLang-Omni source, the SGLang/CANN/PyTorch runtime selected by pyproject_npu.toml, and common audio dependencies. Model weights, host drivers and TorchCodec are not included. Model support on NPU and optional dependencies must be checked separately.

A3 image installation#

Use an ARM64 Linux host with Docker and Ascend A3 devices.

1. Pull the image

Pull the image matching your hardware from Huawei Cloud SWR:

Release: use v<version>-cann9.0.0-a3 for deployments, replacing v<version> with a published version tag.
Development: use main-cann9.0.0-a3 for the latest published development build. Scheduled or manual publications update this rolling tag.

# Release (replace v<version> with a published version):
# IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni:v<version>-cann9.0.0-a3

# Development (A3):
IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni:main-cann9.0.0-a3

docker pull "$IMAGE"

For reproducible deployments, set IMAGE to swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni@sha256:<digest>.

If authentication is required, run docker login swr.cn-southwest-2.myhuaweicloud.com with credentials supplied by the registry administrator before pulling.

2. Start the container

Download your model weights into a host directory and mount it into the container. The example exposes device 0; adjust device IDs and driver paths for your host. --privileged grants broad host access and is only suitable for a trusted host; production deployments should use restricted device access.

MODEL_DIR=<your_own_path>
docker run --rm -it --name omni-npu \
  --privileged --network host --shm-size 8g \
  --device /dev/davinci0 \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm --device /dev/hisi_hdc \
  -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
  -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware:ro \
  -v /usr/local/sbin:/usr/local/sbin:ro \
  -v /etc/ascend_install.info:/etc/ascend_install.info:ro \
  -v "$MODEL_DIR:/models:ro" \
  -e ASCEND_RT_VISIBLE_DEVICES=0 \
  -e OMP_NUM_THREADS=8 \
  "$IMAGE" bash

# Inside the container:
source /usr/local/Ascend/ascend-toolkit/set_env.sh

When selecting another device, update both /dev/davinci0 and ASCEND_RT_VISIBLE_DEVICES. Device indices in the pipeline configuration are relative to the visible devices (gpu: 0 selects the first visible device).

A2 image installation details (click)

Use an ARM64 Linux host with Docker and Ascend A2 / 910B devices.

1. Pull the image

Use v<version>-cann9.0.0-910b for a release, or main-cann9.0.0-910b for the latest published development build.

# Release (replace v<version> with a published version):
# IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni:v<version>-cann9.0.0-910b

# Development (A2 / 910B):
IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni:main-cann9.0.0-910b

docker pull "$IMAGE"

For reproducible deployments, set IMAGE to swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni@sha256:<digest>.

2. Start the container

Set MODEL_DIR to your model directory. The example exposes device 0; adjust device IDs and driver paths for your host. Use --privileged only on a trusted host; production deployments should use restricted device access.

MODEL_DIR=<your_own_path>
docker run --rm -it --name omni-npu \
  --privileged --network host --shm-size 8g \
  --device /dev/davinci0 \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm --device /dev/hisi_hdc \
  -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
  -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware:ro \
  -v /usr/local/sbin:/usr/local/sbin:ro \
  -v /etc/ascend_install.info:/etc/ascend_install.info:ro \
  -v "$MODEL_DIR:/models:ro" \
  -e ASCEND_RT_VISIBLE_DEVICES=0 \
  -e OMP_NUM_THREADS=8 \
  "$IMAGE" bash

# Inside the container:
source /usr/local/Ascend/ascend-toolkit/set_env.sh

When selecting another device, update both /dev/davinci0 and ASCEND_RT_VISIBLE_DEVICES. Pipeline device indices are relative to the visible devices (gpu: 0 selects the first visible device).

Launch the server#

The runtime is already installed. Skip the source installation below and do not run the TorchCodec stack-upgrade script inside this image. Qwen3-TTS dependencies are included; skip its cookbook’s dependency installation. Continue with the API Server Quickstart to launch sgl-omni serve and send requests. Use the container model path under /models and pass the selected model’s NPU pipeline configuration with --config when required. Model-specific inputs belong in the corresponding cookbook, such as the Qwen3-TTS guide; GPU-specific configurations there should not be assumed to work unchanged on NPU.

Option B: Manual install#

Install the Ascend software stack and NPU build of SGLang before installing sglang-omni. The helper script install_npu.sh installs only sglang-omni and the Python dependencies declared in pyproject_npu.toml, using the same installation flow as Docker. Prepare FFmpeg, libsndfile and SoX separately, or pass --install-system-deps as root. The Ascend software stack and SGLang must already be installed.

Prerequisites#

Select mutually compatible versions for your Ascend hardware by following the linked documentation. Python 3.11 is the verified configuration.

Component	Version	Required	Manual installation	Installation
CANN toolkit	Compatible release	Yes	Yes	Official documentation
PyTorch and `torch_npu`	Matching releases	Yes	Yes	Official documentation
`triton-ascend`	Match the selected PyTorch and CANN releases	Yes	Yes	Official documentation
`sgl-kernel-npu`	Match PyTorch, Python, CANN, hardware, and architecture	Yes	Yes	Official documentation
`memfabric-hybrid`	Compatible release	No (PD disaggregation only)	Yes	Official documentation
SGLang for NPU	`sglang-version` in `pyproject_npu.toml`	Yes	Yes	Official documentation

Install sglang-omni#

git clone https://github.com/sgl-project/sglang-omni.git
cd sglang-omni
source /usr/local/Ascend/ascend-toolkit/set_env.sh

# Check the environment and show the installation command without changing files.
bash scripts/npu/install_npu.sh --check

# Install sglang-omni in editable mode.
bash scripts/npu/install_npu.sh

The precheck reads the supported SGLang release from [tool.sglang-omni.npu] in pyproject_npu.toml before swapping the install manifest. This includes development, pre-release, post-release and local builds of that release line. To print the configured release (Python 3.11+, or Python 3.10 with tomli installed), run python scripts/npu/config.py --get sglang-version. It rejects other release lines, including later releases. On a mismatch it reports both the supported line and the installed version. The precheck also verifies the required Python packages, matching torch and torch_npu major-minor versions, NPU availability, and a small NPU matrix multiplication. Run bash scripts/npu/install_npu.sh --help for optional extras, non-editable installation, and environments where devices are intentionally not exposed during the build.

TorchCodec installation for TTS models#

Models in the TTS models utilize torchcodec for high-efficiency, native-streaming audio decoding directly into PyTorch tensors.

The helper script scripts/npu/install_npu_torchcodec.sh automatically installs:

Audio codec: ffmpeg
CANN 9.1.0 stack: toolkit, A3-ops, nnal
PyTorch 2.11 stack: torch, torchvision, torchaudio, torch_npu, torchcodec

To run the installation:

# Specify your device type as the first argument (910b or A3)
bash scripts/npu/install_npu_torchcodec.sh A3

Qwen3-ASR graph execution#

Qwen3-ASR pre-LM encoding and language-model execution submit work on separate NPU streams. With encoder, decode, or prefill graphs enabled, a shared Host task queue can couple those submission paths and cause stalls. Per-stream Host queues let each stream submit work independently; they do not replace device-stream synchronization.

For normal pipeline launches, PER_STREAM_QUEUE=1 is set by default before spawning the worker process. Leave TASK_QUEUE_ENABLE unset (its runtime default is 1) or explicitly set it to 1, and leave ASCEND_LAUNCH_BLOCKING unset or set it to 0 to retain asynchronous launches. Existing shell and pipeline environment settings take precedence over the default.

When pre-LM encoding and any encoder, decode, or prefill graph backend are enabled on NPU, startup rejects incompatible queue or launch-blocking settings instead of silently overriding them. This check does not apply when pre-LM encoding is disabled or all three graph paths are disabled. For direct engine construction outside the pipeline launcher, configure these variables before initializing the NPU runtime; the engine validates them but does not supply the pipeline’s default.

Use a torch_npu build that supports per-stream Host task queues. The feature is experimental and does not support process-level online recovery. Although per-stream queues support TaskQueue modes 1 and 2, NPU graph capture rejects mode 2; this graph configuration therefore requires mode 1. Mode 0 disables the per-stream queues required by this path. Additional queue threads and cross-stream event ordering may add overhead. See the TorchNPU stream-level TaskQueue documentation for runtime constraints.

Model notes#

MiniCPM-o#

MiniCPM-o runs on Ascend 910 with its NPU pipeline configuration:

python -m sglang_omni.cli serve \
  --model-path openbmb/MiniCPM-o-4_5 \
  --config examples/configs/minicpm_o_npu.yaml \
  --port 8000

The configuration uses SGLang’s ascend attention backend, keeps AR decode graphs enabled, and runs the image and audio encoders on their own NPU stream. On NPU the reference audio for voice cloning is decoded through the soundfile fallback because torchcodec is not usable in NPU images. Validated on Ascend 910 / CANN 9.1.0 / torch 2.10 / torch_npu 2.10.0.post6 / SGLang 0.5.22.

Installation — NPU

Contents

Installation — NPU#

Prerequisites#

Option A: Docker (recommended)#

A3 image installation#

Launch the server#

Option B: Manual install#

Prerequisites#

Install sglang-omni#

TorchCodec installation for TTS models#

Qwen3-ASR graph execution#

Model notes#

MiniCPM-o#