# Installation — NPU Install SGLang-Omni on ARM64 Linux with Ascend A3 or A2 / 910B devices. Choose the Docker image for your device or install into an existing Ascend environment. ## Prerequisites Both installation methods require Ascend host drivers and firmware compatible with your hardware and selected CANN release. Follow the [HDK driver and firmware installation guide](https://www.hiascend.com/hardware/firmware-drivers/community) and the [Ascend software compatibility guide](https://docs.sglang.io/docs/hardware-platforms/ascend-npus/ascend_npu) to select and install matching versions. For Docker, match the CANN version in the image tag. Verify the host installation before continuing: ```bash npu-smi info ``` ## Docker (recommended) The ARM64 image includes SGLang-Omni source, the SGLang/CANN/PyTorch runtime selected by [`pyproject_npu.toml`](https://github.com/sgl-project/sglang-omni/blob/main/pyproject_npu.toml), and common audio dependencies. Model weights, host drivers and TorchCodec are not included. Model support on NPU and optional dependencies must be checked separately. ### A3 image installation Use an ARM64 Linux host with Docker and Ascend A3 devices. **1. Pull the image** Pull the image matching your hardware from Huawei Cloud SWR: - **Release:** use `v-cann9.0.0-a3` for deployments, replacing `v` with a published version tag. - **Development:** use `main-cann9.0.0-a3` for the latest published development build. Scheduled or manual publications update this rolling tag. ```bash # Release (replace v with a published version): # IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni:v-cann9.0.0-a3 # Development (A3): IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni:main-cann9.0.0-a3 docker pull "$IMAGE" ``` For reproducible deployments, set `IMAGE` to `swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni@sha256:`. If authentication is required, run `docker login swr.cn-southwest-2.myhuaweicloud.com` with credentials supplied by the registry administrator before pulling. **2. Start the container** Download your model weights into a host directory and mount it into the container. The example exposes device 0; adjust device IDs and driver paths for your host. `--privileged` grants broad host access and is only suitable for a trusted host; production deployments should use restricted device access. ```bash MODEL_DIR= docker run --rm -it --name omni-npu \ --privileged --network host --shm-size 8g \ --device /dev/davinci0 \ --device /dev/davinci_manager \ --device /dev/devmm_svm --device /dev/hisi_hdc \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware:ro \ -v /usr/local/sbin:/usr/local/sbin:ro \ -v /etc/ascend_install.info:/etc/ascend_install.info:ro \ -v "$MODEL_DIR:/models:ro" \ -e ASCEND_RT_VISIBLE_DEVICES=0 \ -e OMP_NUM_THREADS=8 \ "$IMAGE" bash # Inside the container: source /usr/local/Ascend/ascend-toolkit/set_env.sh ``` When selecting another device, update both `/dev/davinci0` and `ASCEND_RT_VISIBLE_DEVICES`. Device indices in the pipeline configuration are relative to the visible devices (`gpu: 0` selects the first visible device). ### A2 / 910B image installation Use an ARM64 Linux host with Docker and Ascend A2 / 910B devices. **1. Pull the image** Use `v-cann9.0.0-910b` for a release, or `main-cann9.0.0-910b` for the latest published development build. ```bash # Release (replace v with a published version): # IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni:v-cann9.0.0-910b # Development (A2 / 910B): IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni:main-cann9.0.0-910b docker pull "$IMAGE" ``` For reproducible deployments, set `IMAGE` to `swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang-omni@sha256:`. **2. Start the container** Set `MODEL_DIR` to your model directory. The example exposes device 0; adjust device IDs and driver paths for your host. Use `--privileged` only on a trusted host; production deployments should use restricted device access. ```bash MODEL_DIR= docker run --rm -it --name omni-npu \ --privileged --network host --shm-size 8g \ --device /dev/davinci0 \ --device /dev/davinci_manager \ --device /dev/devmm_svm --device /dev/hisi_hdc \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware:ro \ -v /usr/local/sbin:/usr/local/sbin:ro \ -v /etc/ascend_install.info:/etc/ascend_install.info:ro \ -v "$MODEL_DIR:/models:ro" \ -e ASCEND_RT_VISIBLE_DEVICES=0 \ -e OMP_NUM_THREADS=8 \ "$IMAGE" bash # Inside the container: source /usr/local/Ascend/ascend-toolkit/set_env.sh ``` When selecting another device, update both `/dev/davinci0` and `ASCEND_RT_VISIBLE_DEVICES`. Pipeline device indices are relative to the visible devices (`gpu: 0` selects the first visible device). ## Manual install Install the Ascend software stack and NPU build of SGLang before installing `sglang-omni`. The helper script [`install_npu.sh`](https://github.com/sgl-project/sglang-omni/blob/main/scripts/npu/install_npu.sh) installs only `sglang-omni` and the Python dependencies declared in [`pyproject_npu.toml`](https://github.com/sgl-project/sglang-omni/blob/main/pyproject_npu.toml), using the same installation flow as Docker. Prepare FFmpeg, libsndfile and SoX separately, or pass `--install-system-deps` as root. The Ascend software stack and SGLang must already be installed. ### Manual installation prerequisites Select mutually compatible versions for your Ascend hardware by following the linked documentation. Python 3.11 is the verified configuration. | Component | Version | Required | Manual installation | Installation | |-----------|---------|----------|---------------------|--------------| | CANN toolkit | Compatible release | Yes | Yes | [Official documentation](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/900/softwareinst/instg/instg_0008.html) | | PyTorch and `torch_npu` | Matching releases | Yes | Yes | [Official documentation](https://www.hiascend.com/developer/software/ai-frameworks/pytorch/download?versionId=177&ids=89dda9ba9de741349efa03687a487678%2C204%2C200%2C1%2C6%2C177%2C) | | `triton-ascend` | Match the selected PyTorch and CANN releases | Yes | Yes | [Official documentation](https://gitcode.com/Ascend/triton-ascend/blob/main/docs/en/quick_start.md) | | `sgl-kernel-npu` | Match PyTorch, Python, CANN, hardware, and architecture | Yes | Yes | [Official documentation](https://github.com/sgl-project/sgl-kernel-npu/releases) | | `memfabric-hybrid` | Compatible release | No (PD disaggregation only) | Yes | [Official documentation](https://docs.sglang.io/docs/hardware-platforms/ascend-npus/ascend_npu) | | SGLang for NPU | `sglang-version` in [`pyproject_npu.toml`](https://github.com/sgl-project/sglang-omni/blob/main/pyproject_npu.toml) | Yes | Yes | [Official documentation](https://docs.sglang.io/docs/hardware-platforms/ascend-npus/ascend_npu) | ### Install sglang-omni ```bash git clone https://github.com/sgl-project/sglang-omni.git cd sglang-omni source /usr/local/Ascend/ascend-toolkit/set_env.sh # Check the environment and show the installation command without changing files. bash scripts/npu/install_npu.sh --check # Install sglang-omni in editable mode. bash scripts/npu/install_npu.sh ``` The precheck reads the supported SGLang release from `[tool.sglang-omni.npu]` in `pyproject_npu.toml` before swapping the install manifest. This includes development, pre-release, post-release and local builds of that release line. To print the configured release (Python 3.11+, or Python 3.10 with `tomli` installed), run `python scripts/npu/config.py --get sglang-version`. It rejects other release lines, including later releases. On a mismatch it reports both the supported line and the installed version. The precheck also verifies the required Python packages, matching `torch` and `torch_npu` major-minor versions, NPU availability, and a small NPU matrix multiplication. Run `bash scripts/npu/install_npu.sh --help` for optional extras, non-editable installation, and environments where devices are intentionally not exposed during the build. ## Verify the installation Run these checks inside the container or the manual installation environment: ```bash npu-smi info python -c 'import sglang_omni, torch, torch_npu; print(torch.npu.is_available())' sgl-omni --help ``` Check that `npu-smi` lists your device, the import prints `True`, and the CLI prints its help. ### Launch the server Use the NPU configuration for your model. For Docker, use the container model path under `/models`. Run `sgl-omni serve --help` for CLI flags and see the [model notes](#model-notes) below for NPU examples and requirements. ## Qwen3-ASR graph execution Qwen3-ASR pre-LM encoding and language-model execution submit work on separate NPU streams. With encoder, decode, or prefill graphs enabled, a shared Host task queue can couple those submission paths and cause stalls. Per-stream Host queues let each stream submit work independently; they do not replace device-stream synchronization. For normal pipeline launches, `PER_STREAM_QUEUE=1` is set by default before spawning the worker process. Leave `TASK_QUEUE_ENABLE` unset (its runtime default is `1`) or explicitly set it to `1`, and leave `ASCEND_LAUNCH_BLOCKING` unset or set it to `0` to retain asynchronous launches. Existing shell and pipeline environment settings take precedence over the default. When pre-LM encoding and any encoder, decode, or prefill graph backend are enabled on NPU, startup rejects incompatible queue or launch-blocking settings instead of silently overriding them. This check does not apply when pre-LM encoding is disabled or all three graph paths are disabled. For direct engine construction outside the pipeline launcher, configure these variables before initializing the NPU runtime; the engine validates them but does not supply the pipeline's default. Use a torch_npu build that supports per-stream Host task queues. The feature is experimental and does not support process-level online recovery. Although per-stream queues support TaskQueue modes `1` and `2`, NPU graph capture rejects mode `2`; this graph configuration therefore requires mode `1`. Mode `0` disables the per-stream queues required by this path. Additional queue threads and cross-stream event ordering may add overhead. See the [TorchNPU stream-level TaskQueue documentation](https://www.hiascend.com/document/detail/en/Pytorch/2610/devguide/fwfeatures/docs/en/framework_feature_guide_pytorch/stream_taskqueue_parallel_delivery.md) for runtime constraints. ## Model notes ### TorchCodec installation for TTS models Some TTS models use `torchcodec` for audio decoding. The NPU Docker images already include the Qwen3-TTS dependencies; skip the Cookbook dependency installation and do not run this stack upgrade inside those containers. The helper script `scripts/npu/install_npu_torchcodec.sh` automatically installs: * **Audio codec:** `ffmpeg` * **CANN 9.1.0 stack:** `toolkit`, `A3-ops`, `nnal` * **PyTorch 2.11 stack:** `torch`, `torchvision`, `torchaudio`, `torch_npu`, `torchcodec` To run the installation: ```bash # Specify your device type as the first argument (910b or A3) bash scripts/npu/install_npu_torchcodec.sh A3 ``` ### MiniCPM-o MiniCPM-o runs on Ascend 910 with its NPU pipeline configuration: ```bash python -m sglang_omni.cli serve \ --model-path openbmb/MiniCPM-o-4_5 \ --config examples/configs/minicpm_o_npu.yaml \ --port 8000 ``` The configuration uses SGLang's `ascend` attention backend, keeps AR decode graphs enabled, and runs the image and audio encoders on their own NPU stream. On NPU the reference audio for voice cloning is decoded through the soundfile fallback because torchcodec is not usable in NPU images. Validated on Ascend 910 / CANN 9.1.0 / torch 2.10 / torch_npu 2.10.0.post6 / SGLang 0.5.22.