MiniCPM-o#

MiniCPM-o 4.5 is a Gemini 2.5 Flash level model for vision, speech, and full-duplex live streaming. SGLang-Omni serves it in two ways:

Mode	Endpoint	Use it for
Chat	`/v1/chat/completions`	One request, one reply, in text or speech
Full duplex	`/v1/realtime` (WebSocket)	A live voice or video call where the model listens and speaks at the same time

Prerequisites#

Follow Installation, then run from the repository root.

For Intel GPUs, follow the MiniCPM-o XPU recipe.

Chat with speech output:

python -m sglang_omni.cli serve --model-path openbmb/MiniCPM-o-4_5 --port 8000

Chat with text output only:

python -m sglang_omni.cli serve --model-path openbmb/MiniCPM-o-4_5 --text-only --port 8000

Ascend NPU#

See Installation — NPU for the NPU launch command, configuration, and validated versions.

Normal Usage#

import base64
from pathlib import Path

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
reference = base64.b64encode(Path("docs/_static/audio/male-voice.wav").read_bytes()).decode("ascii")
response = client.chat.completions.create(
    model="MiniCPM-o-4_5",
    messages=[{"role": "user", "content": "Please say hello."}],
    modalities=["text", "audio"],
    audio={
        "format": "wav",
        "ref_audio": f"data:audio/wav;base64,{reference}",
    },
)
message = response.choices[0].message
Path("reply.wav").write_bytes(base64.b64decode(message.audio.data))
print(message.audio.transcript or message.content)

Video Chat Inputs#

Chat accepts inline video_url content and top-level video or videos fields. Top-level requests without use_audio_in_video retain the existing shared video sampling and resizing policy. They place explicit images, video frames, and independent audio before the final user message’s text, in that order. Predecoded video tensors remain supported, including mixed file/tensor inputs.

Top-level videos without use_audio_in_video decode on parallel decoders. Each of the four preprocessing workers gets min(16, the stage’s CPUs divided by four) FFmpeg threads, split evenly among a request’s videos. Set --preprocessing.factory.video_decode_threads N to choose that count. A value at or below the video reader’s own threads (TORCHCODEC_NUM_THREADS, 8 by default) decodes each video on the reader alone.

Inline videos and top-level requests with use_audio_in_video use MiniCPM timed video sampling. Set use_audio on an inline video, or use_audio_in_video on a top-level request, to include interleaved audio from file or URL sources. Top-level interleaved videos follow explicit images and independent audio.

Timed video frames retain native resolution by default; the MiniCPM image processor performs the final resize. video_max_frames limits the sampled frame count, while video_max_pixels and video_total_pixels resize frames after decoding. Those pixel options do not bound the initial native-resolution allocation.

Native Full-Duplex Ability#

MiniCPM-o 4.5 also serves live full-duplex voice and video calls on /v1/realtime. Launching, settings, the browser demo and the protocol are in Full Duplex Interactive Model.

Text to speech#

/v1/audio/speech reads the given text aloud. MiniCPM-o prefills the text in one Thinker pass and conditions the Talker on each of its tokens, instead of generating the same words one token at a time. Set language to Chinese for Chinese text, and pass a base64 audio data URI as ref_audio to clone a voice.

audio = client.audio.speech.create(
    model="MiniCPM-o-4_5",
    voice="default",
    input="Hello!",
    extra_body={"language": "English"},
)

The speech output is non-streaming. The sampling fields you set, such as temperature, top_p and max_new_tokens, apply to the Talker, which generates the speech; the rest keep the Talker’s defaults.

MiniCPM-o

Contents