Fun-CosyVoice3#

Fun-CosyVoice3-0.5B is a lightweight text-to-speech model (0.5B parameters) from the FunAudioLLM team at Alibaba. It uses a Qwen2.5-0.5B backbone with FSQ speech tokens (vocab = 6561 + 200 special tokens), conditioned on a CAMPPlus speaker embedding and prompt speech tokens extracted via an ONNX speech tokenizer. It supports zero-shot voice cloning, cross-lingual synthesis, and instruction-based style control. The model produces 24 kHz speech at a rate of 25 Hz token frame rate through the preprocessing → tts_engine → vocoder pipeline and the OpenAI-compatible /v1/audio/speech endpoint.

Prerequisites#

Install sglang-omni from source as in Installation.

Fun-CosyVoice3 needs sox and a few extra Python packages. From the repository root, install the extra against this checkout:

apt-get update && apt-get install -y sox
uv pip install -e ".[fun-cosyvoice3]"

Clone the CosyVoice repository with its Matcha-TTS submodule and add both to PYTHONPATH:

COSYVOICE_PATH=/path/to/CosyVoice
COSYVOICE_COMMIT=074ca6dc9e80a2f424f1f74b48bdd7d3fea531cc
MATCHA_TTS_COMMIT=dd9105b34bf2be2230f4aa1e4769fb586a3c824e

git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git ${COSYVOICE_PATH}
git -C ${COSYVOICE_PATH} checkout ${COSYVOICE_COMMIT}
git -C ${COSYVOICE_PATH} submodule update --init --recursive
git -C ${COSYVOICE_PATH}/third_party/Matcha-TTS checkout ${MATCHA_TTS_COMMIT}
export PYTHONPATH="${COSYVOICE_PATH}:${COSYVOICE_PATH}/third_party/Matcha-TTS:$PYTHONPATH"

Do not run pip install -r requirements.txt from the CosyVoice checkout. That file pins torch, torchaudio, transformers, and diffusers versions that conflict with the sglang-omni core pins. Only the fun-cosyvoice3 extra above and the two PYTHONPATH entries are needed; the CosyVoice Flow and HiFT modules import fine against the sglang-omni versions of those shared packages.

The checkpoint includes ONNX models for the speech tokenizer and speaker encoder, which use the onnxruntime already pinned in sglang-omni’s core dependencies.

Download the checkpoint:

hf download FunAudioLLM/Fun-CosyVoice3-0.5B-2512

The pipeline is preprocessing → tts_engine → vocoder. First startup can take several minutes while the tts_engine captures CUDA graphs.

sgl-omni serve \
  --model-path FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
  --port 8000

Apple Silicon#

On Apple Silicon, install the optional Fun-CosyVoice3 extra with the repository installer, then expose Homebrew’s keg-only FFmpeg libraries to TorchCodec:

brew install sox
SGLANG_OMNI_EXTRAS=fun-cosyvoice3 ./install.sh
source .venv-apple/bin/activate
export DYLD_LIBRARY_PATH="$(brew --prefix ffmpeg@7)/lib${DYLD_LIBRARY_PATH:+:$DYLD_LIBRARY_PATH}"

The model runs through PyTorch MPS with the official checkpoint:

sgl-omni serve \
  --model-path FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
  --port 8000

Native MLX runtime#

cosyvoice3_server from the native MLX runtime (sglang_omni_mlx/native) serves the same /v1/audio/speech API on an Apple Silicon Mac from the float16 mlx-community/Fun-CosyVoice3-0.5B-2512-fp16 checkpoint, as one C++ process without PyTorch, ONNX Runtime or Python. Voice cloning runs the reference-audio front end natively too: the checkpoint’s CAMPPlus speaker encoder, and the S3 speech tokenizer v3 from mlx-community/S3TokenizerV3 in the model directory’s speech_tokenizer/.

sglang_omni_mlx/native/scripts/build_runtime.sh ~/omni-runtime

hf download mlx-community/Fun-CosyVoice3-0.5B-2512-fp16 config.json model.safetensors \
  tokenizer.json tokenizer_config.json \
  --revision 18ccb7fbd7246e8cd3420d02f5dd28595cc0fcd9 --local-dir fun-cosyvoice3-mlx-fp16
hf download mlx-community/S3TokenizerV3 config.json model.safetensors \
  --revision b143914b3e912278104824da706edc9c2d317c4e \
  --local-dir fun-cosyvoice3-mlx-fp16/speech_tokenizer

~/omni-runtime/bin/cosyvoice3_server --model-path fun-cosyvoice3-mlx-fp16 \
  --model-name FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --port 8000 \
  --allowed-local-media-path "$PWD"

curl -s http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" \
  -d '{"input": "Hello from Apple Silicon.", "seed": 1}' -o hello.wav

curl -s http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" \
  -d "{\"input\": \"Hello from Apple Silicon.\", \"ref_audio\": \"$PWD/reference.wav\",
       \"ref_text\": \"What the reference recording says.\", \"seed\": 1}" -o cloned.wav

The native server differs from the CUDA server in that:

Reference audio is a WAV file of 0.5 to 30 seconds, sent as a base64 data URL (ref_audio or references[0].data), or as a path or file:// URL inside --allowed-local-media-path; http(s) URLs and other containers return 400. ref_audio with ref_text, or one references item with text, clones the voice as on CUDA; without a transcript the reference conditions only the flow (cross-lingual). Giving both ref_audio and references returns 400. Uploaded voices (/v1/audio/voices) are not served.
voice must be default; language is accepted and unused, as on CUDA.
response_format is wav or pcm; speed must be 1.0; stream_format must be audio.
stream: true sends the whole utterance’s PCM once it is vocoded, since the flow decodes the utterance at once.
max_new_tokens is at most 2048 speech tokens (25 per second, about 82 seconds), where CUDA takes any value; without it the limit is 20 per target text token, as on CUDA. Integer fields take integral floats such as 5.0, as CUDA’s request model does.
Sampling defaults match the CUDA server (temperature 0.7, top_p 0.8, top_k 20, repetition_penalty 1.21, with the repetition-aware redraw); a request without seed gets a random one.
Started with --model-name, it returns 404 for any other model; without that flag, model is ignored, as on CUDA.
Errors are {"detail": "..."}. Requests run one at a time and --max-queued-requests (default 2) more wait; the rest get 503. A request not finished --request-timeout-s (default 600) seconds after it arrives is cancelled with 504, or its stream ends early, and so is one whose client disconnects. A request body over 32 MiB returns 413.

Measured on an Apple M5 (24 GB, macOS 26.3.1), warm, over 8 utterances (76 s of audio): 0.2 s load, real-time factor 0.33, about 2.4 GB peak MLX memory (3.2 GB with a reference). With the same speech tokens, the legacy Python MLX implementation from #1964 takes RTF 0.36 on the same MLX build. Transcribed back with Qwen3-ASR-1.7B-8bit, 16 English corpus sentences give 0.62% WER and 16 Chinese ones 0.97% CER. Against the official ONNX front end, the native one gives identical reference speech tokens on 20 of 20 corpus clips (speaker embedding cosine 0.99999, prompt mel within 7e-4), and with the same seed identical cloned speech tokens; cloned output transcribes at 1.03% WER (English) and 5.83% CER (Chinese, mostly digits and Latin names), the same with either front end. Mac CI checks seven seeded requests’ speech tokens (English, Chinese, Chinese with instructions, a max_new_tokens cap, the default length limit, an English clone and a cross-lingual clone) against sglang_omni_mlx/native/ci/model_golden/fun-cosyvoice3-0.5b-2512.json, and on every chip the mel of two of them vocoded from the recorded tokens, within a tolerance.

For Intel GPUs, follow the Fun-CosyVoice3 XPU recipe.

Synthesizing Speech#

Zero-shot Voice Cloning#

CosyVoice3 clones a voice from a short reference audio clip. ref_audio can be a local path, file URL, data URL, or HTTP URL. ref_text (the transcript of the reference clip) is optional but recommended for better alignment.

Using CURL:

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
    "input": "SGLang-Omni makes text-to-speech fast and easy to deploy.",
    "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
    "ref_text": "We asked over twenty different people, and they all said it was his."
  }' \
  --output output.wav

Using Python:

import requests

resp = requests.post(
    "http://localhost:8000/v1/audio/speech",
    json={
        "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
        "input": "Get the trust fund to the bank early.",
        "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
        "ref_text": "We asked over twenty different people, and they all said it was his.",
    },
)
resp.raise_for_status()
with open("output.wav", "wb") as f:
    f.write(resp.content)

Cross-lingual Synthesis#

CosyVoice3 supports cross-lingual voice cloning where the reference speaker speaks a different language than the synthesis text. Omit ref_text to enter cross-lingual mode.

Using CURL:

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
    "input": "今天天气真好，我们一起出去散步吧。",
    "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav"
  }' \
  --output output.wav

Using Python:

import requests

resp = requests.post(
    "http://localhost:8000/v1/audio/speech",
    json={
        "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
        "input": "今天天气真好，我们一起出去散步吧。",
        "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
    },
)
resp.raise_for_status()
with open("output.wav", "wb") as f:
    f.write(resp.content)

Instruction-based Style Control#

Pass instructions to guide prosody, emotion, or speaking style:

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
    "input": "Welcome to our annual developer conference.",
    "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
    "instructions": "Speak in a cheerful and energetic tone, as if addressing a large audience."
  }' \
  --output output.wav

Speed Control#

Adjust playback speed with speed (default 1.0):

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
    "input": "This is spoken at one point three times normal speed.",
    "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
    "speed": 1.3
  }' \
  --output output.wav

Streaming#

Incremental Flow + HiFT decoding supports streaming. Set stream: true and response_format: "pcm" to emit audio before AR generation completes. The vocoder uses CosyVoice3’s causal chunk loop: 25 speech tokens per hop (~1 s of audio). The first PCM chunk is emitted once AR produces 28 + prompt_pad tokens (where prompt_pad rounds the Flow prompt length to a 25-token multiple). Later hops grow from 25 to 50 to 100 tokens in vocoder to increase batching efficiency without affecting audio continuity or real-time factor, only the decoding granularity changes, not the total synthesis time. The scheduler processes at most one hop per request per step to prevent backlogged streams from monopolizing the GPU. Non-streaming requests decode the entire utterance in one pass.

Optional serving knobs (vocoder factory):

Factory arg	Default	Notes
`token_hop_len`	`25`	Base hop size; must match training chunk size
`token_max_hop_len`	`100`	Cap for `25 → 50 → 100` growth
`disable_hop_growth`	`false`	Leave off; growth ON performs better at high concurrency

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
    "input": "Get the trust fund to the bank early.",
    "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
    "ref_text": "We asked over twenty different people, and they all said it was his.",
    "stream": true,
    "response_format": "pcm"
  }' \
  --output output.pcm

Generation Parameters#

Parameter	Default	Notes
`model`	served model	Served model identifier
`input`	(required)	Text to synthesize
`ref_audio`	`null`	Reference audio for voice cloning (path / URL / data URL)
`ref_text`	`null`	Transcript of the reference audio. Improves cloning quality; omit for cross-lingual mode
`instructions`	`null`	Instruction text for style/prosody/emotion guidance
`speed`	`1.0`	Playback speed multiplier
`temperature`	`0.7`	Sampling temperature
`top_p`	`0.8`	Top-p sampling
`top_k`	`20`	Top-k sampling
`repetition_penalty`	`1.21`	Repetition penalty
`max_new_tokens`	`min(2048, 20x target text tokens)`	Maximum number of generated speech tokens. If omitted, derived from the target text length (capped at 2048); stop tokens are also suppressed until at least `2x` that length has been generated
`seed`	`null`	Random seed for reproducibility
`stream`	`false`	Incremental causal Flow + HiFT; first PCM chunk after `28 + prompt_pad` speech tokens (`prompt_pad` rounds the prompt length to a multiple of 25)

Serving Optimization#

Flow Decoder Batching#

For complete buffered requests, scheduler admission uses exact mel frames (flow_batch_admission_frames, default 8000). Adaptive Flow grouping is enabled by default: it sorts admitted requests by total mel length and lets adjacent requests share one Flow solve when the maximum within-group length gap and global added-padding budget stay within:

flow_merge_max_gap_frames = 384
flow_merge_pad_budget_percent = 25

HiFT grouping is independent and applies its existing hift_max_padding_waste policy to the produced mels. Causal streaming uses a separate Flow + HiFT path.

Increase the normal Flow batching budget only after measuring the target GPU.

sgl-omni serve \
  --model-path FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
  --port 8000 \
  --vocoder.factory.flow_batch_admission_frames 4000

On the other hand, decrease the admission budget to reduce latency and lower peak GPU memory.

Vocoder Configuration#

Vocoder configuration controls batching, precision, and acceleration. The scheduler accepts max_batch_size (16) and max_batch_wait_ms (30) to tune batch assembly. Flow uses dtype (bfloat16) for autocast, while HiFT uses hift_dtype (float32), independent of Flow; bfloat16 shows no speedup on H200 and reduces fidelity. Buffered Flow CUDA Graphs and DiT torch.compile are on by default. The Flow prefix cache keeps the keys and values of each stream’s completed frames for its later hops. It takes the GPU memory left after the TTS engine and the vocoder’s other allocations, within the stage’s gpu_memory_fraction or total_reserve_bytes when one is declared. It leaves free the prefix graphs and the largest step the vocoder admits, both measured at startup (the larger of max_batch_size streams, each a 30 s reference prompt and the TTS engine’s context length of generated tokens, and the largest non-streaming batch that flow_batch_admission_frames lets through), and the runtime headroom (1 - mem_fraction_static of the GPU memory free before the TTS engine loaded, or before the vocoder loaded in a process without the engine). The cache’s size counts its padding block, each stream’s convolution context and the allocator’s rounding. flow_prefix_cache_gb caps that size, a larger value falls back to it with a warning, and 0 turns the cache off. Streaming hops over the Flow prefix cache replay their Euler solve from CUDA graphs captured per frame tier (one at every whole DiT chunk, from a tier holding the shortest hop up to max_batch_size rows of the longest hop). A step above the largest tier runs in eager mode, and enable_flow_prefix_cuda_graph=false turns the graphs off. A stream’s final step replays its whole-history Euler solve from CUDA graphs captured per frame tier with max_batch_size row slots, on the prefill graph ladder up to flow_batch_admission_frames. A final step above the largest tier runs eagerly, and enable_flow_whole_history_cuda_graph=false turns these graphs off. On SM120 the DiT runs FlashAttention 4, the whole-history graphs stay off. When the largest admitted step does not fit on the device, serving runs without the prefix cache. HiFT’s decode replays from CUDA graphs captured per row tier, up to max_batch_size, and per frame tier, from a first hop’s window to a final’s. F0 and the sine source stay eager, and a step wider than the largest frame tier decodes eagerly. The graphs need a float32 HiFT on CUDA with Triton, and enable_hift_cuda_graph=false turns them off. enable_flow_estimator_trt stays opt-in; enabling it turns the DiT compile default off, and enabling both explicitly is rejected.

The TTS engine stage accepts onnx_intra_op_threads (16) for the speech tokenizer and speaker encoder ONNX sessions. Preprocessing takes max_concurrency (8) to limit concurrent reference conditioning requests.

On CUDA the TTS engine replays prompt prefills from full CUDA graphs captured on the prefill batch ladder, which starts at 1 because a prompt that is already cached prefills a single token. The graphs need the fa3, fa4 or flashinfer attention backend, and with any other backend the default prefill runs in eager mode. Opt out with --tts_engine.engine.cuda_graph_backend_prefill disabled. Declaring cuda_graph_bs_prefill keeps the declared list as is.

torch.compile for the DiT backbone#

torch.compile is on by default. It reduces DiT kernel-launch overhead for supported TTS execution paths, including streaming PackedDiT. Startup warmup covers varying batch sizes and mel lengths before serving. Existing CUDA Graph behavior is unchanged. Pass enable_dit_torch_compile=false to run the DiT eager.

sgl-omni serve \
  --model-path FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
  --vocoder.factory.enable_dit_torch_compile false \
  --port 8000

Enabling TensorRT turns this default off.

TensorRT for the DiT backbone#

TensorRT accelerates the DiT by building a cached .plan engine from the bundled ONNX. The CFG batch is frozen at 2 with dynamic mel dimensions; larger request batches are handled by chunking cond/uncond pairs. Enabling it turns the DiT torch.compile default off; enabling both explicitly is rejected.

uv pip install tensorrt

See NVIDIA’s pip install guide for more details.

Enable the flag:

sgl-omni serve \
  --model-path FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
  --vocoder.factory.enable_flow_estimator_trt true \
  --port 8000

On one H200, Flow latency and vocoder RTF comparisons:

Backend	Batch size	Flow latency	Vocoder RTF
eager	1	1131 ms	0.359
torch.compile	1	1070 ms	0.340
TensorRT (CFG batch=2 engine)	1	20 ms	0.012
eager	4	267 ms	0.026
torch.compile	4	220 ms	0.022
TensorRT (chunked 4× CFG pairs)	4	77 ms	0.011

Full SeedTTS EN set (1088 samples) on buffered /v1/audio/speech, one H200:

Backend	Concurrency	Latency mean	RTF mean	Throughput
eager	1	1.091 s	0.241	0.916 req/s
torch.compile	1	1.016 s	0.221	0.984 req/s
TensorRT	1	0.871 s	0.189	1.147 req/s
eager	16	5.690 s	1.300	2.800 req/s
torch.compile	16	3.151 s	0.706	5.059 req/s
TensorRT	16	2.549 s	0.570	6.243 req/s

This result is only demonstrative, since we have further optimization after the evluation of TensorRT is done.

Speculative Decoding#

SpeechSpec’s DSpark draft yuekai/cosyvoice3_llm_dspark proposes 7 speech tokens per step from three of the LLM’s hidden layers, and the LLM verifies them in one forward pass.

sgl-omni serve \
  --model-path FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
  --tts_engine.engine.speculative_algorithm DSPARK \
  --tts_engine.engine.speculative_draft_model_path yuekai/cosyvoice3_llm_dspark \
  --port 8000

Send return_spec_tokens_details: true with a speech request to get its acceptance statistics. Buffered audio adds X-SGLang-Omni-Spec-Accept-Length, X-SGLang-Omni-Spec-Accept-Rate, X-SGLang-Omni-Spec-Num-Correct-Drafts, X-SGLang-Omni-Spec-Num-Proposed-Drafts and X-SGLang-Omni-Spec-Verify-Ct headers; the SSE speech.audio.done event and the raw PCM outcome from GET /v1/audio/speech/{request_id} carry sglext.spec_tokens_details, which also holds the per-verify histogram. spec_accept_length is speech tokens per verify, including the token sampled at prefill, so it can read higher than the scheduler log’s accept len, which counts only verified tokens. spec_accept_rate and spec_num_proposed_drafts, and their headers, are left out when no drafts were proposed or the draft width is adaptive.

Verification reproduces CosyVoice3’s sampler exactly: temperature, top-k/top-p, the repetition penalty, the minimum-length stop mask and the repetition-aware redraw. The speech tokens therefore follow the same distribution as without the draft. A request’s seed fixes every draft and verify draw, so a seeded request reproduces its speech tokens while the draft is on; they differ from the tokens the same seed gives without the draft, which samples differently.

LLM speech-token throughput without the vocoder (Seed-TTS zh, 2,020 utterances, one H200 141GB with SGLang 0.5.22, target and draft on FA3, each batch run to completion), with the default sampling and about 3.0 tokens accepted per verify:

Batch	Baseline	DSpark	Speedup
1	434 tok/s	711 tok/s	1.64x
8	2,421 tok/s	3,794 tok/s	1.57x
16	4,260 tok/s	6,388 tok/s	1.50x
64	11,030 tok/s	15,027 tok/s	1.36x

With SpeechSpec’s benchmark sampling (temperature 0.8, top_p 0.95, top_k 15, repetition_penalty 1.1), the speedups are 1.64x, 1.61x, 1.57x and 1.42x.

End-to-end streaming on the same GPU (benchmark_tts_seedtts --stream, all 2,020 Seed-TTS zh utterances, default sampling), baseline → DSpark:

Concurrency	Throughput	RTF mean	TTFC mean	WER (Qwen3-ASR)
1	2.68 → 3.77 req/s	0.071 → 0.051	0.140 → 0.122 s	0.77% → 0.71%
8	12.37 → 15.59 req/s	0.123 → 0.098	0.230 → 0.227 s	0.75% → 0.79%
16	18.02 → 21.08 req/s	0.169 → 0.146	0.306 → 0.333 s	0.76% → 0.73%

The vocoder takes a growing share of each request as concurrency rises, so the end-to-end gain is largest at low concurrency.

Zero-shot Voice Cloning#

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
    "input": "SGLang-Omni makes text-to-speech fast and easy to deploy.",
    "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
    "ref_text": "We asked over twenty different people, and they all said it was his."
  }' \
  --output output.wav

Python

import requests

resp = requests.post(
    "http://localhost:8000/v1/audio/speech",
    json={
        "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
        "input": "Get the trust fund to the bank early.",
        "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
        "ref_text": "We asked over twenty different people, and they all said it was his.",
    },
)
resp.raise_for_status()
with open("output.wav", "wb") as f:
    f.write(resp.content)

Cross-lingual Synthesis#

CosyVoice3 supports cross-lingual voice cloning where the reference speaker speaks a different language than the synthesis text. Omit ref_text to enter cross-lingual mode.

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
    "input": "今天天气真好，我们一起出去散步吧。",
    "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav"
  }' \
  --output output.wav

Instruction-based Style Control#

Pass instructions to guide prosody, emotion, or speaking style:

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
    "input": "Welcome to our annual developer conference.",
    "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
    "instructions": "Speak in a cheerful and energetic tone, as if addressing a large audience."
  }' \
  --output output.wav

Speed Control#

Adjust playback speed with speed (default 1.0):

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
    "input": "This is spoken at one point three times normal speed.",
    "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
    "speed": 1.3
  }' \
  --output output.wav

Streaming#

Incremental Flow + HiFT decoding is enabled. Set stream: true and use response_format: "pcm" so the server can emit audio before speech-token generation finishes.

The vocoder follows CosyVoice3’s causal chunk loop: 25 speech tokens per hop (pre_lookahead_len=3, 25 Hz → about 1 s of audio per chunk). The first PCM chunk is emitted once the AR stage has produced 28 + prompt_pad tokens, where prompt_pad (0–24) rounds the Flow prompt-token length up to a multiple of 25. Later hops grow 25 → 50 → 100 tokens like the upstream CosyVoice3Model (default; keep growth on). Each scheduler step runs at most one hop per request so a backlogged stream cannot monopolize the GPU. Non-streaming requests still use the buffered Flow + HiFT path for the whole utterance.

Optional serving knobs (vocoder factory; keep tts_engine.factory.token_hop_len in sync if you change the hop):

Factory arg	Default	Notes
`token_hop_len`	`25`	Base hop / AR follow-up flush; must match training chunk size
`token_max_hop_len`	`100`	Cap for `25 → 50 → 100` growth
`disable_hop_growth`	`false`	Leave off unless A/B’ing; growth ON was better at c=16

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
    "input": "Get the trust fund to the bank early.",
    "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
    "ref_text": "We asked over twenty different people, and they all said it was his.",
    "stream": true,
    "response_format": "pcm"
  }' \
  --output output.pcm

Generation Parameters#

Parameter	Default	Notes
`model`	served model	Served model identifier
`input`	(required)	Text to synthesize
`ref_audio`	`null`	Reference audio for voice cloning (path / URL / data URL)
`ref_text`	`null`	Transcript of the reference audio. Improves cloning quality; omit for cross-lingual mode
`instructions`	`null`	Instruction text for style/prosody/emotion guidance
`speed`	`1.0`	Playback speed multiplier
`temperature`	`0.7`	Sampling temperature
`top_p`	`0.8`	Top-p sampling
`top_k`	`20`	Top-k sampling
`repetition_penalty`	`1.21`	Repetition penalty
`max_new_tokens`	`min(2048, 20x target text tokens)`	Maximum number of generated speech tokens. If omitted, derived from the target text length (capped at 2048); stop tokens are also suppressed until at least `2x` that length has been generated
`seed`	`null`	Random seed for reproducibility
`stream`	`false`	Incremental causal Flow + HiFT; first PCM chunk after `28 + prompt_pad` speech tokens (`prompt_pad` rounds the prompt length to a multiple of 25)

Sampling is repetition-aware, as in upstream CosyVoice: a sampled speech token that already appears among the last 10 generated tokens is masked and redrawn once from the full distribution at the request temperature. Greedy requests (top_k = 1) sample without the redraw.

Benchmarking#

Measure causal streaming with Seed-TTS-Eval against a running server:

python -m benchmarks.eval.benchmark_tts_seedtts \
  --model FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
  --port 8000 --lang en --max-concurrency 16 \
  --use-existing-server --generate-only --stream \
  --output-dir results/fun_cosyvoice3_en

Use --lang zh --no-ref-text for the Chinese cross-lingual split. See benchmarks/README.md for the full workflow.

Model Architecture#

Component	Detail
LLM Backbone	Qwen2.5-0.5B (24 layers, hidden=896, 14 heads, 2 KV heads GQA)
Speech Tokenizer	FSQ codebook (vocab=6561) + 200 special tokens, 25 Hz frame rate
Speaker Encoder	CAMPPlus (192-dim embedding, ONNX)
Flow Model	CausalMaskedDiffWithDiT (DiT depth=22, dim=1024, heads=16)
Vocoder	CausalHiFTGenerator (24 kHz output)
Sample Rate	24000 Hz

Known Limitations#

Reference audio required. CosyVoice3 requires a reference audio clip for voice cloning; it does not support text-only synthesis without a speaker reference.
30-second limit. Reference audio must be 30 seconds or shorter for speech token extraction.
Speaker similarity. Providing ref_text (the transcript) yields better voice similarity than omitting it (cross-lingual mode).
Reference shape. The endpoint accepts either ref_audio plus optional ref_text, or one item in references; multiple references are rejected for this checkpoint.
Prompt modes. Provide either ref_text or instructions for the reference prompt, not both. instructions selects CosyVoice3 instruct2 conditioning.
Reference conditioning cache. Local files, data URLs, and byte payloads are cached by audio content and encoder configuration. Mutable HTTP URLs are intentionally encoded on every request instead of being cached by URL alone.
Speed control. Applied once, on the decoded waveform, by the shared /v1/audio/speech response-encoding path.
Voice conversion. Voice conversion is outside the current zero-shot TTS scope.
Streaming decode. Causal Flow + HiFT emit PCM after each hop (hop grows 25 → 50 → 100 by default). Quality can differ slightly from the buffered whole-utterance path. Opt-in TensorRT (enable_flow_estimator_trt) also accelerates streaming hops; it turns the enable_dit_torch_compile default off. TRT freezes DiT attention, so streaming+TRT is not bit-exact with PyTorch streaming. Keep the Module TRT wrapper when streaming: CosyVoice’s raw TRT enqueue is incompatible with packed hop-batch CFG shapes.
Flow batch scope. Flow batching supports the CosyVoice PyTorch estimator and the opt-in TensorRT estimator. Buffered HiFT grouping is independent and uses hift_max_padding_waste. Streaming coalesces first/follow-up hops across requests (_can_batch_stream_chunks, short peer wait) so TTFP stays low under load.
Speculative decoding. The DSpark draft runs on CUDA with TP 1, the TTS engine’s scheduler steps synchronously while it is on, and weight updates are refused while it is loaded.
cosyvoice dependency. The cosyvoice package has no PyPI release and must be installed from GitHub. Matcha-TTS is a required submodule and must also be importable; only the CosyVoice Flow and HiFT paths are used by the vocoder.

Fun-CosyVoice3

Contents

Fun-CosyVoice3#

Prerequisites#

Apple Silicon#

Native MLX runtime#

Synthesizing Speech#

Zero-shot Voice Cloning#

Cross-lingual Synthesis#

Instruction-based Style Control#

Speed Control#

Streaming#

Generation Parameters#

Serving Optimization#

Flow Decoder Batching#

Vocoder Configuration#

torch.compile for the DiT backbone#

TensorRT for the DiT backbone#

Speculative Decoding#

Zero-shot Voice Cloning#

Cross-lingual Synthesis#

Instruction-based Style Control#

Speed Control#

Streaming#

Generation Parameters#

Benchmarking#

Model Architecture#

Known Limitations#