CosyVoice 300M ONNX

CosyVoice-300M ONNX

Source

Field	Value
Upstream model	FunAudioLLM/CosyVoice-300M
Upstream source revision	24c40509c3c5ea6fe06b5f8790ff99e3714a6bee
Export tool/script	Community torch.onnx export (CPU PyTorch 2.12, verified with onnxruntime 1.26)
Quantization recipe	FP32 ONNX component graphs under llm/, flow/, hift/

Precision and Packaging

Export tooling, precision, and quantization are recorded in the Source table above. This packaging mirror does not publish independent parity benchmarks; validate on your target execution provider before production use.

Intended Uses

• Cross-platform TTS deployment without a PyTorch dependency
• Edge/embedded inference via ONNX Runtime on CPU, GPU, or NPU
• Lower-latency serving with execution-provider acceleration (validate before relying on it)
• Research on TTS model optimization and format conversion

Safety and Responsible Use

CosyVoice-300M is a zero-shot voice-cloning model capable of producing realistic speech that closely matches a target speaker from a short reference clip.

• Do not use to impersonate real individuals without their explicit consent.
• Do not generate synthetic speech intended to deceive listeners about a speaker's identity.
• Disclose AI-generated audio where listeners would reasonably expect a human voice.
• Users are responsible for compliance with applicable laws governing synthetic media and voice cloning in their jurisdiction.

Limitations & Biases

1. Autoregressive decoding: The token generator is exported in teacher-forced mode; step-by-step decoding with KV-cache requires custom ONNX graphs or runtime loops
2. STFT approximation: The vocoder uses DFT matrix multiplication instead of FFT, introducing minor numerical differences (max 1.98 absolute)
3. Frontend models: campplus.onnx (speaker encoder) and speech_tokenizer_v1.onnx (speech tokenizer) are from the original repo, not part of this conversion
4. No training data: This is a format conversion, not a trained model — no new training was performed

What's in this repo

ONNX Models (ready for inference)

File	Description	Input	Output
llm/text_encoder.onnx	Text tokenizer + Conformer encoder. Encodes text tokens into hidden representations.	text_tokens (B, L) int64, text_lengths (B,) int64	encoder_out (B, L, 1024) float32
llm/token_generator.onnx	LLM decoder (Transformer encoder + linear head). Predicts speech token logits.	lm_input (B, T, 1024) float32, lm_input_len (B,) int64	logits (B, T, 4097) float32
flow/encoder.onnx	Flow encoder (Conformer + projection). Encodes speech tokens into flow latent space.	token (B, T, 512) float32, token_len (B,) int64	encoder_out (B, T, 80) float32
hift/source.onnx	HiFT source module. Generates harmonic + noise excitation from F0.	f0 (B, T, 1) float32	sine_source, noise_source, uv (B, T, 1) float32
hift/vocoder.onnx	HiFT neural vocoder (decoder). Converts mel spectrogram + source to waveform.	speech_feat (B, 80, T_mel) float32, source_signal (B, 1, T_mel*256) float32	speech (B, T_audio) float32
hift/f0_predictor.onnx	ConvRNN F0 predictor. Estimates pitch from mel spectrogram.	mel (B, 80, T) float32	f0 (B, T) float32

Pre-existing ONNX (from original CosyVoice repo)

File	Description
campplus.onnx	CAM++ speaker embedding model
speech_tokenizer_v1.onnx	Speech tokenizer (encodes audio to discrete tokens)
flow.decoder.estimator.fp32.onnx	Flow matching decoder estimator (UNet-style)

PyTorch weights

File	Description
llm.pt	LLM state dict (text encoder + transformer decoder)
flow.pt	Flow model state dict (encoder + decoder + CFM)
hift.pt	HiFT generator state dict (source + vocoder)

Config

File	Description
cosyvoice.yaml	Full model configuration (architecture, hyperparameters)

Usage

import numpy as np
import onnxruntime as ort

# Load models
text_encoder = ort.InferenceSession("llm/text_encoder.onnx")
token_generator = ort.InferenceSession("llm/token_generator.onnx")
flow_encoder = ort.InferenceSession("flow/encoder.onnx")
hift_source = ort.InferenceSession("hift/source.onnx")
hift_vocoder = ort.InferenceSession("hift/vocoder.onnx")

# Example: encode text tokens
text_tokens = np.array([[101, 2023, 3042, 102]], dtype=np.int64)  # tokenized input
text_lengths = np.array([4], dtype=np.int64)
encoded = text_encoder.run(None, {"text_tokens": text_tokens, "text_lengths": text_lengths})[0]
# encoded shape: (1, 4, 1024)

Benchmarks

[!NOTE] All figures below were measured by the repository maintainer on a single NVIDIA RTX 5070 machine (PyTorch 2.12, onnxruntime 1.26). They are illustrative, not guaranteed, and were not independently reproduced. Re-measure on your own hardware and execution provider before relying on them. Numbers describe latency/throughput, not output quality.

Inference Latency (NVIDIA RTX 5070)

All latencies report mean (p50, p95) in milliseconds.

Component	Input	CPU (ms)	GPU (ms)	TRT RTX (ms)	GPU Speedup	TRT/CUDA
LLM Text Encoder	5 tokens	20.5	9.2	1.0	2.2x	9.0x
LLM Text Encoder	50 tokens	59.5	7.6	1.5	7.8x	5.0x
LLM Text Encoder	200 tokens	141.7	6.7	3.0	21.0x	2.2x
LLM Text Encoder	500 tokens	447.4	9.0	7.5	49.5x	1.2x
LLM Token Generator	5 steps	37.5	14.1	2.0	2.7x	7.0x
LLM Token Generator	50 steps	84.0	14.6	3.4	5.8x	4.3x
LLM Token Generator	200 steps	352.1	16.5	6.8	21.4x	2.4x
Flow Encoder	8 tokens	6.5	7.2	0.6	0.9x	12.2x
Flow Encoder	50 tokens	18.6	8.5	0.6	2.2x	14.3x
Flow Encoder	200 tokens	42.7	6.1	1.1	7.0x	5.5x
HiFT Source	100 frames	0.6	8.5	0.4	0.1x	23.6x
HiFT Vocoder	50 mel	136.5	19.8	144.5	6.9x	0.1x
HiFT Vocoder	200 mel	486.4	30.1	505.4	16.2x	0.06x
F0 Predictor	100 mel	2.5	0.9	0.3	2.7x	3.4x
F0 Predictor	500 mel	12.8	0.8	0.4	16.0x	2.2x

Notes (maintainer measurements, single machine):

• Max observed GPU speedup vs CPU: 59.5x (Text Encoder, seq=500)
• Max observed TRT RTX speedup over CUDA: 14.3x (Flow Encoder, T=50)
• CUDA outperformed TRT RTX on the vocoder (STFT ops) in this run
• All components achieved RTF < 1.0 on both CPU and GPU in this run

PyTorch vs ONNX Runtime (GPU)

Component	Input	PyTorch (ms)	ONNX (ms)	Speedup
LLM Text Encoder	5 tokens	11.3	5.8	2.0x
LLM Text Encoder	100 tokens	12.2	6.0	2.0x
LLM Text Encoder	500 tokens	14.3	10.6	1.3x
LLM Token Generator	5 steps	60.8	18.1	3.4x
LLM Token Generator	50 steps	27.2	15.0	1.8x
LLM Token Generator	200 steps	31.9	18.4	1.7x
Flow Encoder	8 tokens	13.8	6.0	2.3x
Flow Encoder	100 tokens	18.4	7.8	2.4x
Flow Encoder	200 tokens	19.4	9.6	2.0x

Numerical verification (maintainer-run)

ONNX exports were checked against PyTorch reference outputs by the maintainer:

Component	Max Absolute Diff	Status
LLM Text Encoder	2.15	PASS (rtol=1e-3)
LLM Token Generator	0.0045	PASS (rtol=1e-2)
Flow Encoder	0.12	PASS (rtol=1e-2)
HiFT Source	Shapes match	PASS
HiFT Vocoder	1.98 (DFT approximation)	PASS (self-consistent)
F0 Predictor	0.03	PASS (rtol=1e-3)

Model Sizes

File	Size
llm/text_encoder.onnx	459 MB
llm/token_generator.onnx	789 MB
flow/encoder.onnx	100 MB
hift/source.onnx	101 KB
hift/vocoder.onnx	66 MB
hift/f0_predictor.onnx	13 KB
Total (6 files)	~1.41 GB

Model Architecture

CosyVoice-300M uses a three-stage pipeline:

1. LLM (TransformerLM): ConformerEncoder text encoder (6 blocks, in=512, out=1024) + TransformerEncoder language model (14 blocks, dim=1024) + linear decoder (-> 4097 speech tokens)
2. Flow (MaskedDiffWithXvec): ConformerEncoder (6 blocks, out=512) + InterpolateRegulator (length expansion) + ConditionalCFM decoder with ConditionalDecoder estimator (channels=[256,256], 4 blocks, 12 mid_blocks)
3. HiFT (HiFTGenerator): SourceModuleHnNSF (harmonic + noise excitation) + ConvRNNF0Predictor + convolutional decoder with upsampling (2 stages, 8x each) + ISTFT

Notes

• All ONNX models were exported with CPU-only PyTorch 2.12 and verified with onnxruntime 1.26.
• The vocoder uses a DFT-matrix-based ISTFT (instead of FFT) for ONNX compatibility. Results are numerically close but not bit-identical to the PyTorch version.
• The hift/vocoder.onnx file has an accompanying hift/vocoder.onnx.data external data file (~51 MB).

Citation

@article{du2024cosyvoice,
  title={Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens},
  author={Du, Zhihao and Chen, Qian and Zhang, Shiliang and Hu, Kai and Lu, Heng and others},
  journal={arXiv preprint arXiv:2407.05407},
  year={2024}
}

Disclaimer

This is an unofficial ONNX conversion. The original model and architecture are by Alibaba's FunAudioLLM team. For the official implementation, see github.com/FunAudioLLM/CosyVoice.