Skip to content

voxcpm1.md

Latest commit

 

History

History
193 lines (145 loc) · 8.22 KB

voxcpm1.md

File metadata and controls

193 lines (145 loc) · 8.22 KB
Raw

VoxCPM1

VoxCPM1 is a tokenizer-free TTS model from OpenBMB that generates 16 kHz mono speech. The audio.cpp package uses a native GGUF layout with audio.cpp tensor names, embedded config/tokenizer sidecars, folded AudioVAE weights, and materialized SR-conditioning tensors.

Field Value
Family voxcpm1
Model directory models/VoxCPM1-GGUF (0.5B)
Tasks tts, clon
Modes offline, streaming
Languages Model auto-handles supported languages (zh, en, ja, ko validated)
Voice input Optional reference WAV; optional transcript via --reference-text
Built-in voices Not exposed
Output mono 16 kHz WAV

🚀 Installation & Quick Start

1. Install GGUF Model Weights

# Via model manager (recommended)
python3 tools/model_manager_v2.py install voxcpm1_0_5b_q8_0 --models-root models

This downloads the voxcpm-0.5b-q8_0-audiovae-f16.gguf package (~690 MB) to models/VoxCPM1-GGUF/.

2. Compile the CLI Target

chmod +x scripts/build_linux.sh
./scripts/build_linux.sh --backend cpu --target audiocpp_cli

For CUDA (recommended for production):

./scripts/build_linux.sh --backend cuda --target audiocpp_cli

3. Run Inference

Text-to-speech (offline):

./build/linux-cpu-release/bin/audiocpp_cli \
  --task tts \
  --family voxcpm1 \
  --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf \
  --backend cpu \
  --text "Hello from VoxCPM1." \
  --out out.wav

Voice clone (continuation-mode):

./build/linux-cpu-release/bin/audiocpp_cli \
  --task tts \
  --family voxcpm1 \
  --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf \
  --backend cpu \
  --text "Hello from VoxCPM1." \
  --voice-ref assets/resources/b.wav \
  --reference-text "Some call me nature. Others call me Mother Nature." \
  --out out.wav

clon is an alias to the same synthesis path. Route selection does not change the existing reference-audio or continuation-prompt behavior.

Streaming output:

./build/linux-cpu-release/bin/audiocpp_cli \
  --task tts \
  --family voxcpm1 \
  --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf \
  --backend cpu \
  --mode streaming \
  --text "Hello from VoxCPM1 streaming." \
  --request-option retry_badcase=false \
  --out out.wav

📊 Performance Benchmark

Measured on Ubuntu 24.04 with OpenMP optimization (CPU backend):

  • GGUF package size: ~690 MB (Q8_0 LLM + F16 AudioVAE)
  • Startup latency: Instant (<0.01s) via mmap lazy loading
  • Inference speed (short sentence): ~0.25 RTF on CPU (~4x faster than real-time)
  • Idle VRAM (with mem_saver): ~1.4 GB; long text up to ~3.5 GB

🛡️ Options & Customizations

Option Values Default Meaning
--voice-ref WAV path not set Reference speaker audio for voice cloning.
--reference-text text empty Transcript of the reference audio (for clone prompting).
--mode offline, streaming offline Full-output or streaming run mode; streaming does not retry bad cases.
--session-option voxcpm1.mem_saver true, false false Use tighter graph workspaces and release request runtime graphs to reduce resident VRAM.
--session-option voxcpm1.prompt_cache_slots integer 1 Prompt and prompt-audio embedding cache slots. Set 0 to disable prompt caching.
--session-option voxcpm1.weight_type native, f32, f16, bf16, q8_0 native Generator weight storage for the session.
--session-option voxcpm1.lora directory or checkpoint file not set Native LoRA weights with adjacent lora_config.json; merged at session creation.
--session-option voxcpm1.lora_scale finite float 1 Multiplier for the configured alpha/r scaling.
--session-option voxcpm1.audiovae_weight_type native, f32, f16, bf16, q8_0 native AudioVAE weight storage for the session.
--max-tokens integer 4096 Maximum MiniCPM output tokens; default 4096. Zero selects model max_length; negative counts are rejected.
--num-inference-steps integer 10 CFM diffusion sampling steps; default 10. Must be positive.
--guidance-scale float 2.0 CFM classifier-free guidance rate. Must be nonnegative.
--request-option retry_badcase true, false offline: true; streaming: false Auto-retry when generation is detected as a bad case. Streaming ignores explicit true with a warning.
--request-option retry_badcase_max_times integer 3 Maximum total generation attempts, including the initial attempt.
--text-chunk-mode default, tag_aware, japanese, endline tag_aware Long-form text chunking strategy.

Session Graph Arenas

Use these names with --session-option. They size host graph metadata in MiB, not device workspace. The old *_graph_context_mb aliases remain accepted with the same voxcpm1. prefix. Weight-context controls are separate and unchanged.

Option Default MiB
voxcpm1.text_embedding_graph_arena_mb 64
voxcpm1.lm_step_graph_arena_mb 1024
voxcpm1.projection_graph_arena_mb 256
voxcpm1.local_encoder_graph_arena_mb 512
voxcpm1.dit_graph_arena_mb 1024
voxcpm1.audiovae_graph_arena_mb 1024
voxcpm1.audiovae_encoder_graph_arena_mb 1024

🔧 Architecture Notes

  • Base LM: 24-layer MiniCPM transformer (1024 hidden, 73,448 vocab)
  • Residual LM: 6-layer autoregressive refinement
  • Local Encoder: 4-layer feature encoder (FSQ quantization, 8 codebooks)
  • Local DiT: 4-layer diffusion transformer for CFM sampling
  • AudioVAE: Encoder (128 dim, rates [2,5,8,8]) + Decoder (1536 dim, rates [8,8,5,2])
  • Output: 16 kHz mono

The GGUF is produced from the original OpenBMB files with:

python3 tools/community_models/convert_voxcpm1.py --overwrite

The converter stages the OpenBMB PyTorch checkpoint as audio.cpp tensor names, folds AudioVAE weight_v / weight_g pairs through the shared GGUF conversion path, embeds the required sidecars, and writes one native GGUF. The runtime expects that native layout directly; it does not adapt third-party VoxCPM GGUF metadata at load time.

✅ Validation Status

Mode Status Notes
Offline TTS ✅ Works "This is a test run for the fix" → transcribes as "This is a test." (SenseVoice)
Voice Clone ✅ Works 6/6 target sentences transcribe exactly via SenseVoice; continuation-mode with reference audio + transcript
Streaming ✅ Works SSE PCM chunks at native 16 kHz; retries default to false, and an explicit true is ignored with a warning. On CPU, CUDA, HIP and Vulkan each patch is decoded with the AudioVAE's causal-convolution state carried over from the previous patch, so patch boundaries closely match the offline decode. Metal remains on the per-patch fallback and is expected to show the same boundary artifacts; it was not tested.
Reference-only clone ⚠️ Limited ref_start/ref_end fails identically in the golden VoxCPM.cpp — model-level limitation

Regression guard: VoxCPM2 path is untouched (config.v1 default false); still generates 48 kHz speech with byte-identical output.

📦 Model Package

The default package is the standalone GGUF:

Package ID Display Name Format Precision Files
voxcpm1_0_5b_q8_0 VoxCPM 0.5B Q8_0 GGUF gguf q8_0 (LLM) + f16 (AudioVAE) VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf

The GGUF embeds:

  • Hybrid-quantized model (LLM Q8_0, AudioVAE F16)
  • Full config sidecar (config.json)
  • BPE tokenizer sidecars (tokenizer.json, tokenizer_config.json)

No separate tokenizer or config files are needed at inference time.

🔗 References