herimor / voxtream

What it does

• Dynamic speed control. Duration distribution guidance allows for fine-grained speaking rate control, which can be adjusted as the model generates speech.
• Hybrid depth transformer head. 32 Mimi codebooks are generated in 4 steps (3 AR and a single Mixture of Gaussians step), combining high speech quality and low generation complexity.
• Streaming performance. Works 10x times faster than real-time and achieves 30 ms first packet latency in a full-stream on RTX3090 GPU. Real-time streaming support on a single CPU core via GGUF (int8) and llama.cpp.
• Long-form generation. Generation runs for up to 5 minutes with a WER below 2% measured on the LibriSpeech-long benchmark.