Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding

The auto-regressive architecture, like GPTs, is widely used in modern Text-to-Speech (TTS) systems. However, it incurs substantial inference time, particularly due to the challenges in the next-token prediction posed by lengthy sequences of speech tokens. In this work, we introduce VADUSA, one of the first approaches to accelerate auto-regressive TTS through speculative decoding. Our results show that VADUSA not only significantly improves inference speed but also enhances performance by incorporating draft heads to predict future speech content auto-regressively. Furthermore, the inclusion of a tolerance mechanism during sampling accelerates inference without compromising quality. Our approach demonstrates strong generalization across large datasets and various types of speech tokens.

@article{li2025_2410.21951,
  title={ Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding },
  author={ Bohan Li and Hankun Wang and Situo Zhang and Yiwei Guo and Kai Yu },
  journal={arXiv preprint arXiv:2410.21951},
  year={ 2025 }
}