SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models

31 August 2023

Xipeng Qiu

Papers citing "SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models"

50 / 55 papers shown

Title
Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play Yemin Shi Yu Shu Siwei Dong Guangyi Liu Jaward Sesay Jingwen Li Zhiting Hu AuLLM VLM 43 0 0 05 May 2025
Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities X. Zhang Jintao Guo Shanshan Zhao Minghao Fu Lunhao Duan Guo-Hua Wang Qing-Guo Chen Zhao Xu Weihua Luo Kaifu Zhang DiffM 57 0 0 05 May 2025
Deep Audio Watermarks are Shallow: Limitations of Post-Hoc Watermarking Techniques for Speech P. O'Reilly Zeyu Jin Jiaqi Su Bryan Pardo 21 0 0 15 Apr 2025
TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling Liang-Hsuan Tseng Yi-Chang Chen Kuan-Yi Lee Da-shan Shiu Hung-yi Lee AuLLM 52 0 0 09 Apr 2025
UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation Alexander H. Liu Sang-gil Lee Chao-Han Huck Yang Yuan Gong Yu-Chun Wang James Glass Rafael Valle Bryan Catanzaro SSL 42 0 0 02 Mar 2025
From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval Jian Jia Jingtong Gao Ben Xue Junhao Wang Qingpeng Cai Quan Chen Xiangyu Zhao Peng Jiang Kun Gai OffRL 62 0 0 18 Feb 2025
AudioMiXR: Spatial Audio Object Manipulation with 6DoF for Sound Design in Augmented Reality Brandon Woodard Margarita Geleta Joseph J. LaViola Jr. Andrea Fanelli Rhonda Wilson 55 2 0 05 Feb 2025
SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characters Jianping Jiang Weiye Xiao Zhengyu Lin H. Zhang Tianxiang Ren Yang Gao Zhiqian Lin Zhongang Cai Lei Yang Ziwei Liu 79 3 0 29 Nov 2024
Scaling Transformers for Low-Bitrate High-Quality Speech Coding Julian Parker Anton Smirnov Jordi Pons CJ Carr Zack Zukowski Zach Evans Xubo Liu 70 9 0 29 Nov 2024
MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios Xiao-Hang Jiang Yang Ai Rui Zheng Hui-Peng Du Ye-Xin Lu Zhen-Hua Ling 39 0 0 01 Nov 2024
Optimizing Neural Speech Codec for Low-Bitrate Compression via Multi-Scale Encoding Peiji Yang Fengping Wang Yicheng Zhong Huawei Wei Zhisheng Wang 13 0 0 21 Oct 2024
Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant Alan Dao Dinh Bach Vu Huy Hoang Ha AuLLM VLM 57 3 0 20 Oct 2024
DM-Codec: Distilling Multimodal Representations for Speech Tokenization Md Mubtasim Ahasan Md Fahim Tasnim Mohiuddin A K M Mahbubur Rahman Aman Chadha Tariq Iqbal M. A. Amin Md. Mofijul Islam Amin Ahsan Ali 13 0 0 19 Oct 2024
Code Drift: Towards Idempotent Neural Audio Codecs P. O'Reilly Prem Seetharaman Jiaqi Su Zeyu Jin Bryan Pardo 33 0 0 14 Oct 2024
Graded Suspiciousness of Adversarial Texts to Human Shakila Mahjabin Tonni Pedro Faustini Mark Dras AAML 16 0 0 06 Oct 2024
SyllableLM: Learning Coarse Semantic Units for Speech Language Models Alan Baade Puyuan Peng David F. Harwath 39 3 0 05 Oct 2024
Recent Advances in Speech Language Models: A Survey Wenqian Cui Dianzhi Yu Xiaoqi Jiao Ziqiao Meng Guangyan Zhang Qichao Wang Yiwen Guo Irwin King AuLLM 57 14 0 01 Oct 2024
Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models Wenrui Liu Zhifang Guo Jin Xu Yuanjun Lv Yunfei Chu Zhou Zhao Junyang Lin 38 1 0 28 Sep 2024
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions Kai Chen Yunhao Gou Runhui Huang Zhili Liu Daxin Tan ... Qun Liu Jun Yao Lu Hou Hang Xu Hang Xu AuLLM MLLM VLM 58 21 0 26 Sep 2024
Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM Robin Shing-Hei Yuen Timothy Tin-Long Tse Jian Zhu AuLLM 27 3 0 25 Sep 2024
StyleFusion TTS: Multimodal Style-control and Enhanced Feature Fusion for Zero-shot Text-to-speech Synthesis Zhiyong Chen Xinnuo Li Zhiqi Ai Shugong Xu DiffM 18 1 0 24 Sep 2024
Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models Haibin Wu Xuanjun Chen Yi-Cheng Lin Kaiwei Chang Jiawei Du ... Yi-Chiao Wu Xu Tan James Glass Shinji Watanabe Hung-yi Lee 16 0 0 21 Sep 2024
Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility Xiaoyu Liu Xu Li Joan Serra Santiago Pascual 16 3 0 14 Sep 2024
Text-To-Speech Synthesis In The Wild Jee-weon Jung Wangyou Zhang Soumi Maiti Yihan Wu Xin Wang ... Hye-jin Shim Nicholas W. D. Evans Joon Son Chung Shinnosuke Takamichi Shinji Watanabe 19 1 0 13 Sep 2024
LAST: Language Model Aware Speech Tokenization A. Turetzky Yossi Adi 21 2 0 05 Sep 2024
SoCodec: A Semantic-Ordered Multi-Stream Speech Codec for Efficient Language Model Based Text-to-Speech Synthesis Haohan Guo Fenglong Xie Kun Xie Dongchao Yang Dake Guo Xixin Wu Helen Meng 21 4 0 02 Sep 2024
MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer Yuancheng Wang Haoyue Zhan Liwei Liu Ruihong Zeng Haotian Guo Jiachen Zheng Qiang Zhang Shunsi Zhang Shunsi Zhang Zhizheng Wu 23 37 0 01 Sep 2024
Progressive Residual Extraction based Pre-training for Speech Representation Learning Tianrui Wang Jin Li Ziyang Ma Rui Cao Xie Chen ... Meng Ge Xiaobao Wang Yuguang Wang Jianwu Dang Nyima Tashi SSL 30 0 0 31 Aug 2024
SSDM: Scalable Speech Dysfluency Modeling Jiachen Lian Xuanru Zhou Z. Ezzes Jet M J Vonk Brittany Morin D. Baquirin Zachary Mille M. G. Tempini Gopala Anumanchipalli AuLLM 30 1 0 29 Aug 2024
WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling Shengpeng Ji Ziyue Jiang Xize Cheng Yifu Chen Minghui Fang ... Rongjie Huang Yidi Jiang Qian Chen Zhou Zhao Zhou Zhao VLM 45 32 0 29 Aug 2024
dMel: Speech Tokenization made Simple Richard He Bai Tatiana Likhomanenko Ruixiang Zhang Zijin Gu Zakaria Aldeneh Navdeep Jaitly 30 4 0 22 Jul 2024
Overview of Speaker Modeling and Its Applications: From the Lens of Deep Speaker Representation Learning Shuai Wang Zheng-Shou Chen Kong Aik Lee Yan-min Qian Haizhou Li 24 4 0 21 Jul 2024
Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations Kunal Dhawan Nithin Rao Koluguri Ante Jukić Ryan Langman Jagadeesh Balam Boris Ginsburg 36 1 0 03 Jul 2024
Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimization Yuchen Hu Chen Chen Siyin Wang Eng Siong Chng C. Zhang 43 3 0 02 Jul 2024
DASB -- Discrete Audio and Speech Benchmark Pooneh Mousavi Luca Della Libera J. Duret Artem Ploujnikov Cem Subakan Mirco Ravanelli 22 12 0 20 Jun 2024
Articulatory Encodec: Coding Speech through Vocal Tract Kinematics Cheol Jun Cho Peter Wu Tejas S. Prabhune Dhruv Agarwal Gopala K. Anumanchipalli 24 1 0 18 Jun 2024
UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learner Dongchao Yang Haohan Guo Yuanyuan Wang Rongjie Huang Xiang Li Xu Tan Xixin Wu Helen Meng AuLLM 36 15 0 14 Jun 2024
Autoregressive Diffusion Transformer for Text-to-Speech Synthesis Zhijun Liu Shuai Wang Sho Inoue Qibing Bai Haizhou Li DiffM 30 15 0 08 Jun 2024
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers Sanyuan Chen Shujie Liu Long Zhou Yanqing Liu Xu Tan Jinyu Li Sheng Zhao Yao Qian Furu Wei VLM 29 64 0 08 Jun 2024
Neural Codec-based Adversarial Sample Detection for Speaker Verification Xuanjun Chen Jiawei Du Haibin Wu Jyh-Shing Roger Jang Hung-yi Lee 16 2 0 07 Jun 2024
Addressing Index Collapse of Large-Codebook Speech Tokenizer with Dual-Decoding Product-Quantized Variational Auto-Encoder Haohan Guo Fenglong Xie Dongchao Yang Hui Lu Xixin Wu Helen Meng 42 6 0 05 Jun 2024
ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec Shengpeng Ji Jia-li Zuo Minghui Fang Siqi Zheng Qian Chen ... Ziyue Jiang Hai Huang Xize Cheng Rongjie Huang Zhou Zhao 45 7 0 03 Jun 2024
Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback Chen Chen Yuchen Hu Wen Wu Helin Wang Chng Eng Siong Chao Zhang 33 1 0 02 Jun 2024
SpeechAlign: Aligning Speech Generation to Human Preferences Dong Zhang Zhaowei Li Shimin Li Xin Zhang Pengyu Wang Yaqian Zhou Xipeng Qiu ALM AuLLM 32 4 0 08 Apr 2024
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models Zeqian Ju Yuancheng Wang Kai Shen Xu Tan Detai Xin ... Shikun Zhang Jiang Bian Lei He Jinyu Li Sheng Zhao DiffM 20 139 0 05 Mar 2024
Towards audio language modeling -- an overview Haibin Wu Xuanjun Chen Yi-Cheng Lin Kai-Wei Chang Ho-Lam Chung Alexander H. Liu Hung-yi Lee AuLLM 25 28 0 20 Feb 2024
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling Jun Zhan Junqi Dai Jiasheng Ye Yunhua Zhou Dong Zhang ... Jie Fu Tao Gui Tianxiang Sun Yugang Jiang Xipeng Qiu MLLM 21 114 0 19 Feb 2024
APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decoding Yang Ai Xiao-Hang Jiang Ye-Xin Lu Hui-Peng Du Zhenhua Ling 13 20 0 16 Feb 2024
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data Mateusz Lajszczak Guillermo Cámbara Yang Li Fatih Beyhan Arent van Korlaar ... Bartosz Putrycz Soledad López Gambino Kayeon Yoo Elena Sokolova Thomas Drugman LM&MA 22 71 0 12 Feb 2024
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models Yunfei Chu Jin Xu Xiaohuan Zhou Qian Yang Shiliang Zhang Zhijie Yan Chang Zhou Jingren Zhou AuLLM 16 263 0 14 Nov 2023