ResearchTrend.AI
SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models
v1v2 (latest)

SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models

International Conference on Learning Representations (ICLR), 2023
Xin Zhang
Dong Zhang
Shimin Li
Yaqian Zhou
Xipeng Qiu
ArXiv (abs)PDFHTMLHuggingFace (1 upvotes)Github (560★)

Papers citing "SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models"

50 / 74 papers shown
PURE Codec: Progressive Unfolding of Residual Entropy for Speech Codec Learning
PURE Codec: Progressive Unfolding of Residual Entropy for Speech Codec Learning
J. Shi
H. Wang
William Chen
Chenda Li
Wangyou Zhang
Jinchuan Tian
Shinji Watanabe
140
0
0
27 Nov 2025
DUO-TOK: Dual-Track Semantic Music Tokenizer for Vocal-Accompaniment Generation
DUO-TOK: Dual-Track Semantic Music Tokenizer for Vocal-Accompaniment Generation
Rui Lin
Zhiyue Wu
Jiahe Le
Kangdi Wang
Weixiong Chen
Junyu Dai
Tao Jiang
168
1
0
25 Nov 2025
LongCat-Audio-Codec: An Audio Tokenizer and Detokenizer Solution Designed for Speech Large Language Models
LongCat-Audio-Codec: An Audio Tokenizer and Detokenizer Solution Designed for Speech Large Language Models
Xiaohan Zhao
Hongyu Xiang
Shengze Ye
Song Li
Zhengkun Tian
Guanyu Chen
Ke Ding
Guanglu Wan
AuLLM
179
1
0
17 Oct 2025
FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
Jiaqi Li
Y. Qian
Yuxuan Hu
Leying Zhang
Xiaofei Wang
Heng Lu
Manthan Thakker
Jinyu Li
Sheng Zhao
Zhizheng Wu
218
1
0
01 Oct 2025
Comprehend and Talk: Text to Speech Synthesis via Dual Language Modeling
Comprehend and Talk: Text to Speech Synthesis via Dual Language Modeling
Junjie Cao
Yichen Han
Ruonan Zhang
Xiaoyang Hao
Hongxiang Li
Shuaijiang Zhao
Yue Liu
Xiao-Ping Zhng
119
0
0
26 Sep 2025
StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs
StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs
Yuhan Song
Linhao Zhang
Chuhan Wu
Aiwei Liu
Wei Jia
Houfeng Wang
Xiao-bin Zhou
132
0
0
26 Sep 2025
AUV: Teaching Audio Universal Vector Quantization with Single Nested Codebook
AUV: Teaching Audio Universal Vector Quantization with Single Nested Codebook
Yihao Chen
Kai Hu
Long Zhou
Shulin Feng
Xusheng Yang
Hangting Chen
Xie Chen
155
2
0
26 Sep 2025
MBCodec:Thorough disentangle for high-fidelity audio compression
MBCodec:Thorough disentangle for high-fidelity audio compression
Ruonan Zhang
Xiaoyang Hao
Yichen Han
Junjie Cao
Yue Liu
Kai Zhang
111
1
0
21 Sep 2025
FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation
FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation
Luca Della Libera
Cem Subakan
Mirco Ravanelli
112
0
0
19 Sep 2025
DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners
DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners
Xiaoxue Luo
Jinwei Huang
Runyan Yang
Yingying Gao
Junlan Feng
Chao Deng
Shilei Zhang
142
2
0
11 Sep 2025
FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbot
FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbot
Kun Xie
Feiyu Shen
Junjie Li
Fenglong Xie
Xu Tang
Yao Hu
168
10
0
02 Sep 2025
Analysing the Language of Neural Audio Codecs
Analysing the Language of Neural Audio Codecs
J. S. Park
Shinnosuke Takamichi
David M. Chan
Shunsuke Kando
Yuki Saito
Hiroshi Saruwatari
88
0
0
01 Sep 2025
CodecBench: A Comprehensive Benchmark for Acoustic and Semantic Evaluation
CodecBench: A Comprehensive Benchmark for Acoustic and Semantic Evaluation
Ruifan Deng
Yitian Gong
Qinghui Gao
Luozhijie Jin
Qinyuan Cheng
Zhaoye Fei
Shimin Li
Xipeng Qiu
AuLLM
133
2
0
28 Aug 2025
TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling
TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling
Yuancheng Wang
Dekun Chen
Xueyao Zhang
Junan Zhang
Jiaqi Li
Zhizheng Wu
228
4
0
22 Aug 2025
Towards Reliable Audio Deepfake Attribution and Model Recognition: A Multi-Level Autoencoder-Based Framework
Towards Reliable Audio Deepfake Attribution and Model Recognition: A Multi-Level Autoencoder-Based Framework
Andrea Di Pierno
Luca Guarnera
D. Allegra
Sebastiano Battiato
182
0
0
04 Aug 2025
SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec
SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec
Chunyu Qiang
Haoyu Wang
Cheng Gong
Tianrui Wang
Ruibo Fu
...
Zhengqi Wen
C. Zhang
Longbiao Wang
Jianwu Dang
Jianhua Tao
136
5
0
04 Aug 2025
ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models
ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models
Kaizhi Qian
Xulin Fan
Junrui Ni
Slava Shechtman
M. Hasegawa-Johnson
Chuang Gan
Yang Zhang
149
0
0
27 Jul 2025
Step-Audio 2 Technical Report
Step-Audio 2 Technical Report
Boyong Wu
Chao Yan
Chen Hu
Cheng Yi
Chengli Feng
...
Yuanwei Lu
Yuchu Luo
Yuhe Yin
Yumeng Zhan
Y. Zhang
AuLLM
288
0
0
22 Jul 2025
DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE
DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE
Hang Shao
Heting Gao
Yunhang Shen
Jiawei Chen
Zuwei Long
Dong Yang
Ke Li
Xing Sun
AuLLMMoE
222
2
0
27 Jun 2025
MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation
MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation
Yakun Song
Jiawei Chen
Xiaobin Zhuang
Chenpeng Du
Ziyang Ma
...
Dongya Jia
Zhuo Chen
Yuping Wang
Yuping Wang
Xie Chen
192
3
0
31 May 2025
Impact of Frame Rates on Speech Tokenizer: A Case Study on Mandarin and English
Impact of Frame Rates on Speech Tokenizer: A Case Study on Mandarin and English
Haoyang Zhang
Hexin Liu
Xiangyu Zhang
Qiquan Zhang
Yuchen Hu
Junqi Zhao
Fei Tian
Xuerui Yang
Eng Siong Chng
Eng Siong Chng
418
0
0
20 May 2025
Universal Semantic Disentangled Privacy-preserving Speech Representation Learning
Universal Semantic Disentangled Privacy-preserving Speech Representation Learning
Biel Tura Vecino
Subhadeep Maji
Aravind Varier
Antonio Bonafonte
Ivan Valles
...
Roberto Barra-Chicote
Ariya Rastrow
C. Papayiannis
Volker Leutnant
Trevor Wood
307
0
0
19 May 2025
Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space
Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space
Zhengrui Ma
Yang Feng
Chenze Shao
Fandong Meng
Jie Zhou
Min Zhang
261
3
0
19 May 2025
Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities
Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities
Wei Wei
Jintao Guo
Shanshan Zhao
Minghao Fu
Lunhao Duan
...
Guo-Hua Wang
Qing-Guo Chen
Zhao Xu
Weihua Luo
Kaifu Zhang
DiffM
1.1K
30
0
05 May 2025
Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play
Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play
Yemin Shi
Yu Shu
Siwei Dong
Guangyi Liu
Jaward Sesay
Jingwen Li
Zhiting Hu
AuLLMVLM
263
2
0
05 May 2025
Deep Audio Watermarks are Shallow: Limitations of Post-Hoc Watermarking Techniques for Speech
Deep Audio Watermarks are Shallow: Limitations of Post-Hoc Watermarking Techniques for Speech
P. O'Reilly
Zeyu Jin
Jiaqi Su
Bryan Pardo
275
6
0
15 Apr 2025
TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling
TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling
Liang-Hsuan Tseng
Yi-Chang Chen
Kuan-Yi Lee
Da-shan Shiu
Hung-yi Lee
AuLLM
452
13
0
09 Apr 2025
UniWav: Towards Unified Pre-training for Speech Representation Learning and GenerationInternational Conference on Learning Representations (ICLR), 2025
Alexander H. Liu
Sang-gil Lee
Chao-Han Huck Yang
Yuan Gong
Yu-Chun Wang
James Glass
Rafael Valle
Bryan Catanzaro
SSL
268
4
0
02 Mar 2025
From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval
From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval
Jian Jia
Jingtong Gao
Ben Xue
Junhao Wang
Qingpeng Cai
Quan Chen
Xiangyu Zhao
Peng Jiang
Kun Gai
OffRL
328
6
0
18 Feb 2025
AudioMiXR: Spatial Audio Object Manipulation with 6DoF for Sound Design in Augmented Reality
AudioMiXR: Spatial Audio Object Manipulation with 6DoF for Sound Design in Augmented RealityProceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (IMWUT), 2025
Brandon Woodard
Margarita Geleta
Joseph J. LaViola Jr.
Andrea Fanelli
Rhonda Wilson
874
25
0
05 Feb 2025
SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characters
SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous CharactersComputer Vision and Pattern Recognition (CVPR), 2024
Jianping Jiang
Weiye Xiao
Zhengyu Lin
Han Zhang
Tianxiang Ren
Yang Gao
Zhiqian Lin
Zhongang Cai
Lei Yang
Ziwei Liu
338
8
0
29 Nov 2024
Scaling Transformers for Low-Bitrate High-Quality Speech Coding
Scaling Transformers for Low-Bitrate High-Quality Speech Coding
Julian Parker
Anton Smirnov
Jordi Pons
CJ Carr
Zack Zukowski
Zach Evans
Xubo Liu
305
52
0
29 Nov 2024
MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios
MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate ScenariosSpoken Language Technology Workshop (SLT), 2024
Xiao-Hang Jiang
Yang Ai
Rui Zheng
Hui-Peng Du
Ye-Xin Lu
Zhen-Hua Ling
271
10
0
01 Nov 2024
Optimizing Neural Speech Codec for Low-Bitrate Compression via Multi-Scale Encoding
Optimizing Neural Speech Codec for Low-Bitrate Compression via Multi-Scale Encoding
Peiji Yang
Fengping Wang
Yicheng Zhong
Huawei Wei
Zhisheng Wang
185
1
0
21 Oct 2024
Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant
Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant
Alan Dao
Dinh Bach Vu
Huy Hoang Ha
AuLLMVLM
340
6
0
20 Oct 2024
DM-Codec: Distilling Multimodal Representations for Speech Tokenization
DM-Codec: Distilling Multimodal Representations for Speech Tokenization
Md Mubtasim Ahasan
Md Fahim
Tasnim Mohiuddin
A. K. M. Mahbubur Rahman
Vasu Sharma
Tariq Iqbal
M. A. Amin
Md. Mofijul Islam
Amin Ahsan Ali
352
3
0
19 Oct 2024
Code Drift: Towards Idempotent Neural Audio Codecs
Code Drift: Towards Idempotent Neural Audio CodecsIEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024
P. O'Reilly
Prem Seetharaman
Jiaqi Su
Zeyu Jin
Bryan Pardo
925
3
0
14 Oct 2024
Graded Suspiciousness of Adversarial Texts to Human
Graded Suspiciousness of Adversarial Texts to Human
Shakila Mahjabin Tonni
Pedro Faustini
Mark Dras
AAML
206
0
0
06 Oct 2024
SyllableLM: Learning Coarse Semantic Units for Speech Language Models
SyllableLM: Learning Coarse Semantic Units for Speech Language ModelsInternational Conference on Learning Representations (ICLR), 2024
Alan Baade
Puyuan Peng
David Harwath
328
21
0
05 Oct 2024
Recent Advances in Speech Language Models: A Survey
Recent Advances in Speech Language Models: A SurveyAnnual Meeting of the Association for Computational Linguistics (ACL), 2024
Wenqian Cui
Dianzhi Yu
Xiaoqi Jiao
Ziqiao Meng
Guangyan Zhang
Qichao Wang
Yiwen Guo
Irwin King
AuLLM
537
62
0
01 Oct 2024
Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models
Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models
Wenrui Liu
Zhifang Guo
Jin Xu
Yuanjun Lv
Yunfei Chu
Zhou Zhao
Junyang Lin
213
4
0
28 Sep 2024
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid EmotionsComputer Vision and Pattern Recognition (CVPR), 2024
Kai Chen
Yunhao Gou
Runhui Huang
Zhili Liu
Daxin Tan
...
Qun Liu
Jun Yao
Lu Hou
Hang Xu
Hang Xu
AuLLMMLLMVLM
433
42
0
26 Sep 2024
Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM
Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM
Robin Shing-Hei Yuen
Timothy Tin-Long Tse
Jian Zhu
AuLLM
179
4
0
25 Sep 2024
StyleFusion TTS: Multimodal Style-control and Enhanced Feature Fusion for Zero-shot Text-to-speech Synthesis
StyleFusion TTS: Multimodal Style-control and Enhanced Feature Fusion for Zero-shot Text-to-speech SynthesisChinese Conference on Pattern Recognition and Computer Vision (CPRCV), 2024
Zhiyong Chen
Xinnuo Li
Zhiqi Ai
Shugong Xu
DiffM
192
3
0
24 Sep 2024
Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models
Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec modelsSpoken Language Technology Workshop (SLT), 2024
Haibin Wu
Xuanjun Chen
Yi-Cheng Lin
Kaiwei Chang
Jiawei Du
...
Yi-Chiao Wu
Xu Tan
James Glass
Shinji Watanabe
Hung-yi Lee
179
14
0
21 Sep 2024
Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility
Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved IntelligibilityIEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024
Xiaoyu Liu
Xu Li
Joan Serrà
Santiago Pascual
253
5
0
14 Sep 2024
Text-To-Speech Synthesis In The Wild
Text-To-Speech Synthesis In The Wild
Jee-weon Jung
Wangyou Zhang
Soumi Maiti
Yihan Wu
Xin Eric Wang
...
Hye-jin Shim
Nicholas W. D. Evans
Joon Son Chung
Shinnosuke Takamichi
Shinji Watanabe
390
3
0
13 Sep 2024
LAST: Language Model Aware Speech Tokenization
LAST: Language Model Aware Speech Tokenization
A. Turetzky
Yossi Adi
287
8
0
05 Sep 2024
SoCodec: A Semantic-Ordered Multi-Stream Speech Codec for Efficient Language Model Based Text-to-Speech Synthesis
SoCodec: A Semantic-Ordered Multi-Stream Speech Codec for Efficient Language Model Based Text-to-Speech SynthesisSpoken Language Technology Workshop (SLT), 2024
Haohan Guo
Fenglong Xie
Kun Xie
Dongchao Yang
Dake Guo
Xixin Wu
Helen Meng
177
11
0
02 Sep 2024
MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec TransformerInternational Conference on Learning Representations (ICLR), 2024
Yuancheng Wang
Haoyue Zhan
Liwei Liu
Ruihong Zeng
Haotian Guo
Jiachen Zheng
Qiang Zhang
Shunsi Zhang
Shunsi Zhang
Zhizheng Wu
440
148
0
01 Sep 2024