v1v2 (latest)

LLaMA-Omni: Seamless Speech Interaction with Large Language Models

International Conference on Learning Representations (ICLR), 2024

10 September 2024

Qingkai Fang

Shoutao Guo

Yan Zhou

Zhengrui Ma

Shaolei Zhang

Yang Feng

AuLLM

ArXiv (abs)PDF HTML HuggingFace (58 upvotes)

Papers citing "LLaMA-Omni: Seamless Speech Interaction with Large Language Models"

36 / 86 papers shown

Title
VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation Puyuan Peng Shang-Wen Li Abdelrahman Mohamed David Harwath 183 1 0 26 May 2025
Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling Haiyang Sun Shujie Hu Shujie Liu L. Meng Hui Wang ... Yifan Yang Yanqing Liu Sheng Zhao Yan Lu Y. Qian 268 4 0 26 May 2025
Speechless: Speech Instruction Training Without Speech for Low Resource Languages Alan Dao Dinh Bach Vu Huy Hoang Ha Tuan Le Duc Anh Shreyas Gopal Yue Heng Yeo Warren Keng Hoong Low Eng Siong Chng J. Yip SyDa 265 3 0 23 May 2025
JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models Zifan Peng Yule Liu Zhen Sun Mingchen Li Zeren Luo ... Xinlei He Xuechao Wang Yingjie Xue Shengmin Xu Xinyi Huang AuLLM AAML 418 4 0 23 May 2025
VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Models Heyang Liu Yuhao Wang Ziyang Cheng Ronghua Wu Qunshan Gu Yanfeng Wang Yu Wang AuLLM 232 9 0 21 May 2025
Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models Zirui Song Qian Jiang Mingxuan Cui Mingzhe Li Lang Gao ... Zixiang Xu Chenxi Wang Guangxian Ouyang Zhenhao Chen Xiuying Chen AuLLM AAML 236 8 0 21 May 2025
SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information Chih-Kai Yang Neo Ho Yen-Ting Piao Hung-yi Lee AuLLM LRM 478 18 0 19 May 2025
SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation Wenyi Yu Siyin Wang Xiaoyu Yang Xianzhao Chen Xiaohai Tian Jun Zhang Guangzhi Sun Lu Lu Yuping Wang Chao Zhang AuLLM 218 8 0 17 May 2025
WavReward: Spoken Dialogue Models With Generalist Reward Evaluators Shengpeng Ji Tianle Liang Yongqian Li Jialong Zuo Minghui Fang ... Xize Cheng Siqi Zheng Jin Xu Junyang Lin Zhou Zhao AuLLM ALM 341 3 0 14 May 2025
VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model Zuwei Long Chunjiang Ge Chaoyou Fu Heting Gao Lijiang Li ... Jinlong Peng Haoyu Cao Ke Li Rongrong Ji Xing Sun 207 15 0 06 May 2025
Kimi-Audio Technical Report KimiTeam Ding Ding Zeqian Ju Yichong Leng Shixuan Liu ... Zhiyong Yang Aoxiong Yin Ruibin Yuan Yanzhe Zhang Zaida Zhou AuLLM VLM 392 113 0 25 Apr 2025
VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models Weiye Xu Jun Wang Weiyun Wang Zhe Chen Wengang Zhou ... Xiaohua Wang Xizhou Zhu Wenhai Wang Jifeng Dai Jinguo Zhu VLM LRM 392 39 0 21 Apr 2025
On The Landscape of Spoken Language Models: A Comprehensive Survey Siddhant Arora Kai-Wei Chang Chung-Ming Chien Yifan Peng Haibin Wu Yossi Adi Emmanuel Dupoux Hung-yi Lee Karen Livescu Shinji Watanabe 309 55 0 11 Apr 2025
TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling Liang-Hsuan Tseng Yi-Chang Chen Kuan-Yi Lee Da-shan Shiu Hung-yi Lee AuLLM 404 12 0 09 Apr 2025
VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation Yuhao Wang Heyang Liu Ziyang Cheng Ronghua Wu Qunshan Gu Yanfeng Wang Yu Wang 903 18 0 05 Apr 2025
Scaling Analysis of Interleaved Speech-Text Language Models Gallil Maimon Michael Hassid Amit Roth Yossi Adi AuLLM 364 4 0 03 Apr 2025
Vision-Speech Models: Teaching Speech Models to Converse about Images Amélie Royer Moritz Böhle Gabriel de Marmiesse Laurent Mazaré Neil Zeghidour Alexandre Défossez P. Pérez AuLLM VLM 247 1 0 19 Mar 2025
Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs Umberto Cappellazzo Minsu Kim Stavros Petridis 339 7 0 09 Mar 2025
Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision Che Liu Yingji Zhang D. Zhang Weijie Zhang Chenggong Gong ... Junwei Liao Haipang Wu Ji Liu André Freitas Qifan Wang AuLLM 523 6 0 26 Feb 2025
Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction Tianpeng Li Qingbin Liu Tao Zhang Yuanbo Fang Zheng Liang ... Bin Cui Jianhua Xu Haoze Sun Guosheng Dong Xin Wu AuLLM 246 48 0 24 Feb 2025
Soundwave: Less is More for Speech-Text Alignment in LLMsAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Yunke Zhang Zhiheng Liu Fan Bu Ruiyu Zhang Benyou Wang Haoyang Li AuLLM SyDa VLM 258 7 0 18 Feb 2025
Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction Ailin Huang Boyong Wu Bruce Wang Chao Yan Chen Hu ... Jiashuo Liu Wenjin Deng Wuxun Xie Weipeng Ming Wenqing He AuLLM 267 64 0 17 Feb 2025
HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding Jiaxing Zhao Q. Yang Yixing Peng Detao Bai Shimin Yao ... Xiang Chen Shenghao Fu Weixuan chen Xihan Wei Liefeng Bo VGen AuLLM 276 27 0 28 Jan 2025
Baichuan-Omni-1.5 Technical Report Yadong Li Qingbin Liu Tao Zhang Tao Zhang Tian Jin ... Jianhua Xu Haoze Sun Mingan Lin Guosheng Dong Xin Wu AuLLM 316 61 0 28 Jan 2025
AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language ModelsInternational Conference on Learning Representations (ICLR), 2024 Mintong Kang Chejian Xu Yue Liu AAML AuLLM 237 18 0 11 Dec 2024
Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learners Ze Yuan Yanqing Liu Shujie Liu Sheng Zhao AuLLM 257 6 0 06 Dec 2024
SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation Wenyi Yu Siyin Wang Xiaoyu Yang Xianzhao Chen Xiaohai Tian Jing Zhang Guangzhi Sun Lu Lu Longji Xu Chao Zhang AuLLM 320 22 0 27 Nov 2024
Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM Xiong Wang Yangze Li Chaoyou Fu Chunjiang Ge Lei Xie Ke Li Xing Sun Long Ma AuLLM MLLM 367 98 0 01 Nov 2024
Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant Alan Dao Dinh Bach Vu Huy Hoang Ha AuLLM VLM 285 6 0 20 Oct 2024
Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning - A Convex Optimization Perspective H. Fernando Han Shen Parikshit Ram Yi Zhou Horst Samulowitz Nathalie Baracaldo Tianyi Chen CLL 414 10 0 20 Oct 2024
IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities Xin Zhang Xiang Lyu Zhihao Du Qian Chen Dong Zhang ... Yuxuan Wang Bin Zhang Heng Lu Yaqian Zhou Jiaqi Leng AuLLM 252 14 0 09 Oct 2024
Recent Advances in Speech Language Models: A SurveyAnnual Meeting of the Association for Computational Linguistics (ACL), 2024 Wenqian Cui Dianzhi Yu Xiaoqi Jiao Ziqiao Meng Guangyan Zhang Qichao Wang Yiwen Guo Irwin King AuLLM 473 62 0 01 Oct 2024
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid EmotionsComputer Vision and Pattern Recognition (CVPR), 2024 Kai Chen Yunhao Gou Runhui Huang Zhili Liu Daxin Tan ... Qun Liu Jun Yao Lu Hou Hang Xu Hang Xu AuLLM MLLM VLM 397 42 0 26 Sep 2024
MIO: A Foundation Model on Multimodal Tokens Zekun Wang King Zhu Chunpu Xu Wangchunshu Zhou Jiaheng Liu ... Yuanxing Zhang Ge Zhang Ke Xu Jie Fu Wenhao Huang MLLM AuLLM 422 20 0 26 Sep 2024
SpeechVerse: A Large-scale Generalizable Audio Language Model Nilaksh Das Saket Dingliwal S. Ronanki Rohit Paturi David Huang ... Monica Sunkara S. Srinivasan Kyu J. Han Katrin Kirchhoff Katrin Kirchhoff 376 67 0 14 May 2024
The Art of Deception: Robust Backdoor Attack using Dynamic Stacking of Triggers Orson Mengara AAML 289 4 0 03 Jan 2024