SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

18 May 2023

Xipeng Qiu

Papers citing "SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities"

50 / 223 papers shown

Title
Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding Dianwen Ng Kun Zhou Yi-Wen Chao Zhiwei Xiong B. Ma E. Chng 14 0 0 12 May 2025
LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis Qingkai Fang Yan Zhou Shoutao Guo Shaolei Zhang Yang Feng AuLLM 48 0 0 05 May 2025
Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play Yemin Shi Yu Shu Siwei Dong Guangyi Liu Jaward Sesay Jingwen Li Zhiting Hu AuLLM VLM 32 0 0 05 May 2025
Symbolic Representation for Any-to-Any Generative Tasks J. Chen Xiaoye Zhu Y. Wang Tianyang Liu Xinhui Chen ... Yifei Ke J. Liu Yiwen Yuan Julian McAuley Li Li DiffM 29 0 0 24 Apr 2025
A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms Chengkai Huang Hongtao Huang Tong Yu Kaige Xie Junda Wu Shuai Zhang Julian McAuley Dietmar Jannach Lina Yao LRM AI4CE 17 0 0 23 Apr 2025
SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation Keqi Deng Wenxi Chen Xie Chen P. Woodland 38 0 0 22 Apr 2025
EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting Guanrou Yang Chen Yang Qian Chen Ziyang Ma Wenxi Chen ... Fan Yu Zhihao Du Zhifu Gao Shiliang Zhang Xie Chen AuLLM 46 0 0 17 Apr 2025
On The Landscape of Spoken Language Models: A Comprehensive Survey Siddhant Arora Kai-Wei Chang Chung-Ming Chien Yifan Peng Haibin Wu Yossi Adi Emmanuel Dupoux Hung-yi Lee Karen Livescu Shinji Watanabe 36 1 0 11 Apr 2025
Voice Interaction With Conversational AI Could Facilitate Thoughtful Reflection and Substantive Revision in Writing Jiho Kim Philippe Laban Xiang Ánthony' Chen Kenneth C. Arnold 30 0 0 11 Apr 2025
Scaling Analysis of Interleaved Speech-Text Language Models Gallil Maimon Michael Hassid Amit Roth Yossi Adi AuLLM 37 0 0 03 Apr 2025
SViQA: A Unified Speech-Vision Multimodal Model for Textless Visual Question Answering Bingxin Li 23 0 0 01 Apr 2025
Dolphin: A Large-Scale Automatic Speech Recognition Model for Eastern Languages Yangyang Meng Jinpeng Li Guodong Lin Yu Pu G. Wang Hu Du Zhiming Shao Yukai Huang Ke Li Wei-Qiang Zhang ObjD 88 0 0 26 Mar 2025
FinAudio: A Benchmark for Audio Large Language Models in Financial Applications Yupeng Cao Haohang Li Yangyang Yu Shashidhar Reddy Javaji Yueru He ... Xiao-Yang Liu K. P. Subbalakshmi Meikang Qiu Sophia Ananiadou J. Nie AuLLM 59 0 0 26 Mar 2025
CountLLM: Towards Generalizable Repetitive Action Counting via Large Language Model Ziyu Yao Xuxin Cheng Zhiqi Huang Lei Li 45 0 0 22 Mar 2025
Multi-modal Time Series Analysis: A Tutorial and Survey Yushan Jiang Kanghui Ning Zijie Pan Xuyang Shen Jingchao Ni Wenchao Yu Anderson Schneider Haifeng Chen Yuriy Nevmyvaka Dongjin Song AI4TS 53 0 0 17 Mar 2025
Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey Y. Wang Shengqiong Wu Y. Zhang William Yang Wang Ziwei Liu Jiebo Luo Hao Fei LRM 69 7 0 16 Mar 2025
Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations Xue Jiang Xiulian Peng Yuan Zhang Yan-Heng Lu SSL 70 0 0 15 Mar 2025
Adaptive Inner Speech-Text Alignment for LLM-based Speech Translation Henglyu Liu Andong Chen Kehai Chen X. Bai M. Zhong Yuan Qiu Min Zhang 32 0 0 13 Mar 2025
From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM Kshitij Ambilduke Ben Peters Sonal Sannigrahi Anil Keshwani Tsz Kin Lam Bruno Martins Marcely Zanon Boito André F. T. Martins 44 0 0 13 Mar 2025
3D Point Cloud Generation via Autoregressive Up-sampling Ziqiao Meng Qichao Wang Zhipeng Zhou Irwin King Peilin Zhao 3DPC 51 0 0 11 Mar 2025
ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems Siddhant Arora Yifan Peng Jiatong Shi Jinchuan Tian William Chen ... Yosuke Kashiwagi E. Tsunoo Shuichiro Shimizu Vaibhav Srivastav Shinji Watanabe 34 0 0 11 Mar 2025
Text-Speech Language Models with Improved Cross-Modal Transfer by Aligning Abstraction Levels Santiago Cuervo Adel Moumen Yanis Labrak Sameer Khurana Antoine Laurent Mickael Rouvier R. Marxer 67 1 0 08 Mar 2025
S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following with Paralinguistic Information Feng Jiang Zhiyu Lin Fan Bu Yuhao Du Benyou Wang H. Li AuLLM ELM 88 0 0 07 Mar 2025
LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM S. Mohammed Irfan Kurpath Sahal Shaji Mullappilly Jean Lahoud Fahad A Khan Rao Muhammad Anwer Salman Khan Hisham Cholakkal AuLLM 54 0 0 06 Mar 2025
Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cues Varsha Suresh Muhammad Hamza Mughal Christian Theobalt Vera Demberg 51 0 0 05 Mar 2025
InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-training Dingdong Wang Jin Xu Ruihang Chu Zhifang Guo X. Wang Jincenzi Wu Dongchao Yang Shengpeng Ji Junyang Lin AuLLM 78 0 0 04 Mar 2025
Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models Zhifei Xie Mingbao Lin Z. Liu Pengcheng Wu Shuicheng Yan Chunyan Miao AuLLM OffRL LRM 69 5 0 04 Mar 2025
UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation Alexander H. Liu Sang-gil Lee Chao-Han Huck Yang Yuan Gong Yu-Chun Wang James Glass Rafael Valle Bryan Catanzaro SSL 35 0 0 02 Mar 2025
LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement Boyi Kang Xinfa Zhu Zihan Zhang Zhen Ye Mingshuai Liu ... Jun Chen Longshuai Xiao Chao Weng Wei Xue Lei Xie AuLLM 55 3 0 01 Mar 2025
PodAgent: A Comprehensive Framework for Podcast Generation Yujia Xiao Lei He Haohan Guo Fenglong Xie Tan Lee 29 0 0 01 Mar 2025
Does Your Voice Assistant Remember? Analyzing Conversational Context Recall and Utilization in Voice Interaction Models Heeseung Kim Che Hyun Lee S. Park Jiheum Yeom Nohil Park Sangwon Yu Sungroh Yoon 59 0 0 27 Feb 2025
Nexus-O: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision Che Liu Yingji Zhang D. Zhang Weijie Zhang Chenggong Gong ... André Freitas Qifan Wang Z. Xu Rongjuncheng Zhang Yong Dai AuLLM 61 0 0 26 Feb 2025
Mind the Gap! Static and Interactive Evaluations of Large Audio Models Minzhi Li William B. Held Michael Joseph Ryan Kunat Pipatanakul Potsawee Manakul Hao Zhu Diyi Yang AuLLM ALM 51 0 0 21 Feb 2025
FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systems Borui Liao Yulong Xu Jiao Ou Kaiyuan Yang Weihua Jian Pengfei Wan Di Zhang AuLLM 57 0 0 20 Feb 2025
DuplexMamba: Enhancing Real-time Speech Conversations with Duplex and Streaming Capabilities Xiangyu Lu Wang Xu Haoyu Wang Hongyun Zhou Haiyan Zhao Conghui Zhu T. Zhao M. Yang Mamba AuLLM 45 0 0 16 Feb 2025
From No to Know: Taxonomy, Challenges, and Opportunities for Negation Understanding in Multimodal Foundation Models Mayank Vatsa Aparna Bharati S. Mittal Richa Singh 53 0 0 10 Feb 2025
Audio-Language Models for Audio-Centric Tasks: A survey Yi Su Jisheng Bai Qisheng Xu Kele Xu Yong Dou AuLLM 96 1 0 28 Jan 2025
A Comprehensive Survey of Foundation Models in Medicine Wasif Khan Seowung Leem Kyle B. See Joshua K. Wong Shaoting Zhang R. Fang AI4CE LM&MA VLM 93 16 0 17 Jan 2025
SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words Junyi Ao Yuancheng Wang Xiaohai Tian Dekun Chen J. Zhang Lu Lu Y. Wang Haizhou Li Z. Wu AuLLM 66 16 0 17 Jan 2025
OneLLM: One Framework to Align All Modalities with Language Jiaming Han Kaixiong Gong Yiyuan Zhang Jiaqi Wang Kaipeng Zhang D. Lin Yu Qiao Peng Gao Xiangyu Yue MLLM 96 102 0 10 Jan 2025
"Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer? Benjamin Z. Reichman Kartik Talamadupula 35 0 0 07 Jan 2025
Instruction-Guided Scene Text Recognition Yongkun Du Z. Chen Yuchen Su Caiyan Jia Yu-Gang Jiang 50 3 0 03 Jan 2025
OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios Xize Cheng Dongjie Fu Xiaoda Yang Minghui Fang Ruofan Hu ... Rongjie Huang Linjun Li Yu Chen Tao Jin Zhou Zhao 38 1 0 03 Jan 2025
AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues Se Jin Park Yeonju Kim Hyeongseop Rha Bella Godiva Y. Ro 29 1 0 23 Dec 2024
SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization Tan-Hanh Pham Hoang-Nam Le Phu-Vinh Nguyen Chris Ngo Truong Son-Hy AuLLM LRM 70 1 0 21 Dec 2024
The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion Changan Chen Juze Zhang S. K. Lakshmikanth Yusu Fang Ruizhi Shao Gordon Wetzstein L. Fei-Fei Ehsan Adeli VGen 65 3 0 13 Dec 2024
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions Pan Zhang Xiaoyi Dong Yuhang Cao Yuhang Zang Rui Qian ... X. Zhang K. Chen Yu Qiao D. Lin Jiaqi Wang KELM 78 12 0 12 Dec 2024
Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition Zhisheng Zhong Chengyao Wang Yuqi Liu Senqiao Yang Longxiang Tang ... Shaozuo Yu Sitong Wu Eric Lo Shu-Lin Liu Jiaya Jia AuLLM 95 6 0 12 Dec 2024
Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learners Ze Yuan Yanqing Liu Shujie Liu Sheng Zhao AuLLM 74 0 0 06 Dec 2024
SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characters Jianping Jiang Weiye Xiao Zhengyu Lin H. Zhang Tianxiang Ren Yang Gao Zhiqian Lin Zhongang Cai Lei Yang Ziwei Liu 79 3 0 29 Nov 2024