SALMONN: Towards Generic Hearing Abilities for Large Language Models

20 October 2023

Changli Tang

Wenyi Yu

Guangzhi Sun

Papers citing "SALMONN: Towards Generic Hearing Abilities for Large Language Models"

50 / 161 papers shown

Title
StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification Yichen He Yuan Lin Jianchao Wu Hanchong Zhang Yuchen Zhang Ruicheng Le VGen VLM 39 2 0 11 Nov 2024
The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities Zhaofeng Wu Xinyan Velocity Yu Dani Yogatama Jiasen Lu Yoon Kim AIFin 36 10 0 07 Nov 2024
Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward Shashi Kumar Iuliia Thorbecke Sergio Burdisso Esaú Villatoro-Tello M. Errecalde Kadri Hacioğlu Pradeep Rangappa P. Motlícek A. Ganapathiraju Andreas Stolcke 39 1 0 06 Nov 2024
DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models Heng-Jui Chang Hongyu Gong Changhan Wang James R. Glass Yu-An Chung 26 0 0 31 Oct 2024
Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation Maohao Shen Shun Zhang Jilong Wu Zhiping Xiu Ehab AlBadawy Yiting Lu M. Seltzer Qing He 26 2 0 27 Oct 2024
MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark S. Sakshi Utkarsh Tyagi Sonal Kumar Ashish Seth Ramaneswaran Selvakumar Oriol Nieto R. Duraiswami Sreyan Ghosh Dinesh Manocha AuLLM ELM 65 19 0 24 Oct 2024
Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant Alan Dao Dinh Bach Vu Huy Hoang Ha AuLLM VLM 57 3 0 20 Oct 2024
Roadmap towards Superhuman Speech Understanding using Large Language Models Fan Bu Yuhao Zhang X. Wang Benyou Wang Q. Liu H. Li LM&MA ELM AuLLM 30 1 0 17 Oct 2024
The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio Sicong Leng Yun Xing Zesen Cheng Yang Zhou Hang Zhang Xin Li Deli Zhao Shijian Lu Chunyan Miao Lidong Bing 20 5 0 16 Oct 2024
OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities L. Chen Hexiang Hu Mingda Zhang Y. Chen Zifeng Wang ... Pranav Shyam Tianyi Zhou Heng-Chiao Huang Ming Yang Boqing Gong 21 2 0 16 Oct 2024
Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization Changli Tang Yixuan Li Yudong Yang Jimin Zhuang Guangzhi Sun Wei Li Z. Ma Chao Zhang 13 4 0 09 Oct 2024
IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities Xin Zhang Xiang Lyu Zhihao Du Qian Chen Dong Zhang ... Yuxuan Wang Bin Zhang Heng Lu Yaqian Zhou Xipeng Qiu AuLLM 20 4 0 09 Oct 2024
MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Models Kaichen Huang Jiahao Huo Yibo Yan Kun Wang Yutao Yue Xuming Hu 23 2 0 07 Oct 2024
Distilling an End-to-End Voice Assistant Without Instruction Training Data William B. Held Ella Li Michael Joseph Ryan Weiyan Shi Yanzhe Zhang Diyi Yang AuLLM 29 8 0 03 Oct 2024
Open-vocabulary Multimodal Emotion Recognition: Dataset, Metric, and Benchmark Zheng Lian Haiyang Sun Licai Sun Lan Chen Haoyu Chen ... Rui Liu Shan Liang Ya Li Jiangyan Yi Jianhua Tao VLM 15 0 0 02 Oct 2024
Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech Wonjune Kang J. Jia Chunyang Wu Wei Zhou Egor Lakomkin ... Leda Sari Suyoun Kim Ke Li Jay Mahadeokar Ozlem Kalinli AuLLM 14 0 0 02 Oct 2024
Recent Advances in Speech Language Models: A Survey Wenqian Cui Dianzhi Yu Xiaoqi Jiao Ziqiao Meng Guangyan Zhang Qichao Wang Yiwen Guo Irwin King AuLLM 57 14 0 01 Oct 2024
Are Transformers in Pre-trained LM A Good ASR Encoder? An Empirical Study Keyu An Shiliang Zhang Zhijie Yan 13 0 0 26 Sep 2024
MIO: A Foundation Model on Multimodal Tokens Zekun Wang King Zhu Chunpu Xu Wangchunshu Zhou Jiaheng Liu ... Yuanxing Zhang Ge Zhang Ke Xu Jie Fu Wenhao Huang MLLM AuLLM 35 11 0 26 Sep 2024
How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not Francesco Verdini Pierfrancesco Melucci Stefano Perna Francesco Cariaggi Marco Gaido ... Marek Kasztelnik L. Bentivogli Sébastien Bratières P. Merialdo Simone Scardapane AuLLM 20 0 0 25 Sep 2024
Speech Recognition Rescoring with Large Speech-Text Foundation Models Prashanth Gurunath Shivakumar J. Kolehmainen Aditya Gourav Yi Gu Ankur Gandhe Ariya Rastrow I. Bulyko AuLLM 16 0 0 25 Sep 2024
Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation Siyin Wang Wenyi Yu Yudong Yang Changli Tang Yixuan Li ... Jun Zhang Guangzhi Sun Lu Lu Yuxuan Wang Chao Zhang AuLLM LM&MA 62 5 0 25 Sep 2024
MT2KD: Towards A General-Purpose Encoder for Speech, Speaker, and Audio Events Xiaoyu Yang Qiujia Li Chao Zhang P. Woodland 13 0 0 25 Sep 2024
Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs Yang Yuhang Peng Yizhou Eng Siong Chng Xionghu Zhong AuLLM AI4CE 14 0 0 24 Sep 2024
Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLM Fengrun Zhang Wang Geng Hukai Huang Cheng Yi He Qu He Qu AuLLM MoE 18 1 0 24 Sep 2024
OmniBench: Towards The Future of Universal Omni-Language Models Yizhi Li Ge Zhang Yinghao Ma Ruibin Yuan Kang Zhu ... Zhaoxiang Zhang Zachary Liu Emmanouil Benetos Wenhao Huang Chenghua Lin LRM 30 11 0 23 Sep 2024
What Are They Doing? Joint Audio-Speech Co-Reasoning Yingzhi Wang Pooneh Mousavi Artem Ploujnikov Mirco Ravanelli AuLLM 41 0 0 22 Sep 2024
EMMeTT: Efficient Multimodal Machine Translation Training Piotr Żelasko Zhehuai Chen Mengru Wang Daniel Galvez Oleksii Hrinchuk Shuoyang Ding Ke Hu Jagadeesh Balam Vitaly Lavrukhin Boris Ginsburg 15 1 0 20 Sep 2024
Ideal-LLM: Integrating Dual Encoders and Language-Adapted LLM for Multilingual Speech-to-Text Hongfei Xue Wei Ren Xuelong Geng Kun Wei Longhao Li Qijie Shao Linju Yang Kai Diao Lei Xie AuLLM 13 0 0 17 Sep 2024
Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models Potsawee Manakul Guangzhi Sun Warit Sirichotedumrong Kasima Tharnpipitchai Kunat Pipatanakul AuLLM 28 4 0 17 Sep 2024
Enhancing Multilingual Speech Generation and Recognition Abilities in LLMs with Constructed Code-switched Data Jing Xu Daxin Tan Jiaqi Wang Xiao Chen 14 0 0 17 Sep 2024
Chain-of-Thought Prompting for Speech Translation Ke Hu Zhehuai Chen Chao-Han Huck Yang Piotr Żelasko Oleksii Hrinchuk Vitaly Lavrukhin Jagadeesh Balam Boris Ginsburg LRM 21 2 0 17 Sep 2024
A Survey of Foundation Models for Music Understanding Wenjun Li Ying Cai Ziyang Wu Wenyi Zhang Yifan Chen ... Junwei Han Bao Ge Tianming Liu Lin Gan Tuo Zhang 40 0 0 15 Sep 2024
NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training Minglun Han Ye Bai Chen Shen Youjia Huang Mingkun Huang Zehua Lin Linhao Dong Lu Lu Yuxuan Wang 27 0 0 13 Sep 2024
Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions Lingwei Meng Shujie Hu Jiawen Kang Zhaoqing Li Yuejiao Wang Wenxuan Wu Xixin Wu Xunying Liu Helen Meng AuLLM 55 1 0 13 Sep 2024
Salmon: A Suite for Acoustic Language Model Evaluation Gallil Maimon Amit Roth Yossi Adi ELM AuLLM 49 5 0 11 Sep 2024
Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Models A. Sridhar Yinyi Guo Erik M. Visser AuLLM 20 0 0 10 Sep 2024
MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders W. Zhang Shuo Sun Bin Wang Xunlong Zou Zhuohan Liu Yingxu He Geyu Lin Nancy F. Chen A. Aw AuLLM 65 1 0 10 Sep 2024
LLaMA-Omni: Seamless Speech Interaction with Large Language Models Qingkai Fang Shoutao Guo Yan Zhou Zhengrui Ma Shaolei Zhang Yang Feng AuLLM 25 29 0 10 Sep 2024
Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue Junkai Wu Xulin Fan Bo-Ru Lu Xilin Jiang N. Mesgarani M. Hasegawa-Johnson Mari Ostendorf AuLLM ELM 53 0 0 07 Sep 2024
Comparing Discrete and Continuous Space LLMs for Speech Recognition Yaoxun Xu Shi-Xiong Zhang Jianwei Yu Zhiyong Wu Dong Yu AuLLM 14 3 0 01 Sep 2024
Advancing Multi-talker ASR Performance with Large Language Models Mohan Shi Zengrui Jin Yaoxun Xu Yong Xu Shi-Xiong Zhang Kun Wei Yiwen Shao Chunlei Zhang Dong Yu 13 0 0 30 Aug 2024
Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming Zhifei Xie Changqiao Wu AuLLM VGen VLM SyDa LRM 21 52 0 29 Aug 2024
WHISMA: A Speech-LLM to Perform Zero-shot Spoken Language Understanding Mohan Li Cong-Thanh Do Simon Keizer Youmna Farag Svetlana Stoyanchev R. Doddipatla 17 2 0 29 Aug 2024
SSDM: Scalable Speech Dysfluency Modeling Jiachen Lian Xuanru Zhou Z. Ezzes Jet M J Vonk Brittany Morin D. Baquirin Zachary Mille M. G. Tempini Gopala Anumanchipalli AuLLM 30 1 0 29 Aug 2024
WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling Shengpeng Ji Ziyue Jiang Xize Cheng Yifu Chen Minghui Fang ... Rongjie Huang Yidi Jiang Qian Chen Zhou Zhao Zhou Zhao VLM 36 32 0 29 Aug 2024
SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks Kai-Wei Chang Haibin Wu Yu-Kai Wang Yuan-Kuei Wu Hua Shen Wei-Cheng Tseng Iu-thing Kang Shang-Wen Li Hung-yi Lee 26 1 0 23 Aug 2024
Video Emotion Open-vocabulary Recognition Based on Multimodal Large Language Model Mengying Ge Dongkai Tang Mingyang Li VLM 12 1 0 21 Aug 2024
A Transcription Prompt-based Efficient Audio Large Language Model for Robust Speech Recognition Yangze Li Xiong Wang Songjun Cao Yike Zhang Long Ma Lei Xie AuLLM 43 0 0 18 Aug 2024
Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models Yi-Cheng Lin Wei-Chih Chen Hung-yi Lee 18 1 0 14 Aug 2024