SALMONN: Towards Generic Hearing Abilities for Large Language Models

20 October 2023

Changli Tang

Wenyi Yu

Guangzhi Sun

Papers citing "SALMONN: Towards Generic Hearing Abilities for Large Language Models"

50 / 161 papers shown

Title
Multi-Domain Audio Question Answering Toward Acoustic Content Reasoning in The DCASE 2025 Challenge Chao-Han Huck Yang Sreyan Ghosh Qing Wang Jaeyeon Kim Hengyi Hong ... Dinesh Manocha Gunhee Kim Jun Du Rafael Valle Bryan Catanzaro 14 0 0 12 May 2025
BLAB: Brutally Long Audio Bench Orevaoghene Ahia Martijn Bartelds Kabir Ahuja Hila Gonen Valentin Hofmann ... Noah Bennett Shinji Watanabe Noah A. Smith Yulia Tsvetkov Sachin Kumar AuLLM LM&MA VLM 38 0 0 05 May 2025
Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play Yemin Shi Yu Shu Siwei Dong Guangyi Liu Jaward Sesay Jingwen Li Zhiting Hu AuLLM VLM 32 0 0 05 May 2025
Kimi-Audio Technical Report KimiTeam Ding Ding Zeqian Ju Yichong Leng S. Liu ... Z. Yang Aoxiong Yin Ruibin Yuan Y. Zhang Zaida Zhou AuLLM VLM 105 1 0 25 Apr 2025
SARI: Structured Audio Reasoning via Curriculum-Guided Reinforcement Learning Cheng Wen Tingwei Guo Shuaijiang Zhao Wei Zou Xiangang Li OffRL AuLLM LRM 45 1 0 22 Apr 2025
SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning Prabhat Pandey R. Swaminathan K V Vijay Girish Arunasish Sen Jian Xie Grant P. Strimel Andreas Schwarz 29 0 0 12 Apr 2025
On The Landscape of Spoken Language Models: A Comprehensive Survey Siddhant Arora Kai-Wei Chang Chung-Ming Chien Yifan Peng Haibin Wu Yossi Adi Emmanuel Dupoux Hung-yi Lee Karen Livescu Shinji Watanabe 36 1 0 11 Apr 2025
Spatial Audio Processing with Large Language Model on Wearable Devices Ayushi Mishra Yang Bai Priyadarshan Narayanasamy Nakul Garg Nirupam Roy 25 0 0 11 Apr 2025
Scaling Analysis of Interleaved Speech-Text Language Models Gallil Maimon Michael Hassid Amit Roth Yossi Adi AuLLM 35 0 0 03 Apr 2025
Exploring In-Context Learning Capabilities of ChatGPT for Pathological Speech Detection Mahdi Amiri Hatef Otroshi Shahreza Ina Kodrasi 32 0 0 31 Mar 2025
Towards Understanding How Knowledge Evolves in Large Vision-Language Models Sudong Wang Y. Zhang Yao Zhu Jianing Li Zizhe Wang Y. Liu Xiangyang Ji 29 0 0 31 Mar 2025
Scaling Auditory Cognition via Test-Time Compute in Audio Language Models Ting Dang Yan Gao Hong Jia 34 0 0 30 Mar 2025
FinAudio: A Benchmark for Audio Large Language Models in Financial Applications Yupeng Cao Haohang Li Yangyang Yu Shashidhar Reddy Javaji Yueru He ... Xiao-Yang Liu K. P. Subbalakshmi Meikang Qiu Sophia Ananiadou J. Nie AuLLM 57 0 0 26 Mar 2025
Qwen2.5-Omni Technical Report Jin Xu Zhifang Guo Jinzheng He Hangrui Hu Ting He ... K. Dang Bin Zhang X. Wang Yunfei Chu Junyang Lin VGen AuLLM 86 12 0 26 Mar 2025
Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering Gang Li Jizhong Liu Heinrich Dinkel Yadong Niu Junbo Zhang Jian Luan ReLM OffRL LRM 52 5 0 14 Mar 2025
Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Model Ali Vosoughi Dimitra Emmanouilidou H. Gamper 45 0 0 12 Mar 2025
Mellow: a small audio language model for reasoning Soham Deshmukh Satvik Dixit Rita Singh Bhiksha Raj AuLLM ReLM LRM 67 1 0 11 Mar 2025
Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities Sreyan Ghosh Zhifeng Kong Sonal Kumar S. Sakshi Jaehyeon Kim Wei Ping Rafael Valle Dinesh Manocha Bryan Catanzaro MLLM AuLLM LRM 43 4 0 06 Mar 2025
LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant Wei Li Bing Hu Rui Shao Leyang Shen Liqiang Nie 28 2 0 05 Mar 2025
Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models Zhifei Xie Mingbao Lin Z. Liu Pengcheng Wu Shuicheng Yan Chunyan Miao AuLLM OffRL LRM 66 5 0 04 Mar 2025
Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics Siddhant Arora Zhiyun Lu Chung-Cheng Chiu Ruoming Pang Shinji Watanabe 40 2 0 03 Mar 2025
UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation Alexander H. Liu Sang-gil Lee Chao-Han Huck Yang Yuan Gong Yu-Chun Wang James Glass Rafael Valle Bryan Catanzaro SSL 35 0 0 02 Mar 2025
Retrieval-Augmented Speech Recognition Approach for Domain Challenges Peng Shen Xugang Lu Hisashi Kawai RALM 55 0 0 24 Feb 2025
Introducing Visual Perception Token into Multimodal Large Language Model Runpeng Yu Xinyin Ma Xinchao Wang MLLM LRM 66 0 0 24 Feb 2025
Audio-FLAN: A Preliminary Release Liumeng Xue Ziya Zhou J. Pan Z. Li Shuai Fan ... Haohe Liu Emmanouil Benetos Ge Zhang Yike Guo Wei Xue MLLM AuLLM CLIP VLM 52 1 0 23 Feb 2025
Soundwave: Less is More for Speech-Text Alignment in LLMs Y. Zhang Zhiheng Liu Fan Bu Ruiyu Zhang Benyou Wang H. Li AuLLM SyDa VLM 98 0 0 18 Feb 2025
DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data Ke-Han Lu Zhehuai Chen Szu-Wei Fu Chao-Han Huck Yang Jagadeesh Balam Boris Ginsburg Yu-Te Wang Hung-yi Lee AuLLM SyDa 79 5 0 28 Jan 2025
Audio-Language Models for Audio-Centric Tasks: A survey Yi Su Jisheng Bai Qisheng Xu Kele Xu Yong Dou AuLLM 93 1 0 28 Jan 2025
Speech Translation Refinement using Large Language Models Huaixia Dou Xinyu Tian Xinglin Lyu Jie Zhu Junhui Li Lifan Guo 35 0 0 28 Jan 2025
OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia Xuelong Geng Kun Wei Qijie Shao Shuiyun Liu Zhennan Lin ... Yuhang Dai Xinfa Zhu Yue Li Li Zhang Lei Xie 57 3 0 23 Jan 2025
LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations Soumya Dutta Sriram Ganapathy 29 2 0 20 Jan 2025
SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words Junyi Ao Yuancheng Wang Xiaohai Tian Dekun Chen J. Zhang Lu Lu Y. Wang Haizhou Li Z. Wu AuLLM 66 16 0 17 Jan 2025
Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer Siyuan Hou Shansong Liu Ruibin Yuan Wei Xue Ying Shan Mangsuo Zhao Chao Zhang 77 3 0 17 Jan 2025
Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model Z. Ma Zhuo Chen Y. Wang Eng Siong Chng Xie Chen AuLLM LRM 59 7 0 13 Jan 2025
Audio-Language Datasets of Scenes and Events: A Survey Gijs Wijngaard Elia Formisano Michele Esposito M. Dumontier 61 2 0 10 Jan 2025
Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition Rui Liu Hongyu Yuan H. Li 33 0 0 03 Jan 2025
Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning Chun-Yi Kuan Hung-yi Lee AuLLM LRM 42 1 0 03 Jan 2025
OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios Xize Cheng Dongjie Fu Xiaoda Yang Minghui Fang Ruofan Hu ... Rongjie Huang Linjun Li Yu Chen Tao Jin Zhou Zhao 38 1 0 03 Jan 2025
TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization Chia-Yu Hung Navonil Majumder Zhifeng Kong Ambuj Mehrish Rafael Valle Bryan Catanzaro Soujanya Poria Bryan Catanzaro Soujanya Poria 43 4 0 30 Dec 2024
AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues Se Jin Park Yeonju Kim Hyeongseop Rha Bella Godiva Y. Ro 29 1 0 23 Dec 2024
SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization Tan-Hanh Pham Hoang-Nam Le Phu-Vinh Nguyen Chris Ngo Truong Son-Hy AuLLM LRM 70 1 0 21 Dec 2024
A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future Shilin Sun Wenbin An Feng Tian Fang Nan Qidong Liu J. Liu N. Shah Ping Chen 68 2 0 18 Dec 2024
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions Pan Zhang Xiaoyi Dong Yuhang Cao Yuhang Zang Rui Qian ... X. Zhang K. Chen Yu Qiao D. Lin Jiaqi Wang KELM 78 10 0 12 Dec 2024
Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition Zhisheng Zhong Chengyao Wang Yuqi Liu Senqiao Yang Longxiang Tang ... Shaozuo Yu Sitong Wu Eric Lo Shu-Lin Liu Jiaya Jia AuLLM 95 6 0 12 Dec 2024
AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information? Kaixiong Gong Kaituo Feng B. Li Yibing Wang Mofan Cheng ... Jiaming Han Benyou Wang Yutong Bai Z. Yang Xiangyu Yue MLLM AuLLM VLM 69 5 0 03 Dec 2024
A Comparative Study of LLM-based ASR and Whisper in Low Resource and Code Switching Scenario Zheshu Song Z. Ma Yifan Yang Jianheng Zhuo Xie Chen 62 2 0 01 Dec 2024
SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation Wenyi Yu Siyin Wang Xiaoyu Yang Xianzhao Chen Xiaohai Tian J. Zhang Guangzhi Sun Lu Lu Y. Wang Chao Zhang AuLLM 59 6 0 27 Nov 2024
State-Space Large Audio Language Models Saurabhchand Bhati Yuan Gong Leonid Karlinsky Hilde Kuehne Rogerio Feris James Glass 80 0 0 24 Nov 2024
Vision Language Models Are Few-Shot Audio Spectrogram Classifiers Satvik Dixit Laurie M. Heller Chris Donahue VLM 55 5 0 18 Nov 2024
Addressing Hallucinations in Language Models with Knowledge Graph Embeddings as an Additional Modality Viktoriia Chekalina Anton Razzigaev Elizaveta Goncharova Andrey Kuznetsov KELM 62 0 0 18 Nov 2024