SALMONN: Towards Generic Hearing Abilities for Large Language Models

20 October 2023

Changli Tang

Wenyi Yu

Guangzhi Sun

Papers citing "SALMONN: Towards Generic Hearing Abilities for Large Language Models"

50 / 161 papers shown

Title
Language Model Can Listen While Speaking Ziyang Ma Yakun Song Chenpeng Du Jian Cong Zhuo Chen Yuping Wang Y. Wang Xie Chen AuLLM 29 23 0 05 Aug 2024
StreamVoice+: Evolving into End-to-end Streaming Zero-shot Voice Conversion Zhichao Wang Yuanzhe Chen Xinsheng Wang Lei Xie Yuping Wang 16 1 0 05 Aug 2024
MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models Yunwen Xia Hui Fang Emmanouil Benetos Jie Zhang Chong Long Dmitry Bogdanov AuLLM 32 1 0 02 Aug 2024
A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks Jiaqi Wang Hanqi Jiang Yi-Hsueh Liu Chong Ma Xu-Yao Zhang ... Xin Zhang Wei Zhang Dinggang Shen Tianming Liu Shu Zhang VLM AI4TS 29 18 0 02 Aug 2024
SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data Yichen Lu Álvaro Huertas-García Xuankai Chang Hengwei Bian Soumi Maiti Shinji Watanabe 29 1 0 01 Aug 2024
CoMMIT: Coordinated Instruction Tuning for Multimodal Large Language Models Junda Wu Xintong Li Tong Yu Yu-Xiang Wang Xiang Chen Jiuxiang Gu Lina Yao Jingbo Shang Julian McAuley 29 0 0 29 Jul 2024
Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation Junda Wu Zachary Novack Amit Namburi Jiaheng Dai Hao-Wen Dong Zhouhang Xie Carol Chen Julian McAuley 27 0 0 29 Jul 2024
Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight Ziyuan Huang Kaixiang Ji Biao Gong Zhiwu Qing Qinglong Zhang Kecheng Zheng Jian Wang Jingdong Chen Ming Yang LRM 29 0 0 22 Jul 2024
Computer Audition: From Task-Specific Machine Learning to Foundation Models Andreas Triantafyllopoulos Iosif Tsangko Alexander Gebhard A. Mesaros Tuomas Virtanen Björn Schuller 34 4 0 22 Jul 2024
Overview of Speaker Modeling and Its Applications: From the Lens of Deep Speaker Representation Learning Shuai Wang Zheng-Shou Chen Kong Aik Lee Yan-min Qian Haizhou Li 19 4 0 21 Jul 2024
Qwen2-Audio Technical Report Yunfei Chu Jin Xu Qian Yang Haojie Wei Xipin Wei ... Yuanjun Lv Jinzheng He Junyang Lin Chang Zhou Jingren Zhou AuLLM VLM 24 100 0 15 Jul 2024
Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generation Chun-Yi Kuan Chih-Kai Yang Wei-Ping Huang Ke-Han Lu Hung-yi Lee 36 5 0 13 Jul 2024
Pronunciation Assessment with Multi-modal Large Language Models Kaiqi Fu Linkai Peng Nan Yang Shuran Zhou 27 1 0 12 Jul 2024
AffectGPT: Dataset and Framework for Explainable Multimodal Emotion Recognition Zheng Lian Haiyang Sun Licai Sun Jiangyan Yi Bin Liu Jianhua Tao 24 2 0 10 Jul 2024
Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Models Yi-Cheng Lin T. Lin Chih-Kai Yang Ke-Han Lu Wei-Chih Chen Chun-Yi Kuan Hung-yi Lee 18 1 0 09 Jul 2024
Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition Ye Bai Jingping Chen Jitong Chen Wei Chen Zhuo Chen ... Wanyi Zhang Yang Zhang Yawei Zhang Yijie Zheng Ming Zou AuLLM 29 19 0 05 Jul 2024
Controlling Whisper: Universal Acoustic Adversarial Attacks to Control Speech Foundation Models Vyas Raina Mark J. F. Gales AAML 25 0 0 05 Jul 2024
FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs Keyu An Qian Chen Chong Deng Zhihao Du Changfeng Gao ... Bin Zhang Qinglin Zhang Shiliang Zhang Nan Zhao Siqi Zheng AuLLM 19 5 0 04 Jul 2024
Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems Chin Yuen Kwok J. Yip Eng Siong Chng CLL 25 1 0 04 Jul 2024
Investigating Decoder-only Large Language Models for Speech-to-text Translation Chao-Wei Huang Hui Lu Hongyu Gong H. Inaguma Ilia Kulikov Ruslan Mavlyutov Sravya Popuri AuLLM LRM 37 6 0 03 Jul 2024
BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5 Zhehuai Chen He Huang Oleksii Hrinchuk Krishna C. Puvvada Nithin Rao Koluguri Piotr Żelasko Jagadeesh Balam Boris Ginsburg AuLLM RALM 21 10 0 28 Jun 2024
DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment Ke-Han Lu Zhehuai Chen Szu-Wei Fu He Huang Boris Ginsburg Yu-Chiang Frank Wang Hung-yi Lee VLM AuLLM 18 4 0 27 Jun 2024
Streaming Decoder-Only Automatic Speech Recognition with Discrete Speech Units: A Pilot Study Peikun Chen Sining Sun Changhao Shan Qing Yang Lei Xie 27 2 0 27 Jun 2024
Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions M. Baskar Andrew Rosenberg Bhuvana Ramabhadran Neeraj Gaur Zhong Meng 19 3 0 20 Jun 2024
Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding Jizhong Liu Gang Li Junbo Zhang Heinrich Dinkel Yongqing Wang Zhiyong Yan Yujun Wang Bin Wang AuLLM 31 2 0 19 Jun 2024
Instruction Data Generation and Unsupervised Adaptation for Speech Language Models Vahid Noroozi Zhehuai Chen Somshubra Majumdar Steve Huang Jagadeesh Balam Boris Ginsburg SyDa 21 3 0 18 Jun 2024
Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model Hayato Futami Siddhant Arora Yosuke Kashiwagi E. Tsunoo Shinji Watanabe 26 0 0 18 Jun 2024
Improving Text-To-Audio Models with Synthetic Captions Zhifeng Kong Sang-gil Lee Deepanway Ghosal Navonil Majumder Ambuj Mehrish Rafael Valle Soujanya Poria Bryan Catanzaro 37 3 0 18 Jun 2024
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning Zebang Cheng Zhi-Qi Cheng Jun-Yan He Jingdong Sun Kai Wang Yuxiang Lin Zheng Lian Xiaojiang Peng Alexander G. Hauptmann MLLM 16 14 0 17 Jun 2024
Large Language Models for Dysfluency Detection in Stuttered Speech Dominik Wagner Sebastian P. Bayerl Ilja Baumann K. Riedhammer Elmar Nöth Tobias Bocklet 27 3 0 16 Jun 2024
UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learner Dongchao Yang Haohan Guo Yuanyuan Wang Rongjie Huang Xiang Li Xu Tan Xixin Wu Helen Meng AuLLM 31 15 0 14 Jun 2024
DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding Suwon Shon Kwangyoun Kim Yi-Te Hsu Prashant Sridhar Shinji Watanabe Karen Livescu AuLLM 26 2 0 13 Jun 2024
Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models Chun-Yi Kuan Wei-Ping Huang Hung-yi Lee AuLLM 17 1 0 12 Jun 2024
Can Large Language Models Understand Spatial Audio? Changli Tang Wenyi Yu Guangzhi Sun Xianzhao Chen Tian Tan ... Jun Zhang Lu Lu Zejun Ma Yuxuan Wang Chao Zhang 20 4 0 12 Jun 2024
Prompting Large Language Models with Audio for General-Purpose Speech Summarization Wonjune Kang Deb Roy LRM 16 7 0 10 Jun 2024
MaLa-ASR: Multimedia-Assisted LLM-Based ASR Guanrou Yang Ziyang Ma Fan Yu Zhifu Gao Shiliang Zhang Xie Chen AuLLM 23 2 0 09 Jun 2024
What do MLLMs hear? Examining reasoning with text and sound components in Multimodal Large Language Models Enis Berk Çoban Michael I. Mandel Johanna Devaney AuLLM LRM 23 0 0 07 Jun 2024
Wav2Prompt: End-to-End Speech Prompt Generation and Tuning For LLM in Zero and Few-shot Learning Keqi Deng Guangzhi Sun Phil Woodland VLM 28 1 0 01 Jun 2024
The Evolution of Multimodal Model Architectures S. Wadekar Abhishek Chaurasia Aman Chadha Eugenio Culurciello 41 13 0 28 May 2024
CrossCheckGPT: Universal Hallucination Ranking for Multimodal Foundation Models Guangzhi Sun Potsawee Manakul Adian Liusie Kunat Pipatanakul Chao Zhang P. Woodland Mark J. F. Gales HILM MLLM 16 7 0 22 May 2024
Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets Xuelong Geng Tianyi Xu Kun Wei Bingshen Mu Hongfei Xue ... Pengcheng Guo Yuhang Dai Longhao Li Mingchen Shao Lei Xie 28 9 0 03 May 2024
MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition Zheng Lian Haiyang Sun Licai Sun Zhuofan Wen Siyuan Zhang ... Bin Liu Erik Cambria Guoying Zhao Björn W. Schuller Jianhua Tao VLM 23 11 0 26 Apr 2024
WavLLM: Towards Robust and Adaptive Speech Large Language Model Shujie Hu Long Zhou Shujie Liu Sanyuan Chen Hongkun Hao ... Xunying Liu Jinyu Li S. Sivasankaran Linquan Liu Furu Wei AuLLM 21 42 0 31 Mar 2024
An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM Wonkyun Kim Changin Choi Wonseok Lee Wonjong Rhee VLM 40 50 0 27 Mar 2024
A Multimodal Approach to Device-Directed Speech Detection with Large Language Models Dominik Wagner Alexander W. Churchill Siddharth Sigtia Panayiotis Georgiou Matt Mirsamadi Aarshee Mishra Erik Marchi 30 3 0 21 Mar 2024
Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities Zhifeng Kong Arushi Goel Rohan Badlani Wei Ping Rafael Valle Bryan Catanzaro AuLLM LM&MA MLLM 59 73 0 02 Feb 2024
BAT: Learning to Reason about Spatial Sounds with Large Language Models Zhisheng Zheng Puyuan Peng Ziyang Ma Xie Chen Eunsol Choi David F. Harwath LRM 23 13 0 02 Feb 2024
Large Language Models for Time Series: A Survey Xiyuan Zhang Ranak Roy Chowdhury Rajesh K. Gupta Jingbo Shang AI4TS 66 53 0 02 Feb 2024
OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer Yifan Peng Jinchuan Tian William Chen Siddhant Arora Brian Yan ... Kwanghee Choi Jiatong Shi Xuankai Chang Jee-weon Jung Shinji Watanabe VLM OSLM 6 38 0 30 Jan 2024
E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models Hongfei Xue Yuhao Liang Bingshen Mu Shiliang Zhang Mengzhe Chen Qian Chen Lei Xie AuLLM 11 9 0 31 Dec 2023