Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models

14 November 2023

Yunfei Chu

Jin Xu

Xiaohuan Zhou

Qian Yang

Shiliang Zhang

Zhijie Yan

Chang Zhou

Jingren Zhou

AuLLM

ArXiv PDF HTML

Papers citing "Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models"

50 / 211 papers shown

Title
Speech Translation Refinement using Large Language Models Huaixia Dou Xinyu Tian Xinglin Lyu Jie Zhu Junhui Li Lifan Guo 47 0 0 28 Jan 2025
Audio-Language Models for Audio-Centric Tasks: A survey Yi Su Jisheng Bai Qisheng Xu Kele Xu Yong Dou AuLLM 99 1 0 28 Jan 2025
DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data Ke-Han Lu Zhehuai Chen Szu-Wei Fu Chao-Han Huck Yang Jagadeesh Balam Boris Ginsburg Yu-Te Wang Hung-yi Lee AuLLM SyDa 97 5 0 28 Jan 2025
Baichuan-Omni-1.5 Technical Report Yadong Li J. Liu Tao Zhang Tao Zhang S. Chen ... Jianhua Xu Haoze Sun Mingan Lin Zenan Zhou Weipeng Chen AuLLM 67 10 0 28 Jan 2025
FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integration Kai-Tuo Xu Feng-Long Xie Xu Tang Yao Hu 59 4 0 24 Jan 2025
OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia Xuelong Geng Kun Wei Qijie Shao Shuiyun Liu Zhennan Lin ... Yuhang Dai Xinfa Zhu Yue Li Li Zhang Lei Xie 65 3 0 23 Jan 2025
SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words Junyi Ao Yuancheng Wang Xiaohai Tian Dekun Chen J. Zhang Lu Lu Y. Wang Haizhou Li Z. Wu AuLLM 73 16 0 17 Jan 2025
Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model Z. Ma Zhuo Chen Y. Wang Eng Siong Chng Xie Chen AuLLM LRM 62 7 0 13 Jan 2025
Audio-Language Datasets of Scenes and Events: A Survey Gijs Wijngaard Elia Formisano Michele Esposito M. Dumontier 74 2 0 10 Jan 2025
"Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer? Benjamin Z. Reichman Kartik Talamadupula 38 0 0 07 Jan 2025
Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison Tsz Kin Lam Marco Gaido Sara Papi L. Bentivogli Barry Haddow 29 0 0 04 Jan 2025
OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios Xize Cheng Dongjie Fu Xiaoda Yang Minghui Fang Ruofan Hu ... Rongjie Huang Linjun Li Yu Chen Tao Jin Zhou Zhao 41 1 0 03 Jan 2025
Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning Chun-Yi Kuan Hung-yi Lee AuLLM LRM 56 1 0 03 Jan 2025
TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization Chia-Yu Hung Navonil Majumder Zhifeng Kong Ambuj Mehrish Rafael Valle Bryan Catanzaro Soujanya Poria Bryan Catanzaro Soujanya Poria 52 4 0 30 Dec 2024
AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues Se Jin Park Yeonju Kim Hyeongseop Rha Bella Godiva Y. Ro 36 1 0 23 Dec 2024
A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future Shilin Sun Wenbin An Feng Tian Fang Nan Qidong Liu J. Liu N. Shah Ping Chen 78 2 0 18 Dec 2024
LLMs are Also Effective Embedding Models: An In-depth Overview Chongyang Tao Tao Shen Shen Gao Junshuo Zhang Zhen Li Zhengwei Tao Shuai Ma 68 7 0 17 Dec 2024
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions Pan Zhang Xiaoyi Dong Yuhang Cao Yuhang Zang Rui Qian ... X. Zhang K. Chen Yu Qiao D. Lin Jiaqi Wang KELM 84 12 0 12 Dec 2024
AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information? Kaixiong Gong Kaituo Feng B. Li Yibing Wang Mofan Cheng ... Jiaming Han Benyou Wang Yutong Bai Z. Yang Xiangyu Yue MLLM AuLLM VLM 79 5 0 03 Dec 2024
Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey Yunkai Dang Kaichen Huang Jiahao Huo Yibo Yan S. Huang ... Kun Wang Yong Liu Jing Shao Hui Xiong Xuming Hu LRM 96 14 0 03 Dec 2024
A Comparative Study of LLM-based ASR and Whisper in Low Resource and Code Switching Scenario Zheshu Song Z. Ma Yifan Yang Jianheng Zhuo Xie Chen 64 2 0 01 Dec 2024
Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition Yoshiki Masuyama Koichi Miyazaki Masato Murata Mamba 30 0 0 11 Nov 2024
BhasaAnuvaad: A Speech Translation Dataset for 13 Indian Languages Sparsh Jain Ashwin Sankar Devilal Choudhary Dhairya Suman Nikhil Narasimhan Mohammed Safi Ur Rahman Khan Anoop Kunchukuttan Mitesh M. Khapra Raj Dabre 32 2 0 07 Nov 2024
Sing-On-Your-Beat: Simple Text-Controllable Accompaniment Generations Quoc-Huy Trinh Minh-Van Nguyen Trong-Hieu Nguyen-Mau Khoa Tran Thanh Do 23 0 0 03 Nov 2024
DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models Heng-Jui Chang Hongyu Gong Changhan Wang James R. Glass Yu-An Chung 26 0 0 31 Oct 2024
Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Models Hao Yang Lizhen Qu Ehsan Shareghi Gholamreza Haffari AAML 36 3 0 31 Oct 2024
SpeechQE: Estimating the Quality of Direct Speech Translation HyoJung Han Kevin Duh Marine Carpuat 20 0 0 28 Oct 2024
MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark S. Sakshi Utkarsh Tyagi Sonal Kumar Ashish Seth Ramaneswaran Selvakumar Oriol Nieto R. Duraiswami Sreyan Ghosh Dinesh Manocha AuLLM ELM 65 19 0 24 Oct 2024
VoiceBench: Benchmarking LLM-Based Voice Assistants Yiming Chen Xianghu Yue Chen Zhang Xiaoxue Gao R. Tan H. Li ELM AuLLM 34 17 0 22 Oct 2024
Foundation Models for Rapid Autonomy Validation Alec Farid Peter Schleede Aaron Huang Christoffer Heckman 32 0 0 22 Oct 2024
Roadmap towards Superhuman Speech Understanding using Large Language Models Fan Bu Yuhao Zhang X. Wang Benyou Wang Q. Liu H. Li LM&MA ELM AuLLM 33 1 0 17 Oct 2024
Sound Check: Auditing Audio Datasets William Agnew Julia Barnett Annie Chu Rachel Hong Michael Feffer Robin Netzorg Harry H. Jiang Ezra Awumey Sauvik Das 31 1 0 17 Oct 2024
What Do Speech Foundation Models Not Learn About Speech? Abdul Waheed Hanin Atwany Bhiksha Raj Rita Singh SSL 27 1 0 16 Oct 2024
OMCAT: Omni Context Aware Transformer Arushi Goel Karan Sapra Matthieu Le Rafael Valle Andrew Tao Bryan Catanzaro MLLM VLM 16 0 0 15 Oct 2024
Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization Changli Tang Yixuan Li Yudong Yang Jimin Zhuang Guangzhi Sun Wei Li Z. Ma Chao Zhang 23 4 0 09 Oct 2024
Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See Phu Pham Phu Pham Kun Wan Yu-Jhe Li Zeliang Zhang Daniel Miranda Ajinkya Kale Ajinkya Kale Chenliang Xu 22 5 0 08 Oct 2024
An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment Hugo Malard Michel Olvera Stéphane Lathuilière S. Essid VLM 25 0 0 08 Oct 2024
MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Models Kaichen Huang Jiahao Huo Yibo Yan Kun Wang Yutao Yue Xuming Hu 31 2 0 07 Oct 2024
HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis Yuto Nishimura Takumi Hirose Masanari Ohi Hideki Nakayama Nakamasa Inoue VLM 26 1 0 06 Oct 2024
Self-Powered LLM Modality Expansion for Large Speech-Text Models Tengfei Yu Xuebo Liu Zhiyi Hou Liang Ding Dacheng Tao Min Zhang 32 0 0 04 Oct 2024
FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Model Yichen Lu Jiaqi Song Chao-Han Huck Yang Shinji Watanabe 16 0 0 03 Oct 2024
Distilling an End-to-End Voice Assistant Without Instruction Training Data William B. Held Ella Li Michael Joseph Ryan Weiyan Shi Yanzhe Zhang Diyi Yang AuLLM 29 8 0 03 Oct 2024
Open-vocabulary Multimodal Emotion Recognition: Dataset, Metric, and Benchmark Zheng Lian Haiyang Sun Licai Sun Lan Chen Haoyu Chen ... Rui Liu Shan Liang Ya Li Jiangyan Yi Jianhua Tao VLM 23 0 0 02 Oct 2024
Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech Wonjune Kang J. Jia Chunyang Wu Wei Zhou Egor Lakomkin ... Leda Sari Suyoun Kim Ke Li Jay Mahadeokar Ozlem Kalinli AuLLM 24 2 0 02 Oct 2024
VHASR: A Multimodal Speech Recognition System With Vision Hotwords Jiliang Hu Zuchao Li Ping Wang Haojun Ai Lefei Zhang Hai Zhao 16 0 0 01 Oct 2024
CoT-ST: Enhancing LLM-based Speech Translation with Multimodal Chain-of-Thought Yexing Du Ziyang Ma Yifan Yang Keqi Deng Xie Chen Bo Yang Yang Xiang Ming Liu Bing Qin LRM 21 6 0 29 Sep 2024
Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models Yiming Chen Xianghu Yue Xiaoxue Gao Chen Zhang L. F. D’Haro R. Tan Haizhou Li AuLLM 30 0 0 27 Sep 2024
KALE-LM: Unleash The Power Of AI For Science Via Knowledge And Logic Enhanced Large Model Weichen Dai Yezeng Chen Zijie Dai Zhijie Huang Y. Liu ... Chengli Zhong Xinhe Li Zeyu Wang Zhuoying Feng Yi Zhou 33 0 0 27 Sep 2024
Are Transformers in Pre-trained LM A Good ASR Encoder? An Empirical Study Keyu An Shiliang Zhang Zhijie Yan 18 0 0 26 Sep 2024
Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM Robin Shing-Hei Yuen Timothy Tin-Long Tse Jian Zhu AuLLM 30 3 0 25 Sep 2024