Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation

12 November 2022

Taylor Berg-Kirkpatrick

Shlomo Dubnov

CLIP

ArXiv PDF HTML

Papers citing "Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation"

50 / 343 papers shown

Title
Audio-visual Generalized Zero-shot Learning the Easy Way Shentong Mo Pedro Morgado 30 5 0 18 Jul 2024
Audio Conditioning for Music Generation via Discrete Bottleneck Features Simon Rouard Yossi Adi Jade Copet Axel Roebel Alexandre Défossez MGen 39 1 0 17 Jul 2024
OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces Zehan Wang Ziang Zhang Hang Zhang Luping Liu Rongjie Huang Xize Cheng Hengshuang Zhao Zhou Zhao 30 7 0 16 Jul 2024
LiteFocus: Accelerated Diffusion Inference for Long Audio Synthesis Zhenxiong Tan Xinyin Ma Gongfan Fang Xinchao Wang 23 3 0 15 Jul 2024
The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective Zhen Qin Daoyuan Chen Wenhao Zhang Liuyi Yao Yilun Huang Bolin Ding Yaliang Li Shuiguang Deng 48 5 0 11 Jul 2024
Video-to-Audio Generation with Hidden Alignment Manjie Xu Chenxing Li Yong Ren Rilin Chen Yu Gu Yu Gu Dong Yu Dong Yu DiffM VGen 43 11 0 10 Jul 2024
Read, Watch and Scream! Sound Generation from Text and Video Yujin Jeong Yunji Kim Sanghyuk Chun Jiyoung Lee VGen DiffM 25 11 0 08 Jul 2024
Sequential Contrastive Audio-Visual Learning Ioannis Tsiamas Santiago Pascual Chunghsin Yeh Joan Serra 28 2 0 08 Jul 2024
A Reference-free Metric for Language-Queried Audio Source Separation using Contrastive Language-Audio Pretraining Feiyang Xiao Jian Guan Qiaoxi Zhu Xubo Liu Wenbo Wang Shuhan Qi Kejia Zhang Jianyuan Sun Wenwu Wang 30 4 0 06 Jul 2024
MuDiT & MuSiT: Alignment with Colloquial Expression in Description-to-Song Generation Zihao Wang Haoxuan Liu Jiaxing Yu Tao Zhang Yan Liu K. Zhang 52 1 0 03 Jul 2024
PicoAudio: Enabling Precise Timestamp and Frequency Controllability of Audio Events in Text-to-audio Generation Zeyu Xie Xuenan Xu Zhizheng Wu Mengyue Wu 27 7 0 03 Jul 2024
AudioTime: A Temporally-aligned Audio-text Benchmark Dataset Zeyu Xie Xuenan Xu Zhizheng Wu Mengyue Wu AuLLM 40 5 0 03 Jul 2024
SpecMaskGIT: Masked Generative Modeling of Audio Spectrograms for Efficient Audio Synthesis and Beyond Marco Comunità Zhi-Wei Zhong Akira Takahashi Shiqi Yang Mengjie Zhao Koichi Saito Yukara Ikemiya Takashi Shibuya Shusuke Takahashi Yuki Mitsufuji 45 2 0 25 Jun 2024
Exploring compressibility of transformer based text-to-music (TTM) models Vasileios Moschopoulos Thanasis Kotsiopoulos Pablo Peso Parada Konstantinos Nikiforidis Alexandros Stergiadis Gerasimos Papakostas Md. Asif Jalal Jisi Zhang Anastasios Drosou Karthikeyan P. Saravanan 23 0 0 24 Jun 2024
AND: Audio Network Dissection for Interpreting Deep Acoustic Models Tung-Yu Wu Yu-Xiang Lin Tsui-Wei Weng 37 1 0 24 Jun 2024
Improving Text-To-Audio Models with Synthetic Captions Zhifeng Kong Sang-gil Lee Deepanway Ghosal Navonil Majumder Ambuj Mehrish Rafael Valle Soujanya Poria Bryan Catanzaro 45 11 0 18 Jun 2024
GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities Sreyan Ghosh Sonal Kumar Ashish Seth Chandra Kiran Reddy Evuru Utkarsh Tyagi S. Sakshi Oriol Nieto R. Duraiswami Dinesh Manocha AuLLM LRM 41 34 0 17 Jun 2024
Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation Or Tal Alon Ziv Itai Gat Felix Kreuk Yossi Adi 37 13 0 16 Jun 2024
MINT: a Multi-modal Image and Narrative Text Dubbing Dataset for Foley Audio Content Planning and Generation Ruibo Fu Shuchen Shi Hongming Guo Tao Wang Chunyu Qiang ... Zhiyong Wang Yukun Liu Xuefei Liu Shuai Zhang Guanjun Li VGen 19 0 0 15 Jun 2024
Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos Changan Chen Puyuan Peng Ami Baid Zihui Xue Wei-Ning Hsu David F. Harwath Kristen Grauman VGen 37 7 0 13 Jun 2024
Bridging Language Gaps in Audio-Text Retrieval Zhiyong Yan Heinrich Dinkel Yongqing Wang Jizhong Liu Junbo Zhang Yujun Wang Bin Wang VLM 27 4 0 11 Jun 2024
MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation Lu Li T. Zhang Zhiqi Bu Suyuchen Wang Huan He Jie Fu Yonghui Wu Jiang Bian Yong Chen Yoshua Bengio FedML MoMe 92 3 0 11 Jun 2024
BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification June-Woo Kim Miika Toikkanen Yera Choi Seoung-Eun Moon Ho-Young Jung 28 4 0 10 Jun 2024
Zero-Shot Audio Captioning Using Soft and Hard Prompts Yiming Zhang Xuenan Xu Ruoyi Du Haohe Liu Yuan Dong Zheng-Hua Tan Wenwu Wang Zhanyu Ma VLM 33 4 0 10 Jun 2024
Zero-Shot End-To-End Spoken Question Answering In Medical Domain Yanis Labrak Adel Moumen Richard Dufour Mickael Rouvier ELM LM&MA MedIm 29 0 0 09 Jun 2024
Contrastive Learning from Synthetic Audio Doppelgängers Manuel Cherep Nikhil Singh 24 1 0 09 Jun 2024
Retrieval Augmented Generation in Prompt-based Text-to-Speech Synthesis with Context-Aware Contrastive Language-Audio Pretraining Jinlong Xue Yayue Deng Yingming Gao Ya Li RALM VLM 26 4 0 06 Jun 2024
VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling Zeyue Tian Zhaoyang Liu Ruibin Yuan Jiahao Pan Xiaoqiang Huang Xu Tan Xu Tan Qifeng Chen Y. Guo VGen 97 16 0 06 Jun 2024
Audio Mamba: Bidirectional State Space Model for Audio Representation Learning Mehmet Hamza Erol Arda Senocak Jiu Feng Joon Son Chung Mamba 62 18 0 05 Jun 2024
Operational Latent Spaces Scott H. Hawley Austin R. Tackett 13 0 0 04 Jun 2024
M2D-CLAP: Masked Modeling Duo Meets CLAP for Learning General-purpose Audio-Language Representation Daisuke Niizumi Daiki Takeuchi Yasunori Ohishi Noboru Harada Masahiro Yasuda Shunsuke Tsubaki Keisuke Imoto VLM 31 5 0 04 Jun 2024
Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling Jinxing Zhou Dan Guo Yiran Zhong Meng Wang VLM 53 18 0 03 Jun 2024
Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching Yongqi Wang Wenxiang Guo Rongjie Huang Jia-Bin Huang Zehan Wang Fuming You Ruiqi Li Zhou Zhao VGen DiffM 26 11 0 01 Jun 2024
Creative Text-to-Audio Generation via Synthesizer Programming Manuel Cherep Nikhil Singh Jessica Shand 23 3 0 01 Jun 2024
DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generation Zachary Novack Julian McAuley Taylor Berg-Kirkpatrick Nicholas J. Bryan 26 7 0 30 May 2024
Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRI Che Liu Changde Du Xiaoyu Chen Huiguang He 22 2 0 29 May 2024
Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning Yixiao Zhang Yukara Ikemiya Woosung Choi Naoki Murata Marco A. Martínez Ramírez Liwei Lin Gus Xia Wei-Hsiang Liao Yuki Mitsufuji Simon Dixon 49 10 0 28 May 2024
Listenable Maps for Zero-Shot Audio Classifiers Francesco Paissan Luca Della Libera Mirco Ravanelli Cem Subakan 30 4 0 27 May 2024
C3LLM: Conditional Multimodal Content Generation Using Large Language Models Zixuan Wang Qinkai Duan Yu-Wing Tai Chi-Keung Tang 27 3 0 25 May 2024
SoundLoCD: An Efficient Conditional Discrete Contrastive Latent Diffusion Model for Text-to-Sound Generation Xinlei Niu Jing Zhang Christian J. Walder Charles Patrick Martin 19 2 0 24 May 2024
QA-MDT: Quality-aware Masked Diffusion Transformer for Enhanced Music Generation Chang Li Ruoyu Wang Lijuan Liu Jun Du Yixuan Sun Zilu Guo Zhenrong Zhang Yuan Jiang J. Gao Feng Ma 41 1 0 24 May 2024
Images that Sound: Composing Images and Sounds on a Single Canvas Ziyang Chen Daniel Geng Andrew Owens DiffM 48 9 0 20 May 2024
CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing Faegheh Sardari A. Mustafa Philip J. B. Jackson Adrian Hilton 14 2 0 17 May 2024
Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation Manh Luong Khai Nguyen Nhat Ho Reza Haf D.Q. Phung Lizhen Qu 22 12 0 16 May 2024
Dance Any Beat: Blending Beats with Visuals in Dance Video Generation Xuanchen Wang Heng Wang Dongnan Liu Weidong Cai 30 3 0 15 May 2024
Naturalistic Music Decoding from EEG Data via Latent Diffusion Models Emilian Postolache Natalia Polouliakh Hiroaki Kitano Akima Connelly Emanuele Rodolà Luca Cosmo Taketo Akama MedIm DiffM 35 2 0 15 May 2024
FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generation Jianyi Chen Wei Xue Xu Tan Zhen Ye Qi-fei Liu Yi-Ting Guo 37 2 0 13 May 2024
SonifyAR: Context-Aware Sound Generation in Augmented Reality Xia Su Jon E. Froehlich Eunyee Koh Chang Xiao 33 3 0 11 May 2024
FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion Zehan Wang Ziang Zhang Xize Cheng Rongjie Huang Luping Liu ... Haifeng Huang Yang Zhao Tao Jin Peng Gao Zhou Zhao 18 8 0 08 May 2024
Training-Free Deepfake Voice Recognition by Leveraging Large-Scale Pre-Trained Models Alessandro Pianese D. Cozzolino Giovanni Poggi L. Verdoliva 19 5 0 03 May 2024