Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation

12 November 2022

Taylor Berg-Kirkpatrick

Shlomo Dubnov

CLIP

ArXiv PDF HTML

Papers citing "Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation"

50 / 343 papers shown

Title
Active Learning with Task Adaptation Pre-training for Speech Emotion Recognition Dongyuan Li Ying Zhang Yusong Wang Funakoshi Kataro Manabu Okumura 19 1 0 01 May 2024
T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining Yiitan Yuan Zhuo Chen Xubo Liu Haohe Liu Xuenan Xu Dongya Jia Yuanzhe Chen Mark D. Plumbley Wenwu Wang CLIP VLM 35 9 0 27 Apr 2024
HybridVC: Efficient Voice Style Conversion with Text and Audio Prompts Xinlei Niu Jing Zhang Charles Patrick Martin 18 2 0 24 Apr 2024
Music Consistency Models Zhengcong Fei Mingyuan Fan Junshi Huang DiffM 40 5 0 20 Apr 2024
Track Role Prediction of Single-Instrumental Sequences Changheon Han Suhyun Lee Minsam Ko 19 0 0 20 Apr 2024
GEOBIND: Binding Text, Image, and Audio through Satellite Images A. Dhakal Subash Khanal S. Sastry Adeel Ahmad Nathan Jacobs 28 2 0 17 Apr 2024
Long-form music generation with latent diffusion Zach Evans Julian Parker CJ Carr Zack Zukowski Josiah Taylor Jordi Pons MGen DiffM 33 38 0 16 Apr 2024
Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization Navonil Majumder Chia-Yu Hung Deepanway Ghosal Wei-Ning Hsu Rada Mihalcea Soujanya Poria 42 51 0 15 Apr 2024
Audio Dialogues: Dialogues dataset for audio and music understanding Arushi Goel Zhifeng Kong Rafael Valle Bryan Catanzaro AuLLM 29 4 0 11 Apr 2024
UniAV: Unified Audio-Visual Perception for Multi-Task Video Localization Tiantian Geng Teng Wang Yanfu Zhang Jinming Duan Weili Guan Feng Zheng 21 2 0 04 Apr 2024
Weakly-supervised Audio Separation via Bi-modal Semantic Similarity Tanvir Mahmud Saeed Amizadeh K. Koishida Diana Marculescu AI4TS 9 2 0 02 Apr 2024
SMITIN: Self-Monitored Inference-Time INtervention for Generative Music Transformers Junghyun Koo G. Wichern François G. Germain Sameer Khurana Jonathan Le Roux 18 3 0 02 Apr 2024
A Diffusion-Based Generative Equalizer for Music Restoration Eloi Moliner Maija Turunen Filip Elvander Vesa Valimaki 21 5 0 27 Mar 2024
Synthetic training set generation using text-to-audio models for environmental sound classification Francesca Ronchini Luca Comanducci Fabio Antonacci 28 2 0 26 Mar 2024
Correlation of Fréchet Audio Distance With Human Perception of Environmental Audio Is Embedding Dependant Modan Tailleur Junwon Lee Mathieu Lagrange Keunwoo Choi Laurie M. Heller Keisuke Imoto Yuki Okamoto 14 10 0 26 Mar 2024
Building speech corpus with diverse voice characteristics for its prompt-based representation Aya Watanabe Shinnosuke Takamichi Yuki Saito Wataru Nakata Detai Xin Hiroshi Saruwatari 16 0 0 20 Mar 2024
Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt Yongqi Wang Ruofan Hu Rongjie Huang Zhiqing Hong Ruiqi Li Wenrui Liu Fuming You Tao Jin Zhou Zhao 38 9 0 18 Mar 2024
Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models Emilian Postolache Giorgio Mariani Luca Cosmo Emmanouil Benetos Emanuele Rodolà DiffM 14 9 0 18 Mar 2024
Multiscale Matching Driven by Cross-Modal Similarity Consistency for Audio-Text Retrieval Qian Wang Jia-Chen Gu Zhen-Hua Ling 30 2 0 15 Mar 2024
Knowledge Conflicts for LLMs: A Survey Rongwu Xu Zehan Qi Zhijiang Guo Cunxiang Wang Hongru Wang Yue Zhang Wei Xu 198 91 0 13 Mar 2024
Text-to-Audio Generation Synchronized with Videos Shentong Mo Jing Shi Yapeng Tian DiffM VGen 34 17 0 08 Mar 2024
A Detailed Audio-Text Data Simulation Pipeline using Single-Event Sounds Xuenan Xu Xiaohang Xu Zeyu Xie Pingyue Zhang Mengyue Wu Kai Yu 20 6 0 07 Mar 2024
Time Weaver: A Conditional Time Series Generation Model Sai Shankar Narasimhan Shubhankar Agarwal Oguzhan Akcin Sujay Sanghavi Sandeep P. Chinchali DiffM MedIm 26 20 0 05 Mar 2024
Retrieval-Augmented Generation for AI-Generated Content: A Survey Penghao Zhao Hailin Zhang Qinhan Yu Zhengren Wang Yunteng Geng Fangcheng Fu Ling Yang Wentao Zhang Jie Jiang Bin Cui 3DV 110 215 0 29 Feb 2024
A SOUND APPROACH: Using Large Language Models to generate audio descriptions for egocentric text-audio retrieval Andreea-Maria Oncescu João F. Henriques Andrew Zisserman Samuel Albanie A. Sophia Koepke 17 5 0 29 Feb 2024
ArcSin: Adaptive ranged cosine Similarity injected noise for Language-Driven Visual Tasks Yang Liu Xiaomin Yu Gongyu Zhang Christos Bergeles Prokar Dasgupta Alejandro Granados Sebastien Ourselin 35 2 0 27 Feb 2024
EDTC: enhance depth of text comprehension in automated audio captioning Liwen Tan Yin Cao Yi Zhou 27 0 0 27 Feb 2024
Music Style Transfer with Time-Varying Inversion of Diffusion Models Sifei Li Yuxin Zhang Fan Tang Chongyang Ma Weiming Dong Changsheng Xu DiffM 29 11 0 21 Feb 2024
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling Jun Zhan Junqi Dai Jiasheng Ye Yunhua Zhou Dong Zhang ... Jie Fu Tao Gui Tianxiang Sun Yugang Jiang Xipeng Qiu MLLM 27 114 0 19 Feb 2024
Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion Hila Manor T. Michaeli DiffM 19 25 0 15 Feb 2024
Domain Adaptation for Contrastive Audio-Language Models Soham Deshmukh Rita Singh Bhiksha Raj VLM 27 7 0 14 Feb 2024
Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controls Liwei Lin Gus Xia Yixiao Zhang Junyan Jiang 11 12 0 14 Feb 2024
MINT: Boosting Audio-Language Model via Multi-Target Pre-Training and Instruction Tuning Hang Zhao Yifei Xin Zhesong Yu Bilei Zhu Lu Lu Zejun Ma AuLLM 26 4 0 12 Feb 2024
Cacophony: An Improved Contrastive Audio-Text Model Ge Zhu Jordan Darefsky Zhiyao Duan AuLLM 38 11 0 10 Feb 2024
MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models Yixiao Zhang Yukara Ikemiya Gus Xia Naoki Murata Marco A. Martínez Ramírez Wei-Hsiang Liao Yuki Mitsufuji Simon Dixon 42 20 0 09 Feb 2024
Fast Timing-Conditioned Latent Audio Diffusion Zach Evans CJ Carr Josiah Taylor Scott H. Hawley Jordi Pons DiffM 74 98 0 07 Feb 2024
Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities Zhifeng Kong Arushi Goel Rohan Badlani Wei Ping Rafael Valle Bryan Catanzaro AuLLM LM&MA MLLM 59 73 0 02 Feb 2024
PAM: Prompting Audio-Language Models for Audio Quality Assessment Soham Deshmukh Dareen Alharthi Benjamin Elizalde Hannes Gamper Mahmoud Al Ismail Rita Singh Bhiksha Raj Huaming Wang 10 11 0 01 Feb 2024
Binding Touch to Everything: Learning Unified Multimodal Tactile Representations Fengyu Yang Chao Feng Ziyang Chen Hyoungseob Park Daniel Wang ... Ziyao Zeng Xien Chen Rit Gangopadhyay Andrew Owens Alex Wong 36 52 0 31 Jan 2024
EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning Jaeyeon Kim Jaeyoon Jung Jinjoo Lee Sang Hoon Woo CLIP VLM 16 21 0 31 Jan 2024
Masked Audio Modeling with CLAP and Multi-Objective Learning Yifei Xin Xiulian Peng Yan Lu 42 8 0 29 Jan 2024
Exploring Musical Roots: Applying Audio Embeddings to Empower Influence Attribution for a Generative Music Model Julia Barnett Hugo Flores Garcia Bryan Pardo 27 6 0 25 Jan 2024
MM-LLMs: Recent Advances in MultiModal Large Language Models Duzhen Zhang Yahan Yu Jiahua Dong Chenxing Li Dan Su Chenhui Chu Dong Yu OffRL LRM 37 173 0 24 Jan 2024
DITTO: Diffusion Inference-Time T-Optimization for Music Generation Zachary Novack Julian McAuley Taylor Berg-Kirkpatrick Nicholas J. Bryan DiffM 16 32 0 22 Jan 2024
CoAVT: A Cognition-Inspired Unified Audio-Visual-Text Pre-Training Model for Multimodal Processing Xianghu Yue Xiaohai Tian Lu Lu Malu Zhang Zhizheng Wu Haizhou Li 17 0 0 22 Jan 2024
On the Audio Hallucinations in Large Audio-Video Language Models Taichi Nishimura Shota Nakada Masayoshi Kondo VLM 25 5 0 18 Jan 2024
Learning Audio Concepts from Counterfactual Natural Language A. Vosoughi Luca Bondi Ho-Hsiang Wu Chenliang Xu CML 42 2 0 10 Jan 2024
Masked Audio Generation using a Single Non-Autoregressive Transformer Alon Ziv Itai Gat Gaël Le Lan Tal Remez Felix Kreuk Alexandre Défossez Jade Copet Gabriel Synnaeve Yossi Adi 32 36 0 09 Jan 2024
AccidentGPT: Large Multi-Modal Foundation Model for Traffic Accident Analysis Kebin Wu Wenbin Li Xiaofei Xiao 11 2 0 05 Jan 2024
Towards Weakly Supervised Text-to-Audio Grounding Xuenan Xu Ziyang Ma Mengyue Wu Kai Yu AI4TS 25 8 0 05 Jan 2024