Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation

12 November 2022

Taylor Berg-Kirkpatrick

Shlomo Dubnov

CLIP

ArXiv PDF HTML

Papers citing "Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation"

50 / 343 papers shown

Title
A sound description: Exploring prompt templates and class descriptions to enhance zero-shot audio classification Michel Olvera Paraskevas Stamatiadis S. Essid VLM 22 1 0 19 Sep 2024
SoundBeam meets M2D: Target Sound Extraction with Audio Foundation Model Carlos Hernandez-Olivan Marc Delcroix Tsubasa Ochiai Daisuke Niizumi Naohiro Tawara Tomohiro Nakatani Shoko Araki 29 2 0 19 Sep 2024
ViolinDiff: Enhancing Expressive Violin Synthesis with Pitch Bend Conditioning Daewoong Kim Hao-Wen Dong Dasaem Jeong 13 0 0 19 Sep 2024
DETECLAP: Enhancing Audio-Visual Representation Learning with Object Information Shota Nakada Taichi Nishimura Hokuto Munakata Masayoshi Kondo Tatsuya Komatsu CLIP VLM 20 0 0 18 Sep 2024
Augment, Drop & Swap: Improving Diversity in LLM Captions for Efficient Music-Text Representation Learning Ilaria Manco Justin Salamon Oriol Nieto 23 0 0 17 Sep 2024
Evaluation of pretrained language models on music understanding Yannis Vasilakis Rachel M. Bittner Johan Pauwels 20 1 0 17 Sep 2024
FakeMusicCaps: a Dataset for Detection and Attribution of Synthetic Music Generated via Text-to-Music Models Luca Comanducci Paolo Bestagini Stefano Tubaro 35 6 0 16 Sep 2024
Efficient Video to Audio Mapper with Visual Scene Detection Mingjing Yi Ming Li VGen 13 3 0 15 Sep 2024
Towards Diverse and Efficient Audio Captioning via Diffusion Models Manjie Xu Chenxing Li Xinyi Tu Yong Ren Ruibo Fu Wei Liang Dong Yu DiffM 41 1 0 14 Sep 2024
Prevailing Research Areas for Music AI in the Era of Foundation Models Megan Wei M. Modrzejewski Aswin Sivaraman Dorien Herremans MedIm 29 1 0 14 Sep 2024
Language-Queried Target Sound Extraction Without Parallel Training Data Hao Ma Zhiyuan Peng Xu Li Yukai Li Mingjie Shao Qiuqiang Kong Ju Liu VLM 69 1 0 14 Sep 2024
ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds Sreyan Ghosh Sonal Kumar Chandra Kiran Reddy Evuru Oriol Nieto R. Duraiswami Dinesh Manocha VLM 32 3 0 13 Sep 2024
Towards Leveraging Contrastively Pretrained Neural Audio Embeddings for Recommender Tasks Florian Grötschla Luca Strassle Luca A. Lanzendörfer Roger Wattenhofer 19 0 0 13 Sep 2024
Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis Zhiqi Huang Dan Luo Jun Wang Huan Liao Zhiheng Li Zhiyong Wu VGen 45 4 0 13 Sep 2024
STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment Yong Ren Chenxing Li Manjie Xu Wei Liang Yu Gu Rilin Chen Dong Yu VGen DiffM 43 6 0 13 Sep 2024
VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videos Yan-Bo Lin Yu Tian L. Yang Gedas Bertasius Heng Wang VGen 31 7 0 11 Sep 2024
1M-Deepfakes Detection Challenge Zhixi Cai Abhinav Dhall Shreya Ghosh Munawar Hayat D. Kollias Kalin Stefanov Usman Tariq 20 1 0 11 Sep 2024
Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis Qi Yang Binjie Mao Zili Wang Xing Nie Pengfei Gao Ying Guo Cheng Zhen Pengfei Yan Shiming Xiang VGen DiffM 30 5 0 10 Sep 2024
WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusion Teysir Baoueb Xiaoyu Bie Hicham Janati Gaël Richard DiffM 11 0 0 06 Sep 2024
Dynamic Motion Synthesis: Masked Audio-Text Conditioned Spatio-Temporal Transformers Sohan Anisetty James Hays 30 0 0 03 Sep 2024
EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance Jaeyeon Kim Minjeon Jeon Jaeyoon Jung Sang Hoon Woo Jinjoo Lee 23 2 0 02 Sep 2024
Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioning Jaeyeon Kim Jaeyoon Jung Minjeong Jeon Sang Hoon Woo Jinjoo Lee 24 1 0 02 Sep 2024
Dissecting Temporal Understanding in Text-to-Audio Retrieval Andreea-Maria Oncescu João F. Henriques A. Sophia Koepke 17 2 0 01 Sep 2024
Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion Yan Rong Li Liu 19 3 0 01 Sep 2024
Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model Zhen Ye Peiwen Sun Jiahe Lei Hongzhan Lin Xu Tan ... Jianyi Chen Jiahao Pan Qifeng Liu Yike Guo Wei Xue AuLLM 19 11 0 30 Aug 2024
Improving Generalization of Speech Separation in Real-World Scenarios: Strategies in Simulation, Optimization, and Evaluation K. Chen Jiaqi Su Taylor Berg-Kirkpatrick Shlomo Dubnov Zeyu Jin 21 0 0 28 Aug 2024
SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description Zeyu Jin Jia Jia Qixin Wang Kehan Li Shuoyi Zhou Songtao Zhou Xiaoyu Qin Zhiyong Wu 21 10 0 24 Aug 2024
On Class Separability Pitfalls In Audio-Text Contrastive Zero-Shot Learning Tiago Tavares Fabio Ayres Zhepei Wang Paris Smaragdis VLM 21 2 0 23 Aug 2024
Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition -- And Ways to Overcome Them H. Haresamudram Apoorva Beedu Mashfiqui Rabbi Sankalita Saha Irfan Essa Thomas Ploetz 26 4 0 21 Aug 2024
Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound Junwon Lee Jaekwon Im Dabin Kim Juhan Nam VGen 16 9 0 21 Aug 2024
Audio Match Cutting: Finding and Creating Matching Audio Transitions in Movies and Videos Dennis Fedorishin Lie Lu S. Setlur Venu Govindaraju VGen 36 3 0 20 Aug 2024
Unsupervised Composable Representations for Audio Giovanni Bindi P. Esling DiffM OCL CoGe 23 0 0 19 Aug 2024
SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognition Mohamed Osman Daniel Z. Kaplan Tamer Nadeem 24 1 0 14 Aug 2024
Music2Latent: Consistency Autoencoders for Latent Audio Compression Marco Pasini Stefan Lattner George Fazekas 14 6 0 12 Aug 2024
TEAdapter: Supply abundant guidance for controllable text-to-music generation Jialing Zou Jiahao Mei Xudong Nan Jinghua Li Daoguo Dong Liang He 19 0 0 09 Aug 2024
Nested Music Transformer: Sequentially Decoding Compound Tokens in Symbolic Music and Audio Generation Michael Kolle Maximilian Zorn Jongmin Jung Dasaem Jeong 16 0 0 02 Aug 2024
Combining audio control and style transfer using latent diffusion Andreas Maier Yuliya Burankova Anne Hartebrodt David B. Blumenthal DiffM 32 2 0 31 Jul 2024
Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation Junda Wu Zachary Novack Amit Namburi Jiaheng Dai Hao-Wen Dong Zhouhang Xie Carol Chen Julian McAuley 38 1 0 29 Jul 2024
Audio Entailment: Assessing Deductive Reasoning for Audio Understanding Soham Deshmukh Shuo Han Hazim T. Bukhari Benjamin Elizalde Hannes Gamper Rita Singh Bhiksha Raj ReLM LRM AuLLM 22 7 0 25 Jul 2024
I can listen but cannot read: An evaluation of two-tower multimodal systems for instrument recognition Yannis Vasilakis Rachel M. Bittner Johan Pauwels 35 0 0 25 Jul 2024
Describe Where You Are: Improving Noise-Robustness for Speech Emotion Recognition with Text Description of the Environment Seong-Gyun Leem Daniel Fulford J. Onnela David Gard Carlos Busso 28 0 0 25 Jul 2024
Distortion Recovery: A Two-Stage Method for Guitar Effect Removal Ying-Shuo Lee Yueh-Po Peng Jui-Te Wu Ming Cheng Li Su Yi-Hsuan Yang 18 0 0 23 Jul 2024
Computer Audition: From Task-Specific Machine Learning to Foundation Models Andreas Triantafyllopoulos Iosif Tsangko Alexander Gebhard A. Mesaros Tuomas Virtanen Björn Schuller 39 4 0 22 Jul 2024
Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Models S. Nercessian Johannes Imort Ninon Devis Frederik Blang 29 1 0 22 Jul 2024
DSP-informed bandwidth extension using locally-conditioned excitation and linear time-varying filter subnetworks S. Nercessian Alexey Lukin Johannes Imort 27 0 0 22 Jul 2024
MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generation Yun-Han Lan Wen-Yi Hsiao Hao-Chung Cheng Yi-Hsuan Yang 33 7 0 21 Jul 2024
Towards Assessing Data Replication in Music Generation with Music Similarity Metrics on Raw Audio Roser Batlle-Roca Wei-Hsiang Liao Xavier Serra Yuki Mitsufuji Emilia Gómez 37 0 0 19 Jul 2024
Stable Audio Open Zach Evans Julian Parker CJ Carr Zack Zukowski Josiah Taylor Jordi Pons 64 36 0 19 Jul 2024
Enhancing Zero-shot Audio Classification using Sound Attribute Knowledge from Large Language Models Xuenan Xu Pingyue Zhang Ming Yan Ji Zhang Mengyue Wu VLM 16 0 0 19 Jul 2024
Efficient Audio Captioning with Encoder-Level Knowledge Distillation Xuenan Xu Haohe Liu Mengyue Wu Wenwu Wang Mark D. Plumbley 40 1 0 19 Jul 2024