Self-Supervised Learning by Cross-Modal Audio-Video Clustering

28 November 2019

Papers citing "Self-Supervised Learning by Cross-Modal Audio-Video Clustering"

50 / 84 papers shown

Title
Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention Joe Dhanith Shravan Venkatraman Modigari Narendra Vigya Sharma Santhosh Malarvannan 74 0 0 20 Feb 2025
The Sound of Water: Inferring Physical Properties from Pouring Liquids Piyush Bagad Makarand Tapaswi Cees G. M. Snoek Andrew Zisserman 40 0 0 18 Nov 2024
What to align in multimodal contrastive learning? Benoit Dufumier J. Castillo-Navarro D. Tuia Jean-Philippe Thiran 22 3 0 11 Sep 2024
Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models David Kurzendörfer Otniel-Bogdan Mercea A. Sophia Koepke Zeynep Akata VLM CLIP 26 2 0 09 Apr 2024
Siamese Vision Transformers are Scalable Audio-visual Learners Yan-Bo Lin Gedas Bertasius 37 5 0 28 Mar 2024
Self Supervised Correlation-based Permutations for Multi-View Clustering Ran Eisenberg Jonathan Svirsky Ofir Lindenbaum 32 2 0 26 Feb 2024
POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images Antonín Vobecký Oriane Siméoni David Hurych Spyros Gidaris Andrei Bursuc Patrick Pérez Josef Sivic 29 33 0 17 Jan 2024
Collaboratively Self-supervised Video Representation Learning for Action Recognition Jie M. Zhang Zhifan Wan Lanqing Hu Stephen Lin Shuzhe Wu Shiguang Shan TTA 56 1 0 15 Jan 2024
Modality Mixer Exploiting Complementary Information for Multi-modal Action Recognition Sumin Lee Sangmin Woo Muhammad Adi Nugroho Changick Kim 25 0 0 21 Nov 2023
Towards Generalized Multi-stage Clustering: Multi-view Self-distillation Jiatai Wang Zhiwei Xu Xin Wang Tao Li 11 1 0 29 Oct 2023
Deep Neural Networks in Video Human Action Recognition: A Review Zihan Wang Yang Yang Zhi Liu Y. Zheng 51 4 0 25 May 2023
Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning Nikhil Singh Chih-Wei Wu Iroro Orife Mahdi M. Kalayeh 23 2 0 12 Apr 2023
Procedure-Aware Pretraining for Instructional Video Understanding Honglu Zhou Roberto Martín-Martín Mubbasir Kapadia Silvio Savarese Juan Carlos Niebles 23 38 0 31 Mar 2023
Egocentric Auditory Attention Localization in Conversations Fiona Ryan Hao Jiang Abhinav Shukla James M. Rehg V. Ithapu EgoV 24 16 0 28 Mar 2023
SemanticAC: Semantics-Assisted Framework for Audio Classification Yicheng Xiao Yue Ma Shuyan Li Hantao Zhou Ran Liao Xiu Li 13 8 0 12 Feb 2023
Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models Zhiqiu Lin Samuel Yu Zhiyi Kuang Deepak Pathak Deva Ramana VLM 15 100 0 16 Jan 2023
What You Say Is What You Show: Visual Narration Detection in Instructional Videos Kumar Ashutosh Rohit Girdhar Lorenzo Torresani Kristen Grauman 11 4 0 05 Jan 2023
Look, Listen, and Attack: Backdoor Attacks Against Video Action Recognition Hasan Hammoud Shuming Liu Mohammad Alkhrashi Fahad Albalawi Bernard Ghanem AAML 26 8 0 03 Jan 2023
Similarity Contrastive Estimation for Image and Video Soft Contrastive Self-Supervised Learning J. Denize Jaonary Rabarisoa Astrid Orcesi Romain Hérault SSL 14 5 0 21 Dec 2022
Complete Cross-triplet Loss in Label Space for Audio-visual Cross-modal Retrieval Donghuo Zeng Yanan Wang Jianming Wu K. Ikeda 19 4 0 07 Nov 2022
Self-supervised Video Representation Learning with Motion-Aware Masked Autoencoders Haosen Yang Deng Huang Bin Wen Jiannan Wu H. Yao Yi-Xin Jiang Xiatian Zhu Zehuan Yuan 24 19 0 09 Oct 2022
Learning State-Aware Visual Representations from Audible Interactions Himangi Mittal Pedro Morgado Unnat Jain Abhinav Gupta 66 22 0 27 Sep 2022
Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings Yiren Jian Chongyang Gao Soroush Vosoughi SSL 18 15 0 20 Sep 2022
ImageArg: A Multi-modal Tweet Dataset for Image Persuasiveness Mining Zhexiong Liu M. Guo Y. Dai Diane Litman 16 15 0 14 Sep 2022
Modality Mixer for Multi-modal Action Recognition Sumin Lee Sangmin Woo Yeonju Park Muhammad Adi Nugroho Changick Kim 19 10 0 24 Aug 2022
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey Yanbei Chen Massimiliano Mancini Xiatian Zhu Zeynep Akata 30 113 0 24 Aug 2022
ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain Generalization Zdravko Marinov Alina Roitberg David Schneider Rainer Stiefelhagen 17 4 0 19 Aug 2022
Learning Music-Dance Representations through Explicit-Implicit Rhythm Synchronization Jiashuo Yu Junfu Pu Ying Cheng Rui Feng Ying Shan 14 5 0 07 Jul 2022
Self-Supervised Learning for Videos: A Survey Madeline Chantry Schiappa Y. S. Rawat M. Shah SSL 26 131 0 18 Jun 2022
SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning Changan Chen Carl Schissler Sanchit Garg Philip Kobernik Alexander William Clegg P. Calamia Dhruv Batra Philip Robinson Kristen Grauman 3DGS 31 79 0 16 Jun 2022
TransRank: Self-supervised Video Representation Learning via Ranking-based Transformation Recognition Haodong Duan Nanxuan Zhao Kai-xiang Chen Dahua Lin ViT AI4TS 31 19 0 04 May 2022
On Negative Sampling for Audio-Visual Contrastive Learning from Movies Mahdi M. Kalayeh Shervin Ardeshir Lingyi Liu Nagendra Kamath Ashok Chandrashekar SSL 22 3 0 29 Apr 2022
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval Yuying Ge Yixiao Ge Xihui Liu Alex Jinpeng Wang Jianping Wu Ying Shan Xiaohu Qie Ping Luo VLM 9 43 0 26 Apr 2022
Contrastive Language-Action Pre-training for Temporal Localization Mengmeng Xu Erhan Gundogdu ⋆⋆ Maksim Bernard Ghanem M. Donoser Loris Bazzani 25 27 0 26 Apr 2022
Frequency Selective Augmentation for Video Representation Learning Jinhyung Kim Taeoh Kim Minho Shim Dongyoon Han Dongyoon Wee Junmo Kim AI4TS 41 3 0 08 Apr 2022
ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound Yan-Bo Lin Jie Lei Mohit Bansal Gedas Bertasius 31 39 0 06 Apr 2022
Single-Stream Multi-Level Alignment for Vision-Language Pretraining Zaid Khan B. Vijaykumar Xiang Yu S. Schulter Manmohan Chandraker Y. Fu CLIP VLM 20 16 0 27 Mar 2022
Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers A. Bucker Luis F. C. Figueredo Sami Haddadin Ashish Kapoor Shuang Ma Rogerio Bonatti LM&Ro 16 49 0 25 Mar 2022
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training Zhan Tong Yibing Song Jue Wang Limin Wang ViT 111 1,120 0 23 Mar 2022
Audio Self-supervised Learning: A Survey Shuo Liu Adria Mallol-Ragolta Emilia Parada-Cabeleiro Kun Qian Xingshuo Jing Alexander Kathan Bin Hu Bjoern W. Schuller SSL 26 106 0 02 Mar 2022
Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition Zitian Zhang Jie M. Zhang Jian-Shu Zhang Ming Wu Xin Fang Lirong Dai SSL 27 10 0 15 Feb 2022
Visual Acoustic Matching Changan Chen Ruohan Gao P. Calamia Kristen Grauman 16 55 0 14 Feb 2022
Visual Sound Localization in the Wild by Cross-Modal Interference Erasing Xian Liu Rui Qian Hang Zhou Di Hu Weiyao Lin Ziwei Liu Bolei Zhou Xiaowei Zhou 6 25 0 13 Feb 2022
Keyword localisation in untranscribed speech using visually grounded speech models Kayode Olaleye Dan Oneaţă Herman Kamper 19 7 0 02 Feb 2022
Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection A. Haliassos Rodrigo Mira Stavros Petridis M. Pantic CVBM 27 125 0 18 Jan 2022
Bridging Video-text Retrieval with Multiple Choice Questions Yuying Ge Yixiao Ge Xihui Liu Dian Li Ying Shan Xiaohu Qie Ping Luo BDL 16 108 0 13 Jan 2022
Robust Contrastive Learning against Noisy Views Ching-Yao Chuang R. Devon Hjelm Xin Eric Wang Vibhav Vineet Neel Joshi Antonio Torralba Stefanie Jegelka Ya-heng Song NoLa 13 68 0 12 Jan 2022
Exploring Temporal Granularity in Self-Supervised Video Representation Learning Rui Qian Yeqing Li Liangzhe Yuan Boqing Gong Ting Liu Matthew A. Brown Serge J. Belongie Ming-Hsuan Yang Hartwig Adam Yin Cui AI4TS 41 6 0 08 Dec 2021
Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval Nina Shvetsova Brian Chen Andrew Rouditchenko Samuel Thomas Brian Kingsbury Rogerio Feris David F. Harwath James R. Glass Hilde Kuehne ViT 23 129 0 08 Dec 2021
Time-Equivariant Contrastive Video Representation Learning Simon Jenni Hailin Jin SSL AI4TS 135 60 0 07 Dec 2021