Are Multimodal Transformers Robust to Missing Modality?

12 April 2022

Papers citing "Are Multimodal Transformers Robust to Missing Modality?"

44 / 94 papers shown

Title
Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality Huy Q. Le Chu Myaet Thwal Yu Qiao Ye Lin Tun Minh N. H. Nguyen Choong Seon Hong Choong Seon Hong 55 3 0 25 Jan 2024
Exploring Missing Modality in Multimodal Egocentric Datasets Merey Ramazanova Alejandro Pardo Humam Alwassel Bernard Ghanem EgoV 18 4 0 21 Jan 2024
Toward Robust Multimodal Learning using Multimodal Foundational Models Xianbing Zhao Soujanya Poria Xuejiao Li Yixin Chen Buzhou Tang VLM 32 2 0 20 Jan 2024
Towards Robust Multimodal Prompting With Missing Modalities Jaehyuk Jang Yooseung Wang Changick Kim VLM 20 10 0 26 Dec 2023
Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast Guangyin Bao Qi Zhang Duoqian Miao Zixuan Gong Liang Hu Ke Liu Yang Liu Chongyang Shi 29 7 0 21 Dec 2023
Can Physician Judgment Enhance Model Trustworthiness? A Case Study on Predicting Pathological Lymph Nodes in Rectal Cancer Kazuma Kobayashi Yasuyuki Takamizawa M. Miyake Sono Ito Lin Gu Tatsuya Nakatsuka Yu Akagi Tatsuya Harada Y. Kanemitsu Ryuji Hamamoto 15 2 0 15 Dec 2023
Segment Beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation Renjie Wu Hu Wang Feras Dayoub Hsiang-Ting Chen 17 5 0 14 Dec 2023
Adventures of Trustworthy Vision-Language Models: A Survey Mayank Vatsa Anubhooti Jain Richa Singh 9 4 0 07 Dec 2023
Multimodal Representation Learning by Alternating Unimodal Adaptation Xiaohui Zhang Jaehong Yoon Mohit Bansal Huaxiu Yao 21 20 0 17 Nov 2023
Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition Juan Vazquez-Rodriguez G. Lefebvre Julien Cumin James L. Crowley 21 2 0 16 Nov 2023
MOSEL: Inference Serving Using Dynamic Modality Selection Bodun Hu Le Xu Jeongyoon Moon N. Yadwadkar Aditya Akella 11 3 0 27 Oct 2023
What Makes for Robust Multi-Modal Models in the Face of Missing Modalities? Siting Li Chenzhuang Du Yue Zhao Yu Huang Hang Zhao 19 4 0 10 Oct 2023
Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation Md Kaykobad Reza Ashley Prater-Bennette M. Salman Asif 16 4 0 06 Oct 2023
MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks Vinitra Swamy Malika Satayeva Jibril Frej Thierry Bossy Thijs Vogels Martin Jaggi Tanja Kaser Mary-Anne Hartley 11 11 0 25 Sep 2023
A Survey on Image-text Multimodal Models Ruifeng Guo Jingxuan Wei Linzhuang Sun Khai Le-Duc Guiyong Chang Dawei Liu Sibo Zhang Zhengbing Yao Mingjun Xu Liping Bu VLM 21 5 0 23 Sep 2023
Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognition Qitong Wang Long Zhao Liangzhe Yuan Ting Liu Xi Peng 20 11 0 22 Aug 2023
Boosting Multi-modal Model Performance with Adaptive Gradient Modulation Hong Li Xingyu Li Pengbo Hu Yinuo Lei Chunxiao Li Yi Zhou 22 20 0 15 Aug 2023
Learning Unseen Modality Interaction Yunhua Zhang Hazel Doughty Cees G. M. Snoek 14 3 0 22 Jun 2023
Modality Influence in Multimodal Machine Learning Abdelhamid Haouhat Slimane Bellaouar A. Nehar H. Cherroun 16 1 0 10 Jun 2023
VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognition Ziyi Ni Minglun Han Feilong Chen Linghui Meng Jing Shi Shuang Xu Bo Xu 18 0 0 31 May 2023
FRAMM: Fair Ranking with Missing Modalities for Clinical Trial Site Selection B. Theodorou Lucas Glass Cao Xiao Jimeng Sun 14 7 0 30 May 2023
LANISTR: Multimodal Learning from Structured and Unstructured Data Sayna Ebrahimi Sercan Ö. Arik Yihe Dong Tomas Pfister 14 4 0 26 May 2023
Training Transitive and Commutative Multimodal Transformers with LoReTTa Manuel Tran Yashin Dicente Cid Amal Lahiani Fabian J. Theis Tingying Peng Eldad Klaiman 11 2 0 23 May 2023
Rethinking Multimodal Content Moderation from an Asymmetric Angle with Mixed-modality Jialing Yuan Ye Yu Gaurav Mittal Matthew Hall Sandra Sajeev Mei Chen 12 8 0 17 May 2023
MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition Xinyu Gong S. Mohan Naina Dhingra Jean-Charles Bazin Yilei Li Zhangyang Wang Rakesh Ranjan EgoV 44 17 0 12 May 2023
MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis Jinsheng Zheng Daqing Liu Chaoyue Wang Minghui Hu Zuopeng Yang Changxing Ding Dacheng Tao 18 1 0 10 May 2023
Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention Kwanhyung Lee Soojeong Lee Sangchul Hahn Heejung Hyun E. Choi Byungeun Ahn Joohyung Lee 46 8 0 04 May 2023
Incomplete Multimodal Learning for Remote Sensing Data Fusion Yuxing Chen Maofan Zhao Lorenzo Bruzzone 13 3 0 22 Apr 2023
Towards Unified AI Drug Discovery with Multiple Knowledge Modalities Yi Luo Xing Yi Liu Kai Yang Kui Huang Massimo Hong Jiahuan Zhang Yushuai Wu Zaiqing Nie OffRL 24 4 0 17 Apr 2023
Are Data-driven Explanations Robust against Out-of-distribution Data? Tang Li Fengchun Qiao Mengmeng Ma Xiangkai Peng OODD OOD 20 5 0 29 Mar 2023
CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities Konstantinos Kontras Christos Chatzichristos Huy P Phan Johan A. K. Suykens Marina De Vos AI4TS 11 11 0 27 Mar 2023
Multimodal Prompting with Missing Modalities for Visual Recognition Yi-Lun Lee Yi-Hsuan Tsai Wei-Chen Chiu Chen-Yu Lee VPVLM 19 91 0 06 Mar 2023
Rethinking Vision Transformers for MobileNet Size and Speed Yanyu Li Ju Hu Yang Wen Georgios Evangelidis Kamyar Salahi Yanzhi Wang Sergey Tulyakov Jian Ren ViT 19 157 0 15 Dec 2022
Towards Good Practices for Missing Modality Robust Action Recognition Sangmin Woo Sumin Lee Yeonju Park Muhammad Adi Nugroho Changick Kim 22 42 0 25 Nov 2022
Multimodal Transformer for Parallel Concatenated Variational Autoencoders Stephen D. Liang J. Mendel ViT 11 5 0 28 Oct 2022
Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features Gokul Karthik Kumar Karthik Nandakumar VLM CLIP 19 50 0 12 Oct 2022
Missing Modality meets Meta Sampling (M3S): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modality Haozhe Chi Minghua Yang Junhao Zhu Guanhong Wang Gaoang Wang 33 3 0 07 Oct 2022
Husformer: A Multi-Modal Transformer for Multi-Modal Human State Recognition Ruiqi Wang Wonse Jo Dezhong Zhao Weizheng Wang B. Yang Guohua Chen Byung-Cheol Min HAI 11 28 0 30 Sep 2022
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions Paul Pu Liang Amir Zadeh Louis-Philippe Morency 13 59 0 07 Sep 2022
Multimodal Learning with Transformers: A Survey P. Xu Xiatian Zhu David A. Clifton ViT 41 518 0 13 Jun 2022
Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review C. Cui Haichun Yang Yaohong Wang Shilin Zhao Zuhayr Asad Lori A. Coburn K. Wilson Bennett A. Landman Yuankai Huo 15 93 0 25 Mar 2022
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text Hassan Akbari Liangzhe Yuan Rui Qian Wei-Hong Chuang Shih-Fu Chang Yin Cui Boqing Gong ViT 231 573 0 22 Apr 2021
Multi-modal Transformer for Video Retrieval Valentin Gabeur Chen Sun Alahari Karteek Cordelia Schmid ViT 398 532 0 21 Jul 2020
Supervised Multimodal Bitransformers for Classifying Images and Text Douwe Kiela Suvrat Bhooshan Hamed Firooz Ethan Perez Davide Testuggine 51 238 0 06 Sep 2019