Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning

8 April 2023

Yu Yang

Besmira Nushi

Hamid Palangi

Baharan Mirzasoleiman

ArXiv PDF HTML

Papers citing "Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning"

27 / 27 papers shown

Title
Gradient Extrapolation for Debiased Representation Learning Ihab Asaad M. Shadaydeh Joachim Denzler 36 0 0 17 Mar 2025
Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness B. Zhu Jiequan Cui H. Zhang Chi Zhang 74 0 0 12 Mar 2025
Debiased Prompt Tuning in Vision-Language Model without Annotations Chaoquan Jiang Yunfan Yang Rui Hu Jitao Sang VLM 57 0 0 11 Mar 2025
Language Models' Factuality Depends on the Language of Inquiry Tushar Aggarwal Kumar Tanmay Ayush Agrawal Kumar Ayush Hamid Palangi Paul Pu Liang HILM KELM 68 0 0 25 Feb 2025
Refining Skewed Perceptions in Vision-Language Models through Visual Representations Haocheng Dai Sarang Joshi VLM 57 0 0 03 Jan 2025
The Silent Majority: Demystifying Memorization Effect in the Presence of Spurious Correlations Chenyu You Haocheng Dai Yifei Min Jasjeet Sekhon S. Joshi James S. Duncan 53 2 0 01 Jan 2025
RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models Maya Varma Jean-Benoit Delbrouck Zhihong Chen Akshay S. Chaudhari C. Langlotz VLM 37 6 0 06 Nov 2024
Improving Robustness to Multiple Spurious Correlations by Multi-Objective Optimization Nayeong Kim Juwon Kang Sungsoo Ahn Jungseul Ok Suha Kwak 24 0 0 05 Sep 2024
Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effort Jeeyung Kim Ze Wang Qiang Qiu 38 1 0 12 Jul 2024
SLIM: Spuriousness Mitigation with Minimal Human Annotations Xiwei Xuan Ziquan Deng Hsuan-Tien Lin Kwan-Liu Ma 26 1 0 08 Jul 2024
Autoencoder based approach for the mitigation of spurious correlations Srinitish Srinivasan Karthik Seemakurthy 17 0 0 27 Jun 2024
Mind the Graph When Balancing Data for Fairness or Robustness Jessica Schrouff Alexis Bellot Amal Rannen-Triki Alan Malek Isabela Albuquerque A. Gretton Alexander DÁmour Silvia Chiappa OOD CML 26 1 0 25 Jun 2024
Discovering Bias in Latent Space: An Unsupervised Debiasing Approach Dyah Adila Shuai Zhang Boran Han Yuyang Wang AAML LLMSV 27 6 0 05 Jun 2024
Controllable Prompt Tuning For Balancing Group Distributional Robustness Hoang Phan Andrew Gordon Wilson Qi Lei 36 5 0 05 Mar 2024
Spurious Correlations in Machine Learning: A Survey Wenqian Ye Guangtao Zheng Xu Cao Yunsheng Ma Aidong Zhang OOD AAML CML 29 33 0 20 Feb 2024
AttributionScanner: A Visual Analytics System for Model Validation with Metadata-Free Slice Finding Xiwei Xuan Jorge Henrique Piazentin Ono Liang Gou Kwan-Liu Ma Liu Ren 48 0 0 12 Jan 2024
Distilling Vision-Language Models on Millions of Videos Yue Zhao Long Zhao Xingyi Zhou Jialin Wu Chun-Te Chu ... Hartwig Adam Ting Liu Boqing Gong Philipp Krahenbuhl Liangzhe Yuan VLM 21 13 0 11 Jan 2024
FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning Kun Song Huimin Ma Bochao Zou Huishuai Zhang Weiran Huang 13 10 0 23 Oct 2023
Sieve: Multimodal Dataset Pruning Using Image Captioning Models Anas Mahmoud Mostafa Elhoushi Amro Abbas Yu Yang Newsha Ardalani Hugh Leather Ari S. Morcos VLM CLIP 30 18 0 03 Oct 2023
Zero-Shot Robustification of Zero-Shot Models Dyah Adila Changho Shin Lin Cai Frederic Sala 29 18 0 08 Sep 2023
ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations Sreyan Ghosh Chandra Kiran Reddy Evuru Sonal Kumar Utkarsh Tyagi Sakshi Singh Sanjoy Chowdhury Dinesh Manocha OOD 20 1 0 19 Aug 2023
PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts Bang An Sicheng Zhu Michael-Andrei Panaitescu-Liess Chaithanya Kumar Mummadi Furong Huang VLM 20 7 0 02 Aug 2023
Robust Learning with Progressive Data Expansion Against Spurious Correlation Yihe Deng Yu Yang Baharan Mirzasoleiman Quanquan Gu OOD MLT 11 24 0 08 Jun 2023
CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning Hritik Bansal Nishad Singhi Yu Yang Fan Yin Aditya Grover Kai-Wei Chang AAML 24 41 0 06 Mar 2023
On Guiding Visual Attention with Language Specification Suzanne Petryk Lisa Dunlap Keyan Nasseri Joseph E. Gonzalez Trevor Darrell Anna Rohrbach VLM 185 30 1 17 Feb 2022
Unified Vision-Language Pre-Training for Image Captioning and VQA Luowei Zhou Hamid Palangi Lei Zhang Houdong Hu Jason J. Corso Jianfeng Gao MLLM VLM 250 922 0 24 Sep 2019
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 279 39,083 0 01 Sep 2014