Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)

3 May 2022

Papers citing "Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)"

49 / 99 papers shown

Title
Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models Vishaal Udandarao Max F. Burg Samuel Albanie Matthias Bethge VLM 24 8 0 12 Oct 2023
Prompt Backdoors in Visual Prompt Learning Hai Huang Zhengyu Zhao Michael Backes Yun Shen Yang Zhang VLM VPVLM AAML SILM 35 2 0 11 Oct 2023
Robustness May be More Brittle than We Think under Different Degrees of Distribution Shifts Kaican Li Yifan Zhang Lanqing Hong Zhenguo Li Nevin L. Zhang OOD 28 0 0 10 Oct 2023
Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift Yihao Xue Siddharth Joshi Dang Nguyen Baharan Mirzasoleiman VLM 24 4 0 08 Oct 2023
Data Filtering Networks Alex Fang Albin Madappally Jose Amit Jain Ludwig Schmidt Alexander Toshev Vaishaal Shankar CLIP 32 124 0 29 Sep 2023
Preserving Modality Structure Improves Multi-Modal Learning Swetha Sirnam Mamshad Nayeem Rizve Nina Shvetsova Hilde Kuehne M. Shah 23 4 0 24 Aug 2023
ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data M. Varma Jean-Benoit Delbrouck Sarah Hooper Akshay S. Chaudhari C. Langlotz VLM CoGe 40 5 0 22 Aug 2023
Distributionally Robust Classification on a Data Budget Ben Feuer Ameya Joshi Minh Pham C. Hegde OOD 22 2 0 07 Aug 2023
Does Progress On Object Recognition Benchmarks Improve Real-World Generalization? Megan Richards Polina Kirichenko Diane Bouchacourt Mark Ibrahim VLM 64 11 0 24 Jul 2023
On the Connection between Pre-training Data Diversity and Fine-tuning Robustness Vivek Ramanujan Thao Nguyen Sewoong Oh Ludwig Schmidt Ali Farhadi OOD 14 21 0 24 Jul 2023
Robust Visual Question Answering: Datasets, Methods, and Future Challenges Jie Ma Pinghui Wang Dechen Kong Zewei Wang Jun Liu Hongbin Pei Junzhou Zhao OOD 19 18 0 21 Jul 2023
Improving Multimodal Datasets with Image Captioning Thao Nguyen S. Gadre Gabriel Ilharco Sewoong Oh Ludwig Schmidt VLM 19 70 0 19 Jul 2023
T-MARS: Improving Visual Representations by Circumventing Text Feature Learning Pratyush Maini Sachin Goyal Zachary Chase Lipton J. Zico Kolter Aditi Raghunathan VLM 29 33 0 06 Jul 2023
Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity Zhenlin Xu Yi Zhu Tiffany Deng Abhay Mittal Yanbei Chen Manchen Wang Paolo Favaro Joseph Tighe Davide Modolo VLM CoGe 11 7 0 28 Jun 2023
What Makes ImageNet Look Unlike LAION Ali Shirali Moritz Hardt 8 9 0 27 Jun 2023
Utilizing Segment Anything Model For Assessing Localization of GRAD-CAM in Medical Imaging Evan Kellener Ihina Nath Anthony Ngo Thomas Nguyen Joshua Schuman Coen Adler Arnav Kartikeya 11 4 0 24 Jun 2023
Neural Priming for Sample-Efficient Adaptation Matthew Wallingford Vivek Ramanujan Alex Fang Aditya Kusupati Roozbeh Mottaghi Aniruddha Kembhavi Ludwig Schmidt Ali Farhadi VLM 106 13 0 16 Jun 2023
LOVM: Language-Only Vision Model Selection O. Zohar Shih-Cheng Huang Kuan-Chieh Jackson Wang Serena Yeung MLLM 35 12 0 15 Jun 2023
How to Train Your CheXDragon: Training Chest X-Ray Models for Transfer to Novel Tasks and Healthcare Systems Cara Van Uden Jeremy Irvin Mars Huang N. Dean J. Carr A. Ng C. Langlotz OOD 24 1 0 13 May 2023
Accelerating Batch Active Learning Using Continual Learning Techniques Arnav M. Das Gantavya Bhatt M. Bhalerao Vianne R. Gao Rui Yang J. Bilmes VLM CLL 22 9 0 10 May 2023
DataComp: In search of the next generation of multimodal datasets S. Gadre Gabriel Ilharco Alex Fang J. Hayase Georgios Smyrnis ... A. Dimakis J. Jitsev Y. Carmon Vaishaal Shankar Ludwig Schmidt VLM 15 408 0 27 Apr 2023
Unified Out-Of-Distribution Detection: A Model-Specific Perspective Reza Averly Wei-Lun Chao OODD 35 11 0 13 Apr 2023
On Robustness in Multimodal Learning Brandon McKinzie Joseph Cheng Vaishaal Shankar Yinfei Yang Jonathon Shlens Alexander Toshev 30 2 0 10 Apr 2023
TRAK: Attributing Model Behavior at Scale Sung Min Park Kristian Georgiev Andrew Ilyas Guillaume Leclerc A. Madry TDI 11 127 0 24 Mar 2023
Boosting Weakly Supervised Object Detection using Fusion and Priors from Hallucinated Depth Cagri Gungor Adriana Kovashka MDE 13 1 0 20 Mar 2023
VEIL: Vetting Extracted Image Labels from In-the-Wild Captions for Weakly-Supervised Object Detection Arushi Rai Adriana Kovashka 19 0 0 16 Mar 2023
Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need Da-Wei Zhou Han-Jia Ye De-Chuan Zhan Ziwei Liu CLL 28 99 0 13 Mar 2023
The Role of Pre-training Data in Transfer Learning R. Entezari Mitchell Wortsman O. Saukh M. Shariatnia Hanie Sedghi Ludwig Schmidt 38 20 0 27 Feb 2023
Measuring Equality in Machine Learning Security Defenses: A Case Study in Speech Recognition Luke E. Richards Edward Raff Cynthia Matuszek AAML 11 2 0 17 Feb 2023
Effective Robustness against Natural Distribution Shifts for Models with Different Training Data Zhouxing Shi Nicholas Carlini Ananth Balashankar Ludwig Schmidt Cho-Jui Hsieh Alex Beutel Yao Qin OOD 21 9 0 02 Feb 2023
Discovering and Mitigating Visual Biases through Keyword Explanation Younghyun Kim Sangwoo Mo Minkyu Kim Kyungmin Lee Jaeho Lee Jinwoo Shin 26 30 0 26 Jan 2023
Masked Autoencoding Does Not Help Natural Language Supervision at Scale Floris Weers Vaishaal Shankar Angelos Katharopoulos Yinfei Yang Tom Gunter CLIP 18 4 0 19 Jan 2023
Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization Alexandre Ramé Kartik Ahuja Jianyu Zhang Matthieu Cord Léon Bottou David Lopez-Paz MoMe OODD 24 80 0 20 Dec 2022
Reproducible scaling laws for contrastive language-image learning Mehdi Cherti Romain Beaumont Ross Wightman Mitchell Wortsman Gabriel Ilharco Cade Gordon Christoph Schuhmann Ludwig Schmidt J. Jitsev VLM CLIP 49 734 0 14 Dec 2022
A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others Zhiheng Li Ivan Evtimov Albert Gordo C. Hazirbas Tal Hassner Cristian Canton Ferrer Chenliang Xu Mark Ibrahim 26 70 0 09 Dec 2022
PØDA: Prompt-driven Zero-shot Domain Adaptation Mohammad Fahes Tuan-Hung Vu Andrei Bursuc Patrick Pérez Raoul de Charette VLM 36 45 0 06 Dec 2022
Context-Aware Robust Fine-Tuning Xiaofeng Mao YueFeng Chen Xiaojun Jia Rong Zhang Hui Xue Zhao Li VLM CLIP 20 23 0 29 Nov 2022
SuS-X: Training-Free Name-Only Transfer of Vision-Language Models Vishaal Udandarao Ankush Gupta Samuel Albanie VLM MLLM 22 103 0 28 Nov 2022
FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers Jinyu Chen Wenchao Xu Song Guo Junxiao Wang Jie M. Zhang Haozhao Wang FedML 15 32 0 15 Nov 2022
Caption supervision enables robust learners Ben Feuer Ameya Joshi C. Hegde SSL CLIP VLM 32 2 0 13 Oct 2022
APE: Aligning Pretrained Encoders to Quickly Learn Aligned Multimodal Representations Elan Rosenfeld Preetum Nakkiran Hadi Pouransari Oncel Tuzel Fartash Faghri 55 6 0 08 Oct 2022
ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets Damien Teney Yong Lin Seong Joon Oh Ehsan Abbasnejad OOD 360 47 0 01 Sep 2022
Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP Thao Nguyen Gabriel Ilharco Mitchell Wortsman Sewoong Oh Ludwig Schmidt CLIP VLM 38 97 0 10 Aug 2022
Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning Shibani Santurkar Yann Dubois Rohan Taori Percy Liang Tatsunori Hashimoto CLIP VLM 11 41 0 15 Jul 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 258 7,412 0 11 Nov 2021
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning Krishna Srinivasan K. Raman Jiecao Chen Michael Bendersky Marc Najork VLM 197 308 0 02 Mar 2021
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts Soravit Changpinyo P. Sharma Nan Ding Radu Soricut VLM 273 1,077 0 17 Feb 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 293 3,683 0 11 Feb 2021
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 282 39,170 0 01 Sep 2014