Finetune like you pretrain: Improved finetuning of zero-shot vision models

1 December 2022

Papers citing "Finetune like you pretrain: Improved finetuning of zero-shot vision models"

50 / 101 papers shown

Title
Tell, Don't Show!: Language Guidance Eases Transfer Across Domains in Images and Videos Tarun Kalluri Bodhisattwa Prasad Majumder Manmohan Chandraker VLM 24 4 0 08 Mar 2024
On the Convergence of Differentially-Private Fine-tuning: To Linearly Probe or to Fully Fine-tune? Shuqi Ke Charlie Hou Giulia Fanti Sewoong Oh 34 4 0 29 Feb 2024
An Empirical Study Into What Matters for Calibrating Vision-Language Models Weijie Tu Weijian Deng Dylan Campbell Stephen Gould Tom Gedeon VLM 33 7 0 12 Feb 2024
Open-Vocabulary Calibration for Fine-tuned CLIP Shuoyuan Wang Jindong Wang Guoqing Wang Bob Zhang Kaiyang Zhou Hongxin Wei VLM 13 5 0 07 Feb 2024
Enhancing Compositional Generalization via Compositional Feature Alignment Haoxiang Wang Haozhe Si Huajie Shao Han Zhao 29 1 0 05 Feb 2024
AutoFT: Learning an Objective for Robust Fine-Tuning Caroline Choi Yoonho Lee Annie S. Chen Allan Zhou Aditi Raghunathan Chelsea Finn OOD 29 0 0 18 Jan 2024
Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions Oindrila Saha Grant Van Horn Subhransu Maji VLM 32 20 0 04 Jan 2024
A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models Julio Silva-Rodríguez Sina Hajimiri Ismail Ben Ayed Jose Dolz VLM 14 25 0 20 Dec 2023
MAFA: Managing False Negatives for Vision-Language Pre-training Jaeseok Byun Dohoon Kim Taesup Moon VLM 13 3 0 11 Dec 2023
Multitask Learning Can Improve Worst-Group Outcomes Atharva Kulkarni Lucio Dery Amrith Rajagopal Setlur Aditi Raghunathan Ameet Talwalkar Graham Neubig 19 1 0 05 Dec 2023
BioCLIP: A Vision Foundation Model for the Tree of Life Samuel Stevens Jiaman Wu Matthew J Thompson Elizabeth G Campolongo Chan Hee Song ... Wasila M Dahdul Charles V. Stewart Tanya Berger-Wolf Wei-Lun Chao Yu-Chuan Su 20 62 0 30 Nov 2023
A Simple Recipe for Language-guided Domain Generalized Segmentation Mohammad Fahes Tuan-Hung Vu Andrei Bursuc Patrick Pérez Raoul de Charette VLM 16 14 0 29 Nov 2023
Towards Calibrated Robust Fine-Tuning of Vision-Language Models Changdae Oh Hyesu Lim Mijoo Kim Dongyoon Han Junhyeok Park Euiseog Jeong Alexander G. Hauptmann Zhi-Qi Cheng Kyungwoo Song VLM 16 13 0 03 Nov 2023
Fast Trainable Projection for Robust Fine-Tuning Junjiao Tian Yen-Cheng Liu James Seale Smith Z. Kira OOD 20 11 0 29 Oct 2023
Hallucination Detection for Grounded Instruction Generation Lingjun Zhao Khanh Nguyen Hal Daumé HILM 26 7 0 23 Oct 2023
CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models Sreyan Ghosh Ashish Seth Sonal Kumar Utkarsh Tyagi Chandra Kiran Reddy Evuru S. Ramaneswaran S. Sakshi Oriol Nieto R. Duraiswami Dinesh Manocha AuLLM VLM CoGe 30 21 0 12 Oct 2023
Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification Sravanti Addepalli Ashish Ramayee Asokan Lakshay Sharma R. V. Babu VLM 19 15 0 12 Oct 2023
Data Filtering Networks Alex Fang Albin Madappally Jose Amit Jain Ludwig Schmidt Alexander Toshev Vaishaal Shankar CLIP 18 121 0 29 Sep 2023
Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks Hao Chen Jindong Wang Ankit Shah Ran Tao Hongxin Wei Berfin cSimcsek Masashi Sugiyama Bhiksha Raj 22 26 0 29 Sep 2023
GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph Xin Li Dongze Lian Zhihe Lu Jiawang Bai Zhibo Chen Xinchao Wang VLM 27 58 0 24 Sep 2023
Mitigating the Alignment Tax of RLHF Yong Lin Hangyu Lin Wei Xiong Shizhe Diao Zeming Zheng ... Han Zhao Nan Jiang Heng Ji Yuan Yao Tong Zhang MoMe CLL 24 63 0 12 Sep 2023
Zero-Shot Robustification of Zero-Shot Models Dyah Adila Changho Shin Lin Cai Frederic Sala 29 18 0 08 Sep 2023
Fine-tuning can cripple your foundation model; preserving features may be the solution Jishnu Mukhoti Y. Gal Philip H. S. Torr P. Dokania CLL 26 29 0 25 Aug 2023
Overcoming Generic Knowledge Loss with Selective Parameter Update Wenxuan Zhang Paul Janson Rahaf Aljundi Mohamed Elhoseiny KELM CLL 19 10 0 23 Aug 2023
A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision Julio Silva-Rodríguez H. Chakor Riadh Kobbi Jose Dolz Ismail Ben Ayed VLM MedIm 64 33 0 15 Aug 2023
Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptability Seokhyeon Ha S. Jung Jungwook Lee 14 3 0 15 Aug 2023
Bridging the Gap: Exploring the Capabilities of Bridge-Architectures for Complex Visual Reasoning Tasks Kousik Rajesh Mrigank Raman M. A. Karim Pranit Chawla VLM 23 2 0 31 Jul 2023
GIST: Generating Image-Specific Text for Fine-grained Object Classification Kathleen M. Lewis Emily Mu Adrian V. Dalca John Guttag VLM 19 7 0 21 Jul 2023
Ethosight: A Reasoning-Guided Iterative Learning System for Nuanced Perception based on Joint-Embedding & Contextual Label Affinity Hugo Latapie Shan Yu Patrick Hammer K. Thórisson V. Petrosyan ... Mohsen Imani Mike Archbold Tangrui Li Pei Wang Justin W. Hart VLM 17 0 0 20 Jul 2023
Text Descriptions are Compressive and Invariant Representations for Visual Learning Zhili Feng Anna Bair J. Zico Kolter VLM 16 6 0 10 Jul 2023
T-MARS: Improving Visual Representations by Circumventing Text Feature Learning Pratyush Maini Sachin Goyal Zachary Chase Lipton J. Zico Kolter Aditi Raghunathan VLM 26 33 0 06 Jul 2023
Image Background Serves as Good Proxy for Out-of-distribution Data Sen Pei 30 1 0 02 Jul 2023
Neural Priming for Sample-Efficient Adaptation Matthew Wallingford Vivek Ramanujan Alex Fang Aditya Kusupati Roozbeh Mottaghi Aniruddha Kembhavi Ludwig Schmidt Ali Farhadi VLM 106 13 0 16 Jun 2023
LIV: Language-Image Representations and Rewards for Robotic Control Yecheng Jason Ma William Liang Vaidehi Som Vikash Kumar Amy Zhang Osbert Bastani Dinesh Jayaraman LM&Ro 15 119 0 01 Jun 2023
Contextual Vision Transformers for Robust Representation Learning Yu Bao Theofanis Karaletsos ViT 11 13 0 30 May 2023
Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality Harman Singh Pengchuan Zhang Qifan Wang Mengjiao MJ Wang Wenhan Xiong Jingfei Du Yu Chen CoGe VLM 18 24 0 23 May 2023
Universal Domain Adaptation from Foundation Models: A Baseline Study Bin Deng K. Jia VLM 14 6 0 18 May 2023
Consistency Regularization for Domain Generalization with Logit Attribution Matching Han Gao Kaican Li Weiyan Xie Zhi Lin Yongxiang Huang Luning Wang Caleb Chen Cao N. Zhang 13 2 0 13 May 2023
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? Arjun Majumdar Karmesh Yadav Sergio Arnaud Yecheng Jason Ma Claire Chen ... Dhruv Batra Yixin Lin Oleksandr Maksymets Aravind Rajeswaran Franziska Meier LM&Ro 14 171 0 31 Mar 2023
Model-tuning Via Prompts Makes NLP Models Adversarially Robust Mrigank Raman Pratyush Maini J. Zico Kolter Zachary Chase Lipton Danish Pruthi AAML 17 17 0 13 Mar 2023
CLIPood: Generalizing CLIP to Out-of-Distributions Yang Shu Xingzhuo Guo Jialong Wu Ximei Wang Jianmin Wang Mingsheng Long OODD VLM 41 74 0 02 Feb 2023
MolCPT: Molecule Continuous Prompt Tuning to Generalize Molecular Representation Learning Cameron Diao Kaixiong Zhou Zirui Liu Xiao Huang Xia Hu 17 14 0 20 Dec 2022
Geodesic Multi-Modal Mixup for Robust Fine-Tuning Changdae Oh Junhyuk So Hoyoon Byun Yongtaek Lim Minchul Shin Jong-June Jeon Kyungwoo Song 21 24 0 08 Mar 2022
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling Renrui Zhang Rongyao Fang Wei Zhang Peng Gao Kunchang Li Jifeng Dai Yu Qiao Hongsheng Li VLM 184 384 0 06 Nov 2021
Learning to Prompt for Vision-Language Models Kaiyang Zhou Jingkang Yang Chen Change Loy Ziwei Liu VPVLM CLIP VLM 322 2,249 0 02 Sep 2021
Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning Yifan Zhang Bryan Hooi Dapeng Hu Jian Liang Jiashi Feng 71 64 0 12 Feb 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 293 3,683 0 11 Feb 2021
Improved Baselines with Momentum Contrastive Learning Xinlei Chen Haoqi Fan Ross B. Girshick Kaiming He SSL 238 3,359 0 09 Mar 2020
FreeLB: Enhanced Adversarial Training for Natural Language Understanding Chen Zhu Yu Cheng Zhe Gan S. Sun Tom Goldstein Jingjing Liu AAML 221 430 0 25 Sep 2019
Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuning Weifeng Ge Yizhou Yu 86 232 0 28 Feb 2017