v1v2v3 (latest)

Fine-tuning can cripple your foundation model; preserving features may be the solution

25 August 2023

Papers citing "Fine-tuning can cripple your foundation model; preserving features may be the solution"

45 / 45 papers shown

Title
FarSLIP: Discovering Effective CLIP Adaptation for Fine-Grained Remote Sensing Understanding Z. Li W. Yu Dilxat Muhtar X. Zhang Pengfeng Xiao Pedram Ghamisi Xiao Xiang Zhu CLIP VLM 136 0 0 18 Nov 2025
Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler Zixuan Hu Li Shen Zhenyi Wang Yongxian Wei Dacheng Tao AAML 99 0 0 31 Oct 2025
A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space Bingjie Zhang Yibo Yang Renzhe Dandan Guo Jindong Gu Philip Torr Bernard Ghanem 191 0 0 16 Oct 2025
Trade-offs in Cross-Domain Generalization of Foundation Model Fine-Tuned for Biometric Applications Tahar Chettaoui Naser Damer Fadi Boutros CVBM VLM 185 0 0 18 Sep 2025
Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performance Xiang Yuan Jun Shu Deyu Meng Zongben Xu AAML 52 0 0 31 Aug 2025
Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuning Weitao Feng Lixu Wang Tianyi Wei Jie Zhang Chongyang Gao Sinong Zhan Peizhuo Lv Wei Dong AAML OffRL CLL 56 0 0 28 Aug 2025
Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation Chun-Peng Chang Chen-Yu Wang Julian Schmidt Holger Caesar A. Pagani VGen 183 1 0 22 Aug 2025
Infusing fine-grained visual knowledge to Vision-Language Models Nikolaos-Antonios Ypsilantis Kaifeng Chen A. Araújo Ondrej Chum CLL VLM 84 0 0 16 Aug 2025
Gradient Surgery for Safe LLM Fine-Tuning Biao Yi Jiahao Li Baolei Zhang Lihai Nie Tong Li Tiansheng Huang Zheli Liu 78 1 0 10 Aug 2025
Calibrated Language Models and How to Find Them with Label Smoothing J. Huang Peng Lu Qiuhao Zeng 172 1 0 01 Aug 2025
LoRA is All You Need for Safety Alignment of Reasoning LLMs Yihao Xue Baharan Mirzasoleiman MoMe LRM 293 0 0 22 Jul 2025
Subspace-Boosted Model Merging Ronald Skorobogat Karsten Roth Mariana-Iuliana Georgescu MoMe 323 2 0 19 Jun 2025
LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning Gabrel J. Perin Runjin Chen Xuxi Chen Nina S. T. Hirata Zinan Lin Junyuan Hong AAML 255 1 0 18 Jun 2025
Multi-Scale Finetuning for Encoder-based Time Series Foundation Models Zhongzheng Qiao Chenghao Liu Y. Zhang Ming Jin Quang Pham Qingsong Wen P.N. Suganthan Xudong Jiang Savitha Ramasamy AI4TS AI4CE 230 1 0 17 Jun 2025
AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basin Shuo Yang Qihui Zhang Yuyang Liu Yue Huang Xiaojun Jia ... Jiayu Yao Jigang Wang Hailiang Dai Yibing Song Li Yuan 190 8 0 10 Jun 2025
Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets Lei Hsiung Tianyu Pang Yung-Chen Tang Linyue Song Tsung-Yi Ho Pin-Yu Chen Yaoqing Yang 260 6 0 05 Jun 2025
Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning Liang Chen Xueting Han Li Shen Jing Bai Kam-Fai Wong AAML 246 4 0 04 Jun 2025
Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting Chen Huang Skyler Seto Hadi Pouransari Mehrdad Farajtabar Raviteja Vemulapalli Fartash Faghri Oncel Tuzel B. Theobald Josh Susskind CLL 232 0 0 30 May 2025
Unveiling the Basin-Like Loss Landscape in Large Language Models Huanran Chen Yinpeng Dong Zeming Wei Yao Huang Yichi Zhang Hang Su Jun Zhu MoMe 309 5 0 23 May 2025
Shape it Up! Restoring LLM Safety during Finetuning ShengYun Peng Pin-Yu Chen Jianfeng Chi Seongmin Lee Duen Horng Chau LLMAG 228 3 0 22 May 2025
CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuning Biao Yi Tiansheng Huang Baolei Zhang Tong Li Lihai Nie Zheli Liu Li Shen MU AAML 259 5 0 22 May 2025
CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation Anna C. Doris Md Ferdous Alam Amin Heyrani Nobari Faez Ahmed 187 7 0 20 May 2025
MoCLIP: Motion-Aware Fine-Tuning and Distillation of CLIP for Human Motion Generation Gabriel Maldonado Armin Danesh Pazho Ghazal Alinezhad Noghre Vinit Katariya Hamed Tabkhi CLIP VGen 276 0 0 16 May 2025
Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety Zihan Guan Mengxuan Hu Ronghang Zhu Sheng Li Anil Vullikanti AAML 255 10 0 11 May 2025
Vanishing Depth: A Depth Adapter with Positional Depth Encoding for Generalized Image Encoders Paul Koch Jörg Krüger Ankit Chowdhury O. Heimann MDE 216 0 0 25 Mar 2025
SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging Aladin Djuhera S. Kadhe Praneet Adusumilli Syed Zawad Holger Boche MoMe 193 13 0 21 Mar 2025
Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation Yun Wang Tiansheng Huang Li Shen Huanjin Yao Haotian Luo Rui Liu Naiqiang Tan Jiaxing Huang Dacheng Tao AAML MoMe CLL 339 9 0 30 Jan 2025
Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt TemplatesNeural Information Processing Systems (NeurIPS), 2024 Kaifeng Lyu Haoyu Zhao Xinran Gu Dingli Yu Anirudh Goyal Sanjeev Arora ALM 315 83 0 20 Jan 2025
SaLoRA: Safety-Alignment Preserved Low-Rank AdaptationInternational Conference on Learning Representations (ICLR), 2025 Mingjie Li Wai Man Si Michael Backes Yang Zhang Yisen Wang 284 35 0 03 Jan 2025
PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning Shenghui Li Edith C.H. Ngai Fanghua Ye Thiemo Voigt SILM 351 7 0 28 Nov 2024
FEET: A Framework for Evaluating Embedding Techniques Simon A. Lee John Lee Jeffrey N. Chiang 157 5 0 02 Nov 2024
Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Model Divyanshu Aggarwal Sankarshan Damle Navin Goyal Satya Lokam Sunayana Sitaram CLL 199 3 0 21 Oct 2024
Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise PerturbationIEEE Transactions on Information Forensics and Security (IEEE TIFS), 2024 Guozhi Liu Weiwei Lin Tiansheng Huang Ruichao Mo Qi Mu Li Shen AAML 286 28 0 13 Oct 2024
SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data SelectionInternational Conference on Learning Representations (ICLR), 2024 Han Shen Pin-Yu Chen Payel Das Tianyi Chen ALM 237 44 0 09 Oct 2024
Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey Tiansheng Huang Sihao Hu Fatih Ilhan Selim Furkan Tekin Ling Liu AAML 324 78 0 26 Sep 2024
Minimizing Embedding Distortion for Robust Out-of-Distribution Performance Tom Shaked Yuval Goldman Oran Shayer OODD 142 0 0 11 Sep 2024
Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Tiansheng Huang Gautam Bhattacharya Pratik Joshi Josh Kimball Ling Liu AAML MoMe 447 45 0 18 Aug 2024
ICLGuard: Controlling In-Context Learning Behavior for Applicability Authorization Wai Man Si Michael Backes Yang Zhang 145 1 0 09 Jul 2024
Safety Alignment Should Be Made More Than Just a Few Tokens DeepInternational Conference on Learning Representations (ICLR), 2024 Xiangyu Qi Ashwinee Panda Kaifeng Lyu Xiao Ma Subhrajit Roy Ahmad Beirami Prateek Mittal Peter Henderson 207 256 0 10 Jun 2024
kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies Zhongrui Gui Shuyang Sun Runjia Li Jianhao Yuan Zhaochong An Karsten Roth Christian Schroeder de Witt Juil Sock VLM CLL 242 16 0 15 Apr 2024
Vaccine: Perturbation-aware Alignment for Large Language Model Tiansheng Huang Sihao Hu Ling Liu 366 77 0 02 Feb 2024
AutoFT: Learning an Objective for Robust Fine-Tuning Caroline Choi Yoonho Lee Annie S. Chen Allan Zhou Aditi Raghunathan Chelsea Finn OOD 253 1 0 18 Jan 2024
When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and LimitationsInternational Conference on Learning Representations (ICLR), 2023 Aleksandar Petrov Juil Sock Adel Bibi VPVLM 268 35 0 30 Oct 2023
FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot LearningNeural Information Processing Systems (NeurIPS), 2023 Kun Song Huimin Ma Bochao Zou Huishuai Zhang Weiran Huang 260 15 0 23 Oct 2023
CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision (IJCV), 2021 Shiyang Feng Shijie Geng Renrui Zhang Teli Ma Rongyao Fang Zelong Li Jiaming Song Yu Qiao VLM CLIP 1.0K 1,385 0 09 Oct 2021