The Impact of Initialization on LoRA Finetuning Dynamics

The Impact of Initialization on LoRA Finetuning Dynamics

12 June 2024

Papers citing "The Impact of Initialization on LoRA Finetuning Dynamics"

12 / 12 papers shown

Title
Decoupling Angles and Strength in Low-rank Adaptation Massimo Bini Leander Girrbach Zeynep Akata 37 0 0 23 Mar 2025
Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization Ziqing Xu Hancheng Min Lachlan Ewen MacDonald Jinqi Luo Salma Tarmoun Enrique Mallada René Vidal AI4CE 43 0 0 10 Mar 2025
On the Crucial Role of Initialization for Matrix Factorization Bingcong Li Liang Zhang Aryan Mokhtari Niao He 15 1 0 24 Oct 2024
RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates Md. Kowsher Tara Esmaeilbeig Chun-Nam Yu Mojtaba Soltanalian Niloofar Yousefi 22 0 0 14 Oct 2024
ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flaws Hai Huang Randall Balestriero 20 0 0 13 Oct 2024
One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation Fabian Paischer Lukas Hauzenberger Thomas Schmied Benedikt Alkin Marc Peter Deisenroth Sepp Hochreiter 14 1 0 09 Oct 2024
Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization Xinhao Yao Hongjin Qian Xiaolin Hu Gengze Xu Yong Liu Wei Liu Jian Luan Bin Wang 31 0 0 03 Oct 2024
A Survey on LoRA of Large Language Models Yuren Mao Yuhang Ge Yijiang Fan Wenyi Xu Yu Mi Zhonghao Hu Yunjun Gao ALM 52 22 0 08 Jul 2024
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 275 3,784 0 18 Apr 2021
Stable ResNet Soufiane Hayou Eugenio Clerico Bo He George Deligiannidis Arnaud Doucet Judith Rousseau ODL SSeg 38 50 0 24 Oct 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 3,054 0 23 Jan 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,927 0 20 Apr 2018