MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

13 June 2024

Hanqing Wang

Shuo Wang

Guanhua Chen

Papers citing "MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning"

19 / 19 papers shown

Title
Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformations Chongjie Si Zhiyi Shi Xuehui Wang Yichen Xiao Xiaokang Yang Wei-Ming Shen AI4CE 56 0 0 01 Apr 2025
Decoupling Angles and Strength in Low-rank Adaptation Massimo Bini Leander Girrbach Zeynep Akata 40 0 0 23 Mar 2025
Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization Ziqing Xu Hancheng Min Lachlan Ewen MacDonald Jinqi Luo Salma Tarmoun Enrique Mallada René Vidal AI4CE 46 0 0 10 Mar 2025
Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model Wenke Huang Jian Liang Xianda Guo Yiyang Fang Guancheng Wan ... Bin Yang He Li Jiawei Shao Mang Ye Bo Du OffRL LRM MLLM KELM VLM 63 1 0 06 Mar 2025
NEAT: Nonlinear Parameter-efficient Adaptation of Pre-trained Models Yibo Zhong Haoxiang Jiang Lincan Li Ryumei Nakada Tianci Liu Linjun Zhang Huaxiu Yao Haoyu Wang 70 2 0 24 Feb 2025
A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models Mengyang Sun Yihao Wang Tao Feng Dan Zhang Yifan Zhu J. Tang MoE 29 0 0 20 Feb 2025
GoRA: Gradient-driven Adaptive Low Rank Adaptation Haonan He Peng Ye Yuchen Ren Yuan Yuan Lei Chen AI4TS AI4CE 62 0 0 13 Feb 2025
Spectral-Aware Low-Rank Adaptation for Speaker Verification Zhe Li Man-Wai Mak Mert Pilanci Hung-yi Lee H. Meng 41 0 0 07 Jan 2025
DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models Xiaolin Hu Xiang Cheng Peiyu Liu Wei Liu Jian Luan Bin Wang Y. Liu 37 1 0 31 Dec 2024
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models Fan Wang Juyong Jiang Chansung Park Sunghun Kim Jing Tang 86 0 0 08 Dec 2024
Training-Free Bayesianization for Low-Rank Adapters of Large Language Models H. Shi Yibin Wang Ligong Han H. M. Zhang Hao Wang UQCV 83 0 0 07 Dec 2024
Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models Yuheng Lu Bingshuo Qian Caixia Yuan Huixing Jiang Xiaojie Wang CLL 18 0 0 22 Oct 2024
PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning Qibin Wang Xiaolin Hu Weikai Xu Wei Liu Jian Luan Bin Wang 18 1 0 25 Sep 2024
Low-Rank Interconnected Adaptation Across Layers Yibo Zhong Yao Zhou OffRL MoE 36 1 0 13 Jul 2024
A Survey on LoRA of Large Language Models Yuren Mao Yuhang Ge Yijiang Fan Wenyi Xu Yu Mi Zhonghao Hu Yunjun Gao ALM 52 22 0 08 Jul 2024
Scaling Laws for Forgetting When Fine-Tuning Large Language Models Damjan Kalajdzievski CLL 24 8 0 11 Jan 2024
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 275 3,784 0 18 Apr 2021