RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models

24 May 2025

Papers citing "RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models"

46 / 46 papers shown

Title
Quantum-PEFT: Ultra parameter-efficient fine-tuning Toshiaki Koike-Akino F. Tonin Yongtao Wu Frank Zhengqing Wu Leyla Naz Candogan Volkan Cevher MQ 128 5 0 07 Mar 2025
LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization Jui-Nan Yen Si Si Zhao Meng Felix X. Yu Sai Surya Duvvuri Inderjit Dhillon Cho-Jui Hsieh Sanjiv Kumar 47 5 0 27 Oct 2024
On the Crucial Role of Initialization for Matrix Factorization Bingcong Li Liang Zhang Aryan Mokhtari Niao He 109 4 0 24 Oct 2024
Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems Bingcong Li Liang Zhang Niao He 71 6 0 18 Oct 2024
LoRA-Pro: Are Low-Rank Adapters Properly Optimized? Zhengbo Wang Jian Liang Ran He Zilei Wang Tieniu Tan 97 25 0 25 Jul 2024
LoRA-GA: Low-Rank Adaptation with Gradient Approximation Shaowen Wang Linxi Yu Jian Li ALM AI4CE 64 41 0 06 Jul 2024
SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors Vijay Lingam Atula Tejaswi Aditya Vavre Aneesh Shetty Gautham Krishna Gudur Joydeep Ghosh Alexandros G. Dimakis Eunsol Choi Aleksandar Bojchevski Sujay Sanghavi 82 14 0 30 May 2024
Parameter-Efficient Fine-Tuning with Discrete Fourier Transform Ziqi Gao Qichao Wang Aochuan Chen Zijing Liu Bingzhe Wu Liang Chen Jia Li 59 28 0 05 May 2024
ReFT: Representation Finetuning for Language Models Zhengxuan Wu Aryaman Arora Zheng Wang Atticus Geiger Daniel Jurafsky Christopher D. Manning Christopher Potts OffRL 85 67 0 04 Apr 2024
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models Fanxu Meng Zhaohui Wang Muhan Zhang VLM 103 92 0 03 Apr 2024
LoRA+: Efficient Low Rank Adaptation of Large Models Soufiane Hayou Nikhil Ghosh Bin Yu AI4CE 69 161 0 19 Feb 2024
DoRA: Weight-Decomposed Low-Rank Adaptation Shih-yang Liu Chien-Yi Wang Hongxu Yin Pavlo Molchanov Yu-Chiang Frank Wang Kwang-Ting Cheng Min-Hung Chen 64 374 0 14 Feb 2024
Flora: Low-Rank Adapters Are Secretly Gradient Compressors Yongchang Hao Yanshuai Cao Lili Mou 49 47 0 05 Feb 2024
Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models Fangzhao Zhang Mert Pilanci AI4CE 97 20 0 04 Feb 2024
Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization Weiyang Liu Zeju Qiu Yao Feng Yuliang Xiu Yuxuan Xue ... Songyou Peng Yandong Wen Michael J. Black Adrian Weller Bernhard Schölkopf 70 63 0 10 Nov 2023
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models Yixiao Li Yifan Yu Chen Liang Pengcheng He Nikos Karampatziakis Weizhu Chen Tuo Zhao MQ 53 133 0 12 Oct 2023
Enhancing Sharpness-Aware Optimization Through Variance Suppression Bingcong Li G. Giannakis AAML 52 20 0 27 Sep 2023
Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation Shin-Ying Yeh Yu-Guan Hsieh Zhidong Gao Bernard B. W. Yang Giyeong Oh Yanmin Gong 28 78 0 26 Sep 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 206 11,636 0 18 Jul 2023
ReLoRA: High-Rank Training Through Low-Rank Updates Vladislav Lialin Namrata Shivagunde Sherin Muckatira Anna Rumshisky BDL 53 102 0 11 Jul 2023
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models Zhiqiang Hu Lei Wang Yihuai Lan Wanyu Xu Ee-Peng Lim Lidong Bing Xing Xu Soujanya Poria Roy Ka-wei Lee ALM 78 249 0 04 Apr 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 631 13,788 0 15 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 813 12,840 0 27 Feb 2023
Galactica: A Large Language Model for Science Ross Taylor Marcin Kardas Guillem Cucurull Thomas Scialom Anthony Hartshorn Elvis Saravia Andrew Poulton Viktor Kerkez Robert Stojnic ELM ReLM 88 754 0 16 Nov 2022
Training Neural Networks with Fixed Sparse Masks Yi-Lin Sung Varun Nair Colin Raffel FedML 64 200 0 18 Nov 2021
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing Pengcheng He Jianfeng Gao Weizhu Chen 112 1,155 0 18 Nov 2021
FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning Nam Hyeon-Woo Moon Ye-Bin Tae-Hyun Oh FedML 51 120 0 13 Aug 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 155 5,328 0 07 Jul 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 458 3,952 0 18 Apr 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 174 4,209 0 01 Jan 2021
Parameter-Efficient Transfer Learning with Diff Pruning Demi Guo Alexander M. Rush Yoon Kim 50 391 0 14 Dec 2020
AdapterDrop: On the Efficiency of Adapters in Transformers Andreas Rucklé Gregor Geigle Max Glockner Tilman Beck Jonas Pfeiffer Nils Reimers Iryna Gurevych 96 259 0 22 Oct 2020
Sharpness-Aware Minimization for Efficiently Improving Generalization Pierre Foret Ariel Kleiner H. Mobahi Behnam Neyshabur AAML 164 1,323 0 03 Oct 2020
Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent Tian Tong Cong Ma Yuejie Chi 54 116 0 18 May 2020
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning Zhaojiang Lin Andrea Madotto Pascale Fung 82 160 0 08 Apr 2020
PIQA: Reasoning about Physical Commonsense in Natural Language Yonatan Bisk Rowan Zellers Ronan Le Bras Jianfeng Gao Yejin Choi OOD LRM 91 1,724 0 26 Nov 2019
HellaSwag: Can a Machine Really Finish Your Sentence? Rowan Zellers Ari Holtzman Yonatan Bisk Ali Farhadi Yejin Choi 80 2,373 0 19 May 2019
Parameter-Efficient Transfer Learning for NLP N. Houlsby A. Giurgiu Stanislaw Jastrzebski Bruna Morrone Quentin de Laroussilhe Andrea Gesmundo Mona Attariyan Sylvain Gelly 189 4,368 0 02 Feb 2019
A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks Sanjeev Arora Nadav Cohen Noah Golowich Wei Hu 100 284 0 04 Oct 2018
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering Todor Mihaylov Peter Clark Tushar Khot Ashish Sabharwal 74 1,475 0 08 Sep 2018
Know What You Don't Know: Unanswerable Questions for SQuAD Pranav Rajpurkar Robin Jia Percy Liang RALM ELM 192 2,830 0 11 Jun 2018
Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced S. Du Wei Hu Jason D. Lee MLT 119 239 0 04 Jun 2018
Neural Network Acceptability Judgments Alex Warstadt Amanpreet Singh Samuel R. Bowman 174 1,390 0 31 May 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 658 7,080 0 20 Apr 2018
SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation Daniel Cer Mona T. Diab Eneko Agirre I. Lopez-Gazpio Lucia Specia 210 1,870 0 31 Jul 2017
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams Nikita Nangia Samuel R. Bowman 402 4,444 0 18 Apr 2017