Tied-Lora: Enhancing parameter efficiency of LoRA with weight tying

16 November 2023

Adithya Renduchintala

Papers citing "Tied-Lora: Enhancing parameter efficiency of LoRA with weight tying"

33 / 33 papers shown

Title
Concept-Aware LoRA for Domain-Aligned Segmentation Dataset Generation Minho Park S. Park Jungsoo Lee Hyojin Park Kyuwoong Hwang Fatih Porikli Jaegul Choo Sungha Choi 29 0 0 28 Mar 2025
RaSA: Rank-Sharing Low-Rank Adaptation Zhiwei He Zhaopeng Tu Xing Wang Xingyu Chen Z. Wang Jiahao Xu Tian Liang Wenxiang Jiao Z. Zhang Rui Wang ALM 82 1 0 16 Mar 2025
SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation Yixian Shen Qi Bi Jia-Hong Huang Hongyi Zhu Andy D. Pimentel Anuj Pathania 44 0 0 08 Feb 2025
SubTrack your Grad: Gradient Subspace Tracking for Memory and Time Efficient Full-Parameter LLM Training Sahar Rajabi Nayeema Nonta Sirisha Rambhatla 82 0 0 03 Feb 2025
Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation Reza Akbarian Bafghi Carden Bagwell Avinash Ravichandran Ashish Shrivastava M. Raissi 40 0 0 28 Jan 2025
Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning Kaustubh Ponkshe Raghav Singhal Eduard A. Gorbunov Alexey Tumanov Samuel Horváth Praneeth Vepakomma 63 1 0 29 Nov 2024
Closed-form merging of parameter-efficient modules for Federated Continual Learning Riccardo Salami Pietro Buzzega Matteo Mosconi Jacopo Bonato Luigi Sabetta Simone Calderara FedML MoMe CLL 29 2 0 23 Oct 2024
DEAN: Deactivating the Coupled Neurons to Mitigate Fairness-Privacy Conflicts in Large Language Models Chen Qian Dongrui Liu Jie Zhang Yong Liu Jing Shao 24 1 0 22 Oct 2024
LoKO: Low-Rank Kalman Optimizer for Online Fine-Tuning of Large Models Hossein Abdi Mingfei Sun Andi Zhang Samuel Kaski Wei Pan 23 0 0 15 Oct 2024
ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flaws Hai Huang Randall Balestriero 30 0 0 13 Oct 2024
Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning Nusrat Jahan Prottasha Asif Mahmud Md. Shohanur Islam Sobuj Prakash Bhat Md. Kowsher Niloofar Yousefi O. Garibay 30 4 0 11 Oct 2024
Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMs Ruijia Niu D. Wu Rose Yu Yi-An Ma 18 1 0 09 Oct 2024
MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards Sheng Wang Liheng Chen Pengan Chen Jingwei Dong Boyang Xue Jiyue Jiang Lingpeng Kong Chuan Wu MoE 16 7 0 01 Oct 2024
NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models Cheng Lin Lujun Li Dezhi Li Jie Zou Wei Xue Yike Guo AI4TS 28 4 0 18 Aug 2024
SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis Yuchen Mao Hongwei Bran Li Wei Pang G. Papanastasiou G. Yang Chengjia Wang MedIm 32 0 0 13 Aug 2024
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients Ajay Jaiswal Lu Yin Zhenyu (Allen) Zhang Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang 31 14 0 15 Jul 2024
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients Zhenyu (Allen) Zhang Ajay Jaiswal L. Yin Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang VLM 21 1 0 11 Jul 2024
A Survey on LoRA of Large Language Models Yuren Mao Yuhang Ge Yijiang Fan Wenyi Xu Yu Mi Zhonghao Hu Yunjun Gao ALM 52 22 0 08 Jul 2024
See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition Chongjie Si Xiaokang Yang Wei Shen 27 5 0 07 Jul 2024
SBoRA: Low-Rank Adaptation with Regional Weight Updates L. Po Yuyang Liu Haoxuan Wu Tianqi Zhang W. Yu Zeyu Jiang Kun Li 33 0 0 07 Jul 2024
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning Yibo Yang Xiaojie Li Zhongzhu Zhou S. Song Jianlong Wu Liqiang Nie Bernard Ghanem 43 6 0 07 Jun 2024
Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision Minglei Li Peng Ye Yongqi Huang Lin Zhang Tao Chen Tong He Jiayuan Fan Wanli Ouyang MoE 22 4 0 05 Jun 2024
Low-rank finetuning for LLMs: A fairness perspective Saswat Das Marco Romanelli Cuong Tran Zarreen Reza B. Kailkhura Ferdinando Fioretto 22 1 0 28 May 2024
OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning Pengxiang Li Lu Yin Xiaowei Gao Shiwei Liu 19 7 0 28 May 2024
VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections Roy Miles Pradyumna Reddy Ismail Elezi Jiankang Deng VLM 19 3 0 28 May 2024
LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters Klaudia Bałazy Mohammadreza Banaei Karl Aberer Jacek Tabor 38 16 0 27 May 2024
VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks Yang Li Shaobo Han Shihao Ji MoE 22 4 0 24 May 2024
Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning Chongjie Si Xuehui Wang Xue Yang Zhengqin Xu Qingyun Li Jifeng Dai Yu Qiao Xiaokang Yang Wei Shen 26 8 0 23 May 2024
MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts Dengchun Li Yingzi Ma Naizheng Wang Zhengmao Ye Zhiyuan Cheng ... Yan Zhang Lei Duan Jie Zuo Cal Yang Mingjie Tang MoE 24 13 0 22 Apr 2024
FSViewFusion: Few-Shots View Generation of Novel Objects Rukhshanda Hussain Hui Xian Grace Lim Borchun Chen Mubarak Shah Ser Nam Lim DiffM 18 0 0 11 Mar 2024
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Jiawei Zhao Zhenyu (Allen) Zhang Beidi Chen Zhangyang Wang A. Anandkumar Yuandong Tian 27 173 0 06 Mar 2024
DoRA: Weight-Decomposed Low-Rank Adaptation Shih-yang Liu Chien-Yi Wang Hongxu Yin Pavlo Molchanov Yu-Chiang Frank Wang Kwang-Ting Cheng Min-Hung Chen 17 335 0 14 Feb 2024
Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning Wenhan Xia Chengwei Qin Elad Hazan 46 52 0 08 Jan 2024