DoRA: Weight-Decomposed Low-Rank Adaptation

14 February 2024

Papers citing "DoRA: Weight-Decomposed Low-Rank Adaptation"

50 / 63 papers shown

Title
Not All Parameters Matter: Masking Diffusion Models for Enhancing Generation Ability L. Wang Senmao Li Fei Yang Jianye Wang Ziheng Zhang Y. Liu Y. Wang Jian Yang DiffM 52 0 0 06 May 2025
NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation Rob Romijnders Stefanos Laskaridis Ali Shahin Shamsabadi Hamed Haddadi 54 0 0 25 Apr 2025
Parameter-Efficient Checkpoint Merging via Metrics-Weighted Averaging Shi Jie Yu Sehyun Choi MoMe 45 0 0 23 Apr 2025
Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management Hang Zhang Jiuchen Shi Yixiao Wang Quan Chen Yizhou Shan Minyi Guo 25 0 0 19 Apr 2025
DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models Wenjin Ke Zhe Li D. Li Lu Tian E. Barsoum MQ 27 1 0 12 Apr 2025
Pretraining Language Models for Diachronic Linguistic Change Discovery Elisabeth Fittschen Sabrina Li Tom Lippincott Leshem Choshen Craig Messner 23 0 0 07 Apr 2025
QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understanding Binh M. Le Shaoyuan Xu Jinmiao Fu Zhishen Huang Moyan Li Yanhui Guo Hongdong Li Sameera Ramasinghe Bryan Wang 28 0 0 03 Apr 2025
Towards Symmetric Low-Rank Adapters Tales Panoutsos Rodrygo L. T. Santos Flavio Figueiredo 23 0 0 29 Mar 2025
Endo-FASt3r: Endoscopic Foundation model Adaptation for Structure from motion Mona Sheikh Zeinoddin Mobarakol Islam Zafer Tandogdu Greg Shaw Mathew J. Clarkson E. Mazomenos Danail Stoyanov 61 0 0 10 Mar 2025
MoFE: Mixture of Frozen Experts Architecture Jean Seo Jaeyoon Kim Hyopil Shin MoE 71 0 0 09 Mar 2025
Parameter-Efficient Fine-Tuning of Large Language Models via Deconvolution in Subspace Jia-Chen Zhang Yu-Jie Xiong Chun-Ming Xia Dong-Hai Zhu Xi-He Qiu 61 1 0 03 Mar 2025
PaCA: Partial Connection Adaptation for Efficient Fine-Tuning Sunghyeon Woo Sol Namkung Sunwoo Lee Inho Jeong Beomseok Kim Dongsuk Jeon 33 0 0 28 Feb 2025
FedMentalCare: Towards Privacy-Preserving Fine-Tuned LLMs to Analyze Mental Health Status Using Federated Learning Framework S M Sarwar AI4MH 39 0 0 27 Feb 2025
NEAT: Nonlinear Parameter-efficient Adaptation of Pre-trained Models Yibo Zhong Haoxiang Jiang Lincan Li Ryumei Nakada Tianci Liu Linjun Zhang Huaxiu Yao Haoyu Wang 70 2 0 24 Feb 2025
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning Raghav Singhal Kaustubh Ponkshe Rohit Vartak Lav R. Varshney Praneeth Vepakomma FedML 74 0 0 24 Feb 2025
DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations Krishna Sri Ipsit Mantri Carola-Bibiane Schönlieb Bruno Ribeiro Chaim Baskin Moshe Eliasof 36 0 0 09 Feb 2025
SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation Yixian Shen Qi Bi Jia-Hong Huang Hongyi Zhu Andy D. Pimentel Anuj Pathania 46 0 0 08 Feb 2025
LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning Zhekai Du Yinjie Min Jingjing Li Ke Lu Changliang Zou Liuhua Peng Tingjin Chu M. Gong 113 1 0 05 Feb 2025
RandLoRA: Full-rank parameter-efficient fine-tuning of large models Paul Albert Frederic Z. Zhang Hemanth Saratchandran Cristian Rodriguez-Opazo Anton van den Hengel Ehsan Abbasnejad 94 0 0 03 Feb 2025
Sparse High Rank Adapters K. Bhardwaj N. Pandey Sweta Priyadarshi Viswanath Ganapathy Rafael Esteves ... P. Whatmough Risheek Garrepalli M. V. Baalen Harris Teague Markus Nagel MQ 33 4 0 28 Jan 2025
RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation Jun Liu Zhenglun Kong Peiyan Dong Changdi Yang Xuan Shen ... Wei Niu Wenbin Zhang Xue Lin Dong Huang Yanzhi Wang ALM 34 2 0 08 Jan 2025
Spectral-Aware Low-Rank Adaptation for Speaker Verification Zhe Li Man-Wai Mak Mert Pilanci Hung-yi Lee H. Meng 41 0 0 07 Jan 2025
GaLore $+$ : Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection Xutao Liao Shaohui Li Yuhui Xu Zhi Li Y. Liu You He VLM 54 2 0 31 Dec 2024
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models Fan Wang Juyong Jiang Chansung Park Sunghun Kim Jing Tang 91 1 0 08 Dec 2024
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection Jinqi Xiao S. Sang Tiancheng Zhi Jing Liu Qing Yan Linjie Luo Bo Yuan Bo Yuan VLM 81 1 0 26 Nov 2024
RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks Nazia Tasnim Bryan A. Plummer CLL OffRL 72 0 0 25 Nov 2024
Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation Seokil Ham H. Kim Sangmin Woo Changick Kim Mamba 103 0 0 21 Nov 2024
Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass Tong Chen Hao Fang Patrick Xia Xiaodong Liu Benjamin Van Durme Luke Zettlemoyer Jianfeng Gao Hao Cheng KELM 38 2 0 08 Nov 2024
MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba Masakazu Yoshimura Teruaki Hayashi Yota Maeda Mamba 66 2 0 06 Nov 2024
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies L. Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 44 3 0 24 Oct 2024
Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models Yuheng Lu Bingshuo Qian Caixia Yuan Huixing Jiang Xiaojie Wang CLL 32 0 0 22 Oct 2024
MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning Yaming Yang Dilxat Muhtar Yelong Shen Yuefeng Zhan Jianfeng Liu ... Denvy Deng Feng Sun Qi Zhang Weizhu Chen Yunhai Tong MoE MoMe 38 2 0 12 Oct 2024
Parameter-Efficient Fine-Tuning of State Space Models Kevin Galim Wonjun Kang Yuchen Zeng H. Koo Kangwook Lee 29 4 0 11 Oct 2024
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture Jiayi Han Liang Du Hongwei Du Xiangguo Zhou Yiwen Wu Weibo Zheng Donghong Han CLL MoMe MoE 33 2 0 10 Oct 2024
Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMs Ruijia Niu D. Wu Rose Yu Yi-An Ma 18 1 0 09 Oct 2024
Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization Zichen Miao Zhengyuan Yang Kevin Lin Ze Wang Zicheng Liu Lijuan Wang Qiang Qiu 40 3 0 04 Oct 2024
Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization Xinhao Yao Hongjin Qian Xiaolin Hu Gengze Xu Wei Liu Jian Luan B. Wang Y. Liu 45 0 0 03 Oct 2024
Selective Aggregation for Low-Rank Adaptation in Federated Learning Pengxin Guo Shuang Zeng Y. Wang Huijie Fan Feifei Wang Liangqiong Qu FedML 34 8 0 02 Oct 2024
House of Cards: Massive Weights in LLMs Jaehoon Oh Seungjun Shin Dokwan Oh 35 1 0 02 Oct 2024
Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data Atilla Akkus Mingjie Li Junjie Chu Junjie Chu Michael Backes Sinem Sav Sinem Sav SILM SyDa 28 1 0 12 Sep 2024
Multi-Modal Adapter for Vision-Language Models Dominykas Seputis Serghei Mihailov Soham Chatterjee Zehao Xiao VLM 19 1 0 03 Sep 2024
Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement Le Yu Bowen Yu Haiyang Yu Fei Huang Yongbin Li MoMe 27 5 0 06 Aug 2024
A Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM in 3D Segmentation Mothilal Asokan Difei Gao Joya Chen Mike Zheng Shou FedML MedIm 33 1 0 31 Jul 2024
LoRA-Pro: Are Low-Rank Adapters Properly Optimized? Zhengbo Wang Jian Liang Ran He Zilei Wang Tieniu Tan 47 15 0 25 Jul 2024
Low-Rank Interconnected Adaptation Across Layers Yibo Zhong Yao Zhou OffRL MoE 38 1 0 13 Jul 2024
Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences Nikolaos Dimitriadis Pascal Frossard F. Fleuret MoE 57 6 0 10 Jul 2024
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning Hanqing Wang Zeguan Xiao Shuo Wang Guanhua Chen Guanhua Chen 30 19 0 13 Jun 2024
The Impact of Initialization on LoRA Finetuning Dynamics Soufiane Hayou Nikhil Ghosh Bin Yu AI4CE 34 10 0 12 Jun 2024
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning Yibo Yang Xiaojie Li Zhongzhu Zhou S. Song Jianlong Wu Liqiang Nie Bernard Ghanem 43 6 0 07 Jun 2024
Low-Rank Adaption on Transformer-based Oriented Object Detector for Satellite Onboard Processing of Remote Sensing Images Xinyang Pu Feng Xu 32 3 0 04 Jun 2024