PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

3 April 2024

Papers citing "PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models"

50 / 62 papers shown

Title
Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptation Nairouz Mrabah Nicolas Richet Ismail Ben Ayed Eric Granger BDL VLM 42 0 0 16 Apr 2025
RWKVTTS: Yet another TTS based on RWKV-7 Lin Yueyu Liu Xiao 36 0 0 04 Apr 2025
Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformations Chongjie Si Zhiyi Shi Xuehui Wang Yichen Xiao Xiaokang Yang Wei-Ming Shen AI4CE 56 0 0 01 Apr 2025
MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning Jiancheng Zhao Xingda Yu Zhen Yang MoE 43 1 0 27 Mar 2025
Low-resource Machine Translation for Code-switched Kazakh-Russian Language Pair Maksim Borisov Zhanibek Kozhirbayev Valentin Malykh 38 0 0 25 Mar 2025
Hiding Images in Diffusion Models by Editing Learned Score Functions Haoyu Chen Yunqiao Yang Nan Zhong Kede Ma DiffM 60 0 0 24 Mar 2025
Decoupling Angles and Strength in Low-rank Adaptation Massimo Bini Leander Girrbach Zeynep Akata 35 0 0 23 Mar 2025
SALT: Singular Value Adaptation with Low-Rank Transformation Abdelrahman Elsayed Sarim Hashmi Mohammed Elseiagy Hu Wang Mohammad Yaqub Ibrahim Almakky OOD 41 0 0 20 Mar 2025
A Foundational individual Mobility Prediction Model based on Open-Source Large Language Models Zhenlin Qin Leizhen Wang Francisco Camara Pereira Zhenlinag Ma 22 0 0 19 Mar 2025
RaSA: Rank-Sharing Low-Rank Adaptation Zhiwei He Zhaopeng Tu Xing Wang Xingyu Chen Z. Wang Jiahao Xu Tian Liang Wenxiang Jiao Z. Zhang Rui Wang ALM 77 1 0 16 Mar 2025
Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization Ziqing Xu Hancheng Min Lachlan Ewen MacDonald Jinqi Luo Salma Tarmoun Enrique Mallada René Vidal AI4CE 41 0 0 10 Mar 2025
Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model Wenke Huang Jian Liang Xianda Guo Yiyang Fang Guancheng Wan ... Bin Yang He Li Jiawei Shao Mang Ye Bo Du OffRL LRM MLLM KELM VLM 55 1 0 06 Mar 2025
NEAT: Nonlinear Parameter-efficient Adaptation of Pre-trained Models Yibo Zhong Haoxiang Jiang Lincan Li Ryumei Nakada Tianci Liu Linjun Zhang Huaxiu Yao Haoyu Wang 70 2 0 24 Feb 2025
LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits Zikai Zhou Qizheng Zhang Hermann Kumbong Kunle Olukotun MQ 113 0 0 12 Feb 2025
RandLoRA: Full-rank parameter-efficient fine-tuning of large models Paul Albert Frederic Z. Zhang Hemanth Saratchandran Cristian Rodriguez-Opazo Anton van den Hengel Ehsan Abbasnejad 87 0 0 03 Feb 2025
Parameter-Efficient Fine-Tuning for Foundation Models Dan Zhang Tao Feng Lilong Xue Yuandong Wang Yuxiao Dong J. Tang 29 6 0 23 Jan 2025
A Domain Adaptation Framework for Speech Recognition Systems with Only Synthetic data Minh Tran Yutong Pang Debjyoti Paul Laxmi Pandey Kevin Jiang Jinxi Guo Ke Li Shun Zhang X. Zhang Xin Lei AI4CE 26 0 0 21 Jan 2025
tCURLoRA: Tensor CUR Decomposition Based Low-Rank Parameter Adaptation and Its Application in Medical Image Segmentation Guanghua He Wangang Cheng Hancan Zhu Xiaohao Cai Gaohang Yu 36 0 0 10 Jan 2025
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement Tianyu Zheng Ge Zhang Tianhao Shen Xueling Liu Bill Yuchen Lin Jie Fu Wenhu Chen Xiang Yue SyDa 67 102 0 08 Jan 2025
Spectral-Aware Low-Rank Adaptation for Speaker Verification Zhe Li Man-Wai Mak Mert Pilanci Hung-yi Lee H. Meng 36 0 0 07 Jan 2025
GaLore $+$ : Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection Xutao Liao Shaohui Li Yuhui Xu Zhi Li Y. Liu You He VLM 45 2 0 31 Dec 2024
DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models Xiaolin Hu Xiang Cheng Peiyu Liu Wei Liu Jian Luan Bin Wang Y. Liu 32 1 0 31 Dec 2024
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models Fan Wang Juyong Jiang Chansung Park Sunghun Kim Jing Tang 79 0 0 08 Dec 2024
Adaptive Rank, Reduced Forgetting: Knowledge Retention in Continual Learning Vision-Language Models with Dynamic Rank-Selective LoRA Haodong Lu Chongyang Zhao Jason Xue Lina Yao Kristen Moore Dong Gong VLM KELM CLL 70 3 0 01 Dec 2024
Reassessing Layer Pruning in LLMs: New Insights and Methods Yao Lu Hao Cheng Yujie Fang Zeyu Wang Jiaheng Wei Dongwei Xu Qi Xuan Xiaoniu Yang Zhaowei Zhu 61 0 0 23 Nov 2024
MC-LLaVA: Multi-Concept Personalized Vision-Language Model Ruichuan An Sihan Yang Ming Lu Kai Zeng Yulin Luo ... Hao Liang Qi She Shanghang Zhang W. Zhang Wentao Zhang 64 5 0 18 Nov 2024
LoRA vs Full Fine-tuning: An Illusion of Equivalence Reece Shuttleworth Jacob Andreas Antonio Torralba Pratyusha Sharma 14 8 0 28 Oct 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 48 5 0 28 Oct 2024
An Efficient Watermarking Method for Latent Diffusion Models via Low-Rank Adaptation Dongdong Lin Yue Li B. Tondi Bin Li Mauro Barni WIGM 18 0 0 26 Oct 2024
On the Crucial Role of Initialization for Matrix Factorization Bingcong Li Liang Zhang Aryan Mokhtari Niao He 13 1 0 24 Oct 2024
Understanding Layer Significance in LLM Alignment Guangyuan Shi Zexin Lu Xiaoyu Dong Wenlong Zhang Xuanyu Zhang Yujie Feng Xiao-Ming Wu 28 2 0 23 Oct 2024
LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning Yiming Shi Jiwei Wei Yujia Wu Ran Ran Chengwei Sun Shiyuan He Yang Yang ALM 25 1 0 17 Oct 2024
QERA: an Analytical Framework for Quantization Error Reconstruction Cheng Zhang Jeffrey T. H. Wong Can Xiao G. Constantinides Yiren Zhao MQ 25 0 0 08 Oct 2024
Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models Theo Putterman Derek Lim Yoav Gelberg Stefanie Jegelka Haggai Maron AI4CE 32 5 0 05 Oct 2024
FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models Yucheng Xie Fu Feng Ruixiao Shi Jing Wang Xin Geng AI4CE 19 1 0 28 Sep 2024
Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape Tao Li Zhengbao He Yujun Li Yasheng Wang Lifeng Shang X. Huang 37 0 0 22 Sep 2024
Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models Yuncheng Yang Yulei Qin Tong Wu Zihan Xu Gang Li ... Yuchen Shi Ke Li Xing Sun Jie Yang Yun Gu ALM OffRL MoE 38 0 0 28 Aug 2024
SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models Yang Cao 19 2 0 21 Aug 2024
NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models Cheng Lin Lujun Li Dezhi Li Jie Zou Wei Xue Yike Guo AI4TS 28 4 0 18 Aug 2024
LoRA-Pro: Are Low-Rank Adapters Properly Optimized? Zhengbo Wang Jian Liang Ran He Zilei Wang Tieniu Tan 36 15 0 25 Jul 2024
LoRA-PT: Low-Rank Adapting UNETR for Hippocampus Segmentation Using Principal Tensor Singular Values and Vectors Guanghua He Wangang Cheng Hancan Zhu Gaohang Yu 12 1 0 16 Jul 2024
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients Ajay Jaiswal Lu Yin Zhenyu (Allen) Zhang Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang 23 14 0 15 Jul 2024
SLIP: Securing LLMs IP Using Weights Decomposition Yehonathan Refael Adam Hakim Lev Greenberg T. Aviv S. Lokam Ben Fishman Shachar Seidman 28 3 0 15 Jul 2024
Low-Rank Interconnected Adaptation Across Layers Yibo Zhong Yao Zhou OffRL MoE 28 1 0 13 Jul 2024
A Survey on LoRA of Large Language Models Yuren Mao Yuhang Ge Yijiang Fan Wenyi Xu Yu Mi Zhonghao Hu Yunjun Gao ALM 50 22 0 08 Jul 2024
A Fingerprint for Large Language Models Zhiguang Yang Hanzhou Wu 28 9 0 01 Jul 2024
Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead Rickard Brüel-Gabrielsson Jiacheng Zhu Onkar Bhardwaj Leshem Choshen Kristjan Greenewald Mikhail Yurochkin Justin Solomon 23 5 0 17 Jun 2024
The Impact of Initialization on LoRA Finetuning Dynamics Soufiane Hayou Nikhil Ghosh Bin Yu AI4CE 16 9 0 12 Jun 2024
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning Yibo Yang Xiaojie Li Zhongzhu Zhou S. Song Jianlong Wu Liqiang Nie Bernard Ghanem 36 6 0 07 Jun 2024
SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors Vijay Lingam Atula Tejaswi Aditya Vavre Aneesh Shetty Gautham Krishna Gudur Joydeep Ghosh Alexandros G. Dimakis Eunsol Choi Aleksandar Bojchevski Sujay Sanghavi 26 8 0 30 May 2024