LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning

20 November 2023

Papers citing "LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning"

42 / 42 papers shown

Title
Efficient Fine-Tuning of Quantized Models via Adaptive Rank and Bitwidth Changhai Zhou Yuhua Zhou Qian Qiao Weizhong Zhang Cheng Jin MQ 25 0 0 02 May 2025
DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models Wenjin Ke Zhe Li D. Li Lu Tian E. Barsoum MQ 27 1 0 12 Apr 2025
LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation Juzheng Zhang Jiacheng You Ashwinee Panda Tom Goldstein MoMe 51 0 0 10 Apr 2025
1LoRA: Summation Compression for Very Low-Rank Adaptation Alessio Quercia Zhuo Cao Arya Bangun Richard D. Paul Abigail Morrison Ira Assent Hanno Scharr 50 0 0 11 Mar 2025
Towards Superior Quantization Accuracy: A Layer-sensitive Approach Feng Zhang Yanbin Liu Weihua Li Jie Lv Xiaodan Wang Q. Bai MQ 44 0 0 09 Mar 2025
CASP: Compression of Large Multimodal Models Based on Attention Sparsity Mohsen Gholami Mohammad Akbari Kevin Cannons Yong Zhang 63 0 0 07 Mar 2025
An Efficient Row-Based Sparse Fine-Tuning Cen-Jhih Li Aditya Bhaskara 44 0 0 17 Feb 2025
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs Mohammad Mozaffari Amir Yazdanbakhsh Zhao Zhang M. Dehnavi 65 5 0 28 Jan 2025
Fast Matrix Multiplications for Lookup Table-Quantized LLMs Han Guo William Brandon Radostin Cholakov Jonathan Ragan-Kelley Eric P. Xing Yoon Kim MQ 73 12 0 20 Jan 2025
ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization Weibo Zhao Yubin Shi Xinyu Lyu Wanchen Sui Shen Li Yong Li MQ 47 1 0 12 Nov 2024
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies L. Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 44 3 0 24 Oct 2024
QuAILoRA: Quantization-Aware Initialization for LoRA Neal Lawton Aishwarya Padmakumar Judith Gaspers Jack FitzGerald Anoop Kumar Greg Ver Steeg Aram Galstyan MQ 21 0 0 09 Oct 2024
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Cong Guo Feng Cheng Zhixu Du James Kiessling Jonathan Ku ... Qilin Zheng Guanglei Zhou Hai Li-Wei Li Yiran Chen 29 5 0 08 Oct 2024
QERA: an Analytical Framework for Quantization Error Reconstruction Cheng Zhang Jeffrey T. H. Wong Can Xiao G. Constantinides Yiren Zhao MQ 35 0 0 08 Oct 2024
Mixture Compressor for Mixture-of-Experts LLMs Gains More Wei Huang Yue Liao Jianhui Liu Ruifei He Haoru Tan Shiming Zhang Hongsheng Li Si Liu Xiaojuan Qi MoE 36 3 0 08 Oct 2024
Agentic Retrieval-Augmented Generation for Time Series Analysis Chidaksh Ravuru Sagar Srinivas Sakhinana Venkataramana Runkana AI4TS 18 5 0 18 Aug 2024
EfficientQAT: Efficient Quantization-Aware Training for Large Language Models Mengzhao Chen Wenqi Shao Peng Xu Jiahao Wang Peng Gao Kaipeng Zhang Yu Qiao Ping Luo MQ 36 21 0 10 Jul 2024
SBoRA: Low-Rank Adaptation with Regional Weight Updates L. Po Yuyang Liu Haoxuan Wu Tianqi Zhang W. Yu Zeyu Jiang Kun Li 33 0 0 07 Jul 2024
Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs Ashwinee Panda Berivan Isik Xiangyu Qi Sanmi Koyejo Tsachy Weissman Prateek Mittal MoMe 45 12 0 24 Jun 2024
Talking Heads: Understanding Inter-layer Communication in Transformer Language Models Jack Merullo Carsten Eickhoff Ellie Pavlick 51 12 0 13 Jun 2024
Low-Rank Quantization-Aware Training for LLMs Yelysei Bondarenko Riccardo Del Chiaro Markus Nagel MQ 33 8 0 10 Jun 2024
Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity Wentao Guo Jikai Long Yimeng Zeng Zirui Liu Xinyu Yang ... Osbert Bastani Christopher De Sa Xiaodong Yu Beidi Chen Zhaozhuo Xu 26 14 0 05 Jun 2024
One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments Ke Yi Yuhui Xu Heng Chang Chen Tang Yuan Meng Tong Zhang Jia Li MQ 30 2 0 30 May 2024
Compressing Large Language Models using Low Rank and Low Precision Decomposition R. Saha Naomi Sagan Varun Srivastava Andrea J. Goldsmith Mert Pilanci MQ 16 7 0 29 May 2024
PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression Vladimir Malinovskii Denis Mazur Ivan Ilin Denis Kuznedelev Konstantin Burlachenko Kai Yi Dan Alistarh Peter Richtárik MQ 24 18 0 23 May 2024
ReALLM: A general framework for LLM compression and fine-tuning Louis Leconte Lisa Bedin Van Minh Nguyen Eric Moulines MQ 31 0 0 21 May 2024
Collage: Light-Weight Low-Precision Strategy for LLM Training Tao Yu Gaurav Gupta Karthick Gopalswamy Amith R. Mamidala Hao Zhou Jeffrey Huynh Youngsuk Park Ron Diamant Anoop Deoras Jun Huan MQ 41 3 0 06 May 2024
Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models Wanyun Cui Qianle Wang MQ 31 1 0 03 Apr 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 136 301 0 21 Mar 2024
Asymmetry in Low-Rank Adapters of Foundation Models Jiacheng Zhu Kristjan Greenewald Kimia Nadjahi Haitz Sáez de Ocáriz Borde Rickard Brüel-Gabrielsson Leshem Choshen Marzyeh Ghassemi Mikhail Yurochkin Justin Solomon 34 26 0 26 Feb 2024
LLM Inference Unveiled: Survey and Roofline Model Insights Zhihang Yuan Yuzhang Shang Yang Zhou Zhen Dong Zhe Zhou ... Yong Jae Lee Yan Yan Beidi Chen Guangyu Sun Kurt Keutzer 37 77 0 26 Feb 2024
ApiQ: Finetuning of 2-Bit Quantized Large Language Model Baohao Liao Christian Herold Shahram Khadivi Christof Monz CLL MQ 34 12 0 07 Feb 2024
ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks Xiaoxia Wu Haojun Xia Stephen Youn Zhen Zheng Shiyang Chen ... Reza Yazdani Aminabadi Yuxiong He Olatunji Ruwase Leon Song Zhewei Yao 66 8 0 14 Dec 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 206 2,232 0 22 Mar 2023
ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation Z. Yao Xiaoxia Wu Cheng-rong Li Stephen Youn Yuxiong He MQ 63 56 0 15 Mar 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
Latency Adjustable Transformer Encoder for Language Understanding Sajjad Kachuee M. Sharifkhani 24 0 0 10 Jan 2022
Weighted Low Rank Matrix Approximation and Acceleration Elena Tuzhilina Trevor Hastie OffRL 11 5 0 22 Sep 2021
KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation Marzieh S. Tahaei Ella Charlaix V. Nia A. Ghodsi Mehdi Rezagholizadeh 41 22 0 13 Sep 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 278 3,784 0 18 Apr 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 160 399 0 18 Jan 2021
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,927 0 20 Apr 2018