QLoRA: Efficient Finetuning of Quantized LLMs

23 May 2023

Tim Dettmers

Luke Zettlemoyer

Papers citing "QLoRA: Efficient Finetuning of Quantized LLMs"

50 / 298 papers shown

Title
PiCO: Peer Review in LLMs based on the Consistency Optimization Kun-Peng Ning Shuo Yang Yu-Yang Liu Jia-Yu Yao Zhen-Hui Liu Yu Wang Ming Pang Li Yuan ALM 69 8 0 24 Feb 2025
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning Raghav Singhal Kaustubh Ponkshe Rohit Vartak Lav R. Varshney Praneeth Vepakomma FedML 74 0 0 24 Feb 2025
Testing the limits of fine-tuning to improve reasoning in vision language models Luca M. Schulze Buschoff Konstantinos Voudouris Elif Akata Matthias Bethge Joshua B. Tenenbaum Eric Schulz LRM VLM Presented at ResearchTrend Connect \| VLM on 14 Mar 2025 122 0 1 24 Feb 2025
Automatic Input Rewriting Improves Translation with Large Language Models Dayeon Ki Marine Carpuat 38 0 0 23 Feb 2025
Beyond Release: Access Considerations for Generative AI Systems Irene Solaiman Rishi Bommasani Dan Hendrycks Ariel Herbert-Voss Yacine Jernite Aviya Skowron Andrew Trask 60 1 0 23 Feb 2025
Fine-Tuning Qwen 2.5 3B for Realistic Movie Dialogue Generation Kartik Gupta VGen 46 0 0 22 Feb 2025
How Much Do LLMs Hallucinate across Languages? On Multilingual Estimation of LLM Hallucination in the Wild Saad Obaid ul Islam Anne Lauscher Goran Glavas HILM LRM 115 1 0 21 Feb 2025
Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model Emre Can Acikgoz Jeremiah Greer Akul Datta Ze Yang William Zeng Oussama Elachqar Emmanouil Koukoumidis Dilek Hakkani-Tür Gökhan Tür LLMAG 97 3 0 20 Feb 2025
An Efficient Row-Based Sparse Fine-Tuning Cen-Jhih Li Aditya Bhaskara 49 0 0 17 Feb 2025
Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation Vera Neplenbroek Arianna Bisazza Raquel Fernández 97 0 0 17 Feb 2025
SuperMerge: An Approach For Gradient-Based Model Merging Haoyu Yang Zheng Zhang Saket Sathe MoMe 125 0 0 17 Feb 2025
LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits Zikai Zhou Qizheng Zhang Hermann Kumbong Kunle Olukotun MQ 147 0 0 12 Feb 2025
SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation Yixian Shen Qi Bi Jia-Hong Huang Hongyi Zhu Andy D. Pimentel Anuj Pathania 46 0 0 08 Feb 2025
DeepThink: Aligning Language Models with Domain-Specific User Intents Yang Li Mingxuan Luo Yeyun Gong Chen Lin Jian Jiao Yi Liu Kaili Huang LRM ALM ELM 52 0 0 08 Feb 2025
LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning Zhekai Du Yinjie Min Jingjing Li Ke Lu Changliang Zou Liuhua Peng Tingjin Chu M. Gong 113 1 0 05 Feb 2025
Learning to Generate Unit Tests for Automated Debugging Archiki Prasad Elias Stengel-Eskin Justin Chih-Yao Chen Zaid Khan Mohit Bansal ELM 76 1 0 03 Feb 2025
Position: AI Scaling: From Up to Down and Out Yunke Wang Yanxi Li Chang Xu HAI 74 1 0 02 Feb 2025
BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models Yibin Wang H. Shi Ligong Han Dimitris N. Metaxas Hao Wang BDL UQLM 102 6 0 28 Jan 2025
Learning to Summarize from LLM-generated Feedback Hwanjun Song Taewon Yun Yuho Lee Jihwan Oh Gihun Lee Jason (Jinglun) Cai Hang Su 73 2 0 28 Jan 2025
Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models Jingwei Yi Yueqi Xie Bin Zhu Emre Kiciman Guangzhong Sun Xing Xie Fangzhao Wu AAML 51 64 0 28 Jan 2025
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs Nicolas Boizard Kevin El Haddad C´eline Hudelot Pierre Colombo 68 14 0 28 Jan 2025
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs Mohammad Mozaffari Amir Yazdanbakhsh Zhao Zhang M. Dehnavi 65 5 0 28 Jan 2025
Decentralized Low-Rank Fine-Tuning of Large Language Models Sajjad Ghiasvand Mahnoosh Alizadeh Ramtin Pedarsani ALM 64 0 0 26 Jan 2025
The Karp Dataset Mason DiCicco Eamon Worden Conner Olsen Nikhil Gangaram Daniel Reichman Neil T. Heffernan ReLM LRM 56 0 0 24 Jan 2025
Enhancing Biomedical Relation Extraction with Directionality Po-Ting Lai Chih-Hsuan Wei Shubo Tian Robert Leaman Zhiyong Lu 32 0 0 23 Jan 2025
Fast Matrix Multiplications for Lookup Table-Quantized LLMs Han Guo William Brandon Radostin Cholakov Jonathan Ragan-Kelley Eric P. Xing Yoon Kim MQ 79 12 0 20 Jan 2025
Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models Tom Wallace Naser Ezzati-Jivan Beatrice Ombuki-Berman MQ 33 1 0 16 Jan 2025
Merging Feed-Forward Sublayers for Compressed Transformers Neha Verma Kenton W. Murray Kevin Duh AI4CE 45 0 0 10 Jan 2025
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 83 0 0 08 Jan 2025
Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning Zhen Li Yupeng Su Runming Yang C. Xie Z. Wang Zhongwei Xie Ngai Wong Hongxia Yang MQ LRM 30 3 0 06 Jan 2025
Scaling Laws for Floating Point Quantization Training X. Sun Shuaipeng Li Ruobing Xie Weidong Han Kan Wu ... Yangyu Tao Zhanhui Kang C. Xu Di Wang Jie Jiang MQ AIFin 53 0 0 05 Jan 2025
PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations Ruosen Li Teerth Patel Xinya Du LLMAG ALM 52 94 0 03 Jan 2025
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct Haipeng Luo Qingfeng Sun Can Xu Pu Zhao Jian-Guang Lou ... Xiubo Geng Qingwei Lin Shifeng Chen Yansong Tang Dongmei Zhang OSLM LRM 103 402 0 03 Jan 2025
Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets Vatsal Gupta Pranshu Pandya Tushar Kataria Vivek Gupta Dan Roth AAML 53 1 0 03 Jan 2025
GaLore $+$ : Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection Xutao Liao Shaohui Li Yuhui Xu Zhi Li Y. Liu You He VLM 54 2 0 31 Dec 2024
GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inference Chao Zeng Songwei Liu Shu Yang Fangmin Chen Xing Mei Lean Fu MQ 38 0 0 23 Dec 2024
Investigating Length Issues in Document-level Machine Translation Ziqian Peng Rachel Bawden François Yvon 69 1 0 23 Dec 2024
Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference Siyuan Wang Dianyi Wang Chengxing Zhou Zejun Li Zhihao Fan Xuanjing Huang Zhongyu Wei VLM 108 0 0 17 Dec 2024
Empowering LLMs to Understand and Generate Complex Vector Graphics Ximing Xing Juncheng Hu Guotao Liang Jing Zhang Dong Xu Qian Yu 92 7 0 15 Dec 2024
Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization Dongwei Wang Huanrui Yang MQ 82 1 0 08 Dec 2024
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models Fan Wang Juyong Jiang Chansung Park Sunghun Kim Jing Tang 91 1 0 08 Dec 2024
Reinforcement Learning Enhanced LLMs: A Survey Shuhe Wang Shengyu Zhang J. Zhang Runyi Hu Xiaoya Li Tianwei Zhang Jiwei Li Fei Wu G. Wang Eduard H. Hovy OffRL 128 7 0 05 Dec 2024
Unified Parameter-Efficient Unlearning for LLMs Chenlu Ding Jiancan Wu Yancheng Yuan Jinda Lu Kai Zhang Alex Su Xiang Wang Xiangnan He MU KELM 97 6 0 30 Nov 2024
On the Impact of Fine-Tuning on Chain-of-Thought Reasoning Elita Lobo Chirag Agarwal Himabindu Lakkaraju LRM 70 5 0 22 Nov 2024
Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation Seokil Ham H. Kim Sangmin Woo Changick Kim Mamba 115 0 0 21 Nov 2024
Quantized symbolic time series approximation Erin Carson Xinye Chen Cheng Kang AI4TS 74 0 0 20 Nov 2024
When Backdoors Speak: Understanding LLM Backdoor Attacks Through Model-Generated Explanations Huaizhi Ge Yiming Li Qifan Wang Yongfeng Zhang Ruixiang Tang AAML SILM 72 0 0 19 Nov 2024
BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration Yuzong Chen Ahmed F. AbouElhamayed Xilai Dai Yang Wang Marta Andronic G. Constantinides Mohamed S. Abdelfattah MQ 98 0 0 18 Nov 2024
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset Khaoula Chehbouni Jonathan Colaço-Carr Yash More Jackie CK Cheung G. Farnadi 73 0 0 12 Nov 2024
Neural Precision Polarization: Simplifying Neural Network Inference with Dual-Level Precision Dinithi Jayasuriya Nastaran Darabi Maeesha Binte Hashem A. R. Trivedi MQ 21 1 0 06 Nov 2024