Full Parameter Fine-tuning for Large Language Models with Limited Resources

16 June 2023

Qipeng Guo

Xipeng Qiu

ArXiv PDF HTML

Papers citing "Full Parameter Fine-tuning for Large Language Models with Limited Resources"

50 / 78 papers shown

Title
Graph-Based Spectral Decomposition for Parameter Coordination in Language Model Fine-Tuning Hanlu Zhang Yumeng Ma Shuo Wang Guiran Liu B. Zhu 50 2 0 28 Apr 2025
Mixture of Routers Jia-Chen Zhang Yu-Jie Xiong Xi-He Qiu Chun-Ming Xia Fei Dai MoE 59 0 0 30 Mar 2025
Tuning LLMs by RAG Principles: Towards LLM-native Memory Jiale Wei Shuchi Wu Ruochen Liu Xiang Ying Jingbo Shang Fangbo Tao RALM 60 0 0 20 Mar 2025
Keeping Representation Similarity in Finetuning for Medical Image Analysis Wenqiang Zu Shenghao Xie Hao Chen Yiming Liang Lei Ma MedIm OOD 41 0 0 10 Mar 2025
LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAM Yehonathan Refael Iftach Arbel Ofir Lindenbaum Tom Tirer 64 0 0 26 Feb 2025
Compressing Language Models for Specialized Domains Miles Williams G. Chrysostomou Vitor Jeronymo Nikolaos Aletras MQ 39 0 0 25 Feb 2025
SubTrack your Grad: Gradient Subspace Tracking for Memory and Time Efficient Full-Parameter LLM Training Sahar Rajabi Nayeema Nonta Sirisha Rambhatla 85 0 0 03 Feb 2025
How to Alleviate Catastrophic Forgetting in LLMs Finetuning? Hierarchical Layer-Wise and Element-Wise Regularization Shezheng Song Hao Xu Jun Ma Shasha Li Long Peng Qian Wan Xiaodong Liu Jie Yu KELM CLL ALM 65 0 0 23 Jan 2025
AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning Yehonathan Refael Jonathan Svirsky Boris Shustin Wasim Huleihel Ofir Lindenbaum 34 3 0 31 Dec 2024
Grams: Gradient Descent with Adaptive Momentum Scaling Yang Cao Xiaoyu Li Zhao-quan Song ODL 83 2 0 22 Dec 2024
In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning Yifei Duan Liu Li Zirui Zhai Jinxia Yao 74 0 0 17 Dec 2024
Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning Kaustubh Ponkshe Raghav Singhal Eduard A. Gorbunov Alexey Tumanov Samuel Horváth Praneeth Vepakomma 66 1 0 29 Nov 2024
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection Jinqi Xiao S. Sang Tiancheng Zhi Jing Liu Qing Yan Linjie Luo Bo Yuan Bo Yuan VLM 81 1 0 26 Nov 2024
Cautious Optimizers: Improving Training with One Line of Code Kaizhao Liang Lizhang Chen B. Liu Qiang Liu ODL 98 5 0 25 Nov 2024
FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training Philip Zmushko Aleksandr Beznosikov Martin Takáč Samuel Horváth 37 0 0 12 Nov 2024
Beyond The Rainbow: High Performance Deep Reinforcement Learning On A Desktop PC Tyler Clark Mark Towers Christine Evers Jonathon Hare OffRL 33 0 0 06 Nov 2024
Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction Muhammad Tayyab Khan Lequn Chen Ye Han Ng Wenhe Feng Nicholas Yew Jin Tan Seung Ki Moon 24 2 0 06 Nov 2024
KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation Rambod Azimi Rishav Rishav M. Teichmann Samira Ebrahimi Kahou ALM 26 0 0 28 Oct 2024
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks Yongchang Hao Yanshuai Cao Lili Mou MQ 28 2 0 28 Oct 2024
LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics Thomas Robert M. Safaryan Ionut-Vlad Modoranu Dan Alistarh ODL 31 2 0 21 Oct 2024
HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adapter Yumiao Zhao Bo Jiang Xiao Wang Qin Xu Jin Tang VLM 28 0 0 10 Oct 2024
Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Models Zeman Li Xinwei Zhang Peilin Zhong Yuan Deng Meisam Razaviyayn Vahab Mirrokni 15 2 0 09 Oct 2024
SOAP: Improving and Stabilizing Shampoo using Adam Nikhil Vyas Depen Morwani Rosie Zhao Itai Shapira David Brandfonbrener Lucas Janson Sham Kakade Sham Kakade 59 23 0 17 Sep 2024
Achieving Peak Performance for Large Language Models: A Systematic Review Z. R. K. Rostam Sándor Szénási Gábor Kertész 32 3 0 07 Sep 2024
Memory-Efficient LLM Training with Online Subspace Descent Kaizhao Liang Bo Liu Lizhang Chen Qiang Liu 19 7 0 23 Aug 2024
NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models Cheng Lin Lujun Li Dezhi Li Jie Zou Wei Xue Yike Guo AI4TS 28 4 0 18 Aug 2024
Performance of Recent Large Language Models for a Low-Resourced Language Ravindu Jayakody Gihan Dias 16 5 0 31 Jul 2024
An Application of Large Language Models to Coding Negotiation Transcripts Ray Friedman Jaewoo Cho Jeanne Brett Xuhui Zhan Ningyu Han ... Ece Tuncel Laurie R. Weingart Robert Wilken JingJing Yao Zhi-Xue Zhang 30 0 0 18 Jul 2024
Weak-to-Strong Reasoning Yuqing Yang Yan Ma Pengfei Liu LRM 30 13 0 18 Jul 2024
Weighted Grouped Query Attention in Transformers Sai Sena Chinnakonduru Astarag Mohapatra 42 5 0 15 Jul 2024
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients Zhenyu (Allen) Zhang Ajay Jaiswal L. Yin Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang VLM 31 16 0 11 Jul 2024
Large Language Models for Wearable Sensor-Based Human Activity Recognition, Health Monitoring, and Behavioral Modeling: A Survey of Early Trends, Datasets, and Challenges Emilio Ferrara LM&MA 32 13 0 09 Jul 2024
LoRA-GA: Low-Rank Adaptation with Gradient Approximation Shaowen Wang Linxi Yu Jian Li ALM AI4CE 26 27 0 06 Jul 2024
PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs Dan Peng Zhihui Fu Jun Wang 24 12 0 01 Jul 2024
Evaluating the Efficacy of Foundational Models: Advancing Benchmarking Practices to Enhance Fine-Tuning Decision-Making O. Amujo S. Yang 22 1 0 25 Jun 2024
Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients Aashiq Muhamed Oscar Li David Woodruff Mona Diab Virginia Smith 45 7 0 25 Jun 2024
Adam-mini: Use Fewer Learning Rates To Gain More Yushun Zhang Congliang Chen Ziniu Li Tian Ding Chenwei Wu Yinyu Ye Zhi-Quan Luo Ruoyu Sun 34 33 0 24 Jun 2024
SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining Andi Han Jiaxiang Li Wei Huang Mingyi Hong Akiko Takeda Pratik Jawanpuria Bamdev Mishra 36 9 0 04 Jun 2024
VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections Roy Miles Pradyumna Reddy Ismail Elezi Jiankang Deng VLM 27 3 0 28 May 2024
LoQT: Low Rank Adapters for Quantized Training Sebastian Loeschcke M. Toftrup M. Kastoryano Serge J. Belongie Vésteinn Snæbjarnarson MQ 34 3 0 26 May 2024
Efficient Multimodal Large Language Models: A Survey Yizhang Jin Jian Li Yexin Liu Tianjun Gu Kai Wu ... Xin Tan Zhenye Gan Yabiao Wang Chengjie Wang Lizhuang Ma LRM 39 45 0 17 May 2024
A survey on fairness of large language models in e-commerce: progress, application, and challenge Qingyang Ren Zilin Jiang Jinghan Cao Sijia Li Chiqu Li Yiyang Liu Shuning Huo Tiange He Yuan Chen AILaw FaML 35 6 0 15 May 2024
A safety realignment framework via subspace-oriented model fusion for large language models Xin Yi Shunfan Zheng Linlin Wang Xiaoling Wang Liang He 43 20 0 15 May 2024
Pruning as a Domain-specific LLM Extractor Nan Zhang Yanchi Liu Xujiang Zhao Wei Cheng Runxue Bao Rui Zhang Prasenjit Mitra Haifeng Chen 14 9 0 10 May 2024
Large Language Models for Cyber Security: A Systematic Literature Review HanXiang Xu Shenao Wang Ningke Li K. Wang Yanjie Zhao Kai Chen Ting Yu Yang Janet Liu H. Wang 29 23 0 08 May 2024
Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning Jing Xu Jingzhao Zhang 34 6 0 04 May 2024
Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications Charith Chandra Sai Balne S. Bhaduri Tamoghna Roy Vinija Jain Aman Chadha 27 12 0 21 Apr 2024
BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models Qi Luo Hengxu Yu Xiao Li 34 1 0 03 Apr 2024
Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models Zehui Chen Kuikun Liu Qiuchen Wang Wenwei Zhang Jiangning Liu Dahua Lin Kai-xiang Chen Feng Zhao LLMAG ALM AIFin 59 27 0 19 Mar 2024
A Moral Imperative: The Need for Continual Superalignment of Large Language Models Gokul Puthumanaillam Manav Vora Pranay Thangeda Melkior Ornik 29 7 0 13 Mar 2024