Full Parameter Fine-tuning for Large Language Models with Limited Resources

16 June 2023

Qipeng Guo

Xipeng Qiu

ArXiv PDF HTML

Papers citing "Full Parameter Fine-tuning for Large Language Models with Limited Resources"

28 / 78 papers shown

Title
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Jiawei Zhao Zhenyu (Allen) Zhang Beidi Chen Zhangyang Wang A. Anandkumar Yuandong Tian 41 173 0 06 Mar 2024
A Survey of Large Language Models in Cybersecurity Gabriel de Jesus Coelho da Silva Carlos Becker Westphall 27 5 0 26 Feb 2024
Why Transformers Need Adam: A Hessian Perspective Yushun Zhang Congliang Chen Tian Ding Ziniu Li Ruoyu Sun Zhimin Luo 24 40 0 26 Feb 2024
Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks Minju Seo Jinheon Baek James Thorne Sung Ju Hwang RALM 29 9 0 21 Feb 2024
Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuning Shuai Zhao Leilei Gan Anh Tuan Luu Jie Fu Lingjuan Lyu Meihuizi Jia Jinming Wen AAML 21 22 0 19 Feb 2024
Higher Layers Need More LoRA Experts Chongyang Gao Kezhen Chen Jinmeng Rao Baochen Sun Ruibo Liu Daiyi Peng Yawen Zhang Xiaoyuan Guo Jie Yang V. Subrahmanian MoE 13 37 0 13 Feb 2024
InCoRo: In-Context Learning for Robotics Control with Feedback Loops Jiaqiang Ye Zhu Carla Gomez Cano David Vazquez Bermudez Michal Drozdzal LRM 22 7 0 07 Feb 2024
Flora: Low-Rank Adapters Are Secretly Gradient Compressors Yongchang Hao Yanshuai Cao Lili Mou 6 39 0 05 Feb 2024
HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy Yongkang Liu Yiqun Zhang Qian Li Tong Liu Shi Feng Daling Wang Yifei Zhang Hinrich Schütze 30 6 0 26 Jan 2024
IoT in the Era of Generative AI: Vision and Challenges Xin Wang Zhongwei Wan Arvin Hekmati M. Zong Samiul Alam Mi Zhang Bhaskar Krishnamachari 27 15 0 03 Jan 2024
RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair André Silva Sen Fang Martin Monperrus MoMe KELM 62 37 0 25 Dec 2023
ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimization Shuoran Jiang Qingcai Chen Youcheng Pan Yang Xiang Yukang Lin Xiangping Wu Chuanyi Liu Xiaobao Song ODL 28 9 0 23 Dec 2023
Sparse is Enough in Fine-tuning Pre-trained Large Language Models Weixi Song Z. Li Lefei Zhang Hai Zhao Bo Du VLM 16 6 0 19 Dec 2023
ASPEN: High-Throughput LoRA Fine-Tuning of Large Language Models with a Single GPU Zhengmao Ye Dengchun Li Jingqi Tian Tingfeng Lan Jie Zuo ... Hui Lu Yexi Jiang Jian Sha Ke Zhang Mingjie Tang 91 7 0 05 Dec 2023
CoLLiE: Collaborative Training of Large Language Models in an Efficient Way Kai Lv Shuo Zhang Tianle Gu Shuhao Xing Jiawei Hong ... Tengxiao Liu Yu Sun Penousal Machado Hang Yan Xipeng Qiu 35 7 0 01 Dec 2023
Cerbero-7B: A Leap Forward in Language-Specific LLMs Through Enhanced Chat Corpus Generation and Evaluation Federico A. Galatolo M. G. Cimino 24 5 0 27 Nov 2023
ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models Ziniu Li Tian Xu Yushun Zhang Zhihang Lin Yang Yu Ruoyu Sun Zhimin Luo 19 47 0 16 Oct 2023
AdaLomo: Low-memory Optimization with Adaptive Learning Rate Kai Lv Hang Yan Qipeng Guo Haijun Lv Xipeng Qiu ODL 19 20 0 16 Oct 2023
QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources Zhikai Li Xiaoxuan Liu Banghua Zhu Zhen Dong Qingyi Gu Kurt Keutzer MQ 27 7 0 11 Oct 2023
FireAct: Toward Language Agent Fine-tuning Baian Chen Chang Shu Ehsan Shareghi Nigel Collier Karthik Narasimhan Shunyu Yao ALM LLMAG 99 97 0 09 Oct 2023
Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices Bojia Zi Xianbiao Qi Lingzhi Wang Jianan Wang Kam-Fai Wong Lei Zhang 18 40 0 05 Sep 2023
Instruction Tuning for Large Language Models: A Survey Shengyu Zhang Linfeng Dong Xiaoya Li Sen Zhang Xiaofei Sun ... Jiwei Li Runyi Hu Tianwei Zhang Fei Wu Guoyin Wang LM&MA 21 531 0 21 Aug 2023
LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Models Yixuan Weng Zhiqi Wang Huanxuan Liao Shizhu He Shengping Liu Kang Liu Jun Zhao 26 3 0 20 Aug 2023
A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrology Sean Wu Michael Koo L. Blum A. Black Liyo Kao Fabien Scalzo Ira Kurtz LM&MA ELM AI4MH 15 41 0 09 Aug 2023
Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuning Deepanway Ghosal Yew Ken Chia Navonil Majumder Soujanya Poria ALM LRM 30 17 0 05 Jul 2023
Unlocking the Potential of ChatGPT: A Comprehensive Exploration of its Applications, Advantages, Limitations, and Future Directions in Natural Language Processing Walid Hariri AI4MH LM&MA 25 84 0 27 Mar 2023
Diversity Over Size: On the Effect of Sample and Topic Sizes for Argument Mining Datasets Benjamin Schiller Johannes Daxenberger Iryna Gurevych 26 1 0 23 May 2022
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 160 413 0 18 Jan 2021