LISA: Layerwise Importance Sampling for Memory-Efficient Large Language
Model Fine-Tuning

LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning

26 March 2024

Tong Zhang

Papers citing "LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning"

10 / 10 papers shown

Title
COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs Liming Liu Zhenghao Xu Zixuan Zhang Hao Kang Zichong Li Chen Liang Weizhu Chen T. Zhao 42 1 0 24 Feb 2025
Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference Siyuan Wang Dianyi Wang Chengxing Zhou Zejun Li Zhihao Fan Xuanjing Huang Zhongyu Wei VLM 74 0 0 17 Dec 2024
Understanding Layer Significance in LLM Alignment Guangyuan Shi Zexin Lu Xiaoyu Dong Wenlong Zhang Xuanyu Zhang Yujie Feng Xiao-Ming Wu 30 2 0 23 Oct 2024
Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation Guozhi Liu Weiwei Lin Tiansheng Huang Ruichao Mo Qi Mu Li Shen AAML 49 9 0 13 Oct 2024
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning Hanqing Wang Zeguan Xiao Shuo Wang Guanhua Chen Guanhua Chen 22 19 0 13 Jun 2024
SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezing Sheng R. Li Geng Yuan Yuezhen Dai Youtao Zhang Yanzhi Wang Xulong Tang 23 16 0 30 Jan 2024
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 155 399 0 18 Jan 2021
WARP: Word-level Adversarial ReProgramming Karen Hambardzumyan Hrant Khachatrian Jonathan May AAML 243 340 0 01 Jan 2021
PubMedQA: A Dataset for Biomedical Research Question Answering Qiao Jin Bhuwan Dhingra Zhengping Liu William W. Cohen Xinghua Lu 196 791 0 13 Sep 2019