Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive
Low-Rank Gradients

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients

11 July 2024

Zhenyu (Allen) Zhang

Ajay Jaiswal

Jiawei Zhao

Zhangyang Wang

Papers citing "Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients"

10 / 10 papers shown

Title
Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam Tianjin Huang Haotian Hu Zhenyu (Allen) Zhang Gaojie Jin X. Li ... Tianlong Chen Lu Liu Qingsong Wen Zhangyang Wang Shiwei Liu MQ 33 0 0 24 Feb 2025
SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training Tianjin Huang Ziquan Zhu Gaojie Jin Lu Liu Zhangyang Wang Shiwei Liu 36 1 0 12 Jan 2025
AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning Yehonathan Refael Jonathan Svirsky Boris Shustin Wasim Huleihel Ofir Lindenbaum 34 3 0 31 Dec 2024
CompAct: Compressed Activations for Memory-Efficient LLM Training Yara Shamshoum Nitzan Hodos Yuval Sieradzki Assaf Schuster MQ VLM 36 0 0 20 Oct 2024
Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning Wenhan Xia Chengwei Qin Elad Hazan 46 52 0 08 Jan 2024
Exploring Low Rank Training of Deep Neural Networks Siddhartha Rao Kamalakara Acyr F. Locatelli Bharat Venkitesh Jimmy Ba Y. Gal Aidan N. Gomez 48 22 0 27 Sep 2022
FP8 Formats for Deep Learning Paulius Micikevicius Dusan Stosic N. Burgess Marius Cornea Pradeep Dubey ... Naveen Mellempudi S. Oberman M. Shoeybi Michael Siu Hao Wu BDL VLM MQ 67 121 0 12 Sep 2022
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT Sheng Shen Zhen Dong Jiayu Ye Linjian Ma Z. Yao A. Gholami Michael W. Mahoney Kurt Keutzer MQ 225 571 0 12 Sep 2019
Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers Yukuan Yang Shuang Wu Lei Deng Tianyi Yan Yuan Xie Guoqi Li MQ 99 108 0 05 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,927 0 20 Apr 2018