v1v2 (latest)

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

6 March 2024

Yuandong Tian

ArXiv (abs)PDF HTML HuggingFace (189 upvotes)

Papers citing "GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection"

50 / 219 papers shown

Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guarantees

625

11 Jul 2025

LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization

Xujia Wang

Yunjia Qi

Bin Xu

249

06 Jul 2025

Pay Attention to Small Weights

220

26 Jun 2025

A Minimalist Optimizer Design for LLM Pretraining

266

20 Jun 2025

A geometric framework for momentum-based optimizers for low-rank training

229

20 Jun 2025

Subspace-Boosted Model Merging

Ronald Skorobogat

Karsten Roth

Mariana-Iuliana Georgescu

MoMe

400

19 Jun 2025

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity

Konstantinos N. Plataniotis

217

19 Jun 2025

Memory-Efficient Differentially Private Training with Gradient Random Projection

243

18 Jun 2025

Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality

258

17 Jun 2025

AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs

386

17 Jun 2025

Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence

262

16 Jun 2025

Prefix-Tuning+: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention

211

16 Jun 2025

EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction

277

13 Jun 2025

Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner

283

04 Jun 2025

PoLAR: Polar-Decomposed Low-Rank Adapter Representation

256

03 Jun 2025

QKV Projections Require a Fraction of Their Memory

324

03 Jun 2025

MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation

326

02 Jun 2025

Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism

Sameera Ramasinghe

Thalaiyasingam Ajanthan

377

02 Jun 2025

Taming LLMs by Scaling Learning Rates with Gradient Grouping

230

01 Jun 2025

Structured Gradient Guidance for Few-Shot Adaptation in Large Language Models

142

31 May 2025

LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMsJournal of Information Security and Applications (JISA), 2025

Luis Ibanez-Lissen

Lorena Gonzalez-Manzano

José Maria De Fuentes

Nicolas Anciaux

140

30 May 2025

GradPower: Powering Gradients for Faster Language Model Pre-Training

231

30 May 2025

SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training

180

30 May 2025

On the Convergence Analysis of Muon

303

29 May 2025

Two Is Better Than One: Rotations Scale LoRAs

...

243

29 May 2025

Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution

Decebal Constantin Mocanu

279

29 May 2025

MuLoCo: Muon is a practical inner optimizer for DiLoCo

170

29 May 2025

Highly Efficient and Effective LLMs with Multi-Boolean Architectures

Ba-Hien Tran

Van Minh Nguyen

373

28 May 2025

POQD: Performance-Oriented Query Decomposer for Multi-vector retrieval

329

25 May 2025

Efficient Data Selection at Scale via Influence Distillation

329

25 May 2025

KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning

356

24 May 2025

PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Training

Matan Haroush

Daniel Soudry

362

23 May 2025

FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models

321

23 May 2025

NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling

...

Decebal Constantin Mocanu

307

23 May 2025

AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training

482

22 May 2025

Understanding Differential Transformer Unchains Pretrained Self-Attentions

Chaerin Kong

Jiho Jang

Nojun Kwak

459

22 May 2025

Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models

Sajjad Ghiasvand

Haniyeh Ehsani Oskouie

Mahnoosh Alizadeh

Ramtin Pedarsani

AAML VLM

407

21 May 2025

Fine-tuning Quantized Neural Networks with Zeroth-order Optimization

353

19 May 2025

ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updates

567

18 May 2025

AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections

326

18 May 2025

Continuous Subspace Optimization for Continual Learning

488

17 May 2025

Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients

236

03 May 2025

Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics

335

01 May 2025

Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptation

417

16 Apr 2025

STEP: Staged Parameter-Efficient Pre-training for Large Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL), 2025

295

05 Apr 2025

Task-Aware Parameter-Efficient Fine-Tuning of Large Pre-Trained Models at the Edge

261

29 Mar 2025

ASGO: Adaptive Structured Gradient Optimization

430

26 Mar 2025

An Overview of Low-Rank Structures in the Training and Adaptation of Large Models

239

25 Mar 2025

Striving for Simplicity: Simple Yet Effective Prior-Aware Pseudo-Labeling for Semi-Supervised Ultrasound Image SegmentationInternational Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2025

385

18 Mar 2025

ClusComp: A Simple Paradigm for Model Compression and Efficient FinetuningAnnual Meeting of the Association for Computational Linguistics (ACL), 2025

374

17 Mar 2025