MLorc: Momentum Low-rank Compression for Large Language Model Adaptation

With increasing size of large language models (LLMs), full-parameter fine-tuning imposes substantial memory demands. To alleviate this, we propose a novel memory-efficient training paradigm called Momentum Low-rank compression (MLorc). By directly compressing and reconstructing momentum rather than gradients, MLorc avoids imposing a fixed-rank constraint on weight update matrices and better preserves the training dynamics of full-parameter fine-tuning, in contrast to existing low-rank approaches such as LoRA and GaLore. Empirically, MLorc consistently outperforms other memory-efficient training methods, matches or even exceeds the performance of full fine-tuning with a small rank (e.g., $r=4$), and generalizes well across different optimizers -- all while not compromising time or memory efficiency. Furthermore, we provide a theoretical guarantee for its convergence under reasonable assumptions.

@article{shen2025_2506.01897,
  title={ MLorc: Momentum Low-rank Compression for Large Language Model Adaptation },
  author={ Wei Shen and Zhang Yaxiang and Minhui Huang and Mengfan Xu and Jiawei Zhang and Cong Shen },
  journal={arXiv preprint arXiv:2506.01897},
  year={ 2025 }
}