Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning

On-device learning has emerged as a promising direction for AI development, particularly because of its potential to reduce latency issues and mitigate privacy risks associated with device-server communication, while improving energy efficiency. Despite these advantages, significant memory and computational constraints still represent major challenges for its deployment. Drawing on previous studies on low-rank decomposition methods that address activation memory bottlenecks in backpropagation, we propose a novel shortcut approach as an alternative. Our analysis and experiments demonstrate that our method can reduce activation memory usage, even up to $120.09\times$ compared to vanilla training, while also reducing overall training FLOPs up to $1.86\times$ when evaluated on traditional benchmarks.

@article{nguyen2025_2505.05086,
  title={ Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning },
  author={ Le-Trung Nguyen and Ael Quelennec and Van-Tam Nguyen and Enzo Tartaglione },
  journal={arXiv preprint arXiv:2505.05086},
  year={ 2025 }
}