RaSA: Rank-Sharing Low-Rank Adaptation

Low-rank adaptation (LoRA) has been prominently employed for parameter-efficient fine-tuning of large language models (LLMs). However, the limited expressive capacity of LoRA, stemming from the low-rank constraint, has been recognized as a bottleneck, particularly in rigorous tasks like code generation and mathematical reasoning. To address this limitation, we introduce Rank-Sharing Low-Rank Adaptation (RaSA), an innovative extension that enhances the expressive capacity of LoRA by leveraging partial rank sharing across layers. By forming a shared rank pool and applying layer-specific weighting, RaSA effectively increases the number of ranks without augmenting parameter overhead. Our theoretically grounded and empirically validated approach demonstrates that RaSA not only maintains the core advantages of LoRA but also significantly boosts performance in challenging code and math tasks. Code, data and scripts are available at:this https URL.

@article{he2025_2503.12576,
  title={ RaSA: Rank-Sharing Low-Rank Adaptation },
  author={ Zhiwei He and Zhaopeng Tu and Xing Wang and Xingyu Chen and Zhijie Wang and Jiahao Xu and Tian Liang and Wenxiang Jiao and Zhuosheng Zhang and Rui Wang },
  journal={arXiv preprint arXiv:2503.12576},
  year={ 2025 }
}