ResearchTrend.AI
  • Papers
  • Communities
  • Events
  • Blog
  • Pricing
Papers
Communities
Social Events
Terms and Conditions
Pricing
Parameter LabParameter LabTwitterGitHubLinkedInBlueskyYoutube

© 2025 ResearchTrend.AI, All rights reserved.

  1. Home
  2. Papers
  3. 2505.12805
37
0

FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA

19 May 2025
Seanie Lee
Sangwoo Park
Dong Bok Lee
Dominik Wagner
Haebin Seong
Tobias Bocklet
Juho Lee
Sung Ju Hwang
    FedML
ArXivPDFHTML
Abstract

Low-Rank Adaptation (LoRA), which introduces a product of two trainable low-rank matrices into frozen pre-trained weights, is widely used for efficient fine-tuning of language models in federated learning (FL). However, when combined with differentially private stochastic gradient descent (DP-SGD), LoRA faces substantial noise amplification: DP-SGD perturbs per-sample gradients, and the matrix multiplication of the LoRA update (BABABA) intensifies this effect. Freezing one matrix (e.g., AAA) reduces the noise but restricts model expressiveness, often resulting in suboptimal adaptation. To address this, we propose FedSVD, a simple yet effective method that introduces a global reparameterization based on singular value decomposition (SVD). In our approach, each client optimizes only the BBB matrix and transmits it to the server. The server aggregates the BBB matrices, computes the product BABABA using the previous AAA, and refactorizes the result via SVD. This yields a new adaptive AAA composed of the orthonormal right singular vectors of BABABA, and an updated BBB containing the remaining SVD components. This reparameterization avoids quadratic noise amplification, while allowing AAA to better capture the principal directions of the aggregate updates. Moreover, the orthonormal structure of AAA bounds the gradient norms of BBB and preserves more signal under DP-SGD, as confirmed by our theoretical analysis. As a result, FedSVD consistently improves stability and performance across a variety of privacy settings and benchmarks, outperforming relevant baselines under both private and non-private regimes.

View on arXiv
@article{lee2025_2505.12805,
  title={ FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA },
  author={ Seanie Lee and Sangwoo Park and Dong Bok Lee and Dominik Wagner and Haebin Seong and Tobias Bocklet and Juho Lee and Sung Ju Hwang },
  journal={arXiv preprint arXiv:2505.12805},
  year={ 2025 }
}
Comments on this paper