Efficient Zero-Order Federated Finetuning of Language Models for Resource-Constrained Devices

Federated fine-tuning offers a promising approach for tuning Large Language Models (LLMs) on edge devices while preserving data privacy. However, fine-tuning these models on edge devices remains challenging due to high memory, communication, and computational demands. Zero-order optimization with task alignment provides a potential solution, enabling fine-tuning with inference-level memory requirements but requires a longer convergence time. In this paper, we propose Federated Split-Perturbation Zero-order Optimization (FedSPZO) that divides the network into two blocks, applying a different number of perturbations per block in a computationally effective way, achieving faster convergence. Our evaluation shows a $2.5 - 7\times$ reduction in computation overhead compared to zero-order state of the art techniques in federated learning.

@article{ahmed2025_2502.10239,
  title={ Efficient Zero-Order Federated Finetuning of Language Models for Resource-Constrained Devices },
  author={ Mohamed Aboelenien Ahmed and Kilian Pfeiffer and Ramin Khalili and Heba Khdr and Jörg Henkel },
  journal={arXiv preprint arXiv:2502.10239},
  year={ 2025 }
}