Fine-tuning in Federated Learning: a simple but tough-to-beat baseline

We study the performance of federated learning algorithms and their variants in an asymptotic framework. Our starting point is the formulation of federated learning as a multi-criterion objective, where the goal is to minimize each client's loss using information from all of the clients. We analyze a linear regression model, where, for a given client, we theoretically compare the performance of various algorithms in the high-dimensional asymptotic limit. This asymptotic multi-criterion approach naturally models the high-dimensional, many-device nature of federated learning and suggests that personalization is central to federated learning. In this paper, we investigate how some sophisticated personalization algorithms fare against simple fine-tuning baselines. In particular, our theory suggests that Federated Averaging with client fine-tuning is competitive than more intricate meta-learning and proximal-regularized approaches. In addition to being conceptually simpler, our fine-tuning-based methods are computationally more efficient than their competitors. We corroborate our theoretical claims with extensive experiments on federated versions of the EMNIST, CIFAR-100, Shakespeare, and Stack Overflow datasets.