Byzantine-Resilient Federated Learning with Heterogeneous Data Distribution

For mitigating Byzantine behaviors in federated learning (FL), most state-of-the-art approaches, such as Bulyan, tend to leverage the similarity of updates from the benign clients. However, in federated learning (FL), data distribution across clients is typically heterogeneous. This makes Byzantine fault mitigation a very challenging task, as even the updates from the benign clients are quite dissimilar from each other. Hence, most prior methods, which treat any update that differs from the majority of other updates as a Byzantine update, exhibit poor performance. We propose DiverseFL, in which rather than comparing each client's update with other updates to detect Byzantine clients, the FL server compares each client's update with a guiding update of that client. Any client whose update diverges from its associated guiding update is tagged as a Byzantine node. The FL server computes the guiding update for each participating client over a small sample of the client's local data that is received only once before training. For preserving the privacy of the shared samples, DiverseFL creates a Trusted Execution Environment (TEE)-based secure enclave within the FL server to receive each client's samples, compute guiding updates, and perform secure aggregation for global model update. In experiments, DiverseFL achieves improvements of up to ~16% in absolute test accuracy over prior benchmarks, and consistently performs closely to OracleSGD, where the server only aggregates the updates from the benign clients. We also analyze convergence rate of DiverseFL with non-IID data, under simplifying assumptions such as strong convexity of local loss.