WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average

27 May 2024

Papers citing "WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average"

4 / 4 papers shown

Title
Communication Efficient LLM Pre-training with SparseLoCo Amir Sarfi Benjamin Thérien Joel Lidin Eugene Belilovsky 72 1 0 21 Aug 2025
Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning Tolga Dimlioglu A. Choromańska FedML 214 1 0 27 Jul 2025
Model Parallelism With Subnetwork Data Parallelism Vaibhav Singh Zafir Khalid Edouard Oyallon Eugene Belilovsky 269 1 0 11 Jul 2025
Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo Zachary B. Charles Gabriel Teston Lucio Dery Keith Rush Nova Fallen Zachary Garrett Arthur Szlam Arthur Douillard 834 12 0 12 Mar 2025