Don't Use Large Mini-Batches, Use Local SGD

22 August 2018

Papers citing "Don't Use Large Mini-Batches, Use Local SGD"

50 / 271 papers shown

Title
Stochastic Controlled Averaging for Federated Learning with Communication Compression Xinmeng Huang Ping Li Xiaoyun Li 32 195 0 16 Aug 2023
Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup Yan Sun Li Shen Hao Sun Liang Ding Dacheng Tao FedML 19 16 0 30 Jul 2023
DIGEST: Fast and Communication Efficient Decentralized Learning with Local Updates Peyman Gholami H. Seferoglu FedML 8 11 0 14 Jul 2023
Momentum Benefits Non-IID Federated Learning Simply and Provably Ziheng Cheng Xinmeng Huang Pengfei Wu Kun Yuan FedML 27 16 0 28 Jun 2023
Near-Optimal Nonconvex-Strongly-Convex Bilevel Optimization with Fully First-Order Oracles Le‐Yu Chen Yaohua Ma J. Zhang 84 2 0 26 Jun 2023
DropCompute: simple and more robust distributed synchronous training via compute variance reduction Niv Giladi Shahar Gottlieb Moran Shkolnik A. Karnieli Ron Banner Elad Hoffer Kfir Y. Levy Daniel Soudry 25 2 0 18 Jun 2023
Batches Stabilize the Minimum Norm Risk in High Dimensional Overparameterized Linear Regression Shahar Stein Ioushua Inbar Hasidim O. Shayevitz M. Feder 14 0 0 14 Jun 2023
$$\textbf{A}^2\textbf{CiD}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning$ $\textbf{A}^2\textbf{CiD}^2$ : Accelerating Asynchronous Communication in Decentralized Deep Learning Adel Nabli Eugene Belilovsky Edouard Oyallon 13 6 0 14 Jun 2023
On the Computation-Communication Trade-Off with A Flexible Gradient Tracking Approach Yan Huang Jinming Xu 25 2 0 12 Jun 2023
Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization Yan Sun Li Shen Dacheng Tao FedML 20 14 0 09 Jun 2023
A Lightweight Method for Tackling Unknown Participation Statistics in Federated Averaging Shiqiang Wang Mingyue Ji FedML 30 0 0 06 Jun 2023
Stochastic Gradient Langevin Dynamics Based on Quantization with Increasing Resolution Jinwuk Seok Chang-Jae Cho 20 0 0 30 May 2023
FAVANO: Federated AVeraging with Asynchronous NOdes Louis Leconte Van Minh Nguyen Eric Moulines FedML 28 2 0 25 May 2023
Local SGD Accelerates Convergence by Exploiting Second Order Information of the Loss Function Linxuan Pan Shenghui Song FedML 17 2 0 24 May 2023
Loss Spike in Training Neural Networks Zhongwang Zhang Z. Xu 28 4 0 20 May 2023
Faster Federated Learning with Decaying Number of Local SGD Steps Jed Mills Jia Hu Geyong Min FedML 30 7 0 16 May 2023
Hierarchical Weight Averaging for Deep Neural Networks Xiaozhe Gu Zixun Zhang Yuncheng Jiang Tao Luo Ruimao Zhang Shuguang Cui Zhuguo Li 14 5 0 23 Apr 2023
WW-FL: Secure and Private Large-Scale Federated Learning F. Marx T. Schneider Ajith Suresh Tobias Wehrle Christian Weinert Hossein Yalame FedML 17 2 0 20 Feb 2023
Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities Aleksandr Beznosikov Martin Takáč Alexander Gasnikov 21 10 0 15 Feb 2023
EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous Data M. Crawshaw Yajie Bao Mingrui Liu FedML 12 8 0 14 Feb 2023
Delay Sensitive Hierarchical Federated Learning with Stochastic Local Updates Abdulmoneam Ali A. Arafa FedML 34 4 0 09 Feb 2023
Federated Learning with Regularized Client Participation Grigory Malinovsky Samuel Horváth Konstantin Burlachenko Peter Richtárik FedML 20 13 0 07 Feb 2023
FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering Yongxin Guo Xiaoying Tang Tao R. Lin OOD FedML 30 8 0 29 Jan 2023
SuperFedNAS: Cost-Efficient Federated Neural Architecture Search for On-Device Inference Alind Khare A. Agrawal Aditya Annavajjala Payman Behnam Myungjin Lee Hugo Latapie Alexey Tumanov FedML 11 2 0 26 Jan 2023
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression Jaeyong Song Jinkyu Yim Jaewon Jung Hongsun Jang H. Kim Youngsok Kim Jinho Lee GNN 14 25 0 24 Jan 2023
Decentralized Gradient Tracking with Local Steps Yue Liu Tao R. Lin Anastasia Koloskova Sebastian U. Stich 24 36 0 03 Jan 2023
Federated Learning with Flexible Control Shiqiang Wang Jake B. Perazzone Mingyue Ji Kevin S. Chan FedML 28 17 0 16 Dec 2022
FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Data Tailin Zhou Jun Zhang Danny H. K. Tsang FedML 15 57 0 17 Nov 2022
Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities Brian Bartoldson B. Kailkhura Davis W. Blalock 29 47 0 13 Oct 2022
On the Performance of Gradient Tracking with Local Updates Edward Duc Hien Nguyen Sulaiman A. Alghunaim Kun Yuan César A. Uribe 35 18 0 10 Oct 2022
Scaling up Stochastic Gradient Descent for Non-convex Optimisation S. Mohamad H. Alamri A. Bouchachia 34 3 0 06 Oct 2022
STSyn: Speeding Up Local SGD with Straggler-Tolerant Synchronization Feng Zhu Jingjing Zhang Xin Eric Wang 26 3 0 06 Oct 2022
Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learning Haibo Yang Pei-Yuan Qiu Jia Liu FedML 27 12 0 03 Oct 2022
Distributed Non-Convex Optimization with One-Bit Compressors on Heterogeneous Data: Efficient and Resilient Algorithms Ming Xiang Lili Su FedML 18 2 0 03 Oct 2022
$SAGDA: Achieving $\mathcal{O}(ε^{-2})$ Communication Complexity in Federated Min-Max Learning$ SAGDA: Achieving $\mathcal{O}(ε^{-2})$ Communication Complexity in Federated Min-Max Learning Haibo Yang Zhuqing Liu Xin Zhang Jia-Wei Liu FedML 23 0 0 02 Oct 2022
Personalized Federated Learning with Communication Compression El Houcine Bergou Konstantin Burlachenko Aritra Dutta Peter Richtárik FedML 72 9 0 12 Sep 2022
Flexible Vertical Federated Learning with Heterogeneous Parties Timothy Castiglia Shiqiang Wang S. Patterson FedML 22 33 0 26 Aug 2022
Exact Penalty Method for Federated Learning Shenglong Zhou Geoffrey Ye Li FedML 17 0 0 23 Aug 2022
NET-FLEET: Achieving Linear Convergence Speedup for Fully Decentralized Federated Learning with Heterogeneous Data Xin Zhang Minghong Fang Zhuqing Liu Haibo Yang Jia-Wei Liu Zhengyuan Zhu FedML 13 14 0 17 Aug 2022
ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scale Gopinath Chennupati Milind Rao Gurpreet Chadha Aaron Eakin A. Raju ... Andrew Oberlin Buddha Nandanoor Prahalad Venkataramanan Zheng Wu Pankaj Sitpure CLL 16 8 0 19 Jul 2022
On uniform-in-time diffusion approximation for stochastic gradient descent Lei Li Yuliang Wang 48 3 0 11 Jul 2022
Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data Timothy Castiglia Anirban Das Shiqiang Wang S. Patterson FedML 14 48 0 16 Jun 2022
Anchor Sampling for Federated Learning with Partial Client Participation Feijie Wu Song Guo Zhihao Qu Shiqi He Ziming Liu Jing Gao FedML 28 12 0 13 Jun 2022
A principled framework for the design and analysis of token algorithms Hadrien Hendrikx FedML 16 13 0 30 May 2022
FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reduction Yongxin Guo Xiaoying Tang Tao R. Lin FedML 49 27 0 26 May 2022
Test-Time Robust Personalization for Federated Learning Liang Jiang Tao R. Lin FedML OOD TTA 77 43 0 22 May 2022
A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks Mingrui Liu Zhenxun Zhuang Yunwei Lei Chunyang Liao 20 16 0 10 May 2022
Federated Random Reshuffling with Compression and Variance Reduction Grigory Malinovsky Peter Richtárik FedML 16 10 0 08 May 2022
Communication-Efficient Adaptive Federated Learning Yujia Wang Lu Lin Jinghui Chen FedML 19 69 0 05 May 2022
FedGiA: An Efficient Hybrid Algorithm for Federated Learning Shenglong Zhou Geoffrey Ye Li FedML 23 16 0 03 May 2022