SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum

1 October 2019

Papers citing "SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum"

43 / 43 papers shown

Title
Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training Hiroki Naganuma Xinzhi Zhang Man-Chung Yue Ioannis Mitliagkas Philipp A. Witte Russell J. Hewett Yin Tat Lee 63 0 0 25 Apr 2025
EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models Jialiang Cheng Ning Gao Yun Yue Zhiling Ye Jiadi Jiang Jian Sha OffRL 77 0 0 10 Dec 2024
Distributed Sign Momentum with Local Steps for Training Transformers Shuhua Yu Ding Zhou Cong Xie An Xu Zhi-Li Zhang Xin Liu S. Kar 69 0 0 26 Nov 2024
A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needs Yan Sun Li Shen Dacheng Tao FedML 25 0 0 27 Sep 2024
On the Convergence of a Federated Expectation-Maximization Algorithm Zhixu Tao Rajita Chandak Sanjeev R. Kulkarni FedML 30 0 0 11 Aug 2024
Ordered Momentum for Asynchronous SGD Chang-Wei Shi Yi-Rui Yang Wu-Jun Li ODL 56 0 0 27 Jul 2024
A New Theoretical Perspective on Data Heterogeneity in Federated Optimization Jiayi Wang Shiqiang Wang Rong-Rong Chen Mingyue Ji FedML 30 1 0 22 Jul 2024
Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization Xinyu Zhang Weiyu Sun Ying Chen FedML 30 5 0 13 Sep 2023
Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup Yan Sun Li Shen Hao Sun Liang Ding Dacheng Tao FedML 24 17 0 30 Jul 2023
DropCompute: simple and more robust distributed synchronous training via compute variance reduction Niv Giladi Shahar Gottlieb Moran Shkolnik A. Karnieli Ron Banner Elad Hoffer Kfir Y. Levy Daniel Soudry 33 2 0 18 Jun 2023
Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape Yan Sun Li Shen Shi-Yong Chen Liang Ding Dacheng Tao FedML 34 33 0 19 May 2023
On the Convergence of Federated Averaging with Cyclic Client Participation Yae Jee Cho Pranay Sharma Gauri Joshi Zheng Xu Satyen Kale Tong Zhang FedML 44 27 0 06 Feb 2023
Hierarchical Federated Learning with Momentum Acceleration in Multi-Tier Networks Zhengjie Yang Sen Fu Wei Bao Dong Yuan Albert Y. Zomaya FedML 42 5 0 26 Oct 2022
On the Stability Analysis of Open Federated Learning Systems Youbang Sun H. Fernando Tianyi Chen Shahin Shahrampour FedML 29 1 0 25 Sep 2022
On Generalization of Decentralized Learning with Separable Data Hossein Taheri Christos Thrampoulidis FedML 27 10 0 15 Sep 2022
Byzantines can also Learn from History: Fall of Centered Clipping in Federated Learning Kerem Ozfatura Emre Ozfatura Alptekin Kupcu Deniz Gunduz AAML FedML 28 13 0 21 Aug 2022
Accelerated Federated Learning with Decoupled Adaptive Optimization Jiayin Jin Jiaxiang Ren Yang Zhou Lingjuan Lyu Ji Liu Dejing Dou AI4CE FedML 19 51 0 14 Jul 2022
Distributed Adversarial Training to Robustify Deep Neural Networks at Scale Gaoyuan Zhang Songtao Lu Yihua Zhang Xiangyi Chen Pin-Yu Chen Quanfu Fan Lee Martie L. Horesh Min-Fong Hong Sijia Liu OOD 24 12 0 13 Jun 2022
Generalized Federated Learning via Sharpness Aware Minimization Zhe Qu Xingyu Li Rui Duan Yaojiang Liu Bo Tang Zhuo Lu FedML 25 131 0 06 Jun 2022
Communication-Efficient Adaptive Federated Learning Yujia Wang Lu Lin Jinghui Chen FedML 24 70 0 05 May 2022
Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction Junyi Li Feihu Huang Heng-Chiao Huang FedML 19 27 0 03 May 2022
Robust Federated Learning with Connectivity Failures: A Semi-Decentralized Framework with Collaborative Relaying M. Yemini R. Saha Emre Ozfatura Deniz Gündüz Andrea J. Goldsmith FedML 40 8 0 24 Feb 2022
Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank? Sheikh Shams Azam Seyyedali Hosseinalipour Qiang Qiu Christopher G. Brinton FedML 20 20 0 01 Feb 2022
What Do We Mean by Generalization in Federated Learning? Honglin Yuan Warren Morningstar Lin Ning K. Singhal OOD FedML 41 71 0 27 Oct 2021
Exponential Graph is Provably Efficient for Decentralized Deep Training Bicheng Ying Kun Yuan Yiming Chen Hanbin Hu Pan Pan W. Yin FedML 39 83 0 26 Oct 2021
Acceleration in Distributed Optimization under Similarity Helena Lofstrom G. Scutari Tianyue Cao Alexander Gasnikov 19 26 0 24 Oct 2021
Trade-offs of Local SGD at Scale: An Empirical Study Jose Javier Gonzalez Ortiz Jonathan Frankle Michael G. Rabbat Ari S. Morcos Nicolas Ballas FedML 35 19 0 15 Oct 2021
FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learning Charlie Hou K. K. Thekumparampil Giulia Fanti Sewoong Oh FedML 30 14 0 16 Aug 2021
ErrorCompensatedX: error compensation for variance reduced algorithms Hanlin Tang Yao Li Ji Liu Ming Yan 26 9 0 04 Aug 2021
A Field Guide to Federated Optimization Jianyu Wang Zachary B. Charles Zheng Xu Gauri Joshi H. B. McMahan ... Mi Zhang Tong Zhang Chunxiang Zheng Chen Zhu Wennan Zhu FedML 187 411 0 14 Jul 2021
Secure Distributed Training at Scale Eduard A. Gorbunov Alexander Borzunov Michael Diskin Max Ryabinin FedML 21 15 0 21 Jun 2021
FedCM: Federated Learning with Client-level Momentum Jing Xu Sen Wang Liwei Wang Andrew Chi-Chih Yao FedML 22 94 0 21 Jun 2021
Local Adaptivity in Federated Learning: Convergence and Consistency Jianyu Wang Zheng Xu Zachary Garrett Zachary B. Charles Luyang Liu Gauri Joshi FedML 24 39 0 04 Jun 2021
Consensus Control for Decentralized Deep Learning Lingjing Kong Tao R. Lin Anastasia Koloskova Martin Jaggi Sebastian U. Stich 19 75 0 09 Feb 2021
Time-Correlated Sparsification for Communication-Efficient Federated Learning Emre Ozfatura Kerem Ozfatura Deniz Gunduz FedML 36 47 0 21 Jan 2021
FedADC: Accelerated Federated Learning with Drift Control Emre Ozfatura Kerem Ozfatura Deniz Gunduz FedML 40 37 0 16 Dec 2020
Faster Non-Convex Federated Learning via Global and Local Momentum Rudrajit Das Anish Acharya Abolfazl Hashemi Sujay Sanghavi Inderjit S. Dhillon Ufuk Topcu FedML 35 82 0 07 Dec 2020
Federated Composite Optimization Honglin Yuan Manzil Zaheer Sashank J. Reddi FedML 32 58 0 17 Nov 2020
Throughput-Optimal Topology Design for Cross-Silo Federated Learning Othmane Marfoq Chuan Xu Giovanni Neglia Richard Vidal FedML 67 85 0 23 Oct 2020
Federated Learning with Nesterov Accelerated Gradient Zhengjie Yang Wei Bao Dong Yuan Nguyen H. Tran Albert Y. Zomaya FedML 19 29 0 18 Sep 2020
FetchSGD: Communication-Efficient Federated Learning with Sketching D. Rothchild Ashwinee Panda Enayat Ullah Nikita Ivkin Ion Stoica Vladimir Braverman Joseph E. Gonzalez Raman Arora FedML 17 361 0 15 Jul 2020
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization Jianyu Wang Qinghua Liu Hao Liang Gauri Joshi H. Vincent Poor MoMe FedML 16 1,297 0 15 Jul 2020
Quasi-hyperbolic momentum and Adam for deep learning Jerry Ma Denis Yarats ODL 84 129 0 16 Oct 2018