Don't Use Large Mini-Batches, Use Local SGD

22 August 2018

Papers citing "Don't Use Large Mini-Batches, Use Local SGD"

50 / 73 papers shown

Title
Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training Hiroki Naganuma Xinzhi Zhang Man-Chung Yue Ioannis Mitliagkas Philipp A. Witte Russell J. Hewett Yin Tat Lee 63 0 0 25 Apr 2025
FedSat: A Statistical Aggregation Approach for Class Imbalanced Clients in Federated Learning S. Chowdhury Raju Halder FedML 32 0 0 31 Dec 2024
EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models Jialiang Cheng Ning Gao Yun Yue Zhiling Ye Jiadi Jiang Jian Sha OffRL 74 0 0 10 Dec 2024
Distributed Sign Momentum with Local Steps for Training Transformers Shuhua Yu Ding Zhou Cong Xie An Xu Zhi-Li Zhang Xin Liu S. Kar 64 0 0 26 Nov 2024
DEPT: Decoupled Embeddings for Pre-training Language Models Alex Iacob Lorenzo Sani Meghdad Kurmanji William F. Shen Xinchi Qiu Dongqi Cai Yan Gao Nicholas D. Lane VLM 112 0 0 07 Oct 2024
A New Theoretical Perspective on Data Heterogeneity in Federated Optimization Jiayi Wang Shiqiang Wang Rong-Rong Chen Mingyue Ji FedML 28 1 0 22 Jul 2024
The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication Kumar Kshitij Patel Margalit Glasgow Ali Zindari Lingxiao Wang Sebastian U. Stich Ziheng Cheng Nirmit Joshi Nathan Srebro 44 6 0 19 May 2024
Momentum-SAM: Sharpness Aware Minimization without Computational Overhead Marlon Becker Frederick Altrock Benjamin Risse 74 5 0 22 Jan 2024
Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup Yan Sun Li Shen Hao Sun Liang Ding Dacheng Tao FedML 19 16 0 30 Jul 2023
DropCompute: simple and more robust distributed synchronous training via compute variance reduction Niv Giladi Shahar Gottlieb Moran Shkolnik A. Karnieli Ron Banner Elad Hoffer Kfir Y. Levy Daniel Soudry 25 2 0 18 Jun 2023
Faster Federated Learning with Decaying Number of Local SGD Steps Jed Mills Jia Hu Geyong Min FedML 30 7 0 16 May 2023
Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities Aleksandr Beznosikov Martin Takáč Alexander Gasnikov 21 10 0 15 Feb 2023
Delay Sensitive Hierarchical Federated Learning with Stochastic Local Updates Abdulmoneam Ali A. Arafa FedML 29 3 0 09 Feb 2023
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression Jaeyong Song Jinkyu Yim Jaewon Jung Hongsun Jang H. Kim Youngsok Kim Jinho Lee GNN 8 25 0 24 Jan 2023
Federated Learning with Flexible Control Shiqiang Wang Jake B. Perazzone Mingyue Ji Kevin S. Chan FedML 22 17 0 16 Dec 2022
On the Performance of Gradient Tracking with Local Updates Edward Duc Hien Nguyen Sulaiman A. Alghunaim Kun Yuan César A. Uribe 35 18 0 10 Oct 2022
STSyn: Speeding Up Local SGD with Straggler-Tolerant Synchronization Feng Zhu Jingjing Zhang Xin Eric Wang 22 3 0 06 Oct 2022
NET-FLEET: Achieving Linear Convergence Speedup for Fully Decentralized Federated Learning with Heterogeneous Data Xin Zhang Minghong Fang Zhuqing Liu Haibo Yang Jia-Wei Liu Zhengyuan Zhu FedML 10 14 0 17 Aug 2022
ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scale Gopinath Chennupati Milind Rao Gurpreet Chadha Aaron Eakin A. Raju ... Andrew Oberlin Buddha Nandanoor Prahalad Venkataramanan Zheng Wu Pankaj Sitpure CLL 16 8 0 19 Jul 2022
On uniform-in-time diffusion approximation for stochastic gradient descent Lei Li Yuliang Wang 48 3 0 11 Jul 2022
A principled framework for the design and analysis of token algorithms Hadrien Hendrikx FedML 16 13 0 30 May 2022
Federated Random Reshuffling with Compression and Variance Reduction Grigory Malinovsky Peter Richtárik FedML 16 10 0 08 May 2022
Communication-Efficient Adaptive Federated Learning Yujia Wang Lu Lin Jinghui Chen FedML 19 69 0 05 May 2022
FedCos: A Scene-adaptive Federated Optimization Enhancement for Performance Improvement Hao Zhang Tingting Wu Siyao Cheng Jie Liu FedML 30 11 0 07 Apr 2022
Sparse Federated Learning with Hierarchical Personalized Models Xiaofeng Liu Qing Wang Yunfeng Shao Yinchuan Li FedML 36 11 0 25 Mar 2022
Towards Federated Learning on Time-Evolving Heterogeneous Data Yongxin Guo Tao R. Lin Xiaoying Tang FedML 14 30 0 25 Dec 2021
Large-Scale Deep Learning Optimizations: A Comprehensive Survey Xiaoxin He Fuzhao Xue Xiaozhe Ren Yang You 22 14 0 01 Nov 2021
KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Tasks Rishabh Bhardwaj Tushar Vaidya Soujanya Poria OT FedML 55 7 0 06 Oct 2021
Accelerate Distributed Stochastic Descent for Nonconvex Optimization with Momentum Guojing Cong Tianyi Liu 16 0 0 01 Oct 2021
FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learning Charlie Hou K. K. Thekumparampil Giulia Fanti Sewoong Oh FedML 30 14 0 16 Aug 2021
A Field Guide to Federated Optimization Jianyu Wang Zachary B. Charles Zheng Xu Gauri Joshi H. B. McMahan ... Mi Zhang Tong Zhang Chunxiang Zheng Chen Zhu Wennan Zhu FedML 173 411 0 14 Jul 2021
BAGUA: Scaling up Distributed Learning with System Relaxations Shaoduo Gan Xiangru Lian Rui Wang Jianbin Chang Chengjun Liu ... Jiawei Jiang Binhang Yuan Sen Yang Ji Liu Ce Zhang 20 30 0 03 Jul 2021
ResIST: Layer-Wise Decomposition of ResNets for Distributed Training Chen Dun Cameron R. Wolfe C. Jermaine Anastasios Kyrillidis 16 21 0 02 Jul 2021
Implicit Gradient Alignment in Distributed and Federated Learning Yatin Dandi Luis Barba Martin Jaggi FedML 18 31 0 25 Jun 2021
Behavior Mimics Distribution: Combining Individual and Group Behaviors for Federated Learning Hua Huang Fanhua Shang Yuanyuan Liu Hongying Liu FedML 11 14 0 23 Jun 2021
On Large-Cohort Training for Federated Learning Zachary B. Charles Zachary Garrett Zhouyuan Huo Sergei Shmulyian Virginia Smith FedML 16 112 0 15 Jun 2021
Federated Learning with Buffered Asynchronous Aggregation John Nguyen Kshitiz Malik Hongyuan Zhan Ashkan Yousefpour Michael G. Rabbat Mani Malek Dzmitry Huba FedML 11 287 0 11 Jun 2021
Towards Demystifying Serverless Machine Learning Training Jiawei Jiang Shaoduo Gan Yue Liu Fanlin Wang Gustavo Alonso Ana Klimovic Ankit Singla Wentao Wu Ce Zhang 14 121 0 17 May 2021
Relating Adversarially Robust Generalization to Flat Minima David Stutz Matthias Hein Bernt Schiele OOD 22 65 0 09 Apr 2021
Personalized Federated Learning using Hypernetworks Aviv Shamsian Aviv Navon Ethan Fetaya Gal Chechik FedML 25 324 0 08 Mar 2021
Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices Max Ryabinin Eduard A. Gorbunov Vsevolod Plokhotnyuk Gennady Pekhimenko 21 31 0 04 Mar 2021
Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiency Yuyang Deng M. Mahdavi 14 58 0 25 Feb 2021
GIST: Distributed Training for Large-Scale Graph Convolutional Networks Cameron R. Wolfe Jingkang Yang Arindam Chowdhury Chen Dun Artun Bayer Santiago Segarra Anastasios Kyrillidis BDL GNN LRM 41 9 0 20 Feb 2021
Consensus Control for Decentralized Deep Learning Lingjing Kong Tao R. Lin Anastasia Koloskova Martin Jaggi Sebastian U. Stich 19 75 0 09 Feb 2021
Local SGD: Unified Theory and New Efficient Methods Eduard A. Gorbunov Filip Hanzely Peter Richtárik FedML 19 108 0 03 Nov 2020
Demystifying Why Local Aggregation Helps: Convergence Analysis of Hierarchical SGD Jiayi Wang Shiqiang Wang Rong-Rong Chen Mingyue Ji FedML 23 50 0 24 Oct 2020
Throughput-Optimal Topology Design for Cross-Silo Federated Learning Othmane Marfoq Chuan Xu Giovanni Neglia Richard Vidal FedML 51 85 0 23 Oct 2020
Blind Federated Edge Learning M. Amiri T. Duman Deniz Gunduz Sanjeev R. Kulkarni H. Vincent Poor 71 92 0 19 Oct 2020
Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training Shen-Yi Zhao Chang-Wei Shi Yin-Peng Xie Wu-Jun Li ODL 8 8 0 28 Jul 2020
Fast-Convergent Federated Learning Hung T. Nguyen Vikash Sehwag Seyyedali Hosseinalipour Christopher G. Brinton M. Chiang H. Vincent Poor FedML 22 190 0 26 Jul 2020