v1v2v3v4v5v6 (latest)

Don't Use Large Mini-Batches, Use Local SGD

22 August 2018

Papers citing "Don't Use Large Mini-Batches, Use Local SGD"

50 / 280 papers shown

Title
Momentum-SAM: Sharpness Aware Minimization without Computational Overhead Marlon Becker Frederick Altrock Benjamin Risse 453 9 0 22 Jan 2024
Asynchronous Local-SGD Training for Language Modeling Bo Liu Rachita Chhaparia Arthur Douillard Satyen Kale Andrei A. Rusu Jiajun Shen Arthur Szlam MarcÁurelio Ranzato FedML 270 16 0 17 Jan 2024
On the Role of Server Momentum in Federated Learning Jianhui Sun Xidong Wu Heng-Chiao Huang Aidong Zhang FedML 241 20 0 19 Dec 2023
Meta-learning Optimizers for Communication-Efficient Learning Charles-Étienne Joseph Benjamin Thérien A. Moudgil Boris Knyazev Eugene Belilovsky 363 2 0 02 Dec 2023
Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentum Riccardo Zaccone Sai Praneeth Karimireddy Carlo Masone Marco Ciccone FedML 417 3 0 30 Nov 2023
DiLoCo: Distributed Low-Communication Training of Language Models Arthur Douillard Qixuang Feng Andrei A. Rusu Rachita Chhaparia Yani Donchev A. Kuncoro MarcÁurelio Ranzato Arthur Szlam Jiajun Shen 291 74 0 14 Nov 2023
A Quadratic Synchronization Rule for Distributed Deep LearningInternational Conference on Learning Representations (ICLR), 2023 Xinran Gu Kaifeng Lyu Sanjeev Arora Jingzhao Zhang Longbo Huang 297 4 0 22 Oct 2023
Federated Multi-Objective Learning Haibo Yang Zhuqing Liu Jia-Wei Liu Chaosheng Dong Michinari Momma FedML 272 19 0 15 Oct 2023
Enhancing Clustered Federated Learning: Integration of Strategies and Improved MethodologiesInternational Conference on Learning Representations (ICLR), 2023 Yongxin Guo Xiaoying Tang Tao Lin FedML 87 1 0 09 Oct 2023
Minibatch and Local SGD: Algorithmic Stability and Linear Speedup in Generalization Yunwen Lei Tao Sun Mingrui Liu 437 4 0 02 Oct 2023
FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data Hao Sun Li Shen Shi-Yong Chen Jingwei Sun Jing Li Guangzhong Sun Dacheng Tao FedML 187 2 0 18 Sep 2023
Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problemsScientific Reports (Sci Rep), 2023 Maria Chiara Angelini A. Cavaliere Raffaele Marino F. Ricci-Tersenghi 336 5 0 11 Sep 2023
Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to BeatIEEE International Conference on Computer Vision (ICCV), 2023 Erdong Hu Yu-Shuen Tang Anastasios Kyrillidis C. Jermaine FedML 265 13 0 06 Sep 2023
Stochastic Controlled Averaging for Federated Learning with Communication CompressionInternational Conference on Learning Representations (ICLR), 2023 Xinmeng Huang Ping Li Xiaoyun Li 375 247 0 16 Aug 2023
Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear SpeedupIEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023 Yan Sun Li Shen Hao Sun Liang Ding Dacheng Tao FedML 152 26 0 30 Jul 2023
DIGEST: Fast and Communication Efficient Decentralized Learning with Local UpdatesIEEE Transactions on Machine Learning in Communications and Networking (IEEE TMLCN), 2023 Peyman Gholami H. Seferoglu FedML 240 16 0 14 Jul 2023
Momentum Benefits Non-IID Federated Learning Simply and ProvablyInternational Conference on Learning Representations (ICLR), 2023 Ziheng Cheng Xinmeng Huang Pengfei Wu Kun Yuan FedML 523 34 0 28 Jun 2023
Near-Optimal Nonconvex-Strongly-Convex Bilevel Optimization with Fully First-Order Oracles Le‐Yu Chen Yaohua Ma J.N. Zhang 422 10 0 26 Jun 2023
DropCompute: simple and more robust distributed synchronous training via compute variance reductionNeural Information Processing Systems (NeurIPS), 2023 Niv Giladi Shahar Gottlieb Moran Shkolnik A. Karnieli Ron Banner Elad Hoffer Kfir Y. Levy Daniel Soudry 346 4 0 18 Jun 2023
Batches Stabilize the Minimum Norm Risk in High Dimensional Overparameterized Linear Regression Shahar Stein Ioushua Inbar Hasidim O. Shayevitz M. Feder 249 1 0 14 Jun 2023
$$\textbf{A}^2\textbf{CiD}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning$ $\textbf{A}^2\textbf{CiD}^2$ : Accelerating Asynchronous Communication in Decentralized Deep LearningNeural Information Processing Systems (NeurIPS), 2023 Adel Nabli Eugene Belilovsky Edouard Oyallon 334 9 0 14 Jun 2023
On the Computation-Communication Trade-Off with A Flexible Gradient Tracking ApproachIEEE Conference on Decision and Control (CDC), 2023 Yan Huang Jinming Xu 211 7 0 12 Jun 2023
Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed InitializationNeural Information Processing Systems (NeurIPS), 2023 Yan Sun Li Shen Dacheng Tao FedML 206 19 0 09 Jun 2023
A Lightweight Method for Tackling Unknown Participation Statistics in Federated AveragingInternational Conference on Learning Representations (ICLR), 2023 Maroun Touma Mingyue Ji FedML 297 0 0 06 Jun 2023
Stochastic Gradient Langevin Dynamics Based on Quantization with Increasing Resolution Jinwuk Seok Chang-Jae Cho 258 0 0 30 May 2023
FAVANO: Federated AVeraging with Asynchronous NOdesIEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2023 Louis Leconte Van Minh Nguyen Eric Moulines FedML 269 3 0 25 May 2023
Local SGD Accelerates Convergence by Exploiting Second Order Information of the Loss Function Linxuan Pan Shenghui Song FedML 130 2 0 24 May 2023
Loss Spike in Training Neural NetworksJournal of Computational Mathematics (JCM), 2023 Zhongwang Zhang Z. Xu 200 13 0 20 May 2023
Faster Federated Learning with Decaying Number of Local SGD StepsIEEE Transactions on Parallel and Distributed Systems (TPDS), 2023 Jed Mills Jia Hu Geyong Min FedML 156 12 0 16 May 2023
Hierarchical Weight Averaging for Deep Neural NetworksIEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2023 Xiaozhe Gu Zixun Zhang Yuncheng Jiang Yaoyu Zhang Ruimao Zhang Shuguang Cui Zhuguo Li 136 6 0 23 Apr 2023
WW-FL: Secure and Private Large-Scale Federated Learning F. Marx T. Schneider Ajith Suresh Tobias Wehrle Christian Weinert Hossein Yalame FedML 379 5 0 20 Feb 2023
Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational InequalitiesNeural Information Processing Systems (NeurIPS), 2023 Aleksandr Beznosikov Martin Takáč Alexander Gasnikov 284 12 0 15 Feb 2023
EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous DataInternational Conference on Learning Representations (ICLR), 2023 M. Crawshaw Yajie Bao Mingrui Liu FedML 181 10 0 14 Feb 2023
Delay Sensitive Hierarchical Federated Learning with Stochastic Local UpdatesIEEE Transactions on Cognitive Communications and Networking (IEEE TCCN), 2023 Abdulmoneam Ali A. Arafa FedML 237 8 0 09 Feb 2023
Federated Learning with Regularized Client Participation Grigory Malinovsky Samuel Horváth Konstantin Burlachenko Peter Richtárik FedML 248 18 0 07 Feb 2023
FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust ClusteringInternational Conference on Machine Learning (ICML), 2023 Yongxin Guo Xiaoying Tang Tao Lin OOD FedML 318 23 0 29 Jan 2023
SuperFedNAS: Cost-Efficient Federated Neural Architecture Search for On-Device InferenceEuropean Conference on Computer Vision (ECCV), 2023 Alind Khare A. Agrawal Aditya Annavajjala Rohit Das Myungjin Lee Hugo Latapie Alexey Tumanov FedML 190 3 0 26 Jan 2023
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication CompressionInternational Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2023 Jaeyong Song Jinkyu Yim Jaewon Jung Hongsun Jang H. Kim Youngsok Kim Jinho Lee GNN 254 39 0 24 Jan 2023
Decentralized Gradient Tracking with Local StepsOptimization Methods and Software (OMS), 2023 Yue Liu Tao Lin Anastasia Koloskova Sebastian U. Stich 239 57 0 03 Jan 2023
Federated Learning with Flexible ControlIEEE Conference on Computer Communications (INFOCOM), 2022 Maroun Touma Jake B. Perazzone Mingyue Ji Kevin S. Chan FedML 214 20 0 16 Dec 2022
FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous DataIEEE Transactions on Mobile Computing (IEEE TMC), 2022 Tailin Zhou Jun Zhang Danny H. K. Tsang FedML 375 81 0 17 Nov 2022
Compute-Efficient Deep Learning: Algorithmic Trends and OpportunitiesJournal of machine learning research (JMLR), 2022 Brian Bartoldson B. Kailkhura Davis W. Blalock 280 64 0 13 Oct 2022
On the Performance of Gradient Tracking with Local UpdatesIEEE Conference on Decision and Control (CDC), 2022 Edward Duc Hien Nguyen Sulaiman A. Alghunaim Kun Yuan César A. Uribe 215 29 0 10 Oct 2022
Scaling up Stochastic Gradient Descent for Non-convex OptimisationMachine-mediated learning (ML), 2022 S. Mohamad H. Alamri A. Bouchachia 188 3 0 06 Oct 2022
STSyn: Speeding Up Local SGD with Straggler-Tolerant SynchronizationIEEE Transactions on Signal Processing (IEEE Trans. Signal Process.), 2022 Feng Zhu Jingjing Zhang Xin Eric Wang 282 4 0 06 Oct 2022
Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated LearningNeural Information Processing Systems (NeurIPS), 2022 Haibo Yang Pei-Yuan Qiu Jia Liu FedML 311 17 0 03 Oct 2022
Distributed Non-Convex Optimization with One-Bit Compressors on Heterogeneous Data: Efficient and Resilient Algorithms Ming Xiang Lili Su FedML 162 5 0 03 Oct 2022
$SAGDA: Achieving $\mathcal{O}(ε^{-2})$ Communication Complexity in Federated Min-Max Learning$ SAGDA: Achieving $\mathcal{O}(ε^{-2})$ Communication Complexity in Federated Min-Max Learning Haibo Yang Zhuqing Liu Xin Zhang Jia-Wei Liu FedML 247 0 0 02 Oct 2022
Personalized Federated Learning with Communication Compression El Houcine Bergou Konstantin Burlachenko Aritra Dutta Peter Richtárik FedML 222 10 0 12 Sep 2022
Flexible Vertical Federated Learning with Heterogeneous PartiesIEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2022 Timothy Castiglia Maroun Touma S. Patterson FedML 372 41 0 26 Aug 2022