Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delays

15 June 2022

Konstantin Mishchenko

Papers citing "Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delays"

43 / 43 papers shown

Title
Nesterov Method for Asynchronous Pipeline Parallel Optimization Thalaiyasingam Ajanthan Sameera Ramasinghe Yan Zuo Gil Avraham Alexander Long 24 0 0 02 May 2025
A Tale of Two Learning Algorithms: Multiple Stream Random Walk and Asynchronous Gossip Peyman Gholami H. Seferoglu 29 0 0 14 Apr 2025
Orthogonal Calibration for Asynchronous Federated Learning Jiayun Zhang Shuheng Li Haiyu Huang Xiaofan Yu Rajesh K. Gupta Jingbo Shang FedML 62 0 0 21 Feb 2025
Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexity A. Maranjyan A. Tyurin Peter Richtárik 34 2 0 28 Jan 2025
Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous ML T. Dahan Kfir Y. Levy 65 0 0 17 Jan 2025
Asynchronous Stochastic Gradient Descent with Decoupled Backpropagation and Layer-Wise Updates Cabrel Teguemne Fokam Khaleelulla Khan Nazeer Lukas König David Kappel Anand Subramoney 28 0 0 08 Oct 2024
MindFlayer: Efficient Asynchronous Parallel SGD in the Presence of Heterogeneous and Random Worker Compute Times A. Maranjyan Omar Shaikh Omar Peter Richtárik 26 3 0 05 Oct 2024
Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailability Ming Xiang Stratis Ioannidis Edmund Yeh Carlee Joe-Wong Lili Su FedML 29 5 0 26 Sep 2024
Faster Stochastic Optimization with Arbitrary Delays via Asynchronous Mini-Batching Amit Attia Ofir Gaash Tomer Koren 40 0 0 14 Aug 2024
Ordered Momentum for Asynchronous SGD Chang-Wei Shi Yi-Rui Yang Wu-Jun Li ODL 54 0 0 27 Jul 2024
FADAS: Towards Federated Adaptive Asynchronous Optimization Yujia Wang Shiqiang Wang Songtao Lu Jinghui Chen FedML 34 3 0 25 Jul 2024
Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods Tim Tsz-Kit Lau Weijian Li Chenwei Xu Han Liu Mladen Kolar 44 1 0 20 Jun 2024
Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Framework Siyuan Yu Wei Chen H. V. Poor 32 0 0 17 Jun 2024
ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training Adel Nabli Louis Fournier Pierre Erbacher Louis Serrano Eugene Belilovsky Edouard Oyallon FedML 46 1 0 03 Jun 2024
Dual-Delayed Asynchronous SGD for Arbitrarily Heterogeneous Data Xiaolu Wang Yuchang Sun Hoi-To Wai Jun Zhang 41 0 0 27 May 2024
Freya PAGE: First Optimal Time Complexity for Large-Scale Nonconvex Finite-Sum Optimization with Heterogeneous Asynchronous Computations A. Tyurin Kaja Gruntkowska Peter Richtárik 31 3 0 24 May 2024
Asynchronous Federated Stochastic Optimization for Heterogeneous Objectives Under Arbitrary Delays Charikleia Iakovidou Kibaek Kim FedML 35 2 0 16 May 2024
Beyond Noise: Privacy-Preserving Decentralized Learning with Virtual Nodes Sayan Biswas Mathieu Even Anne-Marie Kermarrec Laurent Massoulie Rafael Pires Rishi Sharma M. Vos 38 3 0 15 Apr 2024
Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey Feng Liang Zhen Zhang Haifeng Lu Victor C. M. Leung Yanyi Guo Xiping Hu GNN 37 6 0 09 Apr 2024
Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis Guangchen Lan Dong-Jun Han Abolfazl Hashemi Vaneet Aggarwal Christopher G. Brinton 124 15 0 09 Apr 2024
Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networks Louis Fournier Edouard Oyallon 33 0 0 13 Mar 2024
Queuing dynamics of asynchronous Federated Learning Louis Leconte Matthieu Jonckheere S. Samsonov Eric Moulines FedML 37 5 0 12 Feb 2024
CO2: Efficient Distributed Training with Full Communication-Computation Overlap Weigao Sun Zhen Qin Weixuan Sun Shidi Li Dong Li Xuyang Shen Yu Qiao Yiran Zhong OffRL 53 10 0 29 Jan 2024
Asynchronous SGD on Graphs: a Unified Framework for Asynchronous Decentralized and Federated Optimization Mathieu Even Anastasia Koloskova Laurent Massoulié FedML 38 13 0 01 Nov 2023
AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithms Rustem Islamov M. Safaryan Dan Alistarh FedML 23 12 0 31 Oct 2023
Convergence Analysis of Decentralized ASGD Mauro Dalle Lucca Tosi Martin Theobald 26 2 0 07 Sep 2023
FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs Zhenheng Tang Yuxin Wang Xin He Longteng Zhang Xinglin Pan ... Rongfei Zeng Kaiyong Zhao S. Shi Bingsheng He Xiaowen Chu 38 30 0 03 Sep 2023
Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent Xiaoge Deng Li Shen Shengwei Li Tao Sun Dongsheng Li Dacheng Tao 28 3 0 18 Aug 2023
Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity Konstantin Mishchenko Rustem Islamov Eduard A. Gorbunov Samuel Horváth FedML 33 11 0 29 May 2023
FAVANO: Federated AVeraging with Asynchronous NOdes Louis Leconte Van Minh Nguyen Eric Moulines FedML 28 2 0 25 May 2023
Stochastic Gradient Descent under Markovian Sampling Schemes Mathieu Even 11 28 0 28 Feb 2023
Delayed Stochastic Algorithms for Distributed Weakly Convex Optimization W. Gao Qinhao Deng 16 0 0 30 Jan 2023
SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communication Marco Bornstein Tahseen Rabbani Evana Wang Amrit Singh Bedi Furong Huang FedML 49 18 0 25 Oct 2022
PersA-FL: Personalized Asynchronous Federated Learning Taha Toghani Soomin Lee César A. Uribe FedML 34 6 0 03 Oct 2022
Unbounded Gradients in Federated Learning with Buffered Asynchronous Aggregation Taha Toghani César A. Uribe FedML 35 14 0 03 Oct 2022
Communication-Efficient Federated Learning With Data and Client Heterogeneity Hossein Zakerinia Shayan Talaei Giorgi Nadiradze Dan Alistarh FedML 19 7 0 20 Jun 2022
Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning Anastasia Koloskova Sebastian U. Stich Martin Jaggi FedML 22 77 0 16 Jun 2022
Delay-adaptive step-sizes for asynchronous learning Xuyang Wu Sindri Magnússon Hamid Reza Feyzmahdavian M. Johansson 32 14 0 17 Feb 2022
Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guarantees Hamid Reza Feyzmahdavian M. Johansson 16 20 0 09 Sep 2021
Asynchronous speedup in decentralized optimization Mathieu Even Hadrien Hendrikx Laurent Massoulie 13 4 0 07 Jun 2021
FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout Samuel Horváth Stefanos Laskaridis Mario Almeida Ilias Leondiadis Stylianos I. Venieris Nicholas D. Lane 181 267 0 26 Feb 2021
Communication-Efficient Distributed Deep Learning: A Comprehensive Survey Zhenheng Tang S. Shi Wei Wang Bo-wen Li Xiaowen Chu 19 48 0 10 Mar 2020
Optimal Distributed Online Prediction using Mini-Batches O. Dekel Ran Gilad-Bachrach Ohad Shamir Lin Xiao 171 683 0 07 Dec 2010