Local SGD Converges Fast and Communicates Little

24 May 2018

Papers citing "Local SGD Converges Fast and Communicates Little"

29 / 629 papers shown

Title
Decentralized Deep Learning with Arbitrary Communication Compression Anastasia Koloskova Tao R. Lin Sebastian U. Stich Martin Jaggi FedML 15 232 0 22 Jul 2019
On the Convergence of FedAvg on Non-IID Data Xiang Li Kaixuan Huang Wenhao Yang Shusen Wang Zhihua Zhang FedML 15 2,275 0 04 Jul 2019
Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent Shuheng Shen Linli Xu Jingchang Liu Xianfeng Liang Yifei Cheng ODL FedML 21 24 0 28 Jun 2019
Distributed Optimization for Over-Parameterized Learning Chi Zhang Qianxiao Li 8 4 0 14 Jun 2019
Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations Debraj Basu Deepesh Data C. Karakuş Suhas Diggavi MQ 11 400 0 06 Jun 2019
Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithms Xiangyi Chen Tiancong Chen Haoran Sun Zhiwei Steven Wu Mingyi Hong FedML 11 73 0 04 Jun 2019
Natural Compression for Distributed Deep Learning Samuel Horváth Chen-Yu Ho L. Horvath Atal Narayan Sahu Marco Canini Peter Richtárik 21 150 0 27 May 2019
Fair Resource Allocation in Federated Learning Tian Li Maziar Sanjabi Ahmad Beirami Virginia Smith FedML 14 781 0 25 May 2019
Memorized Sparse Backpropagation Zhiyuan Zhang Pengcheng Yang Xuancheng Ren Qi Su Xu Sun 11 13 0 24 May 2019
MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling Jianyu Wang Anit Kumar Sahu Zhouyi Yang Gauri Joshi S. Kar 15 159 0 23 May 2019
On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimization Hao Yu R. L. Jin 10 50 0 10 May 2019
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization Hao Yu R. L. Jin Sen Yang FedML 27 378 0 09 May 2019
SLSGD: Secure and Efficient Distributed On-device Machine Learning Cong Xie Oluwasanmi Koyejo Indranil Gupta FedML 14 14 0 16 Mar 2019
A Distributed Hierarchical SGD Algorithm with Sparse Global Reduction Fan Zhou Guojing Cong 9 8 0 12 Mar 2019
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication Anastasia Koloskova Sebastian U. Stich Martin Jaggi FedML 14 501 0 01 Feb 2019
Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air Mohammad Mohammadi Amiri Deniz Gunduz 17 53 0 03 Jan 2019
Federated Optimization in Heterogeneous Networks Tian Li Anit Kumar Sahu Manzil Zaheer Maziar Sanjabi Ameet Talwalkar Virginia Smith FedML 11 5,011 0 14 Dec 2018
Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning Tianyi Chen K. Zhang G. Giannakis Tamer Basar OffRL 16 41 0 07 Dec 2018
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD Jianyu Wang Gauri Joshi FedML 17 231 0 19 Oct 2018
Distributed Learning over Unreliable Networks Chen Yu Hanlin Tang Cédric Renggli S. Kassing Ankit Singla Dan Alistarh Ce Zhang Ji Liu OOD 9 59 0 17 Oct 2018
Sparsified SGD with Memory Sebastian U. Stich Jean-Baptiste Cordonnier Martin Jaggi 25 737 0 20 Sep 2018
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms Jianyu Wang Gauri Joshi 11 348 0 22 Aug 2018
Don't Use Large Mini-Batches, Use Local SGD Tao R. Lin Sebastian U. Stich Kumar Kshitij Patel Martin Jaggi 25 429 0 22 Aug 2018
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning Hao Yu Sen Yang Shenghuo Zhu MoMe FedML 22 594 0 17 Jul 2018
Communication Compression for Decentralized Training Hanlin Tang Shaoduo Gan Ce Zhang Tong Zhang Ji Liu 11 270 0 17 Mar 2018
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 278 2,888 0 15 Sep 2016
A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method Simon Lacoste-Julien Mark W. Schmidt Francis R. Bach 119 259 0 10 Dec 2012
Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes Ohad Shamir Tong Zhang 99 570 0 08 Dec 2012
Optimal Distributed Online Prediction using Mini-Batches O. Dekel Ran Gilad-Bachrach Ohad Shamir Lin Xiao 168 683 0 07 Dec 2010