Don't Use Large Mini-Batches, Use Local SGD

22 August 2018

Papers citing "Don't Use Large Mini-Batches, Use Local SGD"

50 / 271 papers shown

Title
Ensemble Distillation for Robust Model Fusion in Federated Learning Tao R. Lin Lingjing Kong Sebastian U. Stich Martin Jaggi FedML 8 1,013 0 12 Jun 2020
An Accurate, Scalable and Verifiable Protocol for Federated Differentially Private Averaging C. Sabater A. Bellet J. Ramon FedML 13 18 0 12 Jun 2020
STL-SGD: Speeding Up Local SGD with Stagewise Communication Period Shuheng Shen Yifei Cheng Jingchang Liu Linli Xu LRM 6 7 0 11 Jun 2020
Extrapolation for Large-batch Training in Deep Learning Tao R. Lin Lingjing Kong Sebastian U. Stich Martin Jaggi 12 36 0 10 Jun 2020
Minibatch vs Local SGD for Heterogeneous Distributed Learning Blake E. Woodworth Kumar Kshitij Patel Nathan Srebro FedML 22 198 0 08 Jun 2020
Local SGD With a Communication Overhead Depending Only on the Number of Workers Artin Spiridonoff Alexander Olshevsky I. Paschalidis FedML 12 19 0 03 Jun 2020
DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging Q. Zhou Yawen Zhang Pengcheng Li Xiaoyong Liu Jun Yang Runsheng Wang Ru Huang FedML 14 2 0 31 May 2020
A Quantitative Survey of Communication Optimizations in Distributed Deep Learning S. Shi Zhenheng Tang X. Chu Chengjian Liu Wei Wang Bo Li GNN AI4CE 15 3 0 27 May 2020
MixML: A Unified Analysis of Weakly Consistent Parallel Learning Yucheng Lu J. Nash Christopher De Sa FedML 11 12 0 14 May 2020
Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks Zhishuai Guo Mingrui Liu Zhuoning Yuan Li Shen Wei Liu Tianbao Yang 15 42 0 05 May 2020
Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging Shigang Li Tal Ben-Nun Giorgi Nadiradze Salvatore Di Girolamo Nikoli Dryden Dan Alistarh Torsten Hoefler 21 14 0 30 Apr 2020
Enhancing Privacy via Hierarchical Federated Learning A. Wainakh Alejandro Sánchez Guinea Tim Grube M. Mühlhäuser FedML 20 45 0 23 Apr 2020
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates Anastasia Koloskova Nicolas Loizou Sadra Boreiri Martin Jaggi Sebastian U. Stich FedML 39 491 0 23 Mar 2020
Communication-Efficient Distributed Deep Learning: A Comprehensive Survey Zhenheng Tang S. Shi Wei Wang Bo-wen Li Xiaowen Chu 19 48 0 10 Mar 2020
LASG: Lazily Aggregated Stochastic Gradients for Communication-Efficient Distributed Learning Tianyi Chen Yuejiao Sun W. Yin FedML 6 14 0 26 Feb 2020
Personalized Federated Learning: A Meta-Learning Approach Alireza Fallah Aryan Mokhtari Asuman Ozdaglar FedML 28 560 0 19 Feb 2020
Is Local SGD Better than Minibatch SGD? Blake E. Woodworth Kumar Kshitij Patel Sebastian U. Stich Zhen Dai Brian Bullins H. B. McMahan Ohad Shamir Nathan Srebro FedML 34 253 0 18 Feb 2020
Scalable and Practical Natural Gradient for Large-Scale Deep Learning Kazuki Osawa Yohei Tsuji Yuichiro Ueno Akira Naruse Chuan-Sheng Foo Rio Yokota 23 36 0 13 Feb 2020
Faster On-Device Training Using New Federated Momentum Algorithm Zhouyuan Huo Qian Yang Bin Gu Heng-Chiao Huang FedML 9 47 0 06 Feb 2020
Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descent Giorgi Nadiradze Ilia Markov Bapi Chatterjee Vyacheslav Kungurtsev Dan Alistarh FedML 6 14 0 16 Jan 2020
Exploiting Unlabeled Data in Smart Cities using Federated Learning A. Albaseer Bekir Sait Ciftler M. Abdallah Ala I. Al-Fuqaha 9 18 0 10 Jan 2020
Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well Vipul Gupta S. Serrano D. DeCoste MoMe 30 55 0 07 Jan 2020
Variance Reduced Local SGD with Lower Communication Complexity Xian-Feng Liang Shuheng Shen Jingchang Liu Zhen Pan Enhong Chen Yifei Cheng FedML 21 152 0 30 Dec 2019
Parallel Restarted SPIDER -- Communication Efficient Distributed Nonconvex Optimization with Optimal Computation Complexity Pranay Sharma Swatantra Kafle Prashant Khanduri Saikiran Bulusu K. Rajawat P. Varshney FedML 15 17 0 12 Dec 2019
Advances and Open Problems in Federated Learning Peter Kairouz H. B. McMahan Brendan Avent A. Bellet M. Bennis ... Zheng Xu Qiang Yang Felix X. Yu Han Yu Sen Zhao FedML AI4CE 69 6,063 0 10 Dec 2019
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates Cong Xie Oluwasanmi Koyejo Indranil Gupta Haibin Lin 16 39 0 20 Nov 2019
Energy Efficient Federated Learning Over Wireless Communication Networks Zhaohui Yang Mingzhe Chen Walid Saad C. Hong M. Shikh-Bahaei 11 680 0 06 Nov 2019
On the Convergence of Local Descent Methods in Federated Learning Farzin Haddadpour M. Mahdavi FedML 19 265 0 31 Oct 2019
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization Farzin Haddadpour Mohammad Mahdi Kamani M. Mahdavi V. Cadambe FedML 12 199 0 30 Oct 2019
Asynchronous Decentralized SGD with Quantized and Local Updates Giorgi Nadiradze Amirmojtaba Sabour Peter Davies Shigang Li Dan Alistarh 9 49 0 27 Oct 2019
Sparsification as a Remedy for Staleness in Distributed Asynchronous SGD Rosa Candela Giulio Franzese Maurizio Filippone Pietro Michiardi 8 1 0 21 Oct 2019
Communication-Efficient Local Decentralized SGD Methods Xiang Li Wenhao Yang Shusen Wang Zhihua Zhang 16 53 0 21 Oct 2019
Central Server Free Federated Learning over Single-sided Trust Social Networks Chaoyang He Conghui Tan Hanlin Tang Shuang Qiu Ji Liu FedML 10 73 0 11 Oct 2019
Distributed Learning of Deep Neural Networks using Independent Subnet Training John Shelton Hyatt Cameron R. Wolfe Michael Lee Yuxin Tang Anastasios Kyrillidis Christopher M. Jermaine OOD 13 35 0 04 Oct 2019
Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints Felix Sattler K. Müller Wojciech Samek FedML 40 965 0 04 Oct 2019
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum Jianyu Wang Vinayak Tantia Nicolas Ballas Michael G. Rabbat 4 200 0 01 Oct 2019
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization Amirhossein Reisizadeh Aryan Mokhtari Hamed Hassani Ali Jadbabaie Ramtin Pedarsani FedML 162 760 0 28 Sep 2019
The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication Sebastian U. Stich Sai Praneeth Karimireddy FedML 12 20 0 11 Sep 2019
Addressing Algorithmic Bottlenecks in Elastic Machine Learning with Chicle Michael Kaufmann K. Kourtis Celestine Mendler-Dünner Adrian Schüpbach Thomas Parnell 6 0 0 11 Sep 2019
Tighter Theory for Local SGD on Identical and Heterogeneous Data Ahmed Khaled Konstantin Mishchenko Peter Richtárik 13 424 0 10 Sep 2019
Gradient Descent with Compressed Iterates Ahmed Khaled Peter Richtárik 16 22 0 10 Sep 2019
Hierarchical Federated Learning Across Heterogeneous Cellular Networks Mehdi Salehi Heydar Abad Emre Ozfatura Deniz Gunduz Ozgur Ercetin FedML 17 308 0 05 Sep 2019
Federated Learning: Challenges, Methods, and Future Directions Tian Li Anit Kumar Sahu Ameet Talwalkar Virginia Smith FedML 11 4,413 0 21 Aug 2019
Federated Learning over Wireless Fading Channels M. Amiri Deniz Gunduz 16 505 0 23 Jul 2019
Decentralized Deep Learning with Arbitrary Communication Compression Anastasia Koloskova Tao R. Lin Sebastian U. Stich Martin Jaggi FedML 15 232 0 22 Jul 2019
Collaborative Machine Learning at the Wireless Edge with Blind Transmitters M. Amiri T. Duman Deniz Gunduz 15 54 0 08 Jul 2019
On the Convergence of FedAvg on Non-IID Data Xiang Li Kaixuan Huang Wenhao Yang Shusen Wang Zhihua Zhang FedML 15 2,275 0 04 Jul 2019
An Accelerated Decentralized Stochastic Proximal Algorithm for Finite Sums Hadrien Hendrikx Francis R. Bach Laurent Massoulie 8 31 0 27 May 2019
Decentralized Bayesian Learning over Graphs Anusha Lalitha Xinghan Wang O. Kilinc Y. Lu T. Javidi F. Koushanfar FedML 28 25 0 24 May 2019
On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimization Hao Yu R. L. Jin 10 50 0 10 May 2019