Don't Use Large Mini-Batches, Use Local SGD

22 August 2018

Papers citing "Don't Use Large Mini-Batches, Use Local SGD"

50 / 271 papers shown

Title
FedShuffle: Recipes for Better Use of Local Work in Federated Learning Samuel Horváth Maziar Sanjabi Lin Xiao Peter Richtárik Michael G. Rabbat FedML 25 21 0 27 Apr 2022
Federated Learning via Inexact ADMM Shenglong Zhou Geoffrey Ye Li FedML 19 58 0 22 Apr 2022
FedCos: A Scene-adaptive Federated Optimization Enhancement for Performance Improvement Hao Zhang Tingting Wu Siyao Cheng Jie Liu FedML 30 11 0 07 Apr 2022
Sparse Federated Learning with Hierarchical Personalized Models Xiaofeng Liu Qing Wang Yunfeng Shao Yinchuan Li FedML 36 11 0 25 Mar 2022
The Role of Local Steps in Local SGD Tiancheng Qin S. Rasoul Etesami César A. Uribe 8 4 0 14 Mar 2022
Scaling the Wild: Decentralizing Hogwild!-style Shared-memory SGD Bapi Chatterjee Vyacheslav Kungurtsev Dan Alistarh FedML 14 2 0 13 Mar 2022
ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally! Konstantin Mishchenko Grigory Malinovsky Sebastian U. Stich Peter Richtárik 11 148 0 18 Feb 2022
Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam Yucheng Lu Conglong Li Minjia Zhang Christopher De Sa Yuxiong He OffRL AI4CE 22 20 0 12 Feb 2022
Partial Model Averaging in Federated Learning: Performance Guarantees and Benefits Sunwoo Lee Anit Kumar Sahu Chaoyang He Salman Avestimehr FedML 17 16 0 11 Jan 2022
AET-SGD: Asynchronous Event-triggered Stochastic Gradient Descent Nhuong V. Nguyen Song Han 21 2 0 27 Dec 2021
Towards Federated Learning on Time-Evolving Heterogeneous Data Yongxin Guo Tao R. Lin Xiaoying Tang FedML 14 30 0 25 Dec 2021
Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient Xiaoge Deng Dongsheng Li Tao Sun Xicheng Lu FedML 11 0 0 08 Dec 2021
Collaborative Learning over Wireless Networks: An Introductory Overview Emre Ozfatura Deniz Gunduz H. Vincent Poor 22 11 0 07 Dec 2021
Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent Wei Zhang Mingrui Liu Yu Feng Xiaodong Cui Brian Kingsbury Yuhai Tu 6 3 0 02 Dec 2021
HADFL: Heterogeneity-aware Decentralized Federated Learning Framework Jing Cao Zirui Lian Weihong Liu Zongwei Zhu Cheng Ji FedML 14 18 0 16 Nov 2021
Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters Xiangru Lian Binhang Yuan Xuefeng Zhu Yulong Wang Yongjun He ... Lei Yuan Hai-bo Yu Sen Yang Ce Zhang Ji Liu VLM 19 34 0 10 Nov 2021
Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective Margalit Glasgow Honglin Yuan Tengyu Ma FedML 25 42 0 05 Nov 2021
Large-Scale Deep Learning Optimizations: A Comprehensive Survey Xiaoxin He Fuzhao Xue Xiaozhe Ren Yang You 22 14 0 01 Nov 2021
Communication-Efficient ADMM-based Federated Learning Shenglong Zhou Geoffrey Ye Li FedML 35 22 0 28 Oct 2021
FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion Sijie Cheng Jingwen Wu Yanghua Xiao Yang Liu Yang Liu FedML 10 67 0 21 Oct 2021
Trade-offs of Local SGD at Scale: An Empirical Study Jose Javier Gonzalez Ortiz Jonathan Frankle Michael G. Rabbat Ari S. Morcos Nicolas Ballas FedML 20 19 0 15 Oct 2021
KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Tasks Rishabh Bhardwaj Tushar Vaidya Soujanya Poria OT FedML 57 7 0 06 Oct 2021
Accelerate Distributed Stochastic Descent for Nonconvex Optimization with Momentum Guojing Cong Tianyi Liu 16 0 0 01 Oct 2021
Communication Efficient Generalized Tensor Factorization for Decentralized Healthcare Networks Jing Ma Qiuchen Zhang Jian Lou Li Xiong S. Bhavani Joyce C. Ho 18 0 0 03 Sep 2021
Statistical Estimation and Inference via Local SGD in Federated Learning Xiang Li Jiadong Liang Xiangyu Chang Zhihua Zhang FedML 22 4 0 03 Sep 2021
FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learning Charlie Hou K. K. Thekumparampil Giulia Fanti Sewoong Oh FedML 30 14 0 16 Aug 2021
RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data Guang Yang Ke Mu Chunhe Song Zhijia Yang Tierui Gong FedML 8 15 0 19 Jul 2021
A Field Guide to Federated Optimization Jianyu Wang Zachary B. Charles Zheng Xu Gauri Joshi H. B. McMahan ... Mi Zhang Tong Zhang Chunxiang Zheng Chen Zhu Wennan Zhu FedML 175 411 0 14 Jul 2021
Sparse Personalized Federated Learning Xiaofeng Liu Yinchuan Li Qing Wang Xu Zhang Yunfeng Shao Yanhui Geng FedML 31 6 0 12 Jul 2021
BAGUA: Scaling up Distributed Learning with System Relaxations Shaoduo Gan Xiangru Lian Rui Wang Jianbin Chang Chengjun Liu ... Jiawei Jiang Binhang Yuan Sen Yang Ji Liu Ce Zhang 23 30 0 03 Jul 2021
ResIST: Layer-Wise Decomposition of ResNets for Distributed Training Chen Dun Cameron R. Wolfe C. Jermaine Anastasios Kyrillidis 16 21 0 02 Jul 2021
Personalized Federated Learning with Gaussian Processes Idan Achituve Aviv Shamsian Aviv Navon Gal Chechik Ethan Fetaya FedML 21 98 0 29 Jun 2021
Implicit Gradient Alignment in Distributed and Federated Learning Yatin Dandi Luis Barba Martin Jaggi FedML 18 31 0 25 Jun 2021
Behavior Mimics Distribution: Combining Individual and Group Behaviors for Federated Learning Hua Huang Fanhua Shang Yuanyuan Liu Hongying Liu FedML 16 14 0 23 Jun 2021
CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation Enda Yu Dezun Dong Yemao Xu Shuo Ouyang Xiangke Liao 6 5 0 21 Jun 2021
STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learning Prashant Khanduri Pranay Sharma Haibo Yang Min-Fong Hong Jia Liu K. Rajawat P. Varshney FedML 17 63 0 19 Jun 2021
Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Optimization Luofeng Liao Li Shen Jia Duan Mladen Kolar Dacheng Tao 11 4 0 18 Jun 2021
Optimality and Stability in Federated Learning: A Game-theoretic Approach Kate Donahue Jon M. Kleinberg FedML 11 45 0 17 Jun 2021
Towards Heterogeneous Clients with Elastic Federated Learning Zichen Ma Yu Lu Zihan Lu Wenye Li Jinfeng Yi Shuguang Cui FedML 11 3 0 17 Jun 2021
On Large-Cohort Training for Federated Learning Zachary B. Charles Zachary Garrett Zhouyuan Huo Sergei Shmulyian Virginia Smith FedML 19 112 0 15 Jun 2021
CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning Haibo Yang Jia Liu Elizabeth S. Bentley FedML 13 16 0 14 Jun 2021
Federated Learning with Buffered Asynchronous Aggregation John Nguyen Kshitiz Malik Hongyuan Zhan Ashkan Yousefpour Michael G. Rabbat Mani Malek Dzmitry Huba FedML 13 288 0 11 Jun 2021
Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning Bokun Wang Zhuoning Yuan Yiming Ying Tianbao Yang FedML 40 9 0 09 Jun 2021
Communication-efficient SGD: From Local SGD to One-Shot Averaging Artin Spiridonoff Alexander Olshevsky I. Paschalidis FedML 21 20 0 09 Jun 2021
SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networks Chaoyang He Emir Ceyani Keshav Balasubramanian M. Annavaram Salman Avestimehr FedML 17 50 0 04 Jun 2021
On Linear Stability of SGD and Input-Smoothness of Neural Networks Chao Ma Lexing Ying MLT 9 44 0 27 May 2021
Fast Federated Learning by Balancing Communication Trade-Offs M. Nori Sangseok Yun Il Kim FedML 18 53 0 23 May 2021
Accelerating Gossip SGD with Periodic Global Averaging Yiming Chen Kun Yuan Yingya Zhang Pan Pan Yinghui Xu W. Yin 15 41 0 19 May 2021
Towards Demystifying Serverless Machine Learning Training Jiawei Jiang Shaoduo Gan Yue Liu Fanlin Wang Gustavo Alonso Ana Klimovic Ankit Singla Wentao Wu Ce Zhang 19 121 0 17 May 2021
LocalNewton: Reducing Communication Bottleneck for Distributed Learning Vipul Gupta Avishek Ghosh Michal Derezinski Rajiv Khanna K. Ramchandran Michael W. Mahoney 30 12 0 16 May 2021