Breaking (Global) Barriers in Parallel Stochastic Optimization with
Wait-Avoiding Group Averaging

Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging

30 April 2020

Giorgi Nadiradze

Salvatore Di Girolamo

Dan Alistarh

Torsten Hoefler

Papers citing "Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging"

5 / 5 papers shown

Title
DropCompute: simple and more robust distributed synchronous training via compute variance reduction Niv Giladi Shahar Gottlieb Moran Shkolnik A. Karnieli Ron Banner Elad Hoffer Kfir Y. Levy Daniel Soudry 25 2 0 18 Jun 2023
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient Max Ryabinin Tim Dettmers Michael Diskin Alexander Borzunov MoE 22 31 0 27 Jan 2023
HammingMesh: A Network Topology for Large-Scale Deep Learning Torsten Hoefler Tommaso Bonato Daniele De Sensi Salvatore Di Girolamo Shigang Li Marco Heddes Jon Belk Deepak Goel Miguel Castro Steve Scott 3DH GNN AI4CE 18 20 0 03 Sep 2022
Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines Shigang Li Torsten Hoefler GNN AI4CE LRM 77 131 0 14 Jul 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 228 4,460 0 23 Jan 2020