COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Training

30 November 2022

Papers citing "COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Training"

5 / 5 papers shown

Title
ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale William Won Taekyung Heo Saeed Rashidi Srinivas Sridharan S. Srinivasan T. Krishna 36 43 0 24 Mar 2023
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining Junyang Lin An Yang Jinze Bai Chang Zhou Le Jiang ... Jie M. Zhang Yong Li Wei Lin Jingren Zhou Hongxia Yang MoE 84 43 0 08 Oct 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 160 413 0 18 Jan 2021
Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems Maxim Naumov John Kim Dheevatsa Mudigere Srinivas Sridharan Xiaodong Wang ... Krishnakumar Nair Isabel Gao Bor-Yiing Su Jiyan Yang M. Smelyanskiy GNN 30 94 0 20 Mar 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,815 0 17 Sep 2019