HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU
Clusters through Integration of Pipelined Model Parallelism and Data
Parallelism

HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism

28 May 2020

Papers citing "HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism"

12 / 12 papers shown

Title
Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey Zhihong Liu Xin Xu Peng Qiao Dongsheng Li OffRL 20 2 0 08 Nov 2024
Accelerating Heterogeneous Tensor Parallelism via Flexible Workload Control Zhigang Wang Xu Zhang Ning Wang Chuanfei Xu Jie Nie Zhiqiang Wei Yu Gu Ge Yu 11 0 0 21 Jan 2024
HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis Shiwei Zhang Lansong Diao Chuan Wu Zongyan Cao Siyu Wang Wei Lin 35 12 0 11 Jan 2024
Demand Layering for Real-Time DNN Inference with Minimized Memory Usage Min-Zhi Ji Saehanseul Yi Chang-Mo Koo Sol Ahn Dongjoo Seo N. Dutt Jong-Chan Kim 40 16 0 08 Oct 2022
Decentralized Training of Foundation Models in Heterogeneous Environments Binhang Yuan Yongjun He Jared Davis Tianyi Zhang Tri Dao Beidi Chen Percy Liang Christopher Ré Ce Zhang 20 90 0 02 Jun 2022
Efficient Pipeline Planning for Expedited Distributed DNN Training Ziyue Luo Xiaodong Yi Guoping Long Shiqing Fan Chuan Wu Jun Yang Wei Lin 20 16 0 22 Apr 2022
Enabling All In-Edge Deep Learning: A Literature Review Praveen Joshi Mohammed Hasanuzzaman Chandra Thapa Haithem Afli T. Scully 21 22 0 07 Apr 2022
Survey on Large Scale Neural Network Training Julia Gusak Daria Cherniuk Alena Shilova A. Katrutsa Daniel Bershatsky ... Lionel Eyraud-Dubois Oleg Shlyazhko Denis Dimitrov Ivan V. Oseledets Olivier Beaumont 22 10 0 21 Feb 2022
HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments Ji Liu Zhihua Wu Dianhai Yu Yanjun Ma Danlei Feng Minxu Zhang Xinxuan Wu Xuefeng Yao Dejing Dou 11 43 0 20 Nov 2021
Towards Efficient Post-training Quantization of Pre-trained Language Models Haoli Bai Lu Hou Lifeng Shang Xin Jiang Irwin King M. Lyu MQ 71 47 0 30 Sep 2021
PanGu- $α$ : Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation Wei Zeng Xiaozhe Ren Teng Su Hui Wang Yi-Lun Liao ... Gaojun Fan Yaowei Wang Xuefeng Jin Qun Liu Yonghong Tian ALM MoE AI4CE 27 212 0 26 Apr 2021
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Deepak Narayanan M. Shoeybi Jared Casper P. LeGresley M. Patwary ... Prethvi Kashinkunti J. Bernauer Bryan Catanzaro Amar Phanishayee Matei A. Zaharia MoE 11 643 0 09 Apr 2021