HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis

11 January 2024

Papers citing "HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis"

7 / 7 papers shown

Title
Nonuniform-Tensor-Parallelism: Mitigating GPU failure impact for Scaled-up LLM Training Daiyaan Arfeen Dheevatsa Mudigere Ankit More Bhargava Gopireddy Ahmet Inci G. R. Ganger 23 0 0 08 Apr 2025
HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs Yongji Wu Xueshen Liu Shuowei Jin Ceyu Xu Feng Qian Ziming Mao Matthew Lentz Danyang Zhuo Ion Stoica MoMe MoE 59 0 0 04 Apr 2025
Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Models Runsheng Benson Guo Utkarsh Anand Arthur Chen Khuzaima Daudjee 34 1 0 01 Nov 2024
Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization Haoyang Li Fangcheng Fu Hao Ge Sheng Lin Xuanyu Wang Jiawen Niu Y. Wang Hailin Zhang Xiaonan Nie Bin Cui MoMe 31 2 0 17 Oct 2024
HybridFlow: A Flexible and Efficient RLHF Framework Guangming Sheng Chi Zhang Zilingfeng Ye Xibin Wu Wang Zhang Ru Zhang Yanghua Peng Haibin Lin Chuan Wu AI4CE 26 66 0 28 Sep 2024
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey Jiangfei Duan Shuo Zhang Zerui Wang Lijuan Jiang Wenwen Qu ... Dahua Lin Yonggang Wen Xin Jin Tianwei Zhang Peng Sun 71 8 0 29 Jul 2024
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,817 0 17 Sep 2019