Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed
Deep Learning

Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

28 January 2022

Lianmin Zheng

Zhuohan Li

Hao Zhang

Yonghao Zhuang

Zhifeng Chen

Yanping Huang

Yida Wang

Yuanzhong Xu

Danyang Zhuo

Eric P. Xing

Joseph E. Gonzalez

Ion Stoica

Papers citing "Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning"

17 / 17 papers shown

Title
Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers Siddharth Singh Prajwal Singhania Aditya K. Ranjan John Kirchenbauer Jonas Geiping ... Abhimanyu Hans Manli Shu Aditya Tomar Tom Goldstein A. Bhatele 94 2 0 12 Feb 2025
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 116 0 0 08 Jan 2025
FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines Jiaao He Jidong Zhai 32 27 0 18 Mar 2024
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism Yanxi Chen Xuchen Pan Yaliang Li Bolin Ding Jingren Zhou LRM 28 31 0 08 Dec 2023
Moirai: Towards Optimal Placement for Distributed Inference on Heterogeneous Devices Beibei Zhang Hongwei Zhu Feng Gao Zhihui Yang Xiaoyang Sean Wang 14 1 0 07 Dec 2023
UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming Hao Lin Ke Wu Jie Li Jun Yu Li Wu-Jun Li 26 1 0 31 Jul 2023
Scheduling Multi-Server Jobs with Sublinear Regrets via Online Learning Hailiang Zhao Shuiguang Deng Zhengzhe Xiang Xueqiang Yan Jianwei Yin Schahram Dustdar Albert Y. Zomaya 20 1 0 11 May 2023
Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent Xiaonan Nie Yi Liu Fangcheng Fu J. Xue Dian Jiao Xupeng Miao Yangyu Tao Bin Cui MoE 19 16 0 06 Mar 2023
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient Max Ryabinin Tim Dettmers Michael Diskin Alexander Borzunov MoE 22 31 0 27 Jan 2023
Does compressing activations help model parallel training? S. Bian Dacheng Li Hongyi Wang Eric P. Xing Shivaram Venkataraman 19 4 0 06 Jan 2023
Efficiently Scaling Transformer Inference Reiner Pope Sholto Douglas Aakanksha Chowdhery Jacob Devlin James Bradbury Anselm Levskaya Jonathan Heek Kefan Xiao Shivani Agrawal J. Dean 21 292 0 09 Nov 2022
MSRL: Distributed Reinforcement Learning with Dataflow Fragments Huanzhou Zhu Bo Zhao Gang Chen Weifeng Chen Yijie Chen Liang Shi Yaodong Yang Peter R. Pietzuch Lei Chen OffRL MoE 11 6 0 03 Oct 2022
Efficient Quantized Sparse Matrix Operations on Tensor Cores Shigang Li Kazuki Osawa Torsten Hoefler 72 31 0 14 Sep 2022
Varuna: Scalable, Low-cost Training of Massive Deep Learning Models Sanjith Athlur Nitika Saran Muthian Sivathanu R. Ramjee Nipun Kwatra GNN 31 80 0 07 Nov 2021
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training Yongbin Li Hongxin Liu Zhengda Bian Boxiang Wang Haichen Huang Fan Cui Chuan-Qing Wang Yang You GNN 9 143 0 28 Oct 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training Jie Ren Samyam Rajbhandari Reza Yazdani Aminabadi Olatunji Ruwase Shuangyang Yang Minjia Zhang Dong Li Yuxiong He MoE 160 413 0 18 Jan 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,817 0 17 Sep 2019