Title
Automated Tensor Model Parallelism with Overlapped Communication for Efficient Foundation Model Training Shengwei Li Zhiquan Lai Yanqi Hao Weijie Liu Ke-shi Ge Xiaoge Deng Dongsheng Li KaiCheng Lu 11 10 0 25 May 2023
ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation Han Zhang Weichong Yin Yewei Fang Lanxin Li Boqiang Duan Zhihua Wu Yu Sun Hao Tian Hua-Hong Wu Haifeng Wang 27 58 0 31 Dec 2021
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation Shuohuan Wang Yu Sun Yang Xiang Zhihua Wu Siyu Ding ... Tian Wu Wei Zeng Ge Li Wen Gao Haifeng Wang ELM 33 79 0 23 Dec 2021
Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems Weijie Zhao Deping Xie Ronglai Jia Yulei Qian Rui Ding Mingming Sun P. Li MoE 57 150 0 12 Mar 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,817 0 17 Sep 2019