Parm: Efficient Training of Large Sparsely-Activated Models with
Dedicated Schedules

Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules

30 June 2024

Wenxiang Lin

Xiaowen Chu

Bo Li

Papers citing "Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules"

6 / 6 papers shown

Title
Accelerating Mixture-of-Experts Training with Adaptive Expert Replication Athinagoras Skiadopoulos Mark Zhao Swapnil Gandhi Thomas Norrie Shrijeet Mukherjee Christos Kozyrakis MoE 91 0 0 28 Apr 2025
Importance Sampling via Score-based Generative Models Heasung Kim Taekyun Lee Hyeji Kim Gustavo de Veciana MedIm DiffM 129 0 0 07 Feb 2025
PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing Xiaozhe Ren Pingyi Zhou Xinfan Meng Xinjing Huang Yadao Wang ... Jiansheng Wei Xin Jiang Teng Su Qun Liu Jun Yao ALM MoE 67 60 0 20 Mar 2023
Tutel: Adaptive Mixture-of-Experts at Scale Changho Hwang Wei Cui Yifan Xiong Ziyue Yang Ze Liu ... Joe Chau Peng Cheng Fan Yang Mao Yang Y. Xiong MoE 92 109 0 07 Jun 2022
Mixture-of-Experts with Expert Choice Routing Yan-Quan Zhou Tao Lei Han-Chu Liu Nan Du Yanping Huang Vincent Zhao Andrew M. Dai Zhifeng Chen Quoc V. Le James Laudon MoE 149 327 0 18 Feb 2022
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,453 0 23 Jan 2020