LocMoE: A Low-Overhead MoE for Large Language Model Training

LocMoE: A Low-Overhead MoE for Large Language Model Training

25 January 2024

Papers citing "LocMoE: A Low-Overhead MoE for Large Language Model Training"

10 / 10 papers shown

Title
Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs Yehui Tang Yichun Yin Yaoyuan Wang Hang Zhou Yu Pan ... Zhe Liu Zhicheng Liu Z. Tu Zilin Ding Zongyuan Zhan MoE 28 0 0 07 May 2025
Plasticity-Aware Mixture of Experts for Learning Under QoE Shifts in Adaptive Video Streaming Zhiqiang He Zhi Liu 36 0 0 14 Apr 2025
Theory on Mixture-of-Experts in Continual Learning Hongbo Li Sen-Fon Lin Lingjie Duan Yingbin Liang Ness B. Shroff MoE MoMe CLL 135 14 0 20 Feb 2025
MoDEM: Mixture of Domain Expert Models Toby Simonds K. K. Jey Han Lau MoE 26 1 0 09 Oct 2024
On-Device Language Models: A Comprehensive Review Jiajun Xu Zhiyuan Li Wei Chen Qun Wang Xin Gao Qi Cai Ziyuan Ling 32 24 0 26 Aug 2024
Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models with Adaptive Expert Placement Yongji Wu Wenjie Qu Tianyang Tao Zhuang Wang Wei Bai Zhuohao Li Yuan Tian Jiaheng Zhang Matthew Lentz Danyang Zhuo 42 3 0 05 Jul 2024
Model Compression and Efficient Inference for Large Language Models: A Survey Wenxiao Wang Wei Chen Yicong Luo Yongliu Long Zhengkai Lin Liye Zhang Binbin Lin Deng Cai Xiaofei He MQ 36 30 0 15 Feb 2024
PanGu- $π$ : Enhancing Language Model Architectures via Nonlinearity Compensation Yunhe Wang Hanting Chen Yehui Tang Tianyu Guo Kai Han ... Qinghua Xu Qun Liu Jun Yao Chao Xu Dacheng Tao 59 15 0 27 Dec 2023
PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing Xiaozhe Ren Pingyi Zhou Xinfan Meng Xinjing Huang Yadao Wang ... Jiansheng Wei Xin Jiang Teng Su Qun Liu Jun Yao ALM MoE 61 59 0 20 Mar 2023
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference Sneha Kudugunta Yanping Huang Ankur Bapna M. Krikun Dmitry Lepikhin Minh-Thang Luong Orhan Firat MoE 119 104 0 24 Sep 2021