Scattered Mixture-of-Experts Implementation

Scattered Mixture-of-Experts Implementation

13 March 2024

Aaron Courville

Papers citing "Scattered Mixture-of-Experts Implementation"

5 / 5 papers shown

Title
Mixture of Parrots: Experts improve memorization more than reasoning Samy Jelassi Clara Mohri David Brandfonbrener Alex Gu Nikhil Vyas Nikhil Anand David Alvarez-Melis Yuanzhi Li Sham Kakade Eran Malach MoE 28 4 0 24 Oct 2024
EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference Yulei Qian Fengcun Li Xiangyang Ji Xiaoyu Zhao Jianchao Tan K. Zhang Xunliang Cai MoE 68 3 0 16 Oct 2024
Scaling Laws for Fine-Grained Mixture of Experts Jakub Krajewski Jan Ludziejewski Kamil Adamczewski Maciej Pióro Michal Krutul ... Krystian Król Tomasz Odrzygó'zd'z Piotr Sankowski Marek Cygan Sebastian Jaszczur MoE 40 53 0 12 Feb 2024
Mixture of Attention Heads: Selecting Attention Heads Per Token Xiaofeng Zhang Yikang Shen Zeyu Huang Jie Zhou Wenge Rong Zhang Xiong MoE 99 42 0 11 Oct 2022
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,817 0 17 Sep 2019