Towards A Unified View of Sparse Feed-Forward Network in Pretraining
Large Language Model

Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model

23 May 2023

Tim Dettmers

Xi Victoria Lin

Xian Li

Papers citing "Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model"

9 / 9 papers shown

Title
UMoE: Unifying Attention and FFN with Shared Experts Yuanhang Yang Chaozheng Wang Jing Li MoE 14 0 0 12 May 2025
The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts Enric Boix Adserà Philippe Rigollet MoE 11 0 0 11 May 2025
Continual Pre-training of MoEs: How robust is your router? Benjamin Thérien Charles-Étienne Joseph Zain Sarwar Ashwinee Panda Anirban Das Shi-Xiong Zhang Stephen Rawls S. Eugene Belilovsky Irina Rish MoE 68 0 0 06 Mar 2025
Mixture of Experts Made Intrinsically Interpretable Xingyi Yang Constantin Venhoff Ashkan Khakzar Christian Schroeder de Witt P. Dokania Adel Bibi Philip H. S. Torr MoE 49 0 0 05 Mar 2025
Scaling Laws for Fine-Grained Mixture of Experts Jakub Krajewski Jan Ludziejewski Kamil Adamczewski Maciej Pióro Michal Krutul ... Krystian Król Tomasz Odrzygó'zd'z Piotr Sankowski Marek Cygan Sebastian Jaszczur MoE 40 53 0 12 Feb 2024
A Survey on Transformer Compression Yehui Tang Yunhe Wang Jianyuan Guo Zhijun Tu Kai Han Hailin Hu Dacheng Tao 29 27 0 05 Feb 2024
Mixture-of-Experts with Expert Choice Routing Yan-Quan Zhou Tao Lei Han-Chu Liu Nan Du Yanping Huang Vincent Zhao Andrew M. Dai Zhifeng Chen Quoc V. Le James Laudon MoE 147 326 0 18 Feb 2022
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 245 1,986 0 31 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,424 0 23 Jan 2020