A Review of Sparse Expert Models in Deep Learning

4 September 2022

Papers citing "A Review of Sparse Expert Models in Deep Learning"

35 / 35 papers shown

Title
The Empirical Impact of Reducing Symmetries on the Performance of Deep Ensembles and MoE Andrei Chernov Oleg Novitskij 43 0 0 24 Feb 2025
Position: AI Scaling: From Up to Down and Out Yunke Wang Yanxi Li Chang Xu HAI 76 1 0 02 Feb 2025
DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference Yujie Zhang Shivam Aggarwal T. Mitra MoE 72 0 0 16 Dec 2024
ELICIT: LLM Augmentation via External In-Context Capability Futing Wang Jianhao Yan Yue Zhang Tao Lin 39 0 0 12 Oct 2024
HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models Bingshen Mu Kun Wei Qijie Shao Yong Xu Lei Xie MoE 37 1 0 30 Sep 2024
Erasure Coded Neural Network Inference via Fisher Averaging Divyansh Jhunjhunwala Neharika Jali Gauri Joshi Shiqiang Wang MoMe FedML 26 1 0 02 Sep 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 40 43 0 09 Jul 2024
RouteFinder: Towards Foundation Models for Vehicle Routing Problems Federico Berto Chuanbo Hua Nayeli Gast Zepeda André Hottung N. Wouda Leon Lan Kevin Tierney J. Park Jinkyoo Park 48 10 0 21 Jun 2024
$$\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts$ $\texttt{MoE-RBench}$ : Towards Building Reliable Language Models with Sparse Mixture-of-Experts Guanjie Chen Xinyu Zhao Tianlong Chen Yu Cheng MoE 62 5 0 17 Jun 2024
GLIMPSE: Generalized Local Imaging with MLPs AmirEhsan Khorashadizadeh Valentin Debarnot Tianlin Liu Ivan Dokmanić 28 0 0 01 Jan 2024
Mixture of Weak & Strong Experts on Graphs Hanqing Zeng Hanjia Lyu Diyi Hu Yinglong Xia Jiebo Luo 23 3 0 09 Nov 2023
Mixture-of-Experts for Open Set Domain Adaptation: A Dual-Space Detection Approach Zhenbang Du Jiayu An Yunlu Tu Jiahao Hong Dongrui Wu MoE 15 1 0 01 Nov 2023
Uncovering the Hidden Cost of Model Compression Diganta Misra Muawiz Chaudhary Agam Goyal Bharat Runwal Pin-Yu Chen VLM 24 0 0 29 Aug 2023
Deep Reinforcement Learning with Task-Adaptive Retrieval via Hypernetwork Yonggang Jin Chenxu Wang Tianyu Zheng Liuyu Xiang Yao-Chun Yang Junge Zhang Jie Fu Zhaofeng He 3DH 27 0 0 19 Jun 2023
Vision Transformers for Mobile Applications: A Short Survey Nahid Alam Steven Kolawole S. Sethi Nishant Bansali Karina Nguyen ViT 16 3 0 30 May 2023
Explaining black box text modules in natural language with language models Chandan Singh Aliyah R. Hsu Richard Antonello Shailee Jain Alexander G. Huth Bin-Xia Yu Jianfeng Gao MILM 16 46 0 17 May 2023
Scaling Expert Language Models with Unsupervised Domain Discovery Suchin Gururangan Margaret Li M. Lewis Weijia Shi Tim Althoff Noah A. Smith Luke Zettlemoyer MoE 15 46 0 24 Mar 2023
ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth S. Bhat R. Birkl Diana Wofk Peter Wonka Matthias Müller VLM MDE 27 483 0 23 Feb 2023
Multipath agents for modular multitask ML systems Andrea Gesmundo 18 1 0 06 Feb 2023
Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective Michael E. Sander J. Puigcerver Josip Djolonga Gabriel Peyré Mathieu Blondel 16 18 0 02 Feb 2023
Adaptive Computation with Elastic Input Sequence Fuzhao Xue Valerii Likhosherstov Anurag Arnab N. Houlsby Mostafa Dehghani Yang You 29 18 0 30 Jan 2023
Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model Yeskendir Koishekenov Alexandre Berard Vassilina Nikoulina MoE 22 28 0 19 Dec 2022
HMOE: Hypernetwork-based Mixture of Experts for Domain Generalization Jingang Qu T. Faney Zehao Wang Patrick Gallinari Soleiman Yousef J. D. Hemptinne OOD 16 7 0 15 Nov 2022
Efficiently Scaling Transformer Inference Reiner Pope Sholto Douglas Aakanksha Chowdhery Jacob Devlin James Bradbury Anselm Levskaya Jonathan Heek Kefan Xiao Shivani Agrawal J. Dean 21 292 0 09 Nov 2022
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 28 109 0 31 Aug 2022
Tutel: Adaptive Mixture-of-Experts at Scale Changho Hwang Wei Cui Yifan Xiong Ziyue Yang Ze Liu ... Joe Chau Peng Cheng Fan Yang Mao Yang Y. Xiong MoE 92 108 0 07 Jun 2022
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT James Lee-Thorp Joshua Ainslie MoE 30 11 0 24 May 2022
Mixture-of-Experts with Expert Choice Routing Yan-Quan Zhou Tao Lei Han-Chu Liu Nan Du Yanping Huang Vincent Zhao Andrew M. Dai Zhifeng Chen Quoc V. Le James Laudon MoE 149 326 0 18 Feb 2022
Tricks for Training Sparse Translation Models Dheeru Dua Shruti Bhosale Vedanuj Goswami James Cross M. Lewis Angela Fan MoE 145 19 0 15 Oct 2021
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference Sneha Kudugunta Yanping Huang Ankur Bapna M. Krikun Dmitry Lepikhin Minh-Thang Luong Orhan Firat MoE 119 105 0 24 Sep 2021
Scalable and Efficient MoE Training for Multitask Multilingual Models Young Jin Kim A. A. Awan Alexandre Muzio Andres Felipe Cruz Salinas Liyang Lu Amr Hendy Samyam Rajbhandari Yuxiong He Hany Awadalla MoE 94 84 0 22 Sep 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 239 2,600 0 04 May 2021
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 239 642 0 21 Apr 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 267 1,808 0 14 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 226 4,453 0 23 Jan 2020