Statistical Advantages of Perturbing Cosine Router in Mixture of Experts

23 May 2024

Papers citing "Statistical Advantages of Perturbing Cosine Router in Mixture of Experts"

5 / 5 papers shown

Title
A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications Siyuan Mu Sen Lin MoE 69 1 0 10 Mar 2025
On Least Square Estimation in Softmax Gating Mixture of Experts Huy Nguyen Nhat Ho Alessandro Rinaldo 28 13 0 05 Feb 2024
FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion Xing Han Huy Nguyen Carl Harris Nhat Ho S. Saria MoE 39 16 0 05 Feb 2024
From Sparse to Soft Mixtures of Experts J. Puigcerver C. Riquelme Basil Mustafa N. Houlsby MoE 114 114 0 02 Aug 2023
A Fine-Grained Analysis on Distribution Shift Olivia Wiles Sven Gowal Florian Stimberg Sylvestre-Alvise Rebuffi Ira Ktena Krishnamurthy Dvijotham A. Cemgil OOD 215 196 0 21 Oct 2021