MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware
Experts

MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts

31 July 2024

Xi Victoria Lin

Akshat Shrivastava

Srinivasan Iyer

Luke Zettlemoyer

Armen Aghajanyan

Papers citing "MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts"

18 / 18 papers shown

Title
Multimodal Long Video Modeling Based on Temporal Dynamic Context Haoran Hao Jiaming Han Yiyuan Zhang Xiangyu Yue 32 0 0 14 Apr 2025
OmniCaptioner: One Captioner to Rule Them All Yiting Lu Jiakang Yuan Zhen Li Shitian Zhao Qi Qin ... Lei Bai Zhibo Chen Peng Gao Bo Zhang Peng Gao MLLM 79 0 0 09 Apr 2025
CoSMoEs: Compact Sparse Mixture of Experts Patrick Huber Akshat Shrivastava Ernie Chang Chinnadhurai Sankar Ahmed Aly Adithya Sagar MoE 24 0 0 28 Feb 2025
R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts Zhongyang Li Ziyue Li Tianyi Zhou MoE 44 0 0 27 Feb 2025
UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths Weijia Mao Z. Yang Mike Zheng Shou MoE 63 0 0 10 Feb 2025
LaB-RAG: Label Boosted Retrieval Augmented Generation for Radiology Report Generation Steven Song Anirudh Subramanyam Irene Madejski Robert L. Grossman MedIm VLM 98 0 0 25 Nov 2024
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization Weiyun Wang Zhe Chen Wenhai Wang Yue Cao Yangzhou Liu ... Jinguo Zhu X. Zhu Lewei Lu Yu Qiao Jifeng Dai LRM 52 45 1 15 Nov 2024
LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models Nam V. Nguyen Thong T. Doan Luong Tran Van Nguyen Quang Pham MoE 55 1 0 01 Nov 2024
Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance Zhangwei Gao Zhe Chen Erfei Cui Yiming Ren Weiyun Wang ... Lewei Lu Tong Lu Yu Qiao Jifeng Dai Wenhai Wang VLM 62 22 0 21 Oct 2024
Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers Yuxin Wen Qingqing Cao Qichen Fu Sachin Mehta Mahyar Najibi VLM 25 4 0 17 Oct 2024
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory Di Wu Hongwei Wang W. Yu Yuwei Zhang Kai-Wei Chang Dong Yu RALM KELM 38 5 0 14 Oct 2024
Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training Gen Luo Xue Yang Wenhan Dou Zhaokai Wang Jifeng Dai Jifeng Dai Yu Qiao Xizhou Zhu VLM MLLM 51 25 0 10 Oct 2024
Aria: An Open Multimodal Native Mixture-of-Experts Model Dongxu Li Yudong Liu Haoning Wu Yue Wang Zhiqi Shen ... Lihuan Zhang Hanshu Yan Guoyin Wang Bei Chen Junnan Li MoE 49 48 0 08 Oct 2024
EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing Haotian Sun Tao Lei Bowen Zhang Yanghao Li Haoshuo Huang Ruoming Pang Bo Dai Nan Du DiffM MoE 73 5 0 02 Oct 2024
Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models Keivan Alizadeh Iman Mirzadeh Hooman Shahrokhi Dmitry Belenko Frank Sun Minsik Cho Mohammad Hossein Sekhavat Moin Nabi Mehrdad Farajtabar MoE 21 1 0 01 Oct 2024
Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers Lirui Wang Xinlei Chen Jialiang Zhao Kaiming He 34 32 0 30 Sep 2024
Mixture-of-Experts with Expert Choice Routing Yan-Quan Zhou Tao Lei Han-Chu Liu Nan Du Yanping Huang Vincent Zhao Andrew M. Dai Zhifeng Chen Quoc V. Le James Laudon MoE 147 323 0 18 Feb 2022
How Does Selective Mechanism Improve Self-Attention Networks? Xinwei Geng Longyue Wang Xing Wang Bing Qin Ting Liu Zhaopeng Tu AAML 26 33 0 03 May 2020