v1v2v3 (latest)

Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model

28 June 2024

Longrong Yang

Dong Shen

Chaoxiang Cai

Fan Yang

Papers citing "Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model"

50 / 86 papers shown

Title
Dynamic-DINO: Fine-Grained Mixture of Experts Tuning for Real-time Open-Vocabulary Object Detection Yehao Lu Minghe Weng Zekang Xiao Rui Jiang Wei Su Guangcong Zheng Ping Lu Xi Li MoE ObjD 44 0 0 23 Jul 2025
Ensembles of Low-Rank Expert Adapters Yinghao Li Vianne Gao Chao Zhang MohamadAli Torkamani 241 1 0 31 Jan 2025
MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning Suning Huang Zheyu Zhang Tianhai Liang Yihan Xu Zhehao Kou Chenhao Lu Guowei Xu Zhengrong Xue Huazhe Xu MoE 229 13 0 19 Oct 2024
MoExtend: Tuning New Experts for Modality and Task Extension Shanshan Zhong Shanghua Gao Zhongzhan Huang Wushao Wen Marinka Zitnik Pan Zhou VLM MLLM MoE 155 9 0 07 Aug 2024
Mixture of Nested Experts: Adaptive Processing of Visual Tokens Gagan Jain Nidhi Hegde Aditya Kusupati Arsha Nagrani Shyamal Buch Prateek Jain Anurag Arnab Sujoy Paul MoE 167 14 0 29 Jul 2024
Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters Yuhang Zhou Zihua Zhao Haolin Li Siyuan Du Jiangchao Yao Ya Zhang Yanfeng Wang MoMe MoE 129 5 0 14 Jun 2024
Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models Yongxin Guo Zhenglin Cheng Xiaoying Tang Tao R. Lin Tao Lin MoE 271 19 0 23 May 2024
Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts Yunxin Li Shenyuan Jiang Baotian Hu Longyue Wang Wanqi Zhong Tong Lu Lin Ma Min Zhang MoE 135 66 0 18 May 2024
CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts Jiachen Li Xinyao Wang Sijie Zhu Chia-Wen Kuo Lu Xu Fan Chen Jitesh Jain Humphrey Shi Longyin Wen MLLM MoE 118 47 0 09 May 2024
AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts Zefang Liu Jiahua Luo MoE KELM 169 19 0 01 May 2024
Mixture of LoRA Experts Xun Wu Shaohan Huang Furu Wei MoMe 159 101 0 21 Apr 2024
MoAI: Mixture of All Intelligence for Large Language and Vision Models Byung-Kwan Lee Beomchan Park Chae Won Kim Yonghyun Ro MLLM VLM 202 27 0 12 Mar 2024
Scaling physics-informed hard constraints with mixture-of-experts N. Chalapathi Yiheng Du Aditi Krishnapriyan AI4CE 145 22 0 20 Feb 2024
LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs Shaoxiang Chen Zequn Jie Lin Ma MoE 228 75 0 29 Jan 2024
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models Bin Lin Zhenyu Tang Yang Ye Jiaxi Cui Bin Zhu ... Jinfa Huang Junwu Zhang Yatian Pang Munan Ning Li-ming Yuan VLM MLLM MoE 209 222 0 29 Jan 2024
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models Damai Dai Chengqi Deng Chenggang Zhao R. X. Xu Huazuo Gao ... Panpan Huang Fuli Luo Chong Ruan Zhifang Sui W. Liang MoE 170 462 0 11 Jan 2024
Mixtral of Experts Albert Q. Jiang Alexandre Sablayrolles Antoine Roux A. Mensch Blanche Savary ... Théophile Gervet Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LLMAG 335 1,336 0 08 Jan 2024
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Zhe Chen Jiannan Wu Wenhai Wang Weijie Su Guo Chen ... Bin Li Ping Luo Tong Lu Yu Qiao Jifeng Dai VLM MLLM 435 1,695 0 21 Dec 2023
Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning Yunhao Gou Zhili Liu Kai Chen Lanqing Hong Hang Xu Aoxue Li Dit-Yan Yeung James T. Kwok Yu Zhang MoE MLLM VLM 264 85 0 19 Dec 2023
Honeybee: Locality-enhanced Projector for Multimodal LLM Junbum Cha Wooyoung Kang Jonghwan Mun Byungseok Roh MLLM 185 176 0 11 Dec 2023
ShareGPT4V: Improving Large Multi-Modal Models with Better Captions Lin Chen Jinsong Li Xiao-wen Dong Pan Zhang Conghui He Yuan Liu Feng Zhao Dahua Lin MLLM VLM 263 826 0 21 Nov 2023
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection Bin Lin Yang Ye Bin Zhu Jiaxi Cui Munan Ning Peng Jin Li-ming Yuan VLM MLLM 523 941 0 16 Nov 2023
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning Jun Chen Deyao Zhu Xiaoqian Shen Xiang Li Zechun Liu Pengchuan Zhang Raghuraman Krishnamoorthi Vikas Chandra Yunyang Xiong Mohamed Elhoseiny MLLM 596 546 0 14 Oct 2023
Improved Baselines with Visual Instruction Tuning Haotian Liu Chunyuan Li Yuheng Li Yong Jae Lee VLM MLLM 294 3,507 0 05 Oct 2023
Qwen Technical Report Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang ... Zhenru Zhang Chang Zhou Jingren Zhou Xiaohuan Zhou Tianhang Zhu OSLM 489 2,429 0 28 Sep 2023
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition Pan Zhang Xiaoyi Wang Bin Wang Yuhang Cao Chao Xu ... Conghui He Xingcheng Zhang Yu Qiao Da Lin Yuan Liu MLLM 360 271 0 26 Sep 2023
Baichuan 2: Open Large-scale Language Models Ai Ming Yang Bin Xiao Bingning Wang Borong Zhang Ce Bian ... Youxin Jiang Yuchen Gao Yupeng Zhang Guosheng Dong Zhiying Wu ELM LRM 554 822 0 19 Sep 2023
Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning Ted Zadouri Ahmet Üstün Arash Ahmadian Beyza Ermics Acyr Locatelli Sara Hooker MoE 152 118 0 11 Sep 2023
MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval Zijun Long George Killick R. McCreadie Gerardo Aragon Camarasa 134 6 0 04 Sep 2023
Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection Longrong Yang Xianpan Zhou Xuewei Li Liang Qiao Zheyang Li Zi-Liang Yang Gaoang Wang Xi Li 156 31 0 28 Aug 2023
Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training Xiao-Ming Wu Dian Zheng Zuhao Liu Weishi Zheng MQ 172 20 0 13 Aug 2023
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities Weihao Yu Zhengyuan Yang Linjie Li Jianfeng Wang Kevin Qinghong Lin Zicheng Liu Xinchao Wang Lijuan Wang MLLM 266 860 0 04 Aug 2023
From Sparse to Soft Mixtures of Experts J. Puigcerver C. Riquelme Basil Mustafa N. Houlsby MoE 331 155 0 02 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 872 13,566 0 18 Jul 2023
MMBench: Is Your Multi-modal Model an All-around Player? Yuanzhan Liu Haodong Duan Yuanhan Zhang Yue Liu Songyang Zhang ... Yuan Liu Conghui He Ziwei Liu Kai-xiang Chen Dahua Lin 276 1,332 0 12 Jul 2023
SVIT: Scaling up Visual Instruction Tuning Bo Zhao Boya Wu Muyang He Tiejun Huang MLLM 165 149 0 09 Jul 2023
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding Yanzhe Zhang Ruiyi Zhang Jiuxiang Gu Jiuxiang Gu Nedim Lipka Diyi Yang Tongfei Sun VLM MLLM 180 266 0 29 Jun 2023
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models Chaoyou Fu Peixian Chen Chunjiang Ge Yulei Qin Mengdan Zhang ... Xiawu Zheng Ke Li Xing Sun Zhenyu Qiu Rongrong Ji ELM MLLM 401 1,034 0 23 Jun 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 1.3K 5,437 0 09 Jun 2023
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only Guilherme Penedo Quentin Malartic Daniel Hesslow Ruxandra-Aimée Cojocaru Alessandro Cappelli Hamza Alobeidli B. Pannier Ebtesam Almazrouei Julien Launay 300 823 0 01 Jun 2023
PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts Yunshui Li Binyuan Hui Zhichao Yin Min Yang Fei Huang Yongbin Li MoE 141 22 0 24 May 2023
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models Sheng Shen Le Hou Yan-Quan Zhou Nan Du Shayne Longpre ... Vincent Zhao Hongkun Yu Kurt Keutzer Trevor Darrell Denny Zhou ALM MoE 221 68 0 24 May 2023
Lifelong Language Pretraining with Distribution-Specialized Experts Wuyang Chen Yan-Quan Zhou Nan Du Yanping Huang James Laudon Zhiwen Chen Claire Cu KELM 139 64 0 20 May 2023
Evaluating Object Hallucination in Large Vision-Language Models Yifan Li Yifan Du Kun Zhou Jinpeng Wang Wayne Xin Zhao Ji-Rong Wen MLLM LRM 473 1,026 0 17 May 2023
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality Qinghao Ye Haiyang Xu Guohai Xu Jiabo Ye Ming Yan ... Junfeng Tian Qiang Qi Ji Zhang Feiyan Huang Jingren Zhou VLM MLLM 450 1,054 0 27 Apr 2023
CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with Mixture-of-Textual-Experts for Passage Retrieval Guangyuan Ma Xing Wu Peng Wang Songlin Hu MoE RALM 108 6 0 20 Apr 2023
Visual Instruction Tuning Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee SyDa VLM MLLM 656 6,090 0 17 Apr 2023
Scaling Vision-Language Models with Sparse Mixture of Experts Sheng Shen Z. Yao Chunyuan Li Trevor Darrell Kurt Keutzer Yuxiong He VLM MoE 115 86 0 13 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 1.9K 15,461 0 27 Feb 2023
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints Aran Komatsuzaki J. Puigcerver James Lee-Thorp Carlos Riquelme Ruiz Basil Mustafa Joshua Ainslie Yi Tay Mostafa Dehghani N. Houlsby MoMe MoE 158 144 0 09 Dec 2022