LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs

29 January 2024

Papers citing "LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs"

44 / 44 papers shown

Title
Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extraction Li Yuan Yi Cai Xudong Shen Qing Li Qingbao Huang Zikun Deng Tao Wang MoMe OffRL MoE 24 0 0 08 May 2025
LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Experts Yimu Wang Mozhgan Nasr Azadani Sean Sedwards Krzysztof Czarnecki MLLM MoE 47 0 0 07 Apr 2025
MoE-Loco: Mixture of Experts for Multitask Locomotion Runhan Huang Shaoting Zhu Yilun Du Hang Zhao MoE 36 0 0 11 Mar 2025
MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Models H. Zhao Wenxuan Song Donglin Wang Xinyang Tong Pengxiang Ding Xuelian Cheng Zongyuan Ge 49 1 0 11 Mar 2025
A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications Siyuan Mu Sen Lin MoE 69 1 0 10 Mar 2025
CoPL: Collaborative Preference Learning for Personalizing LLMs Youngbin Choi Seunghyuk Cho M. Lee Moonjeong Park Yesong Ko Jungseul Ok Dongwoo Kim 50 0 0 03 Mar 2025
A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models Mengyang Sun Yihao Wang Tao Feng Dan Zhang Yifan Zhu J. Tang MoE 29 0 0 20 Feb 2025
Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment Zhili Liu Yunhao Gou Kai Chen Lanqing Hong Jiahui Gao ... Yu Zhang Zhenguo Li Xin Jiang Q. Liu James T. Kwok MoE 93 9 0 20 Feb 2025
Ensembles of Low-Rank Expert Adapters Yinghao Li Vianne Gao Chao Zhang MohamadAli Torkamani 55 0 0 31 Jan 2025
A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future Shilin Sun Wenbin An Feng Tian Fang Nan Qidong Liu J. Liu N. Shah Ping Chen 68 2 0 18 Dec 2024
MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning Lulu Zhao Weihao Zeng Xiaofeng Shi Hua Zhou MoMe MoE 67 0 0 12 Dec 2024
Visual Cue Enhancement and Dual Low-Rank Adaptation for Efficient Visual Instruction Fine-Tuning Pengkun Jiao Bin Zhu Jingjing Chen Chong-Wah Ngo Yu-Gang Jiang VLM OffRL 61 0 0 19 Nov 2024
AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment Y. Fu Zhongzhi Yu Junwei Li Jiayi Qian Yongan Zhang Xiangchi Yuan Dachuan Shi Roman Yakunin Y. Lin 19 2 0 15 Nov 2024
LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models Nam V. Nguyen Thong T. Doan Luong Tran Van Nguyen Quang Pham MoE 48 1 0 01 Nov 2024
R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest Xupeng Chen Zhixin Lai Kangrui Ruan Shichu Chen Jiaxiang Liu Zuozhu Liu 25 1 0 27 Oct 2024
MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning Jingfan Zhang Yi Zhao Dan Chen Xing Tian Huanran Zheng Wei Zhu MoE 23 12 0 23 Oct 2024
LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound Xuechen Guo Wenhao Chai Shi-Yan Li Gaoang Wang 13 5 0 19 Oct 2024
Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants Rafael Ferreira David Semedo João Magalhães 23 1 0 16 Oct 2024
Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMs Ruijia Niu D. Wu Rose Yu Yi-An Ma 13 1 0 09 Oct 2024
Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE Xun Zhu Ying Hu Fanbin Mo Miao Li Ji Wu 31 8 0 26 Sep 2024
FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Models Xiaochen Wang Jiaqi Wang Houping Xiao J. Chen Fenglong Ma MedIm 58 7 0 17 Aug 2024
A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks Jiaqi Wang Hanqi Jiang Yi-Hsueh Liu Chong Ma Xu-Yao Zhang ... Xin Zhang Wei Zhang Dinggang Shen Tianming Liu Shu Zhang VLM AI4TS 29 18 0 02 Aug 2024
RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models Pengkun Jiao Xinlan Wu Bin Zhu Jingjing Chen Chong-Wah Ngo Yu-Gang Jiang 23 1 0 17 Jul 2024
MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models Leyang Shen Gongwei Chen Rui Shao Weili Guan Liqiang Nie MoE 27 0 0 17 Jul 2024
Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models Jinrui Zhang Teng Wang Haigang Zhang Ping Lu Feng Zheng MLLM LRM VLM 19 3 0 16 Jul 2024
A Survey on LoRA of Large Language Models Yuren Mao Yuhang Ge Yijiang Fan Wenyi Xu Yu Mi Zhonghao Hu Yunjun Gao ALM 52 22 0 08 Jul 2024
Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model Longrong Yang Dong Shen Chaoxiang Cai Fan Yang Size Li Di Zhang Xi Li MoE 37 2 0 28 Jun 2024
Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts Tong Zhu Daize Dong Xiaoye Qu Jiacheng Ruan Wenliang Chen Yu Cheng MoE 24 7 0 17 Jun 2024
MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models Jingwei Xu Junyu Lai Yunpeng Huang MoE MoMe 23 2 0 19 May 2024
MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts Dengchun Li Yingzi Ma Naizheng Wang Zhengmao Ye Zhiyuan Cheng ... Yan Zhang Lei Duan Jie Zuo Cal Yang Mingjie Tang MoE 19 13 0 22 Apr 2024
Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models Songtao Jiang Tuo Zheng Yan Zhang Yeying Jin Li Yuan Zuozhu Liu MoE 24 7 0 16 Apr 2024
Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts Weilin Cai Juyong Jiang Le Qin Junwei Cui Sunghun Kim Jiayi Huang 40 7 0 07 Apr 2024
Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models Yang Jiao Shaoxiang Chen Zequn Jie Jing Chen Lin Ma Yueping Jiang MLLM 21 18 0 12 Mar 2024
SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data Jialu Li Jaemin Cho Yi-Lin Sung Jaehong Yoon Mohit Bansal MoMe DiffM 26 8 0 11 Mar 2024
CoTBal: Comprehensive Task Balancing for Multi-Task Visual Instruction Tuning Yanqi Dai Dong Jing Nanyi Fei Zhiwu Lu Nanyi Fei Guoxing Yang Zhiwu Lu 38 2 0 07 Mar 2024
PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA Sheng Wang Boyang Xue Jiacheng Ye Jiyue Jiang Liheng Chen Lingpeng Kong Chuan Wu 15 2 0 24 Feb 2024
SA-MDKIF: A Scalable and Adaptable Medical Domain Knowledge Injection Framework for Large Language Models Tianhan Xu Zhe Hu LingSen Chen Bin Li LM&MA 8 1 0 01 Feb 2024
MM-LLMs: Recent Advances in MultiModal Large Language Models Duzhen Zhang Yahan Yu Jiahua Dong Chenxing Li Dan Su Chenhui Chu Dong Yu OffRL LRM 34 173 0 24 Jan 2024
Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning Yunhao Gou Zhili Liu Kai Chen Lanqing Hong Hang Xu Aoxue Li Dit-Yan Yeung James T. Kwok Yu Zhang MoE MLLM VLM 29 60 0 19 Dec 2023
Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models Haoran Wei Lingyu Kong Jinyue Chen Liang Zhao Zheng Ge Jinrong Yang Jian‐Yuan Sun Chunrui Han Xiangyu Zhang MLLM VLM 61 39 0 11 Dec 2023
MoSA: Mixture of Sparse Adapters for Visual Efficient Tuning Qizhe Zhang Bocheng Zou Ruichuan An Jiaming Liu Shanghang Zhang MoE 20 2 0 05 Dec 2023
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning Jun Chen Deyao Zhu Xiaoqian Shen Xiang Li Zechun Liu Pengchuan Zhang Raghuraman Krishnamoorthi Vikas Chandra Yunyang Xiong Mohamed Elhoseiny MLLM 152 280 0 14 Oct 2023
UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model Jiabo Ye Anwen Hu Haiyang Xu Qinghao Ye Mingshi Yan ... Ji Zhang Qin Jin Liang He Xin Lin Feiyan Huang VLM MLLM 118 83 0 08 Oct 2023
Pathological Visual Question Answering Xuehai He Zhuo Cai Wenlan Wei Yichen Zhang Luntian Mou Eric P. Xing P. Xie 54 24 0 06 Oct 2020