VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

3 November 2021

Papers citing "VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts"

50 / 75 papers shown

Title
Improving Routing in Sparse Mixture of Experts with Graph of Tokens Tam Minh Nguyen Ngoc N. Tran Khai Nguyen Richard G. Baraniuk MoE 59 0 0 01 May 2025
X-Fusion: Introducing New Modality to Frozen Large Language Models Sicheng Mo Thao Nguyen Xun Huang Siddharth Srinivasan Iyer Yijun Li ... Eli Shechtman Krishna Kumar Singh Yong Jae Lee Bolei Zhou Yuheng Li 71 0 0 29 Apr 2025
MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations Ziyang Zhang Yang Yu Yucheng Chen Xulei Yang S. Yeo MedIm 51 1 0 02 Mar 2025
sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging Jingyuan Chen Yuan Yao Mie Anderson Natalie Hauglund Celia Kjaerby Verena Untiet Maiken Nedergaard Jiebo Luo 41 1 0 28 Jan 2025
MM-Path: Multi-modal, Multi-granularity Path Representation Learning -- Extended Version Ronghui Xu Hanyin Cheng Chenjuan Guo Hongfan Gao J. Hu Sean Bin Yang Bin Yang 75 4 0 03 Jan 2025
CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance Chu Myaet Thwal Ye Lin Tun Minh N. H. Nguyen Eui-nam Huh Choong Seon Hong VLM 74 0 0 05 Dec 2024
Driving by the Rules: A Benchmark for Integrating Traffic Sign Regulations into Vectorized HD Map Xinyuan Chang Maixuan Xue Xinran Liu Zheng Pan Xing Wei 40 1 0 31 Oct 2024
Decoding Time Series with LLMs: A Multi-Agent Framework for Cross-Domain Annotation M. Lin Z. Chen Yanchi Liu Xujiang Zhao Zongyu Wu Junxiang Wang Xiang Zhang Suhang Wang Haifeng Chen AI4TS 28 7 0 22 Oct 2024
MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts R. Teo Tan M. Nguyen MoE 31 3 0 18 Oct 2024
Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training Gen Luo Xue Yang Wenhan Dou Zhaokai Wang Jifeng Dai Jifeng Dai Yu Qiao Xizhou Zhu VLM MLLM 62 25 0 10 Oct 2024
Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models Weihao Ye Qiong Wu Wenhao Lin Yiyi Zhou VLM 29 10 0 16 Sep 2024
Multi-Modal Adapter for Vision-Language Models Dominykas Seputis Serghei Mihailov Soham Chatterjee Zehao Xiao VLM 19 1 0 03 Sep 2024
MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts Xi Victoria Lin Akshat Shrivastava Liang Luo Srinivasan Iyer Mike Lewis Gargi Gosh Luke Zettlemoyer Armen Aghajanyan MoE 33 20 0 31 Jul 2024
Coding for Intelligence from the Perspective of Category Wenhan Yang Zixuan Hu Lilang Lin Jiaying Liu Ling-Yu Duan AI4CE 33 1 0 01 Jul 2024
Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model Longrong Yang Dong Shen Chaoxiang Cai Fan Yang Size Li Di Zhang Xi Li MoE 41 2 0 28 Jun 2024
Self Pre-training with Topology- and Spatiality-aware Masked Autoencoders for 3D Medical Image Segmentation Pengfei Gu Yejia Zhang Huimin Li Chaoli Wang D. Z. Chen MedIm 42 1 0 15 Jun 2024
Enhancing Large Vision Language Models with Self-Training on Image Comprehension Yihe Deng Pan Lu Fan Yin Ziniu Hu Sheng Shen James Y. Zou Kai-Wei Chang Wei Wang SyDa VLM LRM 31 36 0 30 May 2024
Multi-Modal Generative Embedding Model Feipeng Ma Hongwei Xue Guangting Wang Yizhou Zhou Fengyun Rao Shilin Yan Yueyi Zhang Siying Wu Mike Zheng Shou Xiaoyan Sun VLM 26 3 0 29 May 2024
OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All Yuanhuiyi Lyu Xueye Zheng Dahun Kim Lin Wang 32 10 0 25 May 2024
A Survey on Vision-Language-Action Models for Embodied AI Yueen Ma Zixing Song Yuzheng Zhuang Jianye Hao Irwin King LM&Ro 67 41 0 23 May 2024
Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification Long Lan Fengxiang Wang Shuyan Li Xiangtao Zheng Zengmao Wang Xinwang Liu VLM 24 7 0 13 Mar 2024
Enhancing Vision-Language Pre-training with Rich Supervisions Yuan Gao Kunyu Shi Pengkai Zhu Edouard Belval Oren Nuriel Srikar Appalaraju Shabnam Ghadar Vijay Mahadevan Zhuowen Tu Stefano Soatto VLM CLIP 62 12 0 05 Mar 2024
Non-autoregressive Sequence-to-Sequence Vision-Language Models Kunyu Shi Qi Dong Luis Goncalves Zhuowen Tu Stefano Soatto VLM 35 3 0 04 Mar 2024
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models Chris Liu Renrui Zhang Longtian Qiu Siyuan Huang Weifeng Lin ... Hao Shao Pan Lu Hongsheng Li Yu Qiao Peng Gao MLLM 126 106 0 08 Feb 2024
GeoDecoder: Empowering Multimodal Map Understanding Feng Qi Mian Dai Zixian Zheng Chao Wang 20 1 0 26 Jan 2024
AliFuse: Aligning and Fusing Multi-modal Medical Data for Computer-Aided Diagnosis Qiuhui Chen Yi Hong MedIm 15 1 0 02 Jan 2024
Weakly-Supervised 3D Visual Grounding based on Visual Linguistic Alignment Xiaoxu Xu Yitian Yuan Qiudan Zhang Wen-Bin Wu Zequn Jie Lin Ma Xu Wang 56 4 0 15 Dec 2023
TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training Chaoya Jiang Wei Ye Haiyang Xu Qinghao Ye Mingshi Yan Ji Zhang Shikun Zhang CLIP VLM 11 4 0 14 Dec 2023
Multimodality of AI for Education: Towards Artificial General Intelligence Gyeong-Geon Lee Lehong Shi Ehsan Latif Yizhu Gao Arne Bewersdorff ... Zheng Liu Hui Wang Gengchen Mai Tiaming Liu Xiaoming Zhai 22 37 0 10 Dec 2023
A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts Huy Nguyen Pedram Akbarian TrungTin Nguyen Nhat Ho 16 10 0 22 Oct 2023
Few-shot Action Recognition with Captioning Foundation Models Xiang Wang Shiwei Zhang Hangjie Yuan Yingya Zhang Changxin Gao Deli Zhao Nong Sang VLM 19 7 0 16 Oct 2023
EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE Junyi Chen Longteng Guo Jianxiang Sun Shuai Shao Zehuan Yuan Liang Lin Dongyu Zhang MLLM VLM MoE 43 9 0 23 Aug 2023
Bridging the Gap: Exploring the Capabilities of Bridge-Architectures for Complex Visual Reasoning Tasks Kousik Rajesh Mrigank Raman M. A. Karim Pranit Chawla VLM 23 2 0 31 Jul 2023
BARTPhoBEiT: Pre-trained Sequence-to-Sequence and Image Transformers Models for Vietnamese Visual Question Answering Khiem Vinh Tran Kiet Van Nguyen N. Nguyen ViT 15 2 0 28 Jul 2023
MESED: A Multi-modal Entity Set Expansion Dataset with Fine-grained Semantic Classes and Hard Negative Entities Y. Li Tingwei Lu Yinghui Li Tianyu Yu Shulin Huang Haitao Zheng Rui Zhang Jun Yuan 37 11 0 27 Jul 2023
Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner Zikang Liu Sihan Chen Longteng Guo Handong Li Xingjian He J. Liu 13 1 0 19 May 2023
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities Peng Wang Shijie Wang Junyang Lin Shuai Bai Xiaohuan Zhou Jingren Zhou Xinggang Wang Chang Zhou VLM MLLM ObjD 16 114 0 18 May 2023
Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literature Ana Claudia Akemi Matsuki de Faria Felype de Castro Bastos Jose Victor Nogueira Alves da Silva Vitor Lopes Fabris Valeska Uchôa Décio Gonccalves de Aguiar Neto C. F. G. Santos 25 22 0 18 May 2023
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation Chaoya Jiang Wei Ye Haiyang Xu Miang yan Shikun Zhang Jie Zhang Fei Huang VLM 19 15 0 08 May 2023
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention Renrui Zhang Jiaming Han Chris Liu Peng Gao Aojun Zhou Xiangfei Hu Shilin Yan Pan Lu Hongsheng Li Yu Qiao MLLM 23 737 0 28 Mar 2023
VideoXum: Cross-modal Visual and Textural Summarization of Videos Jingyang Lin Hang Hua Ming Chen Yikang Li Jenhao Hsiao C. Ho Jiebo Luo 23 30 0 21 Mar 2023
Align and Attend: Multimodal Summarization with Dual Contrastive Losses Bo He Jun Wang Jielin Qiu Trung Bui Abhinav Shrivastava Zhaowen Wang 8 65 0 13 Mar 2023
Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering Zhou Yu Xuecheng Ouyang Zhenwei Shao Mei Wang Jun Yu MLLM 89 11 0 03 Mar 2023
Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts Zhihong Chen Shizhe Diao Benyou Wang Guanbin Li Xiang Wan MedIm 17 29 0 17 Feb 2023
USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval Yan Zhang Zhong Ji Dingrong Wang Yanwei Pang Xuelong Li VLM 13 21 0 17 Jan 2023
UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering Chenlu Zhan Peng Peng Hongsen Wang Tao Chen Hongwei Wang MedIm 6 3 0 21 Dec 2022
OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models Jinze Bai Rui Men Han Yang Xuancheng Ren Kai Dang ... Wenhang Ge Jianxin Ma Junyang Lin Jingren Zhou Chang Zhou 37 15 0 08 Dec 2022
Component Segmentation of Engineering Drawings Using Graph Convolutional Networks Wentai Zhang Joe Joseph Yueyan Yin Liuyue Xie T. Furuhata Soji Yamakawa Kenji Shimada L. Kara 25 12 0 01 Dec 2022
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition Guimin Hu Ting-En Lin Yi Zhao Guangming Lu Yuchuan Wu Yongbin Li 25 110 0 21 Nov 2022
PromptCap: Prompt-Guided Task-Aware Image Captioning Yushi Hu Hang Hua Zhengyuan Yang Weijia Shi Noah A. Smith Jiebo Luo 28 101 0 15 Nov 2022