CoMP: Continual Multimodal Pre-training for Vision Foundation Models

24 March 2025

Papers citing "CoMP: Continual Multimodal Pre-training for Vision Foundation Models"

50 / 67 papers shown

Title
Skin Lesion Phenotyping via Nested Multi-modal Contrastive Learning Dionysis Christopoulos Sotiris Spanos Eirini Baltzi Valsamis Ntouskos Konstantinos Karantzalos 29 0 0 29 May 2025
SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features Michael Tschannen A. Gritsenko Xiao Wang Muhammad Ferjad Naeem Ibrahim Alabdulmohsin ... Basil Mustafa Olivier J. Hénaff Jeremiah Harmsen Andreas Steiner Xiaohua Zhai VLM 108 54 0 21 Feb 2025
Qwen2.5-VL Technical Report S. Bai Keqin Chen Xuejing Liu Jialin Wang Wenbin Ge ... Zesen Cheng Hang Zhang Zhibo Yang Haiyang Xu Junyang Lin VLM 144 430 0 20 Feb 2025
Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning Wujian Peng L. Meng Yuxiao Chen Yiweng Xie Yang Liu Tao Gui Hang Xu Xipeng Qiu Zuxuan Wu Yu-Gang Jiang MLLM VLM 73 4 0 04 Dec 2024
Pixtral 12B Pravesh Agrawal Szymon Antoniak Emma Bou Hanna Baptiste Bout Devendra Singh Chaplot ... Joachim Studnia Sandeep Subramanian Sagar Vaze Thomas Wang Sophia Yang VLM MLLM 67 51 0 09 Oct 2024
LLaVA-OneVision: Easy Visual Task Transfer Bo Li Yuanhan Zhang Dong Guo Renrui Zhang Feng Li Hao Zhang Kaichen Zhang Yanwei Li Ziwei Liu Chunyuan Li MLLM SyDa VLM 79 721 0 06 Aug 2024
DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception Xiaotong Li Fan Zhang Haiwen Diao Yueze Wang Xinlong Wang Ling-yu Duan VLM 60 29 0 11 Jul 2024
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs Shengbang Tong Ellis L Brown Penghao Wu Sanghyun Woo Manoj Middepogu ... Xichen Pan Austin Wang Rob Fergus Yann LeCun Saining Xie 3DV MLLM 80 321 0 24 Jun 2024
Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning Chenyu Yang Xizhou Zhu Jinguo Zhu Weijie Su Junjie Wang ... Lewei Lu Bin Li Jie Zhou Yu Qiao Jifeng Dai VLM CLIP 58 5 0 11 Jun 2024
DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs Lingchen Meng Jianwei Yang Rui Tian Xiyang Dai Zuxuan Wu Jianfeng Gao Yu-Gang Jiang VLM 56 9 0 06 Jun 2024
MoVA: Adapting Mixture of Vision Experts to Multimodal Context Zhuofan Zong Bingqi Ma Dazhong Shen Guanglu Song Hao Shao Dongzhi Jiang Hongsheng Li Yu Liu MoE 58 43 0 19 Apr 2024
Bridging Vision and Language Spaces with Assignment Prediction Jungin Park Jiyoung Lee Kwanghoon Sohn VLM 78 7 0 15 Apr 2024
Long-CLIP: Unlocking the Long-Text Capability of CLIP Beichen Zhang Pan Zhang Xiao-wen Dong Yuhang Zang Jiaqi Wang CLIP VLM 68 128 0 22 Mar 2024
Rotary Position Embedding for Vision Transformer Byeongho Heo Song Park Dongyoon Han Sangdoo Yun 82 41 0 20 Mar 2024
DeepSeek-VL: Towards Real-World Vision-Language Understanding Haoyu Lu Wen Liu Bo Zhang Bing-Li Wang Kai Dong ... Yaofeng Sun Chengqi Deng Hanwei Xu Zhenda Xie Chong Ruan VLM 71 333 0 08 Mar 2024
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models Chris Liu Renrui Zhang Longtian Qiu Siyuan Huang Weifeng Lin ... Hao Shao Pan Lu Hongsheng Li Yu Qiao Peng Gao MLLM 151 112 0 08 Feb 2024
VILA: On Pre-training for Visual Language Models Ji Lin Hongxu Yin Ming-Yu Liu Yao Lu Pavlo Molchanov Andrew Tao Huizi Mao Jan Kautz Mohammad Shoeybi Song Han MLLM VLM 61 382 0 12 Dec 2023
Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding Wujian Peng Sicheng Xie Zuyao You Shiyi Lan Zuxuan Wu VLM CoGe MLLM 51 19 0 30 Nov 2023
MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI Xiang Yue Yuansheng Ni Kai Zhang Tianyu Zheng Ruoqi Liu ... Yibo Liu Wenhao Huang Huan Sun Yu-Chuan Su Wenhu Chen OSLM ELM VLM 148 833 0 27 Nov 2023
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models Ziyi Lin Chris Liu Renrui Zhang Peng Gao Longtian Qiu ... Siyuan Huang Yichi Zhang Xuming He Hongsheng Li Yu Qiao MLLM VLM 50 219 0 13 Nov 2023
To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning Junke Wang Lingchen Meng Zejia Weng Bo He Zuxuan Wu Yu-Gang Jiang MLLM VLM 42 100 0 13 Nov 2023
Improved Baselines with Visual Instruction Tuning Haotian Liu Chunyuan Li Yuheng Li Yong Jae Lee VLM MLLM 89 2,593 0 05 Oct 2023
CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction Size Wu Wenwei Zhang Lumin Xu Sheng Jin Xiangtai Li Wentao Liu Chen Change Loy CLIP VLM 47 71 0 02 Oct 2023
Qwen Technical Report Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang ... Zhenru Zhang Chang Zhou Jingren Zhou Xiaohuan Zhou Tianhang Zhu OSLM 157 1,756 0 28 Sep 2023
Effective Long-Context Scaling of Foundation Models Wenhan Xiong Jingyu Liu Igor Molybog Hejia Zhang Prajjwal Bhargava ... Dániel Baráth Sergey Edunov Mike Lewis Sinong Wang Hao Ma 56 221 0 27 Sep 2023
SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension Bohao Li Rui Wang Guangzhi Wang Yuying Ge Yixiao Ge Ying Shan MLLM ELM 68 534 0 30 Jul 2023
Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution Mostafa Dehghani Basil Mustafa Josip Djolonga Jonathan Heek Matthias Minderer ... Avital Oliver Piotr Padlewski A. Gritsenko Mario Luvcić N. Houlsby ViT 101 110 0 12 Jul 2023
MMBench: Is Your Multi-modal Model an All-around Player? Yuanzhan Liu Haodong Duan Yuanhan Zhang Yue Liu Songyang Zhang ... Jiaqi Wang Conghui He Ziwei Liu Kai-xiang Chen Dahua Lin 44 992 0 12 Jul 2023
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models Chaoyou Fu Peixian Chen Yunhang Shen Yulei Qin Mengdan Zhang ... Xiawu Zheng Ke Li Xing Sun Zhenyu Qiu Rongrong Ji ELM MLLM 62 806 0 23 Jun 2023
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning Wenliang Dai Junnan Li Dongxu Li A. M. H. Tiong Junqi Zhao Weisheng Wang Boyang Albert Li Pascale Fung Steven C. H. Hoi MLLM VLM 75 2,017 0 11 May 2023
Visual Instruction Tuning Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee SyDa VLM MLLM 345 4,607 0 17 Apr 2023
DINOv2: Learning Robust Visual Features without Supervision Maxime Oquab Timothée Darcet Théo Moutakanni Huy Q. Vo Marc Szafraniec ... Hervé Jégou Julien Mairal Patrick Labatut Armand Joulin Piotr Bojanowski VLM CLIP SSL 249 3,205 0 14 Apr 2023
Sigmoid Loss for Language Image Pre-Training Xiaohua Zhai Basil Mustafa Alexander Kolesnikov Lucas Beyer CLIP VLM 83 1,076 0 27 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 815 12,840 0 27 Feb 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 385 4,465 0 30 Jan 2023
FlexiViT: One Model for All Patch Sizes Lucas Beyer Pavel Izmailov Alexander Kolesnikov Mathilde Caron Simon Kornblith Xiaohua Zhai Matthias Minderer Michael Tschannen Ibrahim Alabdulmohsin Filip Pavetić VLM 73 92 0 15 Dec 2022
ResFormer: Scaling ViTs with Multi-Resolution Training Rui Tian Zuxuan Wu Qiuju Dai Hang-Rui Hu Yu Qiao Yu-Gang Jiang ViT 49 35 0 01 Dec 2022
LAION-5B: An open large-scale dataset for training next generation image-text models Christoph Schuhmann Romain Beaumont Richard Vencu Cade Gordon Ross Wightman ... Srivatsa Kundurthy Katherine Crowson Ludwig Schmidt R. Kaczmarczyk J. Jitsev VLM MLLM CLIP 125 3,355 0 16 Oct 2022
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge Dustin Schwenk Apoorv Khandelwal Christopher Clark Kenneth Marino Roozbeh Mottaghi 43 530 0 03 Jun 2022
CoCa: Contrastive Captioners are Image-Text Foundation Models Jiahui Yu Zirui Wang Vijay Vasudevan Legg Yeung Mojtaba Seyedhosseini Yonghui Wu VLM CLIP OffRL 127 1,286 0 04 May 2022
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 283 3,458 0 29 Apr 2022
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training Zhan Tong Yibing Song Jue Wang Limin Wang ViT 195 1,156 0 23 Mar 2022
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning Ahmed Masry Do Xuan Long J. Tan Shafiq Joty Enamul Hoque AIMat 77 617 0 19 Mar 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 694 12,525 0 04 Mar 2022
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning Weixin Liang Yuhui Zhang Yongchan Kwon Serena Yeung James Zou VLM 82 403 0 03 Mar 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong Guosheng Lin MLLM BDL VLM CLIP 446 4,283 0 28 Jan 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 373 7,600 0 11 Nov 2021
BEiT: BERT Pre-Training of Image Transformers Hangbo Bao Li Dong Songhao Piao Furu Wei ViT 175 2,790 0 15 Jun 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 587 5,920 0 29 Apr 2021
InfographicVQA Minesh Mathew Viraj Bagal Rubèn Pérez Tito Dimosthenis Karatzas Ernest Valveny C. V. Jawahar 59 225 0 26 Apr 2021