To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning

13 November 2023

Zuxuan Wu

Papers citing "To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning"

50 / 84 papers shown

Title
Critique Before Thinking: Mitigating Hallucination through Rationale-Augmented Instruction Tuning Zexian Yang Dian Li Dayan Wu Gang Liu Weiping Wang MLLM LRM 36 0 0 12 May 2025
V $^2$ R-Bench: Holistically Evaluating LVLM Robustness to Fundamental Visual Variations Zhiyuan Fan Yumeng Wang Sandeep Polisetty Yi Ren Fung 43 0 0 23 Apr 2025
FUSION: Fully Integration of Vision-Language Representations for Deep Cross-Modal Understanding Zheng Liu Mengjie Liu J. Chen Jingwei Xu Bin Cui Conghui He Wentao Zhang MLLM 57 0 0 14 Apr 2025
Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? Yanbo Wang Jiyang Guan Jian Liang Ran He 41 0 0 14 Apr 2025
Unicorn: Text-Only Data Synthesis for Vision Language Model Training Xiaomin Yu Pengxiang Ding Wenjie Zhang Siteng Huang Songyang Gao Chengwei Qin Kejian Wu Zhaoxin Fan Ziyue Qiao Donglin Wang MLLM SyDa 67 0 0 28 Mar 2025
SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding Mingze Xu Mingfei Gao Shiyu Li Jiasen Lu Zhe Gan Zhengfeng Lai Meng Cao Kai Kang Y. Yang Afshin Dehghan 51 1 0 24 Mar 2025
OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models Jialv Zou Bencheng Liao Qian Zhang Wenyu Liu Xinggang Wang Mamba MLLM 77 1 0 11 Mar 2025
PP-DocBee: Improving Multimodal Document Understanding Through a Bag of Tricks Feng Ni Kui Huang Yao Lu Wenyu Lv Guanzhong Wang Zeyu Chen Y. Liu VLM 42 0 0 06 Mar 2025
DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models Y. Huang Peng Ye Chenyu Huang Jianjian Cao Lin Zhang Baopu Li Gang Yu Tao Chen MoMe MoE 53 0 0 03 Mar 2025
Knowledge Bridger: Towards Training-free Missing Multi-modality Completion Guanzhou Ke Shengfeng He X. Wang Bo Wang Guoqing Chao Y. Zhang Yi Xie HeXing Su 50 0 0 27 Feb 2025
Picking the Cream of the Crop: Visual-Centric Data Selection with Collaborative Agents Zhenyu Liu Yunxin Li Baotian Hu Wenhan Luo Yaowei Wang Min-Ling Zhang 60 0 0 27 Feb 2025
M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance Qingpei Guo Kaiyou Song Zipeng Feng Ziping Ma Qinglong Zhang ... Yunxiao Sun Tai-WeiChang Jingdong Chen Ming Yang Jun Zhou MLLM VLM 77 3 0 26 Feb 2025
MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding Weikang Qiu Zheng Huang Haoyu Hu Aosong Feng Yujun Yan Rex Ying 43 0 0 18 Feb 2025
Visual Large Language Models for Generalized and Specialized Applications Yifan Li Zhixin Lai Wentao Bao Zhen Tan Anh Dao Kewei Sui Jiayi Shen Dong Liu Huan Liu Yu Kong VLM 83 10 0 06 Jan 2025
Efficient Architectures for High Resolution Vision-Language Models Miguel Carvalho Bruno Martins MLLM VLM 35 0 0 05 Jan 2025
Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data Shuhao Gu Jialing Zhang Siyuan Zhou Kevin Yu Zhaohu Xing ... Yufeng Cui Xinlong Wang Yaoqi Liu Fangxiang Feng Guang Liu SyDa VLM MLLM 30 17 0 24 Oct 2024
Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models Yufei Zhan Hongyin Zhao Yousong Zhu Fan Yang Ming Tang Jinqiao Wang MLLM 43 1 0 21 Oct 2024
Locality Alignment Improves Vision-Language Models Ian Covert Tony Sun James Y. Zou Tatsunori Hashimoto VLM 58 3 0 14 Oct 2024
Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training Gen Luo Xue Yang Wenhan Dou Zhaokai Wang Jifeng Dai Jifeng Dai Yu Qiao Xizhou Zhu VLM MLLM 59 25 0 10 Oct 2024
HERM: Benchmarking and Enhancing Multimodal LLMs for Human-Centric Understanding Keliang Li Zaifei Yang Jiahe Zhao Hongze Shen Ruibing Hou Hong Chang Shiguang Shan Xilin Chen VLM 26 0 0 09 Oct 2024
EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment Yifei Xing Xiangyuan Lan Ruiping Wang D. Jiang Wenjun Huang Qingfang Zheng Yaowei Wang Mamba 33 0 0 08 Oct 2024
Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models Yufang Liu Tao Ji Changzhi Sun Yuanbin Wu Aimin Zhou VLM MLLM 30 1 0 04 Oct 2024
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark Wenhao Chai Enxin Song Y. Du Chenlin Meng Vashisht Madhavan Omer Bar-Tal Jeng-Neng Hwang Saining Xie Christopher D. Manning 3DV 71 25 0 04 Oct 2024
FullAnno: A Data Engine for Enhancing Image Comprehension of MLLMs Jing Hao Yuxiang Zhao Song Chen Yanpeng Sun Qiang Chen Gang Zhang Kun Yao Errui Ding Jingdong Wang VLM VGen MLLM 23 5 0 20 Sep 2024
POINTS: Improving Your Vision-language Model with Affordable Strategies Yuan Liu Zhongyin Zhao Ziyuan Zhuang Le Tian Xiao Zhou Jie Zhou VLM 35 5 0 07 Sep 2024
LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation Fangxun Shu Yue Liao Le Zhuo Chenning Xu Guanghao Zhang ... Bolin Li Zhelun Yu Si Liu Hongsheng Li Hao Jiang VLM MoE 27 8 0 28 Aug 2024
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders Min Shi Fuxiao Liu Shihao Wang Shijia Liao Subhashree Radhakrishnan ... Andrew Tao Andrew Tao Zhiding Yu Guilin Liu Guilin Liu MLLM 18 53 0 28 Aug 2024
EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model Feipeng Ma Yizhou Zhou Hebei Li Zilong He Siying Wu Fengyun Rao Siying Wu Fengyun Rao Yueyi Zhang Xiaoyan Sun 29 3 0 21 Aug 2024
Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications Qianqian Xie Dong Li Mengxi Xiao Zihao Jiang Ruoyu Xiang ... Benyou Wang Alejandro Lopez-Lira Qianqian Xie Sophia Ananiadou Junichi Tsujii AIFin AI4TS 30 13 0 20 Aug 2024
xGen-MM (BLIP-3): A Family of Open Large Multimodal Models Le Xue Manli Shu Anas Awadalla Jun Wang An Yan ... Zeyuan Chen Silvio Savarese Juan Carlos Niebles Caiming Xiong Ran Xu VLM 41 91 0 16 Aug 2024
CROME: Cross-Modal Adapters for Efficient Multimodal LLM Sayna Ebrahimi Sercan Ö. Arik Tejas Nama Tomas Pfister 37 1 0 13 Aug 2024
VITA: Towards Open-Source Interactive Omni Multimodal LLM Chaoyou Fu Haojia Lin Zuwei Long Yunhang Shen Meng Zhao ... Ran He Rongrong Ji Yunsheng Wu Caifeng Shan Xing Sun MLLM 29 79 0 09 Aug 2024
Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic Fakhraddin Alwajih Gagan Bhatia Muhammad Abdul-Mageed 21 5 0 25 Jul 2024
Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models Jinrui Zhang Teng Wang Haigang Zhang Ping Lu Feng Zheng MLLM LRM VLM 25 3 0 16 Jul 2024
DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception Xiaotong Li Fan Zhang Haiwen Diao Yueze Wang Xinlong Wang Ling-yu Duan VLM 21 25 0 11 Jul 2024
A Single Transformer for Scalable Vision-Language Modeling Yangyi Chen Xingyao Wang Hao Peng Heng Ji LRM 40 13 0 08 Jul 2024
Lateralization LoRA: Interleaved Instruction Tuning with Modality-Specialized Adaptations Zhiyang Xu Minqian Liu Ying Shen Joy Rimchala Jiaxin Zhang Qifan Wang Yu Cheng Lifu Huang VLM 37 2 0 04 Jul 2024
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output Pan Zhang Xiaoyi Dong Yuhang Zang Yuhang Cao Rui Qian ... Kai Chen Jifeng Dai Yu Qiao Dahua Lin Jiaqi Wang 43 98 0 03 Jul 2024
Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs Sukmin Yun Haokun Lin Rusiru Thushara Mohammad Qazim Bhat Yongxin Wang ... Timothy Baldwin Zhengzhong Liu Eric P. Xing Xiaodan Liang Zhiqiang Shen 24 9 0 28 Jun 2024
MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment Jihao Liu Xin Huang Jinliang Zheng Boxiao Liu Jia Wang Osamu Yoshie Yu Liu Hongsheng Li MLLM SyDa 19 3 0 28 Jun 2024
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs Shengbang Tong Ellis L Brown Penghao Wu Sanghyun Woo Manoj Middepogu ... Xichen Pan Austin Wang Rob Fergus Yann LeCun Saining Xie 3DV MLLM 37 278 0 24 Jun 2024
Unveiling Encoder-Free Vision-Language Models Haiwen Diao Yufeng Cui Xiaotong Li Yueze Wang Huchuan Lu Xinlong Wang VLM 30 27 0 17 Jun 2024
Generative Visual Instruction Tuning Jefferson Hernandez Ruben Villegas Vicente Ordonez VLM 30 3 0 17 Jun 2024
From Pixels to Prose: A Large Dataset of Dense Image Captions Vasu Singla Kaiyu Yue Sukriti Paul Reza Shirkavand Mayuka Jayawardhana Alireza Ganjdanesh Heng Huang A. Bhatele Gowthami Somepalli Tom Goldstein 3DV VLM 23 22 0 14 Jun 2024
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text Qingyun Li Zhe Chen Weiyun Wang Wenhai Wang Shenglong Ye ... Dahua Lin Yu Qiao Botian Shi Conghui He Jifeng Dai VLM OffRL 45 19 0 12 Jun 2024
MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models Tianle Gu Zeyang Zhou Kexin Huang Dandan Liang Yixu Wang ... Keqing Wang Yujiu Yang Yan Teng Yu Qiao Yingchun Wang ELM 33 9 0 11 Jun 2024
DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs Lingchen Meng Jianwei Yang Rui Tian Xiyang Dai Zuxuan Wu Jianfeng Gao Yu-Gang Jiang VLM 22 8 0 06 Jun 2024
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions Lin Chen Xilin Wei Jinsong Li Xiaoyi Dong Pan Zhang ... Li Yuan Yu Qiao Dahua Lin Feng Zhao Jiaqi Wang 72 138 0 06 Jun 2024
The Evolution of Multimodal Model Architectures S. Wadekar Abhishek Chaurasia Aman Chadha Eugenio Culurciello 41 13 0 28 May 2024
VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models Zejun Li Ruipu Luo Jiwen Zhang Minghui Qiu Zhongyu Wei Zhongyu Wei LRM MLLM 52 6 0 27 May 2024