ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

21 November 2023

Conghui He

Dahua Lin

Papers citing "ShareGPT4V: Improving Large Multi-Modal Models with Better Captions"

50 / 467 papers shown

Title
Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation Yunhao Gou Kai Chen Zhili Liu Lanqing Hong Hang Xu Zhenguo Li Dit-Yan Yeung James T. Kwok Yu Zhang MLLM 35 37 0 14 Mar 2024
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization Renjie Pi Tianyang Han Wei Xiong Jipeng Zhang Runtao Liu Rui Pan Tong Zhang MLLM 35 33 0 13 Mar 2024
MoAI: Mixture of All Intelligence for Large Language and Vision Models Byung-Kwan Lee Beomchan Park Chae Won Kim Yonghyun Ro MLLM VLM 40 20 0 12 Mar 2024
Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models Yang Jiao Shaoxiang Chen Zequn Jie Jing Chen Lin Ma Yueping Jiang MLLM 37 18 0 12 Mar 2024
DeepSeek-VL: Towards Real-World Vision-Language Understanding Haoyu Lu Wen Liu Bo Zhang Bing-Li Wang Kai Dong ... Yaofeng Sun Chengqi Deng Hanwei Xu Zhenda Xie Chong Ruan VLM 36 288 0 08 Mar 2024
PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation Junsong Chen Chongjian Ge Enze Xie Yue Wu Lewei Yao Xiaozhe Ren Zhongdao Wang Ping Luo Huchuan Lu Zhenguo Li 128 86 0 07 Mar 2024
Yi: Open Foundation Models by 01.AI 01. AI Alex Young 01.AI Alex Young Bei Chen Chao Li ... Yue Wang Yuxuan Cai Zhenyu Gu Zhiyuan Liu Zonghong Dai OSLM LRM 121 497 0 07 Mar 2024
CoTBal: Comprehensive Task Balancing for Multi-Task Visual Instruction Tuning Yanqi Dai Dong Jing Nanyi Fei Zhiwu Lu Nanyi Fei Guoxing Yang Zhiwu Lu 50 3 0 07 Mar 2024
Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters Weizhi Wang Khalil Mrini Linjie Yang Sateesh Kumar Yu Tian Xifeng Yan Heng Wang 38 16 0 05 Mar 2024
ImgTrojan: Jailbreaking Vision-Language Models with ONE Image Xijia Tao Shuai Zhong Lei Li Qi Liu Lingpeng Kong 39 25 0 05 Mar 2024
Beyond Specialization: Assessing the Capabilities of MLLMs in Age and Gender Estimation Maksim Kuprashevich Grigorii Alekseenko Irina Tolstykh ELM 48 4 0 04 Mar 2024
Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models Lei Li Yuqi Wang Runxin Xu Peiyi Wang Xiachong Feng Lingpeng Kong Qi Liu 37 51 0 01 Mar 2024
The All-Seeing Project V2: Towards General Relation Comprehension of the Open World Weiyun Wang Yiming Ren Hao Luo Tiantong Li Chenxiang Yan ... Qingyun Li Lewei Lu Xizhou Zhu Yu Qiao Jifeng Dai MLLM 40 47 0 29 Feb 2024
A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models Xiujie Song Mengyue Wu Ke Zhu Chunhao Zhang Yanyi Chen LRM ELM 31 3 0 28 Feb 2024
Towards Open-ended Visual Quality Comparison Haoning Wu Hanwei Zhu Zicheng Zhang Erli Zhang Chaofeng Chen ... Qiong Yan Xiaohong Liu Guangtao Zhai Shiqi Wang Weisi Lin AAML 59 49 0 26 Feb 2024
RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis Yao Mu Junting Chen Qinglong Zhang Shoufa Chen Qiaojun Yu ... Wenhai Wang Jifeng Dai Yu Qiao Mingyu Ding Ping Luo 40 21 0 25 Feb 2024
Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models Chaoya Jiang Wei Ye Mengfan Dong Hongrui Jia Haiyang Xu Mingshi Yan Ji Zhang Shikun Zhang VLM MLLM 35 15 0 24 Feb 2024
DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models Yuhang Cao Pan Zhang Xiao-wen Dong Dahua Lin Jiaqi Wang 37 10 0 22 Feb 2024
Visual Hallucinations of Multi-modal Large Language Models Wen Huang Hongbin Liu Minxin Guo Neil Zhenqiang Gong MLLM VLM 32 24 0 22 Feb 2024
Subobject-level Image Tokenization Delong Chen Samuel Cahyawijaya Jianfeng Liu Baoyuan Wang Pascale Fung VLM OCL 46 7 0 22 Feb 2024
CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models Fuwen Luo Chi Chen Zihao Wan Zhaolu Kang Qidong Yan ... Xiaoyue Mi Peng Li Ning Ma Maosong Sun Yang Janet Liu 35 5 0 21 Feb 2024
A Touch, Vision, and Language Dataset for Multimodal Alignment Letian Fu Gaurav Datta Huang Huang Will Panitch Jaimyn Drake Joseph Ortiz Mustafa Mukadam Mike Lambeta Roberto Calandra Ken Goldberg VLM 25 32 0 20 Feb 2024
How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts Yusu Qian Haotian Zhang Yinfei Yang Zhe Gan 83 26 0 20 Feb 2024
ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models Guiming Hardy Chen Shunian Chen Ruifei Zhang Junying Chen Xiangbo Wu Zhiyi Zhang Zhihong Chen Jianquan Li Xiang Wan Benyou Wang VLM SyDa 33 129 0 18 Feb 2024
Cobra Effect in Reference-Free Image Captioning Metrics Zheng Ma Changxin Wang Yawen Ouyang Fei Zhao Jianbing Zhang Shujian Huang Jiajun Chen 30 2 0 18 Feb 2024
Efficient Multimodal Learning from Data-centric Perspective Muyang He Yexin Liu Boya Wu Jianhao Yuan Yueze Wang Tiejun Huang Bo-Lu Zhao MLLM 30 83 0 18 Feb 2024
CoLLaVO: Crayon Large Language and Vision mOdel Byung-Kwan Lee Beomchan Park Chae Won Kim Yonghyun Ro VLM MLLM 24 16 0 17 Feb 2024
Multi-modal preference alignment remedies regression of visual instruction tuning on language model Shengzhi Li Rongyu Lin Shichao Pei 40 20 0 16 Feb 2024
EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models Shangyu Xing Fei Zhao Zhen Wu Tuo An Weihao Chen Chunhui Li Jianbing Zhang Xinyu Dai MLLM MU 42 5 0 15 Feb 2024
Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision Zhaoqing Wang Xiaobo Xia Ziye Chen Xiao He Yandong Guo Mingming Gong Tongliang Liu VLM 14 11 0 14 Feb 2024
World Model on Million-Length Video And Language With Blockwise RingAttention Hao Liu Wilson Yan Matei A. Zaharia Pieter Abbeel VGen 29 59 0 13 Feb 2024
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models Chris Liu Renrui Zhang Longtian Qiu Siyuan Huang Weifeng Lin ... Hao Shao Pan Lu Hongsheng Li Yu Qiao Peng Gao MLLM 126 107 0 08 Feb 2024
MobileVLM V2: Faster and Stronger Baseline for Vision Language Model Xiangxiang Chu Limeng Qiao Xinyu Zhang Shuang Xu Fei Wei ... Xiaofei Sun Yiming Hu Xinyang Lin Bo-Wen Zhang Chunhua Shen VLM MLLM 22 95 0 06 Feb 2024
Instruction Makes a Difference Tosin P. Adewumi Nudrat Habib Lama Alkhaled Elisa Barney VLM MLLM 8 1 0 01 Feb 2024
MouSi: Poly-Visual-Expert Vision-Language Models Xiaoran Fan Tao Ji Changhao Jiang Shuo Li Senjie Jin ... Qi Zhang Xipeng Qiu Xuanjing Huang Zuxuan Wu Yunchun Jiang VLM 24 16 0 30 Jan 2024
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model Xiao-wen Dong Pan Zhang Yuhang Zang Yuhang Cao Bin Wang ... Conghui He Xingcheng Zhang Yu Qiao Dahua Lin Jiaqi Wang VLM MLLM 79 244 0 29 Jan 2024
LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs Shaoxiang Chen Zequn Jie Lin Ma MoE 40 46 0 29 Jan 2024
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models Bin Lin Zhenyu Tang Yang Ye Jiaxi Cui Bin Zhu ... Jinfa Huang Junwu Zhang Yatian Pang Munan Ning Li-ming Yuan VLM MLLM MoE 35 152 0 29 Jan 2024
MM-LLMs: Recent Advances in MultiModal Large Language Models Duzhen Zhang Yahan Yu Jiahua Dong Chenxing Li Dan Su Chenhui Chu Dong Yu OffRL LRM 39 178 0 24 Jan 2024
Red Teaming Visual Language Models Mukai Li Lei Li Yuwei Yin Masood Ahmed Zhenguang Liu Qi Liu VLM 28 30 0 23 Jan 2024
Benchmarking Large Multimodal Models against Common Corruptions Jiawei Zhang Tianyu Pang Chao Du Yi Ren Bo-wen Li Min-Bin Lin MLLM 22 14 0 22 Jan 2024
COCO is "ALL'' You Need for Visual Instruction Fine-tuning Xiaotian Han Yiqi Wang Bohan Zhai Quanzeng You Hongxia Yang VLM MLLM 28 2 0 17 Jan 2024
AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception Yipo Huang Quan Yuan Xiangfei Sheng Zhichao Yang Haoning Wu Pengfei Chen Yuzhe Yang Leida Li Weisi Lin VLM 19 37 0 16 Jan 2024
MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception Yuhao Wang Yusheng Liao Heyang Liu Hongcheng Liu Yu Wang Yanfeng Wang LRM VLM 22 13 0 15 Jan 2024
MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices Xiangxiang Chu Limeng Qiao Xinyang Lin Shuang Xu Yang Yang ... Fei Wei Xinyu Zhang Bo-Wen Zhang Xiaolin Wei Chunhua Shen MLLM 31 33 0 28 Dec 2023
Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning Yunhao Gou Zhili Liu Kai Chen Lanqing Hong Hang Xu Aoxue Li Dit-Yan Yeung James T. Kwok Yu Zhang MoE MLLM VLM 34 62 0 19 Dec 2023
Silkie: Preference Distillation for Large Visual Language Models Lei Li Zhihui Xie Mukai Li Shunian Chen Peiyi Wang Liang Chen Yazheng Yang Benyou Wang Lingpeng Kong MLLM 110 68 0 17 Dec 2023
VILA: On Pre-training for Visual Language Models Ji Lin Hongxu Yin Wei Ping Yao Lu Pavlo Molchanov Andrew Tao Huizi Mao Jan Kautz M. Shoeybi Song Han MLLM VLM 28 350 0 12 Dec 2023
Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator Henry Hengyuan Zhao Pan Zhou Mike Zheng Shou MLLM SyDa 33 7 0 11 Dec 2023
Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects Junyu Lu Ruyi Gan Di Zhang Xiaojun Wu Ziwei Wu Renliang Sun Jiaxing Zhang Pingjian Zhang Yan Song MLLM VLM 17 15 0 08 Dec 2023