ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

1 December 2023

Mu Cai

Haotian Liu

Dennis Park

Siva Karthik Mustikovela

Papers citing "ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts"

50 / 73 papers shown

Title
RESAnything: Attribute Prompting for Arbitrary Referring Segmentation Ruiqi Wang Hao Zhang VLM 44 0 0 03 May 2025
Robotic Visual Instruction Y. Li Ziyang Gong H. Li Xiaoqi Huang Haolan Kang Guangping Bai Xianzheng Ma LM&Ro 66 0 0 01 May 2025
EarthGPT-X: Enabling MLLMs to Flexibly and Comprehensively Understand Multi-Source Remote Sensing Imagery Wei Zhang Miaoxin Cai Yaqian Ning T. Zhang Yin Zhuang He Chen Jun Li Xuerui Mao 36 0 0 17 Apr 2025
Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding Tao Zhang X. Li Zilong Huang Y. Li Weixian Lei XueQing Deng Shihao Chen S. Ji Jiashi Feng MLLM LRM 45 1 0 14 Apr 2025
HistLLM: A Unified Framework for LLM-Based Multimodal Recommendation with User History Encoding and Compression Chen Zhang Bo Hu Weidong Chen Zhendong Mao 39 0 0 14 Apr 2025
MARS: a Multimodal Alignment and Ranking System for Few-Shot Segmentation Nico Catalano Stefano Samele Paolo Pertino Matteo Matteucci 3DPC 45 0 0 10 Apr 2025
URECA: Unique Region Caption Anything Sangbeom Lim J. Kim Heeji Yoon Jaewoo Jung Seungryong Kim 29 0 0 07 Apr 2025
Towards Understanding How Knowledge Evolves in Large Vision-Language Models Sudong Wang Y. Zhang Yao Zhu Jianing Li Zizhe Wang Y. Liu Xiangyang Ji 38 0 0 31 Mar 2025
Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation Niccolo Avogaro Thomas Frick Mattia Rigotti A. Bartezzaghi Filip Janicki C. Malossi Konrad Schindler Roy Assaf MLLM VLM 55 0 0 25 Mar 2025
Towards Online Multi-Modal Social Interaction Understanding X. Li Shijian Deng Bolin Lai Weiguo Pian James M. Rehg Yapeng Tian 41 0 0 25 Mar 2025
V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interaction Yiming Zhao Y. Zeng Yukun Qi Y. Liu Lin Yen-Chen Zehui Chen Xikun Bao Jie Zhao Feng Zhao VLM 53 2 0 22 Mar 2025
Visual Position Prompt for MLLM based Visual Grounding Wei Tang Yanpeng Sun Qinying Gu Zechao Li VLM 43 0 0 19 Mar 2025
Towards Understanding Graphical Perception in Large Multimodal Models Kai Zhang Jianwei Yang J. Inala Chandan Singh Jianfeng Gao Yu Su Chenglong Wang 40 1 0 13 Mar 2025
KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for Open-Vocabulary Robotic Manipulation Zixian Liu Mingtong Zhang Yunzhu Li 49 0 0 13 Mar 2025
GUIDE-CoT: Goal-driven and User-Informed Dynamic Estimation for Pedestrian Trajectory using Chain-of-Thought Sungsik Kim Janghyun Baek Jinkyu Kim Jaekoo Lee LRM 58 1 0 10 Mar 2025
The Role of Background Information in Reducing Object Hallucination in Vision-Language Models: Insights from Cutoff API Prompting Masayo Tomita Katsuhiko Hayashi Tomoyuki Kaneko VLM 43 0 0 24 Feb 2025
Object-centric Binding in Contrastive Language-Image Pretraining Rim Assouel Pietro Astolfi Florian Bordes M. Drozdzal Adriana Romero Soriano OCL VLM CoGe 102 0 0 19 Feb 2025
Magma: A Foundation Model for Multimodal AI Agents Jianwei Yang Reuben Tan Qianhui Wu Ruijie Zheng Baolin Peng ... Seonghyeon Ye Joel Jang Yuquan Deng Lars Liden Jianfeng Gao VLM AI4TS 98 8 0 18 Feb 2025
Vision-Language In-Context Learning Driven Few-Shot Visual Inspection Model Shiryu Ueno Yoshikazu Hayashi Shunsuke Nakatsuka Yusei Yamada Hiroaki Aizawa K. Kato MLLM VLM 84 0 0 13 Feb 2025
Parameter-Efficient Fine-Tuning for Foundation Models Dan Zhang Tao Feng Lilong Xue Yuandong Wang Yuxiao Dong J. Tang 37 6 0 23 Jan 2025
Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks Miran Heo Min-Hung Chen De-An Huang Sifei Liu Subhashree Radhakrishnan Seon Joo Kim Yu-Chun Wang Ryo Hachiuma ObjD VLM 108 2 0 14 Jan 2025
Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos Haobo Yuan X. Li Tao Zhang Zilong Huang Shilin Xu S. Ji Yunhai Tong Lu Qi Jiashi Feng Ming Yang VLM 79 11 0 07 Jan 2025
VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM Yuqian Yuan Hang Zhang Wentong Li Zesen Cheng Boqiang Zhang ... Deli Zhao Wenqiao Zhang Yueting Zhuang Jianke Zhu Lidong Bing 61 5 0 31 Dec 2024
CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models Yeyuan Wang D. Gao Bin Li Rujiao Long Lei Yi Xiaoyan Cai Libin Yang Jinxia Zhang Shanqing Yu Qi Xuan 68 1 0 22 Dec 2024
PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation Ao Wang Hui Chen Jianchao Tan K. Zhang Xunliang Cai Zijia Lin J. Han Guiguang Ding VLM 77 3 0 04 Dec 2024
freePruner: A Training-free Approach for Large Multimodal Model Acceleration Bingxin Xu Yuzhang Shang Yunhao Ge Qian Lou Yan Yan 94 3 0 23 Nov 2024
FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity Hang Hua Qing Liu Lingzhi Zhang Jing Shi Zhifei Zhang Yilin Wang Jianming Zhang Jiebo Luo CoGe VLM 87 6 0 23 Nov 2024
Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts Honglin Li Yuting Gao Chenglu Zhu Jingdong Chen M. Yang Lin Yang MLLM 79 0 0 21 Nov 2024
R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest Xupeng Chen Zhixin Lai Kangrui Ruan Shichu Chen Jiaxiang Liu Zuozhu Liu 33 1 0 27 Oct 2024
Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Models Yucheng Zhou Zhi Rao Jun Wan Jianbing Shen LRM 16 17 0 25 Oct 2024
When Does Perceptual Alignment Benefit Vision Representations? Shobhita Sundaram Stephanie Fu Lukas Muttenthaler Netanel Y. Tamir Lucy Chai Simon Kornblith Trevor Darrell Phillip Isola 47 12 1 14 Oct 2024
ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue Zhangpu Li Changhong Zou Suxue Ma Zhicheng Yang Chen Du ... Xingzhi Sun Jing Xiao Kai Zhang Mei Han Mei Han LM&MA 46 1 0 26 Sep 2024
EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models Jiacheng Zhang Yang Jiao Shaoxiang Chen Jingjing Chen Yu-Gang Jiang 18 1 0 25 Sep 2024
Proactive Schemes: A Survey of Adversarial Attacks for Social Good Vishal Asnani Xi Yin Xiaoming Liu AAML 26 1 0 24 Sep 2024
MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models Mohammad Shahab Sepehri Zalan Fabian Maryam Soltanolkotabi Mahdi Soltanolkotabi MedIm 30 3 0 23 Sep 2024
Towards Efficient and Robust VQA-NLE Data Generation with Large Vision-Language Models Patrick Amadeus Irawan Genta Indra Winata Samuel Cahyawijaya Ayu Purwarianti 25 0 0 23 Sep 2024
Visual Prompting in Multimodal Large Language Models: A Survey Junda Wu Zhehao Zhang Yu Xia Xintong Li Zhaoyang Xia ... Subrata Mitra Dimitris N. Metaxas Lina Yao Jingbo Shang Julian McAuley VLM LRM 38 12 0 05 Sep 2024
Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation Jian Hu Jiayi Lin Junchi Yan Shaogang Gong VLM 23 7 0 27 Aug 2024
PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understanding Dawei Dai Yuanhui Zhang Long Xu Qianlan Yang Xiaojing Shen Shuyin Xia Guoyin Wang LM&MA VLM 21 6 0 18 Aug 2024
ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models Ming-Kuan Wu Xinyue Cai Jiayi Ji Jiale Li Oucheng Huang Gen Luo Hao Fei Xiaoshuai Sun Rongrong Ji MLLM 40 7 0 31 Jul 2024
AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description Junyu Xie Tengda Han Max Bain Arsha Nagrani Gül Varol Weidi Xie Andrew Zisserman VGen 25 7 0 22 Jul 2024
EarthMarker: Visual Prompt Learning for Region-level and Point-level Remote Sensing Imagery Comprehension Wei Zhang Miaoxin Cai Tong Zhang Jun Li Zhuang Yin Xuerui Mao 61 5 0 18 Jul 2024
SPIN: Hierarchical Segmentation with Subpart Granularity in Natural Images Josh Myers-Dean Jarek Reynolds Brian Price Yifei Fan Danna Gurari 32 2 0 12 Jul 2024
IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model Yatai Ji Shilong Zhang Jie Wu Peize Sun Weifeng Chen Xuefeng Xiao Sidi Yang Y. Yang Ping Luo VLM 34 3 0 10 Jul 2024
Multi-Object Hallucination in Vision-Language Models Xuweiyi Chen Ziqiao Ma Xuejun Zhang Sihan Xu Shengyi Qian Jianing Yang David Fouhey Joyce Chai 44 15 0 08 Jul 2024
Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge Yuanze Lin Yunsheng Li Dongdong Chen Weijian Xu Ronald Clark Philip H. S. Torr Lu Yuan LRM VLM 23 8 0 05 Jul 2024
LLaRA: Supercharging Robot Learning Data for Vision-Language Policy Xiang Li Cristina Mata J. Park Kumara Kahatapitiya Yoo Sung Jang ... Kanchana Ranasinghe R. Burgert Mu Cai Yong Jae Lee Michael S. Ryoo LM&Ro 55 23 0 28 Jun 2024
OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding Tao Zhang Xiangtai Li Hao Fei Haobo Yuan Shengqiong Wu Shunping Ji Chen Change Loy Shuicheng Yan LRM MLLM VLM 47 44 0 27 Jun 2024
MACAROON: Training Vision-Language Models To Be Your Engaged Partners Shujin Wu Yi Ren Fung Sha Li Yixin Wan Kai-Wei Chang Heng Ji 28 5 0 20 Jun 2024
AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulation Chuyan Xiong Chengyu Shen Xiaoqi Li Kaichen Zhou Jiaming Liu Ruiping Wang Hao Dong LRM 30 10 0 17 Jun 2024