Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

11 January 2024

Shengbang Tong

Papers citing "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs"

50 / 63 papers shown

Title
Looking Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models Aarti Ghatkesar Uddeshya Upadhyay Ganesh Venkatesh VLM 31 0 0 08 May 2025
VSC: Visual Search Compositional Text-to-Image Diffusion Model Do Huu Dat Nam Hyeonu Po Yuan Mao Tae-Hyun Oh DiffM CoGe 55 0 0 02 May 2025
Multi-Modal Language Models as Text-to-Image Model Evaluators Jiahui Chen Candace Ross Reyhane Askari Hemmat Koustuv Sinha Melissa Hall M. Drozdzal Adriana Romero-Soriano EGVM 60 0 0 01 May 2025
Rethinking Visual Layer Selection in Multimodal LLMs H. Chen Junyan Lin Xinhao Chen Yue Fan Xin Jin Hui Su Jianfeng Dong Jinlan Fu Xiaoyu Shen VLM 95 0 0 30 Apr 2025
AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization Jinda Lu Jinghan Li Yuan Gao Junkang Wu Jiancan Wu X. Wang Xiangnan He 52 0 0 22 Apr 2025
How Can Objects Help Video-Language Understanding? Zitian Tang Shijie Wang Junho Cho Jaewook Yoo Chen Sun 40 0 0 10 Apr 2025
Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis J. Huang Baoxiong Jia Y. Wang Ziyu Zhu Xiongkun Linghu Qing Li Song-Chun Zhu Siyuan Huang 75 3 0 28 Mar 2025
A Vision Centric Remote Sensing Benchmark Abduljaleel Adejumo Faegheh Yeganli Clifford Broni-Bediako Aoran Xiao Naoto Yokoya Mennatullah Siam 58 0 0 20 Mar 2025
TULIP: Towards Unified Language-Image Pretraining Zineng Tang Long Lian Seun Eisape Xudong Wang Roei Herzig Adam Yala Alane Suhr Trevor Darrell David M. Chan VLM CLIP MLLM 95 3 0 19 Mar 2025
4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Wanhua Li Renping Zhou Jiawei Zhou Yingwei Song Johannes Herter Minghan Qin Gao Huang Hanspeter Pfister 3DGS VLM 66 0 0 13 Mar 2025
What's in a Latent? Leveraging Diffusion Latent Space for Domain Generalization Xavier Thomas Deepti Ghadiyaram DiffM 87 0 0 09 Mar 2025
Is Your Video Language Model a Reliable Judge? M. Liu Wensheng Zhang 56 1 0 07 Mar 2025
Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splatting Dominic Maggio Luca Carlone 73 0 0 07 Mar 2025
Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data Haoxin Li Boyang Li CoGe 69 0 0 03 Mar 2025
Learning to Animate Images from A Few Videos to Portray Delicate Human Actions Haoxin Li Yingchen Yu Qilong Wu Hanwang Zhang Boyang Li Song Bai 3DH VGen 78 0 0 01 Mar 2025
Forgotten Polygons: Multimodal Large Language Models are Shape-Blind William Rudman Michal Golovanesky Amir Bar Vedant Palit Yann LeCun Carsten Eickhoff Ritambhara Singh LRM 47 2 0 21 Feb 2025
LOVA3: Learning to Visual Question Answering, Asking and Assessment Henry Hengyuan Zhao Pan Zhou Difei Gao Zechen Bai Mike Zheng Shou 77 8 0 21 Feb 2025
Predicate Hierarchies Improve Few-Shot State Classification Emily Jin Joy Hsu Jiajun Wu OffRL 67 0 0 18 Feb 2025
Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding Kung-Hsiang Huang Can Qin Haoyi Qiu Philippe Laban Shafiq R. Joty Caiming Xiong C. Wu VLM 88 1 0 17 Feb 2025
Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability Zhiyu Zhu Zhibo Jin Jiayu Zhang Nan Yang Jiahao Huang Jianlong Zhou Fang Chen 39 0 0 16 Feb 2025
PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models? Mennatullah Siam VLM 76 1 0 06 Feb 2025
SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training Tianzhe Chu Yuexiang Zhai Jihan Yang Shengbang Tong Saining Xie Dale Schuurmans Quoc V. Le Sergey Levine Yi-An Ma OffRL 70 53 0 28 Jan 2025
ViDDAR: Vision Language Model-Based Task-Detrimental Content Detection for Augmented Reality Yanming Xiu T. Scargill M. Gorlatova 70 2 0 22 Jan 2025
FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance Haicheng Wang Zhemeng Yu Gabriele Spadaro Chen Ju Victor Quétu Enzo Tartaglione Enzo Tartaglione VLM 79 3 0 05 Jan 2025
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training Sanghwan Kim Rui Xiao Mariana-Iuliana Georgescu Stephan Alaniz Zeynep Akata VLM 70 0 0 02 Dec 2024
OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones? Z. Chen Tingzhu Chen Wenjun Zhang Guangtao Zhai 82 3 0 02 Dec 2024
Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs Qizhe Zhang Aosong Cheng Ming Lu Zhiyong Zhuo Minqi Wang Jiajun Cao Shaobo Guo Qi She Shanghang Zhang VLM 88 11 0 02 Dec 2024
Evaluating Vision-Language Models as Evaluators in Path Planning Mohamed Aghzal Xiang Yue E. Plaku Ziyu Yao LRM 72 1 0 27 Nov 2024
Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuning Ji Hyeok Jung Eun Tae Kim S. Kim Joo Ho Lee Bumsoo Kim Buru Chang VLM 112 0 0 24 Nov 2024
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games Davide Paglieri Bartłomiej Cupiał Samuel Coward Ulyana Piterbarg Maciej Wolczyk ... Lerrel Pinto Rob Fergus Jakob Foerster Jack Parker-Holder Tim Rocktaschel LLMAG LRM 106 10 0 20 Nov 2024
Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering Zeping Yu Sophia Ananiadou 67 0 0 17 Nov 2024
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions Moran Yanuka Assaf Ben-Kish Yonatan Bitton Idan Szpektor Raja Giryes VLM 45 2 0 13 Nov 2024
TIPS: Text-Image Pretraining with Spatial awareness Kevis-Kokitsi Maninis Kaifeng Chen Soham Ghosh Arjun Karpur Koert Chen ... Jan Dlabal Dan Gnanapragasam Mojtaba Seyedhosseini Howard Zhou Andre Araujo VLM 30 3 0 21 Oct 2024
Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning Yuxiang Lu Shengcao Cao Yu-xiong Wang 43 1 0 18 Oct 2024
MiCEval: Unveiling Multimodal Chain of Thought's Quality via Image Description and Reasoning Steps Xiongtao Zhou Jie He Lanyu Chen Jingyu Li Haojing Chen Víctor Gutiérrez-Basulto Jeff Z. Pan H. Chen LRM 55 1 0 18 Oct 2024
NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples Baiqi Li Zhiqiu Lin Wenxuan Peng Jean de Dieu Nyandwi Daniel Jiang Zixian Ma Simran Khanuja Ranjay Krishna Graham Neubig Deva Ramanan AAML CoGe VLM 61 20 0 18 Oct 2024
Locality Alignment Improves Vision-Language Models Ian Covert Tony Sun James Y. Zou Tatsunori Hashimoto VLM 64 3 0 14 Oct 2024
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark Wenhao Chai Enxin Song Y. Du Chenlin Meng Vashisht Madhavan Omer Bar-Tal Jeng-Neng Hwang Saining Xie Christopher D. Manning 3DV 77 25 0 04 Oct 2024
FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension Junzhuo Liu X. Yang Weiwei Li Peng Wang ObjD 39 3 0 23 Sep 2024
MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines Dongzhi Jiang Renrui Zhang Ziyu Guo Yanmin Wu Jiayi Lei ... Guanglu Song Peng Gao Yu Liu Chunyuan Li Hongsheng Li MLLM 27 16 0 19 Sep 2024
Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models Weihao Ye Qiong Wu Wenhao Lin Yiyi Zhou VLM 27 10 0 16 Sep 2024
EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models Yupeng Chen Penglin Chen Xiaoyu Zhang Yixian Huang Qian Xie DiffM 36 1 0 15 Sep 2024
No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning Manu Gaur Darshan Singh Makarand Tapaswi 68 1 0 04 Sep 2024
Law of Vision Representation in MLLMs Shijia Yang Bohan Zhai Quanzeng You Jianbo Yuan Hongxia Yang Chenfeng Xu 40 9 0 29 Aug 2024
Visual Agents as Fast and Slow Thinkers Guangyan Sun Mingyu Jin Zhenting Wang Cheng-Long Wang Siqi Ma Qifan Wang Ying Nian Wu Ying Nian Wu Dongfang Liu Dongfang Liu LLMAG LRM 77 12 0 16 Aug 2024
Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models Qirui Jiao Daoyuan Chen Yilun Huang Yaliang Li Ying Shen VLM 27 5 0 08 Aug 2024
Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach Yuxuan Wan Chaozheng Wang Yi Dong Wenxuan Wang Shuqing Li Yintong Huo M. Lyu 3DV 66 10 0 24 Jun 2024
TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Models Junlong Jia Ying Hu Xi Weng Yiming Shi Miao Li ... Baichuan Zhou Ziyu Liu Jie Luo Lei Huang Ji Wu 30 9 0 20 May 2024
Hallucination of Multimodal Large Language Models: A Survey Zechen Bai Pichao Wang Tianjun Xiao Tong He Zongbo Han Zheng Zhang Mike Zheng Shou VLM LRM 80 139 0 29 Apr 2024
Clio: Real-time Task-Driven Open-Set 3D Scene Graphs Dominic Maggio Yun Chang Nathan Hughes Matthew Trang Dan Griffith Carlyn Dougherty Eric Cristofalo Lukas Schmid Luca Carlone 3DV 36 31 0 21 Apr 2024