LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models

17 July 2024

Joshua Adrian Cahyono

Papers citing "LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models"

50 / 62 papers shown

Title
SITE: towards Spatial Intelligence Thorough Evaluation W. Wang Reuben Tan Pengyue Zhu Jianwei Yang Zhengyuan Yang Lijuan Wang Andrey Kolobov Jianfeng Gao Boqing Gong 22 2 0 08 May 2025
OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning Xianhang Li Y. Liu Haoqin Tu Hongru Zhu Cihang Xie VLM 21 0 0 07 May 2025
RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaboration Huajie Tan Xiaoshuai Hao Minglan Lin Pengwei Wang Yaoxu Lyu Mingyu Cao Zhongyuan Wang S. Zhang LM&Ro 24 73 0 06 May 2025
Learning Streaming Video Representation via Multitask Training Yibin Yan Jilan Xu Shangzhe Di Yikun Liu Yudi Shi Qirui Chen Zeqian Li Yifei Huang Weidi Xie CLL 76 123 0 28 Apr 2025
Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark Enxin Song Wenhao Chai Weili Xu Jianwen Xie Yuxuan Liu Gaoang Wang 44 0 0 20 Apr 2025
VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment Yogesh Kulkarni Pooyan Fazli 19 0 0 18 Apr 2025
Resampling Benchmark for Efficient Comprehensive Evaluation of Large Vision-Language Models Teppei Suzuki Keisuke Ozawa VLM 35 0 0 14 Apr 2025
Mavors: Multi-granularity Video Representation for Multimodal Large Language Model Yang Shi Jiaheng Liu Yushuo Guan Z. Wu Y. Zhang ... Bohan Zeng W. Zhang Fuzheng Zhang Wenjing Yang Di Zhang VGen VLM 50 0 0 14 Apr 2025
VideoAds for Fast-Paced Video Understanding: Where Opensource Foundation Models Beat GPT-4o & Gemini-1.5 Pro Zheyuan Zhang Monica Dou Linkai Peng Hongyi Pan Ulas Bagci Boqing Gong VLM 47 0 0 12 Apr 2025
TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection C. Xie Tongxuan Liu Lei Jiang Yuting Zeng J. Guo Yunheng Shen Weizhe Huang Jing Li Xiaohua Xu VLM 41 0 0 05 Apr 2025
VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning Xianwei Zhuang Yuxin Xie Yufan Deng Dongchao Yang Liming Liang Jinghan Ru Yuguo Yin Yuexian Zou 36 0 0 03 Apr 2025
Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features Jewon Lee Ki-Ung Song Seungmin Yang Donguk Lim Jaeyeon Kim Wooksu Shin Bo-Kyeong Kim Yong Jae Lee Tae-Ho Kim VLM 46 0 0 01 Apr 2025
XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery? Fengxiang Wang H. Wang Mingshuo Chen Di Wang Yulin Wang ... L. Lan Wenjing Yang J. Zhang Zhiyuan Liu Maosong Sun 44 2 0 31 Mar 2025
Learning to Instruct for Visual Instruction Tuning Zhihan Zhou Feng Hong Jiaan Luo Jiangchao Yao Dongsheng Li Bo Han Y. Zhang Yanfeng Wang VLM 49 0 0 28 Mar 2025
MAVERIX: Multimodal Audio-Visual Evaluation Reasoning IndeX Liuyue Xie George Z. Wei Avik Kuthiala Ce Zheng Ananya Bal ... Rohan Choudhury Morteza Ziyadi Xu Zhang Hao Yang László A. Jeni 52 0 0 27 Mar 2025
BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding Shuming Liu Chen Zhao Tianqi Xu Bernard Ghanem VLM 59 0 0 27 Mar 2025
From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignment Yucheng Suo Fan Ma Linchao Zhu T. Wang Fengyun Rao Yi Yang LRM 58 0 0 26 Mar 2025
Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning Huajie Tan Yuheng Ji Xiaoshuai Hao Minglan Lin Pengwei Wang Zhongyuan Wang Shanghang Zhang ReLM OffRL LRM 78 0 0 26 Mar 2025
FALCONEye: Finding Answers and Localizing Content in ONE-hour-long videos with multi-modal LLMs Carlos Plou Cesar Borja Ruben Martinez-Cantin Ana C. Murillo 44 0 0 25 Mar 2025
TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model Cheng Yang Yang Sui Jinqi Xiao Lingyi Huang Yu Gong ... Jinghua Yan Y. Bai P. Sadayappan Xia Hu Bo Yuan VLM 39 0 0 24 Mar 2025
Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Models Keda Tao Haoxuan You Yang Sui Can Qin H. Wang VLM MQ 74 0 0 20 Mar 2025
AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understanding Xiao Wang Qingyi Si Jianlong Wu Shiyu Zhu Li Cao Liqiang Nie VLM 57 0 0 16 Mar 2025
FastVID: Dynamic Density Pruning for Fast Video Large Language Models Leqi Shen Guoqiang Gong Tao He Yifeng Zhang Pengzhang Liu Sicheng Zhao Guiguang Ding VLM 50 0 0 14 Mar 2025
VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search Yiming Jia J. Li Xiang Yue Bo Li Ping Nie Kai Zou Wenhu Chen LRM 51 1 0 13 Mar 2025
Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models Bozhi Luan Wengang Zhou Hao Feng Zhe Wang Xiaosong Li H. Li VLM 54 0 0 11 Mar 2025
LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL Yingzhe Peng Gongrui Zhang Miaosen Zhang Zhiyuan You Jie Liu Qipeng Zhu Kai Yang Xingzhong Xu Xin Geng Xu Yang LRM ReLM 64 10 0 10 Mar 2025
SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation Z. Chen Chunwei Wang Xiuwei Chen Hang Xu J. Han Xiandan Liang VLM 47 1 0 09 Mar 2025
Can Atomic Step Decomposition Enhance the Self-structured Reasoning of Multimodal Large Models? Kun Xiang Zhili Liu Zihao Jiang Yunshuang Nie Kaixin Cai ... Yu-Jie Yuan J. Han Lanqing Hong Hang Xu Xiaodan Liang ReLM LRM 36 4 0 08 Mar 2025
CASP: Compression of Large Multimodal Models Based on Attention Sparsity Mohsen Gholami Mohammad Akbari Kevin Cannons Yong Zhang 48 0 0 07 Mar 2025
DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models Saeed Ranjbar Alvar Gursimran Singh Mohammad Akbari Yong Zhang VLM 60 0 0 04 Mar 2025
Adaptive Keyframe Sampling for Long Video Understanding Xi Tang Jihao Qiu Lingxi Xie Yunjie Tian Jianbin Jiao Qixiang Ye 64 0 0 28 Feb 2025
RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete Yuheng Ji Huajie Tan Jiayu Shi Xiaoshuai Hao Yuan Zhang ... Huaihai Lyu Xiaolong Zheng Jiaming Liu Zhongyuan Wang Shanghang Zhang 49 5 0 28 Feb 2025
All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark Davide Testa Giovanni Bonetta Raffaella Bernardi Alessandro Bondielli Alessandro Lenci Alessio Miaschi Lucia Passaro Bernardo Magnini VGen LRM 45 0 0 24 Feb 2025
Unhackable Temporal Rewarding for Scalable Video MLLMs En Yu Kangheng Lin Liang Zhao Yana Wei Zining Zhu ... Jianjian Sun Zheng Ge X. Zhang Jingyu Wang Wenbing Tao 42 1 0 17 Feb 2025
Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence Granite Vision Team Leonid Karlinsky Assaf Arbelle Abraham Daniels A. Nassar ... Sriram Raghavan T. Syeda-Mahmood Peter W. J. Staar Tal Drory Rogerio Feris VLM AI4TS 91 0 0 14 Feb 2025
VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model Xianwei Zhuang Yuxin Xie Yufan Deng Liming Liang Jinghan Ru Yuguo Yin Yuexian Zou MLLM VLM LRM 62 2 0 21 Jan 2025
Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding Z. Chen Mingxiao Li Z. Chen Nan Du Xiaolong Li Yuexian Zou 36 0 0 19 Jan 2025
Supervision-free Vision-Language Alignment Giorgio Giannone Ruoteng Li Qianli Feng Evgeny Perevodchikov Rui Chen Aleix M. Martinez VLM 42 0 0 08 Jan 2025
Multimodal Preference Data Synthetic Alignment with Reward Model Robert Wijaya Ngoc-Bao Nguyen Ngai-man Cheung MLLM SyDa 37 0 0 23 Dec 2024
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces Jihan Yang Shusheng Yang Anjali W. Gupta Rilyn Han Li Fei-Fei Saining Xie LRM 108 45 0 18 Dec 2024
VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models Byung-Kwan Lee Ryo Hachiuma Yu-Chiang Frank Wang Y. Ro Yueh-Hua Wu VLM 69 0 0 02 Dec 2024
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 71 0 0 01 Dec 2024
AtomThink: A Slow Thinking Framework for Multimodal Mathematical Reasoning Kun Xiang Zhili Liu Zihao Jiang Yunshuang Nie Runhui Huang ... Yihan Zeng J. Han Lanqing Hong Hang Xu Xiaodan Liang LRM 87 5 0 18 Nov 2024
MLAN: Language-Based Instruction Tuning Improves Zero-Shot Generalization of Multimodal Large Language Models Jianhong Tu Zhuohao Ni Nicholas Crispino Zihao Yu Michael Bendersky ... Ruoxi Jia Xin Liu Lingjuan Lyu Dawn Song Chenguang Wang VLM MLLM 40 0 0 15 Nov 2024
AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference Janghwan Lee Jiwoong Park Jinseok Kim Yongjik Kim Jungju Oh Jinwook Oh Jungwook Choi 34 1 0 15 Nov 2024
LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models Nam V. Nguyen Thong T. Doan Luong Tran Van Nguyen Quang Pham MoE 27 0 0 01 Nov 2024
JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation Shota Onohara Atsuyuki Miyai Yuki Imajuku Kazuki Egashira Jeonghun Baek Xiang Yue Graham Neubig Kiyoharu Aizawa OSLM 48 1 0 22 Oct 2024
Mitigating Object Hallucination via Concentric Causal Attention Yun Xing Yiheng Li Ivan Laptev Shijian Lu 24 11 0 21 Oct 2024
Harnessing Webpage UIs for Text-Rich Visual Understanding Junpeng Liu Tianyue Ou Yifan Song Yuxiao Qu Wai Lam Chenyan Xiong Wenhu Chen Graham Neubig Xiang Yue 48 1 0 17 Oct 2024
Can We Predict Performance of Large Models across Vision-Language Tasks? Qinyu Zhao Ming Xu Kartik Gupta Akshay Asthana Liang Zheng Stephen Gould 30 0 0 14 Oct 2024