LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

10 July 2024

Feng Li

Renrui Zhang

Hao Zhang

Yuanhan Zhang

Bo Li

Wei Li

Zejun Ma

Papers citing "LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models"

50 / 150 papers shown

Title
OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning Y. Liu Saihui Hou Saijie Hou Jiabao Du Shibei Meng Yongzhen Huang VLM 49 0 0 14 Mar 2025
SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion A. Nassar Andres Marafioti Matteo Omenetti Maksym Lysak Nikolaos Livathinos ... Yusik Kim A. Said Gurbuz Michele Dolfi Miquel Farré Peter W. J. Staar 44 3 0 14 Mar 2025
HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model Jiaming Liu Hao Chen Pengju An Zhuoyang Liu Renrui Zhang ... Chengkai Hou Mengdi Zhao KC alex Zhou Pheng-Ann Heng S. Zhang 58 5 0 13 Mar 2025
CalliReader: Contextualizing Chinese Calligraphy via an Embedding-Aligned Vision-Language Model Yuxuan Luo Jiaqi Tang Chenyi Huang Feiyang Hao Zhouhui Lian VLM 51 0 0 13 Mar 2025
HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding Rui Yang Lin Song Yicheng Xiao Runhui Huang Yixiao Ge Ying Shan Hengshuang Zhao MLLM 57 0 0 12 Mar 2025
Referring to Any Person Qing Jiang Lin Wu Zhaoyang Zeng Tianhe Ren Yuda Xiong Yihao Chen Qin Liu Lei Zhang 61 0 0 11 Mar 2025
PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachability Weijie Zhou Manli Tao Chaoyang Zhao Haiyun Guo Honghui Dong Ming Tang J. T. Wang 43 0 0 11 Mar 2025
KwaiChat: A Large-Scale Video-Driven Multilingual Mixed-Type Dialogue Corpus Xiaoming Shi Zeming Liu Yiming Lei Chenkai Zhang Haitao Leng ... Qingjie Liu Wanxiang Che Shaoguo Liu Size Li Y. Wang 44 1 0 10 Mar 2025
Should VLMs be Pre-trained with Image Data? Sedrick Scott Keh Jean-Pierre Mercat S. Gadre Kushal Arora Igor Vasiljevic ... Shuran Song Russ Tedrake Thomas Kollar Ludwig Schmidt Achal Dave VLM 44 0 0 10 Mar 2025
SplatTalk: 3D VQA with Gaussian Splatting Anh Thai Songyou Peng Kyle Genova Leonidas J. Guibas Thomas Funkhouser 3DGS 75 0 0 08 Mar 2025
Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model Wenke Huang Jian Liang Xianda Guo Yiyang Fang Guancheng Wan ... Bin Yang He Li Jiawei Shao Mang Ye Bo Du OffRL LRM MLLM KELM VLM 60 1 0 06 Mar 2025
ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Task Vittorio Pippi Matthieu Guillaumin S. Cascianelli Rita Cucchiara M. Jaritz Loris Bazzani 57 0 0 06 Mar 2025
SpiritSight Agent: Advanced GUI Agent with One Look Zhiyuan Huang Ziming Cheng Junting Pan Zhaohui Hou Mingjie Zhan LLMAG 75 2 0 05 Mar 2025
LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant Wei Li Bing Hu Rui Shao Leyang Shen Liqiang Nie 28 2 0 05 Mar 2025
Parameter-free Video Segmentation for Vision and Language Understanding Louis Mahon Mirella Lapata VLM 30 1 0 03 Mar 2025
MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts P. Wang Zhongzhi Li Fei Yin Dekang Ran Chenglin Liu Cheng-Lin Liu LRM 34 3 0 28 Feb 2025
RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete Yuheng Ji Huajie Tan Jiayu Shi Xiaoshuai Hao Yuan Zhang ... Huaihai Lyu Xiaolong Zheng Jiaming Liu Zhongyuan Wang Shanghang Zhang 68 5 0 28 Feb 2025
Picking the Cream of the Crop: Visual-Centric Data Selection with Collaborative Agents Zhenyu Liu Yunxin Li Baotian Hu Wenhan Luo Yaowei Wang Min-Ling Zhang 55 0 0 27 Feb 2025
M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance Qingpei Guo Kaiyou Song Zipeng Feng Ziping Ma Qinglong Zhang ... Yunxiao Sun Tai-WeiChang Jingdong Chen Ming Yang Jun Zhou MLLM VLM 65 3 0 26 Feb 2025
Repurposing the scientific literature with vision-language models Anton Alyakin Jaden Stryker Daniel Alber Karl L. Sangwon Brandon Duderstadt ... Laura Snyder Eric Leuthardt Douglas Kondziolka E. Oermann Eric Karl Oermann 81 0 0 26 Feb 2025
VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulation Wei Zhao Pengxiang Ding M. Zhang Zhefei Gong Shuanghao Bai H. Zhao Donglin Wang 77 4 0 24 Feb 2025
MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection Xi Jiang Jian Li Hanqiu Deng Y. Liu Bin-Bin Gao Yifeng Zhou Jialin Li Chengjie Wang Feng Zheng 42 1 0 24 Feb 2025
MOVE: A Mixture-of-Vision-Encoders Approach for Domain-Focused Vision-Language Processing Matvey Skripkin Elizaveta Goncharova Dmitrii Tarasov Andrey Kuznetsov 58 0 0 24 Feb 2025
Forgotten Polygons: Multimodal Large Language Models are Shape-Blind William Rudman Michal Golovanesky Amir Bar Vedant Palit Yann LeCun Carsten Eickhoff Ritambhara Singh LRM 47 2 0 21 Feb 2025
Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images Shengguang Wu Fan-Yun Sun Kaiyue Wen Nick Haber VLM 67 0 0 20 Feb 2025
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization Shuo Xing Yuping Wang Peiran Li Ruizheng Bai Y. Wang Chengxuan Qian Huaxiu Yao Zhengzhong Tu 77 6 0 18 Feb 2025
MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency Dongzhi Jiang Renrui Zhang Ziyu Guo Yanwei Li Yu Qi ... Shen Yan Bo Zhang Chaoyou Fu Peng Gao Hongsheng Li MLLM LRM 77 21 0 13 Feb 2025
3D-Grounded Vision-Language Framework for Robotic Task Planning: Automated Prompt Synthesis and Supervised Reasoning Guoqin Tang Qingxuan Jia Zeyuan Huang Gang Chen Ning Ji Zhipeng Yao 56 0 0 13 Feb 2025
HCMRM: A High-Consistency Multimodal Relevance Model for Search Ads Guobing Gan Kaiming Gao Li Wang Shen Jiang Peng Jiang 54 0 0 09 Feb 2025
Tonguescape: Exploring Language Models Understanding of Vowel Articulation Haruki Sakajo Yusuke Sakai Hidetaka Kamigaito Taro Watanabe 33 0 0 29 Jan 2025
HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding Jiaxing Zhao Q. Yang Yixing Peng Detao Bai Shimin Yao ... Xiang Chen Shenghao Fu Weixuan chen Xihan Wei Liefeng Bo VGen AuLLM 50 5 0 28 Jan 2025
InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling Yi Wang Xinhao Li Ziang Yan Yinan He Jiashuo Yu ... Kai Chen Wenhai Wang Yu Qiao Yali Wang Limin Wang 61 19 0 21 Jan 2025
MMVU: Measuring Expert-Level Multi-Discipline Video Understanding Yilun Zhao Lujing Xie Haowei Zhang Guo Gan Yitao Long ... Xiangru Tang Zhenwen Liang Y. Liu Chen Zhao Arman Cohan 45 5 0 21 Jan 2025
When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis Ruixuan Zhang Beichen Wang Juexiao Zhang Zilin Bian Chen Feng K. Ozbay 31 2 0 17 Jan 2025
ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation Xuanle Zhao Xianzhen Luo Qi Shi C. L. P. Chen Shuo Wang Wanxiang Che Zhiyuan Liu Maosong Sun MLLM 48 2 0 11 Jan 2025
VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling Xinhao Li Yi Wang Jiashuo Yu Xiangyu Zeng Yuhan Zhu ... Yinan He Chenting Wang Yu Qiao Yali Wang L. Wang VLM 73 25 0 31 Dec 2024
Improving Fine-grained Visual Understanding in VLMs through Text-Only Training Dasol Choi Guijin Son Soo Yong Kim Gio Paik Seunghyeok Hong VLM CoGe 72 1 0 17 Dec 2024
Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics Sara Ghazanfari Siddharth Garg Nicolas Flammarion P. Krishnamurthy Farshad Khorrami Francesco Croce VLM 82 0 0 13 Dec 2024
Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM H. Wang Yuxiang Nie Yongjie Ye Deng GuanYu Yanjie Wang Shuai Li Haiyang Yu Jinghui Lu Can Huang VLM MLLM 74 1 0 12 Dec 2024
Falcon-UI: Understanding GUI Before Following User Instructions Huawen Shen Chang-Shu Liu Gengluo Li Xinlong Wang Yu Zhou Can Ma Xiangyang Ji LLMAG 72 4 0 12 Dec 2024
PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation Ao Wang Hui Chen Jianchao Tan K. Zhang Xunliang Cai Zijia Lin J. Han Guiguang Ding VLM 74 3 0 04 Dec 2024
Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation Gianni Franchi Dat Nguyen Trong Nacim Belkhir Guoxuan Xia Andrea Pilzer UQLM 66 0 0 04 Dec 2024
Progress-Aware Video Frame Captioning Zihui Xue Joungbin An Xitong Yang Kristen Grauman 90 1 0 03 Dec 2024
PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos Meng Cao Haoran Tang Haoze Zhao Hangyu Guo J. H. Liu Ge Zhang Ruyang Liu Qiang Sun Ian Reid Xiaodan Liang 90 2 0 02 Dec 2024
Eyes on the Road: State-of-the-Art Video Question Answering Models Assessment for Traffic Monitoring Tasks Joseph Raj Vishal Divesh Basina Aarya Choudhary Bharatesh Chakravarthi 62 1 0 02 Dec 2024
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 83 2 0 01 Dec 2024
ElectroVizQA: How well do Multi-modal LLMs perform in Electronics Visual Question Answering? Pragati Shuddhodhan Meshram Swetha Karthikeyan Bhavya Suma Bhat 90 0 0 27 Nov 2024
VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format Yueqian Wang Xiaojun Meng Y. Wang Jianxin Liang Jiansheng Wei Huishuai Zhang Dongyan Zhao VGen 67 2 0 27 Nov 2024
ChatRex: Taming Multimodal LLM for Joint Perception and Understanding Qing Jiang Gen Luo Yuqin Yang Yuda Xiong Yihao Chen Zhaoyang Zeng Tianhe Ren Lei Zhang VLM LRM 84 6 0 27 Nov 2024
Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding Andong Deng Zhongpai Gao Anwesa Choudhuri Benjamin Planche Meng Zheng Bin Wang Terrence Chen C. L. P. Chen Ziyan Wu AI4TS 67 1 0 25 Nov 2024