LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

10 July 2024

Feng Li

Renrui Zhang

Hao Zhang

Yuanhan Zhang

Bo Li

Wei Li

Zejun Ma

Papers citing "LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models"

50 / 150 papers shown

Title
Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation Jungeun Kim Hyeongwoo Jeon Jongseong Bae Ha Young Kim SLR 70 0 0 25 Nov 2024
Video-Text Dataset Construction from Multi-AI Feedback: Promoting Weak-to-Strong Preference Learning for Video Large Language Models Hao Yi Qingyang Li Y. Hu Fuzheng Zhang Di Zhang Yong Liu VGen 65 0 0 25 Nov 2024
Human-Activity AGV Quality Assessment: A Benchmark Dataset and an Objective Evaluation Metric Zhichao Zhang Wei Sun Xinyue Li Yunhao Li Qihang Ge ... Zhongpeng Ji Fengyu Sun Shangling Jui Xiongkuo Min Guangtao Zhai EGVM 114 1 0 25 Nov 2024
Multimodal large language model for wheat breeding: a new exploration of smart breeding Guofeng Yang Yu Li Yong He Zhenjiang Zhou Lingzhen Ye Hui Fang Yiqi Luo Xuping Feng 62 2 0 20 Nov 2024
VCBench: A Controllable Benchmark for Symbolic and Abstract Challenges in Video Cognition Chenglin Li Qianglong Chen Zhi Li Feng Tao Yin Zhang 24 0 0 14 Nov 2024
Training-free Regional Prompting for Diffusion Transformers Anthony Chen Jianjin Xu Wenzhao Zheng Gaole Dai Y. Wang Renrui Zhang Haofan Wang Shanghang Zhang VLM 34 2 0 04 Nov 2024
PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance Ruyang Liu Haoran Tang Haibo Liu Yixiao Ge Ying Shan Chen Li Jiankun Yang VLM 27 4 0 04 Nov 2024
EfficientEQA: An Efficient Approach for Open Vocabulary Embodied Question Answering Kai Cheng Zhengyuan Li Xingpeng Sun Byung-Cheol Min Amrit Singh Bedi Aniket Bera 25 2 0 26 Oct 2024
Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data Shuhao Gu Jialing Zhang Siyuan Zhou Kevin Yu Zhaohu Xing ... Yufeng Cui Xinlong Wang Yaoqi Liu Fangxiang Feng Guang Liu SyDa VLM MLLM 22 17 0 24 Oct 2024
MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models Ziyu Liu Yuhang Zang Xiaoyi Dong Pan Zhang Yuhang Cao Haodong Duan Conghui He Yuanjun Xiong Dahua Lin Jiaqi Wang 19 1 0 23 Oct 2024
Captions Speak Louder than Images (CASLIE): Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data Xinyi Ling B. Peng Hanwen Du Zhihui Zhu Xia Ning 21 0 0 22 Oct 2024
EVA: An Embodied World Model for Future Video Anticipation Xiaowei Chi Hengyuan Zhang Chun-Kai Fan Xingqun Qi Rongyu Zhang ... Chi-Min Chan Wei Xue Wenhan Luo Shanghang Zhang Yike Guo VGen 25 4 0 20 Oct 2024
Dual-Model Distillation for Efficient Action Classification with Hybrid Edge-Cloud Solution Timothy Wei Hsien Xin Peng Elaine Xu Bryan Zhao Lei Ding Diji Yang 18 0 0 16 Oct 2024
Spatial-Aware Efficient Projector for MLLMs via Multi-Layer Feature Aggregation Shun Qian Bingquan Liu Chengjie Sun Zhen Xu Baoxun Wang 26 0 0 14 Oct 2024
MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models Wenbo Hu Jia-Chen Gu Zi-Yi Dou Mohsen Fayyaz Pan Lu Kai-Wei Chang Nanyun Peng VLM 38 4 0 10 Oct 2024
MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion Onkar Susladkar Jishu Sen Gupta Chirag Sehgal Sparsh Mittal Rekha Singhal DiffM VGen 28 0 0 10 Oct 2024
R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions? Chunyi Li J. Zhang Zicheng Zhang H. Wu Yuan Tian ... Guo Lu Xiaohong Liu Xiongkuo Min Weisi Lin Guangtao Zhai AAML 31 3 0 07 Oct 2024
Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation Fanqing Meng Jiaqi Liao Xinyu Tan Wenqi Shao Quanfeng Lu Kaipeng Zhang Yu Cheng Dianqi Li Yu Qiao Ping Luo VGen EGVM 19 23 0 07 Oct 2024
Intriguing Properties of Large Language and Vision Models Young-Jun Lee ByungSoo Ko Han-Gyu Kim Yechan Hwang Ho-Jin Choi LRM VLM 30 0 0 07 Oct 2024
Organizing Unstructured Image Collections using Natural Language Mingxuan Liu Zhun Zhong Jun Li Gianni Franchi Subhankar Roy Elisa Ricci VLM 18 3 0 07 Oct 2024
Frame-Voyager: Learning to Query Frames for Video Large Language Models Sicheng Yu Chengkai Jin Huanyu Wang Zhenghao Chen Sheng Jin ... Zhenbang Sun Bingni Zhang Jiawei Wu Hao Zhang Qianru Sun 41 5 0 04 Oct 2024
Unified Multi-Modal Interleaved Document Representation for Information Retrieval Jaewoo Lee Joonho Ko Jinheon Baek Soyeong Jeong Sung Ju Hwang 18 0 0 03 Oct 2024
Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks Mengzhao Jia Wenhao Yu Kaixin Ma Tianqing Fang Zhihan Zhang Siru Ouyang Hongming Zhang Meng-Long Jiang Dong Yu VLM 21 2 0 02 Oct 2024
PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation Mike Ranzinger Jon Barker Greg Heinrich Pavlo Molchanov Bryan Catanzaro Andrew Tao 15 4 0 02 Oct 2024
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning Haotian Zhang Mingfei Gao Zhe Gan Philipp Dufter Nina Wenzel ... Haoxuan You Zirui Wang Afshin Dehghan Peter Grasch Yinfei Yang VLM MLLM 36 32 1 30 Sep 2024
From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding Heqing Zou Tianze Luo Guiyang Xie Victor Zhang ... Guangcong Wang Juanyang Chen Zhuochen Wang Hansheng Zhang Huaijian Zhang VLM 21 6 0 27 Sep 2024
LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Chenming Zhu Tai Wang Wenwei Zhang Jiangmiao Pang Xihui Liu 81 29 0 26 Sep 2024
MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines Dongzhi Jiang Renrui Zhang Ziyu Guo Yanmin Wu Jiayi Lei ... Guanglu Song Peng Gao Yu Liu Chunyuan Li Hongsheng Li MLLM 27 16 0 19 Sep 2024
Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution Zuyan Liu Yuhao Dong Ziwei Liu Winston Hu Jiwen Lu Yongming Rao ObjD 56 54 0 19 Sep 2024
Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis Zhiqi Huang Dan Luo Jun Wang Huan Liao Zhiheng Li Zhiyong Wu VGen 27 4 0 13 Sep 2024
Enhancing Long Video Understanding via Hierarchical Event-Based Memory Dingxin Cheng Mingda Li Jingyu Liu Yongxin Guo Bin Jiang Qingbin Liu Xi Chen Bo Zhao 22 4 0 10 Sep 2024
mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding Anwen Hu Haiyang Xu Liang Zhang Jiabo Ye Ming Yan Ji Zhang Qin Jin Fei Huang Jingren Zhou VLM 14 27 0 05 Sep 2024
EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding Muye Huang Han Lai Xinyu Zhang Wenjun Wu Jie Ma Lingling Zhang Jun Liu 20 4 0 03 Sep 2024
UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios Baichuan Zhou Haote Yang Dairong Chen Junyan Ye Tianyi Bai Jinhua Yu Songyang Zhang Dahua Lin Conghui He Weijia Li VLM 45 3 0 30 Aug 2024
SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Manners Ziyu Guo Renrui Zhang Xiangyang Zhu Chengzhuo Tong Peng Gao Chunyuan Li Pheng-Ann Heng VGen 3DPC 37 9 0 29 Aug 2024
LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models Qihang Ge Wei Sun Yu Zhang Yunhao Li Zhongpeng Ji Fengyu Sun Shangling Jui Xiongkuo Min Guangtao Zhai 33 4 0 26 Aug 2024
Vintern-1B: An Efficient Multimodal Large Language Model for Vietnamese Khang T. Doan Bao G. Huynh D. T. Hoang Thuc D. Pham Nhat H. Pham Quan T.M. Nguyen Bang Q. Vo Suong N. Hoang MLLM 18 4 0 22 Aug 2024
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models Jiabo Ye Haiyang Xu Haowei Liu Anwen Hu Ming Yan Qi Qian Ji Zhang Fei Huang Jingren Zhou MLLM VLM 34 9 0 09 Aug 2024
MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models Fanqing Meng J. Wang Chuanhao Li Quanfeng Lu Hao Tian ... Jifeng Dai Yu Qiao Ping Luo Kaipeng Zhang Wenqi Shao VLM 47 17 0 05 Aug 2024
MAVIS: Mathematical Visual Instruction Tuning Renrui Zhang Xinyu Wei Dongzhi Jiang Yichi Zhang Ziyu Guo ... Aojun Zhou Bin Wei Shanghang Zhang Peng Gao Hongsheng Li MLLM 22 24 0 11 Jul 2024
InfiniBench: A Comprehensive Benchmark for Large Multimodal Models in Very Long Video Understanding Kirolos Ataallah Chenhui Gou Eslam Abdelrahman Khushbu Pahwa Jian Ding Mohamed Elhoseiny VLM 22 5 0 28 Jun 2024
Holistic Evaluation for Interleaved Text-and-Image Generation Minqian Liu Zhiyang Xu Zihao Lin Trevor Ashby Joy Rimchala Jiaxin Zhang Lifu Huang EGVM 28 7 0 20 Jun 2024
VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs Rohit K Bharadwaj Hanan Gani Muzammal Naseer F. Khan Salman Khan 45 3 0 14 Jun 2024
MANTIS: Interleaved Multi-Image Instruction Tuning Dongfu Jiang Xuan He Huaye Zeng Cong Wei Max W.F. Ku Qian Liu Wenhu Chen VLM MLLM 20 32 0 02 May 2024
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Marah Abdin Sam Ade Jacobs A. A. Awan J. Aneja Ahmed Hassan Awadallah ... Li Lyna Zhang Yi Zhang Yue Zhang Yunan Zhang Xiren Zhou LRM ALM 36 995 0 22 Apr 2024
CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching Dongzhi Jiang Guanglu Song Xiaoshi Wu Renrui Zhang Dazhong Shen Zhuofan Zong Yu Liu Hongsheng Li VLM 25 20 0 04 Apr 2024
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? Renrui Zhang Dongzhi Jiang Yichi Zhang Haokun Lin Ziyu Guo ... Aojun Zhou Pan Lu Kai-Wei Chang Peng Gao Hongsheng Li 22 165 0 21 Mar 2024
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models Chris Liu Renrui Zhang Longtian Qiu Siyuan Huang Weifeng Lin ... Hao Shao Pan Lu Hongsheng Li Yu Qiao Peng Gao MLLM 116 106 0 08 Feb 2024
VideoLLM: Modeling Video Sequence with Large Language Models Guo Chen Yin-Dong Zheng Jiahao Wang Jilan Xu Yifei Huang ... Yi Wang Yali Wang Yu Qiao Tong Lu Limin Wang MLLM 89 51 0 22 May 2023
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong S. Hoi MLLM BDL VLM CLIP 380 4,010 0 28 Jan 2022