LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models

22 March 2024

Yan Yan

Papers citing "LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models"

36 / 86 papers shown

Title
PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction Long Xing Qidong Huang Xiaoyi Dong Jiajie Lu Pan Zhang ... Yuhang Cao Conghui He Jiaqi Wang Feng Wu Dahua Lin VLM 38 25 0 22 Oct 2024
xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs Michael S Ryoo Honglu Zhou Shrikant B. Kendre Can Qin Le Xue Manli Shu Silvio Savarese R. Xu Caiming Xiong Juan Carlos Niebles VGen 29 12 0 21 Oct 2024
Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation Zixin Wang Dong Gong Sen Wang Zi Huang Yadan Luo VLM 24 0 0 16 Oct 2024
Spatial-Aware Efficient Projector for MLLMs via Multi-Layer Feature Aggregation Shun Qian Bingquan Liu Chengjie Sun Zhen Xu Baoxun Wang 26 0 0 14 Oct 2024
Retrieval Replace Reduction: An effective visual token reduction method via semantic match Yingen Liu Fan Wu Ruihui Li Zhuo Tang KenLi Li VLM 18 0 0 09 Oct 2024
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Cong Guo Feng Cheng Zhixu Du James Kiessling Jonathan Ku ... Qilin Zheng Guanglei Zhou Hai Li-Wei Li Yiran Chen 14 5 0 08 Oct 2024
Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See Phu Pham Phu Pham Kun Wan Yu-Jhe Li Zeliang Zhang Daniel Miranda Ajinkya Kale Ajinkya Kale Chenliang Xu 22 5 0 08 Oct 2024
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark Wenhao Chai Enxin Song Y. Du Chenlin Meng Vashisht Madhavan Omer Bar-Tal Jeng-Neng Hwang Saining Xie Christopher D. Manning 3DV 58 25 0 04 Oct 2024
FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Model Yichen Lu Jiaqi Song Chao-Han Huck Yang Shinji Watanabe 16 0 0 03 Oct 2024
AVG-LLaVA: A Large Multimodal Model with Adaptive Visual Granularity Zhibin Lan Liqiang Niu Fandong Meng Wenbo Li Jie Zhou Jinsong Su VLM 30 2 0 20 Sep 2024
Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Manner Yuzhang Shang Bingxin Xu Weitai Kang Mu Cai Yuheng Li Zehao Wen Zhen Dong Kurt Keutzer Yong Jae Lee Yan Yan 28 0 0 19 Sep 2024
VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation Hanning Chen Yang Ni Wenjun Huang Yezi Liu SungHeon Jeong Fei Wen Nathaniel Bastian Hugo Latapie Mohsen Imani VLM 18 4 0 13 Sep 2024
Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information Yi Chen Jian Xu Xu-Yao Zhang Wen-Zhuo Liu Yang-Yang Liu Cheng-Lin Liu 18 3 0 02 Sep 2024
VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation Shiwei Wu Joya Chen Kevin Qinghong Lin Qimeng Wang Yan Gao Qianli Xu Tong Bill Xu Yao Hu Enhong Chen Mike Zheng Shou VLM 34 0 0 29 Aug 2024
SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs Yuanyang Yin Yaqi Zhao Yajie Zhang Ke Lin Jiahao Wang Xin Tao Pengfei Wan Di Zhang Baoqun Yin Wentao Zhang LRM 31 5 0 21 Aug 2024
HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models in Resource-Constrained Environments Kazi Hasan Ibn Arif JinYi Yoon Dimitrios S. Nikolopoulos Hans Vandierendonck Deepu John Bo Ji MLLM VLM 27 14 0 20 Aug 2024
Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models Fushuo Huo Wenchao Xu Zhong Zhang Haozhao Wang Zhicheng Chen Peilin Zhao VLM MLLM 55 18 0 04 Aug 2024
Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding Renshan Zhang Yibo Lyu Rui Shao Gongwei Chen Weili Guan Liqiang Nie 22 1 0 19 Jul 2024
ACTRESS: Active Retraining for Semi-supervised Visual Grounding Weitai Kang Mengxue Qu Yunchao Wei Yan Yan 21 6 0 03 Jul 2024
Visual Grounding with Attention-Driven Constraint Balancing Weitai Kang Luowei Zhou Junyi Wu Changchang Sun Yan Yan 25 4 0 03 Jul 2024
SegVG: Transferring Object Bounding Box to Segmentation for Visual Grounding Weitai Kang Gaowen Liu Mubarak Shah Yan Yan ObjD 25 6 0 03 Jul 2024
TokenPacker: Efficient Visual Projector for Multimodal LLM Wentong Li Yuqian Yuan Jian Liu Dongqi Tang Song Wang Jie Qin Jianke Zhu Lei Zhang MLLM 34 6 0 02 Jul 2024
DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming Jiaxin Zhang Wentao Yang Songxuan Lai Zecheng Xie Lianwen Jin 32 15 0 27 Jun 2024
LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference Zhongwei Wan Ziang Wu Che Liu Jinfa Huang Zhihong Zhu Peng Jin Longyue Wang Li Yuan VLM 25 28 0 26 Jun 2024
Long Context Transfer from Language to Vision Peiyuan Zhang Kaichen Zhang Bo Li Guangtao Zeng Jingkang Yang Yuanhan Zhang Ziyue Wang Haoran Tan Chunyuan Li Ziwei Liu VLM 55 139 0 24 Jun 2024
Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning Bingchen Zhao Yongshuo Zong Letian Zhang Timothy Hospedales VLM 20 15 0 18 Jun 2024
ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension Tianren Ma Lingxi Xie Yunjie Tian Boyu Yang Yuan Zhang 37 0 0 17 Jun 2024
Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention Weitai Kang Mengxue Qu Jyoti Kini Yunchao Wei Mubarak Shah Yan Yan LM&Ro 3DPC 36 9 0 28 May 2024
Matryoshka Multimodal Models Mu Cai Jianwei Yang Jianfeng Gao Yong Jae Lee VLM 39 25 0 27 May 2024
Efficient Multimodal Large Language Models: A Survey Yizhang Jin Jian Li Yexin Liu Tianjun Gu Kai Wu ... Xin Tan Zhenye Gan Yabiao Wang Chengjie Wang Lizhuang Ma LRM 39 44 0 17 May 2024
Small Language Model Meets with Reinforced Vision Vocabulary Haoran Wei Lingyu Kong Jinyue Chen Liang Zhao Zheng Ge En Yu Jian‐Yuan Sun Chunrui Han Xiangyu Zhang VLM 57 14 0 23 Jan 2024
LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model Yichen Zhu Minjie Zhu Ning Liu Zhicai Ou Xiaofeng Mou Jian Tang 63 89 0 04 Jan 2024
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection Bin Lin Yang Ye Bin Zhu Jiaxi Cui Munan Ning Peng Jin Li-ming Yuan VLM MLLM 182 576 0 16 Nov 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 244 4,186 0 30 Jan 2023
Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention Xiangcheng Liu Tianyi Wu Guodong Guo ViT 36 26 0 28 Sep 2022
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark A. Kalyan ELM ReLM LRM 198 1,089 0 20 Sep 2022