Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time

26 May 2023

Anastasios Kyrillidis

Anshumali Shrivastava

ArXiv PDF HTML

Papers citing "Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time"

50 / 158 papers shown

Title
Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM Zehao Fan Garrett Gagnon Zhenyu Liu Liu Liu 19 0 0 09 May 2025
Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing Piotr Piekos Róbert Csordás Jürgen Schmidhuber MoE VLM 88 1 0 01 May 2025
Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions Yiming Du Wenyu Huang Danna Zheng Zhaowei Wang Sébastien Montella Mirella Lapata Kam-Fai Wong Jeff Z. Pan KELM MU 71 1 0 01 May 2025
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate A. Zandieh Majid Daliri Majid Hadian Vahab Mirrokni MQ 74 0 0 28 Apr 2025
An Empirical Study on Prompt Compression for Large Language Models Z. Zhang Jinyi Li Yihuai Lan X. Wang Hao Wang MQ 42 0 0 24 Apr 2025
From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs Yaxiong Wu Sheng Liang Chen Zhang Y. Wang Y. Zhang Huifeng Guo Ruiming Tang Y. Liu KELM 36 1 0 22 Apr 2025
KeepKV: Eliminating Output Perturbation in KV Cache Compression for Efficient LLMs Inference Yuxuan Tian Zihan Wang Yebo Peng Aomufei Yuan Z. Wang Bairen Yi Xin Liu Yong Cui Tong Yang 29 0 0 14 Apr 2025
Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling Ziran Qin Youru Lv Mingbao Lin Zeren Zhang Danping Zou Weiyao Lin VLM 32 0 0 12 Apr 2025
Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression Hanqi Xiao Yi-Lin Sung Elias Stengel-Eskin Mohit Bansal MQ 31 0 0 10 Apr 2025
Adaptive Computation Pruning for the Forgetting Transformer Zhixuan Lin J. Obando-Ceron Xu Owen He Aaron C. Courville 30 0 0 09 Apr 2025
LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important Manlai Liang JiaMing Zhang Xiong Li Jinlong Li MQ 33 0 0 07 Apr 2025
Saliency-driven Dynamic Token Pruning for Large Language Models Yao Tao Yehui Tang Yun Wang Mingjian Zhu Hailin Hu Yunhe Wang 32 0 0 06 Apr 2025
Using Attention Sinks to Identify and Evaluate Dormant Heads in Pretrained LLMs Pedro Sandoval-Segura Xijun Wang Ashwinee Panda Micah Goldblum Ronen Basri Tom Goldstein David Jacobs 17 0 0 04 Apr 2025
Cognitive Memory in Large Language Models Lianlei Shan Shixian Luo Zezhou Zhu Yu Yuan Yong Wu LLMAG KELM 69 1 0 03 Apr 2025
SQuat: Subspace-orthogonal KV Cache Quantization Hao Wang Ligong Han Kai Xu Akash Srivastava MQ 43 0 0 31 Mar 2025
PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference Weisheng Jin Maojia Song Tej Deep Pala Yew Ken Chia Amir Zadeh Chuan Li Soujanya Poria VLM 47 0 0 30 Mar 2025
WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference Youhui Zuo Sibo Wei C. Zhang Zhuorui Liu Wenpeng Lu Dawei Song VLM 56 0 0 23 Mar 2025
SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs Shibo Jie Yehui Tang Kai Han Zhi-Hong Deng Jing Han 89 0 0 20 Mar 2025
GPU-Accelerated Motion Planning of an Underactuated Forestry Crane in Cluttered Environments M. Vu Gerald Ebmer Alexander Watcher Marc-Philip Ecker Giang Nguyen Tobias Glueck 63 0 0 18 Mar 2025
AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications Haiying Shen Tanmoy Sen 37 0 0 17 Mar 2025
CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences Ziran Qin Yuchen Cao Mingbao Lin Wen Hu Shixuan Fan Ke Cheng Weiyao Lin Jianguo Li 64 3 0 16 Mar 2025
Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models Bozhi Luan Wengang Zhou Hao Feng Zhe Wang Xiaosong Li H. Li VLM 61 0 0 11 Mar 2025
MergeQuant: Accurate 4-bit Static Quantization of Large Language Models by Channel-wise Calibration Jinguang Wang J. Wang Haifeng Sun Tingting Yang Zirui Zhuang Wanyi Ning Yuexi Yin Q. Qi Jianxin Liao MQ MoMe 44 0 0 07 Mar 2025
Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoning Giulio Corallo Orion Weller Fabio Petroni Paolo Papotti MQ VLM 49 0 0 06 Mar 2025
LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression Souvik Kundu Anahita Bhiwandiwalla Sungduk Yu Phillip Howard Tiep Le S. N. Sridhar David Cobbley Hao Kang Vasudev Lal MQ 54 1 0 06 Mar 2025
WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models Jian Yuan Ziwei He Haoli Bai Jingwen Leng Bo Jiang MoMe 34 0 0 03 Mar 2025
Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs Ravi Ghadia Avinash Kumar Gaurav Jain Prashant J. Nair Poulami Das 36 1 0 02 Mar 2025
Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving Qihui Zhou Peiqi Yin Pengfei Zuo James Cheng CLL 35 1 0 01 Mar 2025
Selective Prompt Anchoring for Code Generation Yuan Tian Tianyi Zhang 80 3 0 24 Feb 2025
KVCrush: Key value cache size-reduction using similarity in head-behaviour Gopi Krishna Jha Sameh Gobriel Liubov Talamanova Alexander Kozlov Nilesh Jain MQ 29 0 0 24 Feb 2025
DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance Xuanfan Ni Liyan Xu Chenyang Lyu Longyue Wang Mo Yu Lemao Liu Fandong Meng Jie Zhou Piji Li 45 0 0 24 Feb 2025
Compression Barriers for Autoregressive Transformers Themistoklis Haris Krzysztof Onak 35 1 0 21 Feb 2025
Neural Attention Search Difan Deng Marius Lindauer 85 0 0 21 Feb 2025
CoKV: Optimizing KV Cache Allocation via Cooperative Game Qiheng Sun Hongwei Zhang Haocheng Xia Jiayao Zhang Jinfei Liu Kui Ren VLM 37 0 0 21 Feb 2025
An Efficient Large Recommendation Model: Towards a Resource-Optimal Scaling Law Songpei Xu Shijia Wang Da Guo Xianwen Guo Qiang Xiao Fangjian Li Chuanjiang Luo 76 0 0 17 Feb 2025
Efficient Long-Decoding Inference with Reasoning-Aware Attention Sparsity Junhao Hu Wenrui Huang Weidong Wang Zhenwen Li Tiancheng Hu Zhixia Liu Xusheng Chen Tao Xie Yizhou Shan LRM 43 0 0 16 Feb 2025
BalanceKV: KV Cache Compression through Discrepancy Theory Insu Han Michael Kapralov Ekaterina Kochetkova Kshiteej Sheth A. Zandieh 84 2 0 11 Feb 2025
Enabling Autoregressive Models to Fill In Masked Tokens Daniel Israel Aditya Grover Guy Van den Broeck AI4CE 51 1 0 09 Feb 2025
Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective Yuan Feng Junlin Lv Y. Cao Xike Xie S.Kevin Zhou 71 2 0 06 Feb 2025
QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache Rishabh Tiwari Haocheng Xi Aditya Tomar Coleman Hooper Sehoon Kim Maxwell Horton Mahyar Najibi Michael W. Mahoney K. K. Amir Gholami MQ 43 1 0 05 Feb 2025
Can LLMs Maintain Fundamental Abilities under KV Cache Compression? Xiang Liu Zhenheng Tang Hong Chen Peijie Dong Zeyu Li Xiuze Zhou Bo Li Xuming Hu Xiaowen Chu 83 3 0 04 Feb 2025
Position: AI Scaling: From Up to Down and Out Yunke Wang Yanxi Li Chang Xu HAI 71 1 0 02 Feb 2025
KVDirect: Distributed Disaggregated LLM Inference Shiyang Chen Rain Jiang Dezhi Yu Jinlai Xu Mengyuan Chao Fanlong Meng Chenyu Jiang Wei Xu Hang Liu 40 1 0 28 Jan 2025
Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference Yuan Feng Junlin Lv Yukun Cao Xike Xie S. K. Zhou VLM 53 27 0 28 Jan 2025
Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference WeiZhi Fei Xueyan Niu Guoqing Xie Yingqing Liu Bo Bai Wei Han 28 1 0 22 Jan 2025
HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location Ting Sun Penghan Wang Fan Lai 60 1 0 15 Jan 2025
MPCache: MPC-Friendly KV Cache Eviction for Efficient Private Large Language Model Inference Wenxuan Zeng Ye Dong Jinjin Zhou Junming Ma Jin Tan Runsheng Wang Meng Li 47 0 0 12 Jan 2025
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval Di Liu Meng Chen Baotong Lu Huiqiang Jiang Zhenhua Han ... K. Zhang C. L. P. Chen Fan Yang Y. Yang Lili Qiu 39 29 0 03 Jan 2025
A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression Chenlong Deng Zhisong Zhang Kelong Mao Shuaiyi Li Xinting Huang Dong Yu Zhicheng Dou 36 1 0 23 Dec 2024
Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models Zhisong Zhang Yan Wang Xinting Huang Tianqing Fang H. Zhang Chenlong Deng Shuaiyi Li Dong Yu 75 2 0 21 Dec 2024