Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time

26 May 2023

Anastasios Kyrillidis

Anshumali Shrivastava

ArXiv PDF HTML

Papers citing "Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time"

50 / 158 papers shown

Title
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 34 17 0 01 Jul 2024
$$\text{Memory}^3$: Language Modeling with Explicit Memory$ $\text{Memory}^3$ : Language Modeling with Explicit Memory Hongkang Yang Zehao Lin Wenjin Wang Hao Wu Zhiyu Li ... Yu Yu Kai Chen Feiyu Xiong Linpeng Tang Weinan E 37 11 0 01 Jul 2024
InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management Wonbeom Lee Jungi Lee Junghwan Seo Jaewoong Sim RALM 26 72 0 28 Jun 2024
OutlierTune: Efficient Channel-Wise Quantization for Large Language Models Jinguang Wang Yuexi Yin Haifeng Sun Qi Qi Jingyu Wang Zirui Zhuang Tingting Yang Jianxin Liao 33 2 0 27 Jun 2024
From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models Sean Welleck Amanda Bertsch Matthew Finlayson Hailey Schoelkopf Alex Xie Graham Neubig Ilia Kulikov Zaid Harchaoui 33 45 0 24 Jun 2024
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers Chao Lou Zixia Jia Zilong Zheng Kewei Tu ODL 31 18 0 24 Jun 2024
MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression Tianyu Fu Haofeng Huang Xuefei Ning Genghan Zhang Boju Chen ... Shiyao Li Shengen Yan Guohao Dai Huazhong Yang Yu Wang MQ 44 16 0 21 Jun 2024
Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters Zhiyu Guo Hidetaka Kamigaito Taro Watanabe 19 20 0 18 Jun 2024
D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models Zhongwei Wan Xinjian Wu Yu Zhang Yi Xin Chaofan Tao ... Xin Wang Siqi Luo Jing Xiong Mi Zhang Mi Zhang 27 0 0 18 Jun 2024
CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling Yu Bai Xiyuan Zou Heyan Huang Sanxing Chen Marc-Antoine Rondeau Yang Gao Jackie Chi Kit Cheung 29 3 0 17 Jun 2024
MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding Zayd Muhammad Kawakibi Zuhri Muhammad Farid Adilazuarda Ayu Purwarianti Alham Fikri Aji 29 7 0 13 Jun 2024
Effectively Compress KV Heads for LLM Hao Yu Zelan Yang Shen Li Yong Li Jianxin Wu MQ VLM 31 12 0 11 Jun 2024
SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Models Hengyu Zhang RALM 20 2 0 09 Jun 2024
QCQA: Quality and Capacity-aware grouped Query Attention Vinay Joshi Prashant Laddha Shambhavi Sinha O. J. Omer S. Subramoney 16 4 0 08 Jun 2024
Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction Ke Cheng Wen Hu Zhi Wang Peng Du Jianguo Li Sheng Zhang 34 10 0 07 Jun 2024
QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead A. Zandieh Majid Daliri Insu Han MQ 35 12 0 05 Jun 2024
Loki: Low-Rank Keys for Efficient Sparse Attention Prajwal Singhania Siddharth Singh Shwai He S. Feizi A. Bhatele 32 13 0 04 Jun 2024
Block Transformer: Global-to-Local Language Modeling for Fast Inference Namgyu Ho Sangmin Bae Taehyeon Kim Hyunjik Jo Yireun Kim Tal Schuster Adam Fisch James Thorne Se-Young Yun 45 6 0 04 Jun 2024
DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion Yilong Chen Linhao Zhang Junyuan Shang Zhenyu Zhang Tingwen Liu Shuohuan Wang Yu Sun 25 1 0 03 Jun 2024
Optimizing Foundation Model Inference on a Many-tiny-core Open-source RISC-V Platform Viviane Potocnik Luca Colagrande Tim Fischer L. Bertaccini Daniele Jahier Pagliari Alessio Burrello Luca Benini 18 3 0 29 May 2024
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion Jiayi Yao Hanchen Li Yuhan Liu Siddhant Ray Yihua Cheng Qizheng Zhang Kuntai Du Shan Lu Junchen Jiang 42 14 0 26 May 2024
ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification Yefei He Luoming Zhang Weijia Wu Jing Liu Hong Zhou Bohan Zhuang MQ 35 24 0 23 May 2024
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention William Brandon Mayank Mishra Aniruddha Nrusimha Rameswar Panda Jonathan Ragan-Kelley MQ 36 38 0 21 May 2024
Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression Peiyu Liu Zeming Gao Wayne Xin Zhao Yipeng Ma Tao Wang Ji-Rong Wen MQ 24 4 0 21 May 2024
PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference Dongjie Yang Xiaodong Han Yan Gao Yao Hu Shilin Zhang Hai Zhao 27 49 0 21 May 2024
Layer-Condensed KV Cache for Efficient Inference of Large Language Models Haoyi Wu Kewei Tu MQ 41 17 0 17 May 2024
SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models Haojie Duanmu Zhihang Yuan Xiuhong Li Jiangfei Duan Xingcheng Zhang Dahua Lin MQ 34 18 0 10 May 2024
KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization Tianyi Zhang Jonah Yi Zhaozhuo Xu Anshumali Shrivastava MQ 29 25 0 07 May 2024
Efficient LLM Inference with Kcache Qiaozhi He Zhihua Wu RALM 25 1 0 28 Apr 2024
Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services Jiachen Liu Zhiyu Wu Jae-Won Chung Fan Lai Myungjin Lee Mosharaf Chowdhury 35 22 0 25 Apr 2024
CORM: Cache Optimization with Recent Message for Large Language Model Inference Jincheng Dai Zhuowei Huang Haiyun Jiang Chen Chen Deng Cai Wei Bi Shuming Shi 19 3 0 24 Apr 2024
SnapKV: LLM Knows What You are Looking for Before Generation Yuhong Li Yingbing Huang Bowen Yang Bharat Venkitesh Acyr F. Locatelli Hanchen Ye Tianle Cai Patrick Lewis Deming Chen VLM 75 148 0 22 Apr 2024
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation Chao Jin Zili Zhang Xuanlin Jiang Fangyue Liu Xin Liu Xuanzhe Liu Xin Jin 32 36 0 18 Apr 2024
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding Hanshi Sun Zhuoming Chen Xinyu Yang Yuandong Tian Beidi Chen 33 46 0 18 Apr 2024
Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation Thomas Merth Qichen Fu Mohammad Rastegari Mahyar Najibi LRM RALM 29 8 0 10 Apr 2024
SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget Zihao Wang Shaoduo Gan 30 6 0 07 Apr 2024
Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention Bin Gao Zhuomin He Puru Sharma Qingxuan Kang Djordje Jevdjic Junbo Deng Xingkun Yang Zhou Yu Pengfei Zuo 63 42 0 23 Mar 2024
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression Zhuoshi Pan Qianhui Wu Huiqiang Jiang Menglin Xia Xufang Luo ... Yuqing Yang Chin-Yew Lin H. V. Zhao Lili Qiu Dongmei Zhang VLM 33 88 0 19 Mar 2024
Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference Piotr Nawrot Adrian Lañcucki Marcin Chochowski David Tarjan E. Ponti 28 50 0 14 Mar 2024
CHAI: Clustered Head Attention for Efficient LLM Inference Saurabh Agarwal Bilge Acun Basil Homer Mostafa Elhoushi Yejin Lee Shivaram Venkataraman Dimitris Papailiopoulos Carole-Jean Wu 36 8 0 12 Mar 2024
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM Hao Kang Qingru Zhang Souvik Kundu Geonhwa Jeong Zaoxing Liu Tushar Krishna Tuo Zhao MQ 21 77 0 08 Mar 2024
QAQ: Quality Adaptive Quantization for LLM KV Cache Shichen Dong Wenfang Cheng Jiayu Qin Wei Wang MQ 41 32 0 07 Mar 2024
NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention Tianyi Zhang Jonah Yi Bowen Yao Zhaozhuo Xu Anshumali Shrivastava MQ 22 6 0 02 Mar 2024
No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization J. Yang Byeongwook Kim Jeongin Bae Beomseok Kwon Gunho Park Eunho Yang S. Kwon Dongsoo Lee MQ 34 44 0 28 Feb 2024
Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference Harry Dong Xinyu Yang Zhenyu (Allen) Zhang Zhangyang Wang Yuejie Chi Beidi Chen 27 47 0 14 Feb 2024
On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference Siyu Ren Kenny Q. Zhu 13 27 0 09 Feb 2024
SubGen: Token Generation in Sublinear Time and Memory A. Zandieh Insu Han Vahab Mirrokni Amin Karbasi 18 15 0 08 Feb 2024
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache Zirui Liu Jiayi Yuan Hongye Jin Shaochen Zhong Zhaozhuo Xu Vladimir Braverman Beidi Chen Xia Hu MQ 24 155 0 05 Feb 2024
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization Coleman Hooper Sehoon Kim Hiva Mohammadzadeh Michael W. Mahoney Y. Shao Kurt Keutzer A. Gholami MQ 12 171 0 31 Jan 2024
Transformers are Multi-State RNNs Matanel Oren Michael Hassid Nir Yarden Yossi Adi Roy Schwartz OffRL 19 34 0 11 Jan 2024