Scissorhands: Exploiting the Persistence of Importance Hypothesis for
LLM KV Cache Compression at Test Time

Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time

26 May 2023

Anastasios Kyrillidis

Anshumali Shrivastava

Papers citing "Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time"

8 / 158 papers shown

Title
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems Xupeng Miao Gabriele Oliaro Zhihao Zhang Xinhao Cheng Hongyi Jin Tianqi Chen Zhihao Jia 53 75 0 23 Dec 2023
SGLang: Efficient Execution of Structured Language Model Programs Lianmin Zheng Liangsheng Yin Zhiqiang Xie Chuyue Sun Jeff Huang ... Christos Kozyrakis Ion Stoica Joseph E. Gonzalez Clark W. Barrett Ying Sheng LRM 29 102 0 12 Dec 2023
SparQ Attention: Bandwidth-Efficient LLM Inference Luka Ribar Ivan Chelombiev Luke Hudlass-Galley Charlie Blake Carlo Luschi Douglas Orr 21 45 0 08 Dec 2023
Punica: Multi-Tenant LoRA Serving Lequn Chen Zihao Ye Yongji Wu Danyang Zhuo Luis Ceze Arvind Krishnamurthy 26 34 0 28 Oct 2023
CacheGen: KV Cache Compression and Streaming for Fast Language Model Serving Yuhan Liu Hanchen Li Yihua Cheng Siddhant Ray Yuyang Huang ... Ganesh Ananthanarayanan Michael Maire Henry Hoffmann Ari Holtzman Junchen Jiang 50 41 0 11 Oct 2023
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs Suyu Ge Yunan Zhang Liyuan Liu Minjia Zhang Jiawei Han Jianfeng Gao 4 213 0 03 Oct 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 24 15 0 28 Sep 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 144 365 0 13 Mar 2023