RazorAttention: Efficient KV Cache Compression Through Retrieval Heads

22 July 2024

Papers citing "RazorAttention: Efficient KV Cache Compression Through Retrieval Heads"

25 / 25 papers shown

Title
LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important Manlai Liang JiaMing Zhang Xiong Li Jinlong Li MQ 30 0 0 07 Apr 2025
Cognitive Memory in Large Language Models Lianlei Shan Shixian Luo Zezhou Zhu Yu Yuan Yong Wu LLMAG KELM 60 1 0 03 Apr 2025
EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices Jiyu Chen Shuang Peng Daxiong Luo Fan Yang Renshou Wu Fangyuan Li Xiaoxin Chen 39 0 0 28 Mar 2025
GPU-Accelerated Motion Planning of an Underactuated Forestry Crane in Cluttered Environments M. Vu Gerald Ebmer Alexander Watcher Marc-Philip Ecker Giang Nguyen Tobias Glueck 57 2 0 18 Mar 2025
Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulation Wenlong Meng Fan Zhang Wendao Yao Zhenyuan Guo Y. Li Chengkun Wei Wenzhi Chen AAML 36 1 0 11 Mar 2025
CoKV: Optimizing KV Cache Allocation via Cooperative Game Qiheng Sun Hongwei Zhang Haocheng Xia Jiayao Zhang Jinfei Liu Kui Ren VLM 37 0 0 21 Feb 2025
Compression Barriers for Autoregressive Transformers Themistoklis Haris Krzysztof Onak 35 1 0 21 Feb 2025
HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading Cheng Luo Zefan Cai Hanshi Sun Jinqi Xiao Bo Yuan Wen Xiao Junjie Hu Jiawei Zhao Beidi Chen Anima Anandkumar 56 1 0 18 Feb 2025
Twilight: Adaptive Attention Sparsity with Hierarchical Top- $p$ Pruning C. Lin Jiaming Tang Shuo Yang Hanshuo Wang Tian Tang Boyu Tian Ion Stoica Song Han Mingyu Gao 81 2 0 04 Feb 2025
Explaining Context Length Scaling and Bounds for Language Models Jingzhe Shi Qinwei Ma Hongyi Liu Hang Zhao Jeng-Neng Hwang Serge Belongie Lei Li LRM 60 2 0 03 Feb 2025
GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments Yanyu Chen Ganhong Huang 95 0 0 28 Jan 2025
Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference WeiZhi Fei Xueyan Niu Guoqing Xie Yingqing Liu Bo Bai Wei Han 28 1 0 22 Jan 2025
MPCache: MPC-Friendly KV Cache Eviction for Efficient Private Large Language Model Inference Wenxuan Zeng Ye Dong Jinjin Zhou Junming Ma Jin Tan Runsheng Wang Meng Li 47 0 0 12 Jan 2025
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection Wei Yu Wu Zhuoshi Pan Chao Wang L. Chen Y. Bai Kun Fu Z. Wang Hui Xiong Hui Xiong LLMAG 23 5 0 05 Nov 2024
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference Hanshi Sun Li-Wen Chang Wenlei Bao Size Zheng Ningxin Zheng Xin Liu Harry Dong Yuejie Chi Beidi Chen VLM 78 16 0 28 Oct 2024
Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning Yu Fu Zefan Cai Abedelkadir Asi Wayne Xiong Yue Dong Wen Xiao 30 14 0 25 Oct 2024
TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models Mu Cai Reuben Tan Jianrui Zhang Bocheng Zou Kai Zhang ... Yao Dou J. Park Jianfeng Gao Yong Jae Lee Jianwei Yang 34 12 0 14 Oct 2024
A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts Suyu Ge Xihui Lin Yunan Zhang Jiawei Han Hao Peng 31 4 0 02 Oct 2024
InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference Xiurui Pan Endian Li Qiao Li Shengwen Liang Yizhou Shan Ke Zhou Yingwei Luo Xiaolin Wang Jie Zhang 20 10 0 08 Sep 2024
Attention Heads of Large Language Models: A Survey Zifan Zheng Yezhaohui Wang Yuxin Huang Shichao Song Mingchuan Yang Bo Tang Feiyu Xiong Zhiyu Li LRM 43 1 0 05 Sep 2024
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption Shi Luohe Hongyi Zhang Yao Yao Z. Li Zhao Hai 31 17 0 25 Jul 2024
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention Tsendsuren Munkhdalai Manaal Faruqui Siddharth Gopal LRM LLMAG CLL 79 101 0 10 Apr 2024
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models Soham De Samuel L. Smith Anushan Fernando Aleksandar Botev George-Christian Muraru ... David Budden Yee Whye Teh Razvan Pascanu Nando de Freitas Çağlar Gülçehre Mamba 51 116 0 29 Feb 2024
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 135 208 0 13 Mar 2023
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 234 690 0 27 Aug 2021