Reducing Transformer Key-Value Cache Size with Cross-Layer Attention

Reducing Transformer Key-Value Cache Size with Cross-Layer Attention

21 May 2024

William Brandon

Aniruddha Nrusimha

Jonathan Ragan-Kelley

Papers citing "Reducing Transformer Key-Value Cache Size with Cross-Layer Attention"

12 / 12 papers shown

Title
WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference Youhui Zuo Sibo Wei C. Zhang Zhuorui Liu Wenpeng Lu Dawei Song VLM 56 0 0 23 Mar 2025
GPU-Accelerated Motion Planning of an Underactuated Forestry Crane in Cluttered Environments M. Vu Gerald Ebmer Alexander Watcher Marc-Philip Ecker Giang Nguyen Tobias Glueck 66 0 0 18 Mar 2025
MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections Da Xiao Qingye Meng Shengping Li Xingyuan Yuan MoE AI4CE 54 1 0 13 Feb 2025
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 68 5 0 28 Oct 2024
A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference You Wu Haoyi Wu Kewei Tu 34 3 0 18 Oct 2024
An Evolved Universal Transformer Memory Edoardo Cetin Qi Sun Tianyu Zhao Yujin Tang 63 0 0 17 Oct 2024
ThinK: Thinner Key Cache by Query-Driven Pruning Yuhui Xu Zhanming Jie Hanze Dong Lei Wang Xudong Lu Aojun Zhou Amrita Saha Caiming Xiong Doyen Sahoo 67 14 0 30 Jul 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 34 17 0 01 Jul 2024
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention Tsendsuren Munkhdalai Manaal Faruqui Siddharth Gopal LRM LLMAG CLL 81 101 0 10 Apr 2024
Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention Bin Gao Zhuomin He Puru Sharma Qingxuan Kang Djordje Jevdjic Junbo Deng Xingkun Yang Zhou Yu Pengfei Zuo 63 42 0 23 Mar 2024
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 144 365 0 13 Mar 2023
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,791 0 17 Sep 2019