Efficient Streaming Language Models with Attention Sinks

29 September 2023

Michel Lang

Yuandong Tian

Beidi Chen

Song Han

Mike Lewis

AI4TS

RALM

ArXiv PDF HTML

Papers citing "Efficient Streaming Language Models with Attention Sinks"

50 / 108 papers shown

Title
Unifying KV Cache Compression for Large Language Models with LeanKV Yanqi Zhang Yuwei Hu Runyuan Zhao John C. S. Lui Haibo Chen MQ 100 5 0 04 Dec 2024
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs Akhiad Bercovich Tomer Ronen Talor Abramovich Nir Ailon Nave Assaf ... Ido Shahaf Oren Tropp Omer Ullman Argov Ran Zilberstein Ran El-Yaniv 77 1 0 28 Nov 2024
More Expressive Attention with Negative Weights Ang Lv Ruobing Xie Shuaipeng Li Jiayi Liao X. Sun Zhanhui Kang Di Wang Rui Yan 30 0 0 11 Nov 2024
Context Parallelism for Scalable Million-Token Inference Amy Yang Jingyi Yang Aya Ibrahim Xinfeng Xie Bangsheng Tang Grigory Sizov Jeremy Reizenstein Jongsoo Park Jianyu Huang MoE LRM 60 5 0 04 Nov 2024
SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation Dennis Fucci Marco Gaido Beatrice Savoldi Matteo Negri Mauro Cettolo L. Bentivogli 49 1 0 03 Nov 2024
What is Wrong with Perplexity for Long-context Language Modeling? Lizhe Fang Yifei Wang Zhaoyang Liu Chenheng Zhang Stefanie Jegelka Jinyang Gao Bolin Ding Yisen Wang 58 4 0 31 Oct 2024
MDCure: A Scalable Pipeline for Multi-Document Instruction-Following Gabrielle Kaili-May Liu Bowen Shi Avi Caciularu Idan Szpektor Arman Cohan 58 3 0 30 Oct 2024
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference Hanshi Sun Li-Wen Chang Wenlei Bao Size Zheng Ningxin Zheng Xin Liu Harry Dong Yuejie Chi Beidi Chen VLM 88 16 0 28 Oct 2024
ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage Taewhoo Lee Chanwoong Yoon Kyochul Jang Donghyeon Lee Minju Song Hyunjae Kim Jaewoo Kang ELM 30 1 0 22 Oct 2024
PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction Long Xing Qidong Huang Xiaoyi Dong Jiajie Lu Pan Zhang ... Yuhang Cao Conghui He Jiaqi Wang Feng Wu Dahua Lin VLM 45 26 0 22 Oct 2024
Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Pingyi Chen Zhongyi Shui Chenglu Zhu Lin Yang MedIm 32 4 0 18 Oct 2024
A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference You Wu Haoyi Wu Kewei Tu 34 3 0 18 Oct 2024
Progressive Mixed-Precision Decoding for Efficient LLM Inference Hao Chen Fuwen Tan Alexandros Kouris Royson Lee Hongxiang Fan Stylianos I. Venieris MQ 21 1 0 17 Oct 2024
An Evolved Universal Transformer Memory Edoardo Cetin Qi Sun Tianyu Zhao Yujin Tang 63 0 0 17 Oct 2024
Sequential LLM Framework for Fashion Recommendation Han Liu Xianfeng Tang Tianlang Chen Jiapeng Liu Indu Indu ... Roberto Fernandez Galan Michael D Porter Dongmei Jia Ning Zhang Lian Xiong AI4TS 13 1 0 15 Oct 2024
In-context KV-Cache Eviction for LLMs via Attention-Gate Zihao Zeng Bokai Lin Tianqi Hou Hao Zhang Zhijie Deng 38 1 0 15 Oct 2024
ControlMM: Controllable Masked Motion Generation Ekkasit Pinyoanuntapong Muhammad Usama Saleem Korrawe Karunratanakul Pu Wang Hongfei Xue C. L. P. Chen Chuan Guo Junli Cao J. Ren Sergey Tulyakov VGen 29 4 0 14 Oct 2024
Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective Jinhao Li Jiaming Xu Shan Huang Yonghua Chen Wen Li ... Jiayi Pan Li Ding Hao Zhou Yu Wang Guohao Dai 57 15 0 06 Oct 2024
Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information Yongheng Zhang Qiguang Chen Jingxuan Zhou Peng Wang Jiasheng Si Jin Wang Wenpeng Lu Libo Qin LRM 44 3 0 06 Oct 2024
LongGenBench: Long-context Generation Benchmark Xiang Liu Peijie Dong Xuming Hu Xiaowen Chu RALM 43 8 0 05 Oct 2024
How to Train Long-Context Language Models (Effectively) Tianyu Gao Alexander Wettig Howard Yen Danqi Chen RALM 72 37 0 03 Oct 2024
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration Jintao Zhang Jia wei Pengle Zhang Jun-Jie Zhu Jun Zhu Jianfei Chen VLM MQ 82 18 0 03 Oct 2024
Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices Yuxiang Huang Binhang Yuan Xu Han Chaojun Xiao Zhiyuan Liu RALM 73 1 0 02 Oct 2024
Attention layers provably solve single-location regression P. Marion Raphael Berthier Gérard Biau Claire Boyer 63 2 0 02 Oct 2024
House of Cards: Massive Weights in LLMs Jaehoon Oh Seungjun Shin Dokwan Oh 35 1 0 02 Oct 2024
The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems Linke Song Zixuan Pang Wenhao Wang Zihao Wang XiaoFeng Wang Hongbo Chen Wei Song Yier Jin Dan Meng Rui Hou 43 7 0 30 Sep 2024
A-VL: Adaptive Attention for Large Vision-Language Models Junyang Zhang Mu Yuan Ruiguang Zhong Puhan Luo Huiyou Zhan Ningkang Zhang Chengchen Hu Xiangyang Li VLM 36 1 0 23 Sep 2024
Towards LifeSpan Cognitive Systems Yu Wang Chi Han Tongtong Wu Xiaoxin He Wangchunshu Zhou ... Zexue He Wei Wang Gholamreza Haffari Heng Ji Julian McAuley KELM CLL 88 1 0 20 Sep 2024
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding Jian Chen Vashisth Tiwari Ranajoy Sadhukhan Zhuoming Chen Jinyuan Shi Ian En-Hsu Yen Ian En-Hsu Yen Avner May Tianqi Chen Beidi Chen LRM 31 22 0 20 Aug 2024
Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining Dongyang Liu Shitian Zhao Le Zhuo Weifeng Lin Yu Qiao Xinyue Li Qi Qin Yu Qiao Hongsheng Li Peng Gao MLLM 62 48 0 05 Aug 2024
ThinK: Thinner Key Cache by Query-Driven Pruning Yuhui Xu Zhanming Jie Hanze Dong Lei Wang Xudong Lu Aojun Zhou Amrita Saha Caiming Xiong Doyen Sahoo 67 14 0 30 Jul 2024
Grammar-based Game Description Generation using Large Language Models Tsunehiko Tanaka Edgar Simo-Serra 48 2 0 24 Jul 2024
A deeper look at depth pruning of LLMs Shoaib Ahmed Siddiqui Xin Dong Greg Heinrich Thomas Breuel Jan Kautz David M. Krueger Pavlo Molchanov 29 7 0 23 Jul 2024
RazorAttention: Efficient KV Cache Compression Through Retrieval Heads Hanlin Tang Yang Lin Jing Lin Qingsen Han Shikuan Hong Yiwu Yao Gongyi Wang MQ 29 26 0 22 Jul 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 33 41 0 09 Jul 2024
Let the Code LLM Edit Itself When You Edit the Code Zhenyu He Jun Zhang Shengjie Luo Jingjing Xu Z. Zhang Di He KELM 31 0 0 03 Jul 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 34 17 0 01 Jul 2024
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA Minzheng Wang Longze Chen Cheng Fu Shengyi Liao Xinghua Zhang ... Run Luo Yunshui Li Min Yang Fei Huang Yongbin Li RALM 37 41 0 25 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 64 55 0 11 Jun 2024
Loki: Low-Rank Keys for Efficient Sparse Attention Prajwal Singhania Siddharth Singh Shwai He S. Feizi A. Bhatele 32 13 0 04 Jun 2024
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling Zefan Cai Yichi Zhang Bofei Gao Yuliang Liu Y. Li ... Wayne Xiong Yue Dong Baobao Chang Junjie Hu Wen Xiao 55 83 0 04 Jun 2024
Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models Zhangyue Yin Qiushi Sun Qipeng Guo Zhiyuan Zeng Xiaonan Li ... Qinyuan Cheng Ding Wang Xiaofeng Mou Xipeng Qiu XuanJing Huang LRM 41 3 0 21 May 2024
PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference Dongjie Yang Xiaodong Han Yan Gao Yao Hu Shilin Zhang Hai Zhao 27 49 0 21 May 2024
Memory Mosaics Jianyu Zhang Niklas Nolte Ranajoy Sadhukhan Beidi Chen Léon Bottou VLM 54 3 0 10 May 2024
In-Context Learning with Long-Context Models: An In-Depth Exploration Amanda Bertsch Maor Ivgi Uri Alon Jonathan Berant Matthew R. Gormley Matthew R. Gormley Graham Neubig ReLM AIMat 81 65 0 30 Apr 2024
Efficient LLM Inference with Kcache Qiaozhi He Zhihua Wu RALM 25 1 0 28 Apr 2024
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation Chao Jin Zili Zhang Xuanlin Jiang Fangyue Liu Xin Liu Xuanzhe Liu Xin Jin 32 36 0 18 Apr 2024
When Medical Imaging Met Self-Attention: A Love Story That Didn't Quite Work Out Tristan Piater Niklas Penzel Gideon Stein Joachim Denzler 37 2 0 18 Apr 2024
SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget Zihao Wang Shaoduo Gan 30 6 0 07 Apr 2024
Instruction-Driven Game Engines on Large Language Models Hongqiu Wu Xing-Chen Liu Haizhen Zhao Min Zhang 32 1 0 30 Mar 2024