FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines

18 March 2024

Papers citing "FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines"

19 / 19 papers shown

Title
Taming the Titans: A Survey of Efficient LLM Inference Serving Ranran Zhen J. Li Yixin Ji Z. Yang Tong Liu Qingrong Xia Xinyu Duan Z. Wang Baoxing Huai M. Zhang LLMAG 77 0 0 28 Apr 2025
L3: DIMM-PIM Integrated Architecture and Coordination for Scalable Long-Context LLM Inference Qingyuan Liu Liyan Chen Yanning Yang H. Wang Dong Du Zhigang Mao Naifeng Jing Yubin Xia Haibo Chen 29 0 0 24 Apr 2025
HPU: High-Bandwidth Processing Unit for Scalable, Cost-effective LLM Inference via GPU Co-processing Myunghyun Rhee Joonseop Sim Taeyoung Ahn Seungyong Lee Daegun Yoon Euiseok Kim Kyoung Park Youngpyo Joo Hosik Kim 22 0 0 18 Apr 2025
Cognitive Memory in Large Language Models Lianlei Shan Shixian Luo Zezhou Zhu Yu Yuan Yong Wu LLMAG KELM 104 1 0 03 Apr 2025
Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation Yunkai Liang Zhangyu Chen Pengfei Zuo Zhi Zhou Xu Chen Zhou Yu 86 2 0 26 Mar 2025
LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference G. Wang Shubhangi Upasani Chen Henry Wu Darshan Gandhi Jonathan Li Changran Hu Bo Li Urmish Thakker 77 0 0 11 Mar 2025
Seesaw: High-throughput LLM Inference via Model Re-sharding Qidong Su Wei Zhao X. Li Muralidhar Andoorveedu Chenhao Jiang Zhanda Zhu Kevin Song Christina Giannoula Gennady Pekhimenko LRM 72 0 0 09 Mar 2025
HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading Cheng Luo Zefan Cai Hanshi Sun Jinqi Xiao Bo Yuan Wen Xiao Junjie Hu Jiawei Zhao Beidi Chen Anima Anandkumar 64 1 0 18 Feb 2025
Tensor Product Attention Is All You Need Yifan Zhang Yifeng Liu Huizhuo Yuan Zhen Qin Yang Yuan Q. Gu Andrew Chi-Chih Yao 75 9 0 11 Jan 2025
LoL-PIM: Long-Context LLM Decoding with Scalable DRAM-PIM System Hyucksung Kwon Kyungmo Koo Janghyeon Kim W. Lee Minjae Lee ... Yongkee Kwon Ilkon Kim Euicheol Lim John Kim Jungwook Choi 66 4 0 28 Dec 2024
BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching Yilong Zhao Shuo Yang Kan Zhu Lianmin Zheng Baris Kasikci Yang Zhou Jiarong Xing Ion Stoica 109 5 0 25 Nov 2024
NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference Xuanlin Jiang Yang Zhou Shiyi Cao Ion Stoica Minlan Yu 37 8 0 02 Nov 2024
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference Hanshi Sun Li-Wen Chang Wenlei Bao Size Zheng Ningxin Zheng Xin Liu Harry Dong Yuejie Chi Beidi Chen VLM 88 16 0 28 Oct 2024
InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference Xiurui Pan Endian Li Qiao Li Shengwen Liang Yizhou Shan Ke Zhou Yingwei Luo Xiaolin Wang Jie Zhang 33 10 0 08 Sep 2024
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption Shi Luohe Hongyi Zhang Yao Yao Z. Li Zhao Hai 31 31 0 25 Jul 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 80 0 22 Apr 2024
FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs Shulin Zeng Jun Liu Guohao Dai Xinhao Yang Tianyu Fu ... Zehao Wang Ruoyu Zhang Kairui Wen Xuefei Ning Yu Wang 54 55 0 08 Jan 2024
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU Yixin Song Zeyu Mi Haotong Xie Haibo Chen BDL 120 120 0 16 Dec 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 144 366 0 13 Mar 2023