DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

18 January 2024

Xin Jin

Papers citing "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving"

27 / 27 papers shown

Title
SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models Hang Wu Jianian Zhu Y. Li Haojie Wang Biao Hou Jidong Zhai 23 0 0 12 May 2025
RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM Inference Y. Chen J. Zhang Baotong Lu Qianxi Zhang Chengruidong Zhang ... Chen Chen Mingxing Zhang Yuqing Yang Fan Yang Mao Yang 32 0 0 05 May 2025
Taming the Titans: A Survey of Efficient LLM Inference Serving Ranran Zhen J. Li Yixin Ji Z. Yang Tong Liu Qingrong Xia Xinyu Duan Z. Wang Baoxing Huai M. Zhang LLMAG 77 0 0 28 Apr 2025
GenTorrent: Scaling Large Language Model Serving with An Overley Network Fei Fang Yifan Hua Shengze Wang Ruilin Zhou Y. Liu Chen Qian X. Zhang 46 0 0 27 Apr 2025
Tempo: Application-aware LLM Serving with Mixed SLO Requirements Wei Zhang Zhiyu Wu Yi Mu Banruo Liu Myungjin Lee Fan Lai 51 0 0 24 Apr 2025
Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management Hang Zhang Jiuchen Shi Yixiao Wang Quan Chen Yizhou Shan Minyi Guo 25 0 0 19 Apr 2025
Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents Yueying Li Jim Dai Tianyi Peng 47 1 0 10 Apr 2025
Mitigating KV Cache Competition to Enhance User Experience in LLM Inference Haiying Shen Tanmoy Sen Masahiro Tanaka 76 0 0 17 Mar 2025
eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference Suraiya Tairin Shohaib Mahmud Haiying Shen Anand Iyer MoE 61 0 0 10 Mar 2025
Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation Yingfeng Luo Tong Zheng Yongyu Mu B. Li Qinghong Zhang ... Ziqiang Xu Peinan Feng Xiaoqian Liu Tong Xiao Jingbo Zhu AI4CE 68 0 0 09 Mar 2025
Seesaw: High-throughput LLM Inference via Model Re-sharding Qidong Su Wei Zhao X. Li Muralidhar Andoorveedu Chenhao Jiang Zhanda Zhu Kevin Song Christina Giannoula Gennady Pekhimenko LRM 70 0 0 09 Mar 2025
The Best Instruction-Tuning Data are Those That Fit Dylan Zhang Qirun Dai Hao Peng ALM 113 3 0 06 Feb 2025
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 71 0 0 08 Jan 2025
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval Di Liu Meng Chen Baotong Lu Huiqiang Jiang Zhenhua Han ... K. Zhang C. L. P. Chen Fan Yang Y. Yang Lili Qiu 39 29 0 03 Jan 2025
Context Parallelism for Scalable Million-Token Inference Amy Yang Jingyi Yang Aya Ibrahim Xinfeng Xie Bangsheng Tang Grigory Sizov Jeremy Reizenstein Jongsoo Park Jianyu Huang MoE LRM 57 5 0 04 Nov 2024
EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference Yulei Qian Fengcun Li Xiangyang Ji Xiaoyu Zhao Jianchao Tan K. Zhang Xunliang Cai MoE 65 2 0 16 Oct 2024
Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices Yuxiang Huang Binhang Yuan Xu Han Chaojun Xiao Zhiyuan Liu RALM 73 1 0 02 Oct 2024
LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale Jaehong Cho Minsu Kim Hyunmin Choi Guseul Heo Jongse Park 35 8 0 10 Aug 2024
Teola: Towards End-to-End Optimization of LLM-based Applications Xin Tan Yimin Jiang Yitao Yang Hong-Yu Xu 54 4 0 29 Jun 2024
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion Jiayi Yao Hanchen Li Yuhan Liu Siddhant Ray Yihua Cheng Qizheng Zhang Kuntai Du Shan Lu Junchen Jiang 42 12 0 26 May 2024
Preble: Efficient Distributed Prompt Scheduling for LLM Serving Vikranth Srivatsa Zijian He Reyna Abhyankar Dongming Li Yiying Zhang 40 17 0 08 May 2024
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention Ramya Prabhu Ajay Nayak Jayashree Mohan R. Ramjee Ashish Panwar VLM 52 24 0 07 May 2024
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation Chao Jin Zili Zhang Xuanlin Jiang Fangyue Liu Xin Liu Xuanzhe Liu Xin Jin 32 36 0 18 Apr 2024
FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning Xupeng Miao Gabriele Oliaro Xinhao Cheng Vineeth Kada Ruohan Gao ... April Yang Yingcheng Wang Mengdi Wu Colin Unger Zhihao Jia MoE 88 8 0 29 Feb 2024
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU Yixin Song Zeyu Mi Haotong Xie Haibo Chen BDL 117 114 0 16 Dec 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 144 365 0 13 Mar 2023
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,791 0 17 Sep 2019