SpotServe: Serving Generative Large Language Models on Preemptible
Instances

SpotServe: Serving Generative Large Language Models on Preemptible Instances

27 November 2023

Dahua Lin

Papers citing "SpotServe: Serving Generative Large Language Models on Preemptible Instances"

13 / 13 papers shown

Title
Ascendra: Dynamic Request Prioritization for Efficient LLM Serving Azam Ikram Xiang Li Sameh Elnikety S. Bagchi 75 0 0 29 Apr 2025
Tempo: Application-aware LLM Serving with Mixed SLO Requirements Wei Zhang Zhiyu Wu Yi Mu Banruo Liu Myungjin Lee Fan Lai 51 0 0 24 Apr 2025
Seesaw: High-throughput LLM Inference via Model Re-sharding Qidong Su Wei Zhao X. Li Muralidhar Andoorveedu Chenhao Jiang Zhanda Zhu Kevin Song Christina Giannoula Gennady Pekhimenko LRM 72 0 0 09 Mar 2025
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 127 0 0 08 Jan 2025
SkyServe: Serving AI Models across Regions and Clouds with Spot Instances Ziming Mao Tian Xia Zhanghao Wu Wei-Lin Chiang Tyler Griggs Romil Bhardwaj Zongheng Yang S. Shenker Ion Stoica 44 1 0 03 Nov 2024
LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale Jaehong Cho Minsu Kim Hyunmin Choi Guseul Heo Jongse Park 38 9 0 10 Aug 2024
Teola: Towards End-to-End Optimization of LLM-based Applications Xin Tan Yimin Jiang Yitao Yang Hong-Yu Xu 59 5 0 29 Jun 2024
FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning Xupeng Miao Gabriele Oliaro Xinhao Cheng Vineeth Kada Ruohan Gao ... April Yang Yingcheng Wang Mengdi Wu Colin Unger Zhihao Jia MoE 88 9 0 29 Feb 2024
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 144 366 0 13 Mar 2023
Varuna: Scalable, Low-cost Training of Massive Deep Learning Models Sanjith Athlur Nitika Saran Muthian Sivathanu R. Ramjee Nipun Kwatra GNN 31 80 0 07 Nov 2021
What Makes Good In-Context Examples for GPT- $3$ ? Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin Weizhu Chen AAML RALM 275 1,312 0 17 Jan 2021
Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider Mohammad Shahrad Rodrigo Fonseca Íñigo Goiri G. Chaudhry Paul Batum Jason Cooke Eduardo Laureano Colby Tresness M. Russinovich Ricardo Bianchini 79 601 0 06 Mar 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,817 0 17 Sep 2019