DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

30 June 2022

Reza Yazdani Aminabadi

Yuxiong He

Papers citing "DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale"

42 / 42 papers shown

Title
FloE: On-the-Fly MoE Inference on Memory-constrained GPU Yuxin Zhou Zheng Li J. Zhang Jue Wang Y. Wang Zhongle Xie Ke Chen Lidan Shou MoE 43 0 0 09 May 2025
PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding Bradley McDanel S. Zhang Y. Hu Zining Liu MoE 56 0 0 02 May 2025
Taming the Titans: A Survey of Efficient LLM Inference Serving Ranran Zhen J. Li Yixin Ji Z. Yang Tong Liu Qingrong Xia Xinyu Duan Z. Wang Baoxing Huai M. Zhang LLMAG 77 0 0 28 Apr 2025
Tempo: Application-aware LLM Serving with Mixed SLO Requirements Wei Zhang Zhiyu Wu Yi Mu Banruo Liu Myungjin Lee Fan Lai 51 0 0 24 Apr 2025
Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models Teng Wang Zhangyi Jiang Zhenqi He Wenhan Yang Yanan Zheng Zeyu Li Zifan He Shenyang Tong Hailei Gong LRM 90 1 0 16 Mar 2025
Attention Condensation via Sparsity Induced Regularized Training Eli Sason Darya Frolova Boris Nazarov Felix Goldberd 113 0 0 03 Mar 2025
Importance Sampling via Score-based Generative Models Heasung Kim Taekyun Lee Hyeji Kim Gustavo de Veciana MedIm DiffM 125 0 0 07 Feb 2025
HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location Ting Sun Penghan Wang Fan Lai 86 1 0 15 Jan 2025
Merging Feed-Forward Sublayers for Compressed Transformers Neha Verma Kenton W. Murray Kevin Duh AI4CE 45 0 0 10 Jan 2025
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 92 0 0 08 Jan 2025
LoL-PIM: Long-Context LLM Decoding with Scalable DRAM-PIM System Hyucksung Kwon Kyungmo Koo Janghyeon Kim W. Lee Minjae Lee ... Yongkee Kwon Ilkon Kim Euicheol Lim John Kim Jungwook Choi 66 4 0 28 Dec 2024
DynamicControl: Adaptive Condition Selection for Improved Text-to-Image Generation Q. He Jinlong Peng P. Xu Boyuan Jiang Xiaobin Hu ... Y. Liu Y. Wang Chengjie Wang X. Li J. Zhang DiffM 120 1 0 04 Dec 2024
Unifying KV Cache Compression for Large Language Models with LeanKV Yanqi Zhang Yuwei Hu Runyuan Zhao John C. S. Lui Haibo Chen MQ 103 5 0 04 Dec 2024
Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective Jinhao Li Jiaming Xu Shan Huang Yonghua Chen Wen Li ... Jiayi Pan Li Ding Hao Zhou Yu Wang Guohao Dai 57 15 0 06 Oct 2024
Frame-Voyager: Learning to Query Frames for Video Large Language Models Sicheng Yu Chengkai Jin Huanyu Wang Zhenghao Chen Sheng Jin ... Zhenbang Sun Bingni Zhang Jiawei Wu Hao Zhang Qianru Sun 67 5 0 04 Oct 2024
InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference Xiurui Pan Endian Li Qiao Li Shengwen Liang Yizhou Shan Ke Zhou Yingwei Luo Xiaolin Wang Jie Zhang 28 10 0 08 Sep 2024
SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation Yi-Chia Chen Wei-Hua Li Cheng Sun Yu-Chiang Frank Wang Chu-Song Chen VLM 30 10 0 01 Sep 2024
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding Jian Chen Vashisth Tiwari Ranajoy Sadhukhan Zhuoming Chen Jinyuan Shi Ian En-Hsu Yen Ian En-Hsu Yen Avner May Tianqi Chen Beidi Chen LRM 31 22 0 20 Aug 2024
LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference Qichen Fu Minsik Cho Thomas Merth Sachin Mehta Mohammad Rastegari Mahyar Najibi 33 25 0 19 Jul 2024
Sparse Matrix in Large Language Model Fine-tuning Haoze He Juncheng Billy Li Xuan Jiang Heather Miller MoE 19 3 0 24 May 2024
PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference Dongjie Yang Xiaodong Han Yan Gao Yao Hu Shilin Zhang Hai Zhao 27 49 0 21 May 2024
Preble: Efficient Distributed Prompt Scheduling for LLM Serving Vikranth Srivatsa Zijian He Reyna Abhyankar Dongming Li Yiying Zhang 40 17 0 08 May 2024
Decoding Speculative Decoding Minghao Yan Saurabh Agarwal Shivaram Venkataraman LRM 25 5 0 02 Feb 2024
AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference Xuanlei Zhao Shenggan Cheng Guangyang Lu Jiarui Fang Hao Zhou Bin Jia Ziming Liu Yang You MQ 12 3 0 19 Jan 2024
Train ñ Trade: Foundations of Parameter Markets Tzu-Heng Huang Harit Vishwakarma Frederic Sala AIFin 16 2 0 07 Dec 2023
Splitwise: Efficient generative LLM inference using phase splitting Pratyush Patel Esha Choukse Chaojie Zhang Aashaka Shah Íñigo Goiri Saeed Maleki Ricardo Bianchini 36 196 0 30 Nov 2023
Secure Transformer Inference Protocol Mu Yuan Lan Zhang Xiang-Yang Li 22 3 0 14 Nov 2023
Prompt Cache: Modular Attention Reuse for Low-Latency Inference In Gim Guojun Chen Seung-seob Lee Nikhil Sarda Anurag Khandelwal Lin Zhong 25 71 0 07 Nov 2023
Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language Eghbal A. Hosseini Evelina Fedorenko LLMSV 15 4 0 05 Nov 2023
Punica: Multi-Tenant LoRA Serving Lequn Chen Zihao Ye Yongji Wu Danyang Zhuo Luis Ceze Arvind Krishnamurthy 34 34 0 28 Oct 2023
Norm Tweaking: High-performance Low-bit Quantization of Large Language Models Liang Li Qingyuan Li Bo-Wen Zhang Xiangxiang Chu MQ 25 28 0 06 Sep 2023
ChatGPT in the Age of Generative AI and Large Language Models: A Concise Survey S. Mohamadi G. Mujtaba Ngan Le Gianfranco Doretto Don Adjeroh LM&MA AI4MH 16 21 0 09 Jul 2023
$S$^{3}$: Increasing GPU Utilization during Generative Inference for Higher Throughput$ S $^{3}$ : Increasing GPU Utilization during Generative Inference for Higher Throughput Yunho Jin Chun-Feng Wu David Brooks Gu-Yeon Wei 23 62 0 09 Jun 2023
Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline Zangwei Zheng Xiaozhe Ren Fuzhao Xue Yang Luo Xin Jiang Yang You 24 54 0 22 May 2023
Prompting with Pseudo-Code Instructions Mayank Mishra Prince Kumar Riyaz Ahmad Bhat V. Rudramurthy Danish Contractor Srikanth G. Tamilselvam 30 13 0 19 May 2023
RPTQ: Reorder-based Post-training Quantization for Large Language Models Zhihang Yuan Lin Niu Jia-Wen Liu Wenyu Liu Xinggang Wang Yuzhang Shang Guangyu Sun Qiang Wu Jiaxiang Wu Bingzhe Wu MQ 25 75 0 03 Apr 2023
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment Jared Fernandez Jacob Kahn Clara Na Yonatan Bisk Emma Strubell FedML 16 10 0 13 Feb 2023
Efficiently Scaling Transformer Inference Reiner Pope Sholto Douglas Aakanksha Chowdhery Jacob Devlin James Bradbury Anselm Levskaya Jonathan Heek Kefan Xiao Shivani Agrawal J. Dean 21 292 0 09 Nov 2022
Desiderata for next generation of ML model serving Sherif Akoush Andrei Paleyes A. V. Looveren Clive Cox 8 5 0 26 Oct 2022
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale Samyam Rajbhandari Conglong Li Z. Yao Minjia Zhang Reza Yazdani Aminabadi A. A. Awan Jeff Rasley Yuxiong He 19 282 0 14 Jan 2022
Scalable and Efficient MoE Training for Multitask Multilingual Models Young Jin Kim A. A. Awan Alexandre Muzio Andres Felipe Cruz Salinas Liyang Lu Amr Hendy Samyam Rajbhandari Yuxiong He Hany Awadalla MoE 94 83 0 22 Sep 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,815 0 17 Sep 2019