Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding

15 January 2024

Heming Xia

Zhe Yang

Qingxiu Dong

Peiyi Wang

Zhifang Sui

Papers citing "Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding"

50 / 84 papers shown

Title
Accelerating Large Language Model Reasoning via Speculative Search Zhihai Wang Jie Wang Jilai Pan Xilin Xia Huiling Zhen M. Yuan Jianye Hao Feng Wu ReLM LRM 54 0 0 03 May 2025
Taming the Titans: A Survey of Efficient LLM Inference Serving Ranran Zhen J. Li Yixin Ji Z. Yang Tong Liu Qingrong Xia Xinyu Duan Z. Wang Baoxing Huai M. Zhang LLMAG 77 0 0 28 Apr 2025
Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks Yang Janet Liu Bingjie Yan Tianyuan Zou Jianqing Zhang Zixuan Gu ... J. Li Xiaozhou Ye Ye Ouyang Qiang Yang Y. Zhang ALM 51 0 0 24 Apr 2025
Jupiter: Fast and Resource-Efficient Collaborative Inference of Generative LLMs on Edge Devices Shengyuan Ye Bei Ouyang Liekang Zeng Tianyi Qian Xiaowen Chu Jian Tang Xu Chen 24 0 0 11 Apr 2025
SD $^2$ : Self-Distilled Sparse Drafters Mike Lasby Nish Sinnadurai Valavan Manohararajah Sean Lie Vithursan Thangarasa 46 0 0 10 Apr 2025
SPIRe: Boosting LLM Inference Throughput with Speculative Decoding Sanjit Neelam Daniel Heinlein Vaclav Cvicek Akshay Mishra Reiner Pope LRM 31 0 0 08 Apr 2025
DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding Hossein Entezari Zarch Lei Gao Chaoyi Jiang Murali Annavaram LRM 26 0 0 08 Apr 2025
PipeDec: Low-Latency Pipeline-based Inference with Dynamic Speculative Decoding towards Large-scale Models Haofei Yin Mengbai Xiao Rouzhou Lu Xiao Zhang Dongxiao Yu Guanghui Zhang AI4CE 19 0 0 05 Apr 2025
Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding Aayush Gautam Susav Shrestha Narasimha Annapareddy 41 0 0 28 Mar 2025
A Novel Hat-Shaped Device-Cloud Collaborative Inference Framework for Large Language Models Zuan Xie Yang Xu Hongli Xu Yunming Liao Zhiwei Yao 49 0 0 23 Mar 2025
ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts E. Georganas Dhiraj D. Kalamkar Alexander Kozlov A. Heinecke MQ 36 0 0 17 Mar 2025
Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding J. Li Yixing Xu Haiduo Huang Xuanwu Yin D. Li Edith C. -H. Ngai E. Barsoum 45 0 0 13 Mar 2025
Collaborative Speculative Inference for Efficient LLM Inference Serving Luyao Gao Jianchun Liu Hongli Xu Liusheng Huang 33 0 0 13 Mar 2025
SpecServe: Efficient and SLO-Aware Large Language Model Serving with Adaptive Speculative Decoding Kaiyu Huang Hao Wu Zhubo Shi Han Zou Minchen Yu Qingjiang Shi LRM 36 0 0 07 Mar 2025
EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test Yuhui Li Fangyun Wei Chao Zhang Hongyang R. Zhang 106 3 0 03 Mar 2025
DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting Kai Lv Honglin Guo Qipeng Guo Xipeng Qiu 37 0 0 02 Mar 2025
Tutorial Proposal: Speculative Decoding for Efficient LLM Inference Heming Xia Cunxiao Du Y. Li Qian Liu Wenjie Li 34 0 0 01 Mar 2025
TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding Zhaoxuan Wu Zijian Zhou Arun Verma Alok Prakash Daniela Rus Bryan Kian Hsiang Low 58 0 0 24 Feb 2025
DReSD: Dense Retrieval for Speculative Decoding Milan Gritta Huiyin Xue Gerasimos Lampouras RALM 90 0 0 24 Feb 2025
Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding Z. Wang Muneeza Azmart Ang Li R. Horesh Mikhail Yurochkin 104 0 0 11 Feb 2025
Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding Sukmin Cho S. Choi T. Hwang Jeongyeon Seo Soyeong Jeong Huije Lee Hoyun Song Jong C. Park Youngjin Kwon 46 0 0 08 Feb 2025
Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree Xiangxiang Gao Weisheng Xie Yiwei Xiang Feng Ji 70 5 0 17 Dec 2024
Constrained Decoding with Speculative Lookaheads Nishanth Nakshatri Shamik Roy Rajarshi Das Suthee Chaidaroon Leonid Boytsov Rashmi Gangadharaiah 72 0 0 09 Dec 2024
PLD+: Accelerating LLM inference by leveraging Language Model Artifacts Shwetha Somasundaram Anirudh Phukan Apoorv Saxena 77 1 0 02 Dec 2024
Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration Zhuofan Wen Shangtong Gui Yang Feng 93 2 0 25 Nov 2024
Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding Hyun Ryu Eric Kim 72 3 0 20 Nov 2024
SAM Decoding: Speculative Decoding via Suffix Automaton Yuxuan Hu Ke Wang Jing Zhang Fanjin Zhang C. Li H. Chen Jing Zhang 39 1 0 16 Nov 2024
SSSD: Simply-Scalable Speculative Decoding Michele Marzollo Jiawei Zhuang Niklas Roemer Lorenz K. Müller Lukas Cavigelli LRM 26 1 0 08 Nov 2024
Privacy Risks of Speculative Decoding in Large Language Models Jiankun Wei Abdulrahman Abdulrazzag Tianchen Zhang Adel Muursepp Gururaj Saileshwar 28 2 0 01 Nov 2024
LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators Krishna Teja Chitty-Venkata Siddhisanket Raskar B. Kale Farah Ferdaus Aditya Tanikanti Ken Raffenetti Valerie Taylor M. Emani V. Vishwanath 39 4 0 31 Oct 2024
A Theoretical Perspective for Speculative Decoding Algorithm Ming Yin Minshuo Chen Kaixuan Huang Mengdi Wang 27 3 0 30 Oct 2024
Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspective Shenghao Xie Wenqiang Zu Mingyang Zhao Duo Su Shilong Liu Ruohua Shi Guoqi Li Shanghang Zhang Lei Ma LRM 40 3 0 29 Oct 2024
Dynamic layer selection in decoder-only transformers Theodore Glavas Joud Chataoui Florence Regol Wassim Jabbour Antonios Valkanas Boris N. Oreshkin Mark J. Coates AI4CE 19 0 0 26 Oct 2024
AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration Bradley McDanel LRM 20 2 0 22 Oct 2024
Cerberus: Efficient Inference with Adaptive Parallel Decoding and Sequential Knowledge Enhancement Yuxuan Liu Wenyuan Li Laizhong Cui Hailiang Yang OffRL 18 0 0 17 Oct 2024
Self-Data Distillation for Recovering Quality in Pruned Large Language Models Vithursan Thangarasa Ganesh Venkatesh Mike Lasby Nish Sinnadurai Sean Lie SyDa 33 0 0 13 Oct 2024
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration Heming Xia Yongqi Li Jun Zhang Cunxiao Du Wenjie Li LRM 36 4 0 09 Oct 2024
Root Defence Strategies: Ensuring Safety of LLM at the Decoding Level Xinyi Zeng Yuying Shang Yutao Zhu Jingyuan Zhang Yu Tian AAML 40 2 0 09 Oct 2024
ParallelSpec: Parallel Drafter for Efficient Speculative Decoding Zilin Xiao Hongming Zhang Tao Ge Siru Ouyang Vicente Ordonez Dong Yu 36 5 0 08 Oct 2024
Efficient Inference for Large Language Model-based Generative Recommendation Xinyu Lin Chaoqun Yang Wenjie Wang Yongqi Li Cunxiao Du Fuli Feng See-Kiong Ng Tat-Seng Chua 56 4 0 07 Oct 2024
SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation Aurick Qiao Z. Yao Samyam Rajbhandari Yuxiong He VLM 16 0 0 04 Oct 2024
Mixture of Attentions For Speculative Decoding Matthieu Zimmer Milan Gritta Gerasimos Lampouras Haitham Bou Ammar Jun Wang 63 4 0 04 Oct 2024
Speculative Coreset Selection for Task-Specific Fine-tuning Xiaoyu Zhang Juan Zhai Shiqing Ma Chao Shen Tianlin Li Weipeng Jiang Yang Liu 20 1 0 02 Oct 2024
House of Cards: Massive Weights in LLMs Jaehoon Oh Seungjun Shin Dokwan Oh 35 1 0 02 Oct 2024
Whisper in Medusa's Ear: Multi-head Efficient Decoding for Transformer-based ASR Yael Segal-Feldman Aviv Shamsian Aviv Navon Gill Hetz Joseph Keshet 20 1 0 24 Sep 2024
A-VL: Adaptive Attention for Large Vision-Language Models Junyang Zhang Mu Yuan Ruiguang Zhong Puhan Luo Huiyou Zhan Ningkang Zhang Chengchen Hu Xiangyang Li VLM 36 1 0 23 Sep 2024
What is the Role of Small Models in the LLM Era: A Survey Lihu Chen Gaël Varoquaux ALM 50 23 0 10 Sep 2024
Boosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillation Lujun Gui Bin Xiao Lei Su Weipeng Chen 17 2 0 28 Aug 2024
Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling Xianzhen Luo Yixuan Wang Qingfu Zhu Zhiming Zhang Xuanyu Zhang Qing Yang Dongliang Xu Wanxiang Che 18 3 0 16 Aug 2024
Context-Aware Assistant Selection for Improved Inference Acceleration with Large Language Models Jerry Huang Prasanna Parthasarathi Mehdi Rezagholizadeh Sarath Chandar 43 0 0 16 Aug 2024