REST: Retrieval-Based Speculative Decoding

14 November 2023

Tianle Cai

Papers citing "REST: Retrieval-Based Speculative Decoding"

50 / 59 papers shown

Title
PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding Bradley McDanel S. Zhang Y. Hu Zining Liu MoE 56 0 0 02 May 2025
Efficient Reasoning for LLMs through Speculative Chain-of-Thought Jikai Wang J. Li Lijun Wu M. Zhang LLMAG LRM 64 1 0 27 Apr 2025
PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation Zihao An Huajun Bai Z. Liu Dong Li E. Barsoum 54 0 0 23 Apr 2025
SpecServe: Efficient and SLO-Aware Large Language Model Serving with Adaptive Speculative Decoding Kaiyu Huang Hao Wu Zhubo Shi Han Zou Minchen Yu Qingjiang Shi LRM 36 1 0 07 Mar 2025
Speculative Decoding for Multi-Sample Inference Yiwei Li Jiayi Shi Shaoxiong Feng Peiwen Yuan X. Wang ... Ji Zhang Chuyi Tan Boyuan Pan Yao Hu Kan Li LRM 38 0 0 07 Mar 2025
RASD: Retrieval-Augmented Speculative Decoding Guofeng Quan Wenfeng Feng Chuzhan Hao Guochao Jiang Yuewei Zhang Hao Wang RALM 74 1 0 05 Mar 2025
DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting Kai Lv Honglin Guo Qipeng Guo Xipeng Qiu 39 0 0 02 Mar 2025
Tutorial Proposal: Speculative Decoding for Efficient LLM Inference Heming Xia Cunxiao Du Y. Li Qian Liu Wenjie Li 34 0 0 01 Mar 2025
Speculative Decoding and Beyond: An In-Depth Survey of Techniques Y. Hu Zining Liu Zhenyuan Dong Tianfan Peng Bradley McDanel S. Zhang 85 0 0 27 Feb 2025
From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence Generation up to 100K Tokens Tong Wu Junzhe Shen Zixia Jia Y. Wang Zilong Zheng 78 0 0 26 Feb 2025
Towards Optimal Multi-draft Speculative Decoding Z. Hu Tong Zheng Vignesh Viswanathan Ziyi Chen Ryan Rossi Yihan Wu Dinesh Manocha Heng Huang 42 3 0 26 Feb 2025
LongSpec: Long-Context Speculative Decoding with Efficient Drafting and Verification Penghui Yang Cunxiao Du Fengzhuo Zhang Haonan Wang Tianyu Pang Chao Du Bo An RALM 45 0 0 24 Feb 2025
CodeSwift: Accelerating LLM Inference for Efficient Code Generation Qianhui Zhao L. Zhang Fang Liu Xiaoli Lian Qiaoyuanhe Meng Ziqian Jiao Zetong Zhou Borui Zhang Runlin Guo Jia Li 41 0 0 24 Feb 2025
Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding Tian Jin Ellie Y. Cheng Zack Ankner Nikunj Saunshi Blake M. Elias Amir Yazdanbakhsh Jonathan Ragan-Kelley Suvinay Subramanian Michael Carbin 52 2 0 24 Feb 2025
GRIFFIN: Effective Token Alignment for Faster Speculative Decoding Shijing Hu Jingyang Li Xingyu Xie Zhihui Lu Kim-Chuan Toh Pan Zhou 38 0 0 16 Feb 2025
Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding Z. Wang Muneeza Azmart Ang Li R. Horesh Mikhail Yurochkin 107 1 0 11 Feb 2025
Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding Sukmin Cho S. Choi T. Hwang Jeongyeon Seo Soyeong Jeong Huije Lee Hoyun Song Jong C. Park Youngjin Kwon 51 0 0 08 Feb 2025
Constrained Decoding with Speculative Lookaheads Nishanth Nakshatri Shamik Roy Rajarshi Das Suthee Chaidaroon Leonid Boytsov Rashmi Gangadharaiah 72 0 0 09 Dec 2024
SAM Decoding: Speculative Decoding via Suffix Automaton Yuxuan Hu Ke Wang Jing Zhang Fanjin Zhang C. Li H. Chen Jing Zhang 42 1 0 16 Nov 2024
The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation Lawrence Stewart Matthew Trager Sujan Kumar Gonugondla Stefano Soatto 45 2 0 06 Nov 2024
Privacy Risks of Speculative Decoding in Large Language Models Jiankun Wei Abdulrahman Abdulrazzag Tianchen Zhang Adel Muursepp Gururaj Saileshwar 33 2 0 01 Nov 2024
Interpretable Language Modeling via Induction-head Ngram Models Eunji Kim Sriya Mantena Weiwei Yang Chandan Singh Sungroh Yoon Jianfeng Gao 44 0 0 31 Oct 2024
A Theoretical Perspective for Speculative Decoding Algorithm Ming Yin Minshuo Chen Kaixuan Huang Mengdi Wang 32 4 0 30 Oct 2024
AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration Bradley McDanel LRM 20 2 0 22 Oct 2024
DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure Yunfan Xiong Ruoyu Zhang Yanzeng Li Tianhao Wu Lei Zou 18 5 0 15 Oct 2024
CursorCore: Assist Programming through Aligning Anything Hao Jiang Qi Liu Rui Li Shengyu Ye Shijin Wang 48 1 0 09 Oct 2024
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration Heming Xia Yongqi Li Jun Zhang Cunxiao Du Wenjie Li LRM 44 4 0 09 Oct 2024
ParallelSpec: Parallel Drafter for Efficient Speculative Decoding Zilin Xiao Hongming Zhang Tao Ge Siru Ouyang Vicente Ordonez Dong Yu 39 5 0 08 Oct 2024
Mixture of Attentions For Speculative Decoding Matthieu Zimmer Milan Gritta Gerasimos Lampouras Haitham Bou Ammar Jun Wang 74 4 0 04 Oct 2024
CREST: Effectively Compacting a Datastore For Retrieval-Based Speculative Decoding Sophia Ho Jinsol Park Patrick Wang 24 0 0 08 Aug 2024
Let the Code LLM Edit Itself When You Edit the Code Zhenyu He Jun Zhang Shengjie Luo Jingjing Xu Z. Zhang Di He KELM 31 0 0 03 Jul 2024
OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure Jikai Wang Yi Su Juntao Li Qingrong Xia Zi Ye Xinyu Duan Zhefeng Wang Min Zhang 29 11 0 25 Jun 2024
SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices Ruslan Svirschevski Avner May Zhuoming Chen Beidi Chen Zhihao Jia Max Ryabinin 23 12 0 04 Jun 2024
SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths Kaixuan Huang Xudong Guo Mengdi Wang 32 17 0 30 May 2024
Nearest Neighbor Speculative Decoding for LLM Generation and Attribution Minghan Li Xilun Chen Ari Holtzman Beidi Chen Jimmy Lin Wen-tau Yih Xi Victoria Lin RALM BDL 108 10 0 29 May 2024
Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference Hao Chen Wayne Luk Ka-Fai Cedric Yiu Rui Li Konstantin Mishchenko Stylianos I. Venieris Hongxiang Fan 34 7 0 28 May 2024
Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting Fangcheng Liu Yehui Tang Zhenhua Liu Yunsheng Ni Kai Han Yunhe Wang 33 23 0 29 Apr 2024
Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models Chen Zhang Zhuorui Liu Dawei Song LRM 28 3 0 23 Apr 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 80 0 22 Apr 2024
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding Hanshi Sun Zhuoming Chen Xinyu Yang Yuandong Tian Beidi Chen 33 46 0 18 Apr 2024
SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens Chengbo Liu Yong Zhu 23 0 0 27 Mar 2024
Recurrent Drafter for Fast Speculative Decoding in Large Language Models Aonan Zhang Chong-Jun Wang Yi Wang Xuanyu Zhang Yunfei Cheng 26 15 0 14 Mar 2024
Reliable, Adaptable, and Attributable Language Models with Retrieval Akari Asai Zexuan Zhong Danqi Chen Pang Wei Koh Luke Zettlemoyer Hanna Hajishirzi Wen-tau Yih KELM RALM 36 53 0 05 Mar 2024
Accelerating Greedy Coordinate Gradient via Probe Sampling Yiran Zhao Wenyue Zheng Tianle Cai Xuan Long Do Kenji Kawaguchi Anirudh Goyal Michael Shieh 38 11 0 02 Mar 2024
Retrieval-Augmented Generation for AI-Generated Content: A Survey Penghao Zhao Hailin Zhang Qinhan Yu Zhengren Wang Yunteng Geng Fangcheng Fu Ling Yang Wentao Zhang Jie Jiang Bin Cui 3DV 110 220 0 29 Feb 2024
Grounding Language Models for Visual Entity Recognition Zilin Xiao Ming Gong Paola Cascante-Bonilla Xingyao Zhang Jie Wu Vicente Ordonez VLM 33 8 0 28 Feb 2024
LLM Inference Unveiled: Survey and Roofline Model Insights Zhihang Yuan Yuzhang Shang Yang Zhou Zhen Dong Zhe Zhou ... Yong Jae Lee Yan Yan Beidi Chen Guangyu Sun Kurt Keutzer 37 79 0 26 Feb 2024
HiRE: High Recall Approximate Top- $k$ Estimation for Efficient LLM Inference Yashas Samaga Varun Yerram Chong You Srinadh Bhojanapalli Sanjiv Kumar Prateek Jain Praneeth Netrapalli 44 4 0 14 Feb 2024
Tandem Transformers for Inference Efficient LLMs S. AishwaryaP Pranav Ajit Nair Yashas Samaga Toby Boyd Sanjiv Kumar Prateek Jain Praneeth Netrapalli 6 5 0 13 Feb 2024
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding Yichao Fu Peter Bailis Ion Stoica Hao Zhang 123 137 0 03 Feb 2024