EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

26 January 2024

Papers citing "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty"

50 / 100 papers shown

Title
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration Heming Xia Yongqi Li Jun Zhang Cunxiao Du Wenjie Li LRM 36 4 0 09 Oct 2024
Root Defence Strategies: Ensuring Safety of LLM at the Decoding Level Xinyi Zeng Yuying Shang Yutao Zhu Jingyuan Zhang Yu Tian AAML 40 2 0 09 Oct 2024
ParallelSpec: Parallel Drafter for Efficient Speculative Decoding Zilin Xiao Hongming Zhang Tao Ge Siru Ouyang Vicente Ordonez Dong Yu 30 5 0 08 Oct 2024
Efficient Inference for Large Language Model-based Generative Recommendation Xinyu Lin Chaoqun Yang Wenjie Wang Yongqi Li Cunxiao Du Fuli Feng See-Kiong Ng Tat-Seng Chua 56 4 0 07 Oct 2024
Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective Jinhao Li Jiaming Xu Shan Huang Yonghua Chen Wen Li ... Jiayi Pan Li Ding Hao Zhou Yu Wang Guohao Dai 36 13 0 06 Oct 2024
LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding Doohyuk Jang Sihwan Park J. Yang Yeonsung Jung Jihun Yun Souvik Kundu Sung-Yub Kim Eunho Yang 33 7 0 04 Oct 2024
Mixture of Attentions For Speculative Decoding Matthieu Zimmer Milan Gritta Gerasimos Lampouras Haitham Bou Ammar Jun Wang 63 4 0 04 Oct 2024
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding Yao Teng Han Shi Xian Liu Xuefei Ning Guohao Dai Yu Wang Zhenguo Li Xihui Liu 46 10 0 02 Oct 2024
Approximately Aligned Decoding Daniel Melcer Sujan Kumar Gonugondla Pramuditha Perera Haifeng Qian Wen-Hao Chiang Yanjun Wang Nihal Jain Pranav Garg Xiaofei Ma Anoop Deoras 31 0 0 01 Oct 2024
Improving Multi-candidate Speculative Decoding Xiaofan Lu Yixiao Zeng Feiyang Ma Zixu Yu Marco Levorato 21 0 0 16 Sep 2024
Dynamic Depth Decoding: Faster Speculative Decoding for LLMs Oscar Brown Zhengjie Wang Andrea Do Nikhil Mathew Cheng Yu 16 3 0 30 Aug 2024
Learning Harmonized Representations for Speculative Sampling Lefan Zhang Xiaodan Wang Yanhua Huang Ruiwen Xu 16 9 0 28 Aug 2024
Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling Xianzhen Luo Yixuan Wang Qingfu Zhu Zhiming Zhang Xuanyu Zhang Qing Yang Dongliang Xu Wanxiang Che 16 3 0 16 Aug 2024
Context-Aware Assistant Selection for Improved Inference Acceleration with Large Language Models Jerry Huang Prasanna Parthasarathi Mehdi Rezagholizadeh Sarath Chandar 38 0 0 16 Aug 2024
KOALA: Enhancing Speculative Decoding for LLM via Multi-Layer Draft Heads with Adversarial Learning Kaiqi Zhang Jing Zhao Rui Chen 24 1 0 15 Aug 2024
Efficiency Unleashed: Inference Acceleration for LLM-based Recommender Systems with Speculative Decoding Yunjia Xi Hangyu Wang Bo Chen Jianghao Lin Menghui Zhu W. Liu Ruiming Tang Zhewei Wei W. Zhang Yong Yu OffRL 78 4 0 11 Aug 2024
Let the Code LLM Edit Itself When You Edit the Code Zhenyu He Jun Zhang Shengjie Luo Jingjing Xu Z. Zhang Di He KELM 21 0 0 03 Jul 2024
S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models Parsa Kavehzadeh Mohammadreza Pourreza Mojtaba Valipour Tinashu Zhu Haoli Bai Ali Ghodsi Boxing Chen Mehdi Rezagholizadeh 22 0 0 02 Jul 2024
Adaptive Draft-Verification for Efficient Large Language Model Decoding Xukun Liu Bowen Lei Ruqi Zhang Dongkuan Xu 22 3 0 27 Jun 2024
SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding Zhenglin Wang Jialong Wu Yilong Lai Congzhi Zhang Deyu Zhou LRM ReLM 20 1 0 26 Jun 2024
Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training Yixuan Wang Xianzhen Luo Fuxuan Wei Yijun Liu Qingfu Zhu Xuanyu Zhang Qing Yang Dongliang Xu Wanxiang Che 27 1 0 25 Jun 2024
OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure Jikai Wang Yi Su Juntao Li Qingrong Xia Zi Ye Xinyu Duan Zhefeng Wang Min Zhang 26 11 0 25 Jun 2024
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees Yuhui Li Fangyun Wei Chao Zhang Hongyang R. Zhang 75 1 0 24 Jun 2024
Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters Euiin Yi Taehyeon Kim Hongseok Jeung Du-Seong Chang Se-Young Yun 30 4 0 24 Jun 2024
Optimizing Speculative Decoding for Serving Large Language Models Using Goodput Xiaoxuan Liu Cade Daniel Langxiang Hu Woosuk Kwon Zhuohan Li Xiangxi Mo Alvin Cheung Zhijie Deng Ion Stoica Hao Zhang 18 13 0 20 Jun 2024
PowerInfer-2: Fast Large Language Model Inference on a Smartphone Zhenliang Xue Yixin Song Zeyu Mi Le Chen Yubin Xia Haibo Chen 38 1 0 10 Jun 2024
Block Transformer: Global-to-Local Language Modeling for Fast Inference Namgyu Ho Sangmin Bae Taehyeon Kim Hyunjik Jo Yireun Kim Tal Schuster Adam Fisch James Thorne Se-Young Yun 32 6 0 04 Jun 2024
S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUs Wei Zhong Manasa Bharadwaj 23 5 0 30 May 2024
SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths Kaixuan Huang Xudong Guo Mengdi Wang 19 16 0 30 May 2024
Faster Cascades via Speculative Decoding Harikrishna Narasimhan Wittawat Jitkrittum A. S. Rawat Seungyeon Kim Neha Gupta A. Menon Sanjiv Kumar LRM 28 4 0 29 May 2024
Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference Hao Chen Wayne Luk Ka-Fai Cedric Yiu Rui Li Konstantin Mishchenko Stylianos I. Venieris Hongxiang Fan 26 4 0 28 May 2024
Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs Qingyuan Li Ran Meng Yiduo Li Bo Zhang Yifan Lu Yerui Sun Lin Ma Yuchen Xie MQ 25 0 0 23 May 2024
Atomic Self-Consistency for Better Long Form Generations Raghuveer Thirukovalluru Yukun Huang Bhuwan Dhingra 25 3 0 21 May 2024
EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models Yunsheng Ni Chuanjian Liu Yehui Tang Kai Han Yunhe Wang 13 0 0 13 May 2024
Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models Jonathan Mamou Oren Pereg Daniel Korat Moshe Berchansky Nadav Timor Moshe Wasserblat Roy Schwartz 28 2 0 07 May 2024
Accelerating Production LLMs with Combined Token/Embedding Speculators Davis Wertheimer Joshua Rosenkranz Thomas Parnell Sahil Suneja Pavithra Ranganathan R. Ganti M. Srivatsa 19 3 0 29 Apr 2024
Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting Fangcheng Liu Yehui Tang Zhenhua Liu Yunsheng Ni Kai Han Yunhe Wang 22 21 0 29 Apr 2024
BASS: Batched Attention-optimized Speculative Sampling Haifeng Qian Sujan Kumar Gonugondla Sungsoo Ha Mingyue Shang Sanjay Krishna Gouda Ramesh Nallapati Sudipta Sengupta Xiaofei Ma Anoop Deoras BDL 34 8 0 24 Apr 2024
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding Hanshi Sun Zhuoming Chen Xinyu Yang Yuandong Tian Beidi Chen 25 41 0 18 Apr 2024
Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs Luchang Li Sheng Qian Jie Lu Lunxi Yuan Rui Wang Qin Xie 28 5 0 29 Mar 2024
SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens Chengbo Liu Yong Zhu 18 0 0 27 Mar 2024
Recurrent Drafter for Fast Speculative Decoding in Large Language Models Aonan Zhang Chong-Jun Wang Yi Wang Xuanyu Zhang Yunfei Cheng 16 14 0 14 Mar 2024
Rethinking Generative Large Language Model Evaluation for Semantic Comprehension Fangyun Wei Xi Chen Linzi Luo ELM ALM LRM 19 7 0 12 Mar 2024
LLM Inference Unveiled: Survey and Roofline Model Insights Zhihang Yuan Yuzhang Shang Yang Zhou Zhen Dong Zhe Zhou ... Yong Jae Lee Yan Yan Beidi Chen Guangyu Sun Kurt Keutzer 29 77 0 26 Feb 2024
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding Weilin Zhao Yuxiang Huang Xu Han Wang Xu Chaojun Xiao Xinrong Zhang Yewei Fang Kaihuo Zhang Zhiyuan Liu Maosong Sun 32 10 0 21 Feb 2024
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models Xiaoyu Tian Junru Gu Bailin Li Yicheng Liu Yang Wang Chenxu Hu Kun Zhan Peng Jia Xianpeng Lang Hang Zhao VLM 59 122 0 19 Feb 2024
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding Yichao Fu Peter Bailis Ion Stoica Hao Zhang 118 134 0 03 Feb 2024
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding Heming Xia Zhe Yang Qingxiu Dong Peiyi Wang Yongqi Li Tao Ge Tianyu Liu Wenjie Li Zhifang Sui LRM 11 94 0 15 Jan 2024
I-BERT: Integer-only BERT Quantization Sehoon Kim A. Gholami Z. Yao Michael W. Mahoney Kurt Keutzer MQ 83 332 0 05 Jan 2021
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT Sheng Shen Zhen Dong Jiayu Ye Linjian Ma Z. Yao A. Gholami Michael W. Mahoney Kurt Keutzer MQ 214 505 0 12 Sep 2019