Fast Inference from Transformers via Speculative Decoding

30 November 2022

Yossi Matias

Papers citing "Fast Inference from Transformers via Speculative Decoding"

50 / 482 papers shown

Title
Let the Code LLM Edit Itself When You Edit the Code Zhenyu He Jun Zhang Shengjie Luo Jingjing Xu Z. Zhang Di He KELM 31 0 0 03 Jul 2024
S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models Parsa Kavehzadeh Mohammadreza Pourreza Mojtaba Valipour Tinashu Zhu Haoli Bai Ali Ghodsi Boxing Chen Mehdi Rezagholizadeh 30 0 0 02 Jul 2024
Tree Search for Language Model Agents Jing Yu Koh Stephen Marcus McAleer Daniel Fried Ruslan Salakhutdinov LM&Ro LLMAG LRM 51 56 0 01 Jul 2024
Adaptive Draft-Verification for Efficient Large Language Model Decoding Xukun Liu Bowen Lei Ruqi Zhang Dongkuan Xu 30 3 0 27 Jun 2024
SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding Zhenglin Wang Jialong Wu Yilong Lai Congzhi Zhang Deyu Zhou LRM ReLM 31 3 0 26 Jun 2024
Decoding with Limited Teacher Supervision Requires Understanding When to Trust the Teacher Hyunjong Ok Jegwang Ryu Jaeho Lee 37 0 0 26 Jun 2024
Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training Yixuan Wang Xianzhen Luo Fuxuan Wei Yijun Liu Qingfu Zhu Xuanyu Zhang Qing Yang Dongliang Xu Wanxiang Che 35 3 0 25 Jun 2024
OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure Jikai Wang Yi Su Juntao Li Qingrong Xia Zi Ye Xinyu Duan Zhefeng Wang Min Zhang 29 11 0 25 Jun 2024
Speeding Up Image Classifiers with Little Companions Yang Liu Kowshik Thopalli Jayaraman J. Thiagarajan VLM 21 0 0 24 Jun 2024
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees Yuhui Li Fangyun Wei Chao Zhang Hongyang R. Zhang 85 50 0 24 Jun 2024
From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models Sean Welleck Amanda Bertsch Matthew Finlayson Hailey Schoelkopf Alex Xie Graham Neubig Ilia Kulikov Zaid Harchaoui 33 47 0 24 Jun 2024
Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters Euiin Yi Taehyeon Kim Hongseok Jeung Du-Seong Chang Se-Young Yun 38 4 0 24 Jun 2024
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph Roman Vashurin Ekaterina Fadeeva Artem Vazhentsev Akim Tsvigun Daniil Vasilev ... Timothy Baldwin Timothy Baldwin Maxim Panov Artem Shelmanov Artem Shelmanov HILM 64 8 0 21 Jun 2024
LiveMind: Low-latency Large Language Models with Simultaneous Inference Chuangtao Chen Grace Li Zhang Xunzhao Yin Cheng Zhuo Ulf Schlichtmann Bing Li LRM 34 3 0 20 Jun 2024
Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving Ke Cheng Wen Hu Zhi Wang Hongen Peng Jianguo Li Sheng Zhang 43 7 0 19 Jun 2024
Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding Kaiyan Zhang Jianyu Wang Ning Ding Biqing Qi Ermo Hua Xingtai Lv Bowen Zhou 39 8 0 18 Jun 2024
CleanGen: Mitigating Backdoor Attacks for Generation Tasks in Large Language Models Yuetai Li Zhangchen Xu Fengqing Jiang Luyao Niu D. Sahabandu Bhaskar Ramasubramanian Radha Poovendran SILM AAML 47 6 0 18 Jun 2024
Promises, Outlooks and Challenges of Diffusion Language Modeling Justin Deschenaux Çağlar Gülçehre DiffM 37 2 0 17 Jun 2024
On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion Chenghao Fan Zhenyi Lu Wei Wei Jie Tian Xiaoye Qu Dangyang Chen Yu Cheng MoMe 44 5 0 17 Jun 2024
Optimized Speculative Sampling for GPU Hardware Accelerators Dominik Wagner Seanie Lee Ilja Baumann Philipp Seeberger K. Riedhammer Tobias Bocklet 33 3 0 16 Jun 2024
New Solutions on LLM Acceleration, Optimization, and Application Yingbing Huang Lily Jiaxin Wan Hanchen Ye Manvi Jha Jinghua Wang Yuhong Li Xiaofan Zhang Deming Chen 37 12 0 16 Jun 2024
OpenVLA: An Open-Source Vision-Language-Action Model Moo Jin Kim Karl Pertsch Siddharth Karamcheti Ted Xiao Ashwin Balakrishna ... Russ Tedrake Dorsa Sadigh Sergey Levine Percy Liang Chelsea Finn LM&Ro VLM 37 348 0 13 Jun 2024
Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs Xuan Zhang Chao Du Tianyu Pang Qian Liu Wei Gao Min-Bin Lin LRM AI4CE 44 34 0 13 Jun 2024
Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL Zijin Hong Zheng Yuan Qinggang Zhang Hao Chen Junnan Dong Feiran Huang Xiao Huang 69 50 0 12 Jun 2024
OPTune: Efficient Online Preference Tuning Lichang Chen Jiuhai Chen Chenxi Liu John Kirchenbauer Davit Soselia Chen Zhu Tom Goldstein Tianyi Zhou Heng Huang 34 4 0 11 Jun 2024
When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models Haoran You Yichao Fu Zheng Wang Amir Yazdanbakhsh Yingyan Celine Lin 31 1 0 11 Jun 2024
Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inference Jihwan Bang Juntae Lee Kyuhong Shim Seunghan Yang Simyung Chang 29 5 0 11 Jun 2024
Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters Yixin Song Haotong Xie Zhengyan Zhang Bo Wen Li Ma Zeyu Mi Haibo Chen MoE 29 21 0 10 Jun 2024
Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction Ke Cheng Wen Hu Zhi Wang Peng Du Jianguo Li Sheng Zhang 34 10 0 07 Jun 2024
Proofread: Fixes All Errors with One Tap Renjie Liu Yanxiang Zhang Yun Zhu Haicheng Sun Yuanbo Zhang Michael Xuelin Huang Shanqing Cai Lei Meng Shumin Zhai ALM 31 2 0 06 Jun 2024
To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation Abdul Waheed Karima Kadaoui Muhammad Abdul-Mageed VLM 33 3 0 06 Jun 2024
Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism Jiahao Liu Qifan Wang Jingang Wang Xunliang Cai 25 6 0 06 Jun 2024
SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices Ruslan Svirschevski Avner May Zhuoming Chen Beidi Chen Zhihao Jia Max Ryabinin 23 12 0 04 Jun 2024
Block Transformer: Global-to-Local Language Modeling for Fast Inference Namgyu Ho Sangmin Bae Taehyeon Kim Hyunjik Jo Yireun Kim Tal Schuster Adam Fisch James Thorne Se-Young Yun 45 7 0 04 Jun 2024
Diver: Large Language Model Decoding with Span-Level Mutual Information Verification Jinliang Lu Chen Wang Jiajun Zhang 48 3 0 04 Jun 2024
OccamLLM: Fast and Exact Language Model Arithmetic in a Single Step Owen Dugan Donato Manuel Jimenez Beneto Charlotte Loh Zhuo Chen Rumen Dangovski Marin Soljacic LRM 29 1 0 04 Jun 2024
GRAM: Generative Retrieval Augmented Matching of Data Schemas in the Context of Data Security Xuanqing Liu Luyang Kong Runhui Wang Patrick Song Austin Nevins Henrik Johnson Nimish Amlathe Davor Golac 29 2 0 04 Jun 2024
SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM Quandong Wang Yuxuan Yuan Xiaoyu Yang Ruike Zhang Kang Zhao Wei Liu Jian Luan Daniel Povey Bin Wang 41 0 0 03 Jun 2024
Achieving Sparse Activation in Small Language Models Jifeng Song Kai Huang Xiangyu Yin Boyuan Yang Wei Gao 29 4 0 03 Jun 2024
Decentralized AI: Permissionless LLM Inference on POKT Network D. Olshansky Ramiro Rodríguez Colmeiro Bowen Li MoE 21 3 0 30 May 2024
S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUs Wei Zhong Manasa Bharadwaj 31 5 0 30 May 2024
SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths Kaixuan Huang Xudong Guo Mengdi Wang 32 17 0 30 May 2024
GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment Yao Yao Z. Li Hai Zhao 27 5 0 30 May 2024
Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execution Yechen Xu Xinhao Kong Tingjun Chen Danyang Zhuo LLMAG 22 2 0 29 May 2024
Faster Cascades via Speculative Decoding Harikrishna Narasimhan Wittawat Jitkrittum A. S. Rawat Seungyeon Kim Neha Gupta A. Menon Sanjiv Kumar LRM 44 6 0 29 May 2024
Nearest Neighbor Speculative Decoding for LLM Generation and Attribution Minghan Li Xilun Chen Ari Holtzman Beidi Chen Jimmy Lin Wen-tau Yih Xi Victoria Lin RALM BDL 108 10 0 29 May 2024
Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference Hao Chen Wayne Luk Ka-Fai Cedric Yiu Rui Li Konstantin Mishchenko Stylianos I. Venieris Hongxiang Fan 34 7 0 28 May 2024
Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass Ethan Shen Alan Fan Sarah M Pratt Jae Sung Park Matthew Wallingford Sham Kakade Ari Holtzman Ranjay Krishna Ali Farhadi Aditya Kusupati 33 2 0 28 May 2024
Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection Yun Zhu Jia-Chen Gu Caitlin Sikora Ho Ko Yinxiao Liu ... Lei Shu Liangchen Luo Lei Meng Bang Liu Jindong Chen RALM 22 14 0 25 May 2024
Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs Chenxi Sun Hongzhi Zhang Zijia Lin Jingyuan Zhang Fuzheng Zhang ... Bin Chen Chengru Song Di Zhang Kun Gai Deyi Xiong 38 1 0 24 May 2024