Fast Inference from Transformers via Speculative Decoding

30 November 2022

Yossi Matias

Papers citing "Fast Inference from Transformers via Speculative Decoding"

50 / 482 papers shown

Title
A Declarative System for Optimizing AI Workloads Chunwei Liu Matthew Russo Michael Cafarella Lei Cao Peter Baille Chen Zui Chen Michael Franklin Tim Kraska Samuel Madden Gerardo Vitagliano 34 20 0 23 May 2024
Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs Qingyuan Li Ran Meng Yiduo Li Bo Zhang Yifan Lu Yerui Sun Lin Ma Yuchen Xie MQ 33 0 0 23 May 2024
Modeling Real-Time Interactive Conversations as Timed Diarized Transcripts Garrett Tanzer Gustaf Ahdritz Luke Melas-Kyriazi 16 1 0 21 May 2024
Towards Modular LLMs by Building and Reusing a Library of LoRAs O. Ostapenko Zhan Su E. Ponti Laurent Charlin Nicolas Le Roux Matheus Pereira Lucas Page-Caccia Alessandro Sordoni MoMe 32 30 0 18 May 2024
A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models Mahsa Khoshnoodi Vinija Jain Mingye Gao Malavika Srikanth Aman Chadha OffRL 28 1 0 15 May 2024
Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis Yao Fu 22 19 0 14 May 2024
EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models Yunsheng Ni Chuanjian Liu Yehui Tang Kai Han Yunhe Wang 21 0 0 13 May 2024
SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts R. Prabhakar R. Sivaramakrishnan Darshan Gandhi Yun Du Mingran Wang ... Urmish Thakker Dawei Huang Sumti Jairath Kevin J. Brown K. Olukotun MoE 39 12 0 13 May 2024
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models Wenqi Fan Yujuan Ding Liang-bo Ning Shijie Wang Hengyun Li Dawei Yin Tat-Seng Chua Qing Li RALM 3DV 38 181 0 10 May 2024
KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation Minsik Cho Mohammad Rastegari Devang Naik 32 4 0 08 May 2024
Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models Jonathan Mamou Oren Pereg Daniel Korat Moshe Berchansky Nadav Timor Moshe Wasserblat Roy Schwartz 36 3 0 07 May 2024
Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection Guillem Ramírez Alexandra Birch Ivan Titov 38 8 0 03 May 2024
Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge Bin Xiao Chunan Shi Xiaonan Nie Fan Yang Xiangwei Deng Lei Su Weipeng Chen Bin Cui 26 8 0 01 May 2024
Better & Faster Large Language Models via Multi-token Prediction Fabian Gloeckle Badr Youbi Idrissi Baptiste Rozière David Lopez-Paz Gabriele Synnaeve 22 91 0 30 Apr 2024
Accelerating Production LLMs with Combined Token/Embedding Speculators Davis Wertheimer Joshua Rosenkranz Thomas Parnell Sahil Suneja Pavithra Ranganathan R. Ganti M. Srivatsa 30 3 0 29 Apr 2024
Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting Fangcheng Liu Yehui Tang Zhenhua Liu Yunsheng Ni Kai Han Yunhe Wang 33 23 0 29 Apr 2024
BlockLLM: Multi-tenant Finer-grained Serving for Large Language Models Jiamin Li Le Xu Hong-Yu Xu Aditya Akella 19 1 0 28 Apr 2024
Reinforcement Retrieval Leveraging Fine-grained Feedback for Fact Checking News Claims with Black-Box LLM Xuan Zhang Wei Gao LRM KELM 27 8 0 26 Apr 2024
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding Mostafa Elhoushi Akshat Shrivastava Diana Liskovich Basil Hosmer Bram Wasti ... Saurabh Agarwal Ahmed Roman Ahmed Aly Beidi Chen Carole-Jean Wu LRM 33 82 0 25 Apr 2024
BASS: Batched Attention-optimized Speculative Sampling Haifeng Qian Sujan Kumar Gonugondla Sungsoo Ha Mingyue Shang Sanjay Krishna Gouda Ramesh Nallapati Sudipta Sengupta Xiaofei Ma Anoop Deoras BDL 45 8 0 24 Apr 2024
Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models Chen Zhang Zhuorui Liu Dawei Song LRM 28 3 0 23 Apr 2024
Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing Dujian Ding Ankur Mallick Chi Wang Robert Sim Subhabrata Mukherjee Victor Rühle L. Lakshmanan Ahmed Hassan Awadallah 80 76 0 22 Apr 2024
Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity Tyler Griggs Xiaoxuan Liu Jiaxiang Yu Doyoung Kim Wei-Lin Chiang Alvin Cheung Ion Stoica 40 15 0 22 Apr 2024
SnapKV: LLM Knows What You are Looking for Before Generation Yuhong Li Yingbing Huang Bowen Yang Bharat Venkitesh Acyr F. Locatelli Hanchen Ye Tianle Cai Patrick Lewis Deming Chen VLM 75 148 0 22 Apr 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 80 0 22 Apr 2024
Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration Pengfei Wu Jiahao Liu Zhuocheng Gong Qifan Wang Jinpeng Li Jingang Wang Xunliang Cai Dongyan Zhao 18 1 0 18 Apr 2024
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding Hanshi Sun Zhuoming Chen Xinyu Yang Yuandong Tian Beidi Chen 33 46 0 18 Apr 2024
Language Model Cascades: Token-level uncertainty and beyond Neha Gupta Harikrishna Narasimhan Wittawat Jitkrittum A. S. Rawat A. Menon Sanjiv Kumar UQLM 41 42 0 15 Apr 2024
Improving Recall of Large Language Models: A Model Collaboration Approach for Relational Triple Extraction Zepeng Ding Wenhao Huang Jiaqing Liang Deqing Yang Yanghua Xiao KELM 29 4 0 15 Apr 2024
Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models Siyan Zhao Daniel Israel Guy Van den Broeck Aditya Grover KELM VLM 19 5 0 15 Apr 2024
Exploring and Improving Drafts in Blockwise Parallel Decoding Taehyeon Kim A. Suresh Kishore Papineni Michael Riley Sanjiv Kumar Adrian Benton AI4TS 47 2 0 14 Apr 2024
On Speculative Decoding for Multimodal Large Language Models Mukul Gagrani Raghavv Goel Wonseok Jeon Junyoung Park Mingu Lee Christopher Lott LRM 27 7 0 13 Apr 2024
Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction Haoran Qiu Weichao Mao Archit Patke Shengkun Cui Saurabh Jha Chen Wang Hubertus Franke Zbigniew T. Kalbarczyk Tamer Basar Ravishankar K. Iyer 20 24 0 12 Apr 2024
Reducing hallucination in structured outputs via Retrieval-Augmented Generation Patrice Béchard Orlando Marquez Ayala LLMAG 35 45 0 12 Apr 2024
Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding Jie Ou Yueming Chen Wenhong Tian 51 12 0 10 Apr 2024
CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers Longwei Zou Qingyang Wang Han Zhao Jiangang Kong Yi Yang Yangdong Deng 19 0 0 10 Apr 2024
Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models Bowen Pan Yikang Shen Haokun Liu Mayank Mishra Gaoyuan Zhang Aude Oliva Colin Raffel Rameswar Panda MoE 33 19 0 08 Apr 2024
Training LLMs over Neurally Compressed Text Brian Lester Jaehoon Lee A. Alemi Jeffrey Pennington Adam Roberts Jascha Narain Sohl-Dickstein Noah Constant 32 6 0 04 Apr 2024
The Larger the Better? Improved LLM Code-Generation via Budget Reallocation Michael Hassid Tal Remez Jonas Gehring Roy Schwartz Yossi Adi 34 20 0 31 Mar 2024
SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens Chengbo Liu Yong Zhu 23 0 0 27 Mar 2024
The Unreasonable Ineffectiveness of the Deeper Layers Andrey Gromov Kushal Tirumala Hassan Shapourian Paolo Glorioso Daniel A. Roberts 41 79 0 26 Mar 2024
Efficient Encoder-Decoder Transformer Decoding for Decomposable Tasks Bo-Ru Lu Nikita Haduong Chien-Yu Lin Hao Cheng Noah A. Smith Mari Ostendorf AI4CE 27 0 0 19 Mar 2024
Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inference Baolin Li Yankai Jiang V. Gadepally Devesh Tiwari 27 15 0 19 Mar 2024
MELTing point: Mobile Evaluation of Language Transformers Stefanos Laskaridis Kleomenis Katevas Lorenzo Minto Hamed Haddadi 27 20 0 19 Mar 2024
Recurrent Drafter for Fast Speculative Decoding in Large Language Models Aonan Zhang Chong-Jun Wang Yi Wang Xuanyu Zhang Yunfei Cheng 26 15 0 14 Mar 2024
Token Alignment via Character Matching for Subword Completion Ben Athiwaratkun Shiqi Wang Mingyue Shang Yuchen Tian Zijian Wang Sujan Kumar Gonugondla Sanjay Krishna Gouda Rob Kwiatowski Ramesh Nallapati Bing Xiang 40 4 0 13 Mar 2024
Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs Ben Athiwaratkun Sujan Kumar Gonugondla Sanjay Krishna Gouda Haifeng Qian Hantian Ding ... Liangfu Chen Parminder Bhatia Ramesh Nallapati Sudipta Sengupta Bing Xiang 44 4 0 13 Mar 2024
CHAI: Clustered Head Attention for Efficient LLM Inference Saurabh Agarwal Bilge Acun Basil Homer Mostafa Elhoushi Yejin Lee Shivaram Venkataraman Dimitris Papailiopoulos Carole-Jean Wu 51 8 0 12 Mar 2024
Rethinking Generative Large Language Model Evaluation for Semantic Comprehension Fangyun Wei Xi Chen Linzi Luo ELM ALM LRM 27 7 0 12 Mar 2024
Learning to Decode Collaboratively with Multiple Language Models Zejiang Shen Hunter Lang Bailin Wang Yoon Kim David Sontag 41 28 0 06 Mar 2024