Faster Cascades via Speculative Decoding

Faster Cascades via Speculative Decoding

29 May 2024

Harikrishna Narasimhan

Wittawat Jitkrittum

Sanjiv Kumar

Papers citing "Faster Cascades via Speculative Decoding"

4 / 4 papers shown

Title
Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing Dujian Ding Ankur Mallick Chi Wang Robert Sim Subhabrata Mukherjee Victor Rühle L. Lakshmanan Ahmed Hassan Awadallah 77 73 0 22 Apr 2024
Exploring and Improving Drafts in Blockwise Parallel Decoding Taehyeon Kim A. Suresh Kishore Papineni Michael Riley Sanjiv Kumar Adrian Benton AI4TS 34 2 0 14 Apr 2024
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 135 208 0 13 Mar 2023
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 167 3,504 0 10 Jun 2015