Speculative Decoding

Follow this topic Ask about this topic

Speculative decoding aims to reduce the time and computation needed to generate text by letting a drafter propose future tokens for a target model to check. Approaches include small auxiliary drafters, early-exit or multi-token prediction within the target model, and tree-based or block-parallel drafting; retrieval and task-specific signals can also supply candidates. Exact verification can preserve the target model’s output distribution, while relaxed acceptance rules trade some fidelity for speed. A central challenge is making gains in accepted tokens translate into real latency or throughput improvements across different workloads, hardware, and serving loads.

Latest papers

309 papers72 papers in the last four weeks, up 112% on the four weeks before. 0.4% of all new papers.

FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
May 19, 2026Yaojie Zhang, Jianuo Huang, Junlong Ke +5Parallel Speculative Decoding Speculative Decoding
EPIC Lab, SJTU · School of Software Engineering, HUST · UESTC +3
Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference
May 17, 2026Kangkang Sun, Jianhua Li, Xiuzhen Chen +2Sequential Decision Making Optimal Stopping
Shanghai Key Laboratory of Integrated Administration Technologies for Information Security, School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong, China
Lever: Speculative LLM Inference on Smartphones
May 16, 2026Tuowei Wang, Fengzu Li, Yanfan Sun +2On-Device Language Model Inference LLM Inference
Tsinghua University · Beijing, China · Beihang University +1
PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding
May 15, 2026Shengyin Sun, Yiming Li, Renxi Liu +7Parallel Speculative Decoding Masked Diffusion Models
Department of Computer Science City University of Hong Kong · Huawei Technologies · City University of Hong Kong
An Interpretable Latency Model for Speculative Decoding in LLM Serving
May 14, 2026Linghao Kong, Megan Flynn, Michael Peng +3LLM Serving Speculative Decoding
MIT · Red Hat AI
Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing
May 14, 2026Jie Jiang, Xing Sun, Ruotian Chen +2Speculative Decoding LLM Inference Acceleration
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
May 13, 2026Shuoyang Sun, Chang Dai, Hao Fang +6Adversarial Attacks on LLMs Speculative Decoding
Harbin Institute of Technology, Shenzhen · South China University of Technology · Tsinghua Shenzhen International Graduate School, Tsinghua University +1
Realtime-VLA FLASH: Speculative Inference Framework for Diffusion-based VLAs
May 13, 2026Jiahui Niu, Kefan Gu, Yucheng Zhao +5Efficient VLA Model Inference Speculative Decoding
State Key Lab of Processors, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Dexmal +1
Multi-Token Residual Prediction
May 12, 2026Yufeng Xu, Zishuo Bao, Qian Wang +6Masked Diffusion Models Multi-Token Prediction
New York University · New York University Shanghai · Nous Research +1
D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting
May 12, 2026Tianyu Wu, Yu Yao, Zhenting Qi +7Speculative Decoding LLM Inference Acceleration
1Harvard · 2MIT
CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
May 11, 2026Yuning Han, Yangchenchen Jin, Dylan Zhao +1On-Device Language Model Inference Memory-Efficient Inference
University of Florida Gainesville, FL 32611
SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding
May 11, 2026Anton Plaksin, Sergei Krutikov, Sergei Skvortsov +1LLM Compression Speculative Decoding
Nebius, Amsterdam, Netherlands.
GELATO: Generative Entropy- and Lyapunov-based Adaptive Token Offloading for Device-Edge Speculative LLM Inference
May 11, 2026Zengzipeng Tang, Yuxuan Sun, Wei Chen +2On-Device Language Model Inference Edge Computing
School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China
Attention Drift: What Autoregressive Speculative Decoding Models Learn
May 11, 2026Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek +3Long-Context Language Model Inference Speculative Decoding
Northwestern University · GE Aerospace · fal +1
Test-Time Speculation
May 10, 2026Avinash Kumar, Sujay Sanghavi, Poulami DasTest-Time Adaptation Test-Time Optimization
The University of Texas at Austin
BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
May 9, 2026Yuhang Xu, Kaibin Tian, Yang Tian +6Reinforcement Learning Speculative Decoding
1Shanghai Jiao Tong University · 2Bytedance
PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding
May 9, 2026Zihao An, Taichi Liu, Ziqiong Liu +3Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices, Inc. · Rutgers University · Michigan State University
Fast Byte Latent Transformer
May 8, 2026Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5Speculative Decoding LLM Inference Acceleration
FAIR at Meta · Stanford University · University of Washington
Future Validity is the Missing Statistic: From Impossibility to $Φ$ -Estimation for Grammar-Faithful Speculative Decoding
May 8, 2026Wenhua Nie, Zijie Meng, Kun Zou +5Doob H-Transform Speculative Decoding
National Taiwan University
SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting
May 8, 2026Weijie Shi, Qiang Xu, Fan Deng +9Speculative Decoding LLM Inference Acceleration
Hong Kong University of Science and Technology · MetaX · Zhejiang Normal University +1
CASCADE: Context-Aware Relaxation for Speculative Image Decoding
May 8, 2026Selin Yildirim, Subhajit Dutta Chowdhury, Mohammad Mahdi Kamani +2Speculative Decoding Autoregressive Image Generation
AMD
UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decoding
May 6, 2026Yepeng Weng, Qiao Hu, Takehisa YairiSpeculative Decoding LLM Inference Acceleration
The University of Tokyo · 2Lenovo AI Technology Center · National Center for Mathematics and Interdisciplinary Sciences (NCMIS), AMSS, CAS
Parallel Prefix Verification for Speculative Generation
May 5, 2026Yuncheng Yao, Yuxuan Xia, Shengjie Wang +1Speculative Decoding LLM Inference Acceleration
Duke University · New York University
SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection
May 4, 2026Shikhar ShuklaSpeculative Decoding LLM Inference Acceleration
University of Kentucky, Lexington, KY, USA
CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding
May 4, 2026Yuanyuan Jia, Shunpu Tang, Qianqian YangEfficient VLM Inference Edge Computing
College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, China
SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
May 4, 2026Jincheng Xie, Yawen Ling, Qi Xiao +4Parallel Speculative Decoding LLM Serving
1Tsinghua University · 2AI Infra Team at JDT · 3JD iCity, JD Technology, JD Intelligent Cities Research
Component-Aware Self-Speculative Decoding in Hybrid Language Models
May 1, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóSpeculative Decoding Language Modeling
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding
May 1, 2026Lehan Pan, Ziyang Tao, Ruoyu Pang +3Mixture-of-Experts Inference Speculative Decoding
University of Science and Technology of China · Tianyijiaotong Technology Ltd., Suzhou, China
Position-Aware Drafting for Inference Acceleration in LLM-Based Generative List-Wise Recommendation
Apr 30, 2026Jiaju Chen, Chongming Gao, Chenxiao Fan +4Speculative Decoding LLM Inference Acceleration
University of Science and Technology of China, Hefei, China · Independent Researcher
Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
Apr 29, 2026Hayate Iso, Tiyasa Mitra, Sudipta Mondal +15Reinforcement Learning Speculative Decoding
NVIDIA

May 19, 2026J/K move · Enter open · S save

FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration

Yaojie Zhang, Jianuo Huang, Junlong Ke, Yuhang Han+4

EPIC Lab, SJTU · School of Software Engineering, HUST · UESTC+3

Parallel Speculative Decoding Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding accelerates memory-bound LLM inference without quality degradation by using a fast drafter to propose multiple candidate tokens and the target model to verify them in parallel. However, conventional sequential speculative decoding suffers from mutual waiting between drafting and verification, and repeated exchange of intermediate states further increases memory access overhead. Parallel speculative decoding addresses this limitation by performing drafting and verification within a single target forward pass, allowing future drafts to be prepared while current candidates are being verified. Although effective at small batch sizes, existing parallel speculative decoding methods either require costly continual pretraining with quality degradation or suffer from low acceptance rates. More importantly, this paradigm inherently suffers from uncertainty in both the bonus token and the accepted length, leading to draft verification mismatch and causing throughput gains to collapse at large batch sizes. To address these limitations, we introduce FlexDraft, a lossless speculative decoding framework that flexibly adapts to varying batch sizes through three key designs. (1) Attention Tuning enables block diffusion drafting by tuning only the attention projectors of the final few layers on mask tokens, while keeping the autoregressive path frozen to preserve the target distribution and produce high quality drafts with minimal trainable parameters. (2) Bonus-guided Calibration uses a lightweight MLP conditioned on the resolved bonus token to calibrate draft logits, mitigating draft verification mismatch caused by bonus token uncertainty. (3) Flex Decoding dynamically switches between parallel draft and verify at small batch sizes and sequential draft then verify at large batch sizes, and adjusts verification length based on draft confidence to eliminate redundant computation.