Speculative Decoding

Follow this topic Ask about this topic

Speculative decoding aims to reduce the time and computation needed to generate text by letting a drafter propose future tokens for a target model to check. Approaches include small auxiliary drafters, early-exit or multi-token prediction within the target model, and tree-based or block-parallel drafting; retrieval and task-specific signals can also supply candidates. Exact verification can preserve the target model’s output distribution, while relaxed acceptance rules trade some fidelity for speed. A central challenge is making gains in accepted tokens translate into real latency or throughput improvements across different workloads, hardware, and serving loads.

Latest papers

309 papers72 papers in the last four weeks, up 112% on the four weeks before. 0.4% of all new papers.

Accelerating Speculative Diffusions via Block Verification
Jun 11, 2026Alexander Soen, Hisham Husain, Valentin De Bortoli +1Speculative Decoding for Diffusion Language Models Speculative Decoding
KTH · Google Research† · Google DeepMind‡
Efficient On-Device Diffusion LLM Inference with Mobile NPU
Jun 11, 2026Tuowei Wang, Yanfan Sun, Ju RenAI Accelerator Inference On-Device Language Model Inference
Beihang University China · Tsinghua University China
Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling
Jun 10, 2026Yucheng Li, Huiqiang Jiang, Yang Xu +14Multi-Token Prediction Speculative Decoding
Qwen Team, Alibaba Inc.
VIA-SD: Verification via Intra-Model Routing for Speculative Decoding
Jun 10, 2026Yuchen Xian, Yang He, Yunqiu Xu +1Speculative Decoding LLM Inference Acceleration
ReLER, The State Key Lab of Brain Machine Intelligence, Zhejiang University · College of Artificial Intelligence, Zhejiang University · CFAR, Agency for Science, Technology and Research, Singapore +2
Teaching Diffusion to Speculate Left-to-Right
Jun 10, 2026Lexington Whalen, Yuki Ito, Ryo SakamotoSpeculative Decoding for Diffusion Language Models Blockwise Diffusion Language Models
SB Intuitions, Tokyo, Japan
PathRelax: Parallel-Path Relaxed Speculative Jacobi Decoding for Accelerating Auto-Regressive Text-to-Image Generation
Jun 9, 2026Haodong Lei, Hongsong Wang, Bingxuan Dai +1T2I Generation Speculative Decoding
College of Software Engineering Southeast University · School of Computer Science and Engineering Southeast University · School of Cyber Science and Engineering Southeast University +1
Diffusion Language Model Parallel Decoding via Product-of-Experts Bridge
Jun 6, 2026Juntong Shi, Brian L. Trippe, Jure Leskovec +2Importance Sampling Product of Experts
Stanford University
WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing
Jun 5, 2026Young D. Kwon, Miles Williams, Rui Li +2Speculative Decoding LLM Inference Acceleration
Samsung AI Center, Cambridge, UK
AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding
Jun 4, 2026Runheng Liu, Jincheng Xie, Wen Hu +2Speculative Decoding LLM Inference Acceleration
School of Computer Science and Technology, Beijing Institute of Technology · Department of Mathematical Sciences, Tsinghua University · JDT AI Infra
Multi-SPIN: Multi-Access Speculative Inference for Cooperative Token Generation at the Edge
Jun 3, 2026Haotian Zheng, Zhanwei Wang, Mingyao Cui +3On-Device Language Model Inference Edge Computing
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China
D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
Jun 3, 2026Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6Speculative Decoding for Diffusion Language Models Speculative Decoding
Peking University · Tsinghua University · HKUST +2
TreeFlash: Parallel AR-Approximation for Faster Speculative Decoding
Jun 2, 2026Peer Rheinboldt, Frédéric Berdoz, Roger WattenhoferSpeculative Decoding LLM Inference Acceleration
ETH Zurich
Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning
Jun 2, 2026Yanyu Zhu, Hoilam Pao, Niu Hu +6RL for Language Models Speculative Decoding
1Tsinghua Shenzhen International Graduate School · 3Huawei Noah’s Ark Lab · 4Northwestern Polytechnical University +1
SimSD: Simple Speculative Decoding in Diffusion Language Models
Jun 1, 2026Junxia Cui, Haotian Ye, Runchu Tian +9Masked Diffusion Models Speculative Decoding for Diffusion Language Models
University of California San Diego · University of Illinois Urbana-Champaign · 3Google +1
DFlare: Scaling Up Draft Capacity for Block Diffusion Speculative Decoding
Jun 1, 2026Jiebin Zhang, Zhenghan Yu, Song Liu +9Speculative Decoding LLM Inference Acceleration
School of Computer Science, Peking University · Tencent
Cost-Aware Diffusion Draft Trees for Speculative Decoding
Jun 1, 2026Shuai Zhang, Huachuan Qiu, Hongliang He +1Cost-Aware Inference Speculative Decoding for Diffusion Language Models
1Zhejiang University · 2Westlake University
Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
May 31, 2026Xin Su, Dawid Majchrowski, Fangyuan Yu +5Speculative Decoding LLM Inference Acceleration
Thoughtworks · Nvidia
DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation
May 30, 2026Zining Liu, Yunhai Hu, Tianhua Xia +4Efficient VLM Inference Speculative Decoding
New York University · Cerebras Systems Inc.
TAPS: Target-Aware Prefix Tree Selection for Diffusion-Drafted Speculative Decoding
May 30, 2026Zhuoyu Wang, Junnan Huang, Xinyu ChenTree Search Speculative Decoding
The Hong Kong University of Science and Technology (Guangzhou)
Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism
May 29, 2026Yijiong Yu, Huazheng Wang, Shuai Yuan +2Speculative Decoding LLM Inference Acceleration
aOregon State University · bDeepSolution
BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding
May 29, 2026Liang He, Jingbo Wen, Qishi Zhan +4KV Caching Long-Context Language Model Inference
Shanghai Institute of Optics and Fine Mechanics · The University of Sydney · Marquette University +4
Speculative Decoding Across Languages
May 28, 2026Nirajan Paudel, Michael Ginn, Luc De Nardi +1Multilingual Language Models Speculative Decoding
University of Colorado
TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding
May 28, 2026Jiaming Fan, Daming Cao, Canchen Huang +6Speculative Decoding LLM Inference Acceleration
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Southeast University · Ant Group +1
Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting
May 28, 2026Soowon Oh, Nam Cao, Yujin Kim +4Speculative Decoding Diffusion Language Model Decoding
KAIST AI · Samsung Advanced Institute of Technology
Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding
May 28, 2026Jianuo Huang, Yaojie Zhang, Qituan Zhang +3Parallel Speculative Decoding Speculative Decoding
1EPIC Lab, Shanghai Jiao Tong University · School of Software Engineering, HUST · 3UESTC +2
Draft-OPD: On-Policy Distillation for Speculative Draft Models
May 28, 2026Haodi Lei, Yafu Li, Haoran Zhang +8Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3
Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
May 26, 2026Wenhui Tan, Minghao Li, Xiaoqian Ma +5Multi-Token Prediction LLM Compression
Gaoling School of Artificial Intelligence, Renmin University of China · AI Platform, Xiaohongshu Inc. · University of Electronic Science and Technology of China +1
Beyond the Target: From Imitation to Collaboration in Speculative Decoding
May 24, 2026Jinze Li, Yixing Xu, Guanchen Li +7Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas
Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving
May 22, 2026Kewei Zhang, Jin Wang, Sensen Gao +9VLMs for Autonomous Driving End-to-End Autonomous Driving
1Peking University · 3The University of Hong Kong · 2NVIDIA +1
Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding
May 19, 2026Yuhao Shen, Tianyu Liu, Xinyi Hu +9Speculative Decoding LLM Inference Acceleration
1Zhejiang University · 2Qwen Applications Business Group of Alibaba · University of Science and Technology of China

Jun 11, 2026J/K move · Enter open · S save

Accelerating Speculative Diffusions via Block Verification

Alexander Soen, Hisham Husain, Valentin De Bortoli, Arnaud Doucet

KTH · Google Research† · Google DeepMind‡

Speculative Decoding for Diffusion Language Models Speculative Decoding Diffusion Model Inference Acceleration Diffusion Models

Open paper AskPDF arXiv

Speculative decoding speeds up LLM inference by using a draft model to generate tokens, with an acceptance-rejection scheme that ensures that the output matches the target distribution. Adapting this to continuous diffusions is difficult because speculative sampling requires drawing from a residual distribution. While straightforward in discrete spaces, efficiently sampling this residual in continuous space is non-trivial. Consequently, existing diffusion adaptations either use computationally inefficient sampling techniques or rely on an alternative scheme. In this work, we introduce a novel scheme that efficiently implements the original speculative sampling mechanism for diffusion models. Our approach offers a critical advantage over current methods: it enables us to adapt block verification from LLMs to diffusions -- which provably improves the acceptance rate of drafts. Furthermore, we formalize and analyze the Free Drafter, a heuristic self-speculative drafter for diffusions that requires no training. By enabling block verification, our Free Drafter yields up to a 6.3% speedup over existing speculative methods with no additional training and negligible overhead beyond the existing parallel verification pass.