Speculative Decoding

Follow this topic Ask about this topic

Speculative decoding aims to reduce the time and computation needed to generate text by letting a drafter propose future tokens for a target model to check. Approaches include small auxiliary drafters, early-exit or multi-token prediction within the target model, and tree-based or block-parallel drafting; retrieval and task-specific signals can also supply candidates. Exact verification can preserve the target model’s output distribution, while relaxed acceptance rules trade some fidelity for speed. A central challenge is making gains in accepted tokens translate into real latency or throughput improvements across different workloads, hardware, and serving loads.

Latest papers

309 papers72 papers in the last four weeks, up 112% on the four weeks before. 0.4% of all new papers.

SpecLA: Efficient Speculative Decoding for Linear-Attention Models
Jul 18, 2026Zhibin Wang, Xuying Han, Zhaohua Yang +5Speculative Decoding LLM Inference Acceleration
State Key Laboratory of Novel Software Technology Nanjing University Nanjing, China · Alibaba Group Hangzhou, China
D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding
Jul 16, 2026Tianyu Liu, Yuhao Shen, Rui Cen +7Speculative Decoding LLM Inference Acceleration
1Tencent · 2Independent Researcher
Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
Jul 14, 2026Jincheng Xie, Runheng Liu, Heyan Huang +4Mixture-of-Experts Inference Speculative Decoding
Department of Mathematical Sciences, Tsinghua University, Beijing, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JDT AI Infra +2
Accepted Prefixes Are Not All You Need: A Negative Result on PEFT-Based Block-Diffusion Drafting
Jul 14, 2026Abdurrahman Javat, Allan KazakovSpeculative Decoding LLM Inference Acceleration
Bahçeşehir University, Department of Artificial Intelligence
TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decoding
Jul 13, 2026Quynh Vo, Cong-Duy Nguyen, Ponhvoan Srey +2Efficient VLM Inference Speculative Decoding
National University of Singapore · Nanyang Technological University · Center of AI Research, VinUniversity
Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
Jul 12, 2026Zipeng Gao, Zhi Zheng, Qingrong Xia +5Speculative Decoding LLM Inference Acceleration
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Huawei Technologies Co., Ltd.
A Practical Investigation of Training-free Relaxed Speculative Decoding
Jul 9, 2026Guoxuan Xia, Luka Ribar, Paul BalancaSpeculative Decoding LLM Inference Acceleration
Graphcore
DominoTree: Conditional Tree-Structured Drafting with Domino for Speculative Decoding
Jul 9, 2026Saw S. Lin, Jyh-Shing Roger JangSpeculative Decoding LLM Inference Acceleration
Zhiqi Zhang · Department of Computer Science and Information Engineering, National Taiwan University.
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
Jul 8, 2026Hong-Kai Zheng, Piji LiSpeculative Decoding LLM Inference Acceleration
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China
Trees from Marginals: Autoregressive drafting with factorized priors
Jul 7, 2026Yuma Oda, Ryan Mathieu, Roman Knyazhitskiy +1Speculative Decoding Efficient Language Model Inference
Artur Chakhvadze1
Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding
Jul 7, 2026Yonggan Fu, Lexington Whalen, Abhinav Garg +23Autoregressive Diffusion Autoregressive Language Modeling
NVIDIA · Georgia Tech · Work done during internships at NVIDIA +3
DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
Jul 6, 2026Xin Cheng, Xingkai Yu, Chenze Shao +30LLM Serving Speculative Decoding
1Peking University · 2DeepSeek-AI
Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
Jul 5, 2026Jaeyeon Kim, Jewon Lee, Bo-Kyeong KimLLM Quantization Speculative Decoding
Nota Inc.
AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference
Jul 4, 2026Sadra SaremiOn-Device Language Model Inference Memory-Efficient Inference
Department of Physics, Iran University of Science and Technology, Tehran 16846-13114, Iran.
SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference
Jul 3, 2026Huajun Bai, Weiwei Lv, Huichuan Zheng +2LLM Inference Efficiency LLM Agent Orchestration
Tsinghua University · Meituan
Spec-AUF: Accept-Until-Fail Training under Train-Inference Misalignment for Masked Block Drafters
Jul 2, 2026Tianjian Yang, Meng LiSpeculative Decoding
School of Electronics Engineering and Computer Science, Peking University, Beijing, China · Institute for Artificial Intelligence, Peking University, Beijing, China · School of Integrated Circuits, Peking University, Beijing, China
BlockPilot: Instance-Adaptive Policy Learning for Diffusion-based Speculative Decoding
Jun 30, 2026Hao Zhang, Yiming Hu, Yong Wang +3Speculative Decoding Diffusion Language Model Decoding
AMAP, Alibaba Group
Reasoning-aware Speculative Decoding for Efficient Vision-Language-Action Models in Autonomous Driving
Jun 30, 2026Anh Dung Dinh, Simon Khan, Flora SalimEfficient VLM Inference Speculative Decoding
School of Computer Science and Engineering UNSW Sydney · Air Force Research Laboratory
When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding
Jun 29, 2026Aaryam SharmaSpeculative Decoding LLM Inference Acceleration
Independent Researcher Waterloo, Ontario, Canada
CSD: Content-aware Speculative Decoding for Efficient Image Generation
Jun 26, 2026Mingcheng Wang, Junbo Qiao, Yunchen Li +8Speculative Decoding Inference Acceleration
East China Normal University · Huawei Foundation Model · Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE
EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction
Jun 25, 2026Carrie ChenMulti-Token Prediction LLM Inference Scheduling
Cornell University
HyperDFlash: Hyper-Connection-Aligned Block Speculative Decoding with Gated Residual Reduction
Jun 25, 2026Luxi Lin, Shuang Peng, Rui Ma +7Hyper-Connections Speculative Decoding
ByteDance
Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion
Jun 23, 2026Sahil KadadekarLanguage Model Safety Evaluation Speculative Decoding
Independent Researcher
P-MTP: Efficient Document Parsing via Multi-Token Prediction with Progressive Depth Scaling
Jun 23, 2026Le Xiang, Chenxi Zhai, Shu Wei +5Multi-Token Prediction Efficient VLM Inference
Department of Computer Vision Technology (VIS), Baidu Inc., China · Tsinghua University, Shenzhen International Graduate School, China
Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding
Jun 23, 2026WenHung Lee, Jian-Jia Chen, Xiaolin Lin +6Long-Context Language Model Inference KV-Cache Management
National Yang Ming Chiao Tung University · Technische Universität Darmstadt · Cornell University
RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving
Jun 22, 2026Haifeng Wu, Srinivasan Manoharan, Fangbo Tu +2LLM Inference Serving Speculative Decoding
PayPal
SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation
Jun 18, 2026Shilong Xiang, Zirui Zhang, Lijun Yu +1Visual Autoregressive Models Speculative Decoding
Rutgers University
SafeSpec: Fast and Safe LLM via Dynamic Reflective Sampling
Jun 18, 2026Haotian Xu, Zeyang Zhang, Linbao Li +3Speculative Decoding LLM Inference Acceleration
Zhejiang University, Hangzhou, China · Harbin Institute of Technology, Shenzhen, China · Huawei
EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts
Jun 17, 2026Minseo Kim, Minjae Lee, Seunghyuk Oh +7RL for Language Models Reinforcement Learning
FuriosaAI · University of California, Berkeley
JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
Jun 16, 2026Lanxiang Hu, Zhaoxiang Feng, Yulun Wu +9Speculative Decoding LLM Inference Acceleration
UC San Diego · Zhejiang University · UIUC +2

Jul 18, 2026J/K move · Enter open · S save

SpecLA: Efficient Speculative Decoding for Linear-Attention Models

Zhibin Wang, Xuying Han, Zhaohua Yang, Fuliang Liu+4

State Key Laboratory of Novel Software Technology Nanjing University Nanjing, China · Alibaba Group Hangzhou, China

Speculative Decoding LLM Inference Acceleration Linear Attention

Open paper AskPDF arXiv

Linear-attention models replace the growing KV cache with recurrent states, but autoregressive decoding still reads, updates, and writes these states one token at a time. Speculative decoding can reduce this cost by verifying several draft tokens in one target pass, yet existing speculative systems are designed for Transformer KV caches. For stateful linear-attention targets, verification must follow recurrent dependencies across chains and branches, acceptance must update only the accepted state trajectory, and the drafter must avoid submitting candidates that waste stateful verification work. This paper presents SpecLA, a speculative decoding runtime for stateful linear-attention models. SpecLA verifies chains and trees with topology-aware kernels, stores compact factors produced during verification to recover accepted states, and uses confidence pruning plus a target-aligned EAGLE-style drafter to feed useful candidates to the verifier. On an NVIDIA H100 with a public GDN-1.3B target, SpecLA achieves up to 1.70x end-to-end speedup over autoregressive decoding.