Speculative Decoding

Follow this topic Ask about this topic

Speculative decoding aims to reduce the time and computation needed to generate text by letting a drafter propose future tokens for a target model to check. Approaches include small auxiliary drafters, early-exit or multi-token prediction within the target model, and tree-based or block-parallel drafting; retrieval and task-specific signals can also supply candidates. Exact verification can preserve the target model’s output distribution, while relaxed acceptance rules trade some fidelity for speed. A central challenge is making gains in accepted tokens translate into real latency or throughput improvements across different workloads, hardware, and serving loads.

Latest papers

308 papers71 papers in the last four weeks, up 109% on the four weeks before. 0.4% of all new papers.

S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculation
Mar 26, 2026Ligong Han, Hao Wang, Han Gao +2Speculative Decoding for Diffusion Language Models Diffusion Language Model Decoding
Red Hat AI Innovation · MIT-IBM Watson AI Lab · Iowa State University +1
RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialogue
Mar 24, 2026Long Mai, Junli LiangSpoken Dialogue Systems Speculative Decoding
Trinity College Dublin, Ireland · University College Dublin, Ireland
Cross-Family Speculative Decoding for Polish Language Models on Apple~~Silicon: An Empirical Evaluation of Bielik~~11B with UAG-Extended MLX-LM
Mar 22, 2026Krzysztof FonalSpeculative Decoding LLM Inference Acceleration
Wrocław University of Science and Technology, Poland
ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding
Mar 20, 2026Quan Kong, Yuhao Shen, Yicheng Ji +2Efficient VLM Inference Video-Language Models
Zhejiang University, Hangzhou, China
SJD-PAC: Accelerating Speculative Jacobi Decoding via Proactive Drafting and Adaptive Continuation
Mar 19, 2026Jialiang Kang, Han Shu, Wenshuo Li +2Speculative Decoding Autoregressive Image Generation
Peking University · Huawei Technologies
SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Mar 19, 2026Shenggui Li, Chao Wang, Yikai Zhu +14Speculative Decoding LLM Inference Acceleration
Nanyang Technological University · SpecForge Team · Meituan +2
Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing
Mar 18, 2026Raghavv Goel, Mukul Gagrani, Mingu Lee +1Multi-Token Prediction LLM Inference Acceleration
Qualcomm AI Research
HeiSD: Hybrid Speculative Decoding for Embodied Vision-Language-Action Models with Kinematic Awareness
Mar 18, 2026Zihao Zheng, Zhihao Mao, Sicheng Tian +8Efficient VLA Model Inference Speculative Decoding
School of Computer Science, Peking University · School of Computer Science, China University of Geosciences (Wuhan) · School of Artificial Intelligence, Beijing Normal University +1
MMSpec: Benchmarking Speculative Decoding for Vision-Language Models
Mar 16, 2026Hui Shen, Xin Wang, Ping Zhang +11Speculative Decoding Efficient VLM Inference
University of Michigan · The Ohio State University · Independent +5
When Drafts Evolve: Speculative Decoding Meets Online Learning
Mar 13, 2026Yu-Yang Qian, Hao-Cong Wu, Yichao Fu +2Speculative Decoding LLM Inference Acceleration
State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · University of California, San Diego
Edge-Cloud Collaborative Speech Emotion Captioning via Token-Level Speculative Decoding in Audio-Language Models
Mar 12, 2026Xiangyuan Xue, Jiajun Lu, Yan Gao +3Edge-Cloud Collaborative Inference Speculative Decoding
The University of Auckland, New Zealand · The University of Melbourne, Australia · The University of Cambridge, United Kingdom +1
Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation
Mar 10, 2026Luxi Lin, Zhihang Lin, Zhanpeng Zeng +5Speculative Decoding LLM Inference Acceleration
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Innovation Institute +1
ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
Mar 10, 2026Xinyi Hu, Yuhao Shen, Baolin Zhang +6Speculative Decoding LLM Inference Acceleration
Qwen Applications Business Group of Alibaba · Zhejiang University
ConFu: Contemplate the Future for Better Speculative Sampling
Mar 9, 2026Zongyue Qin, Raghavv Goel, Mukul Gagrani +3Speculative Decoding LLM Inference Acceleration
University of California Los Angeles, United States · Qualcomm AI Research, United States.
EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs
Mar 9, 2026Chang Han, Yijie Hu, Jingling LiuSpeculative Decoding LLM Inference Acceleration
College of Computer Science, Chongqing University
Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding
Mar 5, 2026Ofir Ben ShohamLLM Inference Acceleration Speculative Decoding
Intuit
Speculative Speculative Decoding
Mar 3, 2026Tanishq Kumar, Tri Dao, Avner MaySpeculative Decoding LLM Inference Acceleration
Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI
Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models
Mar 3, 2026Shubhangi Upasani, Ravi Shanker Raju, Bo Li +5Prompt Compression Long-Context Language Model Inference
SambaNova AI
SJD-PV: Speculative Jacobi Decoding with Phrase Verification for Autoregressive Image Generation
Mar 2, 2026Zhehao Yu, Baoquan Zhang, Bingqi Shan +5Speculative Decoding Autoregressive Image Generation
Harbin Institute of Technology, Shenzhen · SIFAR
Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning
Mar 2, 2026Jiebin Zhang, Zhenghan Yu, Liang Wang +8Speculative Decoding LLM Inference Acceleration
National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · Microsoft Research Asia
Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification
Mar 2, 2026Guang Huang, Zeyi WenLLM Inference Acceleration Speculative Decoding
HKUST(GZ), Guangzhou, China · HKUST, Hong Kong, China
Task-Centric Acceleration of Small-Language Models
Feb 27, 2026Dor Tsur, Sharon Adar, Ran LevyLanguage Model Vocabulary Expansion Efficient Language Model Inference
Amazon
LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding
Feb 27, 2026Alexander Samarin, Sergei Krutikov, Anton Shevtsov +3Speculative Decoding LLM Inference Acceleration
Nebius, Amsterdam, Netherlands
Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple
Feb 25, 2026Amirhossein Bozorgkhoo, Igor MolybogSpeculative Decoding LLM Inference Acceleration
Independent Researcher · University of Hawai’i at Manoa, Honolulu, HI, USA.
MineDraft: A Framework for Batch Parallel Speculative Decoding
Feb 24, 2026Zhenwei Tang, Arun Verma, Zijian Zhou +4Speculative Decoding LLM Inference Acceleration
College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1
KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem
Feb 23, 2026Seongjin Cha, Gyuwan Kim, Dongsu Han +2LLM Inference Acceleration Speculative Decoding
School of Electrical Engineering, KAIST, Daejeon, South Korea · Department of Computer Science, University of California, Santa Barbara, CA, USA
Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding
Feb 19, 2026Rahul Thomas, Teo Kitanovski, Micah Goldblum +1Speculative Decoding LLM Inference Acceleration
Ritual · Columbia University · Vanderbilt University
Greedy Multi-Path Block Verification for Faster Decoding in Speculative Sampling
Feb 18, 2026Rahul Thomas, David Hidary, Arka PalSpeculative Decoding LLM Inference Acceleration
Ritual · Columbia University
MoE-Spec: Expert Budgeting for Efficient Speculative Decoding
Feb 17, 2026Bradley McDanel, Steven Li, Sruthikesh Surineni +1Speculative Decoding LLM Inference Acceleration
Franklin and Marshall College · Meta Reality Labs
Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs
Feb 17, 2026Libo Zhang, Zhaoning Zhang, Wangyang Hong +2Speculative Decoding Long-Context VLMs
National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology, Changsha, China.

Mar 26, 2026J/K move · Enter open · S save

S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculation

Ligong Han, Hao Wang, Han Gao, Kai Xu+1

Red Hat AI Innovation · MIT-IBM Watson AI Lab · Iowa State University+1

Speculative Decoding for Diffusion Language Models Diffusion Language Model Decoding Diffusion Model Inference Acceleration Blockwise Diffusion Language Models Speculative Decoding

Open paper AskPDF arXiv

Block-diffusion language models offer a promising path toward faster-than-autoregressive generation by combining block-wise autoregressive decoding with within-block parallel denoising. However, in the few-step regime needed for practical acceleration, standard confidence-thresholded decoding is often brittle: aggressive thresholds hurt quality, while conservative thresholds require unnecessary denoising steps. Existing approaches that address this issue either require additional training or incur extra test-time compute. We present S2D2, a training-free self-speculative decoding framework for block-diffusion language models. Our key observation is that a block-diffusion model becomes autoregressive when the block size is reduced to one, allowing the same pretrained model to act as both drafter and verifier. S2D2 inserts a speculative verification step into standard block-diffusion decoding and uses lightweight routing policies to decide when verification is worth its cost. This yields a hybrid decoding trajectory in which diffusion proposes tokens in parallel, while the autoregressive mode acts as a local sequence-level critic. Across three mainstream block-diffusion families, S2D2 consistently improves the accuracy-speed tradeoff over strong confidence-thresholding baselines. On SDAR, we observe up to

4.7\times

speedup over autoregressive decoding, and up to

1.57\times

over a tuned dynamic decoding baseline while improving accuracy by up to

4.5

points. On LLaDA2.1-Mini, S2D2 remains complementary to built-in self-correction, including a conservative setting where it is

4.4\times

faster than the static baseline with slightly higher accuracy.