Speculative Decoding

Follow this topic Ask about this topic

Speculative decoding aims to reduce the time and computation needed to generate text by letting a drafter propose future tokens for a target model to check. Approaches include small auxiliary drafters, early-exit or multi-token prediction within the target model, and tree-based or block-parallel drafting; retrieval and task-specific signals can also supply candidates. Exact verification can preserve the target model’s output distribution, while relaxed acceptance rules trade some fidelity for speed. A central challenge is making gains in accepted tokens translate into real latency or throughput improvements across different workloads, hardware, and serving loads.

Latest papers

309 papers72 papers in the last four weeks, up 112% on the four weeks before. 0.4% of all new papers.

When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?
Apr 29, 2026Tianyu Liu, Yuhao Shen, Xinyi Hu +8Speculative Decoding LLM Inference Acceleration
1Qwen Applications Business Group of Alibaba · University of Science and Technology of China · 3Zhejiang University
AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices
Apr 28, 2026Ma Zirui, Fan Zhihua, Li Wenxing +4Compute-in-Memory On-Device Language Model Inference
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM
Apr 20, 2026Sravanth Kodavanti, Sowmya Vajrala, Srinivas Miriyala +13On-Device Language Model Inference LLM Inference
Samsung Research Institute Bangalore, India · Samsung Electronics, Suwon, South Korea
WISV: Wireless-Informed Semantic Verification for Distributed Speculative Decoding in Device-Edge LLM Inference
Apr 20, 2026Zixuan Liu, Zhiyong Chen, Nan Xue +4Speculative Decoding Efficient Language Model Inference
Cooperative Medianet Innovation Center and the Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Speculative Decoding for Autoregressive Video Generation
Apr 19, 2026Yuezhou Hu, Jintao ZhangVideo Diffusion Models Autoregressive Diffusion
University of California, Berkeley
EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation
Apr 17, 2026Shuyu Zhang, Lingfeng Pan, Qicheng Wang +6Speculative Decoding LLM Inference Acceleration
School of Computer Science and Technology Xidian University Xi’an, China
Faster LLM Inference via Sequential Monte Carlo
Apr 17, 2026Yahya Emara, Mauricio Barba da Costa, Chi-Chih Chang +4Speculative Decoding LLM Inference Acceleration
Cornell University · Makora · MIT +1
From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning
Apr 16, 2026Kiran Purohit, Ramasuri Narayanam, Soumyabrata PalLanguage Model Self-Verification LLM Answer Verification
IIT Kharagpur · Adobe Research
RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding
Apr 16, 2026Zihong Zhang, Zuchao Li, Lefei Zhang +2Speculative Decoding LLM Inference Acceleration
School of Artiﬁcial Intelligence, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University, Wuhan, China · School of Information Management, Wuhan University, Wuhan, China +1
Acceptance Dynamics Across Cognitive Domains in Speculative Decoding
Apr 16, 2026Seifeldin AbdellatifSpeculative Decoding Text Generation
College of Engineering, Al Ain University Abu Dhabi, United Arab Emirates
ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving
Apr 16, 2026Yuseon Choi, Jingu Lee, Jungjun Oh +5LLM Serving Mixture-of-Experts Inference
KAIST · Samsung Electronics · GIST
ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding
Apr 16, 2026Walaa Amer, Uday das, Fadi KurdahiSpeculative Decoding LLM Inference Acceleration
Department of Electrical Engineering and Computer Sciences, University of California Irvine, Irvine, USA · Advanced Micro Devices, San Jose, USA
Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference
Apr 15, 2026Xuwen Zhou, Fangxin Liu, Chao Wang +5Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Alibaba Cloud Computing
ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decoding
Apr 15, 2026Heming Xia, Yongqi Li, Cunxiao Du +2LLM Inference Acceleration LLM Tool Use
Department of Computing, The Hong Kong Polytechnic University · Peking University
Accelerating Speculative Decoding with Block Diffusion Draft Trees
Apr 14, 2026Liran Ringel, Yaniv RomanoSpeculative Decoding Parallel Speculative Decoding
Department of Computer Science, Technion – Israel Institute of Technology. · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology.
SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding
Apr 14, 2026Shaowen Chen, Zhicheng Liao, Hongwei WangSpeculative Decoding LLM Inference Acceleration
Zhejiang University, Hangzhou, China.
SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration
Apr 14, 2026Zhuofan Wen, Yang FengSpeculative Decoding LLM Inference Acceleration
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding
Apr 11, 2026Jehyeon Bang, Eunyeong Cho, Ranggi Hwang +2Mixture-of-Experts Inference LLM Inference Acceleration
KAIST · UNIST
SMART: When is it Actually Worth Expanding a Speculative Tree?
Apr 9, 2026Lifu Wang, Pan ZhouInference-Time Optimization GPU Acceleration
Singapore Management University, Singapore
ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
Apr 8, 2026Xiangchen Li, Saeid Ghafouri, Jiakun Fan +3LLM Inference Efficiency Disaggregated LLM Serving
Virginia Tech, Blacksburg, Virginia, USA · Queen’s University Belfast, Belfast, Northern Ireland, UK
DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification
Apr 8, 2026Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6Speculative Decoding LLM Inference Acceleration
Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1
NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies
Apr 8, 2026Zhiyang Chen, Daliang Xu, Yinyuan Zhang +3Speculative Decoding LLM Inference Acceleration
Institute for Artificial Intelligence, Peking University, Beijing, China · State Key Laboratory of Networking and Switching Technology; School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China · Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education; School of Computer Science, Peking University, Beijing, China
See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs
Apr 7, 2026Yicheng Ji, Jun Zhang, Jinpeng Chen +4Speculative Decoding LLM Inference Acceleration
ZJU · BUPT
Multi-Drafter Speculative Decoding with Alignment Feedback
Apr 7, 2026Taehyeon Kim, Hojung Jung, Se-Young YunSpeculative Decoding LLM Inference Acceleration
LG AI Research · KAIST AI
Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling
Apr 5, 2026Yongchang Hao, Lili MouSpeculative Decoding LLM Inference Acceleration
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta · Canada CIFAR AI Chair
Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding
Apr 2, 2026Tao Jin, Phuong Minh Nguyen, Naoya InoueSpeculative Decoding LLM Inference Acceleration
Japan Advanced Institute of Science and Technology (JAIST)
SpecTr-GBV: Multi-Draft Block Verification Accelerating Speculative Decoding
Apr 1, 2026Yijun Lin, Jinhao Sheng, Qingyue Cai +1Speculative Decoding LLM Inference Acceleration
Center for Applied Statistics and School of Statistics, Renmin University of China · School of Intelligent Medicine, China Medical University · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing
Drift-AR: Single-Step Visual Autoregressive Generation via Anti-Symmetric Drifting
Mar 30, 2026Zhen Zou, Xiaoxiao Ma, Mingde Yao +3Autoregressive Diffusion Visual Autoregressive Models
University of Science and Technology of China · The Chinese University of Hong Kong, MMLAB · JD Explore Academy +1
SJD-VP: Speculative Jacobi Decoding with Verification Prediction for Autoregressive Image Generation
Mar 28, 2026Bingqi Shan, Baoquan Zhang, Xiaochen Qi +3Speculative Decoding Autoregressive Image Generation
Harbin Institute of Technology, Shenzhen · Institute of Perceptual Intelligence, Pengcheng Laboratory, Shenzhen · SIFAR
TAPS: Task Aware Proposal Distributions for Speculative Sampling
Mar 27, 2026Mohamad Zbib, Mohamad Bazzi, Ammar Mohanna +2Speculative Decoding LLM Inference Acceleration
King Abdullah University of Science and Technology (KAUST) · American University of Beirut (AUB)

Apr 29, 2026J/K move · Enter open · S save

When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?

Tianyu Liu, Yuhao Shen, Xinyi Hu, Baolin Zhang+7

1Qwen Applications Business Group of Alibaba · University of Science and Technology of China · 3Zhejiang University

Speculative Decoding LLM Inference Acceleration KV-Cache Reuse

Open paper AskPDF arXiv

Speculative decoding accelerates LLM inference, but SOTA hidden-state-based drafters suffer from long-range decay: draft accuracy degrades as the speculative step increases. Existing work attributes this decay to train-inference mismatch and proposes test-time training (TTT) as a remedy, yet we observe that long-range decay persists even in TTT-trained drafters. We revisit long-range decay from the perspective of context information preservation. In hidden-state reuse, we argue the target hidden state acts as a biased context compression: it aggregates historical token information according to the attention query at the current position, yielding a compact representation optimized for immediate next-token prediction. This compression can suppress information less relevant to the current query but important for later speculative steps. In contrast, the target model's KV cache serves as an explicit context, retaining the complete set of token-wise KV representations. We therefore posit the KV-Reuse Hypothesis: allowing the draft model to reuse the target KV cache can provide richer signals for long-horizon drafting. To test this hypothesis, we introduce KVShot, a diagnostic framework that compares three reuse paradigms: hidden-only, KV-only, and hybrid. Extensive evaluations on Qwen3-8B show that KV-Reuse improves long-range acceptance, although end-to-end speedups remain marginal under current training pipelines. Our analysis identifies two key structural bottlenecks: shallow drafters struggle to estimate target queries accurately, and draft-side KV projections receive sparse gradient signals. These findings suggest that realizing the full potential of KV-aware decoding requires moving beyond TTT toward block-wise training paradigms. By exposing these bottlenecks, KVShot provides a foundational diagnostic testbed and a clear roadmap for designing next-generation inference architectures.