Speculative Decoding

Follow this topic Ask about this topic

Speculative decoding aims to reduce the time and computation needed to generate text by letting a drafter propose future tokens for a target model to check. Approaches include small auxiliary drafters, early-exit or multi-token prediction within the target model, and tree-based or block-parallel drafting; retrieval and task-specific signals can also supply candidates. Exact verification can preserve the target model’s output distribution, while relaxed acceptance rules trade some fidelity for speed. A central challenge is making gains in accepted tokens translate into real latency or throughput improvements across different workloads, hardware, and serving loads.

Latest papers

308 papers71 papers in the last four weeks, up 109% on the four weeks before. 0.4% of all new papers.

LOCUS: Locate Once, Check Until Settled. Draft-and-Verify Visual Grounding with a Box Head and Coordinate-Token Verification
Oct 26, 2026Muhammad TalhaVision-Language Grounding Visual Grounding
Independent Researcher
T2D: Target-to-Draft Model Pruning & Adaptation for Efficient Speculative Decoding
Oct 25, 2026Shahriar Kabir Nahin, Xiao Ye, Muhao Chen +2Speculative Decoding LLM Inference Acceleration
Bellini College of AI, Cybersecurity, and Computing, University of South Florida · School of Computing and Augmented Intelligence, Arizona State University · Department of Computer Science, University of California, Davis
NgramRL: Accelerating RL Post-Training for Structured Generation via Rollout-Aware N-gram Speculative Decoding
Oct 23, 2026Nilesh Prasad Pandey, Jangseon Park, Jason Kong +2Speculative Decoding LLM Inference Acceleration
University of California San Diego La Jolla, CA, USA · West Virginia University Morgantown, WV, USA
SPECMEM: Accelerating Agentic Tool Calling via Live Memory Management
Oct 22, 2026Zihe Song, Wenting Zhao, Ye Liu +4Tool-Using Language Agents LLM Inference Efficiency
Department of Computer Science, University of Illinois Chicago · Salesforce Research
Speculative Diffusion is a Tale of Two Tradeoffs
Oct 22, 2026Aaron Zorel, Guy Van den Broeck, Aditya Grover +1LLM Inference Speculative Decoding
Independent Researcher · University of California, Los Angeles
Evaluating the Applicability of LLM Inference Optimizations on Apple Silicon
Oct 22, 2026Kyrylo Yemets, Andrii Yaroshevych, Victor Muryn +5LLM Inference On-Device Language Model Inference
MacPaw AI&Research, Kyiv, Ukraine · Department of Artificial Intelligence, Lviv Polytechnic National University, Lviv, Ukraine · Institute of Mathematics of NAS of Ukraine, Kyiv, Ukraine
ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding
Oct 22, 2026Liang He, SiYuan Ma, Qishi Zhan +5Speculative Decoding LLM Inference Acceleration
Tongji University · Nanyang Technological University · Marquette University +4
RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training
Oct 22, 2026Dor Tsur, Maor Ashkenazi, Maximilian Golub +7KV-Cache Compression LLM Inference Acceleration
NVIDIA
Speculative Acceptance Does Not Decay with Position: A Full-Length Diagnostic on Long Chain-of-Thought with Independent Drafts
Oct 22, 2026Zhiming MengSpeculative Decoding CoT Reasoning
RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding
Oct 21, 2026Mingkang Liu, Weiming Meng, Peipei Jiao +4Speculative Decoding LLM Inference Acceleration
Understanding Target Probability Reuse in Speculative Decoding
Oct 21, 2026Jungseob Lee, Giyoon Jang, Chailin Nam +6Speculative Decoding Parallel Speculative Decoding
Korea University · Soongsil University
Target Regeneration Shrinks the Gains of Draft-Training Objectives in Speculative Decoding
Oct 21, 2026Jungseob Lee, Hogun Seong, Euijun Choi +1Speculative Decoding Language Model Distillation
Korea University · Yonsei University Mirae Campus
Adaptive Multi-Token Prediction: How far should we go?
Oct 20, 2026Emile Anand, Mohammed Rehan Parwani, Xinyuan CaoMulti-Token Prediction LLM Inference Acceleration
Georgia Institute of Technology
BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models
Oct 20, 2026Ning Zhang, Shihao Wang, Jinrui Zhang +2LLM Inference Acceleration Multi-Token Prediction
Department of Computing, The Hong Kong Polytechnic University
Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding
Oct 20, 2026Tianyu Wu, Zhuohan Wang, Jonathan Geuter +5Speculative Decoding LLM Inference Acceleration
Harvard University · Massachusetts Institute of Technology
ProVe: Proposer-Verifier Decoding
Oct 20, 2026Ziqi Jin, Weiwen Xu, Jiaxi Li +3LLM Inference Acceleration Speculative Decoding
Nanyang Technological University. · ByteDance.
ReAnchor: Reusing Rejected Verifier States for Diffusion Drafting
Oct 19, 2026Sanghyun Lee, Wonjin Nam, Seungho Jang +4Speculative Decoding Speculative Decoding for Diffusion Language Models
KAIST · Seoul National University
Conjugate Compilation: Self-Improving Agentic Co-Compilation for Speculative Decoding
Oct 18, 2026Annabelle Sujun Tang, Rohan Mahapatra, Christopher Priebe +4LLM Inference Acceleration Speculative Decoding
University of California, San Diego · Massachusetts Institute of Technology · Systalyze +2
When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding
Oct 18, 2026Yongchao Long, Chunrui Zheng, Yuxuan Zhou +10Speculative Decoding LLM Inference Efficiency
Department of Computer Science, Tianjin University of Technology · Baidu Inc. · School of Information Science and Engineering, Shandong University +3
ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction
Oct 17, 2026Yuxuan Zhou, Junyuan Shang, Tianyu Wang +5Multi-Token Prediction Efficient Language Model Inference
ERNIE Team, Baidu · Independent Researcher
FlashMap: Flow Map Speculative Decoding
Oct 17, 2026Jeff Shen, Zhejian Zhou, Ryan Lee +2Speculative Decoding LLM Inference Acceleration
Polymathic AI · Princeton University · USC Information Sciences Institute +2
Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding
Oct 16, 2026Chiatzen Wang, Bill Xu, Haochen Gu +1Speculative Decoding LLM Inference Efficiency
University of Waterloo
DLite: History-free Speculative Decoding for Agentic AI
Oct 16, 2026Jiaming Xu, Hanzhen Wang, Tianlang Zhao +7Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai Innovation Institute · Tsinghua University +1
Trajectory-Triggered Sparse Collaborative Decoding
Oct 16, 2026Amit Ranjan Trivedi, Sruthikesh Surineni, Kyong Ho Lee +3Speculative Decoding LLM Reasoning
University of Illinois Chicago · Meta
Next Token or Vector? Learning How Far to Decode
Oct 16, 2026Sai Sreenivas Chintha, Chidera Biringa, Rishab Sharma +6LLM Inference Acceleration Multi-Token Prediction
Microsoft · University of Massachusetts Amherst
Speculative Forcing for Efficient Autoregressive Video Generation
Oct 15, 2026Hyunwook Choi, Seungryong Kim, Dongyoon Han +1Efficient Video Generation Autoregressive Video Generation
KAIST AI · NAVER AI Lab
LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models
Oct 15, 2026Zixuan Liu, Nan Xue, Rui Bao +4Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Tongji University
HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding
Oct 15, 2026Hanbin Bae, Heejin Choi, Joohyung Lee +3Speculative Decoding LLM Inference Acceleration
AI Solution Team, Samsung Research, Seoul, Republic of Korea
Long-Context Drafting Does Not Require Long-Context Drafters
Oct 14, 2026Aref Jafari, Vansh Bhatia, Matt Qin +5Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices Inc. (AMD) · University of Waterloo
ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models
Oct 14, 2026Bohan Sun, Zunhai Su, Xialie Zhuang +3LLM Inference Acceleration Speculative Decoding
StartLux · University of Chinese Academy of Sciences · The University of Hong Kong

Oct 26, 2026J/K move · Enter open · S save

LOCUS: Locate Once, Check Until Settled. Draft-and-Verify Visual Grounding with a Box Head and Coordinate-Token Verification

Muhammad Talha

Independent Researcher

Vision-Language Grounding Visual Grounding Efficient VLM Inference Speculative Decoding

Open paper AskPDF arXiv

Vision-language models (VLMs) ground a referring expression in one of two ways. They decode the box as coordinate tokens, which is precise but takes one sequential step per token, or they regress it from a hidden state with a small head, which takes one step but loses accuracy at high IoU. The two fail differently: a head usually finds the right object but places its edges loosely, while coordinate tokens place edges precisely. We therefore use the regression head as a speculative drafter for the model's own coordinate tokens. Locus writes the head's box as coordinate tokens and checks them with the same backbone by Jacobi iteration over a shared image prefix. The fixed point is the model's greedy coordinate-token answer, reached in about 5.60 sequential passes on average instead of 18.46 decoding steps; head and verifier are one set of weights, trained jointly. On the 8 RefCOCO, RefCOCO+ and RefCOCOg splits, verification raises Locus-2B's [email protected] from 54.87 for its own head to 59.13, and the one-pass head is ahead at [email protected] of every published model with at most 3B parameters except Florence-2-L. Against Jag, the strongest open regression-head model, at its own size and training budget, Locus ties at [email protected] and, with typical acceptance, gains +4.52 points at [email protected], with paired significance tests. Exact answers come 1.5x faster than vLLM greedy-decoding the same coordinate tokens with the same weights; a pure regression head remains faster still.