Speculative Decoding

Follow this topic Ask about this topic

Speculative decoding aims to reduce the time and computation needed to generate text by letting a drafter propose future tokens for a target model to check. Approaches include small auxiliary drafters, early-exit or multi-token prediction within the target model, and tree-based or block-parallel drafting; retrieval and task-specific signals can also supply candidates. Exact verification can preserve the target model’s output distribution, while relaxed acceptance rules trade some fidelity for speed. A central challenge is making gains in accepted tokens translate into real latency or throughput improvements across different workloads, hardware, and serving loads.

Latest papers

308 papers71 papers in the last four weeks, up 109% on the four weeks before. 0.4% of all new papers.

HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding
Feb 13, 2026Wenhui Liao, Hongliang Li, Pengyu Xie +15Efficient VLM Inference Speculative Decoding
South China University of Technology, Guangzhou, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Nanjing University, Nanjing, China +1
MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
Feb 12, 2026Shuhuai Li, Jianghao Lin, Dongdong Ge +1Mixture-of-Experts Inference Edge Inference
Shanghai University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Standford University, California, United States
TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
Feb 5, 2026Jiyoung Park, Hankyu Jang, Changseok Song +1Adaptive Inference LLM Inference Serving
Moreh, Inc.
Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States
Feb 3, 2026Ximing Dong, Shaowei Wang, Dayi Lin +2Speculative Decoding LLM Inference Acceleration
Centre for Software Excellence, Huawei, Canada · Department of Computer Science, University of Manitoba, Canada · School of Computing, Queen’s University, Canada
JuDi: Revisiting Judge Decoding from First Principles via Training-Free Distributional Divergence
Jan 8, 2026Shengyin Sun, Yiming Li, Renxi Liu +5Speculative Decoding LLM Inference Acceleration
City University of Hong Kong · Huawei Technologies
Entropy-Aware Token Rejection for Improving Speculative Decoding
Dec 29, 2025Tiancheng Su, Meicong Zhang, Guoxiu HeSpeculative Decoding LLM Inference Acceleration
School of Economics and Management East China Normal University
Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problem
Oct 26, 2025Ranran Haoran Zhang, Soumik Dey, Ashirbad Mishra +3Speculative Decoding Efficient Language Model Inference
The Pennsylvania State University, University Park, PA, USA · eBay Inc, San Jose, CA, USA.
Continuous Speculative Decoding for Autoregressive Image Generation
Nov 18, 2024Zili Wang, Zheng Zhang, Kun Ding +3Visual Autoregressive Models Speculative Decoding
School of Artificial Intelligence, University of Chinese Academy of Sciences, China · MAIS, Institute of Automation, Chinese Academy of Sciences, China · JD.COM Inc, China +1

Feb 13, 2026J/K move · Enter open · S save

HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding

Wenhui Liao, Hongliang Li, Pengyu Xie, Xinyu Cai+14

South China University of Technology, Guangzhou, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Nanjing University, Nanjing, China+1

Efficient VLM Inference Speculative Decoding Document Parsing Document Image Understanding

Open paper AskPDF arXiv

Document parsing is a fundamental task in multimodal understanding, supporting a wide range of downstream applications such as information extraction and intelligent document analysis. Benefiting from strong semantic modeling and robust generalization, VLM-based end-to-end approaches have emerged as the mainstream paradigm in recent years. However, these models often suffer from substantial inference latency, as they must autoregressively generate long, full-page sequences when processing long-form documents. While recent hybrid methods mitigate this issue via region-level parallel decoding with VLMs, independent region decoding loses full-page context and might weaken global coherence. To address this issue, we propose Hierarchical Speculative Decoding (HSD), a two-stage local-to-global framework for document parsing. HSD first employs a lightweight pipeline drafter to predict region partitions and generate coarse drafts for each region. The first stage verifies the generated region-level drafts in parallel for efficiency, while the second stage further performs page-level verification on these refined outputs to preserve full-page coherence. Experimental results show that HSD achieves a near-lossless 2.7x speedup with HunyuanOCR on OmniDocBench v1.5 and up to 7.04x speedup on long-document parsing tasks, demonstrating the effectiveness of the proposed method. The code is available at https://github.com/whlscut/HSD.