Speculative Decoding

Follow this topic Ask about this topic

Speculative decoding aims to reduce the time and computation needed to generate text by letting a drafter propose future tokens for a target model to check. Approaches include small auxiliary drafters, early-exit or multi-token prediction within the target model, and tree-based or block-parallel drafting; retrieval and task-specific signals can also supply candidates. Exact verification can preserve the target model’s output distribution, while relaxed acceptance rules trade some fidelity for speed. A central challenge is making gains in accepted tokens translate into real latency or throughput improvements across different workloads, hardware, and serving loads.

Latest papers

309 papers74 papers in the last four weeks, up 118% on the four weeks before. 0.4% of all new papers.

Watermarkable Multi-Draft Speculative Sampling via Poisson Processes
Sep 18, 2026Yanxiao Liu, Sicheng Wan, Zhan Gao +1Speculative Decoding LLM Inference Acceleration
Imperial College London · University of Washington
RheoSampling: Resolving the One-Hot Dilemma in Stochastic Dynamic-Tree Speculative Decoding
Sep 18, 2026Qiao Hu, Yepeng Weng, Bo Zhang +1Speculative Decoding LLM Inference Acceleration
National Center for Mathematics and Interdisciplinary Sciences (NCMIS), AMSS, CAS · The University of Tokyo · Lenovo AI Technology Center +2
SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference
Sep 18, 2026Harish KB, Jagadeeswaran M, Pradheep P +2LLM Inference LLM Inference Acceleration
Computer Science and Engineering Vellore Institute of Technology Vellore, India
To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals
Sep 17, 2026Roy Eisenstadt, Ido Cohen, Edo Cohen-Karlik +2Speculative Decoding LLM Inference Acceleration
Tel Aviv University · Stealth Startup, Tel Aviv
ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
Sep 16, 2026Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao +2Long-Context Language Model Inference LLM Inference Scheduling
University of Southern California, Los Angeles, CA, USA
ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding
Sep 15, 2026Ziyang Ma, Zihong Zhang, Zuchao Li +4Speculative Decoding LLM Inference Acceleration
School of Computer Science, Wuhan University, Wuhan, China · School of Artificial Intelligence, Wuhan University, Wuhan, China · 3Xiaomi, Beijing, China
LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
Sep 15, 2026SangLyul Cho, Langqing Cui, Sehoon Kim +2Efficient Transformer Inference Speculative Decoding
1Seoul National University · 2KAIST
GrowMTP: Can RL Grow Its Own Draft Head?
Sep 15, 2026Minghua He, Lingzhe Zhang, Yuan Liu +2Speculative Decoding LLM Inference Acceleration
WeChat AI, Tencent · Peking University
Evaluating Losslessness in Speculative Decoding Under Finite-Precision Inference
Sep 14, 2026Ilya Koziev, Leonid Sinev, Ivan OseledetsSpeculative Decoding LLM Inference Acceleration
Carryover Drafting: Recycling Rejected States for Speculative Decoding
Sep 13, 2026Jahyun Koo, Sunghyeon Woo, Jaeeun Kil +4Speculative Decoding LLM Inference Acceleration
Seoul National University · NAVER Cloud
Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding
Sep 8, 2026Fengxiang Bie, Yuqing Jian, Yifan Yu +7Language Model Pretraining Speculative Decoding
Together AI · The University of Sydney, Australia · University of Illinois Urbana-Champaign, USA +1
Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training
Sep 7, 2026Zili Wang, Zhaopeng Qiu, Yuekai Zhang +2Context Parallelism Speculative Decoding
NVIDIA
SJD-SV: Speculative Jacobi Decoding with Semantics Verification for Autoregressive Image Generation
Sep 4, 2026Baoquan Zhang, Bingqi Shan, Shihao Fang +3Speculative Decoding Autoregressive Image Generation
Shenzhen Key Laboratory of Internet Information Collaboration, Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory, Shenzhen
Jina-OCR-v1: Efficient Document Parsing with Speculative Decoding and Dense Verifiable Rewards
Sep 2, 2026Alejandro Barón García, Feng Wang, Emilia Garcia Casademont +1Efficient VLM Inference Speculative Decoding
Jina AI by Elastic · New Montgomery Street, San Francisco, CA 94105, USA
SFAD: Speculative Factuality-Aware Decoding
Sep 1, 2026Guanqiao Chen, Di Wang, Lijie HuLLM Hallucination Mitigation Speculative Decoding
1MBZUAI · University of Science and Technology of China · 3Provable Responsible AI and Data Analytics (PRADA) Lab +1
Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models
Aug 31, 2026Jungseob Lee, Seongtae Hong, Dongyub Jude Lee +4Efficient VLM Inference Speculative Decoding
Korea University · Zoom Communications · Soongsil University +2
Ceiling-Clipped Acceptance Histograms Indicate Stranded Speed-up in Block-Diffusion Speculative Decoding
Aug 31, 2026Ephrem WuSpeculative Decoding Diffusion Language Model Decoding
Advanced Micro Devices, Inc., USA
Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
Aug 31, 2026Tong Yuan, Chengxi Liao, Zeyi WenMemory-Augmented Language Models KV Caching
Data Science and Analytics Thrust, Information Hub, The Hong Kong University of Science and Technology (Guangzhou)
Verification-Aware Training for Speculative Decoding
Aug 31, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative Decoding LLM Inference Acceleration
NAVER AI Lab · NAVER AI Search Platform · Korea University
ReTrace: Rejected-Trajectory Conditioning for Speculative Decoding
Aug 30, 2026Luxi Lin, Zhanpeng Zeng, Shuang Peng +2Speculative Decoding LLM Inference Acceleration
1Xiamen University, 2ByteDance
A rigor-matched audit of periodic-step layer skipping for efficient llm inference: conflayers versus swift, with a supplemental analysis of trained routing alternatives
Aug 28, 2026Prateek Kumar SikdarLayer Skipping in Language Model Decoding LLM Inference Acceleration
Accenture
Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting
Aug 27, 2026Xinwei Qiang, Xiang Fang, Chang Chen +3Speculative Decoding LLM Inference Acceleration
University of California San Diego
Trajectory-Level Speculative Decoding for Diffusion Language Models
Aug 27, 2026Tianxiang Pan, Baitao Gong, Mo Guang +5Speculative Decoding for Diffusion Language Models Diffusion Language Model Decoding
Li Auto Inc., Shanghai, China
AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs
Aug 26, 2026Sheng Liang, Yongyue Zhang, Nathanael Brian +4Speculative Decoding LLM Inference Acceleration
Huawei Technologies Co., Ltd. · University of Science and Technology of China
The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts
Aug 26, 2026Aidar Amankulov, Denis MamatinSpeculative Decoding Mixture-of-Experts Inference
ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping
Aug 25, 2026Zhi-Kai Chen, Jun-Jie Tao, Wei-Xiang Mao +2Speculative Decoding LLM Inference Acceleration
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Nanjing University, China
AgentSpec: Speculative Decoding for Batch Inference of LLM Agents
Aug 25, 2026Xin Wang, Ziming Miao, Yi Zhu +4Speculative Decoding LLM Inference Acceleration
The Ohio State University · Microsoft Research · University of Michigan
TreeWY: Speculative Verification for Gated DeltaNet Hybrids
Aug 21, 2026Sneha Murthy GhantasalaSpeculative Decoding LLM Inference Acceleration
Thomson Reuters
Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosis
Aug 21, 2026Yantao Li, Huanlin Gao, Fang Zhao +12Speculative Decoding Speculative Decoding for Diffusion Language Models
National Key Laboratory for Novel Software Technology, Nanjing University · Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom
LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding
Aug 20, 2026Matan Rusanovsky, Yoav Miron, Roy Uziel +5Speculative Decoding LLM Inference Acceleration
NVIDIA

Sep 18, 2026J/K move · Enter open · S save

Watermarkable Multi-Draft Speculative Sampling via Poisson Processes

Yanxiao Liu, Sicheng Wan, Zhan Gao, Deniz Gündüz

Imperial College London · University of Washington

Speculative Decoding LLM Inference Acceleration LLM Watermarking

Open paper AskPDF arXiv

Large language models (LLMs) have achieved state-of-the-art performance across a wide range of tasks, motivating two important aspects of deployment: inference efficiency and output provenance, which can be tackled by speculative sampling and watermarking, respectively. However, recent works have shown that combining these two goals is highly nontrivial and can be potentially impossible. In this work, we develop a novel multi-draft speculative sampling algorithm based on Poisson processes that improves the frontier of this fundamental trade-off. The proposed algorithm has strong sampling efficiency on its own and, more interestingly, is naturally watermarkable: we can embed an unbiased watermark without degrading speculative acceptance. Moreover, our algorithm is based on an exact list-coupling-without-communication scheme, which yields a drafter invariance property that benefits both sampling and watermarking. It is the first multi-draft, drafter-invariant speculative sampling scheme that maintains both watermark strength and sampling efficiency, and we experimentally verify its strong performance in both aspects.