Speculative Decoding

Follow this topic Ask about this topic

Speculative decoding aims to reduce the time and computation needed to generate text by letting a drafter propose future tokens for a target model to check. Approaches include small auxiliary drafters, early-exit or multi-token prediction within the target model, and tree-based or block-parallel drafting; retrieval and task-specific signals can also supply candidates. Exact verification can preserve the target model’s output distribution, while relaxed acceptance rules trade some fidelity for speed. A central challenge is making gains in accepted tokens translate into real latency or throughput improvements across different workloads, hardware, and serving loads.

Latest papers

309 papers72 papers in the last four weeks, up 112% on the four weeks before. 0.4% of all new papers.

S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices
Aug 15, 2026Haochen Huang, Shengxuan Qiu, Meng LiLLM Inference Acceleration On-Device Language Model Inference
Institute for Artificial Intelligence, Peking University, Beijing, China · School of Integrated Circuits, Peking University, Beijing, China · School of Electronics Engineering and Computer Science, Peking University, Beijing, China
From Positionwise Confidence to Prefix Scheduling: Verifier Skipping in Speculative Decoding
Aug 14, 2026Haoxuan Luo, Jameson Sandler, Ferdinando FiorettoSpeculative Decoding Efficient Language Model Inference
University of Virginia
DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees
Aug 13, 2026Tianyi Li, Yaxin Luo, Xinyi Shang +1Speculative Decoding Diffusion Language Model Decoding
VILA Lab, MBZUAI
SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
Aug 13, 2026Divya Jyoti Bajpai, Kishan Kumar Upadhyay, Manjesh Kumar HanawalEdge-Cloud Collaborative Inference Edge Computing
MLiONS, Department of IEOR, IIT Bombay Mumbai, Maharashtra-400076, India
Decoupled Contrastive Decoding via Expert-Aligned Drafting
Aug 13, 2026Zhixuan Liu, Zhichen Dong, Yuanfu Wang +1Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
Alignment Drift in Single-Model Speculative Decoding for ASR: Mechanism, Correction, and Cost
Aug 13, 2026Xinyu Wang, Huapeng Zhou, Ziyu Zhao +5Audio-Text Alignment Speculative Decoding
1Boson AI · 2McGill University
TreeSpark: Calibrated, Load-Adaptive Draft Trees for Semi-Autoregressive Speculative Decoding
Aug 12, 2026Huapeng Zhou, Huayu Wang, Xinyu WangSpeculative Decoding LLM Inference Acceleration
Boson AI · University of Washington · McGill University
MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
Aug 11, 2026Eunjeong Kim, Yeong Jun Jeon, Myeonggyun HanOn-Device Language Model Inference LLM Inference Scheduling
Kyungpook National University Daegu, Republic of Korea
WA-SpecDec: World-Aware Speculative Decoding for Vision-Language-Action Models
Aug 9, 2026Zikang Wen, Yuning Zhang, Dong YuanEfficient VLA Model Inference Speculative Decoding
The University of Sydney
LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization
Aug 9, 2026Zexun Lin, Yuan Feng, Junlin Lv +2Inference-Time Optimization Speculative Decoding for Diffusion Language Models
Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, Jiangsu, China
DBLAST: Dependent Block Drafting for Stochastic Speculative Decoding
Aug 5, 2026Amirmohammad Karimi, Chao Gao, Negar HassanpourSpeculative Decoding LLM Inference Acceleration
Huawei Technologies Canada Co., Ltd.
EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding
Aug 5, 2026Sangwoo Ha, Hyunwoo Seo, Yurim Jo +2AI Accelerator Inference On-Device Language Model Inference
KAIST Daejeon, South Korea
SPARK: Learning from Verification in Speculative GRPO Rollouts
Aug 5, 2026Nhat Minh Pham, Duy Tung Doan, Quang Huy Phung +3RL for Language Model Reasoning Speculative Decoding
VNU University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam · Viettel AI, Viettel Group, Hanoi, Vietnam
Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes
Aug 4, 2026Tao Jin, Phuong Minh Nguyen, Zhenzhu Yan +2Speculative Decoding LLM Inference Acceleration
1Japan Advanced Institute of Science and Technology (JAIST) · 2Tohoku University
Approximate Speculative Decoding
Aug 4, 2026Yuannuo Feng, Zegang Peng, Yuxin Xie +5Speculative Decoding LLM Inference Acceleration
School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2
AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding
Aug 4, 2026Shuang Liang, Hao Mark Chen, Zhiwen Mo +4Mixture-of-Experts Inference Speculative Decoding
Imperial College London · Tile-AI
xPress: Parallel Refinement for Diffusion Drafters in Speculative Decoding
Aug 3, 2026Zheng Wang, Davis Wertheimer, Yu Chin Fabian Lim +4Speculative Decoding for Diffusion Language Models Speculative Decoding
University of Illinois Urbana-Champaign · IBM
From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decoding
Aug 3, 2026Zixian Li, Tong Li, Chi Xie +2Speculative Decoding LLM Inference Acceleration
OPPO AI Center
Bole: Efficient Tree Speculation for Hybrid-Attention Language Models
Aug 3, 2026Li Wang, Yi Su, Xiabao Wu +9Speculative Decoding LLM Inference Acceleration
State Key Laboratory for Novel Software Technology, Nanjing University · Ant Group · Shanghai Jiao Tong University +1
OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling
Aug 1, 2026Zhiheng Zhang, Mujie Xu, Feiyu Sun +1Speculative Decoding LLM Inference Acceleration
The University of Tokyo · Peking University · Nanjing University
CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding
Aug 1, 2026Aofan Liu, Jingxiang Meng, Fangxin Liu +1Speculative Decoding LLM Inference Acceleration
A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
Jul 30, 2026Yuesong Liu, Yuan Zeng, Min Lyu +3Long-Context Language Model Inference Speculative Decoding
University of Science and Technology of China
Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
Jul 29, 2026Weiye Shi, Fanxu Meng, Muhan ZhangSpeculative Decoding LLM Inference Acceleration
Institute for Artificial Intelligence, Peking University
Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes
Jul 29, 2026Tianyu Wang, Yuxuan Zhou, Wenbin Wang +3Speculative Decoding LLM Inference Acceleration
1Independent Researcher · 2Baidu Inc. · 3Zhejiang University
AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
Jul 28, 2026Hong Liu, Rui Cen, Junhan Shi +10LLM Inference Scheduling Speculative Decoding for Diffusion Language Models
Tencent Inc.
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
Jul 27, 2026Dengke HanExpert Offloading On-Device Language Model Inference
Adversarial Prompts for Acceptance Collapse in Speculative Decoding
Jul 23, 2026Run Wang, Chaoyi Zhou, Xi Liu +7Adversarial Prompt Generation Speculative Decoding
Clemson University · Wayne State University · University of Washington
Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Jul 23, 2026Alagappan ValliappanMulti-Token Prediction Long-Context Language Model Inference
NVIDIA
AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
Jul 21, 2026Yu-Yang Qian, Hao-Cong Wu, Chen Chen +4Speculative Decoding for Diffusion Language Models Speculative Decoding
State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Huawei Foundation Model Dept
Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
Jul 19, 2026Param ChordiyaOn-Device Language Model Inference Speculative Decoding
University of California, San Diego

Aug 15, 2026J/K move · Enter open · S save

S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices

Haochen Huang, Shengxuan Qiu, Meng Li

Institute for Artificial Intelligence, Peking University, Beijing, China · School of Integrated Circuits, Peking University, Beijing, China · School of Electronics Engineering and Computer Science, Peking University, Beijing, China

LLM Inference Acceleration On-Device Language Model Inference Speculative Decoding Mixture-of-Experts Inference

Open paper AskPDF arXiv

Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency, naively combining them often incurs excessive verification overhead and poor expert reuse, limiting their effectiveness in memory-bound edge settings. In this work, we propose S2-MoE, an efficient self-speculative decoding framework for MoE inference on edge devices. S2-MoE reduces redundant verification through routing-aware adaptive speculative expansion, improves verification efficiency with reuse-aware expert gating, and aligns draft and target execution via shared context. Implemented in llama

.

cpp, S2-MoE achieves up to

5.3\times

speedup (about

2.0\times

on average) over standard autoregressive decoding across diverse MoE models and datasets on edge devices. Code is available at https://github.com/angerybob/S2-MoE.