Parallel Speculative Decoding

Follow this topic Ask about this topic

Latest papers

28 papers12 papers in the last four weeks, with none the four weeks before. 0.1% of all new papers.

JetSpec++: Foresight Parallel Drafting with Self Refinements
Oct 14, 2026Zhaoxiang Feng, Lanxiang Hu, Haoran Yuan +6Parallel Speculative Decoding Speculative Decoding
UC San Diego · UIUC · Nanjing University +2
Contextual Correction of Parallel Draftsfor Speculative Decoding
Oct 8, 2026Roy Eisenstadt, Harel Lidar, Tal Gonen +5Speculative Decoding LLM Inference Acceleration
Stealth Startup, Tel Aviv · Tel Aviv University · Ben-Gurion University
Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
Oct 7, 2026Yunxiao Zhao, Changxiao CaiSpeculative Decoding Parallel Speculative Decoding
Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, USA.
SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling
Oct 4, 2026Jeonghoon Park, Seongwoon Jo, Jongwon Lee +1Long-Context Language Model Inference Speculative Decoding
UNIST
DRelay: Global Draft Context for Prefix-Aware Parallel Speculative Decoding Repair
Oct 1, 2026Zhuoyu Wang, Junnan Huang, Xinyu ChenParallel Speculative Decoding Speculative Decoding
The Hong Kong University of Science and Technology (Guangzhou)
UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decoding
Sep 30, 2026Chumeng Liang, Linxuan Wang, Xinyu Peng +6Parallel Speculative Decoding Speculative Decoding
Inclusion AI · University of Illinois Urbana-Champaign · Purdue University
AS $^2$ D: Accelerating On-Demand Audio Understanding on Mobile Devices
Sep 29, 2026Yunzhe Li, Kyoungjun Park, Hongzi Zhu +1On-Device Language Model Inference Language Model Decoding
The University of Texas at Austin · Shanghai Jiao Tong University
LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
Sep 29, 2026Hao-Yuan He, Peng-Fei Liu, Si Shen +1Long-Context Language Model Inference Blockwise Diffusion Language Models
Draft in Parallel, Condition Through Depth: Adjacent Causal Injection for Speculative Decoding
Sep 28, 2026Haohui Zhang, Keyu Chen, Haocheng Sun +4Parallel Speculative Decoding Speculative Decoding
Shanghai Jiao Tong University · Tencent YouTu Lab · Xiamen University
Whisper-Flash: Acoustically Conditioned Parallel Drafting for Faster Whisper Decoding
Sep 26, 2026Huapeng Zhou, Huayu Wang, Junkai Wu +2Automatic Speech Recognition Parallel Speculative Decoding
Boson AI · University of Washington · Independent Researcher +1
When Parallel Drafter Meets Parallel Speculative Decoding
Sep 23, 2026Fuliang Liu, Xue Li, Kun Qian +4Parallel Speculative Decoding Speculative Decoding
State Key Laboratory of Novel Software Technology, Nanjing University · Alibaba Group
H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
Sep 21, 2026Weifan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn +6KV-Cache Management Speculative Decoding
Harvard University · Capital One · Red Hat +2
Approximate Speculative Decoding
Aug 4, 2026Yuannuo Feng, Zegang Peng, Yuxin Xie +5Speculative Decoding LLM Inference Acceleration
School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2
CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding
Aug 1, 2026Aofan Liu, Jingxiang Meng, Fangxin Liu +1Speculative Decoding LLM Inference Acceleration
Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
Jul 19, 2026Param ChordiyaOn-Device Language Model Inference Speculative Decoding
University of California, San Diego
Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
Jul 12, 2026Zipeng Gao, Zhi Zheng, Qingrong Xia +5Speculative Decoding LLM Inference Acceleration
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Huawei Technologies Co., Ltd.
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
Jul 8, 2026Hong-Kai Zheng, Piji LiSpeculative Decoding LLM Inference Acceleration
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China
D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
Jun 3, 2026Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6Speculative Decoding for Diffusion Language Models Speculative Decoding
Peking University · Tsinghua University · HKUST +2
TreeFlash: Parallel AR-Approximation for Faster Speculative Decoding
Jun 2, 2026Peer Rheinboldt, Frédéric Berdoz, Roger WattenhoferSpeculative Decoding LLM Inference Acceleration
ETH Zurich
Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism
May 29, 2026Yijiong Yu, Huazheng Wang, Shuai Yuan +2Speculative Decoding LLM Inference Acceleration
aOregon State University · bDeepSolution
Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding
May 28, 2026Jianuo Huang, Yaojie Zhang, Qituan Zhang +3Parallel Speculative Decoding Speculative Decoding
1EPIC Lab, Shanghai Jiao Tong University · School of Software Engineering, HUST · 3UESTC +2
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
May 19, 2026Yaojie Zhang, Jianuo Huang, Junlong Ke +5Parallel Speculative Decoding Speculative Decoding
EPIC Lab, SJTU · School of Software Engineering, HUST · UESTC +3
PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding
May 15, 2026Shengyin Sun, Yiming Li, Renxi Liu +7Parallel Speculative Decoding Masked Diffusion Models
Department of Computer Science City University of Hong Kong · Huawei Technologies · City University of Hong Kong
SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
May 4, 2026Jincheng Xie, Yawen Ling, Qi Xiao +4Parallel Speculative Decoding LLM Serving
1Tsinghua University · 2AI Infra Team at JDT · 3JD iCity, JD Technology, JD Intelligent Cities Research
Accelerating Speculative Decoding with Block Diffusion Draft Trees
Apr 14, 2026Liran Ringel, Yaniv RomanoSpeculative Decoding Parallel Speculative Decoding
Department of Computer Science, Technion – Israel Institute of Technology. · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology.
DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification
Apr 8, 2026Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6Speculative Decoding LLM Inference Acceleration
Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1
Speculative Speculative Decoding
Mar 3, 2026Tanishq Kumar, Tri Dao, Avner MaySpeculative Decoding LLM Inference Acceleration
Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI
MineDraft: A Framework for Batch Parallel Speculative Decoding
Feb 24, 2026Zhenwei Tang, Arun Verma, Zijian Zhou +4Speculative Decoding LLM Inference Acceleration
College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1

Oct 14, 2026J/K move · Enter open · S save

JetSpec++: Foresight Parallel Drafting with Self Refinements

Zhaoxiang Feng, Lanxiang Hu, Haoran Yuan, Yu-Yang Qian+5

UC San Diego · UIUC · Nanjing University+2

Parallel Speculative Decoding Speculative Decoding LLM Inference Acceleration Parallel Decoding

Open paper AskPDF arXiv

Speculative decoding's effectiveness is governed by two quantities: the acceptance rate of drafted tokens and the cost of producing them. We show the two objectives can be co-optimized with algorithm-system co-design, and present JetSpec++, which retains a strong learned draft model while hiding its cost almost entirely. JetSpec++ combines three components: intra-block self refinement, which trains the draft model to iteratively correct its own parallel predictions within a block of tokens; inter-block foresight drafting, which begins the next block before the current one is verified; and pipelined drafting and verification, which overlaps the two stages so drafting's marginal wall-clock cost approaches zero. In comparison with state-of-the-art speculative decoding methods, JetSpec++ attains a wall-clock speedup of up to

7.45\times

over autoregressive (AR) decoding and

1.54\times

over DSpark, with up to 1,361 tokens per second for a single request on GB200, and an average acceptance length of 8.94 across seven benchmarks, while preserving the target model's output distribution. Our code and models are available at https://github.com/hao-ai-lab/JetSpec.