Parallel Speculative Decoding

Latest papers

37 papers7 papers in the last four weeks, down 22% on the four weeks before. 0.0% of all new papers.

HyFlash: A Prefix-Memory Prior for On-Device Speculative Decoding
Nov 5, 2026Jongwoo Ko, Pashmina CameronParallel Speculative Decoding Speculative Decoding
Microsoft Applied Sciences Group
Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding
Nov 3, 2026Hoang Anh Duy Le, Huibing Dong, Alireza Khadem +6Speculative Decoding LLM Inference Acceleration
Google · Rice University · Google Research +1
SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding
Nov 2, 2026Tianyu Liu, Jun Zhang, Wenjie Huang +12LLM Inference Acceleration On-Policy Distillation
Qwen Business Unit of Alibaba · University of Science and Technology of China · Zhejiang University +4
SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding
Nov 2, 2026Kunjun Li, Wenhao Chai, Yihao Liang +7Speculative Decoding LLM Inference Acceleration
University of California, Berkeley · Princeton University · Stanford University +1
Weights That Stay: On-Chip Megakernels for Serial Heads in Speculative Decoding
Oct 31, 2026Xinwei Qiang, Ziyi Xu, Zaifeng Pan +2LLM Inference Acceleration Parallel Speculative Decoding
University of California, San Diego · RadixArk
CopyLoop: Fast Looped Transformer Decoding via Context Copying and Shallow Screening
Oct 28, 2026Wenyu Du, Stephen ChungEfficient Language Model Inference Recurrent Transformers
DualverseAI
Understanding Target Probability Reuse in Speculative Decoding
Oct 21, 2026Jungseob Lee, Giyoon Jang, Chailin Nam +6Speculative Decoding Parallel Speculative Decoding
Korea University · Soongsil University
ProVe: Proposer-Verifier Decoding
Oct 20, 2026Ziqi Jin, Weiwen Xu, Jiaxi Li +3LLM Inference Acceleration Speculative Decoding
Nanyang Technological University. · ByteDance.
JetSpec++: Foresight Parallel Drafting with Self Refinements
Oct 14, 2026Zhaoxiang Feng, Lanxiang Hu, Haoran Yuan +6Parallel Speculative Decoding Speculative Decoding
UC San Diego · UIUC · Nanjing University +2
Contextual Correction of Parallel Draftsfor Speculative Decoding
Oct 8, 2026Roy Eisenstadt, Harel Lidar, Tal Gonen +5Speculative Decoding LLM Inference Acceleration
Stealth Startup, Tel Aviv · Tel Aviv University · Ben-Gurion University
Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
Oct 7, 2026Yunxiao Zhao, Changxiao CaiSpeculative Decoding Parallel Speculative Decoding
Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, USA.
SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling
Oct 4, 2026Jeonghoon Park, Seongwoon Jo, Jongwon Lee +1Long-Context Language Model Inference Speculative Decoding
UNIST
DRelay: Global Draft Context for Prefix-Aware Parallel Speculative Decoding Repair
Oct 1, 2026Zhuoyu Wang, Junnan Huang, Xinyu ChenParallel Speculative Decoding Speculative Decoding
The Hong Kong University of Science and Technology (Guangzhou)
UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decoding
Sep 30, 2026Chumeng Liang, Linxuan Wang, Xinyu Peng +7Parallel Speculative Decoding Speculative Decoding
Inclusion AI · University of Illinois Urbana-Champaign · Purdue University
AS $^2$ D: Accelerating On-Demand Audio Understanding on Mobile Devices
Sep 29, 2026Yunzhe Li, Kyoungjun Park, Hongzi Zhu +1On-Device Language Model Inference Language Model Decoding
The University of Texas at Austin · Shanghai Jiao Tong University
LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
Sep 29, 2026Hao-Yuan He, Peng-Fei Liu, Chen Chen +6Long-Context Language Model Inference Blockwise Diffusion Language Models
Draft in Parallel, Condition Through Depth: Adjacent Causal Injection for Speculative Decoding
Sep 28, 2026Haohui Zhang, Keyu Chen, Haocheng Sun +4Parallel Speculative Decoding Speculative Decoding
Shanghai Jiao Tong University · Tencent YouTu Lab · Xiamen University
Whisper-Flash: Acoustically Conditioned Parallel Drafting for Faster Whisper Decoding
Sep 26, 2026Huapeng Zhou, Huayu Wang, Junkai Wu +2Automatic Speech Recognition Parallel Speculative Decoding
Boson AI · University of Washington · Independent Researcher +1
When Parallel Drafter Meets Parallel Speculative Decoding
Sep 23, 2026Fuliang Liu, Xue Li, Kun Qian +4Parallel Speculative Decoding Speculative Decoding
State Key Laboratory of Novel Software Technology, Nanjing University · Alibaba Group
H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
Sep 21, 2026Weifan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn +6KV-Cache Management Speculative Decoding
Harvard University · Capital One · Red Hat +2
Approximate Speculative Decoding
Aug 4, 2026Yuannuo Feng, Zegang Peng, Yuxin Xie +5Speculative Decoding LLM Inference Acceleration
School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2
CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding
Aug 1, 2026Aofan Liu, Jingxiang Meng, Fangxin Liu +1Speculative Decoding LLM Inference Acceleration
Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
Jul 19, 2026Param ChordiyaOn-Device Language Model Inference Speculative Decoding
University of California, San Diego
Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
Jul 12, 2026Zipeng Gao, Zhi Zheng, Qingrong Xia +5Speculative Decoding LLM Inference Acceleration
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Huawei Technologies Co., Ltd.
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
Jul 8, 2026Hong-Kai Zheng, Piji LiSpeculative Decoding LLM Inference Acceleration
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China
D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
Jun 3, 2026Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6Speculative Decoding for Diffusion Language Models Speculative Decoding
Peking University · Tsinghua University · HKUST +2
TreeFlash: Parallel AR-Approximation for Faster Speculative Decoding
Jun 2, 2026Peer Rheinboldt, Frédéric Berdoz, Roger WattenhoferSpeculative Decoding LLM Inference Acceleration
ETH Zurich
Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism
May 29, 2026Yijiong Yu, Huazheng Wang, Shuai Yuan +2Speculative Decoding LLM Inference Acceleration
aOregon State University · bDeepSolution
Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding
May 28, 2026Jianuo Huang, Yaojie Zhang, Qituan Zhang +3Parallel Speculative Decoding Speculative Decoding
1EPIC Lab, Shanghai Jiao Tong University · School of Software Engineering, HUST · 3UESTC +2
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
May 19, 2026Yaojie Zhang, Jianuo Huang, Junlong Ke +5Parallel Speculative Decoding Speculative Decoding
EPIC Lab, SJTU · School of Software Engineering, HUST · UESTC +3

Nov 5, 2026J/K move · Enter open · S save

HyFlash: A Prefix-Memory Prior for On-Device Speculative Decoding

Jongwoo Ko, Pashmina Cameron

Microsoft Applied Sciences Group

Parallel Speculative Decoding Speculative Decoding Memory-Efficient Inference Memory-Augmented Language Models Hybrid Attention

Open paper AskPDF arXiv

Parallel block drafting accelerates speculative decoding by predicting multiple future positions in a single backbone pass, but on-device deployment constrains both drafter capacity and context storage. Under tight memory budgets, drafters must model long-range dependencies and fine-grained draft-position interactions without the growing KV-cache overhead of full-context attention. We propose HyFlash, a compact parallel drafter that introduces a prefix-memory prior to separate long-range context modeling from fine-grained draft-position interactions with limited capacity. Local attention captures recent context and draft-position interactions, while recurrent memory compresses historical target features into fixed-size states. This inductive bias enables effective local and long-range modeling with bounded draft-side memory. Experiments show that HyFlash achieves higher acceptance than full-context attention with substantially less KV storage and with larger gains for shallow drafters and long contexts.