From Positionwise Confidence to Prefix Scheduling: Verifier Skipping in Speculative Decoding

Haoxuan Luo; Jameson Sandler; Ferdinando Fioretto

cs.CRAug 14, 2026

From Positionwise Confidence to Prefix Scheduling: Verifier Skipping in Speculative Decoding

Authors: Haoxuan Luo, Jameson Sandler, Ferdinando Fioretto

Organizations: University of Virginia

Abstract

Speculative decoding is a leading technique to reduce the cost of autoregressive generation by using a small drafter to propose several tokens, which are then verified in parallel by a larger target model. Speculative diffusion decoding (SDD) further removes sequential drafting by generating every position in a draft block in parallel with a discrete diffusion model. However, SDD still invokes the target on every block, leaving verification as a potential bottleneck. This paper recognizes that this creates a new control handle: whether to invoke the verifier at all. Thus, we study verifier skipping, a lossy policy that commits a selected draft prefix directly, and ask which confidence signal should schedule it. Interestingly, our study finds that better token predictors need not yield better schedulers: skips require contiguous high-confidence prefixes, while short skips can induce additional drafting rounds. To study this mismatch, we compare raw confidence with learned marginal and conditional survival scores under the same policy, using Strict SDD, lenience, and top- $k$ acceptance as baselines. On HumanEval with DiffuCoder-7B-Instruct and Qwen3-32B, all three confidence signals save $9.6\%$ to $13.5\%$ of verifier calls at the same observed pass@1 as Strict SDD. Surprisingly, raw confidence saves the most; marginal survival has higher positionwise AUROC than raw confidence at most positions, yet neither learned signal dominates online. Our analysis shows that verifier skipping is a useful new lossy axis and, surprisingly, its key challenge is prefix scheduling rather than token prediction alone.

Figures & tables

Figure 1: HumanEval pass@1 versus decode throughput with DiffuCoder-7B-Instruct and Qwen3-32B, using draft length

\gamma=32

. Raw confidence, marginal survival, and conditional survival use the same verifier-skipping policy; Strict SDD, lenience, and top-

k

acceptance are shown as baselines. Curves are obtained by sweeping the fixed parameter grid for each method.

Figure 2: Confidence-guided verifier skipping. The outer path shows the decoding loop: the drafter generates a block, then the policy either commits the selected prefix directly without verification or executes a strict round. The center shows how position scores are used to select the longest prefix passing the local and prefix gates. The skip test checks the minimum skip length, staleness, and remaining guards.

Signal	Position score $q_{j}$	Prefix score $C_{k}$
Raw confidence	$p_{\mathrm{draft}}(\hat{x}_{j})$	$\left(\prod_{j=1}^{k}q_{j}\right)^{1/k}$
Marginal survival	$\hat{a}_{j}$	$\min_{j\leq k}\hat{a}_{j}$
Conditional survival	$\hat{s}_{j}$	$\prod_{j=1}^{k}\hat{s}_{j}$

Table 1: Position and prefix scores for the three confidence signals.

Figure 3: Positionwise BCE for marginal and conditional survival on HumanEval with draft length

\gamma=32

. Dashed curves show position-only baselines estimated from the training folds.

Figure 4: HumanEval pass@1 and strict token agreement versus relative verifier calls with draft length

\gamma=32

. Curves connect the plotted settings for each method; faded markers show additional settings.

Panel of Figure 5 — Figure 5: Effect of $K_{\min}$ on verifier calls, draft fragmentation, and throughput for raw confidence. The remaining policy thresholds are fixed, and all points use the same timing setup.

Signal	Original	Permuted mean
Raw confidence	8.75	1.59
Marginal survival	6.86	1.00
Conditional survival	8.76	2.11

Table 2: Mean feasible prefix length before and after permutation. Per-position BCE and AUROC are unchanged by construction.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Positionwise AUROC on the held-out HumanEval folds with draft length

\gamma=32

. The conditional evaluation includes positions with at least ten positive and ten negative examples.

Signal	Policy	Pass@1	Tokens/s	Relative calls	Relative blocks
Raw confidence	Full	0.8902	58.50	0.8170	1.135
	Local only	0.8902	58.24	0.8146	1.135
	Geometric only	0.8598	60.82	0.7659	1.118
Conditional survival	Full	0.8841	58.11	0.8175	1.137
	Product only	0.8841	58.11	0.8175	1.137
	Local only	0.7805	58.96	0.7896	1.130

Appendix

Table 3: Fixed threshold gate ablation on HumanEval at

\gamma=32

. Variants are not retuned. Calls and blocks are normalized by Strict SDD.

$P$	Pass@1	Tokens/s	Rel. calls	Agreement
3	0.5854	63.42	0.7939	0.7417
4	0.6768	60.50	0.8502	0.7542
8	0.7256	56.80	0.9540	0.6989

Appendix

Table 4: Periodic skipping on HumanEval at

\gamma=32

. Relative calls are normalized by Strict SDD, and agreement denotes strict token agreement. Shadow verification is excluded from calls and throughput.

Dataset	Method	$S_{\max}$	Score	Tokens/s	Relative calls	Strict agreement
MBPP+	Strict SDD	–	0.7593	23.03	1.0000	1.0000
	Raw confidence	32	0.7646	25.64	0.8318	0.7503
	Marginal survival	32	0.7434	24.14	0.9106	0.9344
	Conditional survival	32	0.7513	23.35	0.9577	0.9544
MATH subset	Strict SDD	–	0.4733	15.85	1.0000	1.0000
	Raw confidence	24	0.4767	16.53	0.9359	0.9656

Appendix

Table 5: Transfer results with DiffuCoder-7B-Instruct and Qwen2.5-72B-Instruct at

\gamma=16

. Score is the fraction of MBPP+ problems passing the plus tests and normalized answer exact match for the MATH subset. Calls are normalized by Strict SDD. Shadow verification is excluded from calls and throughput.

Method	Values
Lenience	$\ell\in\{1.0,0.99,0.975,0.95,0.90,0.80,0.70,0.50,0.30\}$
Top- $k$	$k\in\{2,3,5\}$ ; $k=1$ is Strict SDD
Raw confidence	$\eta_{b}\in\{0.90,0.95\}$ and $\eta_{c}\in\{0.995,0.99,0.98,0.97,0.95,0.93,0.90\}$ , using every pairing; additional pairs $(0.80,0.93)$ , $(0.80,0.85)$ , $(0.70,0.70)$ , and $(0.60,0.50)$
Marginal survival	$\eta_{m}\in\{0.85,0.90,0.93,0.95,0.97,0.98,0.99\}$
Conditional survival	$\eta_{b}=0.90$ and $\eta_{c}\in\{0.99,0.98,0.97,0.95,0.93,0.90,0.85,0.80\}$

Appendix

Table 6: Hyperparameter grids used in the reported sweeps.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

Aug 4, 2026

Approximate Speculative Decoding

Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel. Under standard greedy verification, decoding stops at the first draft token that differs from the target argmax, discarding the remaining target-scored suffix. Although accepting such a mismatch changes the decoding trajectory, it can make a contiguous suffix reusable when its tokens remain target-greedy under the realized prefix. In this paper, we introduce Approximate Speculative Decoding (ASD), a training-free verifier that replaces binary first-mismatch truncation with budgeted longest-prefix selection. ASD accepts selected mismatches subject to a local target-logit regret gate, a per-block exception cap, and a persistent request-level regret budget, then reuses the contiguous target-greedy suffix without additional approximate decisions or target-model forward passes. ASD requires neither a new draft model nor fine-tuning, and exactly reduces to standard greedy verification when the budget is zero. Experiments show that ASD improves fixed-workload throughput by

3.05\%

15.26\%

over matched strict verification and averages a

7.78\%

gain across seven Qwen3-14B + DSpark-14B tasks. On DeepSeek-V4-Flash (284B) with DSpark it also raises verifier-side acceptance by roughly

10\%

16\%

on GSM8K and MATH-500 in an FP4-to-FP8 compatibility setting. The source code is publicly available at: https://github.com/Kissmetothemoon/ASD

Yuannuo Feng, Zegang Peng, Yuxin Xie +5

School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2

Speculative Decoding LLM Inference Acceleration Autoregressive Decoding Parallel Speculative Decoding

Ask PDF

Jun 3, 2026

D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models

Speculative decoding accelerates autoregressive large language model inference by drafting multiple tokens and verifying them in a single target-model forward pass. Recent diffusion-based drafters generate an entire block of tokens in parallel but usually commit to a single draft sequence per verification: once the first mismatch occurs, all subsequent draft tokens are discarded, resulting in a limited acceptance rate. Naively batching more draft candidate sequences only introduces a marginal improvement, as redundant or poorly placed branches increase the cost of drafting and verification without proportionally increasing the number of accepted tokens. We propose D^2SD, a dual diffusion draft speculative decoding framework that organizes candidates into a confidence-guided prefix tree, where the first diffusion drafter generates a block along with per-position confidence scores that are used to identify the most likely rejection boundary and select the top-K prefix ranges for recovery; the second variable-prefix diffusion drafter re-anchors at each selected prefix and proposes alternative continuations in one batched pass; the resulting shared-prefix candidates are jointly verified via cascade attention. Empirically, D^2SD shows clear improvements over both the underlying diffusion approach and strong autoregressive speculative decoding baselines.

Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6

Peking University · Tsinghua University · HKUST +2

Speculative Decoding for Diffusion Language Models Speculative Decoding Diffusion Language Model Inference LLM Inference Acceleration Diffusion Language Model Decoding Parallel Speculative Decoding

Ask PDF

Apr 8, 2026

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose Dynamic Verification Relaxed Speculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: https://github.com/comeusr/diversed.

Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6

Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1

Speculative Decoding LLM Inference Acceleration Parallel Speculative Decoding

Ask PDF

Abstract

Speculative Decoding

Explore similar work

Approximate Speculative Decoding

D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification