Speculative Acceptance Does Not Decay with Position: A Full-Length Diagnostic on Long Chain-of-Thought with Independent Drafts

Zhiming Meng

cs.AIOct 22, 2026

Speculative Acceptance Does Not Decay with Position: A Full-Length Diagnostic on Long Chain-of-Thought with Independent Drafts

Authors: Zhiming Meng

Abstract

The claim that speculative-decoding acceptance rate decays as a chain-of-thought grows longer has become a de facto industry narrative, motivating draft-model co-design and position-aware training. Nearly all direct evidence comes from trained drafters whose offline training length is far shorter than inference, so a purely positional law has never been isolated. We present the first systematic, full-length measurement of acceptance rate as a function of generation position on MATH-500 (all 500 problems), under greedy and sampling decoding, with two independent drafts against DeepSeek-R1-Distill-Qwen-7B: a same-family draft (DeepSeek-R1-Distill-Qwen-1.5B) and a cross-family draft (Qwen2.5-1.5B-Instruct), yielding about 1.78M verification rounds. Using within-problem paired tests, we find that after isolating a greedy-decoding repetition pathology, acceptance rate does not decay with position in any of the four configurations within the supported ranges (greedy 0-8K, sampling 0-12K); the second half of each trajectory is slightly but significantly higher than the first. A controlled temperature sweep shows that the apparent sampling-mode degradation of high-entropy reflection tokens is a visibility artifact of argmax agreement, not a drop in expected acceptance probability. Rejections concentrate with high precision on reflection trigger tokens such as "wait", "however" and "check", an entropy-selective failure that a lightweight three-arm LoRA trajectory-alignment intervention (uniform, reflection-marked, entropy-weighted) fails to improve in mean accepted length. We thus supply the positional baseline that independent draft-target pairs were missing, and localize the residual failure to decision-point entropy rather than position.

Figures & tables

Figure 1: Acceptance rate vs. generation position. (a) Same-family, all problems: the greedy curve rises and saturates after 12K—almost entirely driven by loop problems. (b) Same-family after removing them: no downward trend, and if anything a slight rise (greedy

0.91

–

0.94

, sampling

0.856

–

0.882

). (c) Cross-family (loops removed) shows the same shape at a lower level. Effective support: greedy 0–8K, sampling 0–12K; beyond these ranges few problems remain and the curves become noisy (Sec. 8 ), so we do not draw conclusions there.

Config	Metric	Diff	95% CI	Up/Down	$p$
same/greedy	def1	$+0.031$	$[+0.026,+0.036]$	327/126	$1.2{\times}10^{-21}$
same/sample	def1	$+0.028$	$[+0.023,+0.033]$	354/130	$5.2{\times}10^{-25}$
xfam/greedy	def1	$+0.069$	$[+0.063,+0.076]$	393/61	$1.8{\times}10^{-60}$
xfam/sample	def1	$+0.059$	$[+0.053,+0.065]$	407/78	$9.6{\times}10^{-55}$
same/sample	$\alpha$	$+0.010$	$[+0.008,+0.013]$	333/151	$8.3{\times}10^{-17}$
xfam/sample	$\alpha$	$+0.030$	$[+0.027,+0.034]$	365/120	$8.7{\times}10^{-30}$

Table 1: Within-problem paired half-difference (second

-

first) after loop removal. def1

=\mathbb{E}[\mathrm{accepted}]/\gamma

;

\alpha=\mathbb{E}[\min(1,p/q)]

is the true expected acceptance probability; it is reported only for sampling configurations because greedy decoding does not record position-level

p,q

(Sec. 3 ). 95% CIs are problem-level bootstrap (10,000 resamples); “Up/Down” counts problems whose second-half acceptance exceeds / falls below the first half (zero differences excluded);

p

is the exact two-sided sign test. def2 (argmax-match rate) curves appear in Fig. 1 and are excluded from paired tests because def2 is not an expectation over the sampling law (Sec. 6 ).

Config	Full	Loop removed
same/greedy	$+0.064$	$+0.025$
same/sample	$+0.006$	$+0.009$
xfam/greedy	$+0.172$	$+0.090$
xfam/sample	$+0.060$	$+0.056$

Table 2: Aggregate half-difference before/after loop removal. Unlike Table 1 , rounds are pooled across problems and split at the global median position, so loop problems (which contribute most long-position rounds) dominate the pre-removal column; the paired column of Table 1 weights every problem equally and is the primary statistic.

Figure 2: First-half vs. second-half acceptance (def1), before and after loop removal. Bars use the problem-weighted paired estimator of Table 1 : each problem contributes one half-pair, so the 46 loop problems have bounded influence and the greedy excess shrinks only modestly after removal (

+0.040\to+0.031

; sampling

+0.027\to+0.028

). The dominant loop effect is a round-weighting phenomenon: in the pooled statistics (Table 2 :

+0.064\to+0.025

) loop rounds dominate long positions, and the position curves flatten accordingly (Fig. 1 (a) vs. (b)).

Figure 3: def2 (visible argmax agreement, both modes) and

\alpha

(true expected acceptance probability, sampling only—greedy records no position-level

p,q

) by token type, non-loop. Failure is entropy-selective: number/operator are near-perfect, reflection tokens are the clear failure point, and cross-family misalignment amplifies it.

Figure 4: Mismatch structure at reflection triggers.

Figure 5: Within-problem position quintiles: acceptance is flat-to-rising after loop removal.

$T$	MAL	def2	$\alpha$	reflection def2	reflection $\alpha$
0.3	4.986	0.917	0.927	0.707	0.785
0.6	4.967	0.886	0.922	0.636	0.802
0.9	4.884	0.836	0.915	0.520	0.809

Table 3: Temperature sweep (same-family sampling, first 100 problems, loop removed). def2 falls sharply with

T

, but the true expected acceptance probability

\alpha

barely moves—the apparent degradation of reflection tokens is a visibility artifact.

Figure 6: MAL at 500 problems under the three intervention arms and the untrained base draft (same-family; error bars: standard error). No arm improves over the uniform training control in either decoding mode.

Arm	MAL samp	MAL greedy	$\Delta_{\mathrm{samp}}$ [95% CI]	$\Delta_{\mathrm{greedy}}$ [95% CI]
base (no training)	$4.990{\pm}.012$	$5.060{\pm}.013$	–	–
uniform	$4.961{\pm}.013$	$5.077{\pm}.013$	–	–
reflection-marked	$4.947{\pm}.013$	$5.058{\pm}.013$	$-0.0138$ $[-0.032,+0.005]$	$-0.0192$ $[-0.042,+0.003]$
entropy-weighted	$4.954{\pm}.013$	$5.072{\pm}.013$	$-0.0072$ $[-0.026,+0.012]$	$-0.0055$ $[-0.008,-0.003]$

Table 4: Intervention results at 500 problems (same-family, both decoding modes). MAL is reported with standard errors.

\Delta

columns are problem-paired differences against the uniform arm—the proper control, since it undergoes identical training on the same trajectories with flat weighting—computed from unrounded per-problem MAL, so a

\Delta

need not equal the difference of the two rounded column entries; 95% CIs are from paired

t

-tests. No arm improves over the uniform control; answer accuracy is identical across arms (McNemar

\chi^{2}{\approx}0

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Cross-family counterparts of Figs. 3 and 5 : (left) token-type breakdown and (right) within-problem position quintiles.

Figure 8: Token-type breakdown for the intervention experiment (100-problem diagnostic subset).

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

Jun 29, 2026

When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

Speculative decoding accelerates language model inference by using a fast drafter to propose candidate tokens that are then verified by a larger target model. Existing theory largely studies the stochastic, distribution-preserving setting, where the goal is to exactly sample from the target distribution. In contrast, many practical systems use greedy decoding, relaxed acceptance rules, or tree-based candidate sets, where success is governed by local ranking and threshold events rather than exact distributional equality. We develop a theory for these regimes. We identify that many common acceptance criteria have rejection regions that can be characterized as lower level sets of the target distribution. For these, we characterize the exact KL divergence required for rejection yielding exact certificates and sharp margin-based bounds for strict greedy decoding, additive and multiplicative relaxed acceptance, top-(m) relaxed criteria, and entropy-thresholded acceptance. We then extend the framework to greedy tree decoding, deriving exact and margin-only certificates for when the target greedy token remains covered by the drafter's top-(m) candidates. Finally, we evaluate the resulting certificates on Qwen3 models, showing that relaxed and tree-based criteria substantially enlarge the region of certified acceptance, especially on decoding steps with low target model distribution margin. These results complement existing distribution-preserving analyses of speculative decoding by characterizing the deterministic local acceptance events common in practical inference systems.

Aaryam Sharma

Independent Researcher Waterloo, Ontario, Canada

Speculative Decoding LLM Inference Acceleration

Ask PDF

Oct 20, 2026

Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding

In speculative decoding, a lightweight draft model proposes tokens that the target model verifies in parallel, speeding up generation without changing the target distribution. D-PACE (Wu et al., 2026) trains such drafters with a loss that weights each position by its contribution to the expected accepted length. Both its weights and its cross-entropy (CE) loss use only the draft's probability of the recorded target token, treating the target as a point mass. Under rejection sampling, however, a sampled draft token is accepted with probability equal to the overlap of the draft and target distributions, one minus their total variation (TV). We derive D-PAL (Dynamic Position-wise Acceptance Loss) from the exact expected accepted length, using this overlap as the acceptance signal. This yields overlap-based position weights with log acceptance (LA) as the per-position loss, and the objective reduces to D-PACE for a point-mass target. Since the weights are detached, any per-position loss can replace LA in practice. We show that forward KL divergence and Rényi divergence of order 1/2 match or exceed LA, and that Rényi-1/2 provably bounds the true acceptance rate from above and below. Replacing CE with Rényi-1/2 improves acceptance length under both D-PACE's weights and ours. On six benchmarks with a Qwen3-4B target, D-PAL raises mean acceptance length over D-PACE from 4.50 to 4.57 at T=0 and from 4.20 to 4.28 at T=1, and consistently achieves higher throughput at both temperatures.

Tianyu Wu, Zhuohan Wang, Jonathan Geuter +5

Harvard University · Massachusetts Institute of Technology

Speculative Decoding LLM Inference Acceleration

Ask PDF

Apr 16, 2026

Acceptance Dynamics Across Cognitive Domains in Speculative Decoding

Speculative decoding accelerates large language model (LLM) inference. It uses a small draft model to propose a tree of future tokens. A larger target model then verifies these tokens in a single batched forward pass. Despite the growing body of work on speculative methods, the degree to which the cognitive characteristics of a task affect acceptance probability remains largely unexplored. We present an empirical study of tree-based speculative decoding acceptance dynamics. Our study spans four well-established NLP benchmark domains: code generation, mathematical reasoning, logical reasoning, and open-ended chat. For this, we use TinyLlama-1.1B as the draft model against Llama-2-7B-Chat-GPTQ as the target. Over 99,768 speculative nodes collected from 200 prompts, we derive per-domain acceptance rates, expected accepted lengths, depth-acceptance profiles, and entropy-acceptance correlations. We find that task type is a stronger predictor of acceptance than tree depth. Furthermore, only the chat domain consistently yields an expected accepted length exceeding 1.0 token per step. We also show that the entropy-acceptance correlation is consistently negative but weak across all domains (rho in [-0.20, -0.15]). Counterintuitively, chat produces the highest entropy yet the highest acceptance rate. We attribute this divergence to the lexical predictability of RLHF-aligned register. These findings have direct implications for domain-aware speculation budgets and draft-model selection strategies. Index Terms--speculative decoding, large language model inference, tree attention, draft model, acceptance probability, LLM efficiency

Seifeldin Abdellatif

College of Engineering, Al Ain University Abu Dhabi, United Arab Emirates

Speculative Decoding Text Generation

Ask PDF

Abstract

Speculative Decoding

Explore similar work

When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding

Acceptance Dynamics Across Cognitive Domains in Speculative Decoding