Speculative Acceptance Does Not Decay with Position: A Full-Length Diagnostic on Long Chain-of-Thought with Independent Drafts

Zhiming Meng

cs.AIOct 22, 2026

Speculative Acceptance Does Not Decay with Position: A Full-Length Diagnostic on Long Chain-of-Thought with Independent Drafts

Authors: Zhiming Meng

Abstract

The claim that speculative-decoding acceptance rate decays as a chain-of-thought grows longer has become a de facto industry narrative, motivating draft-model co-design and position-aware training. Nearly all direct evidence comes from trained drafters whose offline training length is far shorter than inference, so a purely positional law has never been isolated. We present the first systematic, full-length measurement of acceptance rate as a function of generation position on MATH-500 (all 500 problems), under greedy and sampling decoding, with two independent drafts against DeepSeek-R1-Distill-Qwen-7B: a same-family draft (DeepSeek-R1-Distill-Qwen-1.5B) and a cross-family draft (Qwen2.5-1.5B-Instruct), yielding about 1.78M verification rounds. Using within-problem paired tests, we find that after isolating a greedy-decoding repetition pathology, acceptance rate does not decay with position in any of the four configurations within the supported ranges (greedy 0-8K, sampling 0-12K); the second half of each trajectory is slightly but significantly higher than the first. A controlled temperature sweep shows that the apparent sampling-mode degradation of high-entropy reflection tokens is a visibility artifact of argmax agreement, not a drop in expected acceptance probability. Rejections concentrate with high precision on reflection trigger tokens such as "wait", "however" and "check", an entropy-selective failure that a lightweight three-arm LoRA trajectory-alignment intervention (uniform, reflection-marked, entropy-weighted) fails to improve in mean accepted length. We thus supply the positional baseline that independent draft-target pairs were missing, and localize the residual failure to decision-point entropy rather than position.

Figures & tables

Figure 1: Acceptance rate vs. generation position. (a) Same-family, all problems: the greedy curve rises and saturates after 12K—almost entirely driven by loop problems. (b) Same-family after removing them: no downward trend, and if anything a slight rise (greedy

0.91

–

0.94

, sampling

0.856

–

0.882

). (c) Cross-family (loops removed) shows the same shape at a lower level. Effective support: greedy 0–8K, sampling 0–12K; beyond these ranges few problems remain and the curves become noisy (Sec. 8 ), so we do not draw conclusions there.

Config	Metric	Diff	95% CI	Up/Down	$p$
same/greedy	def1	$+0.031$	$[+0.026,+0.036]$	327/126	$1.2{\times}10^{-21}$
same/sample	def1	$+0.028$	$[+0.023,+0.033]$	354/130	$5.2{\times}10^{-25}$
xfam/greedy	def1	$+0.069$	$[+0.063,+0.076]$	393/61	$1.8{\times}10^{-60}$
xfam/sample	def1	$+0.059$	$[+0.053,+0.065]$	407/78	$9.6{\times}10^{-55}$
same/sample	$\alpha$	$+0.010$	$[+0.008,+0.013]$	333/151	$8.3{\times}10^{-17}$
xfam/sample	$\alpha$	$+0.030$	$[+0.027,+0.034]$	365/120	$8.7{\times}10^{-30}$

Table 1: Within-problem paired half-difference (second

-

first) after loop removal. def1

=\mathbb{E}[\mathrm{accepted}]/\gamma

;

\alpha=\mathbb{E}[\min(1,p/q)]

is the true expected acceptance probability; it is reported only for sampling configurations because greedy decoding does not record position-level

p,q

(Sec. 3 ). 95% CIs are problem-level bootstrap (10,000 resamples); “Up/Down” counts problems whose second-half acceptance exceeds / falls below the first half (zero differences excluded);

p

is the exact two-sided sign test. def2 (argmax-match rate) curves appear in Fig. 1 and are excluded from paired tests because def2 is not an expectation over the sampling law (Sec. 6 ).

Config	Full	Loop removed
same/greedy	$+0.064$	$+0.025$
same/sample	$+0.006$	$+0.009$
xfam/greedy	$+0.172$	$+0.090$
xfam/sample	$+0.060$	$+0.056$

Table 2: Aggregate half-difference before/after loop removal. Unlike Table 1 , rounds are pooled across problems and split at the global median position, so loop problems (which contribute most long-position rounds) dominate the pre-removal column; the paired column of Table 1 weights every problem equally and is the primary statistic.

Figure 2: First-half vs. second-half acceptance (def1), before and after loop removal. Bars use the problem-weighted paired estimator of Table 1 : each problem contributes one half-pair, so the 46 loop problems have bounded influence and the greedy excess shrinks only modestly after removal (

+0.040\to+0.031

; sampling

+0.027\to+0.028

). The dominant loop effect is a round-weighting phenomenon: in the pooled statistics (Table 2 :

+0.064\to+0.025

) loop rounds dominate long positions, and the position curves flatten accordingly (Fig. 1 (a) vs. (b)).

Figure 3: def2 (visible argmax agreement, both modes) and

\alpha

(true expected acceptance probability, sampling only—greedy records no position-level

p,q

) by token type, non-loop. Failure is entropy-selective: number/operator are near-perfect, reflection tokens are the clear failure point, and cross-family misalignment amplifies it.

Figure 4: Mismatch structure at reflection triggers.

Figure 5: Within-problem position quintiles: acceptance is flat-to-rising after loop removal.

$T$	MAL	def2	$\alpha$	reflection def2	reflection $\alpha$
0.3	4.986	0.917	0.927	0.707	0.785
0.6	4.967	0.886	0.922	0.636	0.802
0.9	4.884	0.836	0.915	0.520	0.809

Table 3: Temperature sweep (same-family sampling, first 100 problems, loop removed). def2 falls sharply with

T

, but the true expected acceptance probability

\alpha

barely moves—the apparent degradation of reflection tokens is a visibility artifact.

Figure 6: MAL at 500 problems under the three intervention arms and the untrained base draft (same-family; error bars: standard error). No arm improves over the uniform training control in either decoding mode.

Arm	MAL samp	MAL greedy	$\Delta_{\mathrm{samp}}$ [95% CI]	$\Delta_{\mathrm{greedy}}$ [95% CI]
base (no training)	$4.990{\pm}.012$	$5.060{\pm}.013$	–	–
uniform	$4.961{\pm}.013$	$5.077{\pm}.013$	–	–
reflection-marked	$4.947{\pm}.013$	$5.058{\pm}.013$	$-0.0138$ $[-0.032,+0.005]$	$-0.0192$ $[-0.042,+0.003]$
entropy-weighted	$4.954{\pm}.013$	$5.072{\pm}.013$	$-0.0072$ $[-0.026,+0.012]$	$-0.0055$ $[-0.008,-0.003]$

Table 4: Intervention results at 500 problems (same-family, both decoding modes). MAL is reported with standard errors.

\Delta

columns are problem-paired differences against the uniform arm—the proper control, since it undergoes identical training on the same trajectories with flat weighting—computed from unrounded per-problem MAL, so a

\Delta

need not equal the difference of the two rounded column entries; 95% CIs are from paired

t

-tests. No arm improves over the uniform control; answer accuracy is identical across arms (McNemar

\chi^{2}{\approx}0

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Cross-family counterparts of Figs. 3 and 5 : (left) token-type breakdown and (right) within-problem position quintiles.

Figure 8: Token-type breakdown for the intervention experiment (100-problem diagnostic subset).

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding
Jun 29, 2026Aaryam SharmaSpeculative Decoding LLM Inference Acceleration
Independent Researcher Waterloo, Ontario, Canada
Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding
Oct 20, 2026Tianyu Wu, Zhuohan Wang, Jonathan Geuter +5Speculative Decoding LLM Inference Acceleration
Harvard University · Massachusetts Institute of Technology
Acceptance Dynamics Across Cognitive Domains in Speculative Decoding
Apr 16, 2026Seifeldin AbdellatifSpeculative Decoding Text Generation
College of Engineering, Al Ain University Abu Dhabi, United Arab Emirates

Jun 29, 2026J/K move · Enter open · S save

When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

Aaryam Sharma

Independent Researcher Waterloo, Ontario, Canada

Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding accelerates language model inference by using a fast drafter to propose candidate tokens that are then verified by a larger target model. Existing theory largely studies the stochastic, distribution-preserving setting, where the goal is to exactly sample from the target distribution. In contrast, many practical systems use greedy decoding, relaxed acceptance rules, or tree-based candidate sets, where success is governed by local ranking and threshold events rather than exact distributional equality. We develop a theory for these regimes. We identify that many common acceptance criteria have rejection regions that can be characterized as lower level sets of the target distribution. For these, we characterize the exact KL divergence required for rejection yielding exact certificates and sharp margin-based bounds for strict greedy decoding, additive and multiplicative relaxed acceptance, top-(m) relaxed criteria, and entropy-thresholded acceptance. We then extend the framework to greedy tree decoding, deriving exact and margin-only certificates for when the target greedy token remains covered by the drafter's top-(m) candidates. Finally, we evaluate the resulting certificates on Qwen3 models, showing that relaxed and tree-based criteria substantially enlarge the region of certified acceptance, especially on decoding steps with low target model distribution margin. These results complement existing distribution-preserving analyses of speculative decoding by characterizing the deterministic local acceptance events common in practical inference systems.

Abstract

Speculative Decoding

Explore similar work

When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding

Acceptance Dynamics Across Cognitive Domains in Speculative Decoding