Speculative decoding has been treated purely as a systems optimization that speeds up large language model decoding without changing the model behavior. Under this view, the growing effort to train better draft models has been focused solely on acceptance rate. We argue that this view does not hold true for long-horizon agent reasoning. Measuring frontier open-weight models on complex agentic tasks with different speculative decoding approaches, we found that numerical noise grows within verification windows, and the draft acceptance results of the thinking process can change the output distribution of the model after 30 steps, thus noticeably influencing the reasoning accuracy of long-horizon agents in random directions. In this work, we improve the reasoning capability of target models by controlling such noise. While we cannot change the overall token distribution verified by the target model, we can encode critical reasoning with highest precision by training the draft model to be rejected. We identify these points in agent traces and train the draft model to favor adversarial preference. At inference the trained draft and the target part ways precisely at these decision points, verification restarts, so that critical thinking tokens are sampled from the target model and encoded under its least noisy conditions. This approach makes critical reasoning better indexed and more durable over a long horizon. On complex agentic coding benchmarks, our approach raises end-to-end success rate with no change to the target model or its output distribution. Our results recast the speculative decoding mechanism from a pure accelerator into an active allocator for context precision that improves the accuracy of long-horizon agents by training 0.06% to 0.28% of hosted parameters.
Figures & tables
Figure 1: Single-pass numerical noise ∣Δlogp∣ by verify slot between repeated forwards on an identical prefix. (a) Mean deviation, with the per-slot slope. (b, c) Fraction of tokens shifted by more than 1 and 2 nats (log scale). (d) Mean deviation relative to slot 0, with least-squares fits: both models grow by about 4% of their slot-0 level per slot. Slopes with 95% bootstrap CIs over positions: +0.0195[+0.0174,+0.0214] (GLM-5.3) and +0.0094[+0.0078,+0.0110] (DeepSeek-V4.1-Flash).
Figure 2: Long-range deviations of verify-written vs. decode-written context on GLM-5.3: ratio of assistant tokens whose emitted-token log-probability changes by more than each threshold the verify path and decode path, in 20-turn bins after the source turn. Error bars are 95% bootstrap CIs over trajectories. For deviations beyond 4 nats the ratio rises from 0.98 (turns 11–30) to 1.09 (31–50) and 1.16 (51–70, shaded; [0.98,1.36] ), then returns to 0.99 (71–90) since fixed teacher-forcing tokens correct the biases.
Figure 3: The two production drafts, with the parameters trained in the deployed adversarial drafts in red and frozen parameters in blue (inputs not drawn). (a) GLM-5.3 DFlash predicts a block of seven tokens from the projected target features; we train its six transformer layers and the final norm (2.15B parameters), and freeze the feature projection, the candidate selector and the target’s embedding and LM head. (b) DeepSeek-V4.1-Flash DSPARK predicts a block of five tokens through three MoE stages; we train the attention sublayers (FP8 projections through quantization-aware training), the norms, the hyper-connection mixing and the bigram head (449M parameters), and freeze the experts, the router, the input projection and the target’s embedding and LM head.
Figure 4: Held-out preference accuracy of trained drafts on tasks unseen in training, best configuration per preference and direction; error bars are 95% binomial confidence intervals, the dashed line is chance. Top: the two preferences we deploy. Bottom: two reference preferences we do not use. For thinking length the preferred side is the shorter thinking; for right vs. wrong it is the right decision, and the bars shown are the best of 36 (forward) and 12 (adversarial) configurations whose medians are 0.52 and 0.46.
Figure 5: (a, b) Pass rates (%) of each target model served with its unbiased draft (base) and with the adversarial draft (adv.), five independent end-to-end runs each. Dots are runs, the thick line is their mean (printed below), the error bar is the 95% confidence interval of the mean. Both panels span the same 25 points on the vertical axis, so spreads are comparable across benchmarks. The draft changes nothing but which tokens are proposed; the output distribution is identical under exact verification. (c, d) Top-1 agreement of the draft with the emitted token at each slot of the draft block, unbiased draft model against the deployed adversarial draft, teacher-forced on the target’s hidden states over held-out continuations (GLM-5.3-nvfp4: 30 continuations of 512 tokens; DeepSeek-V4.1-Flash: 240 continuations of 512 tokens, 160 at run-tests turns and 80 at decision points). The red numbers denote the change in points.
Figure 6: Direction of the trained preference against DeepSWE pass rate. The unbiased and adversarial points are the means of Figure 5 (a); the desirable draft models are trained on identical datasets with only the preference reversed.
Figure 7: DeepSWE pass rate of DeepSeek-V4.1-Flash served with drafts trained on different preferences. The band denotes the 95% confidence interval of the unbiased draft’s mean. Only the adversarial run-tests draft model shows meaningful improvement.
Speculative decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose outputs that a stronger target model verifies. However, its token-centric nature allows erroneous steps to propagate. Prior approaches mitigate this using external reward models, but incur additional latency, computational overhead, and limit generalizability. We propose SpecGuard, a verification-aware speculative decoding framework that performs step-level verification using only model-internal signals. At each step, SpecGuard samples multiple draft candidates and selects the most consistent step, which is then validated using an ensemble of two lightweight model-internal signals: (i) an attention-based grounding score that measures attribution to the input and previously accepted steps, and (ii) a log-probability-based score that captures token-level confidence. These signals jointly determine whether a step is accepted or recomputed using the target, allocating compute selectively. Experiments across a range of reasoning benchmarks show that SpecGuard improves accuracy by 3.6% while reducing latency by ~11%, outperforming both SD and reward-guided SD.
Kiran Purohit, Ramasuri Narayanam, Soumyabrata Pal
Speculative decoding accelerates large language model (LLM) inference. It uses a small draft model to propose a tree of future tokens. A larger target model then verifies these tokens in a single batched forward pass. Despite the growing body of work on speculative methods, the degree to which the cognitive characteristics of a task affect acceptance probability remains largely unexplored. We present an empirical study of tree-based speculative decoding acceptance dynamics. Our study spans four well-established NLP benchmark domains: code generation, mathematical reasoning, logical reasoning, and open-ended chat. For this, we use TinyLlama-1.1B as the draft model against Llama-2-7B-Chat-GPTQ as the target. Over 99,768 speculative nodes collected from 200 prompts, we derive per-domain acceptance rates, expected accepted lengths, depth-acceptance profiles, and entropy-acceptance correlations. We find that task type is a stronger predictor of acceptance than tree depth. Furthermore, only the chat domain consistently yields an expected accepted length exceeding 1.0 token per step. We also show that the entropy-acceptance correlation is consistently negative but weak across all domains (rho in [-0.20, -0.15]). Counterintuitively, chat produces the highest entropy yet the highest acceptance rate. We attribute this divergence to the lexical predictability of RLHF-aligned register. These findings have direct implications for domain-aware speculation budgets and draft-model selection strategies. Index Terms--speculative decoding, large language model inference, tree attention, draft model, acceptance probability, LLM efficiency
Seifeldin Abdellatif
College of Engineering, Al Ain University Abu Dhabi, United Arab Emirates
Speculative decoding (SPD) accelerates large language model (LLM) inference by letting a smaller draft model propose multiple future tokens that are verified in parallel by a larger target model. The dominant SPD paradigm treats the target model as the sole reliable teacher, accepting a draft token only when it exactly matches the target prediction. This design implicitly assumes that the target is always the better choice at every position. In practice, this assumption does not hold. Although the draft is the weaker model overall, it is not uniformly inferior at the token level. In a meaningful fraction of cases where draft and target disagree, the draft's choice is the one that leads to the correct final answer. Inspired by this, we introduce Collaborative Speculative Decoding (CoSpec), a generalization of SPD that no longer treats the target model as the sole token-level authority. CoSpec trains an arbitration policy via reinforcement learning to decide whether to accept tokens from the draft or target model, selectively accepting draft tokens at mismatches when doing so is likely to yield a correct final answer. Experimental results show that CoSpec maintains substantial speedups while surpassing target-only performance. By shifting the emphasis from imitation to collaboration, CoSpec suggests a new perspective on speculative decoding.
Jinze Li, Yixing Xu, Guanchen Li +7
Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas