Speculative decoding accelerates language-model generation by verifying multiple draft tokens in a single target-model pass. However, better draft predictions and longer accepted continuations do not necessarily produce faster generation. To examine this gap, we present FACT (Fixed-Budget Allocation and Cross-Step Training), an experimental framework for studying how drafter training, candidate allocation, and execution conditions jointly affect acceptance and latency. Using a lightweight feature-prediction drafter with a frozen Llama-3-8B-Instruct target under greedy decoding, we first examine how training translates into accepted tokens. Backpropagating through predicted-feature rollouts with greater weight on early steps improves acceptance by 15-24% over uniformly weighted detached training, yet average rollout proxy cross-entropy does not reliably rank drafters by acceptance. Beyond prediction quality, the placement of draft candidates also matters: with a fixed 256-node budget and matched per-depth widths, replacing static allocation with confidence-guided allocation increases speedup over autoregressive decoding from 1.46× to 2.02× on HumanEval and from 1.31× to 1.75× on MT-Bench. This allocation benefit extends to an independently trained drafter, but stronger training does not consistently amplify it, showing that the two improvements do not necessarily reinforce each other. Even these speed gains depend on execution conditions: changing only host-thread settings can nearly eliminate speedup without changing acceptance. Together, these findings show why training and allocation must be evaluated jointly using both acceptance and end-to-end latency under controlled execution conditions.
Figures & tables
Figure 1: From draft proposals to generation speed. A lightweight drafter proposes candidate continuations, which are arranged in a tree and checked by the target model. Accepted draft tokens and a target-generated token advance the output. We study drafter training, candidate allocation under fixed depth-wise budgets, and execution conditions, measuring both useful progress and total decoding time.
Figure 2: Training on predicted features. The drafter D predicts the next feature from a recorded feature and token. 1 ○ Teacher forcing trains one step on recorded features only. 2 ○ Detached rollout reuses each prediction as the next input but stops gradients between steps. 3 ○ Full-gradient rollout keeps the reuse and lets later losses reach earlier steps. 4 ○ Step weights control each prediction’s contribution to the loss: uniform ( γ=1 ) or step-decayed ( γ=0.5 ).
Figure 3: Training improvements persist across seeds. Each marker shows mean draft acceptance over 50 test prompts for one training seed. All configurations use a static 256-node tree. Percentages compare step-weighted training with detached rollout after averaging the two seeds. K denotes decoding depth, not training rollout length.
Figure 4: Agreement with the target at each draft depth. Fraction of positions where the drafter’s top-ranked token equals the target’s greedy next token, measured along the target’s continuation of 50 test prompts per benchmark with decode-time feature inputs. Thick lines average training seeds 1 and 2, and thin lines show each seed. The checkpoints are the 10,000-step drafters of Figure 3 .
Workload
Allocation
τ
ms/token
Speedup
MT-Bench
Static
1.92
13.96
1.31×
MT-Bench
Dynamic
2.66
10.32
1.75×
HumanEval
Static
2.27
12.47
1.46×
HumanEval
Dynamic
3.22
8.93
2.02×
Table 1: Main allocation comparison. Scaled drafter, K=4 , 256 candidates, and 50 test prompts per workload. τ excludes the target-generated token; latency includes prefill. Figure 5 reports uncertainty on paired latency savings.
Figure 5: Allocation gains transfer across drafters. Points show paired latency savings, Lstatic−Ldynamic , under identical depth-wise candidate budgets. Positive values favor dynamic allocation. Bars show 95% bootstrap intervals over 50 test prompts after pooling repetitions within each prompt. Both panels use the same horizontal scale.
Figure 6: Per-prompt speedups for the scaled drafter at K=4 . Each point is one of the 50 test prompts per workload, with repeated runs averaged within the prompt; HumanEval and MT-Bench pool ten runs and the other workloads three. Boxes show quartiles and medians. Speedup is the per-prompt ratio to autoregressive decoding from the same run, and the dotted line marks autoregressive speed.
Figure 7: Execution conditions change speed without changing acceptance. Speedup over autoregressive decoding as host-thread count varies. Each point averages paired speedup ratios over 50 development prompts from one run per setting. The dotted line denotes autoregressive speed. Acceptance remains unchanged across caps within each configuration.
Figure 8: Tokens per target call and speedup for each prompt. Each point is one test prompt averaged over five runs per setting, using the scaled drafter. Tokens per target call include the target-generated token. The static tree at 208 threads, not shown, matches the static tree at eight threads. The dotted line marks autoregressive speed.
Figure 9: Time per decoding step by phase. Means over 25 HumanEval development prompts with the scaled drafter at K=4 and phase timers enabled, weighting each prompt by its number of steps. Labels give the total time and the tokens emitted per step. Timers can perturb execution, so these runs are diagnostic and are not used for the speedups reported elsewhere.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Fixed budgets, different allocation. Illustrative widths (3,3,2) and draft probabilities for the target continuation “is Paris,”. (a) Static quotas omit the target’s comma continuation, so two draft tokens are accepted before the mismatch; (b) confidence-guided (dynamic) selection places both depth-3 slots below “Paris” and accepts three. Accepted counts exclude the target-generated correction or bonus token.
Setting
Reported value
Target and timing precision
Llama-3-8B-Instruct, frozen; bfloat16
Execution platform
One NVIDIA H100; default host-thread cap 8
Candidate budget and depth
256 candidates; K∈{4,8}
Timing prompts and output cap
50 test prompts per benchmark; 128 tokens
Timing accounting
Includes prefill; excludes loading; actual emitted-token count
Timing repetition
3–10 repeats per configuration; main K=4 HumanEval and MT-Bench comparisons span 3 sessions
Appendix
Table 2: Reported settings. Training-step matching is not compute matching. The 60-prompt quality probes are separate from the timing experiments.
Workload
K
Baseline G
Scaled G
EAGLE G
I
MT-Bench
4
3.43 [3.22,3.64]
3.65 [3.44,3.85]
3.60 [3.41,3.79]
0.22 [ −0.00 ,0.44]
MT-Bench
8
3.46 [3.19,3.73]
3.37 [3.11,3.63]
3.00 [2.76,3.23]
−0.09 [ −0.40 ,0.20]
Alpaca
4
3.35 [3.17,3.53]
3.00 [2.78,3.22]
3.70 [3.51,3.91]
−0.35 [ −0.58 , −0.12 ]
Alpaca
8
3.72 [3.42,3.99]
3.19 [2.93,3.46]
3.05 [2.72,3.41]
−0.52 [ −0.82 , −0.22 ]
GSM8K
4
3.57 [3.32,3.86]
3.62 [3.42,3.83]
3.75 [3.57,3.94]
0.05 [ −0.30 ,0.37]
GSM8K
8
3.16 [2.96,3.37]
3.90 [3.64,4.17]
3.57 [3.34,3.81]
0.74 [0.41,1.07]
Appendix
Table 3: Dynamic-allocation savings G and interaction I , in ms/token. Entries give estimates and 95% intervals. Negative I means smaller absolute savings for the scaled drafter.
Objective
CE 1
CE 1:8
τ , K=4
τ , K=8
Decayed weights, full gradient
2.43
3.33
1.69
1.74
Teacher forcing
2.29
7.62
1.49
1.50
Uniform, detached
3.04
3.56
1.37
1.37
Uniform, full gradient
3.30
3.77
1.34
1.35
Appendix
Table 4: Exploratory four-arm control, seed 0, 80,000 feature rows and 10,000 optimizer steps. MT-Bench, static 256-node tree, earlier prompt set. CE is proxy cross-entropy. The eight-step CE evaluation is shared across objectives.
Figure 11: Acceptance against training time and training data in the exploratory run (seed 0, static 256-node tree, earlier prompt set). The left panel shows MT-Bench acceptance at K=4 . Step-weighted checkpoints are placed at the recorded rate of 27.0 h per 10,000 steps, and the other objectives at their recorded 10,000-step times, with uniform full-gradient training taken from its training log. The right panel shows step-weighted drafters trained for 10,000 steps on 20,000 to 80,000 feature rows.
Benchmark
Metric
Difference
95% interval
HumanEval
pass@1
+0.033
[−0.033,0.100]
GSM8K
Exact match
0.000
[−0.067,0.067]
Appendix
Table 5: Separate paired quality probes, 60 held-out prompts per benchmark. Differences are speculative minus autoregressive performance, on a 0–1 scale. Intervals quantify uncertainty in the difference and do not establish equivalence.
Speculative decoding accelerates large language model inference with a small drafter, whose per-guess acceptance rate determines the achievable speedup. Feature-level drafters share a design choice: every token, easy or hard, is drafted under a single forward pass. Many of the tokens the target rejects are ones the drafter is itself unsure about yet the target answers decisively. To address this, we introduce Parallel Reconsideration: guided by its own uncertainty, the drafter reconsiders its hesitant guesses with extra passes that run in parallel with the one-pass draft, adding negligible latency. A co-designed training recipe and serving-time drafting algorithm append these reconsidered guesses to the original draft tree as refined candidates. Across models and benchmarks, on average, Parallel Reconsideration improves over the baselines by 5.3% to 10.2% on acceptance length and by 3.5% to 9.2% on speedup.
Speculative decoding accelerates large language model inference by using a low-cost draft model to propose tokens that the full-size target model verifies in parallel. Parallel and semi-autoregressive (semi- AR) drafters improve drafting efficiency by proposing an entire block in a single forward pass, but training them raises a new difficulty: the draft distribution for a given position depends on where the decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted. Existing training objectives typically rely on block-local surrogates that ignore this cross-round coupling, and therefore do not directly optimize the global decoding efficiency. In this work, we develop a theoretical framework for training and evaluating these drafters by representing speculative decoding as a Markov reward process. This formulation yields the Expected Decoding Rounds (EDR) objective, which weights local rejection costs by state occupancies and exactly equals the expected number of decoding rounds. Unlike prior surrogate objectives, EDR introduces no auxiliary hyperparameters. We then derive an exact temporal-difference gradient that supports unbiased stochastic optimization from target-model rollouts. The same framework also yields an exact offline evaluator for round counts, enabling paired drafter comparisons on shared target rollouts without running speculative decoding. Finetuning two state-of-the- art drafters, DSpark and DFly, with EDR consistently improves mean accepted length and outperforms existing training objectives across nine benchmarks spanning math reasoning, code generation, and chat.
Yunxiao Zhao, Changxiao Cai
Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, USA.
Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to generate multiple candidate tokens that are verified by the target model in a single forward pass. Its speedup is largely determined by the acceptance length, yet existing draft-model training methods mainly optimize cross-entropy or Kullback-Leibler (KL) divergence as proxies. These objectives encourage distribution matching but do not directly optimize acceptance length, and the acceptance mechanism also differs between greedy and sampling-based decoding. In this work, we propose acceptance-length-aware training losses that directly optimize the expected number of accepted tokens within a speculative window. For greedy verification, we derive an expected accepted length (EAL) loss that explicitly maximizes expected acceptance length. For sampling-based decoding, we introduce a window total variation (WTV) loss that optimizes the overlap between temperature-scaled draft and target distributions while accounting for sequential acceptance dependencies. Both objectives can be further combined with a group-relative reinforcement learning stage (GRPO) using simulated acceptance length as the reward. Experiments across different target and draft models, tasks, and decoding settings show that our losses consistently improve acceptance length over KL-based training. WTV provides particularly strong gains under sampling-based decoding, while EAL better matches greedy verification. These results show that directly optimizing the acceptance objective, with losses tailored to the decoding mode, is more effective than conventional distribution-matching objectives.
Tianhua Xia, Mugilan Ganesan, Yifei Feng +3
New York University · University of Waterloo · Independent Researcher