Speculative decoding accelerates language-model generation by verifying multiple draft tokens in a single target-model pass. However, better draft predictions and longer accepted continuations do not necessarily produce faster generation. To examine this gap, we present FACT (Fixed-Budget Allocation and Cross-Step Training), an experimental framework for studying how drafter training, candidate allocation, and execution conditions jointly affect acceptance and latency. Using a lightweight feature-prediction drafter with a frozen Llama-3-8B-Instruct target under greedy decoding, we first examine how training translates into accepted tokens. Backpropagating through predicted-feature rollouts with greater weight on early steps improves acceptance by 15-24% over uniformly weighted detached training, yet average rollout proxy cross-entropy does not reliably rank drafters by acceptance. Beyond prediction quality, the placement of draft candidates also matters: with a fixed 256-node budget and matched per-depth widths, replacing static allocation with confidence-guided allocation increases speedup over autoregressive decoding from 1.46× to 2.02× on HumanEval and from 1.31× to 1.75× on MT-Bench. This allocation benefit extends to an independently trained drafter, but stronger training does not consistently amplify it, showing that the two improvements do not necessarily reinforce each other. Even these speed gains depend on execution conditions: changing only host-thread settings can nearly eliminate speedup without changing acceptance. Together, these findings show why training and allocation must be evaluated jointly using both acceptance and end-to-end latency under controlled execution conditions.
Figures & tables
Figure 1: From draft proposals to generation speed. A lightweight drafter proposes candidate continuations, which are arranged in a tree and checked by the target model. Accepted draft tokens and a target-generated token advance the output. We study drafter training, candidate allocation under fixed depth-wise budgets, and execution conditions, measuring both useful progress and total decoding time.
Figure 2: Training on predicted features. The drafter D predicts the next feature from a recorded feature and token. 1 ○ Teacher forcing trains one step on recorded features only. 2 ○ Detached rollout reuses each prediction as the next input but stops gradients between steps. 3 ○ Full-gradient rollout keeps the reuse and lets later losses reach earlier steps. 4 ○ Step weights control each prediction’s contribution to the loss: uniform ( γ=1 ) or step-decayed ( γ=0.5 ).
Figure 3: Training improvements persist across seeds. Each marker shows mean draft acceptance over 50 test prompts for one training seed. All configurations use a static 256-node tree. Percentages compare step-weighted training with detached rollout after averaging the two seeds. K denotes decoding depth, not training rollout length.
Figure 4: Agreement with the target at each draft depth. Fraction of positions where the drafter’s top-ranked token equals the target’s greedy next token, measured along the target’s continuation of 50 test prompts per benchmark with decode-time feature inputs. Thick lines average training seeds 1 and 2, and thin lines show each seed. The checkpoints are the 10,000-step drafters of Figure 3 .
Workload
Allocation
τ
ms/token
Speedup
MT-Bench
Static
1.92
13.96
1.31×
MT-Bench
Dynamic
2.66
10.32
1.75×
HumanEval
Static
2.27
12.47
1.46×
HumanEval
Dynamic
3.22
8.93
2.02×
Table 1: Main allocation comparison. Scaled drafter, K=4 , 256 candidates, and 50 test prompts per workload. τ excludes the target-generated token; latency includes prefill. Figure 5 reports uncertainty on paired latency savings.
Figure 5: Allocation gains transfer across drafters. Points show paired latency savings, Lstatic−Ldynamic , under identical depth-wise candidate budgets. Positive values favor dynamic allocation. Bars show 95% bootstrap intervals over 50 test prompts after pooling repetitions within each prompt. Both panels use the same horizontal scale.
Figure 6: Per-prompt speedups for the scaled drafter at K=4 . Each point is one of the 50 test prompts per workload, with repeated runs averaged within the prompt; HumanEval and MT-Bench pool ten runs and the other workloads three. Boxes show quartiles and medians. Speedup is the per-prompt ratio to autoregressive decoding from the same run, and the dotted line marks autoregressive speed.
Figure 7: Execution conditions change speed without changing acceptance. Speedup over autoregressive decoding as host-thread count varies. Each point averages paired speedup ratios over 50 development prompts from one run per setting. The dotted line denotes autoregressive speed. Acceptance remains unchanged across caps within each configuration.
Figure 8: Tokens per target call and speedup for each prompt. Each point is one test prompt averaged over five runs per setting, using the scaled drafter. Tokens per target call include the target-generated token. The static tree at 208 threads, not shown, matches the static tree at eight threads. The dotted line marks autoregressive speed.
Figure 9: Time per decoding step by phase. Means over 25 HumanEval development prompts with the scaled drafter at K=4 and phase timers enabled, weighting each prompt by its number of steps. Labels give the total time and the tokens emitted per step. Timers can perturb execution, so these runs are diagnostic and are not used for the speedups reported elsewhere.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Fixed budgets, different allocation. Illustrative widths (3,3,2) and draft probabilities for the target continuation “is Paris,”. (a) Static quotas omit the target’s comma continuation, so two draft tokens are accepted before the mismatch; (b) confidence-guided (dynamic) selection places both depth-3 slots below “Paris” and accepts three. Accepted counts exclude the target-generated correction or bonus token.
Setting
Reported value
Target and timing precision
Llama-3-8B-Instruct, frozen; bfloat16
Execution platform
One NVIDIA H100; default host-thread cap 8
Candidate budget and depth
256 candidates; K∈{4,8}
Timing prompts and output cap
50 test prompts per benchmark; 128 tokens
Timing accounting
Includes prefill; excludes loading; actual emitted-token count
Timing repetition
3–10 repeats per configuration; main K=4 HumanEval and MT-Bench comparisons span 3 sessions
Appendix
Table 2: Reported settings. Training-step matching is not compute matching. The 60-prompt quality probes are separate from the timing experiments.
Workload
K
Baseline G
Scaled G
EAGLE G
I
MT-Bench
4
3.43 [3.22,3.64]
3.65 [3.44,3.85]
3.60 [3.41,3.79]
0.22 [ −0.00 ,0.44]
MT-Bench
8
3.46 [3.19,3.73]
3.37 [3.11,3.63]
3.00 [2.76,3.23]
−0.09 [ −0.40 ,0.20]
Alpaca
4
3.35 [3.17,3.53]
3.00 [2.78,3.22]
3.70 [3.51,3.91]
−0.35 [ −0.58 , −0.12 ]
Alpaca
8
3.72 [3.42,3.99]
3.19 [2.93,3.46]
3.05 [2.72,3.41]
−0.52 [ −0.82 , −0.22 ]
GSM8K
4
3.57 [3.32,3.86]
3.62 [3.42,3.83]
3.75 [3.57,3.94]
0.05 [ −0.30 ,0.37]
GSM8K
8
3.16 [2.96,3.37]
3.90 [3.64,4.17]
3.57 [3.34,3.81]
0.74 [0.41,1.07]
Appendix
Table 3: Dynamic-allocation savings G and interaction I , in ms/token. Entries give estimates and 95% intervals. Negative I means smaller absolute savings for the scaled drafter.
Objective
CE 1
CE 1:8
τ , K=4
τ , K=8
Decayed weights, full gradient
2.43
3.33
1.69
1.74
Teacher forcing
2.29
7.62
1.49
1.50
Uniform, detached
3.04
3.56
1.37
1.37
Uniform, full gradient
3.30
3.77
1.34
1.35
Appendix
Table 4: Exploratory four-arm control, seed 0, 80,000 feature rows and 10,000 optimizer steps. MT-Bench, static 256-node tree, earlier prompt set. CE is proxy cross-entropy. The eight-step CE evaluation is shared across objectives.
Figure 11: Acceptance against training time and training data in the exploratory run (seed 0, static 256-node tree, earlier prompt set). The left panel shows MT-Bench acceptance at K=4 . Step-weighted checkpoints are placed at the recorded rate of 27.0 h per 10,000 steps, and the other objectives at their recorded 10,000-step times, with uniform full-gradient training taken from its training log. The right panel shows step-weighted drafters trained for 10,000 steps on 20,000 to 80,000 feature rows.
Benchmark
Metric
Difference
95% interval
HumanEval
pass@1
+0.033
[−0.033,0.100]
GSM8K
Exact match
0.000
[−0.067,0.067]
Appendix
Table 5: Separate paired quality probes, 60 held-out prompts per benchmark. Differences are speculative minus autoregressive performance, on a 0–1 scale. Intervals quantify uncertainty in the difference and do not establish equivalence.