SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding
Authors: Tianyu Liu, Jun Zhang, Wenjie Huang, Yuhao Shen, Tianqi Xu, Xinyue Hao, Hongyaoxing Gu, Xuyang Liu, Baolin Zhang, Yating Zhu, Shuang Ge, Jun Dai, Lei Chen, Yue Li, Mingchen Wan
Organizations: Qwen Business Unit of Alibaba · University of Science and Technology of China · Zhejiang University · Carnegie Mellon University · University of Edinburgh · University of Chinese Academy of Sciences · The Hong Kong Polytechnic University
Speculative decoding (SD) accelerates large language model inference by reducing the sequential cost of token-by-token generation. Advanced SD methods reduce proposal cost with parallel drafters that predict entire blocks in one forward pass, but overall speedup still depends on acceptance length. On-policy distillation (OPD) can push well-trained drafters beyond acceptance-length plateaus, yet conventional pipelines collect supervision through sequential draft-and-verify rollouts. Our key insight is that on-policy supervision need not inherit these dependencies: a draft block can be supervised using only its reference prefix and draft-sampled history. We propose SpecOPD, a branch-parallel OPD framework that parallelizes draft sampling and target supervision across independent branches drawn from the evolving drafter policy at reference anchors. We pack these branches into proposal trees to share target-prefix computation, using tree masks for full attention, branch-consistent indexer groups for sparse attention, and indexed state replay for linear attention to preserve each branch's history. We train the drafter with an E2E-TV surrogate that directly targets acceptance length. Our study spans 8B full-attention (GQA), 27B full-linear (GQA--GDN), and 125B sparse-linear (QSA--GDN) targets, three parallel draft architectures (DFlash, DSpark, and DFlash V2), and four task domains (dialogue, coding, mathematics, and agentic). SpecOPD achieves up to 41.45% higher mean acceptance length, 16.09× supervised-token throughput over sequential-rollout OPD under native configurations, and up to 3.56× generation speedup over autoregressive decoding.
Figures & tables
Figure 1: SpecOPD improves training throughput. Qwen3.8-27B with DFlash V2 on eight high-performance GPUs. (a) Sequential draft-and-verify rollouts versus parallel drafting and target scoring, shown on separate time axes. (b) With tokens denoting predictions contributing to the loss, unprofiled measurements show 4.95× update throughput and 3.25× supervised tokens per update, yielding 16.09× supervised-token throughput.
Figure 2: Proposal tree construction. ① We randomly sample anchors from a reference response and extract target context features. ② The current drafter samples continuations in parallel at these anchors, conditioned on their respective reference contexts. ③ We attach the sampled continuations to their anchors, forming a tree with a shared reference trunk and separate draft branches. Branch independence enables parallel collection, while shared ancestors expose reusable target computation.
Figure 3: Branch-parallel scoring with full attention ( ). The tree attention mask lets packed queries share trunk KV while restricting each row to itself and its ancestors. Rows and columns follow the same packed-token order. Blue arrows connect each anchor to all query rows in its draft branch; Prompt aggregates the shared prompt prefix. Terminal samples walk and music use predecessor readouts and need no query rows.
Figure 4: Branch-parallel scoring with sparse attention ( ). Left-hand tokens are predicted from the predecessor queries named in the right-hand rows. ① All rows reuse complete trunk group keys, including the group ending at Enjoy . Incomplete boundary groups remain in the causal tail. For the branch at evening , predicting with uses the query at evening and reads [a, quiet, evening] as a 3/4 causal tail; predicting soft uses the query at with , which completes [a, quiet, evening, with] (4/4) and requires a branch-specific indexer group key. Anchor and bell icons mark branch points and rebuilding. ② Indexer queries select eligible complete groups with top- k ; attention reads selected KV plus the causal tail, which bypasses top- k .
Figure 5: Branch-parallel scoring with linear attention ( ). A shared trunk pass caches an FP32 GDN state after Enjoy at the C=64 boundary (63 prompt tokens omitted). In the indexed recurrent kernel, each thread block (CTA) loads its state tile from this checkpoint into private registers ( Copy State ). Replay Tokens uses cached trunk features to advance it through a or a → quiet → evening , restoring the anchor state. Update State then processes branch tokens, updating the state and reading it with each token’s query to produce GDN outputs. Checkpoints remain immutable; causal convolution follows branch ancestry.
Setup
Dialogue
Coding
Math
Agentic
Average
Temp.
Target
Draft
MT-Bench
HumanEval
MATH-500
SWE
Terminal
DFlash
2.764
3.262
3.453
2.147
1.620
2.649
8B
↪ SpecOPD
3.432
4.177
4.571
3.617
2.940
3.747
DSpark
3.495
3.879
4.153
3.707
3.135
3.674
8B
↪ SpecOPD
4.121
4.876
5.223
4.301
3.676
4.439
DFlash V2
4.143
4.845
5.555
5.653
5.319
5.103
Table 1: Mean acceptance length ( ↑ ), including bonus tokens. Darker blue indicates longer Mean AL.
Figure 6: Measured Qwen3.8-27B output throughput (tokens/s) with BF16 , TP=1, FA3 and CUDA Graphs.
Figure 7: Training efficiency on Qwen3.8-27B / DFlash V2. Tokens count supervised predictions contributing to the loss.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Qwen3-8B acceptance during training at T=1 and concurrency four. Both drafters propose seven tokens with eight verification positions. Checkpoints are equally spaced; lines connect observed checkpoints. Panels use different y-axis ranges.
Figure 10: 125B acceptance during training at concurrency 16. Rows show T=0 and T=1 . The selected checkpoint is step 22,000; lines connect observations through step 32,000.
Figure 11: Public DFlash V2 acceptance during SpecOPD training with an FP8 Flash-Next target. Rows use inference block sizes four and eight. Diamonds mark each curve’s highest observed AL; percentages use that checkpoint’s matched initialization. Initial points and horizontal references use the baselines paired with step 22,000. Lines guide the eye across GPU/configuration changes rather than depict a fixed-configuration trajectory.
Steps
Training method
MT-Bench
HumanEval
MATH-500
Average
0
Original draft
3.9787
4.6225
5.2519
4.6177
8k
Off-policy E2E-TV
3.6011
4.1086
4.5854
4.0983
8k
SpecOPD
3.9966
4.7602
5.4699
4.7422
10k
Off-policy CE
3.8377
4.4905
5.1448
4.4910
10k
SpecOPD
4.0675
4.7668
5.4726
4.7690
12k
Off-policy CE
3.8042
4.4771
5.1311
4.4708
Appendix
Table 2: On-policy and off-policy training on Qwen3.8-27B / DFlash V2 at matched optimizer steps and reference-sequence batches. AL includes bonus tokens ( ↑ ).
Figure 12: Training gains at both DFlash inference windows. SpecOPD uses the step-10,000 checkpoint; B8/B16 denote inference block size. Columns show T=0 / T=1 ; rows show AL and throughput. Percentages compare training at a fixed window. All weights are trained with block size 16.
T
Task
ΔW(0)
ΔW(10,000)
Δtrain(8)
Δtrain(16)
Interaction
0
MT-Bench
0.0604
0.6896
1.6417
2.2708
0.6292
0
HumanEval
0.3455
0.6524
1.9004
2.2073
0.3069
0
MATH-500
0.0780
0.4560
2.3867
2.7647
0.3780
1
MT-Bench
0.0729
0.5708
1.5312
2.0292
0.4979
1
HumanEval
0.2866
0.6748
1.6992
2.0874
0.3882
1
MATH-500
0.0973
0.3167
2.2967
2.5160
0.2193
Appendix
Table 3: Common-anchor effects on S7 , in accepted draft tokens. ΔW is window 16 minus 8; Δtrain is step 10,000 minus initialization. Interaction is ΔW(10,000)−ΔW(0) .
Input Draft qθ ; frozen target p ; M reference-sequence accumulation slots; branch width W .
Predict draft blocks and candidate supports with qθ .
5
Sample each branch: yb,i∼qθ(⋅∣x≤ab,yb,<i) .
6
Pack proposal trees; score prediction predecessors with the frozen target.
Appendix
Algorithm 1 One SpecOPD optimization step
Determinism disabled
Determinism enabled
Comparison
Max. abs.
Rel. L2 (%)
Mean KL
Max. abs.
Rel. L2 (%)
Mean KL
C1/R
4.1094
14.25
0.05840
0
0
0
CB/R
3.4414
14.82
0.04958
0
0
0
T/R
3.7695
15.01
0.05192
0
0
0
CB/C1
3.0625
11.35
0.02743
0
0
0
T/CB
4.2539
14.69
0.04382
0
0
0
Appendix
Table 4: 125B target-logit differences across execution layouts. A/B compares candidate A against reference B; KL is the prediction-position mean of DKL(pB∥pA) . The largest listed error occurs at the first draft prediction after the three-token prefix in the first sample.
Figure 13: Analytical work and conditional wall-time gains. (a) Uncached independent versus shared-prefix scoring with L=2,048 , W=7 ( P=6 ), CGDN=64 , and uniformly placed anchors. GDN counts include suffix replay; attention counts causal pairs. (b) Equation 25 for component acceleration r and no added overhead ( δ=0 ).
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Speculative decoding accelerates large language model inference by pairing a target model with a lightweight draft model whose proposed tokens are verified in parallel. A common way to build draft models, like EAGLE3 or DFlash is supervised fine-tuning (SFT) on target-generated trajectories. However, we observe that SFT quickly plateaus: the draft model's acceptance length on test data stops improving. The reason is an offline-to-inference mismatch: In SFT, the drafter learns from fixed target-generated trajectories, whereas during speculative decoding it is evaluated on blocks proposed under its own policy. This motivates on-policy distillation (OPD), where the target model supervises the drafter on draft-induced states. Yet OPD remains difficult for draft models, as they cannot reliably roll out complete sequences independently, whereas target-assisted generation makes the collected sequences follow the target distribution and thus eliminates the on-policy signal. We therefore propose Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions. This allows the drafter to learn from target feedback on both accepted and rejected proposals, focusing training on the draft-induced errors that limit speculative acceptance. Experiments show that Draft-OPD achieves over 5× lossless acceleration for thinking models across diverse tasks, improving over EAGLE-3 and DFlash by 23% and 13%.
Haodi Lei, Yafu Li, Haoran Zhang +8
Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3
Speculative decoding accelerates large language model inference with a small drafter, whose per-guess acceptance rate determines the achievable speedup. Feature-level drafters share a design choice: every token, easy or hard, is drafted under a single forward pass. Many of the tokens the target rejects are ones the drafter is itself unsure about yet the target answers decisively. To address this, we introduce Parallel Reconsideration: guided by its own uncertainty, the drafter reconsiders its hesitant guesses with extra passes that run in parallel with the one-pass draft, adding negligible latency. A co-designed training recipe and serving-time drafting algorithm append these reconsidered guesses to the original draft tree as refined candidates. Across models and benchmarks, on average, Parallel Reconsideration improves over the baselines by 5.3% to 10.2% on acceptance length and by 3.5% to 9.2% on speedup.
Speculative decoding accelerates large language model inference by employing a compact draft model to propose candidate tokens that are verified in parallel by a target model. Conventional training relies on soft-label distillation, which forces the draft model to mimic the target's full output distribution, including high-entropy regions that degrade acceptance rates and increase early rejection risk. We propose RCSD (Reinforcement and Continual Learning for Speculative Decoding), a training framework that aligns draft model optimization with the system-level objective of maximizing acceptance length. RCSD leverages naturally occurring accepted and rejected sequences during speculative inference to construct implicit preference pairs, with masked loss to avoid penalizing target-aligned tokens. Evaluated across multiple benchmarks, RCSD improves average acceptance length by 35% over HASS and achieves 1.35x faster end-to-end inference than HASS on practical workloads, without compromising generation quality.