SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding

Tianyu Liu; Jun Zhang; Wenjie Huang; Yuhao Shen; Tianqi Xu; Xinyue Hao; Hongyaoxing Gu; Xuyang Liu

cs.AINov 2, 2026

SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding

Authors: Tianyu Liu, Jun Zhang, Wenjie Huang, Yuhao Shen, Tianqi Xu, Xinyue Hao, Hongyaoxing Gu, Xuyang Liu,

Organizations: Qwen Business Unit of Alibaba · University of Science and Technology of China · Zhejiang University · Carnegie Mellon University ·

Abstract

Speculative decoding (SD) accelerates large language model inference by reducing the sequential cost of token-by-token generation. Advanced SD methods reduce proposal cost with parallel drafters that predict entire blocks in one forward pass, but overall speedup still depends on acceptance length. On-policy distillation (OPD) can push well-trained drafters beyond acceptance-length plateaus, yet conventional pipelines collect supervision through sequential draft-and-verify rollouts. Our key insight is that on-policy supervision need not inherit these dependencies: a draft block can be supervised using only its reference prefix and draft-sampled history. We propose SpecOPD, a branch-parallel OPD framework that parallelizes draft sampling and target supervision across independent branches drawn from the evolving drafter policy at reference anchors. We pack these branches into proposal trees to share target-prefix computation, using tree masks for full attention, branch-consistent indexer groups for sparse attention, and indexed state replay for linear attention to preserve each branch's history. We train the drafter with an E2E-TV surrogate that directly targets acceptance length. Our study spans 8B full-attention (GQA), 27B full-linear (GQA--GDN), and 125B sparse-linear (QSA--GDN) targets, three parallel draft architectures (DFlash, DSpark, and DFlash V2), and four task domains (dialogue, coding, mathematics, and agentic). SpecOPD achieves up to 41.45% higher mean acceptance length, $16.09\times$ supervised-token throughput over sequential-rollout OPD under native configurations, and up to $3.56\times$ generation speedup over autoregressive decoding.

Figures & tables

Figure 1: SpecOPD improves training throughput. Qwen3.8-27B with DFlash V2 on eight high-performance GPUs. (a) Sequential draft-and-verify rollouts versus parallel drafting and target scoring, shown on separate time axes. (b) With tokens denoting predictions contributing to the loss, unprofiled measurements show

4.95\times

update throughput and

3.25\times

supervised tokens per update, yielding

16.09\times

supervised-token throughput.

Figure 2: Proposal tree construction. ① We randomly sample anchors from a reference response and extract target context features. ② The current drafter samples continuations in parallel at these anchors, conditioned on their respective reference contexts. ③ We attach the sampled continuations to their anchors, forming a tree with a shared reference trunk and separate draft branches. Branch independence enables parallel collection, while shared ancestors expose reusable target computation.

Figure 3: Branch-parallel scoring with full attention ( ). The tree attention mask lets packed queries share trunk KV while restricting each row to itself and its ancestors. Rows and columns follow the same packed-token order. Blue arrows connect each anchor to all query rows in its draft branch; Prompt aggregates the shared prompt prefix. Terminal samples walk and music use predecessor readouts and need no query rows.

Figure 4: Branch-parallel scoring with sparse attention ( ). Left-hand tokens are predicted from the predecessor queries named in the right-hand rows. ① All rows reuse complete trunk group keys, including the group ending at Enjoy . Incomplete boundary groups remain in the causal tail. For the branch at evening , predicting with uses the query at evening and reads [a, quiet, evening] as a 3/4 causal tail; predicting soft uses the query at with , which completes [a, quiet, evening, with] (4/4) and requires a branch-specific indexer group key. Anchor and bell icons mark branch points and rebuilding. ② Indexer queries select eligible complete groups with top- $k$ ; attention reads selected KV plus the causal tail, which bypasses top- $k$ .

Figure 5: Branch-parallel scoring with linear attention ( ). A shared trunk pass caches an FP32 GDN state after Enjoy at the $C=64$ boundary (63 prompt tokens omitted). In the indexed recurrent kernel, each thread block (CTA) loads its state tile from this checkpoint into private registers ( Copy State ). Replay Tokens uses cached trunk features to advance it through a or a $\to$ quiet $\to$ evening , restoring the anchor state. Update State then processes branch tokens, updating the state and reading it with each token’s query to produce GDN outputs. Checkpoints remain immutable; causal convolution follows branch ancestry.

Setup			Dialogue	Coding	Math	Agentic		Average
Temp.	Target	Draft	MT-Bench	HumanEval	MATH-500	SWE	Terminal	Average
		DFlash	2.764	3.262	3.453	2.147	1.620	2.649
	8B	$\hookrightarrow$ SpecOPD	3.432	4.177	4.571	3.617	2.940	3.747
		DSpark	3.495	3.879	4.153	3.707	3.135	3.674
	8B	$\hookrightarrow$ SpecOPD	4.121	4.876	5.223	4.301	3.676	4.439
		DFlash V2	4.143	4.845	5.555	5.653	5.319	5.103

Table 1: Mean acceptance length (

\uparrow

), including bonus tokens. Darker blue indicates longer Mean AL.

Figure 6: Measured Qwen3.8-27B output throughput (tokens/s) with BF16 , TP=1, FA3 and CUDA Graphs.

Figure 7: Training efficiency on Qwen3.8-27B / DFlash V2. Tokens count supervised predictions contributing to the loss.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Qwen3-8B acceptance during training at

T=1

and concurrency four. Both drafters propose seven tokens with eight verification positions. Checkpoints are equally spaced; lines connect observed checkpoints. Panels use different y-axis ranges.

Figure 10: 125B acceptance during training at concurrency 16. Rows show

T=0

and

T=1

. The selected checkpoint is step 22,000; lines connect observations through step 32,000.

Figure 11: Public DFlash V2 acceptance during SpecOPD training with an FP8 Flash-Next target. Rows use inference block sizes four and eight. Diamonds mark each curve’s highest observed AL; percentages use that checkpoint’s matched initialization. Initial points and horizontal references use the baselines paired with step 22,000. Lines guide the eye across GPU/configuration changes rather than depict a fixed-configuration trajectory.

Steps	Training method	MT-Bench	HumanEval	MATH-500	Average
0	Original draft	3.9787	4.6225	5.2519	4.6177
8k	Off-policy E2E-TV	3.6011	4.1086	4.5854	4.0983
8k	SpecOPD	3.9966	4.7602	5.4699	4.7422
10k	Off-policy CE	3.8377	4.4905	5.1448	4.4910
10k	SpecOPD	4.0675	4.7668	5.4726	4.7690
12k	Off-policy CE	3.8042	4.4771	5.1311	4.4708

Appendix

Table 2: On-policy and off-policy training on Qwen3.8-27B / DFlash V2 at matched optimizer steps and reference-sequence batches. AL includes bonus tokens (

\uparrow

Figure 12: Training gains at both DFlash inference windows. SpecOPD uses the step-10,000 checkpoint; B8/B16 denote inference block size. Columns show

T=0

T=1

; rows show AL and throughput. Percentages compare training at a fixed window. All weights are trained with block size 16.

$T$	Task	$\Delta_{W}(0)$	$\Delta_{W}(10{,}000)$	$\Delta_{\rm train}(8)$	$\Delta_{\rm train}(16)$	Interaction
0	MT-Bench	0.0604	0.6896	1.6417	2.2708	0.6292
0	HumanEval	0.3455	0.6524	1.9004	2.2073	0.3069
0	MATH-500	0.0780	0.4560	2.3867	2.7647	0.3780
1	MT-Bench	0.0729	0.5708	1.5312	2.0292	0.4979
1	HumanEval	0.2866	0.6748	1.6992	2.0874	0.3882
1	MATH-500	0.0973	0.3167	2.2967	2.5160	0.2193

Appendix

Table 3: Common-anchor effects on

S_{7}

, in accepted draft tokens.

\Delta_{W}

is window 16 minus 8;

\Delta_{\rm train}

is step 10,000 minus initialization. Interaction is

\Delta_{W}(10{,}000)-\Delta_{W}(0)

Input Draft $q_{\theta}$ ; frozen target $p$ ; $M$ reference-sequence accumulation slots; branch width $W$ .
1	Zero draft gradients.
2	for accumulation slot $m=1,\ldots,M$ do
3	Extract frozen target features; sample response anchors $\{a_{b}\}$ .
4	Predict draft blocks and candidate supports with $q_{\theta}$ .
5	Sample each branch: $y_{b,i}\sim q_{\theta}(\cdot\mid x_{\leq a_{b}},y_{b,<i})$ .
6	Pack proposal trees; score prediction predecessors with the frozen target.

Appendix

Algorithm 1 One SpecOPD optimization step

	Determinism disabled			Determinism enabled
Comparison	Max. abs.	Rel. L2 (%)	Mean KL	Max. abs.	Rel. L2 (%)	Mean KL
C1/R	4.1094	14.25	0.05840	0	0	0
CB/R	3.4414	14.82	0.04958	0	0	0
T/R	3.7695	15.01	0.05192	0	0	0
CB/C1	3.0625	11.35	0.02743	0	0	0
T/CB	4.2539	14.69	0.04382	0	0	0

Appendix

Table 4: 125B target-logit differences across execution layouts. A/B compares candidate A against reference B; KL is the prediction-position mean of

D_{\rm KL}(p_{B}\|p_{A})

. The largest listed error occurs at the first draft prediction after the three-token prefix in the first sample.

Figure 13: Analytical work and conditional wall-time gains. (a) Uncached independent versus shared-prefix scoring with

L=2{,}048

W=7

(

P=6

C_{\rm GDN}=64

, and uniformly placed anchors. GDN counts include suffix replay; attention counts causal pairs. (b) Equation 25 for component acceleration

r

and no added overhead (

\delta=0

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,171 papers

Draft-OPD: On-Policy Distillation for Speculative Draft Models
May 28, 2026Haodi Lei, Yafu Li, Haoran Zhang +8Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3
Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding
Nov 3, 2026Hoang Anh Duy Le, Huibing Dong, Alireza Khadem +6Speculative Decoding LLM Inference Acceleration
Google · Rice University · Google Research +1
RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding
Oct 21, 2026Mingkang Liu, Weiming Meng, Peipei Jiao +4Speculative Decoding LLM Inference Acceleration

May 28, 2026J/K move · Enter open · S save

Draft-OPD: On-Policy Distillation for Speculative Draft Models

Haodi Lei, Yafu Li, Haoran Zhang, Shunkai Zhang+7

Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong+3

Speculative Decoding LLM Inference Acceleration Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

Speculative decoding accelerates large language model inference by pairing a target model with a lightweight draft model whose proposed tokens are verified in parallel. A common way to build draft models, like EAGLE3 or DFlash is supervised fine-tuning (SFT) on target-generated trajectories. However, we observe that SFT quickly plateaus: the draft model's acceptance length on test data stops improving. The reason is an offline-to-inference mismatch: In SFT, the drafter learns from fixed target-generated trajectories, whereas during speculative decoding it is evaluated on blocks proposed under its own policy. This motivates on-policy distillation (OPD), where the target model supervises the drafter on draft-induced states. Yet OPD remains difficult for draft models, as they cannot reliably roll out complete sequences independently, whereas target-assisted generation makes the collected sequences follow the target distribution and thus eliminates the on-policy signal. We therefore propose Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions. This allows the drafter to learn from target feedback on both accepted and rejected proposals, focusing training on the draft-induced errors that limit speculative acceptance. Experiments show that Draft-OPD achieves over

5\times

lossless acceleration for thinking models across diverse tasks, improving over EAGLE-3 and DFlash by 23% and 13%.

Abstract

LLM Inference Acceleration

Explore similar work

Draft-OPD: On-Policy Distillation for Speculative Draft Models

Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding

RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding