On-policy distillation (OPD) provides dense token-level supervision on student rollouts, but this supervision can become unreliable when the student and teacher diverge, either because of a large initial policy gap or compounding errors in long-horizon reasoning and tool use. We introduce Divergence-Triggered Dynamic Distillation (DTDD), an adaptive OPD framework that uses segment-level student-teacher divergence to trigger teacher takeover during training. When divergence triggers an intervention during reasoning, the student prefix continues to receive standard OPD supervision. The teacher then generates a recovery suffix, which the student learns through behavior cloning, using cross-entropy on the teacher-generated tokens. We evaluate DTDD in three regimes: long-horizon reasoning, distillation from a base initialization without offline supervised fine-tuning (SFT), and multi-turn agents. We also compare alternative trigger designs and teacher-intervention rates. DTDD reduces the post-warmup peak gradient norm by 8.6× and, from a base initialization, reaches 21.3% held-out math accuracy in 2.6× fewer optimizer updates than OPD; both methods finish near 22%. It improves best-observed performance by 4.8 and 4.1 percentage points on ALFWorld with LoRA and agentic code, respectively. Ablations show that segment-averaged triggers outperform token-spike triggers. Behavior cloning is also more effective than the tested OPD surrogate for training teacher-generated recoveries. Across 23 experimental arms, takeover tends to help when the teacher substantially outperforms the OPD-trained student, provided that the teacher can recover from student-visited states, the student can learn those recoveries, and teacher intervention is neither excessive nor insufficient.
Figures & tables
Figure 1: Representative failures of local OPD supervision: (A) slow learning from a base initialization, (B) late instability under long generation budgets, and (C) compounding errors after an incorrect action in multi-turn interaction. Appendix A reports measured examples of (A) and (B).
Figure 2: Reasoning realization of DTDD . (A) The student generates a segmented rollout. (B) The first threshold crossing selects the rewind boundary. (C) The teacher regenerates the suffix from the retained student prefix. (D) The prefix is trained with OPD and the suffix with behavior cloning; without a crossing, DTDD reduces to OPD.
Table 3
Figure 3: DTDD suppresses late OPD spikes on DAPO-Math-17K at 15K/18K response budgets.
Figure 4: Held-out accuracy for Qwen3-1.7B-Base ← Qwen3-4B-Base-GRPO on DAPO-Math-17K. The length-gated control starts near 25% takeover and adapts thereafter.
Figure 5: Fixed-threshold dynamics ( τ=0.87 ) for Qwen3-1.7B-Base ← Qwen3-4B on OpenThoughts3.
Math reasoning: student Qwen3-1.7B-Base; teacher Qwen3-4B (non-thinking); data OpenThoughts3
Method
AIME24
AIME25
AMC23
Average ↑
Teacher
21.04
20.62
67.66
36.44
OPD
5.62
5.42
27.50
12.85
DTDD fixed τ=0.87
6.88 +1.26
7.08 +1.66
31.41 +3.91
15.12 +2.27
DTDD rewind, target 90%→5%
7.29 +1.67
4.79 −0.63
32.19 +4.69
14.76 +1.91
DTDD restart, target 90%→5%
8.75 +3.13
5.21 −0.21
33.28 +5.78
15.75 +2.90
Table 3: Best-observed results (%; selection details in Appendix B.3 ). Subscripts show percentage-point changes from OPD; bold marks the best trained method.
Figure 6: DTDD minus OPD, averaged over all matched checkpoints, against the headroom plain OPD leaves below its teacher. Points left of zero are settings where OPD ends above the teacher’s own score. Ringed markers identify arms analyzed by the validity, learnability, and dose controls in Appendix F .
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Two representative limitations of vanilla OPD. Left: Qwen3-1.7B-Base and Qwen3-1.7B-SFT are distilled from the same Qwen3-4B-Base-GRPO teacher on DAPO-Math-17K with a 7,168-token generation budget. The teacher, data, optimizer, and generation budget are held fixed; the SFT initialization has a large early advantage, while the base initialization eventually reaches a comparable plateau. Held-out accuracy is the mean avg@16 score on AIME 2024, AIME 2025, and AMC 2023. Right: DeepSeek-R1-Distill-Qwen-1.5B is distilled from JustRL-DeepSeek-1.5B on DAPO-Math-17K with a 15,360-token generation budget. OPD develops a large late gradient spike despite otherwise decreasing gradient norms.
Setting
Student
Teacher
Training data
Budget
Evaluation
Rollout length
R1-Distill-Qwen-1.5B
JustRL-DeepSeek-1.5B
DAPO-Math-17K
7,168 / 15,360 / 18,432 tokens
AIME24/25, AMC23
Base student
Qwen3-1.7B-Base
Qwen3-4B-Base-GRPO
DAPO-Math-17K
7,168 tokens
AIME24/25, AMC23
Base student
Qwen3-1.7B-Base
Qwen3-4B (non-thinking)
OpenThoughts3
7,168 tokens
AIME24/25, AMC23
ALFWorld
Qwen2.5-3B-Instruct
GiGPO-7B
ALFWorld train
30 turns; 512 tokens/action
Seen / unseen success
WebShop
Qwen2.5-3B-Instruct
GiGPO-7B (WebShop)
Rebuilt 120K catalog
15 turns; 512 tokens/action
Success
Agentic code
Qwen3-1.7B-SFT
DemyAgent-GRPO-4B
Open-AgentRL-30K (code only)
8 turns / 12,288 tokens
LCB v1/v6, HO-200 full/sub
Appendix
Table 4: Experimental settings. The budget specifies the maximum generated tokens for reasoning tasks and the maximum number of environment or tool turns for interactive tasks.
Study
Budget
Segment
Mode
Controller
Initialization / data
Length sweep
7,168 / 15,360 / 18,432
256 tokens
rewind
fixed τ=0.7 , ϵ=0.05
DeepSeek-R1-Distill-Qwen-1.5B; DAPO-Math-17K
DAPO base, low initial coverage
7,168
256 tokens
rewind
fixed τ=0.2 ; initial takeover ≈25%
Qwen3-1.7B-Base; DAPO-Math-17K
DAPO base, medium coverage
7,168
256 tokens
restart
target rate 45%
Qwen3-1.7B-Base; DAPO-Math-17K
DAPO base, high initial coverage
7,168
64 / 128 / 256 tokens
rewind
fixed τ=.475/.362/.296 ; initial takeover ≈90%
Qwen3-1.7B-Base; DAPO-Math-17K
OpenThoughts3 base
7,168
256 tokens
rewind / restart
target rate 90%→5% , or fixed τ=0.87
Qwen3-1.7B-Base; OpenThoughts3
Appendix
Table 5: Reasoning configurations. Target rate is a rollout-level control parameter; the realized rate and teacher-token dose are measured from the generated batches. A dash denotes vanilla OPD.
Math reasoning: student Qwen3-1.7B-Base; teacher Qwen3-4B (non-thinking); data OpenThoughts3
Method
AIME24
AIME25
AMC23
Average ↑
Teacher
21.04
20.62
67.66
36.44
OPD
5.62
5.42
27.50
12.85
DTDD fixed τ=0.87
6.88 +1.25
5.42 +0.00
31.25 +3.75
14.51 +1.67
DTDD rewind, target 90%→5%
8.96 +3.33
4.58 −0.83
29.53 +2.03
14.36 +1.51
DTDD restart, target 90%→5%
8.75 +3.13
5.21 −0.21
33.28 +5.78
15.75 +2.90
Appendix
Table 7: Matched-checkpoint results (%). All trained methods within each block are evaluated at the same update: 200 for math, 400 for ALFWorld, and 60 ∗ for agentic code. Subscripts show the absolute percentage-point change from OPD; bold marks the best trained method. For code, Held-out averages the two HO-200 splits and LCB averages LiveCodeBench v1 and v6.
Figure 8: Complete 7K training dynamics for DeepSeek-R1-Distill-Qwen-1.5B distilled from JustRL-DeepSeek-1.5B on DAPO-Math-17K. This setting serves as the stable control. The OPD telemetry begins at update 61; held-out comparisons use checkpoints available for both methods.
Figure 9: Complete 15K training dynamics. The OPD gradient and entropy spikes coincide with increasing truncation and decreasing teacher overlap. DTDD limits each component of this event while preserving held-out accuracy.
Figure 10: Complete 18K training dynamics. DTDD suppresses the late OPD instability in gradient norm, entropy, truncation, and teacher overlap. Its intervention rate decreases over training and remains responsive to the late discrepancy increase.
Budget
OPD signal
Peak gradient
Updates >2
Accuracy
15K
Unclipped reproduction
14.33
22
54.50
15K
Clamp [−5,5]
12.48
21
55.26
15K
Clamp [−1,1]
0.95
0
55.05
18K
Unclipped reproduction
12.42
21
54.62
18K
Clamp [−5,5]
11.28
21
53.72
18K
Clamp [−1,1]
0.96
0
53.54
Appendix
Table 8: Clipped-OPD control at update 279. Accuracy is held-out average@16 (%); gradient statistics exclude the first 50 updates and are measured before global gradient clipping.
Table 9: Base-initialization dose comparison on DAPO-Math-17K (average@16, %). Updates 120 and 240 provide common early and late checkpoints for all reported arms.
DTDD controller
N
Last Δ
Last-five Δ
All-shared Δ
OPD late range
Fixed τ=0.87
10
+1.67
+1.68
+1.49
1.46
Rewind, target 90%→5%
10
+1.51
+1.47
+1.03
1.46
Restart, target 90%→5%
10
+2.90
+1.53
+1.21
1.46
Appendix
Table 10: OpenThoughts3 controller gains over OPD through update 200. Accuracy differences and OPD ranges are in percentage points; summaries are defined in Appendix B.4 .
Table 11: Per-benchmark base-initialization results (average@16, %) at shared update 240 for DAPO-Math-17K and 200 for OpenThoughts3.
Arm
≥21.3%
≥22.29%
≥22.75%
Best (step)
(reference)
(OPD seed 0)
(OPD mean, n=2 )
OPD, seed 0
180
200
–
22.41 (220)
OPD, seed 1 ∗
240
240
240
23.21 (240)
DTDD rewind, C=64
60
60
–
22.43 (60)
DTDD rewind, C=128
80
220
–
22.57 (220)
DTDD rewind, C=256
70
100
180
22.88 (180)
Appendix
Table 12: First evaluated update reaching each accuracy level on DAPO-Math-17K (average@16, %). The two OPD-derived targets use update 240; “–” denotes a target not reached by that update. ∗ After update 120, OPD seed 1 is evaluated every 40 updates. Crossing times reflect the available evaluation checkpoints.
Figure 11: Controller dynamics for the three high-coverage DAPO-Math rewind arms. Faint curves show per-update values and solid curves show seven-update moving averages. (a) Rollout-level takeover rate; the dotted line is the initial calibration target. (b) Segment divergence normalized by each arm’s fixed threshold (solid: mean; dashed: 90th percentile). (c) Fraction of generated training tokens written by the teacher. (d) Mean takeover position, conditional on triggering and converted from window index to token position.
Trigger / method
Teacher-suffix objective
Optimizer step
20
40
60
80
100
120
OPD (seed 0)
–
8.58
12.67
12.36
14.01
13.47
14.65
Segment, DTDD (seed 0)
BC
12.83
18.82
20.49
21.60
20.90
20.90
OPD (seed 1)
–
9.69
10.36
11.91
13.99
12.83
14.43
Segment, DTDD (seed 1)
BC
12.47
19.65
20.16
21.34
21.25
20.76
Token spike
Unified KD
8.84
9.53
12.47
12.14
12.81
13.51
Appendix
Table 13: Base-student controller and objective controls on DAPO-Math-17K (held-out average@16, %). A dash denotes an unscheduled evaluation.
Controller / objective
Median trigger
Trigger rate
Teacher-written tokens/update
Time/update
DTDD (segment + BC)
segment 5
71%
0.52M
15.1 min (4 × A100); 19.5 min (4 × L40S)
Token spike + unified KD
token 91
79%
0.40M
10.2 min
Token spike + BC
token 88
82%
0.43M
13.0 min
Segment + unified KD
segment 1
80%
0.67M
14.2 min
AdaSwitch ( K=3 )
token 14
100%
0.86M
31.3 min
OPD
–
–
0
5.4 min (2 × A100)
Appendix
Table 14: Later-training controller and runtime measurements for the base-student controls. Trigger position is reported in each controller’s native granularity. Runtime follows each completed job’s allocation; GPU models are shown where recorded, and times are not hardware-normalized.
Method
Optimizer step
20
40
60
80
100
120
OPD
9.13±0.79
11.52±1.63
12.14±0.32
14.00±0.01
13.15±0.45
14.54±0.16
DTDD
12.65±0.25
19.23±0.59
20.32±0.23
21.47±0.18
21.08±0.25
20.83±0.10
Appendix
Table 15: Two-seed mean ±SD on DAPO-Math-17K (average@16, %).
Training route
Optimizer step
20
40
60
80
100
120
160
200
240
OPD from base
8.58
12.67
12.36
14.01
13.47
14.65
17.01
22.29
22.29
DTDD from base
12.83
18.82
20.49
21.60
20.90
20.90
21.02
21.96
22.40
SFT → OPD
21.55
21.27
20.63
22.81
22.45
22.03
22.31
21.67
21.77
Appendix
Table 16: Fixed-update comparison of direct OPD, online takeover, and offline SFT warm start (held-out average@16, %). A dash denotes an unscheduled checkpoint.
Table 17: Serial cost to comparable observed checkpoints. Training time sums logged step times. Stage times are rounded; totals use unrounded measurements. Accelerator types differ across routes.
Method
Seen
Unseen
Average
Qwen2.5-3B-Instruct ← GiGPO-7B, LoRA
OPD
70.71
69.40
70.06
DTDD
84.29
70.90
77.59
Qwen2.5-3B-Instruct ← GiGPO-7B, full fine-tuning
OPD
82.14
76.87
79.50
DTDD
82.14
77.61
79.88
Appendix
Table 18: ALFWorld success rate (%) at update 400, the last common checkpoint. Full-model training provides a capacity control for LoRA.
Figure 12: ALFWorld action-level takeover (Qwen2.5-3B-Instruct ← GiGPO-7B). The controller withdraws as discrepancy contracts and reactivates around the update-180 disruption.
Figure 13: ALFWorld action-level controller dynamics. Top: takeover rate and action discrepancy. Bottom: intervention turn and its depth within the episode, conditional on takeover. The student is Qwen2.5-3B-Instruct, the teacher is GiGPO-7B, and the maximum episode length is 30 turns.
Table 20: Complete inventory for Figure 6 . Scores and headroom h are percentages; mean Δ is the DTDD –OPD difference in percentage points across N matched checkpoints. Step identifies the OPD endpoint used to compute headroom. Math uses average@16, ALFWorld and WebShop use success rate, and code uses LiveCodeBench v1 average@8.
Condition
Setting
Observation
Result
Interpretation
Little remaining gap
7K/15K/18K distilled math student
OPD finishes within 5–9% of teacher
Stability gain; accuracy tie
Stability without higher accuracy
Valid, learnable gap
Base math student
OPD learns slowly; teacher continuation is learnable
High dose reaches the reference score earlier
Faster early learning
Warm, healthy student
Agentic code, ρT=2.8%
Small in-distribution gain
+1.03 points at update 130
Sparse intervention can help
Excessive dose
Agentic code, ρT≈50%
Teacher behavior replaces on-policy interaction
−5.80 aggregate points at update 130
Excessive replacement
Invalid teacher
Rebuilt WebShop
Teacher 18.69% < OPD 21.21%
DTDD 11.62% at update 180
Teacher mismatch
Limited learnability
ALFWorld, 0.5B ← 7B
Matched mean: OPD 0.79% , DTDD 0.96%
+0.17 points; both remain near zero
Limited student transfer
Appendix
Table 21: Boundary cases and their empirical interpretation. Scores use the native evaluation metric of each environment.
On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.
On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conventional off-policy distillation with substantially less data. However, standard token-level OPD can provide only fragmented corrections along an erroneous student trajectory and cannot unfold a complete and correct repair path. Motivated by this limitation, we propose Step-Level On-Policy Distillation (SOPD), which combines the long-horizon correction of supervised fine-tuning (SFT) with the on-policy advantage of OPD to provide step-level supervision over complete student-generated trajectories. We show that, at different limits of step length, SOPD reduces to SFT or approximates OPD. Compared with SFT, the teacher responses in SOPD are conditioned on student trajectories and therefore align more closely with student-visited states; compared with OPD, SOPD provides longer-horizon corrections rather than fragmented token-level guidance. Across both reasoning and agent tasks, SOPD substantially outperforms conventional SFT and OPD. For example, on ALFWorld, SOPD improves the average success rate by 13.4 points over Vanilla OPD. We hope this work offers a new perspective for future research on distillation methods.
Changhui Sun, Lanbo Liu, Hang Lei +11
State Key Laboratory for Novel Software Technology, Nanjing University, China
On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, Supervision Fidelity Decay (SFD): as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce Lookahead Group Reward (\ours{}). Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by 2.57 points over OPD for a 7B student, with gains increasing in longer-generation and reaching +4.92 points on AIME-26 at 39k tokens.
Yanjiang Liu, Jie Lou, Xinyan Guan +7
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu