DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

Qijia He; Zequn Li; Spencer Luo; Tianle Chen; Pradyumna Narayana; Zixian Ma; Ranjay Krishna; Lan Nie

cs.LGOct 13, 2026

DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

Authors: Qijia He, Zequn Li, Spencer Luo, Tianle Chen, Pradyumna Narayana, Zixian Ma, Ranjay Krishna, Lan Nie,

Organizations: Google · University of Washington · Boston University

Abstract

On-policy distillation (OPD) provides dense token-level supervision on student rollouts, but this supervision can become unreliable when the student and teacher diverge, either because of a large initial policy gap or compounding errors in long-horizon reasoning and tool use. We introduce Divergence-Triggered Dynamic Distillation (DTDD), an adaptive OPD framework that uses segment-level student-teacher divergence to trigger teacher takeover during training. When divergence triggers an intervention during reasoning, the student prefix continues to receive standard OPD supervision. The teacher then generates a recovery suffix, which the student learns through behavior cloning, using cross-entropy on the teacher-generated tokens. We evaluate DTDD in three regimes: long-horizon reasoning, distillation from a base initialization without offline supervised fine-tuning (SFT), and multi-turn agents. We also compare alternative trigger designs and teacher-intervention rates. DTDD reduces the post-warmup peak gradient norm by $8.6\times$ and, from a base initialization, reaches $21.3\%$ held-out math accuracy in $2.6\times$ fewer optimizer updates than OPD; both methods finish near $22\%$ . It improves best-observed performance by $4.8$ and $4.1$ percentage points on ALFWorld with LoRA and agentic code, respectively. Ablations show that segment-averaged triggers outperform token-spike triggers. Behavior cloning is also more effective than the tested OPD surrogate for training teacher-generated recoveries. Across 23 experimental arms, takeover tends to help when the teacher substantially outperforms the OPD-trained student, provided that the teacher can recover from student-visited states, the student can learn those recoveries, and teacher intervention is neither excessive nor insufficient.

Figures & tables

Figure 1: Representative failures of local OPD supervision: (A) slow learning from a base initialization, (B) late instability under long generation budgets, and (C) compounding errors after an incorrect action in multi-turn interaction. Appendix A reports measured examples of (A) and (B).

Figure 2: Reasoning realization of DTDD . (A) The student generates a segmented rollout. (B) The first threshold crossing selects the rewind boundary. (C) The teacher regenerates the suffix from the retained student prefix. (D) The prefix is trained with OPD and the suffix with behavior cloning; without a crossing, DTDD reduces to OPD.

Table 3

Figure 3: DTDD suppresses late OPD spikes on DAPO-Math-17K at 15K/18K response budgets.

Figure 4: Held-out accuracy for Qwen3-1.7B-Base

\leftarrow

Qwen3-4B-Base-GRPO on DAPO-Math-17K. The length-gated control starts near

25\%

takeover and adapts thereafter.

Figure 5: Fixed-threshold dynamics (

\tau=0.87

) for Qwen3-1.7B-Base

\leftarrow

Qwen3-4B on OpenThoughts3.

Math reasoning: student Qwen3-1.7B-Base; teacher Qwen3-4B (non-thinking); data OpenThoughts3
Method	AIME24	AIME25	AMC23	Average $\uparrow$
Teacher	21.04	20.62	67.66	36.44
OPD	5.62	5.42	27.50	12.85
DTDD fixed $\tau=0.87$	6.88 $+1.26$	7.08 $+1.66$	31.41 $+3.91$	15.12 $+2.27$
DTDD rewind, target $90\%\!\rightarrow\!5\%$	7.29 $+1.67$	4.79 $-0.63$	32.19 $+4.69$	14.76 $+1.91$
DTDD restart, target $90\%\!\rightarrow\!5\%$	8.75 $+3.13$	5.21 $-0.21$	33.28 $+5.78$	15.75 $+2.90$

Table 3: Best-observed results (%; selection details in Appendix B.3 ). Subscripts show percentage-point changes from OPD; bold marks the best trained method.

Figure 6: DTDD minus OPD, averaged over all matched checkpoints, against the headroom plain OPD leaves below its teacher. Points left of zero are settings where OPD ends above the teacher’s own score. Ringed markers identify arms analyzed by the validity, learnability, and dose controls in Appendix F .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Two representative limitations of vanilla OPD. Left: Qwen3-1.7B-Base and Qwen3-1.7B-SFT are distilled from the same Qwen3-4B-Base-GRPO teacher on DAPO-Math-17K with a 7,168-token generation budget. The teacher, data, optimizer, and generation budget are held fixed; the SFT initialization has a large early advantage, while the base initialization eventually reaches a comparable plateau. Held-out accuracy is the mean avg@16 score on AIME 2024, AIME 2025, and AMC 2023. Right: DeepSeek-R1-Distill-Qwen-1.5B is distilled from JustRL-DeepSeek-1.5B on DAPO-Math-17K with a 15,360-token generation budget. OPD develops a large late gradient spike despite otherwise decreasing gradient norms.

Setting	Student	Teacher	Training data	Budget	Evaluation
Rollout length	R1-Distill-Qwen-1.5B	JustRL-DeepSeek-1.5B	DAPO-Math-17K	7,168 / 15,360 / 18,432 tokens	AIME24/25, AMC23
Base student	Qwen3-1.7B-Base	Qwen3-4B-Base-GRPO	DAPO-Math-17K	7,168 tokens	AIME24/25, AMC23
Base student	Qwen3-1.7B-Base	Qwen3-4B (non-thinking)	OpenThoughts3	7,168 tokens	AIME24/25, AMC23
ALFWorld	Qwen2.5-3B-Instruct	GiGPO-7B	ALFWorld train	30 turns; 512 tokens/action	Seen / unseen success
WebShop	Qwen2.5-3B-Instruct	GiGPO-7B (WebShop)	Rebuilt 120K catalog	15 turns; 512 tokens/action	Success
Agentic code	Qwen3-1.7B-SFT	DemyAgent-GRPO-4B	Open-AgentRL-30K (code only)	8 turns / 12,288 tokens	LCB v1/v6, HO-200 full/sub

Appendix

Table 4: Experimental settings. The budget specifies the maximum generated tokens for reasoning tasks and the maximum number of environment or tool turns for interactive tasks.

Study	Budget	Segment	Mode	Controller	Initialization / data
Length sweep	7,168 / 15,360 / 18,432	256 tokens	rewind	fixed $\tau=0.7$ , $\epsilon=0.05$	DeepSeek-R1-Distill-Qwen-1.5B; DAPO-Math-17K
DAPO base, low initial coverage	7,168	256 tokens	rewind	fixed $\tau=0.2$ ; initial takeover $\approx 25\%$	Qwen3-1.7B-Base; DAPO-Math-17K
DAPO base, medium coverage	7,168	256 tokens	restart	target rate $45\%$	Qwen3-1.7B-Base; DAPO-Math-17K
DAPO base, high initial coverage	7,168	64 / 128 / 256 tokens	rewind	fixed $\tau=.475/.362/.296$ ; initial takeover $\approx 90\%$	Qwen3-1.7B-Base; DAPO-Math-17K
OpenThoughts3 base	7,168	256 tokens	rewind / restart	target rate $90\%\!\rightarrow\!5\%$ , or fixed $\tau=0.87$	Qwen3-1.7B-Base; OpenThoughts3

Appendix

Table 5: Reasoning configurations. Target rate is a rollout-level control parameter; the realized rate and teacher-token dose are measured from the generated batches. A dash denotes vanilla OPD.

Domain	Evaluation set	Sampling / metric
Math	AIME24, AIME25, AMC23	16 samples, $T=0.7$ , top- $p=0.95$ ; average@16
ALFWorld	140 seen, 134 unseen tasks	Student-only environment success
Code	LCB v1/v6, HO-200 full/sub	8 samples, $T=1.0$ , top- $p=0.6$ ; execution accuracy

Appendix

Table 6: Held-out evaluation protocol.

Math reasoning: student Qwen3-1.7B-Base; teacher Qwen3-4B (non-thinking); data OpenThoughts3
Method	AIME24	AIME25	AMC23	Average $\uparrow$
Teacher	21.04	20.62	67.66	36.44
OPD	5.62	5.42	27.50	12.85
DTDD fixed $\tau=0.87$	6.88 $+1.25$	5.42 $+0.00$	31.25 $+3.75$	14.51 $+1.67$
DTDD rewind, target $90\%\!\rightarrow\!5\%$	8.96 $+3.33$	4.58 $-0.83$	29.53 $+2.03$	14.36 $+1.51$
DTDD restart, target $90\%\!\rightarrow\!5\%$	8.75 $+3.13$	5.21 $-0.21$	33.28 $+5.78$	15.75 $+2.90$

Appendix

Table 7: Matched-checkpoint results (%). All trained methods within each block are evaluated at the same update: 200 for math, 400 for ALFWorld, and 60 ∗ for agentic code. Subscripts show the absolute percentage-point change from OPD; bold marks the best trained method. For code, Held-out averages the two HO-200 splits and LCB averages LiveCodeBench v1 and v6.

Figure 8: Complete 7K training dynamics for DeepSeek-R1-Distill-Qwen-1.5B distilled from JustRL-DeepSeek-1.5B on DAPO-Math-17K. This setting serves as the stable control. The OPD telemetry begins at update 61; held-out comparisons use checkpoints available for both methods.

Figure 9: Complete 15K training dynamics. The OPD gradient and entropy spikes coincide with increasing truncation and decreasing teacher overlap. DTDD limits each component of this event while preserving held-out accuracy.

Figure 10: Complete 18K training dynamics. DTDD suppresses the late OPD instability in gradient norm, entropy, truncation, and teacher overlap. Its intervention rate decreases over training and remains responsive to the late discrepancy increase.

Budget	OPD signal	Peak gradient	Updates $>2$	Accuracy
15K	Unclipped reproduction	14.33	22	54.50
15K	Clamp $[-5,5]$	12.48	21	55.26
15K	Clamp $[-1,1]$	0.95	0	55.05
18K	Unclipped reproduction	12.42	21	54.62
18K	Clamp $[-5,5]$	11.28	21	53.72
18K	Clamp $[-1,1]$	0.96	0	53.54

Appendix

Table 8: Clipped-OPD control at update 279. Accuracy is held-out average@16 (%); gradient statistics exclude the first 50 updates and are measured before global gradient clipping.

Method	Window $C$	Controller	Update 120	Update 240
OPD	–	–	14.65	22.29
DTDD rewind	256	fixed $\tau=0.2$ , length-gated (initial takeover $\approx 25\%$ )	14.81	21.58
DTDD restart	–	target rate $45\%$	20.49	21.67
DTDD rewind	64	fixed $\tau=0.475$ (initial takeover $\approx 90\%$ )	21.32	21.04
DTDD rewind	128	fixed $\tau=0.362$ (initial takeover $\approx 90\%$ )	20.90	22.40
DTDD rewind	256	fixed $\tau=0.296$ (initial takeover $\approx 90\%$ )	21.23	21.39

Appendix

Table 9: Base-initialization dose comparison on DAPO-Math-17K (average@16, %). Updates 120 and 240 provide common early and late checkpoints for all reported arms.

DTDD controller	$N$	Last $\Delta$	Last-five $\Delta$	All-shared $\Delta$	OPD late range
Fixed $\tau=0.87$	10	$+1.67$	$+1.68$	$+1.49$	1.46
Rewind, target $90\%\!\rightarrow\!5\%$	10	$+1.51$	$+1.47$	$+1.03$	1.46
Restart, target $90\%\!\rightarrow\!5\%$	10	$+2.90$	$+1.53$	$+1.21$	1.46

Appendix

Table 10: OpenThoughts3 controller gains over OPD through update 200. Accuracy differences and OPD ranges are in percentage points; summaries are defined in Appendix B.4 .

Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B-Base-GRPO, DAPO-Math-17K, 7,168 tokens
Method	AIME24	AIME25	AMC23	Average
Teacher: Qwen3-4B-Base-GRPO	24.17	18.12	64.22	35.50
OPD	11.25	9.38	46.25	22.29
DTDD restart, target rate $45\%$	11.25	7.50	46.25	21.67
DTDD rewind, initial takeover $\approx 90\%$ , $C=128$	11.67	8.33	47.19	22.40
DTDD fixed $\tau{=}0.2$ , initial takeover $\approx 25\%$	11.88	7.71	45.16	21.58

Appendix

Table 11: Per-benchmark base-initialization results (average@16, %) at shared update 240 for DAPO-Math-17K and 200 for OpenThoughts3.

Arm	$\geq 21.3\%$	$\geq 22.29\%$	$\geq 22.75\%$	Best (step)
	(reference)	(OPD seed 0)	(OPD mean, $n=2$ )
OPD, seed 0	180	200	–	22.41 (220)
OPD, seed 1 ∗	240	240	240	23.21 (240)
DTDD rewind, $C=64$	60	60	–	22.43 (60)
DTDD rewind, $C=128$	80	220	–	22.57 (220)
DTDD rewind, $C=256$	70	100	180	22.88 (180)

Appendix

Table 12: First evaluated update reaching each accuracy level on DAPO-Math-17K (average@16, %). The two OPD-derived targets use update 240; “–” denotes a target not reached by that update. ∗ After update 120, OPD seed 1 is evaluated every 40 updates. Crossing times reflect the available evaluation checkpoints.

Figure 11: Controller dynamics for the three high-coverage DAPO-Math rewind arms. Faint curves show per-update values and solid curves show seven-update moving averages. (a) Rollout-level takeover rate; the dotted line is the initial calibration target. (b) Segment divergence normalized by each arm’s fixed threshold (solid: mean; dashed: 90th percentile). (c) Fraction of generated training tokens written by the teacher. (d) Mean takeover position, conditional on triggering and converted from window index to token position.

Trigger / method	Teacher-suffix objective	Optimizer step
		20	40	60	80	100	120
OPD (seed 0)	–	8.58	12.67	12.36	14.01	13.47	14.65
Segment, DTDD (seed 0)	BC	12.83	18.82	20.49	21.60	20.90	20.90
OPD (seed 1)	–	9.69	10.36	11.91	13.99	12.83	14.43
Segment, DTDD (seed 1)	BC	12.47	19.65	20.16	21.34	21.25	20.76
Token spike	Unified KD	8.84	9.53	12.47	12.14	12.81	13.51

Appendix

Table 13: Base-student controller and objective controls on DAPO-Math-17K (held-out average@16, %). A dash denotes an unscheduled evaluation.

Controller / objective	Median trigger	Trigger rate	Teacher-written tokens/update	Time/update
DTDD (segment + BC)	segment 5	71%	0.52M	15.1 min (4 $\times$ A100); 19.5 min (4 $\times$ L40S)
Token spike + unified KD	token 91	79%	0.40M	10.2 min
Token spike + BC	token 88	82%	0.43M	13.0 min
Segment + unified KD	segment 1	80%	0.67M	14.2 min
AdaSwitch ( $K=3$ )	token 14	100%	0.86M	31.3 min
OPD	–	–	0	5.4 min (2 $\times$ A100)

Appendix

Table 14: Later-training controller and runtime measurements for the base-student controls. Trigger position is reported in each controller’s native granularity. Runtime follows each completed job’s allocation; GPU models are shown where recorded, and times are not hardware-normalized.

Method	Optimizer step
	20	40	60	80	100	120
OPD	$9.13_{\pm 0.79}$	$11.52_{\pm 1.63}$	$12.14_{\pm 0.32}$	$14.00_{\pm 0.01}$	$13.15_{\pm 0.45}$	$14.54_{\pm 0.16}$
DTDD	$12.65_{\pm 0.25}$	$19.23_{\pm 0.59}$	$20.32_{\pm 0.23}$	$21.47_{\pm 0.18}$	$21.08_{\pm 0.25}$	$20.83_{\pm 0.10}$

Appendix

Table 15: Two-seed mean ±SD on DAPO-Math-17K (average@16, %).

Training route	Optimizer step
	20	40	60	80	100	120	160	200	240
OPD from base	8.58	12.67	12.36	14.01	13.47	14.65	17.01	22.29	22.29
DTDD from base	12.83	18.82	20.49	21.60	20.90	20.90	21.02	21.96	22.40
SFT $\rightarrow$ OPD	21.55	21.27	20.63	22.81	22.45	22.03	22.31	21.67	21.77

Appendix

Table 16: Fixed-update comparison of direct OPD, online takeover, and offline SFT warm start (held-out average@16, %). A dash denotes an unscheduled checkpoint.

Route	Measured stages	Checkpoint (accuracy)	Serial time	GPU-h	Teacher-written tokens
Online DTDD (seed 1)	online training (22.0 h)	Step 80 (21.34)	22.0 h	88	43M
SFT $\rightarrow$ OPD	teacher data (26.9 h) $+$ SFT (11.7 h) $+$ merge (0.1 h) $+$ OPD (4.0 h)	Step 20 (21.55)	$\approx$ 42.8 h	$\approx$ 66	95M

Appendix

Table 17: Serial cost to comparable observed checkpoints. Training time sums logged step times. Stage times are rounded; totals use unrounded measurements. Accelerator types differ across routes.

Qwen2.5-3B-Instruct $\leftarrow$ GiGPO-7B, LoRA
Method	Seen	Unseen	Average
OPD	70.71	69.40	70.06
DTDD	84.29	70.90	77.59
Qwen2.5-3B-Instruct $\leftarrow$ GiGPO-7B, full fine-tuning
OPD	82.14	76.87	79.50
DTDD	82.14	77.61	79.88

Appendix

Table 18: ALFWorld success rate (%) at update 400, the last common checkpoint. Full-model training provides a capacity control for LoRA.

Figure 12: ALFWorld action-level takeover (Qwen2.5-3B-Instruct

\leftarrow

GiGPO-7B). The controller withdraws as discrepancy contracts and reactivates around the update-180 disruption.

Figure 13: ALFWorld action-level controller dynamics. Top: takeover rate and action discrepancy. Bottom: intervention turn and its depth within the episode, conditional on takeover. The student is Qwen2.5-3B-Instruct, the teacher is GiGPO-7B, and the maximum episode length is 30 turns.

Qwen3-1.7B-SFT $\leftarrow$ DemyAgent-GRPO-4B, code interaction
Method	Dose	Held-out	LCB	Avg.
Teacher	–	63.16	58.11	60.63
OPD	0%	25.60	42.26	33.93
DTDD ( $\tau=0.50$ )	2.8%	31.50	38.43	34.96
DTDD ( $\tau=0.26$ )	$\sim$ 50%	19.78	36.48	28.13

Appendix

Table 19: Agentic code execution accuracy (average@8, %) at update 130. Held-out averages the two in-distribution splits; LCB averages LiveCodeBench v1 and v6.

Arm	Teacher	OPD endpoint	Step	$h$ (%)	$N$	Mean $\Delta$
DAPO-Math: R1-Distill-1.5B student / JustRL-1.5B teacher
7K rewind	58.45	54.44	260	6.86	10	$-0.04$
15K rewind	58.45	54.15	279	7.37	10	$-0.62$
18K rewind	58.45	55.38	279	5.26	14	$-1.20$
18K, no exploration floor	58.45	55.38	279	5.26	6	$-0.53$
Hard-gap, fixed threshold	58.45	53.12	150	9.12	7	$-0.47$

Appendix

Table 20: Complete inventory for Figure 6 . Scores and headroom

h

are percentages; mean

\Delta

is the DTDD –OPD difference in percentage points across

N

matched checkpoints. Step identifies the OPD endpoint used to compute headroom. Math uses average@16, ALFWorld and WebShop use success rate, and code uses LiveCodeBench v1 average@8.

Condition	Setting	Observation	Result	Interpretation
Little remaining gap	7K/15K/18K distilled math student	OPD finishes within 5–9% of teacher	Stability gain; accuracy tie	Stability without higher accuracy
Valid, learnable gap	Base math student	OPD learns slowly; teacher continuation is learnable	High dose reaches the reference score earlier	Faster early learning
Warm, healthy student	Agentic code, $\rho_{T}=2.8\%$	Small in-distribution gain	$+1.03$ points at update 130	Sparse intervention can help
Excessive dose	Agentic code, $\rho_{T}\approx 50\%$	Teacher behavior replaces on-policy interaction	$-5.80$ aggregate points at update 130	Excessive replacement
Invalid teacher	Rebuilt WebShop	Teacher 18.69% $<$ OPD 21.21%	DTDD 11.62% at update 180	Teacher mismatch
Limited learnability	ALFWorld, 0.5B $\leftarrow$ 7B	Matched mean: OPD $0.79\%$ , DTDD $0.96\%$	$+0.17$ points; both remain near zero	Limited student transfer

Appendix

Table 21: Boundary cases and their empirical interpretation. Scores use the native evaluation metric of each environment.

Long-Horizon Reasoning

See all 28 papers

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
Sep 2, 2026Zhiwei Zhang, Zechen Sun, Fei Zhao +6On-Policy Distillation RL Post-Training
AllSpark Team
Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning
Aug 17, 2026Changhui Sun, Lanbo Liu, Hang Lei +11Language Model Distillation On-Policy Distillation
State Key Laboratory for Novel Software Technology, Nanjing University, China
Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
May 29, 2026Yanjiang Liu, Jie Lou, Xinyan Guan +7CoT Distillation Language Model Distillation
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu

Sep 2, 2026J/K move · Enter open · S save

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation

Zhiwei Zhang, Zechen Sun, Fei Zhao, Kang Peng+5

AllSpark Team

On-Policy Distillation RL Post-Training

Open paper AskPDF arXiv

On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.

Abstract

Long-Horizon Reasoning

Explore similar work

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation

Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation