ATI-OPD: Adaptive Teacher Intervention for Stable Long-Sequence On-Policy Distillation

Jiannan Guan; Youxin Jiang; Yutai Hou; Qiguang Chen; Yirong Zeng; Haitao Wang; Dengyun Peng; Zheng Yan

cs.CLOct 28, 2026

ATI-OPD: Adaptive Teacher Intervention for Stable Long-Sequence On-Policy Distillation

Authors: Jiannan Guan, Youxin Jiang, Yutai Hou, Qiguang Chen, Yirong Zeng, Haitao Wang, Dengyun Peng, Zheng Yan,

Organizations: Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Huawei Technologies Co., Ltd.

Abstract

On-policy distillation (OPD) trains a student on its own responses using token-level feedback from a teacher. As responses grow longer, the student's prefix can depart from the teacher's own reasoning, weakening its feedback on later tokens. Updates based on this feedback can reinforce the same deviations in later responses and destabilize training. Prior approaches such as length control and loss reweighting limit learning from problematic responses but leave their prefixes unchanged. Fixed cutoffs can also discard later reasoning. We propose Adaptive Teacher Intervention for On-Policy Distillation (ATI-OPD) to improve subsequent teacher guidance by revising the student's reasoning prefix. ATI-OPD replaces the earliest high-disagreement segment with a short teacher segment, then lets the student resume its own reasoning from the revised text. To handle renewed disagreement, it excludes persistently divergent suffixes from the distillation loss while retaining useful student continuation. Training diagnostics show less teacher-student disagreement, fewer gradient spikes, and fewer extremely long responses. Across five mathematical reasoning benchmarks and two teacher-student pairs, ATI-OPD improves average student-only accuracy over standard OPD.

Figures & tables

Figure 1: Vanilla OPD on JustRL-DeepSeek exhibits prediction disagreement and training instability. (a) Gap measures the absolute difference between teacher and student log-probabilities for the same sampled token and prefix. (b) Across training steps, entropy rebounds, gradient norms spike, and more responses reach the max token limit.

Figure 2: ATI-OPD replaces a high-divergence student segment with a teacher segment and lets the student generate a new continuation. Persistent divergence after the handoff determines a supervision cutoff. Later tokens remain in the generated response but are excluded from distillation.

JustRL-DeepSeek-1.5B $\rightarrow$ DeepSeek-R1-Distill-Qwen-1.5B
Method	AIME24	AIME25	AMC23	AMO	HMMT	Average
Student	$29.58_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 2.08}}$	$21.88_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.89}}$	$72.34_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.77}}$	$1.44_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 0.48}}$	$16.10_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.60}}$	$28.27_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 0.74}}$
+GRPO [ 20 ]	$31.04_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 2.11}}$	$24.58_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.97}}$	$71.88_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.78}}$	$1.44_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 0.48}}$	$13.83_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.50}}$	$28.55_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 0.75}}$
+OPD [ 12 ]	$\underline{46.25}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 2.28}}$	$\mathbf{33.96}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 2.16}}$	$\underline{87.03}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.33}}$	$\underline{2.08}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 0.57}}$	$\underline{22.92}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.83}}$	$\underline{38.45}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 0.78}}$
+ESR [ 34 ]	$37.92_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 2.21}}$	$28.75_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 2.07}}$	$78.44_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.63}}$	$1.76_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 0.53}}$	$16.86_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.63}}$	$32.74_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 0.77}}$
+ATI-OPD	$\mathbf{49.17}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 2.28}}$	$\underline{33.54}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 2.15}}$	$\mathbf{88.59}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.26}}$	$\mathbf{2.72}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 0.65}}$	$\mathbf{24.24}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 1.87}}$	$\mathbf{39.65}_{\scriptscriptstyle{\color[rgb]{0.45,0.45,0.45}\pm 0.78}}$

Table 1: Student-only accuracy (%) on five mathematical reasoning benchmarks. ESR uses a fixed-prefix supervision implementation [ 34 ] . Subscripts report inference standard errors in 16 independent responses per question. Bold / underline mark the best / second-best results per pair.

Figure 3: Training diagnostics for JustRL-DeepSeek: absolute gap, entropy, response-length tail (last 32 training steps), top-16 coverage, gradient norm, and clipping rate.

Figure 4: Gains over each OPD baseline on three shared tasks (%). $\dagger$ : published results.

Figure 5: Local handoff diagnostics for JustRL-DeepSeek: (a) student K3 after the original and replacement segments, shown as curves and a scatter plot; (b) post-handoff gap ratios. Appendix C.4 defines the matched student windows.

Figure 6: Teacher-segment supervision for JustRL-DeepSeek: (a) five-task average accuracy; (b) top-16 coverage and token-zlib compression ratio (higher values indicate greater compressibility). Both diagnostics include teacher tokens for OPD, but exclude them for Masked and SFT.

Figure 7: JustRL-DeepSeek ablations and outcomes: (a) benchmark accuracy; (b) rescue/harm rates among originally incorrect/correct intervened responses, respectively; inset percentages use all rollouts. (c) Online gain (%) by supervision cutoff (Cut/No cut) and 128-token local K3 change ( $\downarrow$ : decrease; $\leftrightarrow/\uparrow$ : no decrease). Error bars are step-cluster bootstrap 95% confidence intervals; see Appendix C.5 for details.

Figure 8: Intervention dynamics for JustRL-DeepSeek: (a) intervention rate; (b) first trigger position relative to the original response length; (c) intervention rate versus online gain over all rollouts (%). Colors mark training steps; arrows connect consecutive 32-step phase centers.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Training data	DAPO-Math-17K
Max prompt length	1,024 tokens
Max response length	$N_{\max}=23{,}552$ tokens
Batch size	64
Rollouts per prompt	4
Training steps	217

Appendix

Table 2: Main settings for ATI-OPD training, student-only evaluation, and local intervention.

Method	AIME24	AIME25	AMC23	AMO-Bench	Avg.
OPD	46.46	37.71	86.41	3.04	43.40
ATI-OPD	48.54	39.17	87.66	3.53	44.72

Appendix

Table 3: Exploratory student-only accuracy (%) with a 7B student. Evaluation uses temperature 1.0, top-

p=1.0

, no top-

k

truncation, with 16 responses per question. Avg. equally weights the four available tasks.

JustRL-DeepSeek-1.5B $\rightarrow$ DeepSeek-R1-Distill-Qwen-1.5B
Method	$\bm{\Delta}$ AIME24	$\bm{\Delta}$ AIME25	$\bm{\Delta}$ AMC23	$\bm{\Delta}$ Avg.
POPD †	+1.70	+0.50	$-1.40$	+0.27
TOPD-0.50 †	+1.00	+0.80	$-0.70$	+0.37
TOPD-0.25 †	+2.00	$-0.10$	$-1.20$	+0.23
TOPD-0.10 †	+1.40	$-1.80$	$-4.10$	$-1.50$
ATI-OPD	+2.92	$-0.42$	+1.56	+1.35

Appendix

Table 4: Absolute accuracy gains over each method’s own OPD baseline on three shared benchmarks (%). The number following TOPD denotes the truncation length as a fraction of the original maximum response length.

\dagger

denotes gains calculated from the accuracies published in the corresponding work [ 30 ] .

$L_{T}$	AIME24	AIME25	AMC23	AMO-Bench	HMMT Feb26	Avg.
128	49.17	33.54	88.59	2.72	24.24	39.65
512	48.33	36.88	87.19	2.72	21.59	39.34
1024	46.46	36.46	85.94	2.56	22.92	38.87

Appendix

Table 5: Sensitivity to teacher segment length for JustRL-DeepSeek (accuracy, %), with

L_{S}=128

. The first four benchmarks use the training-step-217 evaluation; HMMT February 2026 uses the corresponding standalone checkpoint evaluation. Avg. equally weights the five tasks.

$\tau$	AIME24	AIME25	AMC23	AMO-Bench	HMMT Feb26	Avg.
0.10	47.71	37.50	86.72	3.37	24.43	39.94
0.15	49.17	33.54	88.59	2.72	24.24	39.65
0.30	47.71	34.58	88.91	2.88	24.05	39.63

Appendix

Table 6: Sensitivity to the intervention trigger threshold for JustRL-DeepSeek (accuracy, %). Avg. is the equal-weight mean of five benchmarks.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 517 papers

From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation
Sep 29, 2026Yuhao Wang, Ruiyang Ren, Yinan Zhang +3Language Model Distillation On-Policy Distillation
Nanyang Technological University, Singapore · Baidu Inc.
Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
May 29, 2026Yanjiang Liu, Jie Lou, Xinyan Guan +7CoT Distillation Language Model Distillation
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu
Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation
Sep 6, 2026Jingang Zhou, Yuyi Zhou, Haiyang Guo +6Language Model Distillation On-Policy Distillation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Science · Ant Group

Sep 29, 2026J/K move · Enter open · S save

From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation

Yuhao Wang, Ruiyang Ren, Yinan Zhang, Ruiqing Zhang+2

Nanyang Technological University, Singapore · Baidu Inc.

Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) trains a student on its own reasoning trajectories using feedback from a stronger teacher. Teacher interventions can improve these trajectories, but also change the distribution on which the student learns. Our controlled studies show that rollout quality alone is an incomplete criterion for allocating teacher guidance. Deeper intervention yields diminishing gains in rollout accuracy while increasing off-policy load. In a training probe with a restricted rollout horizon, peak student accuracy and performance retention favor different intervention strengths. The preferred intervention depth and placement also vary across benchmarks. These findings motivate MAESTRO, which uses local policy disagreement to jointly adapt when the teacher takes over and how long it generates. Its {policy disagreement score} combines teacher-weighted candidate coverage with local distribution similarity and is aggregated within reasoning paragraphs. Across eight mathematical reasoning benchmarks, MAESTRO achieves the highest macro-average accuracy among the compared methods for both 0.6B and 1.7B Qwen3 students, with the 1.7B student leading on every benchmark. MAESTRO also reduces average training response length by 67.3% relative to standard OPD. The code is available at https://github.com/yhao-wang/MAESTRO.

Abstract

On-Policy Distillation

Explore similar work

From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation