On-policy distillation (OPD) trains a student on its own reasoning trajectories using feedback from a stronger teacher. Teacher interventions can improve these trajectories, but also change the distribution on which the student learns. Our controlled studies show that rollout quality alone is an incomplete criterion for allocating teacher guidance. Deeper intervention yields diminishing gains in rollout accuracy while increasing off-policy load. In a training probe with a restricted rollout horizon, peak student accuracy and performance retention favor different intervention strengths. The preferred intervention depth and placement also vary across benchmarks. These findings motivate MAESTRO, which uses local policy disagreement to jointly adapt when the teacher takes over and how long it generates. Its {policy disagreement score} combines teacher-weighted candidate coverage with local distribution similarity and is aggregated within reasoning paragraphs. Across eight mathematical reasoning benchmarks, MAESTRO achieves the highest macro-average accuracy among the compared methods for both 0.6B and 1.7B Qwen3 students, with the 1.7B student leading on every benchmark. MAESTRO also reduces average training response length by 67.3% relative to standard OPD. The code is available at https://github.com/yhao-wang/MAESTRO.
Figures & tables
Figure 1: The core motivation of MAESTRO is to balance the benefit of teacher correction against the off-policy shift introduced by teacher generation.
Figure 2: Scaling trade-off of teacher intervention. We vary teacher intervention depth and measure its effect on (a) trajectory accuracy and (b) trajectory-normalized off-policy load.
Figure 3: Teacher intervention under a limited rollout length. (a–b) We vary intervention strength to examine peak performance and training stability. (c–f) We fix the intervention schedule and vary post-intervention student continuation to examine training efficiency.
Figure 4: Adaptive allocation of teacher intervention. (a) Harder problems require greater intervention depth. (b) The preferred intervention position also varies with reasoning progress.
Method
AIME24
AIME25
AIME26
AMC23
HMMT26
HMMT25
MATH500
Olymp.
Avg.
Qwen3-0.6B-Non-Thinking
Student
1.77
2.40
0.73
24.45
0.76
3.85
44.10
16.36
11.80
SFT
4.90
7.60
4.06
34.92
1.89
3.02
59.45
26.74
17.82
KD
4.17
7.19
4.79
35.23
2.94
2.71
57.75
27.15
17.74
GRPO
8.23
15.21
10.00
46.56
7.86
6.04
68.60
35.42
24.74
TRD
4.58
8.54
3.96
36.33
3.79
2.81
57.60
26.93
18.07
Table 1: Main results on eight mathematical reasoning benchmarks. We report mean accuracy (%) on each benchmark and the macro-average across all benchmarks.
Figure 5: Ablation, sensitivity, and training-efficiency analysis of MAESTRO. (a) Component ablation. (b) PDS threshold sensitivity. (c) Average training response length.
Figure 6: Intervention behavior across problem difficulty and training. (a) Average intervention usage per trajectory. (b) Hard-case trigger rate across the same benchmarks. (c) Intervention ratio during training, measured as the fraction of teacher-generated tokens in each response.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Sensitivity of MAESTRO to the paragraph-level aggregation hyperparameters. (a) Sensitivity to the weighting strength α . (b) Sensitivity to the decay scale τ . The default configuration, α=0.5 and τ=12 , achieves the highest accuracy in both sweeps.
Figure 8: Comparison of teacher usage between MAESTRO and Relay-OPD. (a) Intervention ratio throughout training. (b) Average teacher-token usage per training step, showing comparable overall teacher budgets between the two methods. (c) Distribution of teacher-token volume across training steps. Although their aggregate usage is similar, MAESTRO exhibits a different temporal allocation of teacher guidance.
On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy. With a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark, outperforming standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for 1.7B, with consistent gains at 0.6B. Training trajectory length is reduced by over 50%.
Haolei Xu, Xiaowen Xu, Haiwen Hong +5
1Zhejiang University · 2Yuvion Team, Alibaba Group
On-policy distillation (OPD), which supervises a student on its own sampled trajectories, has emerged as a data-efficient post-training method for improving reasoning while avoiding the reward dependence of reinforcement learning and the catastrophic forgetting often observed in standard supervised fine-tuning. However, standard OPD typically computes teacher supervision under noisy student-generated contexts and often relies on a single stochastic teacher rollout per prompt. As a result, the supervision signal can be high-variance: the sampled teacher trajectory can be incorrect, uninformative, or poorly matched to the student's current reasoning behavior. To address this limitation, we propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation. BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the curated teacher trajectory. Rather than distilling from the first sampled teacher rollout, BRTS samples a small pool of teacher trajectories and selects the auxiliary trajectory using a simple priority rule: correctness first, student alignment second. When multiple correct teacher trajectories are available, BRTS chooses the one most aligned with the student's current behavior; when unconditioned teacher samples fail on harder prompts, it invokes a ground-truth-conditioned recovery step to elicit a natural derivation. The selected trajectory is then used to provide reliable teacher-context supervision inside the OPD loop, augmented with an auxiliary loss on the teacher trajectory. Experiments on AIME 2024, AIME 2025, and AMC 2023 show that BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets. Our code is available at https://github.com/BWGZK-keke/BRTS.
Ke Zhang, Yunjie Tian, Dongdi Zhao +4
Johns Hopkins University · TikTok · University of California, San Diego +1
On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, \textbf{Supervision Fidelity Decay (SFD)}: as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce \textbf{Lookahead Group Reward (\ours{})}. Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by \textbf{2.57} points over OPD for a 7B student, with gains increasing in longer-generation and reaching +\textbf{4.92} points on AIME-26 at 39k tokens.
Yanjiang Liu, Jie Lou, Xinyan Guan +7
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu