On-policy self-distillation (OPSD) supplies dense token-level feedback on trajectories sampled from the student's own policy, a richer training signal than the outcome-level rewards of reinforcement learning. This feedback comes from a teacher conditioned on a full reference solution unavailable to the student. The reference solution specifies the target but not how to move from the student's current error toward it, creating a solution-conditioned shortcut risk. We introduce AIR-OPD, an adaptive iterative repair framework for on-policy distillation that provides error-to-repair supervision. Given a failed response, a guidance generator synthesizes repair guidance for the current error. The student samples an on-policy retry with this guidance. If the retry remains incorrect, the generator produces new repair guidance for the newly observed error. At each round, a fixed teacher receives the guidance as privileged context and supervises the student on an error-aligned region of its latest failed response. Outcome-aware stage weighting favors early repair stages and credits stages whose immediate retry passes verification. We train AIR-OPD on the DAPO-Math-17K dataset and evaluate on AIME24, AIME25, and HMMT25, alongside out-of-distribution tests on MMLU-Pro and GPQA. We examine two guidance sources, self-guidance from the current student policy and external guidance from a larger model. For both Qwen3-4B and Qwen3-8B, AIR-OPD attains the best mathematical-reasoning averages, improving over the strongest baseline by up to 3.6 points, while preserving base-model performance on the out-of-distribution benchmarks.
Figures & tables
Figure 1: Overview of Air-OPD . When an on-policy rollout fails verification, the guidance generator G quotes the first flawed step as an error anchor at and synthesizes repair guidance gt . A frozen teacher conditioned on gt then supervises the student over the error-aligned repair region Rt , and the student retries with the guidance in context. A verified retry ends the trajectory, and outcome-aware stage weighting combines the stage losses with weights wt∝γtρst .
Mathematical Reasoning
Out-of-Distribution
Method
AIME24
AIME25
HMMT25
Math Avg.
MMLU-Pro
GPQA
OOD Avg.
Qwen3-4B
Base model
75.0
66.4
42.2
61.2
51.5
62.9
57.2
SFT
70.6
61.4
42.8
58.2
43.1
52.0
47.6
GRPO
75.6
67.5
43.9
62.3
51.3
62.4
56.8
OPSD
76.4
69.4
43.3
63.1
50.4
60.2
55.3
Table 1: Main results on mathematical-reasoning (Avg@12) and out-of-distribution (Avg@4) benchmarks. Both Air-OPD variants surpass all baselines on Math Avg. at each scale while preserving base-model performance on the OOD group. Self-G / External-G generate repair guidance with the current student policy and Qwen3.5-27B, respectively. Bold / underlined : best/second-best within each block.
Setting
Math Avg.
Air-OPD
65.8
Sample selection (OPSD)
All rollouts
63.1
Failed rollouts only
62.5
Supervision region
w/o error alignment
64.7
Table 2: Ablation results (Qwen3-4B, Self-G ). We test failure-only sample selection for OPSD and ablate the supervision region and the stage-weighting components of Air-OPD .
Figure 2: Effect of the repair budget K (Qwen3-4B). Math Avg. (left axis) rises through the default K=3 and then flattens, while each variant’s number of supervised stages (dashed, right axis; % of the samples OPSD supervises) grows with diminishing increments.
Figure 3: Training dynamics on the mathematical benchmarks (Qwen3-4B). Both Air-OPD variants lead OPSD and GRPO on the three-benchmark average from the first checkpoint onward. Curves show Avg@12 at checkpoints saved every 20 optimizer steps, and dashed lines mark the base model.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Guidance generator, first repair round. The template diagnoses the initial failed response y0 under the privileged reference solution and final answer.
Figure 5: Guidance generator, later repair rounds. The template replaces the current guidance after a retry that is still incorrect, and additionally receives the guidance the student has already seen.
Parameter
SFT
GRPO
OPSD
Air-OPD
Optimization and adaptation
Optimizer
AdamW
AdamW
AdamW
AdamW
Learning rate
5×10−6
5×10−6
5×10−6
5×10−6
Effective batch size
32
32
32
32
Training precision
BF16
BF16
BF16
BF16
LoRA rank / alpha
64 / 128
64 / 128
64 / 128
64 / 128
Appendix
Table 3: Training settings for SFT, GRPO, OPSD, and Air-OPD . Lengths are measured in tokens; the SFT limit applies to the full supervised sequence. A dash denotes a setting that is not applicable. For Air-OPD , one rollout is sampled per dialogue state: the initial response can be followed by at most K−1 retries.
Parameter
Value
Maximum new tokens
38,912
Maximum model length (tokens)
40,960
Thinking mode
Enabled
Sampling temperature
1.0
Top- p
0.95
Top- k
–
Appendix
Table 4: Inference settings for mathematical-reasoning evaluation. The base model and all trained methods use these settings on AIME24, AIME25, and HMMT25, without repair guidance.
On-policy self-distillation (OPSD) uses reference solutions as privileged hindsight to supervise student-generated reasoning trajectories. However, reference-based guidance may explain a correct solution without addressing why the student's own reasoning fails. This reasoning mismatch between the guidance provided and the correction needed can encourage the student to borrow correct conclusions while leaving its reasoning errors unresolved. Moreover, applying the same hindsight throughout the trajectory risks a distillation trap, where unnecessary constraints on valid reasoning compete with correction of substantive errors. To address these issues, we propose Root-Cause-Guided On-Policy Distillation (RC-OPD), which uses repairs of the student's own reasoning to provide guidance that addresses its specific errors while building on valid progress. For each failed attempt, RC-OPD locates the earliest substantive error, develops a local correction, and uses the corrected intermediate result as an anchor for the valid prefix. An iterative diagnosis--repair--continuation process tests the repairs through student continuation, identifying further errors within a fixed repair budget. For repair chains that reach a correct answer, root--cause--guided distillation uses failure diagnoses and corrective goals to supervise the erroneous segments, while anchor-guided distillation supports the corresponding valid prefixes with reasoning chains leading to the repaired intermediate results. We evaluate RC-OPD across multiple datasets and model scales. Extensive experiments and analyses show that it mitigates reasoning mismatch and the distillation trap, yielding substantial performance gains.
Chenglei Shen, Haoyang Yao, Weijie Yu +3
Gaoling School of Artificial Intelligence, Renmin University of China · School of Software and Microelectronics, Peking University · School of Information Technology and Management, University of International Business and Economics
On-policy self-distillation (OPSD) trains a question-only student with token-level feedback from a teacher given training-only privileged information (PI). OPSD therefore provides dense, on-policy supervision, and is free of a larger external teacher, but its effectiveness rests on how PI is designed and utilized. Our preliminary diagnostics suggest a significant gap between teacher utility and student learnability, where a small fraction of high-disagreement tokens dominate the distillation signal, and short teacher continuations at these positions further expose more explicit PI leakage than transferable correction cues, indicating a strong intent on injecting PI-conditioned shortcuts. We propose Adaptive On-Policy Self-Distillation (AOPSD), which adapts what information the teacher receives and how strongly its feedback influences learning. AOPSD encodes each solution as a reasoning DAG, orders problems by the student's evolving capability, and reveals only the affordable subgraph and its next frontier as PI. For high-disagreement tokens, AOPSD utilizes short teacher continuations as probes to encourage useful guidance while mitigating PI-conditioned shortcuts among teacher supervisions. On HMMT25, AIME24, AIME25, and BRUMo25, AOPSD achieves 72.5% Pass@8, which is 6.7 percentage points above OPSD and 4.2 above the strongest competing baseline while reducing 15 percentage points of training time at lower cost.
Jiacheng Du, Weiwei Xie, Tianyi Du +3
National University of Singapore · Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher, mitigating the distribution mismatch of off-policy supervision. While prior work has primarily trained on problems a student can largely solve, we ask whether OPD can improve a student when training is restricted to problems it currently fails. We evaluate five representative OPD mechanisms with Qwen-3 and Phi-4 models under a unified competition-mathematics protocol. We find that OPD does not improve accuracy on this hard subset, with most mechanisms instead degrading performance despite continued decreases in training KL divergence. All eight configurations show a decline, and the two-sided sign test gives p=0.0078. In contrast, the same pipeline improves accuracy by 4.6 percentage points when training is not concentrated on failures, showing that the failure is conditional on the training regime rather than systematic. These findings suggest that OPD is effective when supervision is dominated by correct student prefixes, but can reinforce existing errors when training is concentrated on problems the student currently fails.
Shuo Yang, Zheyu Zhang, Mahito Sugiyama +1
Technical University of Munich · National Institute of Informatics · Munich Center for Machine Learning