Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillation

Rui Li; Liyang He; Zheng Zhang; Zhenya Huang; Linbo Zhu; Qi Liu

cs.LGOct 2, 2026

Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillation

Authors: Rui Li, Liyang He, Zheng Zhang, Zhenya Huang, Linbo Zhu, Qi Liu

Organizations: University of Science and Technology of China · Nanyang Technological University

Abstract

On-policy self-distillation (OPSD) supplies dense token-level feedback on trajectories sampled from the student's own policy, a richer training signal than the outcome-level rewards of reinforcement learning. This feedback comes from a teacher conditioned on a full reference solution unavailable to the student. The reference solution specifies the target but not how to move from the student's current error toward it, creating a solution-conditioned shortcut risk. We introduce AIR-OPD, an adaptive iterative repair framework for on-policy distillation that provides error-to-repair supervision. Given a failed response, a guidance generator synthesizes repair guidance for the current error. The student samples an on-policy retry with this guidance. If the retry remains incorrect, the generator produces new repair guidance for the newly observed error. At each round, a fixed teacher receives the guidance as privileged context and supervises the student on an error-aligned region of its latest failed response. Outcome-aware stage weighting favors early repair stages and credits stages whose immediate retry passes verification. We train AIR-OPD on the DAPO-Math-17K dataset and evaluate on AIME24, AIME25, and HMMT25, alongside out-of-distribution tests on MMLU-Pro and GPQA. We examine two guidance sources, self-guidance from the current student policy and external guidance from a larger model. For both Qwen3-4B and Qwen3-8B, AIR-OPD attains the best mathematical-reasoning averages, improving over the strongest baseline by up to 3.6 points, while preserving base-model performance on the out-of-distribution benchmarks.

Figures & tables

Figure 1: Overview of Air-OPD . When an on-policy rollout fails verification, the guidance generator

\mathcal{G}

quotes the first flawed step as an error anchor

a_{t}

and synthesizes repair guidance

g_{t}

. A frozen teacher conditioned on

g_{t}

then supervises the student over the error-aligned repair region

R_{t}

, and the student retries with the guidance in context. A verified retry ends the trajectory, and outcome-aware stage weighting combines the stage losses with weights

w_{t}\propto\gamma^{t}\rho^{s_{t}}

Qwen3-4B
	Mathematical Reasoning				Out-of-Distribution
Method	AIME24	AIME25	HMMT25	Math Avg.	MMLU-Pro	GPQA	OOD Avg.
Base model	75.0	66.4	42.2	61.2	51.5	62.9	57.2
SFT	70.6	61.4	42.8	58.2	43.1	52.0	47.6
GRPO	75.6	67.5	43.9	62.3	51.3	62.4	56.8
OPSD	76.4	69.4	43.3	63.1	50.4	60.2	55.3

Table 1: Main results on mathematical-reasoning (Avg@12) and out-of-distribution (Avg@4) benchmarks. Both Air-OPD variants surpass all baselines on Math Avg. at each scale while preserving base-model performance on the OOD group. Self-G / External-G generate repair guidance with the current student policy and Qwen3.5-27B, respectively. Bold / underlined : best/second-best within each block.

Sample selection (OPSD)
Setting	Math Avg.
Air-OPD	65.8
All rollouts	63.1
Failed rollouts only	62.5
Supervision region
w/o error alignment	64.7

Table 2: Ablation results (Qwen3-4B, Self-G ). We test failure-only sample selection for OPSD and ablate the supervision region and the stage-weighting components of Air-OPD .

Figure 2: Effect of the repair budget

K

(Qwen3-4B). Math Avg. (left axis) rises through the default

K=3

and then flattens, while each variant’s number of supervised stages (dashed, right axis; % of the samples OPSD supervises) grows with diminishing increments.

Figure 3: Training dynamics on the mathematical benchmarks (Qwen3-4B). Both Air-OPD variants lead OPSD and GRPO on the three-benchmark average from the first checkpoint onward. Curves show Avg@12 at checkpoints saved every 20 optimizer steps, and dashed lines mark the base model.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Guidance generator, first repair round. The template diagnoses the initial failed response $y_{0}$ under the privileged reference solution and final answer.

Panel of Figure 5 — Figure 5: Guidance generator, later repair rounds. The template replaces the current guidance after a retry that is still incorrect, and additionally receives the guidance the student has already seen.

Optimization and adaptation
Parameter	SFT	GRPO	OPSD	Air-OPD
Optimizer	AdamW	AdamW	AdamW	AdamW
Learning rate	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$
Effective batch size	32	32	32	32
Training precision	BF16	BF16	BF16	BF16
LoRA rank / alpha	64 / 128	64 / 128	64 / 128	64 / 128

Appendix

Table 3: Training settings for SFT, GRPO, OPSD, and Air-OPD . Lengths are measured in tokens; the SFT limit applies to the full supervised sequence. A dash denotes a setting that is not applicable. For Air-OPD , one rollout is sampled per dialogue state: the initial response can be followed by at most

K-1

retries.

Parameter	Value
Maximum new tokens	38,912
Maximum model length (tokens)	40,960
Thinking mode	Enabled
Sampling temperature	1.0
Top- $p$	0.95
Top- $k$	–

Appendix

Table 4: Inference settings for mathematical-reasoning evaluation. The base model and all trained methods use these settings on AIME24, AIME25, and HMMT25, without repair guidance.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 157 papers

Learning from Repaired Reasoning: Root-Cause-Guided On-Policy Distillation
Oct 2, 2026Chenglei Shen, Haoyang Yao, Weijie Yu +3Language Model Distillation On-Policy Self-Distillation
Gaoling School of Artificial Intelligence, Renmin University of China · School of Software and Microelectronics, Peking University · School of Information Technology and Management, University of International Business and Economics
Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning
Sep 26, 2026Jiacheng Du, Weiwei Xie, Tianyi Du +3On-Policy Self-Distillation CoT Distillation
National University of Singapore · Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
Correcting or Continuing? What On-Policy Distillation Learns from Student Errors
Oct 13, 2026Shuo Yang, Zheyu Zhang, Mahito Sugiyama +1Language Model Distillation On-Policy Distillation
Technical University of Munich · National Institute of Informatics · Munich Center for Machine Learning

Oct 2, 2026J/K move · Enter open · S save

Learning from Repaired Reasoning: Root-Cause-Guided On-Policy Distillation

Chenglei Shen, Haoyang Yao, Weijie Yu, Song Jin+2

Gaoling School of Artificial Intelligence, Renmin University of China · School of Software and Microelectronics, Peking University · School of Information Technology and Management, University of International Business and Economics

Language Model Distillation On-Policy Self-Distillation CoT Distillation LLM Self-Correction

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) uses reference solutions as privileged hindsight to supervise student-generated reasoning trajectories. However, reference-based guidance may explain a correct solution without addressing why the student's own reasoning fails. This reasoning mismatch between the guidance provided and the correction needed can encourage the student to borrow correct conclusions while leaving its reasoning errors unresolved. Moreover, applying the same hindsight throughout the trajectory risks a distillation trap, where unnecessary constraints on valid reasoning compete with correction of substantive errors. To address these issues, we propose Root-Cause-Guided On-Policy Distillation (RC-OPD), which uses repairs of the student's own reasoning to provide guidance that addresses its specific errors while building on valid progress. For each failed attempt, RC-OPD locates the earliest substantive error, develops a local correction, and uses the corrected intermediate result as an anchor for the valid prefix. An iterative diagnosis--repair--continuation process tests the repairs through student continuation, identifying further errors within a fixed repair budget. For repair chains that reach a correct answer, root--cause--guided distillation uses failure diagnoses and corrective goals to supervise the erroneous segments, while anchor-guided distillation supports the corresponding valid prefixes with reasoning chains leading to the repaired intermediate results. We evaluate RC-OPD across multiple datasets and model scales. Extensive experiments and analyses show that it mitigates reasoning mismatch and the distillation trap, yielding substantial performance gains.

Abstract

On-Policy Self-Distillation

Explore similar work

Learning from Repaired Reasoning: Root-Cause-Guided On-Policy Distillation

Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning

Correcting or Continuing? What On-Policy Distillation Learns from Student Errors