Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

Kaixiang Zhao; Tianrun Yu; Shangzhe Li; Yuxiao Yang; Amanda Hughes; Weitong Zhang; Taylor W. Killian

cs.LGOct 14, 2026

Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

Authors: Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes, Weitong Zhang, Taylor W. Killian

Organizations: Brigham Young University · University of North Carolina at Chapel Hill

Abstract

On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.

Figures & tables

Figure 1: Where the supervision of each method falls after the student errs, and who writes it.

Figure 2: Overview of CriticOPD. The student rewrites an incorrect rollout from the last error that the teacher critiques. The teacher scores the rewrite with the critique, and the student learns from it without the critique.

Figure 3: Teacher (

\Lambda^{T}

) and student (

\Lambda^{S}

) continuation accuracy along (a) correct and (b) incorrect rollouts.

Figure 5: An example of critique-guided rewriting. Left: the original solution, with the critiqued paragraphs marked. Middle: the critique, which quotes each erroneous sentence, gives a fix, and ends with the reference answer. Right: the suffix that the student writes from the last error with the critique.

Figure 6: Recoverability of the teacher and the student on original rollouts (dashed) and critique-guided rewrites (solid), plotted against the position relative to the cut point. On the rewrite, the teacher sees the critique, and the student does not.

Method	AIME24	AIME25	AMC23	MATH500	OlympiadBench	Minerva	Avg
Teacher	62.19	48.12	85.99	94.83	72.26	45.54	68.16
Student: Qwen3-0.6B-Non-Thinking
Student	2.08	1.35	18.98	45.52	16.94	13.69	16.43
SFT @139	4.62 ±0.49	6.84 ±0.12	30.08 ±0.20	58.60 ±0.37	27.24 ±0.20	15.59 ±0.25	23.83 ±0.06
KD @139	4.51 ±0.63	6.15 ±0.73	28.75 ±0.80	57.88 ±0.69	26.62 ±0.52	15.09 ±0.88	23.17 ±0.49
GRPO @80	3.82 ±0.71	3.78 ±1.27	26.46 ±0.92	53.72 ±1.60	23.45 ±1.14	15.12 ±0.71	21.06 ±1.00

Table 1: Results on six mathematical reasoning benchmarks. Entries are means and standard deviations over three seeds, and the subscript after each method gives the selected step. Bold marks the best student at each scale.

Variant	AIME	Avg	$\bm{\Delta}$
CriticOPD	34.79	54.57	–
(a) Content of the feedback
w/o answer	34.58	53.72	$-$ 0.85
w/o critique	28.22	48.67	$-$ 5.90
critique + answer $\rightarrow$ generic	33.85	53.11	$-$ 1.46
(b) Writer of the suffix

Table 2: Ablations on Qwen3-1.7B with seed

42

\Delta

is the change in Avg from CriticOPD, and from CriticOPD without the answer in (b) and (c). AIME averages AIME 2024 and AIME 2025.

Figure 7: Occupancy mismatch and teacher reliability on the states that each method supervises, and the average accuracy of the Qwen3-1.7B student.

Figure 8: (a) Advantage per token of two continuations of

64

tokens at the cut point, which either continue or correct the error, under the scoring of OPD and of CriticOPD. Numbers give correct minus continue. (b) Success rate of the student of CriticOPD minus that of OPD from the same prefixes on OlympiadBench, with 95% confidence bands.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Expected accuracy of the teacher and the Qwen3-0.6B student from prefixes of the student on (a) correct and (b) incorrect rollouts.

Figure 10: (a) Outcomes of the continuations of the teacher and (b) fraction of the listed errors that are real, for the Qwen3-0.6B student.

Figure 11: Success rates of the teacher and the Qwen3-0.6B student around the cut point.

Figure 12: Token positions of the prefixes that each method supervises (colored) and of the inference prefixes (gray), with the total variation distance in each title. The dotted line marks the training limit of

16{,}384

tokens.

Figure 13: Hidden states of the student on the prefixes that each method supervises, projected onto the first two principal components of the inference states. The contour encloses $95\%$ of the inference states, and each title gives the classifier bound on the total variation distance.

Setting	Value
Training
Prompts per step	$128$
Rollouts per prompt	$1$ (GRPO: $8$ )
Optimizer updates per step	$1$
Learning rate	$10^{-6}$ , constant, no warmup
Maximum prompt length	$2{,}048$

Appendix

Table 3: Hyperparameters of all methods.

Figure 14: Training cost of the Qwen3-1.7B students. (a) A100 GPU-hours until the selected step, including the generation of offline data. (b) Seconds per training step, averaged over five steps.

Method	AIME24	AIME25	AMC23	MATH500	OlympiadBench	Minerva	Avg
Teacher	73.54	60.62	92.24	96.22	77.35	47.43	74.57
Student: Olmo-3-7B-Instruct-SFT, teacher: Olmo-3.1-32B-Instruct
Student	6.15	4.17	30.50	64.80	30.09	29.09	27.47
SFT @40	31.56	26.15	59.45	79.50	51.91	31.66	46.70
KD @80	35.42	30.83	72.33	86.83	59.48	33.27	53.03
GRPO @120	10.00	7.60	37.46	69.85	36.28	30.70	31.98

Appendix

Table 4: Results with OLMo-3 models on six mathematical reasoning benchmarks with seed

42

. The subscript after each method gives the step, and bold marks the best student.

Method	MMLU-Pro	MMLU	ARC-C	OpenBookQA	HumanEval+	MBPP+	Avg
Teacher	69.75	80.09	94.94	93.95	82.00	74.70	82.57
Student	38.10	55.24	77.69	74.10	62.50	56.50	60.69
SFT	50.80	63.29	84.41	79.70	63.10	55.40	66.12
KD	50.85	63.78	84.28	79.30	64.00	57.10	66.55
GRPO	51.41	61.49	81.68	78.75	63.70	58.50	65.92
OPD	56.61	67.14	87.07	83.80	66.30	61.90	70.47

Appendix

Table 5: Results outside mathematics for Qwen3-1.7B. Code uses EvalPlus pass@1 on the extended tests.

Variant	AIME24	AIME25	AMC23	MATH500	Olympiad	Minerva	Avg
CriticOPD	40.52	29.06	70.18	89.40	60.59	37.64	54.57
(a) Content of the feedback
w/o answer	41.56	27.60	67.43	88.75	59.04	37.91	53.72
w/o critique	34.48	21.96	60.33	85.55	54.83	34.89	48.67
critique + answer $\rightarrow$ generic	40.00	27.71	67.39	88.05	58.91	36.63	53.11
(b) Writer of the suffix

Appendix

Table 6: Per-benchmark results of the ablations on Qwen3-1.7B with seed

42

. The rows follow Table 2 .

Figure 15: Sensitivity of CriticOPD on the Qwen3-1.7B student with seed

42

. The star marks the default, and the numbers in (b) give the share of truncated critiques.

Figure 16: Training dynamics, averaged over seeds and smoothed over five steps.

Figure 17: Average over six benchmarks when responses longer than the budget count as incorrect, over three seeds of Qwen3-1.7B.

Method	AIME length	Avg@16k	Avg@32k
OPD	$8{,}487$	49.99	50.08
FastOPD	$11{,}910$	50.27	52.55
CriticOPD	$12{,}457$	50.32	54.65
scoring w/o feedback	$12{,}065$	50.62	52.26
critique $\rightarrow$ generic	$12{,}051$	50.86	52.87

Appendix

Table 7: Length-controlled comparison on Qwen3-1.7B, averaged over three seeds.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 488 papers

Correcting or Continuing? What On-Policy Distillation Learns from Student Errors
Oct 13, 2026Shuo Yang, Zheyu Zhang, Mahito Sugiyama +1Language Model Distillation On-Policy Distillation
Technical University of Munich · National Institute of Informatics · Munich Center for Machine Learning
Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation
Oct 2, 2026Xiang Chen, Futao Su, Kong Wang +2Language Model Distillation On-Policy Distillation
Tongji University · Xiaohongshu · Shanghai Jiao Tong University +1
Pass the Baton: Trajectory-Relayed On-Policy Distillation
Jul 28, 2026Haolei Xu, Xiaowen Xu, Haiwen Hong +5Language Model Distillation On-Policy Distillation
1Zhejiang University · 2Yuvion Team, Alibaba Group

Oct 13, 2026J/K move · Enter open · S save

Correcting or Continuing? What On-Policy Distillation Learns from Student Errors

Shuo Yang, Zheyu Zhang, Mahito Sugiyama, Gjergji Kasneci

Technical University of Munich · National Institute of Informatics · Munich Center for Machine Learning

Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher, mitigating the distribution mismatch of off-policy supervision. While prior work has primarily trained on problems a student can largely solve, we ask whether OPD can improve a student when training is restricted to problems it currently fails. We evaluate five representative OPD mechanisms with Qwen-3 and Phi-4 models under a unified competition-mathematics protocol. We find that OPD does not improve accuracy on this hard subset, with most mechanisms instead degrading performance despite continued decreases in training KL divergence. All eight configurations show a decline, and the two-sided sign test gives p=0.0078. In contrast, the same pipeline improves accuracy by 4.6 percentage points when training is not concentrated on failures, showing that the failure is conditional on the training regime rather than systematic. These findings suggest that OPD is effective when supervision is dominated by correct student prefixes, but can reinforce existing errors when training is concentrated on problems the student currently fails.

Abstract

On-Policy Distillation

Explore similar work

Correcting or Continuing? What On-Policy Distillation Learns from Student Errors

Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation

Pass the Baton: Trajectory-Relayed On-Policy Distillation