A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning

Mayug Maniparambil; Ayush Rai; Ekansh Verma; Noel E. O'Connor; Fergal Reid

cs.AIOct 9, 2026

A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning

Authors: Mayug Maniparambil, Ayush Rai, Ekansh Verma, Noel E. O'Connor, Fergal Reid

Organizations: Dublin City University

Abstract

On-policy self-distillation promises to replace the sparse outcome signal of verifiable-reward RL with dense token-level credit: a teacher conditioned on the reference solution scores the student's own rollout, and the resulting distributional shift is distilled back into the policy. We investigate whether this scoring signal actually contains corrective credit. Using gold error spans in student rollouts, we test the two signals consumed by self-distillation objectives, the full-distribution KL and the token log-ratio, for two properties: whether they localize errors and whether they push probability away from erroneous student tokens. They exhibit neither of these properties nor can they reliably distinguish correct from incorrect rollouts within the same problem. We argue that this failure is structural: locating an error in a reasoning trace is itself a search problem, while a scoring teacher is never explicitly asked to perform that search. The same model, with the same privileged information, localizes errors substantially better when allowed to generate a critique. This forms the motivation for critique-conditioned self-distillation: first generate a critique of the current rollout, then condition the scoring teacher on that critique. The resulting signal localizes errors, is corrective within error spans, and shifts probability toward tokens that initiate backtracking such as Wait'', But'', and ``Actually''. We further show that the critique is useful both at test time, where resuming a wrong rollout with the critique in context converts math problems the model cannot solve by sampling, and in cold-start GRPO, where it provides an advantage signal on prompts for which verifiable rewards alone provide none.

Figures & tables

Figure 1: The reference-conditioned teacher barely reacts at the student’s errors. Content KL (top) and

\Delta\log p

of the student’s token (bottom) around gold error sentences (blue) and clean sentences (orange), aligned to sentence onset and end; Qwen3-1.7B scoring its own rollouts. Bands: 95% CI.

Qwen3-1.7B (355 / 516 / 794 problems)
	K&K		L5-7		L7-10
conditioning	loc	corr	loc	corr	loc	corr
reference	$\mathbf{-0.17}$	$+0.11$	$+0.14$	$-0.14$	$+0.07$	$\mathbf{+0.17}$
critique, oracle	$\mathbf{+0.75}$	$\mathbf{-0.62}$	$\mathbf{+0.99}$	$\mathbf{-0.90}$	$\mathbf{+0.78}$	$\mathbf{-0.54}$
critique, self	$+0.16$	$-0.07$	$\mathbf{+0.25}$	$-0.14$	$\mathbf{+0.18}$	$-0.07$
Olmo-3-7B-Think (197 / 398 / 631 problems)

Table 1: Error-span Z at the first gold error, for three students each scoring its own rollouts. Localization: log content KL (positive = reacts at the error); correction:

\Delta\log p

of the student’s token (negative = corrective). Bold: 95% CI excludes 0.

Figure 2: The reference-conditioned signal cannot tell correct rollouts from wrong ones. Pooled AUC of trace-mean

\Delta\log p

, correct vs. wrong rollouts, by privileged context and teacher.

	localized $\pm 5$			quote-match			parse
critic	K&K	L5-7	L7-10	K&K	L5-7	L7-10	K&K	L5-7	L7-10
gpt-5.2 (oracle)	57	62	67	98	96	95	100	100	100
Qwen3-8B	26	22	27	96	82	83	99	86	85
Qwen3-1.7B (self)	20	18	21	77	64	68	90	83	83

Table 2: Generating finds the errors that scoring cannot. Percent of wrong rollouts with a critique quote within five sentences of a gold error (localized

\pm 5

), quotes found verbatim in the rollout (quote-match), and critiques in the required format (parse).

Figure 3: With a critique in context, the same teacher reacts at the errors. Same layout as Figure 1 , with gpt-5.2 critiques in place of the reference; Qwen3-1.7B scoring its own rollouts. The error curve leaves its control on every panel.

Table 6

cut	context at the cut	all (39)	unsolved by sampling (19)
fixed depth	cue	10.9	4.3
critique’s flagged sentence	cue + critique item	13.4	$\mathbf{9.3}$

Table 5: At test time the critique’s content converts math problems the model cannot solve by sampling. Olmo-3-32B-Think resumes its own wrong L7-10 rollouts from a cut, eight samples per rollout; sample accuracy in percent on all problems and on the problems unsolved in eight samples. Bold = 95% CI of the paired difference to the control excludes 0.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Cumulative solved training rollouts on four certified-cold Knights & Knaves puzzles, critique-conditioned advantage against the GRPO control. Olmo-3-7B-Think as student, critic, and teacher.

method	critic	in-domain	9	10	11	OOD
base model	–	0.666	0.517	0.415	0.275	0.403
GRPO	–	0.700	0.595	0.532	0.400	0.509
GRPO + critique	gpt-5-mini	0.744	0.680	0.568	0.445	0.564
RLCSD	–	0.745	0.688	0.583	0.430	0.567
RLCSD + critique	gpt-5-mini	0.754	0.698	0.623	0.497	0.606
RLCSD + critique	self	0.735	0.693	0.595	0.490	0.593

Appendix

Table 6: Accuracy after 100 steps on Olmo-3-7B-Instruct. In-domain is Knights & Knaves with 4–8 people, OOD the mean over 9, 10 and 11 people. The critique term is confined to the quoted sentence and the 50 tokens after it.

critic	arm	critiqued rollouts	prompt tokens	completion tokens	step time	share
gpt-5-mini (API)	RLCSD + critique	16	5,600	2,970	255 s	21%
gpt-5-mini (API)	GRPO + critique	18	4,960	2,890	256 s	22%
self (in-engine)	RLCSD + critique	18	6,770	175	219 s	6%

Appendix

Table 7: Cost of the critique pass per training step, mean over 100 steps, in the regime of Table 6 . Tokens are per critique.

conditioning	teacher	task	loc	95% CI	corr	95% CI	$n$
reference	1.7B	K&K	$-0.17$	$[-0.33,-0.01]$	$+0.11$	$[-0.04,+0.26]$	317
		L5-7	$+0.14$	$[-0.01,+0.29]$	$-0.14$	$[-0.30,+0.02]$	430
		L7-10	$+0.07$	$[-0.08,+0.19]$	$+0.17$	$[+0.07,+0.28]$	690
reference	8B	K&K	$+0.13$	$[-0.03,+0.29]$	$-0.16$	$[-0.35,+0.02]$	317
		L5-7	$+0.05$	$[-0.11,+0.20]$	$-0.06$	$[-0.23,+0.10]$	430
		L7-10	$-0.05$	$[-0.18,+0.08]$	$+0.06$	$[-0.06,+0.19]$	690

Appendix

Table 8: Error-span Z at the first gold error with 95% confidence intervals, Qwen3-1.7B rollouts, scale-up sample, every conditioning and teacher. Rows: the reference solution or a critique placed in the scoring teacher’s context; teacher Qwen3-1.7B (the student itself) or Qwen3-8B; critiques by gpt-5.2 (oracle), by the 1.7B student on its own rollouts, or by Qwen3-8B. Localization on log content KL (positive = the distribution reacts at the error); correction on

\Delta\log p

of the student’s token (negative = corrective).

n

= rollouts entering the cell.

task	teacher	reference alone	critique placebo alone	sign% loc	sign% corr
K&K	1.7B	$-0.13^{*}$	$-0.06$	68.1	55.5
K&K	8B	$-0.15^{*}$	$-0.08$	72.2	55.8
L5-7	1.7B	$-0.15$	$-0.07$	74.8	55.6
L5-7	8B	$-0.12$	$-0.14$	74.8	56.7
L7-10	1.7B	$-0.27^{*}$	$-0.29^{*}$	67.1	46.0
L7-10	8B	$-0.25^{*}$	$-0.27^{*}$	68.5	49.6

Appendix

Table 9: Negative controls and reliability, Qwen3-1.7B rollouts, scale-up sample. Left: error-span Z of each privileged context alone, without subtracting its placebo (reference alone; a donor critique alone), which should sit at zero. Right: per-rollout sign consistency of the oracle-critique signal, the share of rollouts whose first error window departs from its own control windows in the expected direction, on each channel. ∗ : 95% CI excludes 0.

Figure 5: Reference-conditioned signal on all three tasks. Content KL (top row of each block) and

\Delta\log p

of the student’s token (bottom row) around gold error sentences (blue) and clean-prefix sentences (orange), aligned to the sentence onset (upper block) and to the sentence end (lower block). Qwen3-1.7B scoring its own rollouts with the reference solution in context, minus the shuffled-reference placebo; scale-up sample.

Figure 6: Critique-conditioned signal on all three tasks. Same quantities, alignments, and rollouts as the figure above, with gpt-5.2’s critique of the rollout in the teacher’s context in place of the reference, minus the donor-critique placebo. Teacher Qwen3-1.7B.

Figure 7: Reference-conditioned signal with Qwen3-8B as the scoring teacher on the Qwen3-1.7B rollouts. Layout as Fig. 1 : content KL (top) and

\Delta\log p

of the student’s token (bottom), Knights & Knaves and DeepMath 5-7, onset- and end-aligned; reference minus shuffled-reference placebo; scale-up sample.

Figure 8: Critique-conditioned signal with Qwen3-8B as the scoring teacher on the Qwen3-1.7B rollouts, gpt-5.2 critiques. Layout as Fig. 3 : content KL (top) and

\Delta\log p

of the student’s token (bottom), Knights & Knaves and DeepMath 5-7, onset- and end-aligned; critique minus donor-critique placebo; scale-up sample.

Figure 9: Critique-conditioned signal with the student’s own critiques: Qwen3-1.7B writes the critique, then scores its own rollout with that critique in context. Layout as Fig. 3 : content KL (top) and

\Delta\log p

of the student’s token (bottom), Knights & Knaves and DeepMath 5-7, onset- and end-aligned; critique minus donor-critique placebo; scale-up sample. Note the smaller vertical scale.

task	teacher	trace-mean KL	trace-mean $\Delta\log p$
K&K	1.7B	0.598 [0.451, 0.740]	0.396 [0.260, 0.536]
K&K	8B	0.435 [0.295, 0.582]	0.630 [0.490, 0.759]
L5-7	1.7B	0.609 [0.448, 0.780]	0.487 [0.329, 0.634]
L5-7	8B	0.495 [0.341, 0.668]	0.529 [0.364, 0.686]
L7-10	1.7B	0.602 [0.490, 0.709]	0.425 [0.317, 0.527]
L7-10	8B	0.566 [0.456, 0.679]	0.481 [0.368, 0.596]

Appendix

Table 10: Within-problem pairwise AUC of the reference-conditioned trace-mean signal, correct vs. wrong sibling rollouts of the same problem, with 95% CI. 0.5 = chance; no cell excludes it.

context	1.7B teacher	8B teacher
length only	.54 / .56	.54 / .56
reference (dense)	.57 / .58	.65 / .64
permuted reference	.60 / .60	.66 / .71
irrelevant reference (placebo)	.57 / .58	.70 / .58
answer only (sparse)	.69 / .75	.78 / .85
critique	.71 / .66	.68 / .69

Appendix

Table 11: Trained probe: out-of-fold AUC of an L2 logistic regression on per-trace summaries of each privileged context’s signal, DeepMath L5-7 / L7-10, both teachers. The reference block never beats its content-free placebos.

critique	task	loc	95% CI	corr	95% CI
quote + reason, 1.7B teacher	K&K	$+0.75$	$[+0.54,+0.97]$	$-0.62$	$[-0.86,-0.40]$
	L5-7	$+0.99$	$[+0.82,+1.16]$	$-0.90$	$[-1.14,-0.65]$
	L7-10	$+0.78$	$[+0.63,+0.92]$	$-0.54$	$[-0.71,-0.35]$
reason only, 1.7B teacher	K&K	$+0.48$	$[+0.31,+0.67]$	$-0.47$	$[-0.67,-0.26]$
	L5-7	$+0.75$	$[+0.60,+0.89]$	$-0.58$	$[-0.81,-0.36]$
	L7-10	$+0.61$	$[+0.48,+0.75]$	$-0.30$	$[-0.45,-0.16]$

Appendix

Table 12: Ablating the critique, with 95% confidence intervals: error-span Z when the gpt-5.2 critique in the teacher’s context is complete (quote + reason), stripped of its quotes (reason only), or stripped of its reasons (quote only). Qwen3-1.7B rollouts, scale-up sample; scoring teacher Qwen3-1.7B in the upper block and Qwen3-8B in the lower. Critique minus donor-critique placebo throughout.

student	signal	ablation	K&K	L5-7 / L7-10
Qwen3-1.7B	localization	remove quote	$+0.30^{*}$	$+0.35^{}$ / $+0.19^{}$
		remove reason	$-0.00$	$+0.09$ / $+0.02$
	correction	remove quote	$-0.14$	$-0.39^{}$ / $-0.27^{}$
		remove reason	$-0.35^{*}$	$-0.44^{}$ / $-0.32^{}$
Olmo-3-7B-Think	localization	remove quote	$+0.28^{*}$ $[+0.15,+0.42]$	$+0.27^{}$ $[+0.19,+0.34]$ / $+0.22^{}$ $[+0.16,+0.28]$
		remove reason	$+0.09$ $[-0.06,+0.25]$	$+0.05$ $[-0.04,+0.14]$ / $+0.01$

Appendix

Table 13: Paired per-rollout ablation drops with 95% CI: error-span Z of the complete critique minus that of the ablated critique on the same rollouts, gpt-5.2 critiques. Upper block: Qwen3-1.7B rollouts and teacher, scale-up sample. Lower block: Olmo-3-7B-Think rollouts and teacher, big set. A positive drop on localization or a negative drop on correction means the ablation removed signal; ∗ : CI excludes 0.

	Qwen3-1.7B (scale-up)	Olmo-3-7B-Think (big set)	Olmo-3-32B-Think (minimal set)
student = teacher = self-critic	yes (plus 8B teacher / critic)	yes	yes
rollout cap	16,384	32,768	32,768
sampling	temp 1.0, top-p 0.95, top-k 20, $n{=}8$	same	same
gold-span grader	deepseek-reasoner / v4-flash	claude-sonnet-5	claude-sonnet-5
juries	gpt-5.2 + claude-sonnet-5	same (gpt-5.2 independent)	same
oracle critiques	gpt-5.2, same prompt	gpt-5.2, same prompt	gpt-5.2, same prompt

Appendix

Table 14: The three error-span studies side by side: models, rollout sampling, gold-span pipeline, reference solutions, and the number of problems and rollouts that enter error-span Z on each task (Knights & Knaves / DeepMath 5-7 / DeepMath 7-10).

student	conditioning	task	loc	95% CI	corr	95% CI	$n$
Olmo-3-7B	reference	K&K	$+0.20$	$[+0.01,+0.39]$	$-0.00$	$[-0.18,+0.18]$	271
		L5-7	$+0.11$	$[-0.03,+0.25]$	$-0.08$	$[-0.23,+0.08]$	584
		L7-10	$-0.01$	$[-0.11,+0.09]$	$-0.03$	$[-0.14,+0.08]$	958
	critique, oracle	K&K	$+0.63$	$[+0.39,+0.87]$	$-0.29$	$[-0.55,-0.03]$	271
		L5-7	$+0.89$	$[+0.72,+1.07]$	$-0.59$	$[-0.78,-0.40]$	584
		L7-10	$+0.88$	$[+0.75,+1.01]$	$-0.49$	$[-0.63,-0.33]$	958

Appendix

Table 15: Error-span Z at the first gold error with 95% confidence intervals for the two Olmo-3 students, each scoring its own rollouts: Olmo-3-7B-Think on the big set and Olmo-3-32B-Think on the minimal set. Rows: reference solution in context; gpt-5.2 critique in context; the student’s own critique in context. Localization on log content KL, correction on

\Delta\log p

of the student’s token;

n

= rollouts entering the cell.

student	critic	task	$\pm 2$	$\pm 5$	$\pm 10$	med. dist	quote-match	parse
Qwen3-1.7B	gpt-5.2 (oracle)	K&K / L5-7 / L7-10	48 / 54 / 59	57 / 62 / 67	65 / 69 / 73	3 / 2 / 1	98 / 96 / 95	100
	Qwen3-8B		15 / 13 / 16	26 / 22 / 27	40 / 31 / 37	17 / 18 / 13	96 / 82 / 83	99 / 86 / 85
	Qwen3-1.7B self, relaxed		9 / 12 / 13	20 / 18 / 21	33 / 26 / 31	16 / 23 / 16	77 / 64 / 68	90 / 83 / 83
	Qwen3-1.7B self, strict		4 / 8 / 9	9 / 13 / 15	16 / 19 / 22	12 / 16 / 13	33 / 42 / 45	43 / 54 / 53
Olmo-3-7B	gpt-5.2 (oracle)		31 / 37 / 46	39 / 45 / 55	43 / 51 / 64	16 / 8 / 3	97 / 89 / 94	100
	Olmo-3-7B self, relaxed		3 / 6 / 7	7 / 9 / 12	10 / 12 / 17	71 / 61 / 42	74 / 58 / 61	92 / 84 / 87

Appendix

Table 16: Critique localization gate for the two Olmo-3 students (oracle and self-critic) and the full window sweep for the Qwen3-1.7B rollouts. Percent of wrong rollouts with a gold span for which a quote lands within

w

sentences of a gold error sentence; lenient tier; relaxed parse for self-critics. Olmo-3 rollouts run 800–1,500 sentences against 450–550 for Qwen, so absolute rates are comparable only within a student.

	Qwen2.5-7B-Instruct		Qwen2.5-Math-7B
conditioning	loc	corr	loc	corr
reference	$+0.18$	$-0.34$	$+0.27$	$-0.26$
critique, oracle	$\mathbf{+1.25}$	$\mathbf{-0.55}$	$\mathbf{+0.80}$	$+0.13$

Appendix

Table 17: On human-labelled error steps, the reference is flat and the critique reacts. ProcessBench, each solution teacher-forced through the Qwen2.5 model that wrote it. Error-step Z as in Table 1 : loc = log content KL (positive = reacts at the labelled step), corr =

\Delta\log p

of the student’s token (negative = corrective), hint minus placebo. Bold = 95% CI excludes 0. The critique is written by gpt-5 without the label.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 154 papers

Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation
Aug 10, 2026Xuan-Phi Nguyen, Zeyu Leo Liu, Yang Li +4On-Policy Self-Distillation Token-Level Credit Assignment
Salesforce AI Research
Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
Oct 14, 2026Kaixiang Zhao, Tianrun Yu, Shangzhe Li +4On-Policy Distillation Knowledge Distillation
Brigham Young University · University of North Carolina at Chapel Hill
Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
Jun 17, 2026Siyi Gu, Jialin Chen, Sophia Zhou +2RL for Language Model Reasoning Language Model Distillation
Yale University

Aug 10, 2026J/K move · Enter open · S save

Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation

Xuan-Phi Nguyen, Zeyu Leo Liu, Yang Li, Shrey Pandit+3

Salesforce AI Research

On-Policy Self-Distillation Token-Level Credit Assignment Step-Level Credit Assignment in RL Self-Distillation Learning Using Privileged Information

Open paper AskPDF arXiv

On-policy self-distillation aims to improve upon reinforcement learning from verifiable rewards (RLVR) by providing token-level scores derived from privileged information, such as reference solutions or critic feedback. These scores are treated as estimates of token-level action values, yet they answer a fundamentally different question: how the model's prediction changes when its input context is enriched, rather than how the expected outcome changes when a token is changed. We examine this gap along three dimensions: (i) whether the token-level score tracks task success; (ii) whether feedback generated from the same rollout causes the score to reflect agreement with its own description, and whether using feedback from other rollouts in the group mitigates this self-referential effect; and (iii) what behavior the resulting training objective actually reinforces. In experiments on AIME 2025, the implemented score distinguishes correct from incorrect rollouts at approximately chance level (AUC=0.505); using feedback from a different rollout does not consistently improve this discrimination; and all training configurations achieve only 24.2-33.9% Avg@4, compared with 64.2% for outcome-only GRPO. Moreover, the highest-entropy token decile accounts for 57-71% of the total absolute token-advantage mass, despite the score being least informative about reasoning quality in this regime. By contrast, similar experiments on SciKnowEval Biology improves held-out Avg@8 by 28.0%, while its corresponding trajectory scores achieve AUCs of 0.81-0.92. Together, these results suggest that dense credit assignment through distillation can be effective when its likelihood-based scores are empirically validated as meaningful proxies for outcome-relevant credit. When this alignment does not hold, however, the resulting supervision can fail to generalize and may substantially underperform outcome-based RL.

Abstract

On-Policy Self-Distillation

Explore similar work

Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation

Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation