Calibrating Teacher--Student Discrepancy for On-Policy Distillation

Qiangqiang He; Jin Li; MingCai Chen

cs.AISep 18, 2026

Calibrating Teacher--Student Discrepancy for On-Policy Distillation

Authors: Qiangqiang He, Jin Li, MingCai Chen

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · College of Software Engineering, Southeast University, Nanjing, China · Nanjing University of Posts and Telecommunications, Nanjing, China

Abstract

On-policy distillation (OPD) improves reasoning models by learning the token-level discrepancy between a stronger teacher and an on-policy student. However, this discrepancy does not purely reflect the capability gap between the teacher and the student: it also contains deviations arising from the teacher itself, which are consequently mixed into the observed teacher--student discrepancy and indiscriminately learned by standard OPD during training. This issue is further exacerbated by privileged OPD, where privileged information induces larger teacher-side likelihood shifts, thereby encouraging the student to learn more of the teacher's own deviation. We introduce Calibrated On-Policy Distillation (Cal-OPD), which estimates the teacher's self-deviation region through positive and negative privileged interventions and calibrates the original teacher--student discrepancy by retaining only the component that lies beyond this region. Experiments on mathematical reasoning benchmarks show that, while retaining only about 52--65% of the original teacher--student discrepancy as the optimization signal, Cal-OPD consistently outperforms standard OPD and its variants across model scales.

Figures & tables

Figure 1: Conceptual comparison of standard OPD, privileged OPD, and Cal-OPD. Cal-OPD removes the TSD-explained discrepancy and retains only the residual beyond the TSD region.

Task-Agnostic Instructions
Symbol	Contextual Intervention
$c_{\mathrm{inst}}^{\mathrm{pos}}$	Please reason through the problem carefully and thoroughly . Verify intermediate steps and provide a complete, rigorous solution.
$c_{\mathrm{inst}}^{\mathrm{neg}}$	Please solve the problem quickly and directly . Avoid unnecessary elaboration or extensive verification and reach the final answer efficiently.
Evaluative Feedback
$c_{\mathrm{eval}}^{\mathrm{pos}}$	A gold-standard verifier has judged that the following solution reaches the correct final answer . The reasoning is rigorous, coherent, and mathematically sound .
$c_{\mathrm{eval}}^{\mathrm{neg}}$	A gold-standard verifier has judged that the following solution does not reach the correct final answer . The reasoning is flawed, incoherent, and mathematically unreliable .

Table 1: Contextual interventions used to probe TSD. The shared problem context

x

is omitted for brevity. For answer- and solution-level interventions, the positive and negative variants share the same template and differ only in the inserted privileged content.

Figure 2: TSD across contextual interventions. Significant TSD emerges even under task-agnostic interventions, and the affected token positions are largely retained under richer privileged contexts.

Figure 3: Consistency of TSD under contrasting interventions. TSD occurs at overlapping token positions, remains directionally aligned, and is dominated by a shared common-mode component.

Highest $\rho_{\tau}$
#	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	#	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	#	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$
1	maybe	97.4	0.59	7	seems	93.6	0.41	13	wait	90.0	0.42
2	however	97.4	0.63	8	another	92.9	0.50	14	check	89.6	0.28
3	therefore	95.9	0.46	9	since	91.8	0.44	15	let	89.5	0.50
4	consider	95.4	0.38	10	try	91.8	0.40	16	here	89.4	0.36
5	altern.	94.3	0.58	11	think	91.6	0.40	17	because	89.4	0.40

Table 2: Token forms with the highest and lowest significant-TSD rates

\rho_{\tau}(v)

under

c_{\mathrm{sol}}^{\mathrm{pos}}

\tau=0.01

, restricted to forms occurring more than 20,000 times.

\overline{|\Delta|}

denotes the mean TSD magnitude over significant occurrences. Values are rounded for display; rankings are based on the unrounded

\rho_{\tau}(v)

. For compactness, altern. abbreviates alternatively .

Model / Method	AMC23	AIME24	AIME25	AIME26	HMMT26	MATH500	Avg.
4B-2507 $\rightarrow$ Qwen3-1.7B
Student	80.8	40.6	32.5	29.2	21.8	90.2	49.2
Teacher	94.8	56.5	53.1	54.2	25.2	95.4	63.2
OPD	82.0	39.6	35.0	34.6	22.9	90.6	50.8
ExOPD	81.7	43.1	35.4	34.2	25.6	90.6	51.8
EOPD	81.6	44.0	36.7	38.8	19.7	90.6	51.9

Table 3: Main results on mathematical reasoning benchmarks. The best result among distillation methods for each teacher–student configuration is shown in bold . Subscripts on Cal-OPD indicate gains over the corresponding student. For compactness, 4B-2507 and 30B-2507 denote Qwen3-4B-Thinking-2507 and Qwen3-30B-A3B-Thinking-2507, respectively.

Intervention Set	AMC23	AIME24	AIME25	AIME26	HMMT26	MATH500	Avg.
$\mathcal{C}_{\mathrm{inst}}$	83.9	40.8	34.8	34.2	25.2	90.3	51.5
$\mathcal{C}_{\mathrm{eval}}$	84.5	45.0	37.1	36.3	24.8	90.8	53.1
$\mathcal{C}_{\mathrm{ans}}$	84.2	40.8	34.8	37.3	23.9	90.9	52.0
$\mathcal{C}_{\mathrm{sol}}$	81.3	40.6	30.0	30.8	22.0	89.2	49.0

Table 4: Effect of contextual interventions used for TSD estimation on Cal-OPD performance for Qwen3-4B-Thinking-2507

\rightarrow

Qwen3-1.7B.

Figure 4: Training dynamics of Cal-OPD under different TSD-estimation interventions for Qwen3-4B-Thinking-2507

\rightarrow

Qwen3-1.7B.

Figure 5: Effect of the relaxation factor

\lambda

on Cal-OPD performance and retained discrepancy ratio.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Model	Mode	Max Length	Temp.	Top- $p$
Solution generation	Qwen3-235B-A22B-Instruct-2507	No-thinking	12,288	1.0	1.0
Student rollout	Qwen3-1.7B	Thinking	16,384	1.0	1.0
Teacher rescoring	Qwen3-8B	Thinking	Fixed rollout	–	–

Appendix

Table 5: Configurations used for preparing the data for TSD analysis. Solution generation and student rollout collection use one generation per question. Teacher rescoring does not involve sampling, since the student trajectory is kept fixed.

Panel of Figure 6 — Figure 6: Prevalence of significant TSD across thresholds and teacher models. Although the absolute prevalence decreases with stricter thresholds, substantial TSD continues to emerge under task-agnostic interventions, while solution-level privilege consistently broadens the affected set.

Panel of Figure 7 — Figure 7: Retention of significant TSD across thresholds and teacher models. Less informative interventions retain nearly all affected token positions under solution-level privilege, whereas retention in the reverse direction is substantially lower.

Panel of Figure 8 — Figure 8: Semantic consistency of TSD across thresholds and teacher models. Contrasting interventions consistently exhibit substantial positional overlap, high directional agreement, and predominantly shared deviation across thresholds and model scales.

#	Highest $\rho_{\tau}$			Lowest $\rho_{\tau}$			Highest $\rho_{\tau}$			Lowest $\rho_{\tau}$
	$c_{\mathrm{inst}}^{\mathrm{pos}}$						$c_{\mathrm{inst}}^{\mathrm{neg}}$
	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$
1	maybe	74.4	0.066		1.0	0.069	maybe	84.4	0.110		1.5	0.080
2	consider	73.1	0.071	###	2.1	0.071	consider	84.2	0.106	0	2.9	0.116
3	try	66.1	0.086	0	2.2	0.095	however	82.8	0.157	###	3.4	0.082
4	however	65.2	0.086	∘	2.6	0.068	therefore	82.1	0.170	∘	3.7	0.083

Appendix

Table 6: Top-24 and bottom-24 token forms under

c_{\mathrm{inst}}^{\mathrm{pos}}

and

c_{\mathrm{inst}}^{\mathrm{neg}}

\tau=0.01

. The final row reports unweighted means of

\rho_{\tau}

and

\overline{|\Delta|}

over each 24-token subset.

#	Highest $\rho_{\tau}$			Lowest $\rho_{\tau}$			Highest $\rho_{\tau}$			Lowest $\rho_{\tau}$
	$c_{\mathrm{eval}}^{\mathrm{pos}}$						$c_{\mathrm{eval}}^{\mathrm{neg}}$
	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$
1	maybe	77.5	0.079	}{	1.0	0.072	maybe	83.8	0.100	}{	1.1	0.076
2	consider	76.2	0.074	###	1.1	0.047	however	83.2	0.139	###	1.3	0.053
3	earlier	72.4	0.082	0	2.5	0.111	consider	82.6	0.091	0	2.9	0.124
4	however	72.2	0.102	frac	2.8	0.058	earlier	79.9	0.113	frac	3.1	0.064

Appendix

Table 7: Top-24 and bottom-24 token forms under

c_{\mathrm{eval}}^{\mathrm{pos}}

and

c_{\mathrm{eval}}^{\mathrm{neg}}

\tau=0.01

. The final row reports unweighted means of

\rho_{\tau}

and

\overline{|\Delta|}

over each 24-token subset.

#	Highest $\rho_{\tau}$			Lowest $\rho_{\tau}$			Highest $\rho_{\tau}$			Lowest $\rho_{\tau}$
	$c_{\mathrm{ans}}^{\mathrm{pos}}$						$c_{\mathrm{ans}}^{\mathrm{neg}}$
	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$
1	maybe	83.3	0.254	}{	1.1	0.070	maybe	83.7	0.362	}{	1.1	0.071
2	however	82.2	0.210	###	1.6	0.058	however	82.3	0.163	###	1.6	0.055
3	therefore	80.6	0.210	0	2.8	0.143	consider	81.0	0.090	0	2.8	0.131
4	consider	80.1	0.090	frac	3.1	0.066	therefore	80.6	0.358	frac	3.1	0.062

Appendix

Table 8: Top-24 and bottom-24 token forms under

c_{\mathrm{ans}}^{\mathrm{pos}}

and

c_{\mathrm{ans}}^{\mathrm{neg}}

\tau=0.01

. The final row reports unweighted means of

\rho_{\tau}

and

\overline{|\Delta|}

over each 24-token subset.

#	Highest $\rho_{\tau}$			Lowest $\rho_{\tau}$			Highest $\rho_{\tau}$			Lowest $\rho_{\tau}$
	$c_{\mathrm{sol}}^{\mathrm{pos}}$						$c_{\mathrm{sol}}^{\mathrm{neg}}$
	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$	Token	$\boldsymbol{\rho_{\tau}}$ (%)	$\mathbf{\overline{\|\Delta\|}}$
1	maybe	97.4	0.594	}{	2.8	0.251	maybe	96.2	0.305	}{	2.0	0.141
2	however	97.4	0.625	###	3.2	0.131	however	95.3	0.358	###	2.9	0.102
3	therefore	95.9	0.460	0	5.0	0.356	therefore	95.0	0.351	0	4.3	0.250
4	consider	95.4	0.380	9	6.8	0.384	consider	93.5	0.230	9	5.9	0.270

Appendix

Table 9: Top-24 and bottom-24 token forms under

c_{\mathrm{sol}}^{\mathrm{pos}}

and

c_{\mathrm{sol}}^{\mathrm{neg}}

\tau=0.01

. The final row reports unweighted means of

\rho_{\tau}

and

\overline{|\Delta|}

over each 24-token subset.

Category	Hyperparameter	Value
Optimizer
	Learning rate	$1\times 10^{-6}$
	LR warmup ratio	0.0
	Weight decay	0.01
	LR scheduler	cosine
	Gradient clip	1.0

Appendix

Table 10: Shared training configuration for all methods.

Method	Method-specific hyperparameters
OPD	None beyond the shared configuration.
ExOPD	exopd_lambda =1.25; loss_max_clamp =null; loss_agg_mode =token-mean. LoRA: rank=64, alpha=128, target=all-linear.
EOPD	eopd_entropy_threshold =0.8; eopd_alpha =1.0; teacher max logprobs=17.
Uni-OPD	Batch structure: 16 questions per step, 16 rollouts per question (256 trajectories per step); mini-batch size 16 questions. Loss: loss_agg_mode =token-mean; loss_max_clamp =null. Margin calibration: target_correct_ratio =0.5; margin_scope =group; trajectory_reduce =mean; margin_direction =spread; margin_delta =0.4.
Privileged-OPD	Max prompt length=12,288 to accommodate reference solutions; longer solutions are truncated. Otherwise identical to OPD.

Appendix

Table 11: Method-specific hyperparameters. Entries marked “none” use the shared configuration unchanged.

Panel of Figure 9 — Figure 9: Training dynamics of OPD and Cal-OPD on Qwen3-1.7B.

Panel of Figure 10 — Figure 10: Training dynamics comparison between privileged OPD and Cal-OPD with solution-level calibration.

Method	Transfer	Avg. time per step (min)	Total time for 100 steps (h)
Cal-OPD	30B-A3B-2507 $\rightarrow$ 8B	50.05	$\sim$ 83.4
OPD	30B-A3B-2507 $\rightarrow$ 8B	35.19	$\sim$ 58.7
Cal-OPD	30B-A3B-2507 $\rightarrow$ 4B	48.76	$\sim$ 81.3
OPD	30B-A3B-2507 $\rightarrow$ 4B	31.26	$\sim$ 52.1
Cal-OPD	4B-2507 $\rightarrow$ 1.7B	14.90	$\sim$ 24.8
OPD	4B-2507 $\rightarrow$ 1.7B	18.80	$\sim$ 31.3

Appendix

Table 12: Efficiency comparison between Cal-OPD and OPD across different teacher–student configurations.

Panel of Figure 11 — Figure 11: Gradient norm dynamics of OPD training across different teacher–student pairs.

Method	AMC23	AIME24	AIME25	AIME26	HMMT26	MATH500	Avg.
OPD	82.0	39.6	35.0	34.6	22.9	90.6	50.8
Advantage-Sync	81.7	39.8	35.4	35.0	22.2	90.5	50.8
Token-Sync	81.3	38.3	34.6	34.0	22.3	89.8	50.1
TSD-Filter ( $\tau_{\mathrm{TSD}}=0.1$ )	83.6	44.0	37.1	36.0	24.4	90.9	52.7
TSD-Filter ( $\tau_{\mathrm{TSD}}=0.05$ )	81.7	41.7	35.4	34.8	22.9	90.1	51.1
TSD-Filter ( $\tau_{\mathrm{TSD}}=0.01$ )	80.3	38.8	33.5	32.9	21.6	89.3	49.4

Appendix

Table 13: Retention-matched control on teacher–student discrepancy for Qwen3-4B-Thinking-2507

\rightarrow

Qwen3-1.7B.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 511 papers

Jun 8, 2026

SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling

On-policy distillation (OPD) trains a student on its own trajectories with dense per-token supervision from a stronger teacher, and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its effectiveness implicitly relies on two assumptions that frequently break in practice: trajectory-level alignment between the student and the teacher, and uniform token-level reliability of the teacher's preferences. We therefore propose Sign-Gated On-Policy Distillation (SG-OPD), which uses a binary verifier as a trust signal for the teacher at two complementary granularities: phased teacher sampling mixes in verifier-endorsed teacher rollouts at cold-start, and a sign-consistency gate extrapolates the distillation update on tokens where the teacher agrees with the verifier-correct direction and interpolates it where it disagrees. Experiments on competition-level mathematical reasoning benchmarks show that SG-OPD consistently outperforms standard OPD, with average gains of 1.98 and 7.50 at the per-sample and per-question levels, respectively.

Haoran Xu, Hongyu Wang, Yifei Gao +3

Zhejiang University · Hunan University · Tianjin University +2

Language Model Distillation On-Policy Distillation Teacher-Student Learning

Ask PDF

Oct 2, 2026

Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher, but providing such supervision for every rollout requires substantial teacher computation. We introduce Success-Referenced On-Policy Distillation (SR-OPD), which reduces this cost by selecting which prompts and rollouts receive teacher supervision. When the student produces both successful and failed rollouts for the same prompt, a successful rollout can serve as a natural reference for selecting failed rollouts. SR-OPD therefore focuses on such prompts and prioritizes failed rollouts whose hidden-state trajectories show sustained divergence from a successful reference, while accounting for estimated teacher-input cost. Across three teacher-student pairs and six mathematical reasoning benchmarks, SR-OPD uses only 3.46-5.02% of the teacher-input tokens required by Vanilla OPD in the one-pass setting while maintaining comparable reasoning performance. Under a controlled setting matched to 5% of Vanilla OPD's teacher-input budget, further experiments support both key design choices: focusing supervision on prompts with both successful and failed rollouts, and using successful rollouts to guide failure selection. These results indicate that a student's own successful behavior can serve as a useful reference for allocating teacher supervision under a fixed teacher-input budget.

Xiang Chen, Futao Su, Kong Wang +2

Tongji University · Xiaohongshu · Shanghai Jiao Tong University +1

Language Model Distillation On-Policy Distillation Knowledge Distillation Selective Knowledge Distillation

Ask PDF

Sep 29, 2026

From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation

On-policy distillation (OPD) trains a student on its own reasoning trajectories using feedback from a stronger teacher. Teacher interventions can improve these trajectories, but also change the distribution on which the student learns. Our controlled studies show that rollout quality alone is an incomplete criterion for allocating teacher guidance. Deeper intervention yields diminishing gains in rollout accuracy while increasing off-policy load. In a training probe with a restricted rollout horizon, peak student accuracy and performance retention favor different intervention strengths. The preferred intervention depth and placement also vary across benchmarks. These findings motivate MAESTRO, which uses local policy disagreement to jointly adapt when the teacher takes over and how long it generates. Its {policy disagreement score} combines teacher-weighted candidate coverage with local distribution similarity and is aggregated within reasoning paragraphs. Across eight mathematical reasoning benchmarks, MAESTRO achieves the highest macro-average accuracy among the compared methods for both 0.6B and 1.7B Qwen3 students, with the 1.7B student leading on every benchmark. MAESTRO also reduces average training response length by 67.3% relative to standard OPD. The code is available at https://github.com/yhao-wang/MAESTRO.

Yuhao Wang, Ruiyang Ren, Yinan Zhang +3

Nanyang Technological University, Singapore · Baidu Inc.

Language Model Distillation On-Policy Distillation

Ask PDF

Abstract

On-Policy Distillation

Explore similar work

SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling

Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation

From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation