Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models

Tiejin Chen; Xiaoou Liu; Hanyang Chen; Hua Wei

cs.LGOct 25, 2026

Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models

Authors: Tiejin Chen, Xiaoou Liu, Hanyang Chen, Hua Wei

Organizations: Arizona State University

Abstract

On-policy self-distillation (OPSD) trains a student on its own reasoning trajectories using token-level supervision from a reference-conditioned teacher initialized from the same model. However, access to the reference can induce teacher behaviors that undermine independent reasoning in thinking models, leading to reference hallucinations, where the student cites an unavailable reference to justify reasoning steps, and reconsideration suppression, where the teacher discourages verification and revision at uncertain steps. We introduce Steered-Teacher OPSD (ST-OPSD) to jointly address these failure modes through activation steering. ST-OPSD extracts two directions from model-generated reasoning trajectories and applies them to the teacher's hidden states during distillation target computation, suppressing reference-invoking behavior and encouraging reconsideration with marker-family calibration. The student learns from the resulting teacher distributions without being steered. Experiments on AIME and HMMT show that ST-OPSD improves accuracy over vanilla OPSD and other interventions while restoring reconsideration and reference-hallucination rates to base-model levels.

Figures & tables

Figure 1: An analogy of reference-conditioned supervision in OPSD: a tutor prescribes solution steps and discourages verification, potentially leaving the student unable to reason independently on new problems without a reference answer.

Figure 2: Overview of the ST-OPSD framework. We construct steering vectors and apply them to the teacher to reduce reference invocation and encourage reconsideration during on-policy self-distillation. The student learns from the steered teacher while generating reasoning from the question alone.

Model	Method	AIME24			AIME25			HMMT25			Avg.
Model	Method	Acc $\uparrow$	Hes./1k $\uparrow$	Ref $\downarrow$	Acc $\uparrow$	Hes./1k $\uparrow$	Ref $\downarrow$	Acc $\uparrow$	Hes./1k $\uparrow$	Ref $\downarrow$	Acc $\uparrow$	Hes./1k $\uparrow$	Ref $\downarrow$
Qwen3-4B	Base	70.8	12.9	10.8	65.8	13.1	15.8	41.7	15.1	14.2	59.4	13.7	13.6
	Vanilla OPSD	69.2	13.4	59.2	56.7	12.9	68.3	31.7	15.4	72.5	52.5	13.9	66.7
	Purified OPSD	70.8	14.1	10.0	64.2	13.1	17.5	43.3	18.7	8.3	59.4	15.3	11.9
	Random SV	69.2	13.2	56.7	49.2	13.3	78.3	31.7	16.6	79.2	50.0	14.4	71.4
	ST-OPSD	74.2	13.8	12.5	64.2	14.4	14.2	45.0	16.8	12.5	61.1	15.0	13.1

Table 1: Results on three mathematical reasoning benchmarks. Acc. denotes avg@4 accuracy (%), Hes./1k denotes the number of hesitation markers per 1,000 generated tokens, and Ref. (%) denotes the percentage of responses containing reference citations. Across all evaluated models, ST-OPSD improves average accuracy over Vanilla OPSD while bringing hesitation frequency and reference-citation rates closer to base-model levels.

Model	$\alpha_{\mathrm{ref}}$	$\alpha_{\mathrm{recon}}$	$\lVert v\rVert_{2}/\lVert out\rVert_{2}$
Qwen3-4B	2.29	9.50	12.7
Qwen3-8B	4.25	12.8	8.9
OLMo-7B	0.80	3.50	18.7

Table 2: Steering coefficients and vector norms for each model.

Panel of Figure 3 — Figure 3: Ablation studies with Qwen3-4B. (a) Method variants under the default training setting. Base + SV applies inference-time steering, Prompted OPSD uses teacher instructions, and Recon. only and Ref. only apply each steering direction separately. (b) Vanilla OPSD and ST-OPSD trained on Omni-Math. (c) Combining ST-OPSD with the selected subset improves upon the previous best ST-OPSD result, obtained with the default training data. Selected denotes examples where the teacher contributes new reasoning ideas. Panels (a) and (c) report mean accuracy across AIME24, AIME25, and HMMT25. All scores are percentages.

	Time (s/step)		Throughput (token/s)
Method	Gen.	Train.	Gen.	Train.
Vanilla OPSD	181.2	71.0	850	2,176
ST-OPSD	214.2	86.7	845	2,092
Purified OPSD	222.0	117.0	844	1,602

Table 3: Efficiency comparison on OLMo. Gen. and Train. denote generation and training, respectively. Lower time and higher throughput indicate better efficiency. The results show that ST-OPSD roughly maintains the speed.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Hesitation markers grouped into the three reconsideration families used in ST-OPSD.

Panel of Figure 5 — Figure 5: Inventory of reference-citation markers grouped by expression type.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 157 papers

Rethinking On-Policy Self-Distillation for Thinking Models
Jul 6, 2026Simran Kaur, Narutatsu Ri, Yinghui He +2LLM Self-Correction On-Policy Self-Distillation
Princeton Language and Intelligence, Princeton University
One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation
Oct 22, 2026Khouloud Saadi, Di WangOn-Policy Self-Distillation Knowledge Distillation
Department of Computer Science, King Abdullah University of Science and Technology, KAUST, Thuwal, Saudi Arabia
Recursive Self-Improvement via On-Policy Distillation for Reasoning
Sep 25, 2026Shangjian Yin, Zehao Zhao, Kavosh Asadi +7Language Model Self-Improvement On-Policy Self-Distillation
Meta AI · University of California, Riverside · Work done at Meta

Jul 6, 2026J/K move · Enter open · S save

Rethinking On-Policy Self-Distillation for Thinking Models

Simran Kaur, Narutatsu Ri, Yinghui He, Liam Fowl+1

Princeton Language and Intelligence, Princeton University

LLM Self-Correction On-Policy Self-Distillation On-Policy Distillation Context Distillation Learning Using Privileged Information Large Reasoning Models

Open paper AskPDF arXiv

Self-distillation is a promising recipe for self-improvement in language models. In this setting, a model can serve as its own teacher when given privileged information, such as a solution to a math problem. This seems especially appealing for thinking models, which can use test-time reasoning to absorb the privileged information. Surprisingly, we show that privileged self-distillation degrades thinking models on long reasoning traces: across five Qwen3 and OLMo thinking models evaluated on AIME24, AIME25, and HMMT25, privileged-context distillation causes a relative drop of up to 17% in avg@16 accuracy. The degradation scales with the amount of privileged context withheld from the student and is most pronounced at long rollout budgets, where thinking models otherwise obtain their largest gains. This failure mode is not specific to self-distillation: on-policy distillation (OPD) improves thinking models, but privileged OPD reverses these gains. Our diagnostics link this failure mode to how privileged teacher context reshapes learning at high-entropy forking positions, where multiple continuations remain plausible and may lead to different reasoning paths. Privileged context lowers fork rates in thinking-model rollouts but not in instruction-model rollouts. This leads to an interesting dichotomy, where privileged context can help instruction-tuned models but hurts stronger thinking models. The effect is visible when the student begins a self-correction branch, where privileged OPD penalizes sampled reconsideration tokens that vanilla OPD supports. Thinking models trained with a privileged teacher produce fewer verification, backtracking, and hedging markers, even after length normalization. These findings indicate that self-distillation for strong thinking models requires attention to token-level signal, especially around correction and reasoning steps.

Abstract

On-Policy Self-Distillation

Explore similar work

Rethinking On-Policy Self-Distillation for Thinking Models

One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation

Recursive Self-Improvement via On-Policy Distillation for Reasoning