How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation

Xiaoyu Ma; Haoyue Liu; Zhichao Wang; Jionghao Zhu; Xiaoying Tang

cs.LGOct 4, 2026

How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation

Authors: Xiaoyu Ma, Haoyue Liu, Zhichao Wang, Jionghao Zhu, Xiaoying Tang

Organizations: School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNiI-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)

Abstract

On-policy distillation (OPD) improves the reasoning capabilities of small language models through token-level teacher supervision on student-generated trajectories. Yet can teachers that excel at solving problems independently also guide student reasoning effectively? Prior work shows that when student prefixes follow reasoning paths that differ from the teacher's own or contain errors, teachers can be less accurate when continuing from these prefixes than when solving problems independently. To this end, we propose Prep-OPD, which uses reinforcement learning (RL) before distillation to train the teacher to adapt to the student's existing reasoning state and correct course when errors arise. Training optimizes teacher continuations from fixed student prefixes using final-answer correctness as the reward. The prepared teacher then trains the student through trajectory guidance and token-level supervision. We evaluate Prep-OPD on eight mathematical reasoning benchmarks, using Qwen3-4B-Instruct-2507 as the teacher and Qwen3-0.6B and Qwen3-1.7B as students. With the 4B teacher and 1.7B student, Prep-OPD improves average accuracy over standard OPD and the strongest baseline, Relay-OPD, by 8.28 and 2.30 percentage points, respectively. Controlled experiments further show that teacher RL conditioned on student-generated prefixes yields higher student accuracy than problem-start teacher RL with and without handoff on Qwen3-1.7B. Reusing the same prepared teacher also improves Qwen3-0.6B.

Figures & tables

Figure 1: Motivation and main results. (a) A solution from scratch and two continuations from the same erroneous student prefix. (b) Qwen3-1.7B mean benchmark accuracy (%; Table 1 ).

Figure 2: Overview of Prep-OPD. Stage 1 fixes prefixes from $S_{0}$ and trains $T_{\phi}$ , initialized from $T_{0}$ , with outcome rewards and suffix-only GRPO. Stage 2 freezes $T_{P}$ for bounded handoffs and token-level supervision of mixed trajectories. Blue/purple tokens denote student/teacher generation; distillation covers both token types. Inference uses the student alone.

Method	Teacher	AIME			AMC	HMMT		Olympiad	MATH	Avg.
		2024	2025	2026	2023	Feb. 26	Nov. 25	Bench	500
Qwen3-0.6B Student
Initial student ( $S_{0}$ )	–	2.08	1.25	0.00	17.19	0.00	1.25	14.15	38.35	9.28
GRPO	–	9.17	13.23	12.08	46.56	6.25	7.08	35.15	65.60	24.39
OPD	$T_{B}$	13.33	12.50	14.17	51.25	8.33	9.17	39.93	70.60	27.41
TRD	$T_{0}$	4.58	7.92	5.94	39.61	6.72	4.69	29.63	61.60	20.09

Table 1: Mathematical reasoning accuracy (%). Avg. is the unweighted mean over eight benchmarks. Distillation runs for both the

1.7

B and

0.6

B students use

16{,}164

training problems. Bold/underline indicate the best/second-best reported scores within each student block.

Figure 3: Training efficiency on Qwen3-1.7B. (a) Accuracy versus cumulative student training compute for the checkpoint selected on the development set for each distillation method. Filled markers identify Pareto-optimal points among these selected checkpoints; the staircase gives their best accuracy within each compute budget. (b) Mean rollout length over the full 126-step student training run for each method.

Figure 4: Teacher continuation across prefix depths. Accuracy at each depth weights the eight benchmarks equally, and the depth mean weights the four continuation positions equally. The

0\%

point denotes problem-start solving and is excluded from this mean. Panel titles report the mean recovery gap in percentage points. Shading shows the pointwise gap between problem-start and continuation accuracy, not uncertainty. Ordinary RL denotes problem-start RL.

Teacher	No handoff	With handoff	$\Delta$ handoff (pp)
$T_{0}$	39.96	45.33	$+5.36$
$T_{B}$	39.37	45.35	$+5.98$
$T_{P}$	41.26	47.65	$+6.39$

Table 2: Ablations of teacher preparation, handoff, and teacher roles. Results are Qwen3-1.7B student accuracies (%), averaged over the eight benchmarks with equal weights.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	LR	Optimizer	Batch/mini	Schedule	Additional settings
Teacher RL ( $T_{B}$ , $T_{P}$ )	$10^{-6}$	AdamW	128/128	constant	GRPO group size $G=8$ ; response cap 16,384 tokens
OPD students	$10^{-6}$	AdamW	128/128	constant	One PPO optimization epoch; dynamic batching
Handoff students	$10^{-6}$	AdamW	128/128	constant	Handoff training channel ( train.sh )
FastOPD	$10^{-6}$	AdamW	128/128	constant	Training rollout cap 4,096 tokens
IW-OPD	$10^{-6}$	AdamW	128/128	constant	Importance-weight parameter $\gamma=0.5$

Appendix

Table 3: Training hyperparameters. Batch/mini denotes the configured training batch and PPO mini-batch sizes. The prepared teacher is denoted by

T_{P}

throughout the paper.

Arm	Teacher	Channel	Budget	Role
$S_{0}$	–	–	–	untrained floor
OPD	$T_{B}$	standard	16K	main distilled baseline
TRD	$T_{B}$	offline rewrite	16K	re-run of Jiang et al. (2026)
IW-OPD	$T_{B}$	standard	16K	importance-weighted OPD
FastOPD	$T_{B}$	truncated rollouts	4K	re-run of Zhang et al. (2026)
Relay-OPD	$T_{B}$	relay	16K	equal-RL handoff baseline

Appendix

Table 4: Full-corpus Qwen3-1.7B distillation configurations (

16{,}164

student-training problems). Budget is the student-training response cap.

			Student-prefix depth
Benchmark	Teacher	$C_{T}$	$20\%$	$40\%$	$60\%$	$80\%$	$P_{T}$	$G_{T}$
AIME 2024	$T_{0}$	60.10	60.94	56.15	53.33	54.17	56.15	3.95
	$T_{B}$	67.92	60.42	54.79	50.10	44.90	52.55	15.37
	$T_{P}$	59.27	61.46	59.48	53.65	56.15	57.68	1.58
AIME 2025	$T_{0}$	41.04	45.31	46.46	38.23	38.13	42.03	-0.99
	$T_{B}$	53.13	44.58	41.35	35.73	38.02	39.92	13.21

Appendix

Table 5: Detailed teacher continuation results. Problem-start accuracy

C_{T}

and continuation accuracy at four student-prefix depths are reported in percent.

P_{T}

is the equal-weight mean over the four depths;

G_{T}=C_{T}-P_{T}

is the recovery gap in percentage points.

Benchmark	Relay ( $T_{0}$ )	Prep ( $T_{P}$ )	$\Delta$	Relay ( $T_{0}$ )	Prep ( $T_{P}$ )	$\Delta$
	Qwen3-0.6B			Qwen3-1.7B
AIME 2024	13.33	17.50	$+4.17$	43.65	48.33	$+4.69$
AIME 2025	14.17	18.33	$+4.17$	28.33	31.67	$+3.33$
AIME 2026	18.33	15.00	$-3.33$	31.15	32.08	$+0.94$
AMC 2023	50.62	57.50	$+6.88$	72.50	75.62	$+3.12$
HMMT Feb. 2026	8.33	10.61	$+2.27$	23.77	25.00	$+1.23$

Appendix

Table 6: Comparison with Relay-OPD using the original teacher. Accuracy is reported in percent;

\Delta

is Prep-OPD minus Relay-OPD in percentage points. Bold marks the higher accuracy within each student size. Avg. weights the eight benchmarks equally. Differences are computed before rounding.

Method	Seed	AIME			AMC	HMMT		Olympiad	MATH	Avg.
		2024	2025	2026	2023	Feb. 26	Nov. 25	Bench	500
Prep-OPD ( $T_{P}$ )	42	48.33	31.67	32.08	75.62	25.00	20.83	60.44	87.20	47.65
	43	45.83	30.83	34.17	75.00	23.48	20.83	59.93	86.30	47.05
	44	43.33	31.67	37.50	73.75	26.52	23.33	58.30	85.00	47.42
Relay-OPD ( $T_{B}$ )	42	38.33	31.25	30.83	74.69	23.48	20.94	58.00	85.25	45.35
	43	40.00	29.58	29.58	74.38	23.48	20.83	56.67	84.45	44.87

Appendix

Table 7: Comparison across random seeds on Qwen3-1.7B. Accuracy is reported in percent. The seed-42 rows reproduce Table 1 ; seeds 43 and 44 are additional runs. Avg. is the unweighted mean over the eight benchmarks, computed before rounding the benchmark entries.

Explore similar work

From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation
Sep 29, 2026Yuhao Wang, Ruiyang Ren, Yinan Zhang +3Language Model Distillation On-Policy Distillation
Nanyang Technological University, Singapore · Baidu Inc.
Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers
Oct 5, 2026Xiaoyu Chen, Bo Shao, Tiangang Zhu +5RL for Language Model Reasoning Selective Knowledge Distillation
Institute of Software, Chinese Academy of Sciences · Microsoft · Work done during an internship at Microsoft +1
Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation
Oct 2, 2026Xiang Chen, Futao Su, Kong Wang +2Language Model Distillation On-Policy Distillation
Tongji University · Xiaohongshu · Shanghai Jiao Tong University +1

Sep 29, 2026J/K move · Enter open · S save

From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation

Yuhao Wang, Ruiyang Ren, Yinan Zhang, Ruiqing Zhang+2

Nanyang Technological University, Singapore · Baidu Inc.

Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) trains a student on its own reasoning trajectories using feedback from a stronger teacher. Teacher interventions can improve these trajectories, but also change the distribution on which the student learns. Our controlled studies show that rollout quality alone is an incomplete criterion for allocating teacher guidance. Deeper intervention yields diminishing gains in rollout accuracy while increasing off-policy load. In a training probe with a restricted rollout horizon, peak student accuracy and performance retention favor different intervention strengths. The preferred intervention depth and placement also vary across benchmarks. These findings motivate MAESTRO, which uses local policy disagreement to jointly adapt when the teacher takes over and how long it generates. Its {policy disagreement score} combines teacher-weighted candidate coverage with local distribution similarity and is aggregated within reasoning paragraphs. Across eight mathematical reasoning benchmarks, MAESTRO achieves the highest macro-average accuracy among the compared methods for both 0.6B and 1.7B Qwen3 students, with the 1.7B student leading on every benchmark. MAESTRO also reduces average training response length by 67.3% relative to standard OPD. The code is available at https://github.com/yhao-wang/MAESTRO.

Abstract

Explore similar work

From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation

Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers

Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation