Teaching a Moving Student: Rethinking the Curriculum of On-Policy Distillation

Lingxiang Hu; Tianle Xia; Yiding Sun; Ming Xu; Linfang Shang; Lan Xu; Ning Zheng; Wei Xu

cs.CLSep 6, 2026

Teaching a Moving Student: Rethinking the Curriculum of On-Policy Distillation

Authors: Lingxiang Hu, Tianle Xia, Yiding Sun, Ming Xu, Linfang Shang, Lan Xu, Ning Zheng, Wei Xu,

Organizations: Tencent

Abstract

In on-policy distillation (OPD), the student determines which states receive teacher supervision. As its policy evolves, earlier response prefixes become less likely even though teacher-student disagreement on them persists. Under matched trajectory and optimization budgets, neither more queries nor more frequent rollout resampling is uniformly beneficial. Current-policy rollouts outperform initial-policy rollouts at shorter response budgets, but this ranking reverses at longer budgets. Replaying initial-policy rollouts after current-policy training improves accuracy, whereas replaying fixed recent rollouts does not reproduce the gain. We propose R-OPD, a gradient-triggered curriculum that adaptively selects when to revisit initial-student trajectories. When changes in mean gradients fall within minibatch-level variation for two consecutive comparisons, training switches from the next iteration onward to initial-policy replay. Across eight mathematics benchmarks and three training-data orders, R-OPD improves average accuracy over continued current-policy sampling by 2.25/4.44 percentage points at 16K/32K for a 0.6B student and 2.81/6.15 points for a 1.7B student. With a 30B-A3B teacher, it improves 8B accuracy by 4.05 points at 32K. At 32K, R-OPD exceeds a fixed schedule of 40 current-policy updates followed by 20 replay updates by 1.39/1.66/1.85 points for 0.6B/1.7B/8B. At the same generation cap, R-OPD also produces longer responses, suggesting that well-timed revisits help students use more of their reasoning capacity.

Figures & tables

Figure 1: R-OPD follows the student, then revisits earlier states. (a) Current-policy sampling follows the evolving student; replay returns to initial-policy states. (b) Earlier prefixes become less likely while teacher–student predictions remain misaligned. (c) At the gradient-triggered iteration $\tau$ , R-OPD finishes the current-policy stage; every later iteration uses the initial-policy bank.

Training queries	Initial rollouts only	Resample every 11 steps	Resample every step
8	21.16	24.09	23.61
48	20.18	24.40	24.22
Full (14,080)	19.05	24.51	25.57

Table 1: Query count and rollout sampling interact. Average 16K accuracy (%) on AIME24, AIME25, and AMC23. All configurations use 14,080 trajectories and 110 optimization steps. Per-benchmark results and uncertainty estimates appear in Appendix D.1 .

Teacher	Current-policy advantage at 16K			Initial-policy advantage at 32K
	Acc.	pass@ $n$	Extraction	Acc.
Qwen3-4B	+0.99	+10.38	+38.87	+4.39
JustRL-1.5B	+6.15	+16.09	+39.52	+2.07

Table 2: The relative value of rollout sources changes with response budget. Eight-benchmark average differences in percentage points: current-policy minus initial-policy at 16K, and initial-policy minus current-policy accuracy at 32K. Positive values favor the source named in the column header. The 16K results re-score prefixes of the same 32K generations. For pass@

n

n=32

on the six competition benchmarks and

n=4

on MATH-500 and Olympiad-Bench. Extraction is the fraction of responses with an extractable boxed answer.

Figure 2: Rollout-source advantages depend on response budget, while earlier prefixes become less likely. (a) Initial-policy minus current-policy accuracy after 40 optimization steps, averaged equally over eight benchmarks under two teachers. Budgets re-score saved 32K responses; error bars show 95% paired-bootstrap intervals at 32K. (b) Response-weighted cumulative distributions of the change in initial-policy prefix log-probability from initialization to 20 steps (orange dashed) or 40 steps (blue solid). Negative changes mean the prefixes become less likely; the light fill marks the gap between the curves. Each response has equal weight; see Appendix E.2 for sampling details.

Comparison	Schedule	16K	32K	Tokens
Source	Current-policy sampling	26.41	26.52	8,254
	Fixed recent bank	26.12	26.21	7,997
	Initial-policy replay	27.13	29.37	12,116
Order	Current then initial	27.77	30.55	13,015
Order	Initial then current	26.16	27.24	11,369

Table 3: The source and order of replay affect accuracy. Eight-benchmark mean accuracy (%) for Qwen3-0.6B. Source compares 20-step continuations after 40 current-policy steps; the fixed recent bank comes from the 40-step student. Order compares two 20-step stages after 20 current-policy steps. All endpoints use 7,200 trajectories and 60 updates. 16K re-scores saved 32K responses; Tokens is mean 32K output length. Bold marks the best accuracy within each comparison.

Student checkpoint / continuation	Initial-policy prefixes	$R_{40}$ prefixes
Common starting checkpoint ( $R_{40}$ )	0.448	0.427
Continue current-policy sampling	0.404	0.370
Replay fixed recent rollouts	0.404	0.370
Replay initial-policy rollouts	0.370	0.400

Table 4: Rollout source determines where KL decreases. Full-vocabulary reverse KL (nats; lower is better) on fixed diagnostic prefixes generated by the initial and 40-step students. All continuations start from the same

R_{40}

checkpoint and add 20 optimization steps. Responses are weighted equally; the full matrix is in Table 13 .

Figure 3: R-OPD across response budgets. Eight-benchmark accuracy after 60 updates: current-policy sampling versus R-OPD. Curves average three runs for 0.6B and show one run for 1.7B/8B; Table 3 a summarizes a separate three-data-order set at every size. Shorter budgets re-score saved 32K responses. Teachers: Qwen3-4B-Instruct-2507 in (a,b), Qwen3-30B-A3B-Instruct-2507 in (c).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Queries	$C$	$U$	Evaluation
Query–sampling grid	8 / 48 / Full	14,080	110	Three benchmarks
Matched rollout sources	48	4,800	40	Three and eight benchmarks
Source and order	48	7,200	60	Eight benchmarks
R-OPD (0.6B, 1.7B, 8B)	48	7,200	60	Eight benchmarks
Fixed-onset scan (0.6B)	48	7,200	60	Eight benchmarks

Appendix

Table 6: Experiment budgets and evaluation protocols. Each comparison matches consumed trajectories

C

and optimization steps

U

. Three-benchmark evaluation uses direct 16K generations; eight-benchmark evaluation uses saved 32K responses and their shorter prefixes. The matched rollout comparison reports both protocols.

Main student	Qwen3-0.6B, non-thinking
Curriculum-transfer students	Qwen3-1.7B and Qwen3-8B, non-thinking
8B curriculum teacher	Qwen3-30B-A3B-Instruct-2507
Main teacher	Qwen3-4B-Instruct-2507 (shared tokenizer)
Secondary teacher	JustRL-DeepSeek-1.5B (cross-tokenizer)
Actor/ref parameters	FP32
Teacher scoring	FP32 parameters, BF16 autocast

Appendix

Table 7: Shared training configuration. Training queries come from the filtered English subset of DAPO-Math-17K ( Yu et al., 2025 ) ; Appendix C gives the construction.

Benchmark	Questions	Responses per question	Total responses
AIME24	30	32	960
AIME25	30	32	960
AIME26	30	32	960
AMC23	83	32	2,656
HMMT25-Nov	30	32	960
HMMT26-Feb	33	32	1,056

Appendix

Table 8: Evaluation sets and sampling counts per checkpoint.

Query set	Queries	Accuracy	pass@32
Full	14,080	25.57	58.11
Difficulty-selected	48	23.98	58.51
Random	48	24.22	59.22

Appendix

Table 9: Query selection at a matched training budget. Qwen3-0.6B with the 4B teacher; all conditions use 14,080 trajectories, 110 updates, and per-step resampling. Accuracy and observed pass@32 (%) are equal-weight averages over AIME24, AIME25, and AMC23 at a direct 16K limit.

Metric	$N$	$M$	AIME24	AIME25	AMC23	Avg.
Accuracy	8	1	8.85	14.90	39.72	21.16
	8	10	11.46	15.42	45.41	24.09
	8	110	10.83	15.42	44.58	23.61
	48	1	7.29	14.90	38.37	20.18
	48	10	11.98	15.31	45.90	24.40
	48	110	12.08	15.10	45.48	24.22

Appendix

Table 10: Per-benchmark results for the query-count and sampling grid (%). Avg. weights AIME24, AIME25, and AMC23 equally;

N

and

M

denote query and collection-round counts.

Figure 4: Response categories by benchmark and sampling configuration. Labels give the query count and collection rounds. Bars show the four mutually exclusive combinations of answer extraction and reaching the 16K limit; extraction counts answers regardless of correctness. The average weights the three benchmarks equally.

Teacher: Qwen3-4B-Instruct-2507
	AIME24		AIME25		AMC23		Average
Rollout source	Acc.	pass@32	Acc.	pass@32	Acc.	pass@32	Acc.	pass@32	Extraction
Current-policy	11.35	50.00	15.83	40.00	43.49	81.93	23.56	57.31	66.72
Initial-policy	8.44	40.00	17.08	30.00	37.27	63.86	20.93	44.62	26.57
Teacher: JustRL-DeepSeek-1.5B
Current-policy	15.31	53.33	19.48	46.67	50.64	84.34	28.48	61.45	63.41

Appendix

Table 11: Independent direct-16K evaluation on three benchmarks (%). Current-policy mean accuracy exceeds initial-policy accuracy by 2.63 points with Qwen3 (95% interval

[-0.03,5.33]

) and 6.72 with JustRL (

[3.65,9.80]

Figure 5: Accuracy and response length across generation budgets. Each curve plots eight-benchmark mean accuracy against mean response length at 4K, 8K, 16K, 24K, and 32K; filled markers denote 32K. Color and line style identify the rollout source, and the panels separate the two teachers. The horizontal axis uses decimal thousands of tokens. Per-benchmark accuracy differences and 32K length ratios appear in Table 12 ; all aggregates weight benchmarks equally.

Teacher	Benchmark	4K	8K	16K	24K	32K	Length ratio at 32K
Qwen3-4B	AIME24	-4.06	-3.23	-1.56	+2.40	+6.77	$1.76\times$
	AIME25	-4.06	-3.85	+1.25	+3.65	+5.94	$1.82\times$
	AIME26	-3.12	-2.60	+0.94	+1.04	+2.71	$1.67\times$
	AMC23	-11.11	-6.36	-4.48	-0.94	+4.33	$1.85\times$
	HMMT25-Nov	-1.25	-1.15	+0.10	+2.50	+4.06	$1.79\times$
	HMMT26-Feb	-2.75	-3.22	-2.37	-0.28	+2.08	$1.83\times$

Appendix

Table 12: Accuracy differences and 32K response-length ratios by benchmark. Accuracy differences are initial-policy minus current-policy in percentage points. Positive values favor initial-policy training. Ratios are initial-policy/current-policy mean response length at 32K. Scores use prefixes of saved 32K responses; sample counts and benchmark weighting follow Appendix B.1 .

Start	Scoring student / continuation	Steps	Initial	$R_{20}$	$R_{40}$
—	Initial student	0	0.774	0.624	0.628
	$R_{20}$	20	0.548	0.505	0.523
	$R_{40}$	40	0.448	0.409	0.427
$R_{40}$	Continue current-policy sampling	60	0.404	0.361	0.370
	Replay fixed recent rollouts	60	0.404	0.360	0.370
	Replay initial-policy rollouts	60	0.370	0.378	0.400

Appendix

Table 13: Complete KL matrix on fixed diagnostic prefixes. Full-vocabulary reverse KL (nats; lower is better),

E_{i,j}

in Equation 7 , for the Qwen3-0.6B student and fixed Qwen3-4B-Instruct-2507 teacher. Rows specify the student used for scoring; columns specify the student that generated the diagnostic responses. Each column uses the same 384 responses and sampled positions across all scoring students, averaging positions within each response and then weighting responses equally. Steps denote total optimization steps; Start identifies the checkpoint restored for each 20-step continuation.

Training history	4K	8K	16K	24K	32K	Tokens
Initial $\rightarrow$ initial	12.85	18.66	25.24	28.93	30.31	21,357
Initial $\rightarrow$ current	17.67	22.87	26.62	27.58	27.71	11,166
Current $\rightarrow$ initial	14.23	20.71	27.13	30.06	29.37	12,116
Current $\rightarrow$ current	17.94	23.06	26.41	26.52	26.52	8,254
Current $\rightarrow$ fixed recent	18.15	23.10	26.12	26.20	26.21	7,997

Appendix

Table 14: Continuation source after 40 optimization steps. Eight-benchmark average accuracy (%) across response budgets. All endpoints have 60 steps; mean response length uses complete 32K responses. Comparisons sharing a first-stage source restore the same model and optimizer. The fixed recent bank is generated by the 40-step current-policy student.

16K retained prefixes
Benchmark	Initial then initial	Initial then current	Current then initial	Current then current
AIME24	10.10	11.04	11.15	12.71
AIME25	16.67	16.25	16.15	16.77
AIME26	13.23	12.92	13.02	10.83
AMC23	38.48	42.73	43.83	44.69
HMMT26-Feb	5.68	9.09	8.71	8.43

Appendix

Table 15: Per-benchmark accuracy for the four continuations (%). Each column lists the first 40 and final 20 steps, as defined in Table 14 . Average weights benchmarks equally and uses unrounded values.

Condition	Steps	4K	8K	16K	24K	32K	Tokens
Current only	40	17.28	21.86	25.86	27.18	27.44	12,011
Replay after 20 steps	40	11.20	17.10	24.40	28.89	32.02	22,343

Appendix

Table 16: Early replay at a matched 40-update budget. Eight-benchmark average accuracy (%) across response budgets. Current only is

R_{40}

; early replay adds 20 initial-policy steps to

R_{20}

. Mean response length uses complete 32K responses.

Benchmark	4K	8K	16K	32K
AIME24	-1.67	-1.77	+1.46	+4.90
AIME25	-4.69	-2.29	+1.04	+0.94
AIME26	-3.54	-1.67	+2.40	+3.33
AMC23	-6.40	-3.84	+2.52	+6.59
HMMT25-Nov	-0.31	+1.04	+1.46	+2.29
HMMT26-Feb	-1.42	-1.04	+0.76	+2.27

Appendix

Table 17: Per-benchmark advantage of current-policy training followed by initial-policy replay. Values are accuracy differences in percentage points relative to the reverse order; positive values favor current then initial. The average weights benchmarks equally.

Continuation	16K	32K	Tokens at 32K
Full initial-policy replay	24.40	32.02	22,343
Equal-response mix	26.38	28.25	12,346
Source-balanced mix	27.00	30.09	14,445
Replay: no EOS	25.07	31.88	22,111
Replay: EOS only	23.64	25.57	13,870

Appendix

Table 18: Loss controls for 20 replay-stage updates after

R_{20}

. Eight-benchmark accuracy (%) and mean 32K output length. All endpoints use 40 updates in total. Full replay is the early-replay baseline in Table 16 ; the mixtures use equal response counts per source.

Condition	exact $\\|g\\|_{2}$	Cosine vs. current-policy
$\theta_{0}$ current-policy	$19.426\pm 8.018$	—
$\theta_{55}$ current-policy	$5.419\pm 2.525$	—
$\theta_{55}$ initial-policy	$13.717\pm 4.781$	$0.127\ (0.092)$
$\theta_{110}$ current-policy	$5.061\pm 3.090$	—
$\theta_{110}$ initial-policy	$14.261\pm 5.043$	$0.034\ (0.078)$

Appendix

Table 19: Gradients on fixed model parameters. Means and standard deviations are over 16 diagnostic minibatches. Norms use all parameters; cross-source cosine uses a fixed subset of gradient coordinates. Parentheses give its standard error over paired minibatches.

Quantity	Value
0.6B / 1.7B teacher	Qwen3-4B-Instruct-2507
8B teacher	Qwen3-30B-A3B-Instruct-2507
Query set	48 random queries; ten occurrences each per iteration
Iterations / optimization steps	15 / 60
Responses per iteration / initial bank	480 / five batches of 480
Trigger threshold / persistence	$D\leq V$ / two consecutive comparisons

Appendix

Table 20: Curriculum settings shared across student sizes. Each student generates its own initial-policy bank and measures its own gradients.

Student	Run	$\tau$	16K accuracy (%)			32K accuracy (%)
			Current	Fixed	R-OPD	Current	Fixed	R-OPD
0.6B	1	5	26.41	27.13	28.96	26.52	29.37	31.52
	2	5	25.94	27.52	28.38	26.07	29.62	30.48
	3	7	26.77	27.49	28.53	26.91	29.68	30.83
1.7B	1	7	38.97	41.21	41.92	39.13	43.74	45.14
	2	6	38.73	41.27	41.65	38.86	43.62	45.43

Appendix

Table 21: Paired accuracy and R-OPD trigger by training-data order. Eight-benchmark average accuracy (%); each row gives one training-data shuffle of the same 48 queries.

\tau

is in training iterations.

Student	Method	4K	8K	16K	24K	32K	Tokens at 32K
0.6B	Current	17.94	23.06	26.41	26.52	26.52	8,254
	Fixed	14.23	20.71	27.13	30.06	29.37	12,116
	R-OPD	14.72	21.61	28.96	31.28	31.52	13,842
1.7B	Current	29.06	35.59	38.97	39.13	39.13	6,887
	Fixed	27.11	34.52	41.21	44.44	43.74	11,025
	R-OPD	26.77	34.66	41.92	44.61	45.14	11,980

Appendix

Table 22: Individual-run accuracy across response budgets. Eight-benchmark mean accuracy (%) and mean 32K output length for one trained endpoint per method and student size. Fixed denotes

R_{40}F_{20}

. Teachers: Qwen3-4B-Instruct-2507 for 0.6B/1.7B; Qwen3-30B-A3B-Instruct-2507 for 8B.

Student	Benchmark	16K			32K
		Current	Fixed	R-OPD	Current	Fixed	R-OPD
0.6B	AIME24	12.71	11.15	13.33	12.71	15.10	16.56
	AIME25	16.77	16.15	18.44	16.88	16.88	19.90
	AIME26	10.83	13.02	15.31	10.94	14.48	17.29
	AMC23	44.69	43.83	46.69	44.88	48.61	51.77
	HMMT25-Nov	5.73	6.67	7.40	5.83	6.77	8.85

Appendix

Table 23: Benchmark variation in the individual runs. Accuracy (%) for the endpoints in Table 22 . Fixed denotes

R_{40}F_{20}

. Average weights the eight benchmarks equally.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 514 papers

Proposing Better Rollouts for On-Policy Distillation
Oct 27, 2026Sirui Chen, Yunzhe Qi, Haobo Xu +4Language Model Distillation On-Policy Distillation
University of Illinois Urbana-Champaign · Stanford University
From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation
Sep 29, 2026Yuhao Wang, Ruiyang Ren, Yinan Zhang +3Language Model Distillation On-Policy Distillation
Nanyang Technological University, Singapore · Baidu Inc.
When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
Oct 8, 2026Siyan Zhao, Yonggan Fu, Jindong Jiang +8Language Model Distillation On-Policy Distillation
NVIDIA

Oct 27, 2026J/K move · Enter open · S save

Proposing Better Rollouts for On-Policy Distillation

Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou+3

University of Illinois Urbana-Champaign · Stanford University

Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the current student. We show that this choice can create a compounding teacher--student mismatch, because it requires the teacher to provide effective supervision on whichever trajectories the student samples, including ones less reachable by the teacher itself. On such trajectories, teacher supervision can become noisy or even misleading, leading to undesirable student patterns such as repetition. Moreover, since the updated student in turn defines the rollout policy of subsequent rounds, these patterns persist and potentially amplify over training. This failure mode suggests that improving OPD requires treating the rollout policy itself as an optimization target, rather than only modifying the distillation objective. We propose Mixture Proposal On-Policy Distillation (MPOPD), which derives a student-anchored, teacher-guided rollout distribution from a local trust-region objective based on relative teacher support. The resulting target is a geometric mixture of the student and teacher policies, and we introduce a proposal policy that approximates this mixture, enabling rollout generation in a single forward pass. Experiments on 16 math, knowledge, and medical reasoning benchmarks show consistent gains over OPD and recent variants. On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%, suggesting that rollout policy design provides an effective way to improve OPD.

Abstract

Language Model Distillation

Explore similar work

Proposing Better Rollouts for On-Policy Distillation

From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation

When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better