In on-policy distillation (OPD), the student determines which states receive teacher supervision. As its policy evolves, earlier response prefixes become less likely even though teacher-student disagreement on them persists. Under matched trajectory and optimization budgets, neither more queries nor more frequent rollout resampling is uniformly beneficial. Current-policy rollouts outperform initial-policy rollouts at shorter response budgets, but this ranking reverses at longer budgets. Replaying initial-policy rollouts after current-policy training improves accuracy, whereas replaying fixed recent rollouts does not reproduce the gain. We propose R-OPD, a gradient-triggered curriculum that adaptively selects when to revisit initial-student trajectories. When changes in mean gradients fall within minibatch-level variation for two consecutive comparisons, training switches from the next iteration onward to initial-policy replay. Across eight mathematics benchmarks and three training-data orders, R-OPD improves average accuracy over continued current-policy sampling by 2.25/4.44 percentage points at 16K/32K for a 0.6B student and 2.81/6.15 points for a 1.7B student. With a 30B-A3B teacher, it improves 8B accuracy by 4.05 points at 32K. At 32K, R-OPD exceeds a fixed schedule of 40 current-policy updates followed by 20 replay updates by 1.39/1.66/1.85 points for 0.6B/1.7B/8B. At the same generation cap, R-OPD also produces longer responses, suggesting that well-timed revisits help students use more of their reasoning capacity.
Figures & tables
Figure 1: R-OPD follows the student, then revisits earlier states. (a) Current-policy sampling follows the evolving student; replay returns to initial-policy states. (b) Earlier prefixes become less likely while teacher–student predictions remain misaligned. (c) At the gradient-triggered iteration τ , R-OPD finishes the current-policy stage; every later iteration uses the initial-policy bank.
Training queries
Initial rollouts only
Resample every 11 steps
Resample every step
8
21.16
24.09
23.61
48
20.18
24.40
24.22
Full (14,080)
19.05
24.51
25.57
Table 1: Query count and rollout sampling interact. Average 16K accuracy (%) on AIME24, AIME25, and AMC23. All configurations use 14,080 trajectories and 110 optimization steps. Per-benchmark results and uncertainty estimates appear in Appendix D.1 .
Teacher
Current-policy advantage at 16K
Initial-policy advantage at 32K
Acc.
pass@ n
Extraction
Acc.
Qwen3-4B
+0.99
+10.38
+38.87
+4.39
JustRL-1.5B
+6.15
+16.09
+39.52
+2.07
Table 2: The relative value of rollout sources changes with response budget. Eight-benchmark average differences in percentage points: current-policy minus initial-policy at 16K, and initial-policy minus current-policy accuracy at 32K. Positive values favor the source named in the column header. The 16K results re-score prefixes of the same 32K generations. For pass@ n , n=32 on the six competition benchmarks and n=4 on MATH-500 and Olympiad-Bench. Extraction is the fraction of responses with an extractable boxed answer.
Figure 2: Rollout-source advantages depend on response budget, while earlier prefixes become less likely. (a) Initial-policy minus current-policy accuracy after 40 optimization steps, averaged equally over eight benchmarks under two teachers. Budgets re-score saved 32K responses; error bars show 95% paired-bootstrap intervals at 32K. (b) Response-weighted cumulative distributions of the change in initial-policy prefix log-probability from initialization to 20 steps (orange dashed) or 40 steps (blue solid). Negative changes mean the prefixes become less likely; the light fill marks the gap between the curves. Each response has equal weight; see Appendix E.2 for sampling details.
Comparison
Schedule
16K
32K
Tokens
Source
Current-policy sampling
26.41
26.52
8,254
Fixed recent bank
26.12
26.21
7,997
Initial-policy replay
27.13
29.37
12,116
Order
Current then initial
27.77
30.55
13,015
Initial then current
26.16
27.24
11,369
Table 3: The source and order of replay affect accuracy. Eight-benchmark mean accuracy (%) for Qwen3-0.6B. Source compares 20-step continuations after 40 current-policy steps; the fixed recent bank comes from the 40-step student. Order compares two 20-step stages after 20 current-policy steps. All endpoints use 7,200 trajectories and 60 updates. 16K re-scores saved 32K responses; Tokens is mean 32K output length. Bold marks the best accuracy within each comparison.
Student checkpoint / continuation
Initial-policy prefixes
R40 prefixes
Common starting checkpoint ( R40 )
0.448
0.427
Continue current-policy sampling
0.404
0.370
Replay fixed recent rollouts
0.404
0.370
Replay initial-policy rollouts
0.370
0.400
Table 4: Rollout source determines where KL decreases. Full-vocabulary reverse KL (nats; lower is better) on fixed diagnostic prefixes generated by the initial and 40-step students. All continuations start from the same R40 checkpoint and add 20 optimization steps. Responses are weighted equally; the full matrix is in Table 13 .
Figure 3: R-OPD across response budgets. Eight-benchmark accuracy after 60 updates: current-policy sampling versus R-OPD. Curves average three runs for 0.6B and show one run for 1.7B/8B; Table 3 a summarizes a separate three-data-order set at every size. Shorter budgets re-score saved 32K responses. Teachers: Qwen3-4B-Instruct-2507 in (a,b), Qwen3-30B-A3B-Instruct-2507 in (c).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Queries
C
U
Evaluation
Query–sampling grid
8 / 48 / Full
14,080
110
Three benchmarks
Matched rollout sources
48
4,800
40
Three and eight benchmarks
Source and order
48
7,200
60
Eight benchmarks
R-OPD (0.6B, 1.7B, 8B)
48
7,200
60
Eight benchmarks
Fixed-onset scan (0.6B)
48
7,200
60
Eight benchmarks
Appendix
Table 6: Experiment budgets and evaluation protocols. Each comparison matches consumed trajectories C and optimization steps U . Three-benchmark evaluation uses direct 16K generations; eight-benchmark evaluation uses saved 32K responses and their shorter prefixes. The matched rollout comparison reports both protocols.
Main student
Qwen3-0.6B, non-thinking
Curriculum-transfer students
Qwen3-1.7B and Qwen3-8B, non-thinking
8B curriculum teacher
Qwen3-30B-A3B-Instruct-2507
Main teacher
Qwen3-4B-Instruct-2507 (shared tokenizer)
Secondary teacher
JustRL-DeepSeek-1.5B (cross-tokenizer)
Actor/ref parameters
FP32
Teacher scoring
FP32 parameters, BF16 autocast
Appendix
Table 7: Shared training configuration. Training queries come from the filtered English subset of DAPO-Math-17K ( Yu et al., 2025 ) ; Appendix C gives the construction.
Benchmark
Questions
Responses per question
Total responses
AIME24
30
32
960
AIME25
30
32
960
AIME26
30
32
960
AMC23
83
32
2,656
HMMT25-Nov
30
32
960
HMMT26-Feb
33
32
1,056
Appendix
Table 8: Evaluation sets and sampling counts per checkpoint.
Query set
Queries
Accuracy
pass@32
Full
14,080
25.57
58.11
Difficulty-selected
48
23.98
58.51
Random
48
24.22
59.22
Appendix
Table 9: Query selection at a matched training budget. Qwen3-0.6B with the 4B teacher; all conditions use 14,080 trajectories, 110 updates, and per-step resampling. Accuracy and observed pass@32 (%) are equal-weight averages over AIME24, AIME25, and AMC23 at a direct 16K limit.
Metric
N
M
AIME24
AIME25
AMC23
Avg.
Accuracy
8
1
8.85
14.90
39.72
21.16
8
10
11.46
15.42
45.41
24.09
8
110
10.83
15.42
44.58
23.61
48
1
7.29
14.90
38.37
20.18
48
10
11.98
15.31
45.90
24.40
48
110
12.08
15.10
45.48
24.22
Appendix
Table 10: Per-benchmark results for the query-count and sampling grid (%). Avg. weights AIME24, AIME25, and AMC23 equally; N and M denote query and collection-round counts.
Figure 4: Response categories by benchmark and sampling configuration. Labels give the query count and collection rounds. Bars show the four mutually exclusive combinations of answer extraction and reaching the 16K limit; extraction counts answers regardless of correctness. The average weights the three benchmarks equally.
AIME24
AIME25
AMC23
Average
Rollout source
Acc.
pass@32
Acc.
pass@32
Acc.
pass@32
Acc.
pass@32
Extraction
Teacher: Qwen3-4B-Instruct-2507
Current-policy
11.35
50.00
15.83
40.00
43.49
81.93
23.56
57.31
66.72
Initial-policy
8.44
40.00
17.08
30.00
37.27
63.86
20.93
44.62
26.57
Teacher: JustRL-DeepSeek-1.5B
Current-policy
15.31
53.33
19.48
46.67
50.64
84.34
28.48
61.45
63.41
Appendix
Table 11: Independent direct-16K evaluation on three benchmarks (%). Current-policy mean accuracy exceeds initial-policy accuracy by 2.63 points with Qwen3 (95% interval [−0.03,5.33] ) and 6.72 with JustRL ( [3.65,9.80] ).
Figure 5: Accuracy and response length across generation budgets. Each curve plots eight-benchmark mean accuracy against mean response length at 4K, 8K, 16K, 24K, and 32K; filled markers denote 32K. Color and line style identify the rollout source, and the panels separate the two teachers. The horizontal axis uses decimal thousands of tokens. Per-benchmark accuracy differences and 32K length ratios appear in Table 12 ; all aggregates weight benchmarks equally.
Teacher
Benchmark
4K
8K
16K
24K
32K
Length ratio at 32K
Qwen3-4B
AIME24
-4.06
-3.23
-1.56
+2.40
+6.77
1.76×
AIME25
-4.06
-3.85
+1.25
+3.65
+5.94
1.82×
AIME26
-3.12
-2.60
+0.94
+1.04
+2.71
1.67×
AMC23
-11.11
-6.36
-4.48
-0.94
+4.33
1.85×
HMMT25-Nov
-1.25
-1.15
+0.10
+2.50
+4.06
1.79×
HMMT26-Feb
-2.75
-3.22
-2.37
-0.28
+2.08
1.83×
Appendix
Table 12: Accuracy differences and 32K response-length ratios by benchmark. Accuracy differences are initial-policy minus current-policy in percentage points. Positive values favor initial-policy training. Ratios are initial-policy/current-policy mean response length at 32K. Scores use prefixes of saved 32K responses; sample counts and benchmark weighting follow Appendix B.1 .
Start
Scoring student / continuation
Steps
Initial
R20
R40
—
Initial student
0
0.774
0.624
0.628
R20
20
0.548
0.505
0.523
R40
40
0.448
0.409
0.427
R40
Continue current-policy sampling
60
0.404
0.361
0.370
Replay fixed recent rollouts
60
0.404
0.360
0.370
Replay initial-policy rollouts
60
0.370
0.378
0.400
Appendix
Table 13: Complete KL matrix on fixed diagnostic prefixes. Full-vocabulary reverse KL (nats; lower is better), Ei,j in Equation 7 , for the Qwen3-0.6B student and fixed Qwen3-4B-Instruct-2507 teacher. Rows specify the student used for scoring; columns specify the student that generated the diagnostic responses. Each column uses the same 384 responses and sampled positions across all scoring students, averaging positions within each response and then weighting responses equally. Steps denote total optimization steps; Start identifies the checkpoint restored for each 20-step continuation.
Training history
4K
8K
16K
24K
32K
Tokens
Initial → initial
12.85
18.66
25.24
28.93
30.31
21,357
Initial → current
17.67
22.87
26.62
27.58
27.71
11,166
Current → initial
14.23
20.71
27.13
30.06
29.37
12,116
Current → current
17.94
23.06
26.41
26.52
26.52
8,254
Current → fixed recent
18.15
23.10
26.12
26.20
26.21
7,997
Appendix
Table 14: Continuation source after 40 optimization steps. Eight-benchmark average accuracy (%) across response budgets. All endpoints have 60 steps; mean response length uses complete 32K responses. Comparisons sharing a first-stage source restore the same model and optimizer. The fixed recent bank is generated by the 40-step current-policy student.
Benchmark
Initial then initial
Initial then current
Current then initial
Current then current
16K retained prefixes
AIME24
10.10
11.04
11.15
12.71
AIME25
16.67
16.25
16.15
16.77
AIME26
13.23
12.92
13.02
10.83
AMC23
38.48
42.73
43.83
44.69
HMMT26-Feb
5.68
9.09
8.71
8.43
Appendix
Table 15: Per-benchmark accuracy for the four continuations (%). Each column lists the first 40 and final 20 steps, as defined in Table 14 . Average weights benchmarks equally and uses unrounded values.
Condition
Steps
4K
8K
16K
24K
32K
Tokens
Current only
40
17.28
21.86
25.86
27.18
27.44
12,011
Replay after 20 steps
40
11.20
17.10
24.40
28.89
32.02
22,343
Appendix
Table 16: Early replay at a matched 40-update budget. Eight-benchmark average accuracy (%) across response budgets. Current only is R40 ; early replay adds 20 initial-policy steps to R20 . Mean response length uses complete 32K responses.
Benchmark
4K
8K
16K
32K
AIME24
-1.67
-1.77
+1.46
+4.90
AIME25
-4.69
-2.29
+1.04
+0.94
AIME26
-3.54
-1.67
+2.40
+3.33
AMC23
-6.40
-3.84
+2.52
+6.59
HMMT25-Nov
-0.31
+1.04
+1.46
+2.29
HMMT26-Feb
-1.42
-1.04
+0.76
+2.27
Appendix
Table 17: Per-benchmark advantage of current-policy training followed by initial-policy replay. Values are accuracy differences in percentage points relative to the reverse order; positive values favor current then initial. The average weights benchmarks equally.
Continuation
16K
32K
Tokens at 32K
Full initial-policy replay
24.40
32.02
22,343
Equal-response mix
26.38
28.25
12,346
Source-balanced mix
27.00
30.09
14,445
Replay: no EOS
25.07
31.88
22,111
Replay: EOS only
23.64
25.57
13,870
Appendix
Table 18: Loss controls for 20 replay-stage updates after R20 . Eight-benchmark accuracy (%) and mean 32K output length. All endpoints use 40 updates in total. Full replay is the early-replay baseline in Table 16 ; the mixtures use equal response counts per source.
Condition
exact ∥g∥2
Cosine vs. current-policy
θ0 current-policy
19.426±8.018
—
θ55 current-policy
5.419±2.525
—
θ55 initial-policy
13.717±4.781
0.127(0.092)
θ110 current-policy
5.061±3.090
—
θ110 initial-policy
14.261±5.043
0.034(0.078)
Appendix
Table 19: Gradients on fixed model parameters. Means and standard deviations are over 16 diagnostic minibatches. Norms use all parameters; cross-source cosine uses a fixed subset of gradient coordinates. Parentheses give its standard error over paired minibatches.
Quantity
Value
0.6B / 1.7B teacher
Qwen3-4B-Instruct-2507
8B teacher
Qwen3-30B-A3B-Instruct-2507
Query set
48 random queries; ten occurrences each per iteration
Iterations / optimization steps
15 / 60
Responses per iteration / initial bank
480 / five batches of 480
Trigger threshold / persistence
D≤V / two consecutive comparisons
Appendix
Table 20: Curriculum settings shared across student sizes. Each student generates its own initial-policy bank and measures its own gradients.
Student
Run
τ
16K accuracy (%)
32K accuracy (%)
Current
Fixed
R-OPD
Current
Fixed
R-OPD
0.6B
1
5
26.41
27.13
28.96
26.52
29.37
31.52
2
5
25.94
27.52
28.38
26.07
29.62
30.48
3
7
26.77
27.49
28.53
26.91
29.68
30.83
1.7B
1
7
38.97
41.21
41.92
39.13
43.74
45.14
2
6
38.73
41.27
41.65
38.86
43.62
45.43
Appendix
Table 21: Paired accuracy and R-OPD trigger by training-data order. Eight-benchmark average accuracy (%); each row gives one training-data shuffle of the same 48 queries. τ is in training iterations.
Student
Method
4K
8K
16K
24K
32K
Tokens at 32K
0.6B
Current
17.94
23.06
26.41
26.52
26.52
8,254
Fixed
14.23
20.71
27.13
30.06
29.37
12,116
R-OPD
14.72
21.61
28.96
31.28
31.52
13,842
1.7B
Current
29.06
35.59
38.97
39.13
39.13
6,887
Fixed
27.11
34.52
41.21
44.44
43.74
11,025
R-OPD
26.77
34.66
41.92
44.61
45.14
11,980
Appendix
Table 22: Individual-run accuracy across response budgets. Eight-benchmark mean accuracy (%) and mean 32K output length for one trained endpoint per method and student size. Fixed denotes R40F20 . Teachers: Qwen3-4B-Instruct-2507 for 0.6B/1.7B; Qwen3-30B-A3B-Instruct-2507 for 8B.
Student
Benchmark
16K
32K
Current
Fixed
R-OPD
Current
Fixed
R-OPD
0.6B
AIME24
12.71
11.15
13.33
12.71
15.10
16.56
AIME25
16.77
16.15
18.44
16.88
16.88
19.90
AIME26
10.83
13.02
15.31
10.94
14.48
17.29
AMC23
44.69
43.83
46.69
44.88
48.61
51.77
HMMT25-Nov
5.73
6.67
7.40
5.83
6.77
8.85
Appendix
Table 23: Benchmark variation in the individual runs. Accuracy (%) for the endpoints in Table 22 . Fixed denotes R40F20 . Average weights the eight benchmarks equally.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the current student. We show that this choice can create a compounding teacher--student mismatch, because it requires the teacher to provide effective supervision on whichever trajectories the student samples, including ones less reachable by the teacher itself. On such trajectories, teacher supervision can become noisy or even misleading, leading to undesirable student patterns such as repetition. Moreover, since the updated student in turn defines the rollout policy of subsequent rounds, these patterns persist and potentially amplify over training. This failure mode suggests that improving OPD requires treating the rollout policy itself as an optimization target, rather than only modifying the distillation objective. We propose Mixture Proposal On-Policy Distillation (MPOPD), which derives a student-anchored, teacher-guided rollout distribution from a local trust-region objective based on relative teacher support. The resulting target is a geometric mixture of the student and teacher policies, and we introduce a proposal policy that approximates this mixture, enabling rollout generation in a single forward pass. Experiments on 16 math, knowledge, and medical reasoning benchmarks show consistent gains over OPD and recent variants. On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%, suggesting that rollout policy design provides an effective way to improve OPD.
Sirui Chen, Yunzhe Qi, Haobo Xu +4
University of Illinois Urbana-Champaign · Stanford University
On-policy distillation (OPD) trains a student on its own reasoning trajectories using feedback from a stronger teacher. Teacher interventions can improve these trajectories, but also change the distribution on which the student learns. Our controlled studies show that rollout quality alone is an incomplete criterion for allocating teacher guidance. Deeper intervention yields diminishing gains in rollout accuracy while increasing off-policy load. In a training probe with a restricted rollout horizon, peak student accuracy and performance retention favor different intervention strengths. The preferred intervention depth and placement also vary across benchmarks. These findings motivate MAESTRO, which uses local policy disagreement to jointly adapt when the teacher takes over and how long it generates. Its {policy disagreement score} combines teacher-weighted candidate coverage with local distribution similarity and is aggregated within reasoning paragraphs. Across eight mathematical reasoning benchmarks, MAESTRO achieves the highest macro-average accuracy among the compared methods for both 0.6B and 1.7B Qwen3 students, with the 1.7B student leading on every benchmark. MAESTRO also reduces average training response length by 67.3% relative to standard OPD. The code is available at https://github.com/yhao-wang/MAESTRO.
Yuhao Wang, Ruiyang Ren, Yinan Zhang +3
Nanyang Technological University, Singapore · Baidu Inc.
On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.