On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its reliance on frequently refreshed student rollouts often incurs substantial generation cost. We introduce REVO, an off-policy distillation framework that improves rollout efficiency by reusing each student rollout for multi-step learner updates. REVO addresses prefix-level and current-token policy mismatch through stabilized prefix weighting and one-step resampling from the current student, which enables repeated updates without regenerating full trajectories. To prioritize informative token positions within reused rollouts, REVO uses the variance of the student-teacher log-probability ratio to quantify the remaining token-level learning signal and guide repeated optimization. Across multiple student-teacher scales, REVO with only 50 rollout iterations matches or exceeds OPD baselines trained for 200 iterations on both in-domain and cross-domain reasoning benchmarks.
Figures & tables
Figure 1: Overview of REVO. One-step resampling refreshes token-level teacher supervision at stored prefixes, while stabilized prefix weights and variance-guided token priorities support repeated updates.
Figure 2: Rollout efficiency of REVO. For Qwen3-1.7B Base, REVO at 50 training steps surpasses standard OPD at 200 steps, measured by mean Avg@16 over AIME24, AIME25, and AMC23.
Figure 3: High- versus low-RKL-variance updates under REVO for Qwen3-1.7B Base → 4B ( δ=0.005 ). Accuracy is mean Avg@16 on AIME24, AIME25, and AMC23.
Figure 4: RKL variance and gradient reliability across 1,280 stored prefixes. Exact Et is strongly rank-correlated with the logit-space gradient SNR of the K=16 resampling estimator, with Spearman’s ρ=0.93 .
Avg@16
Avg@4
Method
Step
AIME24
AIME25
AMC23
MATH-500
OlympiadBench
Minerva
Mean
Student: 0.6B Base Teacher: 1.7B
Base
0
0.0
0.0
1.4
3.3
1.7
2.3
1.5
OPD
50
0.8
1.0
18.4
45.2
14.8
11.0
15.2
OPD
200
2.1
1.0
23.3
48.4
17.9
10.0
17.1
EOPD
50
1.5
0.6
18.8
45.0
13.3
10.7
15.0
Table 1: Results across three student–teacher scales. AIME24, AIME25, and AMC23 use Avg@16; MATH-500, OlympiadBench, and Minerva use Avg@4. Mean is the average across the six benchmarks. Best results within each scale are bold; blue shading highlights REVO. Green/red subscripts indicate gains/losses in percentage points relative to OPD at Step 50 within the same scale.
Student: 0.6B Base Teacher: 1.7B
Student: 1.7B Base Teacher: 4B
Student: 4B Base Teacher: 8B
Benchmark
REVO
OPD
EOPD
FiRe
REVO
OPD
EOPD
FiRe
REVO
OPD
EOPD
FiRe
GPQA@4
20.2
13.8
18.1
13.3
27.3
17.5
20.8
16.8
40.0
38.5
37.7
38.8
HumanEval@4
32.6
30.8
29.7
31.6
54.3
51.2
52.3
49.2
77.9
75.8
77.0
75.2
MMLU-Redux@1
48.0
46.9
46.9
45.0
65.2
63.2
63.8
62.1
79.6
79.0
80.1
79.5
Mean
33.6
30.5
31.6
29.9
48.9
44.0
45.6
42.7
65.8
64.4
64.9
64.5
Table 2: Cross-domain evaluation across three student–teacher scales. All experiments distill a larger post-trained teacher into a smaller Qwen3 Base student. REVO is evaluated at Step 50, while OPD, EOPD, and FiRe are evaluated at Step 200. Mean is the unweighted average across the three benchmarks. Best results within each scale are bold; blue shading highlights REVO.
Method
Avg@16
OPD
15.1
OPD(U10)
15.0
OPD(U10) + PPO-Style Clipped IS
16.7
OPD(U10) + One-Step Resampling
17.8
+ Prefix Correction
18.2
RKL-variance reweighting
Table 3: Ablation study using a Qwen3-1.7B Base student distilled from a Qwen3-4B teacher. Results are reported at Step 50 as Avg@16 over AIME24, AIME25, and AMC23. PPO-style clipped IS and one-step resampling are alternative current-token corrections. Prefix correction and RKL-variance weighting are added to the one-step resampling variant.
Figure 5: Effect of the number of learner updates per rollout batch. Increasing the reuse rate accelerates early learning, while both configurations eventually reach similar performance.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Offline diagnostic of RKL variance and finite-sample gradient reliability. We collect 1280 student-visited prefix states and store the full-vocabulary student and teacher conditional distributions, from which the RKL variance, squared logit-gradient signal, and sampling variance of the K=16 one-step resampling estimator are computed offline. Left: RKL variance is strongly rank-correlated with gradient SNR ( ρ=0.93 ), with the approximate scaling Γt∝Et1.12 . Right: the squared gradient signal scales approximately as ∥ut∥22∝Et2.06 , while the estimator variance scales as σt2∝Et0.94 .
Figure 7: Distribution of the one-step resampling RKL-variance estimator EtK . A substantial fraction of positions have zero or near-zero estimated variance, while the nonzero values span several orders of magnitude and exhibit a long right tail. The inset reports the probability mass at exactly zero and in (0,10−8) .
Llama 3.2 3B
Gemma 3 4B
Method
AMC23
MATH-500
MMLU-Redux
AMC23
MATH-500
MMLU-Redux
Base
0.3
0.3
6.3
0.3
0.5
11.5
Sampled-Token OPD
1.9
3.4
12.5
13.4
34.7
53.4
EOPD
1.1
2.4
6.7
12.8
33.9
27.1
FiRe
1.7
2.6
9.2
12.5
33.1
48.6
REVO
6.4
8.9
28.3
16.9
37.7
54.1
Appendix
Table 4: Evaluation on additional model families. AMC23, MATH-500, and MMLU-Redux are evaluated with Avg@16, Avg@4, and Avg@1, respectively. REVO is evaluated at Step 50, while all other methods are evaluated at Step 200. The highest score in each column is shown in bold.
ai,t(1),…,ai,t(K)∼i.i.d.Pθ(⋅∣ci,t)
Appendix
Algorithm 1 Training procedure for REVO
Figure 8: One-step resampling in OPD without rollout reuse. (a) Rollouts use T=1 , top- p=1 . (b) Rollouts use the evaluation setting T=0.7 , top- p=0.95 . Resampling gives a modest gain in (a) and substantially mitigates the degradation in (b). Results are Avg@16 over AIME24, AIME25, and AMC23.
Category
Hyperparameter
Setting
Method
Candidate samples K
16
RKL-variance threshold δ
0.005
Reweighting coefficient τ
0.75
Weight normalization
Mean 1 over valid tokens
Prefix-ratio cap C
4
Training
Training data
DAPO-Math-17K
Appendix
Table 5: Implementation details and hyperparameters used in the main experiments.
K
1
4
16
64
Avg@16
17.6
17.8
17.8
18.1
Appendix
Table 6: Effect of the resampling size K for a Qwen3-1.7B Base student distilled from a Qwen3-4B teacher, evaluated with Avg@16 over AIME24, AIME25, and AMC23, with other REVO components disabled.
Varying τ ( δ=0.005 )
Varying δ ( τ=0.75 )
τ
Avg@16
δ
Avg@16
0.60
18.8
0.001
19.3
0.70
19.7
0.005 †
19.6
0.75 †
19.6
0.050
18.7
0.80
19.1
0.200
18.9
Appendix
Table 7: Hyperparameter sensitivity of RKL-variance reweighting for a Qwen3-1.7B Base student distilled from a Qwen3-4B teacher. Results are Avg@16 over AIME24, AIME25, and AMC23. † denotes the default configuration.
Prioritization signal
Threshold
High-weight fraction
Avg@16
Uniform weighting
–
–
18.2
Sampled KL
0.02
34.0%
17.3
Sampled entropy
0.27
32.6%
16.5
RKL variance (REVO)
0.005
32.8%
19.6
Appendix
Table 8: Comparison of token-level prioritization signals for a Qwen3-1.7B Base student distilled from a Qwen3-4B teacher. The high-weight fraction is the fraction of valid token positions assigned weight τ before normalization during training. Results are Avg@16 over AIME24, AIME25, and AMC23.
Component
Time (s)
Share
Student rollout
128.7
32.4%
Initial student/teacher scoring
15.5
3.9%
9 additional student log-prob evaluations
52.1
13.1%
9 additional teacher scoring passes
33.8
8.5%
9 additional resampling signal constructions ( K=16 )
14.2
3.6%
10 actor forward/backward/update passes
100.0
25.2%
Appendix
Table 9: Per-step wall-clock breakdown of REVO with 10 learner updates per rollout batch (U10), measured using a Qwen3-1.7B Base student and Qwen3-4B teacher on 4× NVIDIA A100-SXM4-80GB GPUs.
Method
Rollout iterations
Prompts per iteration
Relative rollout budget
Avg@16
REVO
50
8
1×
19.6
OPD
200
8
4×
16.8
OPD
200
16
8×
17.5
OPD
200
32
16×
18.4
Appendix
Table 10: Effect of rollout batch size for OPD with a Qwen3-1.7B Base student distilled from a Qwen3-4B teacher. Relative rollout budget is normalized to the default REVO setting. Results are Avg@16 over AIME24, AIME25, and AMC23.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the current student. We show that this choice can create a compounding teacher--student mismatch, because it requires the teacher to provide effective supervision on whichever trajectories the student samples, including ones less reachable by the teacher itself. On such trajectories, teacher supervision can become noisy or even misleading, leading to undesirable student patterns such as repetition. Moreover, since the updated student in turn defines the rollout policy of subsequent rounds, these patterns persist and potentially amplify over training. This failure mode suggests that improving OPD requires treating the rollout policy itself as an optimization target, rather than only modifying the distillation objective. We propose Mixture Proposal On-Policy Distillation (MPOPD), which derives a student-anchored, teacher-guided rollout distribution from a local trust-region objective based on relative teacher support. The resulting target is a geometric mixture of the student and teacher policies, and we introduce a proposal policy that approximates this mixture, enabling rollout generation in a single forward pass. Experiments on 16 math, knowledge, and medical reasoning benchmarks show consistent gains over OPD and recent variants. On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%, suggesting that rollout policy design provides an effective way to improve OPD.
Sirui Chen, Yunzhe Qi, Haobo Xu +4
University of Illinois Urbana-Champaign · Stanford University
On-policy distillation (OPD) transfers teacher knowledge to a student through feedback on student-generated rollouts, but generating complete rollouts and computing per-token teacher feedback are computationally expensive. Existing truncation methods save computation by stopping rollouts early. However, they do not compensate for truncated token gradients, which biases the gradient estimates and changes Vanilla OPD's original training objective. Furthermore, their truncation decisions rely on assumptions about supervision quality or criteria that may not generalize across settings. We introduce Stochastic Truncated On-Policy Distillation (STOPD), which accelerates OPD by randomly truncating student rollouts and correcting the resulting sampling bias with blockwise inverse-probability weighting (IPW). At each training step, we sample a stopping position for each rollout before generation and let the student generate until that position or natural termination. We obtain teacher feedback on every generated token and use blockwise IPW to compensate for the lower probability of retaining later tokens while controlling the weights. We prove that this weighting preserves Vanilla OPD's gradient sum in expectation and derive one-step update-error bounds for SGD, Adam, and Muon. We evaluate STOPD across multiple teacher-student model pairs, task domains, and optimizers. STOPD reduces generated tokens by 25-48% while maintaining accuracy comparable to Vanilla OPD, achieving 1.3-2.2x training speedups.
On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at https://github.com/HancCui/opd_hacking.