REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse

Yuxiao Yang; Shangzhe Li; Tianrun Yu; Kaixiang Zhao; Taylor W. Killian; Weitong Zhang

cs.LGSep 28, 2026

REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse

Authors: Yuxiao Yang, Shangzhe Li, Tianrun Yu, Kaixiang Zhao, Taylor W. Killian, Weitong Zhang

Organizations: University of North Carolina at Chapel Hill · Brigham Young University

Abstract

On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its reliance on frequently refreshed student rollouts often incurs substantial generation cost. We introduce REVO, an off-policy distillation framework that improves rollout efficiency by reusing each student rollout for multi-step learner updates. REVO addresses prefix-level and current-token policy mismatch through stabilized prefix weighting and one-step resampling from the current student, which enables repeated updates without regenerating full trajectories. To prioritize informative token positions within reused rollouts, REVO uses the variance of the student-teacher log-probability ratio to quantify the remaining token-level learning signal and guide repeated optimization. Across multiple student-teacher scales, REVO with only 50 rollout iterations matches or exceeds OPD baselines trained for 200 iterations on both in-domain and cross-domain reasoning benchmarks.

Figures & tables

Figure 1: Overview of REVO. One-step resampling refreshes token-level teacher supervision at stored prefixes, while stabilized prefix weights and variance-guided token priorities support repeated updates.

Figure 2: Rollout efficiency of REVO. For Qwen3-1.7B Base, REVO at 50 training steps surpasses standard OPD at 200 steps, measured by mean Avg@16 over AIME24, AIME25, and AMC23.

Figure 3: High- versus low-RKL-variance updates under REVO for Qwen3-1.7B Base $\rightarrow$ 4B ( $\delta=0.005$ ). Accuracy is mean Avg@16 on AIME24, AIME25, and AMC23.

Figure 4: RKL variance and gradient reliability across 1,280 stored prefixes. Exact $E_{t}$ is strongly rank-correlated with the logit-space gradient SNR of the $K=16$ resampling estimator, with Spearman’s $\rho=0.93$ .

		Avg@16			Avg@4
Method	Step	AIME24	AIME25	AMC23	MATH-500	OlympiadBench	Minerva	Mean
Student: 0.6B Base Teacher: 1.7B
Base	0	0.0	0.0	1.4	3.3	1.7	2.3	1.5
OPD	50	0.8	1.0	18.4	45.2	14.8	11.0	15.2
OPD	200	2.1	1.0	23.3	48.4	17.9	10.0	17.1
EOPD	50	1.5	0.6	18.8	45.0	13.3	10.7	15.0

Table 1: Results across three student–teacher scales. AIME24, AIME25, and AMC23 use Avg@16; MATH-500, OlympiadBench, and Minerva use Avg@4. Mean is the average across the six benchmarks. Best results within each scale are bold; blue shading highlights REVO. Green/red subscripts indicate gains/losses in percentage points relative to OPD at Step 50 within the same scale.

	Student: 0.6B Base Teacher: 1.7B				Student: 1.7B Base Teacher: 4B				Student: 4B Base Teacher: 8B
Benchmark	REVO	OPD	EOPD	FiRe	REVO	OPD	EOPD	FiRe	REVO	OPD	EOPD	FiRe
GPQA@4	20.2	13.8	18.1	13.3	27.3	17.5	20.8	16.8	40.0	38.5	37.7	38.8
HumanEval@4	32.6	30.8	29.7	31.6	54.3	51.2	52.3	49.2	77.9	75.8	77.0	75.2
MMLU-Redux@1	48.0	46.9	46.9	45.0	65.2	63.2	63.8	62.1	79.6	79.0	80.1	79.5
Mean	33.6	30.5	31.6	29.9	48.9	44.0	45.6	42.7	65.8	64.4	64.9	64.5

Table 2: Cross-domain evaluation across three student–teacher scales. All experiments distill a larger post-trained teacher into a smaller Qwen3 Base student. REVO is evaluated at Step 50, while OPD, EOPD, and FiRe are evaluated at Step 200. Mean is the unweighted average across the three benchmarks. Best results within each scale are bold; blue shading highlights REVO.

RKL-variance reweighting
Method	Avg@16
OPD	15.1
OPD(U10)	15.0
OPD(U10) + PPO-Style Clipped IS	16.7
OPD(U10) + One-Step Resampling	17.8
+ Prefix Correction	18.2

Table 3: Ablation study using a Qwen3-1.7B Base student distilled from a Qwen3-4B teacher. Results are reported at Step 50 as Avg@16 over AIME24, AIME25, and AMC23. PPO-style clipped IS and one-step resampling are alternative current-token corrections. Prefix correction and RKL-variance weighting are added to the one-step resampling variant.

Figure 5: Effect of the number of learner updates per rollout batch. Increasing the reuse rate accelerates early learning, while both configurations eventually reach similar performance.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Offline diagnostic of RKL variance and finite-sample gradient reliability. We collect $1280$ student-visited prefix states and store the full-vocabulary student and teacher conditional distributions, from which the RKL variance, squared logit-gradient signal, and sampling variance of the $K{=}16$ one-step resampling estimator are computed offline. Left: RKL variance is strongly rank-correlated with gradient SNR ( $\rho=0.93$ ), with the approximate scaling $\Gamma_{t}\propto E_{t}^{1.12}$ . Right: the squared gradient signal scales approximately as $\|u_{t}\|_{2}^{2}\propto E_{t}^{2.06}$ , while the estimator variance scales as $\sigma_{t}^{2}\propto E_{t}^{0.94}$ .

Figure 7: Distribution of the one-step resampling RKL-variance estimator $\widehat{E}_{t}^{K}$ . A substantial fraction of positions have zero or near-zero estimated variance, while the nonzero values span several orders of magnitude and exhibit a long right tail. The inset reports the probability mass at exactly zero and in $(0,10^{-8})$ .

	Llama 3.2 3B			Gemma 3 4B
Method	AMC23	MATH-500	MMLU-Redux	AMC23	MATH-500	MMLU-Redux
Base	0.3	0.3	6.3	0.3	0.5	11.5
Sampled-Token OPD	1.9	3.4	12.5	13.4	34.7	53.4
EOPD	1.1	2.4	6.7	12.8	33.9	27.1
FiRe	1.7	2.6	9.2	12.5	33.1	48.6
REVO	6.4	8.9	28.3	16.9	37.7	54.1

Appendix

Table 4: Evaluation on additional model families. AMC23, MATH-500, and MMLU-Redux are evaluated with Avg@16, Avg@4, and Avg@1, respectively. REVO is evaluated at Step 50, while all other methods are evaluated at Step 200. The highest score in each column is shown in bold.

a_{i,t}^{(1)},\ldots,a_{i,t}^{(K)}\overset{\mathrm{i.i.d.}}{\sim}P_{\theta}(\cdot\mid c_{i,t})

Appendix

Algorithm 1 Training procedure for REVO

Figure 8: One-step resampling in OPD without rollout reuse. (a) Rollouts use $T=1$ , top- $p=1$ . (b) Rollouts use the evaluation setting $T=0.7$ , top- $p=0.95$ . Resampling gives a modest gain in (a) and substantially mitigates the degradation in (b). Results are Avg@16 over AIME24, AIME25, and AMC23.

Category	Hyperparameter	Setting
Method	Candidate samples $K$	16
	RKL-variance threshold $\delta$	0.005
	Reweighting coefficient $\tau$	0.75
	Weight normalization	Mean 1 over valid tokens
	Prefix-ratio cap $C$	4
Training	Training data	DAPO-Math-17K

Appendix

Table 5: Implementation details and hyperparameters used in the main experiments.

$K$	1	4	16	64
Avg@16	17.6	17.8	17.8	18.1

Appendix

Table 6: Effect of the resampling size

K

for a Qwen3-1.7B Base student distilled from a Qwen3-4B teacher, evaluated with Avg@16 over AIME24, AIME25, and AMC23, with other REVO components disabled.

Varying $\tau$ ( $\delta=0.005$ )		Varying $\delta$ ( $\tau=0.75$ )
$\tau$	Avg@16	$\delta$	Avg@16
0.60	18.8	0.001	19.3
0.70	19.7	0.005 †	19.6
0.75 †	19.6	0.050	18.7
0.80	19.1	0.200	18.9

Appendix

Table 7: Hyperparameter sensitivity of RKL-variance reweighting for a Qwen3-1.7B Base student distilled from a Qwen3-4B teacher. Results are Avg@16 over AIME24, AIME25, and AMC23.

\dagger

denotes the default configuration.

Prioritization signal	Threshold	High-weight fraction	Avg@16
Uniform weighting	–	–	18.2
Sampled KL	0.02	34.0%	17.3
Sampled entropy	0.27	32.6%	16.5
RKL variance (REVO)	0.005	32.8%	19.6

Appendix

Table 8: Comparison of token-level prioritization signals for a Qwen3-1.7B Base student distilled from a Qwen3-4B teacher. The high-weight fraction is the fraction of valid token positions assigned weight

\tau

before normalization during training. Results are Avg@16 over AIME24, AIME25, and AMC23.

Component	Time (s)	Share
Student rollout	128.7	32.4%
Initial student/teacher scoring	15.5	3.9%
9 additional student log-prob evaluations	52.1	13.1%
9 additional teacher scoring passes	33.8	8.5%
9 additional resampling signal constructions ( $K=16$ )	14.2	3.6%
10 actor forward/backward/update passes	100.0	25.2%

Appendix

Table 9: Per-step wall-clock breakdown of REVO with 10 learner updates per rollout batch (U10), measured using a Qwen3-1.7B Base student and Qwen3-4B teacher on

4\times

NVIDIA A100-SXM4-80GB GPUs.

Method	Rollout iterations	Prompts per iteration	Relative rollout budget	Avg@16
REVO	50	8	$1\times$	19.6
OPD	200	8	$4\times$	16.8
OPD	200	16	$8\times$	17.5
OPD	200	32	$16\times$	18.4

Appendix

Table 10: Effect of rollout batch size for OPD with a Qwen3-1.7B Base student distilled from a Qwen3-4B teacher. Relative rollout budget is normalized to the default REVO setting. Results are Avg@16 over AIME24, AIME25, and AMC23.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 514 papers

Proposing Better Rollouts for On-Policy Distillation
Oct 27, 2026Sirui Chen, Yunzhe Qi, Haobo Xu +4Language Model Distillation On-Policy Distillation
University of Illinois Urbana-Champaign · Stanford University
Does On-Policy Distillation Need to Complete Every Rollout?
Oct 15, 2026Zhengxin Zhang, Maosen Tang, Ning Wang +6On-Policy Distillation Language Model Distillation
Cornell University · Georgia Tech · Alibaba Group
Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
Oct 2, 2026Han Cui, Jianhao Yan, Yun Luo +3On-Policy Distillation Language Model Distillation
Westlake University

Oct 27, 2026J/K move · Enter open · S save

Proposing Better Rollouts for On-Policy Distillation

Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou+3

University of Illinois Urbana-Champaign · Stanford University

Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the current student. We show that this choice can create a compounding teacher--student mismatch, because it requires the teacher to provide effective supervision on whichever trajectories the student samples, including ones less reachable by the teacher itself. On such trajectories, teacher supervision can become noisy or even misleading, leading to undesirable student patterns such as repetition. Moreover, since the updated student in turn defines the rollout policy of subsequent rounds, these patterns persist and potentially amplify over training. This failure mode suggests that improving OPD requires treating the rollout policy itself as an optimization target, rather than only modifying the distillation objective. We propose Mixture Proposal On-Policy Distillation (MPOPD), which derives a student-anchored, teacher-guided rollout distribution from a local trust-region objective based on relative teacher support. The resulting target is a geometric mixture of the student and teacher policies, and we introduce a proposal policy that approximates this mixture, enabling rollout generation in a single forward pass. Experiments on 16 math, knowledge, and medical reasoning benchmarks show consistent gains over OPD and recent variants. On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%, suggesting that rollout policy design provides an effective way to improve OPD.

Abstract

Language Model Distillation

Explore similar work

Proposing Better Rollouts for On-Policy Distillation

Does On-Policy Distillation Need to Complete Every Rollout?

Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective