Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation

Harrison Zhao; Lei Huang; Fei Liu; Wen Shi; Zijian Jin

cs.AIOct 19, 2026

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation

Authors: Harrison Zhao, Lei Huang, Fei Liu, Wen Shi, Zijian Jin

Organizations: Meta

Abstract

On-policy self-distillation (OPSD) supervises student-generated trajectories with a reference-conditioned version of the same base model. We ask whether every trajectory supplies distinct supervision that warrants its distillation cost. Gradient measurements on Qwen3-1.7B at initialization reveal that correct and incorrect responses have similar individual gradient magnitudes, but incorrect responses have approximately three times the coherent gradient share. Their estimated population-mean directions also largely overlap. A signal--noise model fitted at a small group size tracks alignment at larger group sizes without refitting. Motivated by these measurements, we introduce correctness-filtered OPSD (C-OPSD), which generates one response per problem and distills only a subset of incorrect responses. Removing unselected rows before both distillation forwards avoids their teacher/student forward and student backward work. Across Qwen3-1.7B, 4B, and 8B on AIME24, AIME25, and HMMT25, C-OPSD achieves on-par performance with full OPSD. In a measured micro-batch-size-four implementation, executed rows decrease by approximately 68%, and with retained rows packed into full micro-batches, training takes about 40% less time while rollout generation remains unchanged. Together, the measurements and training results connect outcome-conditioned supervision redundancy to more compute-efficient distillation.

Figures & tables

Figure 1: Outcome-conditioned gradient measurements. Qwen3-1.7B at initialization, using 3,000 trajectories and base-weight gradients. (a) Fitted coherent shares are 6.0% and 18.6%, despite similar individual gradient magnitudes. (b) The response-weighted mixture

\mu

full = p

\mu

C + (1

-

\mu

I , with p = 0.291, is approximately 2.87 ∘ from the incorrect mean direction. Vectors use measured mean magnitudes and the noise-corrected mean-direction cosine; the tip inset is an equal-axis 3

\times

enlargement. These angles describe estimated population means. The following subsections give the measurement procedure and derivation.

Population	Trajectories	Mean p 0	p 0 = 0	p 0 $\geq$ 0.875
All	2,027	0.34	45.8%	21.5%
Correct	680	0.80	0.0%	62.1%
Incorrect	1,347	0.11	68.9%	1.0%

Table 1: Offline pass rate p 0 of the problems behind the measured trajectories, computed on the 2,027 of 3,000 trajectories whose problems appear in the probe.

Population	Within-population cosine	Mean-gradient magnitude $\\|$ $\mu$ $\\|$	Noise scale $\sigma$	Coherent share q
Correct	0.2042	0.3776	1.4911	6.0%
Incorrect	0.4778	0.6339	1.3253	18.6%

Table 2: Signal–noise parameters fitted at group size four.

Figure 2: Measured alignment and fixed-model predictions. Markers show within-population gradient cosine; curves use the parameters fitted at group size four, without refitting at larger sizes. Hollow markers identify the fit group size, and filled markers show measurements at sizes 16, 64, and 218. The right panel shows measured minus predicted cosine. Group size counts trajectories aggregated for analysis, not rollouts per problem. All measurements are from Qwen3-1.7B at initialization.

Group size	Correct: predicted	Correct: measured	Incorrect: predicted	Incorrect: measured
16	0.506	0.521	0.785	0.790
64	0.804	0.800	0.936	0.940
218	0.933	0.934	0.980	0.982

Table 3: Predicted and measured within-population cosine at different group sizes.

Checkpoint	Correct q	Incorrect q	Ratio	Corrected cross cosine
Initialization	6.4%	19.1%	2.99	0.975
Step 25	8.3%	25.9%	3.12	0.980
Step 50	11.3%	33.2%	2.94	0.941
Step 100	11.5%	33.9%	2.95	0.943

Table 4: Coherent share and noise-corrected cross-population cosine during C-OPSD training, group size 16.

Figure 3: One rollout per problem and physically compacted distillation batches. Full OPSD processes all generated trajectories. C-OPSD checks correctness, retains s = min(k, n

-

k) incorrect responses, and removes unselected rows before both distillation forwards. The example n = 4, k = 1 produces one-row tensors rather than four-row tensors with masked entries. The reference-conditioned teacher is frozen; only the student is updated. Generation is unchanged. Empty selections execute one fully masked fallback row.

Arm	Retained trajectories	Count for the same batch	Role
Full OPSD	All trajectories	n	Performance reference
Correct-only	All correct trajectories	k	Opposite outcome allocation
C-OPSD	First s incorrect trajectories	s = min(k, n $-$ k)	Proposed filtered objective
Random selection	Outcome-blind subset of all trajectories	s	Count-matched selection control
All-incorrect	All incorrect trajectories	n $-$ k	Higher-retention comparison

Table 5: Supervision-allocation strategies.

Setting	Value
Optimizer-step rollout count	32 distinct problems; one response per problem
Student rollout mode	Non-thinking; vLLM ( Kwon et al., 2023 ) colocated generation
Maximum rollout length	1,024 tokens
Rollout sampling	Temperature 1.1; top-p 0.95; top-k 20
Learning rate	$5\times 10^{-6}$ ; constant schedule
Token-divergence clip parameter	0.05 for 1.7B/4B; 0.06 for 8B

Table 6: Training and evaluation settings.

Result block	Per-device batch	Gradient accumulation	Ranks	Total rollouts/step
All experiments	4	8	1	32

Table 7: Batch configurations across experiments.

Model	Method	AIME24	AIME25	HMMT25	Mean	SD	Retention
1.7B	Base	49.10	33.89	23.26	35.42	—	—
	SFT	50.42	38.59	24.44	37.82	0.19	100%
	GRPO	49.26	36.16	23.66	36.36	0.42	100%
	OPSD	56.19	41.85	28.74	42.26	0.29	100%
	C-OPSD	56.18	41.50	28.51	42.06	0.34	26%
4B	Base	74.38	65.35	43.75	61.16	—	—

Table 8: Best checkpoint per benchmark, Avg@48. Scores are percentages, averaged over training runs: five for the OPSD arms, three for SFT and GRPO, one for the untrained base. SD is the standard deviation of the per-run aggregate. A rule separates the baselines from the two OPSD arms within each model size. Retention is the fraction of generated responses distilled. Executed-row measurements are presented separately in Section 7.1.

Arm	Retention	Incorrect among retained	Mean p 0	Mean accuracy (SD)
Correct-only	33%	0%	0.83	40.35 (0.20)
Random selection	26%	66%	0.42	41.58 (0.31)
C-OPSD	26%	100%	0.09	42.06 (0.34)
All-incorrect	68%	100%	0.09	42.17 (0.38)

Table 9: Outcome allocation: best checkpoint per benchmark, Avg@48, mean over five training runs. Offline p 0 estimates the retained problems’ pass rates from a separate eight-rollout probe. Full OPSD, at 100% retention, scores 42.26 under the same protocol.

Strategy	Expected retained trajectories/step	Shared-direction signal	Noise	Estimated SNR	Mean accuracy
Correct-only	10.56	3.86	4.85	0.80	40.35
Random selection	8.32	4.51	3.99	1.13	41.58
C-OPSD	8.32	5.27	3.82	1.38	42.06

Table 10: Estimated SNR and measured ablation results for three count-matched supervision-allocation strategies.

Distillation operation	Full OPSD rows	C-OPSD mean rows	Executed-row reduction
Student forward	4	1.29	$\approx$ 68%
Teacher forward	4	1.29	$\approx$ 68%
Student backward	4	1.29	$\approx$ 68%

Table 11: Executed distillation rows at micro-batch size four.

Model scale	Generation	Student f+b	Teacher f	Full total	Filtered total	Modeled saving
1.7B	101.7	305.2	157.3	564.1	250.9	55.5%
4B	262.3	786.8	405.5	1454.6	646.8	55.5%
8B	501.5	1504.3	775.3	2781.1	1236.6	55.5%

Table 12: Modeled FLOPs per optimizer step (TFLOPs), using the shared n = 4 configuration and lengths in Equation (9).

	100-step training (min)			Per step (s)
Model	OPSD	C-OPSD	Saving	OPSD	C-OPSD
1.7B	14.2	8.4	41%	7.86	4.79
4B	19.8	11.8	40%	12.36	6.95
8B	22.6	13.7	40%	12.98	8.05

Table 13: Measured wall-clock time on a single GB300 GPU.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Dataset	Method	Step 25	Step 50	Step 75	Step 100
1.7B	AIME24	OPSD	53.33	54.76	56.00	55.10
1.7B	AIME24	C-OPSD	52.64	54.03	56.04	55.14
1.7B	AIME25	OPSD	39.60	41.32	41.01	39.75
1.7B	AIME25	C-OPSD	39.03	40.40	41.38	40.38
1.7B	HMMT25	OPSD	24.57	27.08	28.74	27.89
1.7B	HMMT25	C-OPSD	24.43	25.42	27.49	28.46

Appendix

Table A1: Per-dataset accuracy (Avg@48) at each evaluation checkpoint, averaged over the five training runs per arm used in Table 8 . Bold indicates the higher score between OPSD and C-OPSD within each model, dataset, and checkpoint.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 148 papers

DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation
Aug 26, 2026Yutong Chen, Guangfu Guo, Zhichao Xu +1Adaptive Knowledge Distillation On-Policy Self-Distillation
Department of Computer Science, Clemson University, 105 Sikes Hall, Clemson, SC 29634, USA · Department of Computer Science, University of Utah
Correcting or Continuing? What On-Policy Distillation Learns from Student Errors
Oct 13, 2026Shuo Yang, Zheyu Zhang, Mahito Sugiyama +1Language Model Distillation On-Policy Distillation
Technical University of Munich · National Institute of Informatics · Munich Center for Machine Learning
Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
Aug 31, 2026Yi Ding, Ruqi ZhangRL for Language Models Language Model Distillation
Department of Computer Science, Purdue University, USA

Aug 26, 2026J/K move · Enter open · S save

DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation

Yutong Chen, Guangfu Guo, Zhichao Xu, Kunpeng Liu

Department of Computer Science, Clemson University, 105 Sikes Hall, Clemson, SC 29634, USA · Department of Computer Science, University of Utah

Adaptive Knowledge Distillation On-Policy Self-Distillation Language Model Distillation Knowledge Distillation

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) combines student generated trajectories with dense supervision from a privileged teacher of the same model. Its pointwise clipping protects the student from locally incompatible teacher directions, but the fixed teacher receives no feedback about the supervision the student rejects, which limits effective learning from the teacher. We introduce DualOPSD, which alternates updates in a coupled teacher-student system. The student retains OPSD's clipped forward KL objective. The privileged teacher adapts to the updated student after each student step. A residual target based on the frozen initial model preserves the effect of privileged context during teacher adaptation. Across Qwen3-8B/4B/1.7B and three math benchmarks, DualOPSD improves avg@12 accuracy over OPSD by 9.83 percentage points. Further diagnostic analyses show lower bidirectional KL, less persistent mismatch, and retained sensitivity to privileged reference. Together, these results demonstrate that DualOPSD can be a more effective form of privileged self-distillation. Code is available at: https://github.com/on1262/DualOPSD.

Abstract

On-Policy Self-Distillation

Explore similar work

DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation

Correcting or Continuing? What On-Policy Distillation Learns from Student Errors

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement