Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning

Md. Ismail Hossain; Humaira Kousar; Isidora Chara Tourni

cs.LGSep 29, 2026

Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning

Authors: Md. Ismail Hossain, Humaira Kousar, Isidora Chara Tourni

Organizations: North South University · KAIST · Andria Labs

Abstract

On-policy self-distillation (OPSD) trains a student to match a privileged teacher distribution along its own sampled trajectory. Standard OPSD applies this supervision to unverified student rollouts while conditioning the teacher on privileged context, typically a reference solution. We separate these roles in a factorial analysis and find that scaffold correctness has a stronger effect on downstream accuracy than context correctness. Unverified scaffolds create an imitation gap because the teacher can use information unavailable to the student. This gap shrinks with model scale, yet OPSD continues to supervise mostly unverified trajectories. In contrast, verified scaffolds remain effective even when the teacher is conditioned on the student's own unsuccessful rollout. Based on this finding, we introduce OASIS, which retains the OPSD objective but supervises mostly verified by label on-policy trajectories and replaces written solutions with unverified model-generated attempts as the teacher context. OASIS therefore requires only final-answer labels. Across Qwen3-1.7B, 4B, and 8B on AIME 2024, AIME 2025, and HMMT 2025, OASIS improves over the base model by 3.2--3.8 points on average, while OPSD's gain falls from 3.05 points at 1.7B to 0.14 at 8B. At 8B, OASIS improves over OPSD by 3.05 points, showing that verified on-policy scaffolds preserve the effectiveness of self-distillation as models scale.

Figures & tables

Figure 1: The teacher’s advantage is concentrated on failed trajectories and decreases with model scale. (a) Fraction of continuations reaching the correct answer from prefixes of student rollouts, with and without the gold reference

r

. (b) Scaffolds supervised at 1.7B: only 28% of OPSD’s reach a verified answer, and most of the rest are truncated. (c) Average gain over the base model.

Figure 2: Anatomy of the teacher signal. (a) Sequential attribution of the teacher–student KL gap. (b) Total-variation distance from the gold-context teacher for alternative contexts, with 95% bootstrap intervals. (c) Teacher statistics on verified and non-verified prefixes.

		Qwen3-1.7B		Qwen3-4B
Prefix	Context	Recovers	$\boldsymbol{\Delta}$	Recovers	$\boldsymbol{\Delta}$
Failed	Gold reference	0.77	+0.40	0.67	+0.17
	Answer only	0.61	+0.23	0.59	+0.09
	Gold, answer removed	0.59	+0.22	0.57	+0.08
	Gold, answer removed ∗	0.51	+0.13	0.50	+0.01
	Unrelated reference	0.32	$-0.06$	0.37	$-0.13$

Table 1: The teacher’s advantage is concentrated on failed trajectories. Fraction of continuations that reach the correct answer after resuming from a prefix of a student rollout, averaged over cut fractions

f\in\{0.1,\ldots,0.9\}

. The student receives no context.

\Delta

is the difference from the student-alone condition of the same run. ∗ Restricted to problems where the answer does not appear elsewhere in the redacted reference (failed: 69 at 1.7B, 72 at 4B; verified: 37 at 1.7B, 52 at 4B).

Figure 3: OPSD versus OASIS. (1) OPSD distills along a single unverified rollout

y

, so most updates land where the teacher’s edge from the hidden reference

r

is large. (2) OASIS samples

K

rollouts, verifies them, and takes the shortest verified one as scaffold

y^{*}

and another same-problem rollout as context

r

, typically unverified as context

r

, where the teacher’s edge is small. (3) The student is trained against the frozen teacher with the unchanged OPSD loss.

Model	Method	AIME24	AIME25	HMMT25	Average
Qwen3-1.7B	Base	51.50	36.70	23.10	37.10
	SFT	48.40	36.70	22.70	35.93
	GRPO	51.10	38.30	23.70	37.70
	OPSD	53.88	40.80	25.76	40.15
	AVSD	51.85	38.61	26.39	38.95
	OASIS (ours)	54.17	41.11	26.94	40.74

Table 2: Competition mathematics benchmark performance (Avg@12, %). Best result per block in bold , second best underlined .

\Delta_{\text{Ours}-\text{OPSD}}

highlights the growing margin of OASIS over OPSD as model scale increases.

Benchmark	Metric	OPSD	OASIS
AIME 2024	Avg@12	$74.4\pm 0.71$	$\mathbf{76.6\pm 0.23}$
	Pass@12	$\mathbf{86.6\pm 0.00}$	$\mathbf{86.6\pm 0.00}$
	Maj@12	$80.00\pm 0.00$	$\mathbf{81.1\pm 1.57}$
AIME 2025	Avg@12	67.9 $\pm 1.33$	$\mathbf{69.1\pm 0.23}$
	Pass@12	83.3 $\pm 0.00$	$\mathbf{86.6\pm 0.00}$
	Maj@12	73.3 $\pm 3.33$	$\mathbf{80.0\pm 0.00}$

Table 3: Qwen3-4B reasoning benchmark results. Results are reported as mean

\pm

standard deviation over 3 evaluation seeds with 12 samples per problem.

Model	Method	Seen	Trained	Rollouts	Written solution?	AIME24	AIME25	HMMT25	Avg.
		Training budget				Accuracy
Qwen3-4B	OPSD †	12.8K	12.8K	12.8K	Yes	76.40	68.30	46.10	63.60
	AVSD †	12.8K	12.8K	12.8K	Yes	76.20	68.30	44.20	62.90
	OASIS	6.4K	$\approx$ 3.2K	51.2K	No	76.66	69.17	45.27	63.70
Qwen3-8B	OPSD †	12.8K	12.8K	12.8K	Yes	77.80	70.80	45.80	64.80
	AVSD †	12.8K	12.8K	12.8K	Yes	75.40	69.60	47.10	64.03

Table 4: Written-supervision and rollout trade-offs. This table reports annotation, rollout, and supervised-example trade-offs. Problems seen and problems that received a gradient over training, rollouts generated, and whether written solutions are required. Scores are Avg@12. Best per column within each model in bold .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Effect of the per-entry clip. (a) Fraction of the positive KL mass removed by the clip. Values above one indicate that the clipped terms exceed the total positive KL mass, resulting in a negative value of the clipped objective. (b) Fraction of tokens with at least one clipped vocabulary entry. The dotted line marks

\kappa=0.05

, used by OPSD and all experiments in this work. Proposition 2 shows that clipped entries lose the teacher’s upward pull and receive a downward logit gradient.

	Qwen3-1.7B		Qwen3-4B
	OPSD	OASIS	OPSD	OASIS
Problems receiving gradient	100%	51.7%	100%	50.8%
Selected scaffolds reaching the answer	28.3%	100%	30.6%	100%
Candidate rollouts hitting the length cap	62.0%	58.0%	64.4%	63.1%
Mean supervised length (tokens)	859	603	883	646
Generated tokens	2.76M	43.5M	2.80M	44.8M

Appendix

Table 5: Training composition and dynamics. Statistics over 100 optimizer steps with 64 problems per step and

K=8

rollouts per problem for OASIS. “Problems receiving gradient” is the fraction of candidate problems with at least one verified rollout. “Selected scaffolds reaching the answer” is computed only among the scaffolds selected for supervision, while “Candidate rollouts hitting the length cap” is computed over all generated candidate rollouts. OASIS supervises fewer tokens because problems without a verified rollout are excluded and selected scaffolds are shorter on average, while its additional cost comes from rollout generation. Entropy, teacher–student top-1 agreement, and clipped-token fraction are measured on training batches, with early and late values computed over steps 2–34 and 68–100, respectively.

Quantity at the supervised state	Verified rollouts	Failed rollouts
Share of OPSD’s training scaffolds	28.3%	71.7%
KL $(q_{t}\,\\|\,p_{t})$ per token	0.274	0.181
Teacher entropy	0.297	0.461
Teacher probability on the sampled token	0.834	0.775
Teacher top-1 $=$ sampled token	0.862	0.814
Tokens with a clipped entry ( $\kappa=0.05$ )	0.288	0.323

Appendix

Table 6: Teacher signal at verified and failed states. Measurements use the gold reference on 200 rollouts per condition for Qwen3-1.7B. “Sampled” denotes the token actually produced by the student at the supervised state.

Figure 5: Training dynamics at two scales (rows) for four metrics (columns). Lines represent trailing means with

\pm 1

standard deviation bands. Brackets report the OASIS minus OPSD gap averaged over the shaded final third of training. At 1.7B, OASIS maintains a higher correct rate, less truncation, lower student entropy, and stable teacher agreement, whereas OPSD drifts. At 4B, both methods track each other for most of training and drift near the end, which aligns with the clipping effect described in Proposition 2 . Metrics are computed on each method’s own training batches. Differences therefore reflect both changes in model behavior and changes in the selected trajectory distribution.

Figure 6: Token budget over 100 optimizer steps. (a) Total tokens generated by OPSD and OASIS, shown on a logarithmic scale. OASIS generates roughly sixteen times more tokens because it samples

K=8

rollouts per problem. (b) Tokens receiving distillation supervision. OASIS supervises about 30% fewer tokens because problems without a verified rollout are excluded and verified scaffolds are shorter on average. (c) Teacher contexts used by OASIS: a failed rollout with a wrong answer, a failed rollout truncated at the token limit, or another verified rollout when every sampled rollout is correct.

Scaffold	Supervised length	Avg@12	Pass@12	Maj@12
Shortest verified †	603	$54.17\pm 0.58$	$80.00\pm 0.00$	$68.88\pm 1.57$
Longest verified	909	$52.33\pm 0.51$	$73.33\pm 1.92$	$60.00\pm 1.92$

Appendix

Table 7: Effect of scaffold selection on Qwen3-1.7B using AIME 2024. Both variants use the same verified rollouts and supervise the same set of problems. Results are reported as mean

\pm

standard deviation over three evaluation seeds.

$\boldsymbol{K}$	Generated tokens	Avg@12	Pass@12	Maj@12
4	22.8M	$52.77\pm 0.29$	$76.67\pm 0.33$	$66.67\pm 0.96$
8 †	43.5M	$54.17\pm 0.58$	$80.00\pm 0.00$	$68.88\pm 1.57$

Appendix

Table 8: Effect of the number of rollouts on Qwen3-1.7B using AIME 2024. Generated tokens are measured over 100 optimizer steps. Results are reported as mean

\pm

standard deviation over three evaluation seeds.

Optimization & Batching
Hyperparameter	Qwen3-1.7B	Qwen3-4B	Qwen3-8B
Optimizer steps	100	100	100
Problems per step	64	64	64
Problems trained per step	$\leq 32$	$\leq 32$	$\leq 32$
Rollouts per problem ( $K$ , OASIS)	8	8	8
Learning rate	$5\times 10^{-6}$ , linear decay, gradient clip $\\|g\\|\leq 0.1$

Appendix

Table 9: Training configuration. Hyperparameters are identical across model sizes

Explore similar work

On-Policy Self-Distillation without Any Supervision
Aug 6, 2026Yijiang Li, Bingyang Wang, Yijun Liang +3Unsupervised On-Policy Self-Distillation Self-Distillation Framework
UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1
E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation
Oct 4, 2026Yifei Liu, Minghao Fang, Xinyu Gu +6Unsupervised On-Policy Self-Distillation
The Chinese University of Hong Kong · Zhejiang University · Shanghai Artificial Intelligence Laboratory +5
Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning
Sep 27, 2026Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil +3Unsupervised On-Policy Self-Distillation Efficient On-Policy Distillation
ACI PLC, Bangladesh · University of Dhaka, Bangladesh · BRAC University, Bangladesh +2

Aug 6, 2026, cs.LGJ/K move · Enter open · S save

On-Policy Self-Distillation without Any Supervision

Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian+2

UC San Diego · Georgia Institute of Technology · University of Maryland, College Park+1

Unsupervised On-Policy Self-Distillation Self-Distillation Framework On-Policy Self-Consistency

Open paper AskPDF arXiv

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at https://github.com/williamium3000/u-opsd.

Abstract

Explore similar work

On-Policy Self-Distillation without Any Supervision

E2^22-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning

E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation