On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

Leon Sick; Lukas Hoyer; Yannick Strümpler; Christian Reisswig; Timo Ropinski; Federico Tombari

cs.CLOct 23, 2026

On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

Authors: Leon Sick, Lukas Hoyer, Yannick Strümpler, Christian Reisswig, Timo Ropinski, Federico Tombari

Organizations: Google · Ulm University

Abstract

On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. We push supervision to the limit: guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and privileged reasoning traces without access to any human-curated prompts or answers. We find that self-distillation on fully synthetic privileged information enables a model to improve across multiple domains while retaining its capabilities on domains it is not self-trained on. We demonstrate the generality of this framework across text and multimodal domains, achieving consistent self-improvement on standard language benchmarks and Visual Question Answering. Our findings demonstrate that foundation models possess rich latent capabilities that can be amplified without external task supervision or curated problem distributions.

Figures & tables

Figure 1: On-Policy Self-Training loop. Seeded only by topic descriptors, the model generates and filters its own question-answer pairs, then trains on them via on-policy self-distillation from a privileged EMA teacher, iterating for data-free self-improvement.

Figure 2: Model training. As in OPSD ( Zhao et al., 2026a ) , the student sees only the question and matches the next-token distribution of a teacher that also sees privileged information. In OPST, this information is self-generated, and the teacher is an EMA of the student.

Gemma 4 E2B
	Method	Signal	Gen. Answer	MMLU	GPQA	GSM8K ∗	HellaSwag	Average
	Base			37.5	28.8	–	48.7	38.3
	+ SFT	Dense	Target	38.3 +0.8	30.8 +2.0	–	50.8 +2.1	40.0 +1.7
	+ GRPO	Sparse	Reward	37.3 -0.2	30.3 +1.5	–	50.3 +1.6	39.3 +1.0
w/o Think.	+ OPST (Ours)	Dense	Context	47.7 +10.2	32.8 +4.0	–	56.2 +7.5	45.6 +7.3
	Base			69.9	27.8	72.7	63.1	58.4

Table 1: Synthetic-data results. Performance comparison on language reasoning benchmarks for Gemma 4 models trained on self-generated data only.


	Human-Curated
Method	Question	Answer	MMLU	GPQA	GSM8K ∗	HellaSwag	Average
Base			37.5	28.8	–	48.7	38.3
+ Supervised SFT	$\checkmark$	$\checkmark$	47.3 +9.8	31.3 +2.5	–	52.8 +4.1	43.8 +5.5
+ Supervised OPSD	$\checkmark$	$\checkmark$	49.6 +12.1	33.3 +4.5	–	56.8 +8.1	46.6 +8.3
+ u-OPSD ‡	$\checkmark$	$\times$	45.9 +8.4	33.3 +4.5	–	54.7 +6.0	44.6 +6.3

Table 2: Results across various levels of supervision. Performance comparison on language reasoning benchmarks for Gemma 4 models across supervision levels. Methods above the light rule train on human-curated reference solutions. Bold marks the best result among the remaining methods.

Gemma 4 E2B
		In-domain				Out-of-domain
	Training Topics	MMLU	HellaSwag	GSM8K ∗	Avg.	GPQA	ARC-C	MATH-500 ∗	Winogrande	Avg.
	Base	37.5	48.7	72.7	53.0	28.8	39.1	56.5	50.0	43.6
Reference	+ Single domain	47.7 +10.2	56.2 +7.5	80.9 +8.2	61.6 +8.6
	+ MMLU + HellaSwag	46.9 +9.4	55.6 +6.9	74.3 +1.6	58.9 +5.9	30.3 +1.5	44.9 +5.8	48.0 -8.5	51.8 +1.8	43.8 +0.2
	+ HellaSwag + GSM8K	42.6 +5.1	54.6 +5.9	77.0 +4.3	58.1 +5.1	31.3 +2.5	43.0 +3.9	55.8 -0.7	51.8 +1.8	45.5 +1.9

Table 3: Multi-domain results. Performance comparison on in-domain and out-of-domain benchmarks for Gemma 4 E2B self-trained on combinations of topic descriptors. Single domain reports separate models, each self-trained only on the topics of the respective benchmark.

Gemma 4 E2B
		Privileged Info		VQAv2	POPE
	Method	Caption	Spatial	Acc. ( $\uparrow$ )	F1 ( $\uparrow$ )	Acc. ( $\uparrow$ )	Rec. ( $\uparrow$ )
	Base			49.1	82.2	83.2	72.9
	+ Visual OPST	$\times$	$\checkmark$	55.7 +6.6	84.7 +2.5	85.1 +1.9	77.4 +4.5
	+ Visual OPST	$\checkmark$	$\times$	54.0 +4.9	83.0 +0.8	83.9 +0.7	74.0 +1.1
Ours	+ Visual OPST	$\checkmark$	$\checkmark$	56.0 +6.9	84.6 +2.4	85.0 +1.8	77.4 +4.5

Table 4: Multimodal results. Performance comparison on vision-language benchmarks for Gemma 4 models self-trained on unlabeled images only.

Sample selection	MMLU
Base	37.5
+ OPST w/o verification	42.7
+ Verification	45.4
+ Consensus gate	47.7

Table 5: Ablations on Gemma 4 E2B on MMLU. Default settings are highlighted in gray.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

rate $\beta$	MMLU
1.00 (no EMA)	38.1
0.20	44.2
0.05	47.7
0.01	45.2

Appendix

Table 6: OPST component ablations on Gemma 4 E2B evaluated on MMLU. Default settings are highlighted in gray.

Setting	MMLU	GPQA	GSM8K	HellaSwag	Average
w/o Think.	47.7 $\,\pm\,$ 0.8	32.8 $\,\pm\,$ 1.0	–	56.2 $\,\pm\,$ 0.8	45.6 $\,\pm\,$ 0.3
w/ Think.	71.1 $\,\pm\,$ 0.1	34.3 $\,\pm\,$ 1.0	80.9 $\,\pm\,$ 1.2	65.6 $\,\pm\,$ 0.5	63.0 $\,\pm\,$ 0.4

Appendix

Table 7: Variance across seeds. Mean and standard deviation of OPST on Gemma 4 E2B over three seeds. Means correspond to Table 1 .

Method	MMLU	GPQA	GSM8K	HellaSwag	Average
Qwen3-4B-Base
Base	72.0	32.3	79.7	82.8	66.7
+ OPST (Ours)	75.1 +3.1	37.4 +5.1	93.0 +13.3	85.4 +2.5	72.7 +6.0

Appendix

Table 8: Results on Qwen3-4B-Base. Performance of OPST applied to a pretrained Qwen3 base model, evaluated with the protocol described in Appendix A.6 .

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 172 papers

On-Policy Self-Distillation without Any Supervision
Aug 6, 2026Yijiang Li, Bingyang Wang, Yijun Liang +3On-Policy Self-Distillation Language Model Self-Improvement
UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1
Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline
May 20, 2026Tony Lee, Percy LiangLanguage Model Self-Verification Self-Training for Language Models
Stanford University
Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
Sep 29, 2026Zheng Zhang, Xinyue Tan, Lufei Li +3On-Policy Self-Distillation Language Model Distillation
School of Information Science and Technology, ShanghaiTech University · State Key Laboratory of General Artificial Intelligence, BIGAI · Hefei University of Technology

Aug 6, 2026J/K move · Enter open · S save

On-Policy Self-Distillation without Any Supervision

Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian+2

UC San Diego · Georgia Institute of Technology · University of Maryland, College Park+1

On-Policy Self-Distillation Language Model Self-Improvement Self-Consistency in Language Models

Open paper AskPDF arXiv

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at https://github.com/williamium3000/u-opsd.

Abstract

Language Model Self-Improvement

Explore similar work

On-Policy Self-Distillation without Any Supervision

Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline

Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models