Continually Post-training Small Models via Synthetic Self-distillation

Woosung Koh; Hyunsoo Lee; Kyungjae Lee; Haeju Park; Dahyun Lee; Dasol Hwang; Moontae Lee

cs.LGOct 9, 2026

Continually Post-training Small Models via Synthetic Self-distillation

Authors: Woosung Koh, Hyunsoo Lee, Kyungjae Lee, Haeju Park, Dahyun Lee, Dasol Hwang, Moontae Lee

Organizations: KAIST AI · Work done during an internship at LG AI Research · LG AI Research

Abstract

Continual post-training faces a dual challenge of acquisition and retention: models must learn new capabilities while preserving prior knowledge. While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with in-context demonstrations, it relies on strong in-context learning and instruction-following abilities. Consequently, SDFT fails on smaller models critical for low-latency, on-device use. To enable continual learning at smaller scales, we first identify the root cause of failure. We show that poor acquisition and forgetting correlate linearly with the post-trained model's distribution shift, revealing the necessity of in-distribution learning. We thus introduce Synthetic Self-Distillation (SynthSelf), imitating SDFT's in-distribution rewrite by outsourcing the task to a minimally larger, same-family aligner. Coupled with off-policy supervision and rejection sampling, this provides the high-quality supervision small models require without breaking their original distribution. Evaluations spanning 4 acquisition datasets (including a mixture) and 14 retention benchmarks show SynthSelf consistently outperforms 6 methods (including off- and on-policy distillation) across 3 small model scales. Extensive further analyses confirm SynthSelf paves the way toward effective small-model continual learning.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of SynthSelf performance across three model scales ( 0.5 , 1.5 , and 3B ).

Figure 2: Distribution shift negatively correlates with ideal continual learning. (a) Experiment 1: Post-training distribution shift strongly correlates with forgetting existing capabilities across GPQA-Diamond, ARC-C, MMLU-Pro, HumanEval, and MBPP, regardless of data source or type. (b) - (e) Experiment 2: Transforming the same dataset from its original OOD form (HI) to more in-distribution forms (MID, LO) monotonically improves (c) target domain (math) retention, (d) target domain performance, and (e) cross-domain (coding) retention.

Panel of Figure 3 — Figure 3: SDFT fails at smaller model sizes as its ability to self-teach deteriorates.

		Baseline					SynthSelf (ours)
Model		Base	SFT	SDFT	DFT	ASFT	SS -1 $\Delta$	$\Delta$ Off-PD	$\Delta$ On-PD	SS -2 $\Delta$	$\Delta$ Off-PD	$\Delta$ On-PD
Science: Chemistry
0.5B	Acq.	25.4	56.8	38.5	35.5	45.6	63.7	+21.1	+34.5	59.4	+10.1	+21.5
	Ret.	31.8	28.2	30.3	28.7	30.3	27.8	-2.4	+1.2	29.5	-0.8	+5.1
1.5B	Acq.	31.0	64.9	53.6	56.8	48.5	67.1	+16.4	+33.3	65.3	+12.0	+22.9
	Ret.	43.9	44.1	41.8	43.8	45.2	45.0	+0.4	+1.9	44.6	+0.0	+2.5

Table 1: SynthSelf leads acquisition (Acq.) in all 4 benchmarks and ties the best baseline on 14 retention tasks (Ret.), over 4 baselines, off-/on-policy distillation, and three base models. Green and red background shading on Avg. rows is min-max normalized across each row. The 14 retention tasks span instruction following, commonsense, diverse knowledge, code, and math. The mixture contains Chemistry, Biology, Materials, and Physics in both the train and test splits.

\Delta

Off-PD and

\Delta

On-PD are SynthSelf minus off- or on-policy distillation (%p).

	Acquisition $\uparrow$		KL (NLL) $\downarrow$		Data-gen Accuracy (%) $\uparrow$		Data-gen PFLOPs $\downarrow$
Aligner	Sci.	Tool	Sci.	Tool	Sci.	Tool	Sci.	Tool
More In-distribution Aligners
Qwen2.5-7B	66.8	55.7	0.995	0.650	99.0	84.1	13.9	39.0
Qwen2.5-14B	64.1	60.1	1.033	0.666	99.1	81.8	36.6	76.0
Stronger but More Out-of-distribution Aligners
Qwen3.5-27B	65.1	50.5	1.440	0.873	90.3	95.8	70.0	142.1

Table 2: SynthSelf performs better with in-distribution aligners. Averaged over the Qwen2.5-0.5B , 1.5B , and 3B base models, using aligners that are more in-distribution (measured by NLL) is a stronger predictor of performance than the strength of the aligner (measured by data-generation accuracy). Data-generation accuracy represents the reward

R(\pi_{A})

in Eq. 3 . It is additionally advantageous that smaller, same-family aligners are the most compute-efficient when generating off-policy data. Bold indicates best value across each column.

Figure 6: SynthSelf outperforms SDFT across diverse prompt variations, including a version prompt-matched to SynthSelf . On average, SynthSelf outperforms prompt-matched SDFT by +18.96%p and +1.68%p on acquisition and retention, respectively.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Training Dataset (HuggingFace Path)	Uploaded
Pre-cutoff : math released before the cutoff
microsoft/orca-math-word-problems-200k	2024-03
openai/gsm8k ( main )	2022-04
AI-MO/NuminaMath-1.5	2025-02
EleutherAI/hendrycks_math ( algebra )	2023-09
TIGER-Lab/MathInstruct	2023-09

Appendix

Table 3: The training data. The

23

corpora Experiment 1 trains on, each taken as published and named by the HuggingFace path it is loaded from. Groupings are domain and upload date against the base model’s 2024-09 cutoff, and the dates are each repository’s createdAt read from the HuggingFace API.

		Per-subject domain			Science-mix
Subject	L-3	Gold	Train	Test	Train	Test
Chemistry	3,542	2,674	2,139	535	668	133
Biology	679	356	285	71	285	71
Material	1,283	969	775	194	668	133
Physics	1,498	1,114	891	223	668	133
Total	7,002	5,113	4,090	1,023	2,289	470

Appendix

Table 4: Train/test split of the Science-mix corpus. SciKnowEval L-3 gold, split

80

20

at seed

0

Setting	Value
Adaptation	full fine-tuning of all parameters
Learning rate	$5\times 10^{-5}$
Epochs	$2$
Effective batch	$16$ sequences
Optimizer	paged AdamW, 8-bit
Schedule	cosine; warmup ratio $0.1$

Appendix

Table 5: Configuration for every cell in Tabs. 1 and 2 . Unlisted values are library defaults.

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
Jul 2, 2026Meng Wang, Haohan Zhao, Wenzhuo Liu +7Continual Learning for LLMs On-Policy Self-Distillation
Centre for Artificial Intelligence and Robotics, HKISI, CAS · Institute of Automation, CAS · University of Chinese Academy of Sciences +1
iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
Sep 21, 2026Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber +4Continual Learning for LLMs Language Model Distillation
Imperial College London · Politecnico di Torino · UCL Centre for AI
On-Policy Self-Distillation without Any Supervision
Aug 6, 2026Yijiang Li, Bingyang Wang, Yijun Liang +3On-Policy Self-Distillation Language Model Self-Improvement
UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1

Jul 2, 2026J/K move · Enter open · S save

Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training

Meng Wang, Haohan Zhao, Wenzhuo Liu, Lu Yang+6

Centre for Artificial Intelligence and Robotics, HKISI, CAS · Institute of Automation, CAS · University of Chinese Academy of Sciences+1

Continual Learning for LLMs On-Policy Self-Distillation RL Post-Training LLM Post-Training Catastrophic Forgetting

Open paper AskPDF arXiv

Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods such as GRPO adapt more conservatively and better preserve prior capabilities. Further analyses reveal that denser self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training. Our code is available at https://github.com/Moenupa/SDPO-CL.

Abstract

LLM Post-Training

Explore similar work

Denser ≠\neq= Better: Limits of On-Policy Self-Distillation for Continual Post-Training

iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs

On-Policy Self-Distillation without Any Supervision

Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training