Self-Distillation for Supervised Fine-Tuning

Wenxuan Zhu; Bang An; Shuming Liu; Wenqing Cui; Tianqi Xu; Bing Li; Yibo Yang; Peter Wonka

cs.CLOct 18, 2026

Self-Distillation for Supervised Fine-Tuning

Authors: Wenxuan Zhu, Bang An, Shuming Liu, Wenqing Cui, Tianqi Xu, Bing Li, Yibo Yang, Peter Wonka,

Organizations: King Abdullah University of Science and Technology · Shanghai Jiao Tong University

Abstract

Supervised fine-tuning treats the expert demonstration or gold response as a one-hot target, suppressing valid alternative continuations that the model would otherwise assign probability to. We propose Gold-Conditioned Distillation (GCD), a method that turns expert demonstrations into model-native supervision. GCD uses one model as both teacher and student: the teacher conditions on a prompt containing the gold response in context, the student conditions on the original prompt as at inference, and the student is trained to minimize the forward KL divergence to the teacher's next-token distribution at each gold response token. This lets the model use its prior knowledge to teach itself the new task: correctness is anchored by the gold response, while the soft target preserves plausible alternatives. Across various training tasks on Qwen and Llama models, GCD improves new-task accuracy over SFT and generally reduces degradation on general benchmarks.

Figures & tables

Figure 1: Overview of Gold-Conditioned Distillation (GCD). The student predicts the next token from the query $x$ and the gold-response prefix $y_{<t}$ ; the teacher predicts it from a privileged context that additionally contains the full gold response $y$ and a repeat prompt. The student is trained with a forward KL loss to match the teacher’s gold-conditioned next-token distribution, and the teacher is an exponential moving average of the student.

Qwen2.5-3B-Instruct
Method	Tooluse	Chemistry	Physics & Material	Biology	Avg.
Base	34.02	30.77	44.83	76.67	46.57
SFT	60.83	65.09	63.79	85.94	68.91
SDFT	55.67	63.31	52.30	83.53	66.38
GCD	66.30	66.67	63.22	86.45	70.66
Qwen3-4B-Base

Table 1: New-task accuracy (%) across four downstream tasks and four backbone models. Bold and underline mark the best and second-best among SFT / SDFT / GCD per (model, task) cell (Base excluded).

Model	Method	New Task	Previous Tasks					Avg.
Model	Method	New Task	MMLU	HellaSwag	Winogrande	TruthfulQA	IFEval	Avg.
New Task: Tooluse
Qwen2.5-3B-Instruct	Base	34.02	65.45	74.85	69.69	58.63	53.33	64.39
	SFT	60.83	64.39	72.41	69.14	52.17	40.28	59.68
	SDFT	55.67	65.10	74.22	69.22	58.65	41.22	61.68
	GCD	66.30	64.71	72.62	69.93	52.84	38.49	59.72

Table 2: Previous capability retention on five general benchmarks after fine-tuning on Tooluse (%). Bold and underline denote the best and second-best results among SFT / SDFT / GCD per (model, column) cell (Base excluded).

Figure 2: Learning New Knowledge from Scratch. We report evaluation accuracy as a function of training epochs for Qwen2.5-3B-Instruct, Qwen3-4B-Base and Llama-3.1-8B-Instruct on LiveNews dataset.

Model	Method	New Task	Previous Tasks					Avg.
Model	Method	New Task	MMLU	HellaSwag	Winogrande	TruthfulQA	IFEval	Avg.
New Task: LiveNews
Qwen2.5-3B-Instruct	Base	1.50	65.45	74.85	69.69	58.63	53.33	64.39
	SFT	36.75	64.86	74.37	67.80	36.78	43.41	57.44
	SDFT	10.25	65.12	74.80	69.69	53.94	35.12	59.73
	GCD	38.00	65.19	74.24	69.06	36.54	44.48	57.90

Table 3: Previous capability retention on five general-purpose benchmarks after fine-tuning on LiveNews (%). The gray Base row gives pre-fine-tuning performance. Bold and underline denote the best and second-best among SFT / SDFT / GCD per (model, column) cell (Base excluded).

	Base	SFT	GCD	Teacher $\pi_{T}$
$\pi(\text{gold})$	0.10	0.94	0.86	0.55
Mass on teacher top-5	0.62	0.05	0.13	0.45
Mass on base top-5	0.88	0.05	0.10	0.46
Top-5 Jaccard w/ base	0.99	0.14	0.27	0.26

Table 4: Token-level distribution statistics at low-confidence gold positions on Tooluse (Qwen2.5-3B-Instruct). Compared with SFT, GCD assigns less saturated probability to the gold token while preserving substantially more mass on both the gold-conditioned teacher’s top alternatives and the base model’s natural continuations.

Panel of Figure 3 — Figure 3: Comparison of GCD with alternative regularization or soft-label baselines.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Answer format	#Train	#Eval
Tooluse	API tool calling	API name + JSON args	4,046	97
Chemistry	Mixed scientific QA (chemistry-heavy)	4-option MCQ	2,674	507
Physics & Material	Physics + materials science	4-option MCQ	1,257	174
Biology	School- and graduate-level biology	2/3/4-option MCQ	1,107	583
LiveNews	Post-training-cutoff news QA	Free-form answer	400	400

Appendix

Table 5: Dataset overview. #Train and #Eval give the number of training and held-out evaluation examples.

Figure 4: Learning-rate sensitivity of SDFT versus our GCD on LiveNews. Each panel shows accuracy across four training epochs for one model. Gray curves are SDFT runs, darker for higher learning rates; the red curve is GCD at its single reported learning rate. SDFT’s best learning rate is model-specific and improves the method only marginally, while GCD dominates the entire SDFT envelope on every panel.

EMA rate $\alpha$	LiveNews	Tooluse
$0$ (frozen teacher)	34.25	64.95
$0.01$ (default)	38.00	66.30
$0.05$	35.75	62.89
$0.2$	36.25	16.49
$1.0$ (teacher $=$ student)	20.50	0 0.00

Appendix

Table 6: Effect of the EMA update rate

\alpha

on GCD (Qwen2.5-3B-Instruct, new-task accuracy in %, final checkpoint).

KL direction	LiveNews	Tooluse
Forward (default)	38.00	66.30
Reverse	11.00	55.67

Appendix

Table 7: Effect of the KL direction on GCD (Qwen2.5-3B-Instruct, new-task accuracy in %, final checkpoint).

Method	Qwen2.5-3B-Instruct	Qwen3-4B-Base	Llama-3.1-8B-Instruct
SFT	61.1 $\pm$ 0.7	63.2 $\pm$ 0.6	66.0 $\pm$ 0.4
SDFT	56.4 $\pm$ 0.6	61.2 $\pm$ 0.8	35.6 $\pm$ 0.7
GCD	66.7 $\pm$ 0.6	69.6 $\pm$ 0.5	66.9 $\pm$ 0.4

Appendix

Table 8: Tooluse accuracy (%) over five random seeds, reported as mean

\pm

standard deviation.

Supervised Fine-Tuning

Supervised fine-tuning adapts pretrained models by learning from labeled examples, instructions, demonstrations, and task-specific trajectories.

See all 222 papers

Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
Apr 13, 2026Yinghui He, Simran Kaur, Adithya Bhaskar +7Language Model Self-Improvement Self-Supervised Knowledge Distillation
Princeton University · University of Toronto · Carnegie Mellon University
SoFT: Soft Targets for Generalizable LLM Fine-Tuning
Sep 26, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +7LLM Fine-Tuning Fine-Tuning
HKUST, HKUST-GZ
iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
Sep 21, 2026Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber +4Continual Learning for LLMs Language Model Distillation
Imperial College London · Politecnico di Torino · UCL Centre for AI

Apr 13, 2026J/K move · Enter open · S save

Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision

Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang+6

Princeton University · University of Toronto · Carnegie Mellon University

Language Model Self-Improvement Self-Supervised Knowledge Distillation Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training. Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations. Collecting such supervision can be costly or unavailable. We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations. SD-Zero trains a single model to play two roles: a Generator, which produces an initial response, and a Reviser, which conditions on that response and its binary reward to produce an improved response. We then perform on-policy self-distillation to distill the reviser into the generator, using the reviser's token distributions conditioned on the generator's response and its reward as supervision. In effect, SD-Zero trains the model to transform binary rewards into dense token-level self-supervision. On math and code reasoning benchmarks with Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SD-Zero improves performance by at least 10% over the base models and outperforms strong baselines, including Rejection Fine-Tuning (RFT), GRPO, and Self-Distillation Fine-Tuning (SDFT), under the same question set and training sample budget. Extensive ablation studies show two novel characteristics of our proposed algorithm: (a) token-level self-localization, where the reviser can identify the key tokens that need to be revised in the generator's response based on reward, and (b) iterative self-evolution, where the improving ability to revise answers can be distilled back into generation performance with regular teacher synchronization. Code: https://github.com/princeton-pli/Self-Distillation-Zero.

Abstract

Supervised Fine-Tuning

Explore similar work

Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision

SoFT: Soft Targets for Generalizable LLM Fine-Tuning

iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs