Self-Specialized Teachers for Domain Post-Training

Yifei Li; Rongman Xu; Lingling Zhang; Muye Huang; Zihan Ma; Jiashuai Liu; Hang Yan; Heng Wang

cs.AIAug 16, 2026

Self-Specialized Teachers for Domain Post-Training

Authors: Yifei Li, Rongman Xu, Lingling Zhang, Muye Huang, Zihan Ma, Jiashuai Liu, Hang Yan, Heng Wang

Organizations: School of Computer Science and Technology, Xi’an Jiaotong University · MOE KLNN Lab, Xi’an Jiaotong University

Abstract

Target-only post-training can improve performance in a specialized domain while degrading behaviors that a general-purpose base model acquired before adaptation. We study this problem when target-domain data are available but a representative replay corpus is not. We propose self-specialized teacher distillation (SSTD), a two-stage procedure that first trains a copy of the base model into a domain teacher, then distills its token distribution to a student on prefixes sampled from the student itself. Teacher training combines standard target supervision with base-aware key-token weighting and distribution alignment to the frozen base model; on-policy distillation then places domain feedback on states the student can encounter at inference time. On financial numerical reasoning, medical question answering, and legal holding identification, SSTD retains much of the target improvement of direct fine-tuning while improving the mean score on the evaluated general suite by 4.8--5.0 points at the reported operating point. The pattern persists across Qwen3 sizes and on Gemma backbones. SSTD requires neither an external teacher nor general replay data.

Figures & tables

Figure 1: A finite replay set directly constrains only a small, partial subset of a base model’s general behavior. During target-domain post-training, general behaviors outside that coverage receive no direct retention signal and may drift, even as the model gains the desired domain expertise.

Figure 2: Student-aware teacher training with on-policy distillation. In Stage I, a copy of the base model specializes to the domain while remaining close to the frozen student distribution. In Stage II, the student samples its own trajectories and receives dense token-level supervision from the resulting teacher.

		Finance		Medical		Legal
Model	Method	Target $\uparrow$	General $\uparrow$	Target $\uparrow$	General $\uparrow$	Target $\uparrow$	General $\uparrow$
Qwen3-1.7B	Base	34.2	57.8	63.2	58.1	69.4	59.3
	Naive SFT	61.5	50.4	75.8	51.7	82.0	53.5
	OPSD	56.8	55.0	72.6	55.8	78.5	57.2
	OPD	55.5	53.0	71.8	54.3	77.6	55.6
	SSTD (ours)	59.5	55.2	74.2	56.7	80.1	58.4

Table 1: Domain adaptation and retention across domains. Target is FinQA execution accuracy, MedMCQA accuracy, or CaseHOLD accuracy, respectively. General is the mean of MMLU, HellaSwag, and ARC-C.

Teacher objective	Target	General	$\Delta$ T	$\Delta$ G
SFT	58.0	53.6	–	–
SFT + key-token	59.0	54.2	+1.0	+0.6
SFT + alignment	58.6	54.7	+0.6	+1.1
SFT + both terms	59.5	55.2	+1.5	+1.6

Table 2: Ablation of student-aware teacher construction on Finance. Deltas use the teacher-SFT row as the reference.

Source	Target	$\Delta$ T	General	$\Delta$ G
Base	34.2	–	57.8	–
Gold targets	62.3	+28.1	49.1	-8.7
Teacher rollouts	58.7	+24.5	52.1	-5.7
Offline prefixes	57.6	+23.4	53.8	-4.0
Student prefixes	59.5	+25.3	55.2	-2.6

Table 3: Effect of student-generated supervision on Finance.

Figure 3: Generalization of replay-free domain adaptation across Qwen3 model sizes on Finance. SSTD reduces the loss on the general suite at every scale while retaining a similar target-domain improvement. Values show change from the base model.

Backbone	Method	Target $\uparrow$	General $\uparrow$
Gemma-3-1B	Naive SFT	55.7	47.9
	SSTD	54.4	51.8
Gemma-3-4B	Naive SFT	65.1	55.4
	SSTD	63.8	59.6

Table 4: Backbone generalization on Finance. General Avg. uses the same evaluation suite as Table 1 .

Method	Ext. teacher	Gen. replay	GPU h	Teacher tok.
Naive SFT	No	No	1.0 $\times$	–
OPSD	No	No	1.1 $\times$	–
OPD	Yes	No	2.6 $\times$	180M
SSTD	No	No	1.7 $\times$	120M

Table 5: Training-cost accounting on the Finance setting. “External teacher” refers to a model outside the student family; “Teacher tok.” denotes generated teacher tokens.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Dataset and version	Train	Validation	Test	Input–output format	Metric
Finance	FinQA, official EMNLP 2021 release	6,251	883	1,147	Report, question $\rightarrow$ executable program	Execution accuracy
Medical	MedMCQA, official release	182,822	4,183	6,150	Question and four options $\rightarrow$ option label	Accuracy
Legal	CaseHOLD, LexGLUE split	45,000	3,900	3,900	Case excerpt and five holdings $\rightarrow$ option label	Accuracy

Appendix

Table 6: Target-domain datasets. FinQA is evaluated by executing the predicted program, while MedMCQA and CaseHOLD are evaluated by exact multiple-choice accuracy.

Backbone	Method	Target	General Avg.
Gemma-3-1B	Naive SFT	55.7	47.9
	SSTD	54.4	51.8
Gemma-3-4B	Naive SFT	65.1	55.4
	SSTD	63.8	59.6

Appendix

Table 7: Cross-family results on Finance. These runs use the same Finance protocol as the primary experiments. Target is FinQA execution accuracy and General Avg. is the mean of MMLU, HellaSwag, and ARC-C.

Item	Setting
Primary checkpoint	Qwen/Qwen3-1.7B
Student initialization	Pretrained base checkpoint $S_{0}$
Teacher initialization	Independent copy of $S_{0}$
Frozen reference	Unmodified copy of $S_{0}$
Optimizer	AdamW
Adam parameters	$\beta_{1}=0.9$ , $\beta_{2}=0.95$ , $\epsilon=10^{-8}$

Appendix

Table 8: Primary optimization configuration. The values shown here define the candidate configuration used for all Qwen3-1.7B experiments. Domain-specific update counts are reported separately in Table 9 .

Domain	Teacher updates	Student updates	Eval. interval
Finance	1,200	1,000	100
Medical	3,000	2,500	250
Legal	1,800	1,500	150

Appendix

Table 9: Domain-specific update budgets. Teacher and student update counts are fixed before test evaluation.

Domain	Method	Target	MMLU	HellaSwag	ARC-C	General Avg.
Finance	Base	34.2	57.0	64.5	51.9	57.8
	Naive SFT	61.5	49.9	55.1	46.2	50.4
	OPSD	56.8	54.1	58.0	52.9	55.0
	OPD	55.5	51.9	56.5	50.6	53.0
	SSTD	59.5	54.3	58.2	53.1	55.2
Medical	Base	63.2	57.3	64.7	52.3	58.1

Appendix

Table 10: Complete Qwen3-1.7B results. Target denotes execution accuracy for FinQA and classification accuracy for MedMCQA and CaseHOLD. General Avg. is the arithmetic mean of MMLU, HellaSwag, and ARC-C.

Method	42	43	44	Mean $\pm$ Std.	42	43	44	Mean $\pm$ Std.
	Target				General Avg.
Naive SFT	61.2	61.9	61.4	$61.5\pm 0.36$	50.1	50.8	50.3	$50.4\pm 0.36$
OPSD	56.5	57.1	56.8	$56.8\pm 0.30$	54.8	55.3	54.9	$55.0\pm 0.26$
OPD	55.1	55.8	55.6	$55.5\pm 0.36$	52.7	53.4	52.9	$53.0\pm 0.36$
SSTD	59.2	59.9	59.4	$59.5\pm 0.36$	54.9	55.6	55.1	$55.2\pm 0.36$

Appendix

Table 11: Finance results by random seed.

Method	42	43	44	Mean $\pm$ Std.	42	43	44	Mean $\pm$ Std.
	Target				General Avg.
Naive SFT	75.5	76.1	75.8	$75.8\pm 0.30$	51.4	52.0	51.7	$51.7\pm 0.30$
OPSD	72.3	72.9	72.6	$72.6\pm 0.30$	55.5	56.1	55.8	$55.8\pm 0.30$
OPD	71.5	72.2	71.7	$71.8\pm 0.36$	54.0	54.6	54.3	$54.3\pm 0.30$
SSTD	73.9	74.6	74.1	$74.2\pm 0.36$	56.4	57.1	56.6	$56.7\pm 0.36$

Appendix

Table 12: Medical results by random seed.

Method	42	43	44	Mean $\pm$ Std.	42	43	44	Mean $\pm$ Std.
	Target				General Avg.
Naive SFT	81.7	82.3	82.0	$82.0\pm 0.30$	53.2	53.9	53.4	$53.5\pm 0.36$
OPSD	78.2	78.8	78.5	$78.5\pm 0.30$	56.9	57.5	57.2	$57.2\pm 0.30$
OPD	77.3	77.9	77.6	$77.6\pm 0.30$	55.3	55.9	55.6	$55.6\pm 0.30$
SSTD	79.8	80.4	80.1	$80.1\pm 0.30$	58.1	58.8	58.3	$58.4\pm 0.36$

Appendix

Table 13: Legal results by random seed.

Method	GPU-hours	Wall-clock	Peak memory	Teacher queries	Teacher tokens
Naive SFT	64	8.0 h	46 GB	0	0
OPSD	70	8.8 h	51 GB	0	0
OPD	166	20.8 h	72 GB	1.41M	180M
SSTD teacher stage	43	5.4 h	58 GB	0	0
SSTD student stage	66	8.3 h	64 GB	0.94M	120M
SSTD total	109	13.7 h	64 GB	0.94M	120M

Appendix

Table 15: D. Computational Cost: Qwen3-1.7B Finance. Teacher queries denote teacher forward evaluations at student-generated positions. Teacher tokens count all token positions for which teacher logits are computed.

Explore similar work

RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting
May 29, 2026Yuduo Li, Xiaofeng Shi, Qian Kou +2Supervised Fine-Tuning Fine-Tuning
Beijing Academy of Artificial Intelligence (BAAI) · Beijing Jiaotong University (BJTU)
Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation
Sep 12, 2026Yilei Tu, Zihao Li, Shaoxiong Ji +2Implicit Reasoning in Language Models Language Model Distillation
University of British Columbia · Shanghai Artificial Intelligence Laboratory · University of Helsinki +2
Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
Sep 28, 2026Xin Li, Hao Jiang, Xin Gao +6Language Model Distillation Multi-Teacher Knowledge Distillation
Nanyang Technological University · Yale University · University of Manchester

May 29, 2026J/K move · Enter open · S save

RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting

Yuduo Li, Xiaofeng Shi, Qian Kou, Longbin Yu+1

Beijing Academy of Artificial Intelligence (BAAI) · Beijing Jiaotong University (BJTU)

Supervised Fine-Tuning Fine-Tuning Domain Adaptation for LLMs On-Policy Distillation Catastrophic Forgetting

Open paper AskPDF arXiv

Domain-specific supervised fine-tuning (SFT) often improves in-domain performance at the cost of degrading a model's general capabilities. We view this degradation through two practical gaps in domain SFT: a supervision-compatibility gap, where domain targets differ in style and reasoning format from the original model's natural responses, and a trajectory-preservation gap, where teacher-forced SFT optimizes fixed target tokens without constraining the model's behavior on its own generated prefixes. This process fails to preserve the model's original behavior. We propose RAFT (Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting), a two-stage framework that addresses both factors. First, RAFT constructs model-compatible supervision through self-conditioned rewriting, semantic filtering, and answer fusion. Second, RAFT performs Answer-Conditioned On-Policy Distillation, where the original instruction-tuned model provides soft targets on student-generated trajectories while being conditioned on the fused answer as helpful context. We further introduce top-K temperature distillation and EMA-based adaptive loss balancing to stabilize the domain-general trade-off. Across three instruction-tuned backbones and five domains, RAFT improves average domain accuracy by 23.2% over standard SFT, while recovering part of the SFT-induced degradation on MS-Bench and IFEval, with relative improvements of 18.2% and 10.2%, respectively. These results show that coupling data refinement with trajectory-level preservation provides an effective recipe for domain fine-tuning with alleviated forgetting.

Abstract

Explore similar work

RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting

Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation