SoFT: Soft Targets for Generalizable LLM Fine-Tuning

Huihao Jing; Wenbin Hu; Shaojin Chen; Haochen Shi; Zhongwei Xie; Guijia Zhang; Yuxuan Liu; Haoyu Huang

cs.LGSep 26, 2026

SoFT: Soft Targets for Generalizable LLM Fine-Tuning

Authors: Huihao Jing, Wenbin Hu, Shaojin Chen, Haochen Shi, Zhongwei Xie, Guijia Zhang, Yuxuan Liu, Haoyu Huang, +2 more

Organizations: HKUST, HKUST-GZ

Abstract

Distillation enables student language models to acquire new capabilities from expert teachers. However, integrating knowledge from multi-teacher, multi-domain demonstrations into a single student remains challenging. We study supervised fine-tuning (SFT) in this setting, where students must acquire diverse capabilities while maintaining generalization beyond the training tasks. Our experiments reveal varying trade-offs between in-distribution learning and out-of-distribution generalization across SFT methods, motivating more explicit control over this balance. To this end, we propose soft-target fine-tuning (SoFT) to balance learning from teacher demonstrations with retaining the Base model's existing capabilities. SoFT sets a minimum target probability for each demonstrated token while making the smallest KL change to the Base distribution. The resulting objective couples learning from demonstrations with adaptively weighted regularization toward the Base model. We further use domain-specific gradient budgets to control this balance and determine a probability threshold for each trajectory. Experiments on mixed-domain reasoning and agentic tasks show that SoFT achieves the best overall performance among the compared methods, with improvements in both in-distribution capability acquisition and out-of-distribution generalization.

Figures & tables

Figure 1: SoFT overview and mixed-data outcomes. Left: schematic targets; SFT uses a one-hot target, while SoFT retains Base-supported alternatives. Right: changes relative to Base across reasoning and agentic domains.

Figure 2: From a shared gradient budget (left), to a sequence-specific floor (middle), to token-level SoFT targets (right). Values are illustrative.

Setting	Domain	ID (cases)	OOD (cases)
Reasoning	Math	OpenR1-Math (300)	GSM8K (1,319)
	Code	APPS (250)	MBPP+ (378)
	Science	Science MCQ (1,000)	ARC-Challenge (608)
	General	RiddleSense (1,018)	BoolQ (708)
Agentic	Customer	$\tau^{2}$ (60)	$\tau^{2}$ (300)
	Coding	LiveCodeBench (694)	BigCodeBench (2,280)

Table 1: Evaluation suites and case counts before repeated attempts. BigCodeBench counts evaluation cases.

Model	Method	Math	APPS	Riddle	Science	GSM8K	ARC-C	BoolQ	MBPP+	GM
		In-distribution (ID)				Out-of-distribution (OOD)
Qwen2.5-1.5B	Base	34.00	6.80	66.50	76.50	87.34	85.03	82.91	60.85	50.76
	SFT	37.00	8.00	70.73	78.30	87.11	88.49	82.77	63.76	53.46
	Confidence	33.33	6.00	71.81	79.00	85.06	88.82	83.05	62.70	50.84
	Rho-1	35.00	6.80	72.00	77.30	86.05	87.99	83.47	60.58	51.66
	DFT	31.00	6.40	68.66	73.40	83.55	86.35	81.21	60.58	49.40

Table 2: Reasoning performance (%) under the fixed four-attempt schedule. Bold marks column bests per model size, including ties. Math: OpenR1-Math; Riddle: RiddleSense.

Model	Method	$\tau^{2}$	LCB	Search	TMax	$\tau^{2}$	BCB	BFCL V4	FRAMES	GM
		In-distribution (ID)				Out-of-distribution (OOD)
Qwen3-8B	Base	48.33	39.05	23.80	7.33	20.67	53.51	40.20	46.12	30.11
	SFT	35.00	37.32	26.20	16.00	32.00	58.20	21.10	46.00	31.58
	Confidence	40.00	35.59	17.60	5.00	22.00	56.32	34.15	48.67	26.70
	Rho-1	58.33	40.20	22.80	9.00	29.33	53.51	40.60	45.15	32.93
	DFT	56.67	37.32	23.00	5.33	27.67	52.02	41.32	44.66	30.18

Table 3: Agentic performance (%) under the fixed four-attempt schedule. Bold marks column bests, including ties. LCB: LiveCodeBench; BCB: BigCodeBench.

Figure 3: Benchmark GM gain versus policy drift relative to SFT. The pale rounded region highlights the three SoFT points. Colors denote methods, shapes denote settings, and the horizontal axis uses a symmetric log scale near zero.

Figure 4: First-attempt output lengths for all eight methods across eight benchmark means per setting. Boxes: Q1–Q3; center lines: medians; diamonds: means. Agentic lengths cover scorer-facing outputs, not total rollout cost.

Figure 5: Realized acquisition

A_{\theta}

versus benchmark GM for seven methods at each reasoning scale. Horizontal bars show exploratory resampling ranges; dashed lines mark Base GM. The validation subset was used during hyperparameter selection.

Figure 6: Target KL from Base as the gradient budget varies. Top: absolute KL of the constructed target; bottom: KL relative to each trajectory’s one-hot endpoint. Stars mark selected budgets.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Task-data source	Teacher	Train	Link	Min	Mean	Max
Math	GLM cleaned	GLM-5.1	450	source	1,371	5894.3	8,187
Math	Kimi cleaned	Kimi-K2.5	690	source	2,361	6561.5	8,191
Math	Sky-T1	QwQ-32B / STILL-2	2,400	source	269	3927.4	8,165
Math	OpenThoughts-4 math	Qwen3-235B-A22B	2,000	source	1,535	6658.6	8,192
Math	Mixture-of-Thoughts	DeepSeek-R1	3,460	source	468	4109.7	8,189
Code	Sky-T1	QwQ-32B / STILL-2	3,600	source	798	4898.2	8,191

Appendix

Table 4: Reasoning training sources, teachers, and full-sequence token statistics. Validation examples are excluded.

Domain group	Task-data source	Teacher	Train	Link	Min	Mean	Max
Coding	LiveCodeBench v1	DeepSeek V4 Flash	1,282	source	155	683.0	4,333
Customer	Nemotron interactive	Upstream; unspecified	1,200	source	1,837	3972.1	8,534
Customer	$\tau^{2}$ retail/airline	DeepSeek V4 Flash	2,997	source	4,899	9127.8	22,431
Search	Nemotron search	Upstream; unspecified	600	source	3,892	18791.6	32,701
Tool Interaction	TMax-SFT	Qwen3.6-27B	595	source	1,661	5575.2	30,575
	OpenThoughts-Agent	Upstream; unspecified	387	source	949	2730.1	17,593

Appendix

Table 5: Agentic training sources, teachers, and full-sequence token statistics. Validation examples are excluded.

Figure 7: Token-weighted target allocation from Base caches. Lines: mean

a_{t}

by Base probability; circle areas: token fractions; legend percentages: fraction with

a_{t}>0

. The 8B panel pools source data into four domain groups.

Figure 8: Final policy drift versus Base-success retention on reasoning tasks. Drift is

\mathrm{KL}(\pi_{0}\|\pi_{\theta})

over held-out teacher-forced prefixes; retention is the equal-weight mean over eight benchmarks.

Figure 9: Training objectives averaged over 50-step (reasoning) or 10-step (agentic) windows. Objective scales vary across methods; benchmark scores report task performance.

Figure 10: Predictive entropy (nats). Left/middle: recorded held-out evaluations. Right: training entropy in 10-step windows. Populations differ across panels; missing points are not imputed.

Figure 11: Logged gradient norms (log scale), using the windows in Figure 9 . Magnitudes depend on objective scaling and trainer conventions.

Figure 12: Recorded learning-rate schedules without smoothing. Identical schedules overlap.

	Qwen2.5-1.5B			Qwen2.5-7B
Domain	SFT	DFT	SoFT	SFT	DFT	SoFT
General	20.6	41.6	4.0	43.4	50.1	8.8
Math	24.9	47.2	6.0	47.2	52.9	11.3
Science	18.8	37.1	7.5	36.9	44.4	11.8
Code	18.9	40.0	10.2	38.4	44.2	16.0

Appendix

Table 6: Realized acquisition by reasoning domain (%). Values use the same 32 validation examples per domain as Figure 5 .

Explore similar work

RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting
May 29, 2026Yuduo Li, Xiaofeng Shi, Qian Kou +2Supervised Fine-Tuning Fine-Tuning
Beijing Academy of Artificial Intelligence (BAAI) · Beijing Jiaotong University (BJTU)
Learning to Adapt SFT Data for Better Reasoning Generalization
May 26, 2026Lisong Sun, Li Wang, Chen Zhang +4Supervised Fine-Tuning LLM Post-Training
Beihang University · Tsinghua University · Nanyang Technological University +2
iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
Sep 21, 2026Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber +4Continual Learning for LLMs Language Model Distillation
Imperial College London · Politecnico di Torino · UCL Centre for AI

May 29, 2026, cs.LGJ/K move · Enter open · S save

RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting

Yuduo Li, Xiaofeng Shi, Qian Kou, Longbin Yu+1

Beijing Academy of Artificial Intelligence (BAAI) · Beijing Jiaotong University (BJTU)

Supervised Fine-Tuning Fine-Tuning Domain Adaptation for LLMs On-Policy Distillation Catastrophic Forgetting

Open paper AskPDF arXiv

Domain-specific supervised fine-tuning (SFT) often improves in-domain performance at the cost of degrading a model's general capabilities. We view this degradation through two practical gaps in domain SFT: a supervision-compatibility gap, where domain targets differ in style and reasoning format from the original model's natural responses, and a trajectory-preservation gap, where teacher-forced SFT optimizes fixed target tokens without constraining the model's behavior on its own generated prefixes. This process fails to preserve the model's original behavior. We propose RAFT (Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting), a two-stage framework that addresses both factors. First, RAFT constructs model-compatible supervision through self-conditioned rewriting, semantic filtering, and answer fusion. Second, RAFT performs Answer-Conditioned On-Policy Distillation, where the original instruction-tuned model provides soft targets on student-generated trajectories while being conditioned on the fused answer as helpful context. We further introduce top-K temperature distillation and EMA-based adaptive loss balancing to stabilize the domain-general trade-off. Across three instruction-tuned backbones and five domains, RAFT improves average domain accuracy by 23.2% over standard SFT, while recovering part of the SFT-induced degradation on MS-Bench and IFEval, with relative improvements of 18.2% and 10.2%, respectively. These results show that coupling data refinement with trajectory-level preservation provides an effective recipe for domain fine-tuning with alleviated forgetting.

Abstract

Explore similar work

RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting

Learning to Adapt SFT Data for Better Reasoning Generalization

iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs