Beyond Distribution Matching: Selective Margin Preservation for Supervised Fine-Tuning

Yuxuan Wang; Guangxiang Zhao; Xinyu Tan; Yahui Han; Xinxiang Yin; Zhonghai Wu; Xiangzheng Zhang; Lin Sun

cs.LGOct 14, 2026

Beyond Distribution Matching: Selective Margin Preservation for Supervised Fine-Tuning

Authors: Yuxuan Wang, Guangxiang Zhao, Xinyu Tan, Yahui Han, Xinxiang Yin, Zhonghai Wu, Xiangzheng Zhang, Lin Sun,

Organizations: Peking University · Xidian University · Qiyuan Tech · The University of Hong Kong

Abstract

Supervised fine-tuning (SFT) changes logit margins among non-target tokens at teacher-forced prefixes. Matching a local reference distribution keeps those probabilities and also penalizes margin expansions. We ask whether a soft auxiliary can preserve selected local non-target margin relations without freezing that distribution. We introduce Selective Rank-Preserving SFT (SRP-SFT), a soft margin target: a hinge on reference-scaled lower bounds for pairs from the reference's top- $K$ non-target tokens, applied at fixed teacher-forced prefixes. The hinge stops once the requested bound is met. We derive the KL cost of violating a margin, characterize the feasible region and its conditional utility, and give a finite-step condition for reducing aggregate violations. In one-epoch experiments on Qwen3-4B-Base and UltraFeedback, this target raises MATH-500 and code-generation scores over CE and LP-SFT at all three training seeds. At seed 1234, MATH-500 pass@1 rises from 49.56% to 54.56% and the four-metric code mean rises from 65.91% to 69.50% relative to LP-SFT. Ranking controls, held-out cache diagnostics, sensitivity sweeps, and six additional backbones show where these score changes appear.

Figures & tables

Figure 1: The three objectives preserve different local quantities. CE fits the observed token by maximizing its likelihood. LP-SFT matches a reference distribution. SRP-SFT imposes selected margin lower bounds and stops penalizing a pair once its bound is met.

Figure 2: The selective margin target. The frozen reference selects plausible non-target alternatives and orients their pairs. CE fits the excluded target. The current student receives a pairwise penalty only below the requested margin. A satisfied pair supplies no direct auxiliary gradient. Both models use the same prefix.

Method	MATH-500		AIME		GSM-Plus mini		LogiQA2	BBH-5	ARC-C	Hella- Swag	MMLU	Phy-Oly	Mean
	p@1	p@4	p@1	p@4	Strict	Flexible	Acc.	EM	Norm.	Norm.	5-shot	Acc.
Base †	48.11	71.23	1.81	5.17	51.92	60.21	44.15	16.40	52.30	73.70	73.00	8.90	42.24
CE	27.98	54.91	0.54	1.99	45.62	43.83	43.32	13.60	53.07	72.16	68.90	5.93	35.99
GEM	22.21	47.74	0.54	1.94	46.58	47.38	43.70	12.80	49.49	73.40	68.83	5.51	35.01
DFT	47.73	59.57	0.66	2.19	45.58	44.50	39.63	0.40	47.70	65.96	65.23	3.81	35.25
ASFT	33.41	57.22	2.03	5.42	47.50	48.79	37.09	18.40	55.03	70.72	67.56	8.47	37.64

Table 1: One-epoch Qwen3-4B-Base reasoning results on UltraFeedback (seed 1234, percentages). All fine-tuned methods use the same training implementation, software and hardware environment, and common training hyperparameters, including a 478-step cosine schedule and 15-step warmup. Mean averages all twelve metrics from the same checkpoint. Bold/underline mark best/second-best. † is the untouched Base.

Method	HumanEval+		MBPP+		Mean
	pass@1	pass@4	pass@1	pass@4
Base †	45.49	71.56	46.12	72.50	58.92
CE	44.18	69.56	44.66	65.79	56.04
GEM	22.93	46.34	26.40	52.13	36.95
DFT	64.12	69.98	57.24	63.56	63.72
ASFT	64.79	77.18	57.09	67.90	66.74

Table 2: One-epoch code results for Qwen3-4B-Base on UltraFeedback (seed 1234, percentages). Mean averages HumanEval+ and MBPP+ pass@1/pass@4.

Method	Reasoning			Code
	seed 1234	seed 42	seed 3407	seed 1234	seed 42	seed 3407
CE	35.99	36.03	35.85	56.04	54.33	56.60
GEM	35.01	34.89	35.52	36.95	33.93	36.88
DFT	35.25	33.37	35.77	63.72	61.35	58.38
ASFT	37.64	40.51	37.46	66.74	66.37	67.87
EAFT	34.91	35.81	35.89	57.12	57.32	57.65

Table 3: Three-seed reasoning and code means (percentages). Reasoning averages twelve metrics from the same checkpoint. Code averages four metrics.

Loss	MATH	HE+	MBPP+	Mean
CE	27.98	44.18	44.66	38.94
LP-SFT	49.56	57.90	51.87	53.11
RankOnly ( $\rho=0$ )	41.89	55.37	49.96	49.07
RankNet	46.41	59.09	52.63	52.71
ListMLE	52.15	61.71	56.18	56.68
Spearman	46.74	56.68	52.08	51.83

Table 4: Fixed-configuration loss screen (pass@1, percentages). Mean averages MATH, HumanEval+, and MBPP+. Code columns use the Table 2 evaluation. † Same target as LP-SFT. Definitions are in Appendix H .

Figure 3: Mechanism diagnostics on the held-out cache. Distributional fidelity and margin preservation disagree on this cache: SRP-SFT has less contraction despite larger local KL. Small points are tokens. Large markers are pair-weighted means.

Figure 4: Preservation demand and task utility need not move together. Each panel varies one of

\rho

\mu

, and

K

. Bands mark the best score. The score averages MATH pass@1 and mean code pass@1.

Method	$\alpha$	MATH		HE+		MBPP+
		p@1	p@4	p@1	p@4	p@1	p@4
CE	1	27.98	54.91	44.18	69.56	44.66	65.79
CE-TM	1.081	28.98	55.87	45.88	70.44	45.73	65.33
LP-SFT	1	49.56	77.52	57.90	81.67	51.87	72.23
LP-TM	1.099	51.96	78.08	59.82	82.39	54.84	73.06
SRP-SFT	1	54.56	80.50	62.84	83.87	56.24	75.06

Table 5: Entropy-matched temperature control (seed 1234, percentages). LP-TM and CE-TM are calibrated to the SRP-SFT held-out entropy.

Figure 5: Additional-backbone pass@1 on MATH-500, GSM8K, HumanEval+, and MBPP+. Math and code use separate domain-trained checkpoints.

\Delta

is SRP-SFT minus the mean of CE and LP-SFT.

Figure 6: Ten-epoch trajectories with SRP

\rho=0.5

. MATH averages pass@1 and pass@4. Code averages pass@1 and pass@4 on both benchmarks.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Method	HumanEval+			MBPP+
	p@1	p@4	p@16	p@1	p@4	p@16
Base †	45.49	71.56	87.81	46.12	72.50	84.13
CE	44.18	69.56	84.10	44.66	65.79	76.73
LP-SFT	57.90	81.67	90.50	51.87	72.23	81.07
SRP-SFT	62.84	83.87	91.76	56.24	75.06	84.64

Appendix

Table 6: Matched code results for Base, CE, LP-SFT, and SRP-SFT (seed 1234, percentages), adding pass@16 to Table 2 . All entries use unbiased estimates from 20 samples per task.

Method	Metric mean	Benchmark mean
Base †	42.24	43.07
CE	35.99	38.27
GEM	35.01	37.44
DFT	35.25	35.87
ASFT	37.64	39.39
EAFT	34.91	37.38

Appendix

Table 7: Equal metric versus equal benchmark weighting for the seed-1234 reasoning screen (percentages). Both aggregates require all twelve metrics.

Method	Reasoning	Code
CE	$35.96\pm 0.10$	$55.66\pm 1.18$
LP-SFT	$43.10\pm 0.46$	$66.17\pm 0.27$
SRP-SFT	$44.96\pm 0.55$	$69.48\pm 0.02$

Appendix

Table 8: Mean

\pm

sample standard deviation across training seeds 1234, 42, and 3407 (percentages). Code averages four metrics within each seed before computing the statistics.

Method	HE+ p@1	p@4	MBPP+ p@1	p@4	Code mean
CE	42.74	66.61	43.06	64.92	54.33
LP-SFT	58.29	82.35	51.68	72.24	66.14
GEM	20.15	43.99	22.12	47.63	33.47
DFT	61.74	68.60	54.40	60.44	61.30
SRP ( $\rho=1$ )	62.80	84.39	56.19	74.49	69.47
LP-TM (target $\rho=1$ )	62.16	83.29	53.77	72.37	67.90

Appendix

Table 9: Seed-42 code results (percentages). The seed-42 LP-TM code evaluation uses

\alpha=1.0989

Method	MATH-500		AIME		GSM-Plus mini		LogiQA2	BBH-5	ARC-C	Hella- Swag	MMLU	Phy-Oly	Mean
	p@1	p@4	p@1	p@4	Strict	Flexible	Acc.	EM	Norm.	Norm.	5-shot	Acc.
Training seed 42
CE	31.72	58.62	0.59	2.16	45.42	45.08	42.75	2.80	53.41	71.87	69.07	8.90	36.03
GEM	20.65	47.71	0.43	1.61	47.71	48.00	44.34	8.00	53.33	72.94	69.29	4.66	34.89
DFT	31.51	42.05	0.56	1.17	50.67	52.25	31.81	0.80	51.02	65.91	66.73	5.93	33.37
ASFT	45.50	68.85	2.73	6.94	52.92	54.12	37.15	15.60	53.58	70.75	67.80	10.17	40.51

Appendix

Table 10: Independent seed-42 and seed-3407 reasoning results (percentages). Each mean averages all twelve metrics from the same checkpoint. Both SRP-SFT rows use

K=10

\rho=1

\mu=1

Training seed 42
Method	HumanEval+		MBPP+		Mean
	pass@1	pass@4	pass@1	pass@4
CE	42.74	66.61	43.06	64.92	54.33
GEM	20.00	44.48	22.91	48.31	33.93
DFT	61.86	68.59	54.48	60.49	61.35
ASFT	63.02	76.25	57.78	68.45	66.37

Appendix

Table 11: Independent-seed EvalPlus results (percentages). Mean averages HumanEval+ and MBPP+ pass@1/pass@4 using all 20 samples per task. All panels use generation seed 1234, temperature 0.8, top-

p

1, and token/context caps 4096/8192.

Setting	Mathematics	Code
Retained training examples	96,949	95,966
Example presentations per epoch	97,024	96,000
Sequence length (tokens)	4,096	4,096
Effective batch size	128	128
Epochs	1	1
Optimizer updates / cosine horizon	758	750

Appendix

Table 12: Cross-backbone training budgets, shared by CE, LP-SFT, and SRP-SFT for every backbone. Example presentations include deterministic sampler padding to complete gradient-accumulation windows.

Initialization (repository ID)	Revision	Math tokens	Code tokens
Qwen/Qwen3.5-2B-Base	b1485b2fa6df	306,157,988	234,957,303
Qwen/Qwen3.5-0.8B-Base	dc7cdfe2ee41	306,157,988	234,957,303
ibm-granite/granite-4.1-3b	c0650403e44e	294,685,290	224,987,190
mistralai/Ministral-3-3B-Instruct-2512-BF16	b6d637bef239	283,617,839	220,836,597
google/gemma-3-4b-it	093f9f388b31	305,993,476	240,191,774
HuggingFaceTB/SmolLM3-3B	a07cc9a04f16	292,833,133	224,365,722

Appendix

Table 13: Pinned cross-backbone initializations (12-character revision prefixes) and supervised-token counts after tokenization and truncation. Counts cover one copy of each retained example. Token budgets vary with the tokenizer.

Method	Qwen3.5- 2B Base	Qwen3.5- 0.8B Base	Granite- 4.1-3B Instruct
Base	10.94	1.65	6.54
CE	5.42	1.77	17.38
LP-SFT	6.83	1.58	19.46
SRP-SFT	11.31	1.77	19.31

Appendix

Table 14: AIME 2021–2025 pass@1 results for Qwen3.5-2B Base, Qwen3.5-0.8B Base, and Granite-4.1-3B Instruct.

Method	Ministral- 3-3B Instruct	Gemma- 3-4B it	SmolLM3- 3B Instruct
Base	18.42	9.62	8.67
CE	17.48	5.58	30.31
LP-SFT	20.88	9.52	30.06
SRP-SFT	21.46	8.33	29.23

Appendix

Table 15: AIME 2021–2025 pass@1 results for Ministral-3-3B Instruct, Gemma-3-4B it, and SmolLM3-3B Instruct.

Qwen3.5-2B Base
Method	GSM8K		MATH-500
	pass@1	pass@16	pass@1	pass@16
Base	80.70	96.51	69.25	89.40
CE	73.46	94.69	56.95	88.60
LP-SFT	76.46	95.75	65.59	91.80
SRP-SFT	78.13	96.21	70.39	92.40

Appendix

Table 16: Reasoning checks for Qwen3.5-2B Base, Qwen3.5-0.8B Base, and Granite-4.1-3B Instruct (percentages). Columns report GSM8K and MATH-500 pass@1/pass@16.

Ministral-3-3B Instruct
Method	GSM8K		MATH-500
	pass@1	pass@16	pass@1	pass@16
Base	80.20	97.35	66.53	93.80
CE	88.03	96.97	73.69	91.60
LP-SFT	89.93	97.27	74.76	91.80
SRP-SFT	90.36	97.42	75.30	91.20

Appendix

Table 17: Reasoning checks for Ministral-3-3B Instruct, Gemma-3-4B it, and SmolLM3-3B Instruct (percentages). Columns report GSM8K and MATH-500 pass@1/pass@16.

	Qwen3.5-2B Base		Qwen3.5-0.8B Base
Method	Human Eval+	MBPP+	Human Eval+	MBPP+
Base	38.17	37.51	20.67	19.64
CE	30.61	23.86	16.65	15.09
LP-SFT	38.75	34.27	21.34	18.64
SRP-SFT	39.76	35.21	21.89	20.17

Appendix

Table 18: rStar-Coder adaptation on the two base-model backbones (EvalPlus pass@1 percentages).

Granite-4.1-3B Instruct
Method	HumanEval+	MBPP+	Mean
Base	76.25	59.33	67.79
CE	65.49	51.71	58.60
LP-SFT	73.05	55.87	64.46
SRP-SFT	76.22	56.73	66.48
Ministral-3-3B Instruct

Appendix

Table 19: rStar-Coder adaptation on four instruction-tuned backbones (EvalPlus pass@1 percentages). Mean averages HumanEval+ and MBPP+.

Backbone	MATH	GSM8K	HE+	MBPP+
Granite	-0.70	+0.04	+3.17	+0.86
SmolLM3	+0.28	-0.12	+0.58	+1.47
Ministral	+0.54	+0.43	+1.62	-0.61
Gemma	-2.35	+0.00	+4.09	+3.13
Qwen 2B	+4.80	+1.67	+1.01	+0.94
Qwen 0.8B	+0.25	-1.83	+0.55	+1.53

Appendix

Table 20: Direct cross-backbone

\Delta_{\mathrm{LP}}=\mathrm{SRP}-\mathrm{LP}

in pass@1 percentage points, using the same source data as Figure 5 . Negative values are retained.

Grid	Value	MATH p@1 $\uparrow$	Code avg. $\uparrow$	Avg. $\uparrow$	Flip $\downarrow$	Contr. $\downarrow$
$\rho$	0.00	41.02	53.08	47.05	.0828	.4123
	0.25	47.80	53.57	50.69	.0740	.3549
	0.50	50.12	54.65	52.39	.0666	.2663
	0.75	52.78	58.56	55.67	.0596	.1563
	1.00	54.56	59.54	57.05	.0565	.1098
	1.25	56.83	61.51	59.17	.0583	.0950

Appendix

Table 21: One-factor sensitivity of

\rho

\mu

, and

K

. MATH is MATH-500 pass@1. Code avg. Averages HumanEval+ and MBPP+ pass@1. Avg. Averages the two domains. Flip and Contr. are lower-is-better diagnostics.

Figure 7: Performance–geometry frontier over the reported $\rho$ sweep. Bubble area is the active-pair rate. Lower contraction is preferable.

Variant	UF NLL	Math/ Code	Flip	Contr.	KL
CE	.917	38.94	.1540	.5995	.3592
$\rho=0$	.919	49.06	.0828	.4123	.0797
Include-GT	.922	52.58	.0705	.2750	.0564
$\rho=.5$ (sensitivity ref.)	.922	53.14	.0666	.2663	.0523
Adjacent-rank pairs	.927	55.26	.0573	.3238	.0470
Top1-vs-rest	.919	50.72	.0946	.2961	.0873

Appendix

Table 22: Objective and sparse-pair ablations around the

\rho=0.5

sensitivity reference. The Random-

K

arm is omitted.

Variant	MATH p@1	MATH p@4	Code mean	Flip	Contr.	KL
Full pairs	54.56	80.50	69.50	0.0565	0.1098	0.0748
Adjacent	57.65	82.02	69.04	0.0520	0.1346	0.0431

Appendix

Table 23: SRP pair-graph controls at

\rho=1

K=10

\mu=1

(training seed 1234). Full pairs is the headline checkpoint, and its task scores are the main-table evaluation. Adjacent pairs is a separately trained checkpoint. Task scores are percentages. Code mean averages the four matched HumanEval+/MBPP+ pass@1/pass@4 metrics. Diagnostics use the 998-example held-out cache. Both checkpoints have 478 training steps. Flip is a fraction. Contr. is in logit units.

Method	Pairs	Tokens/s	Wall (s)	Peak (GiB)
CE	–	6748	175.31	19.91
LP-SFT	–	6743	180.81	19.91
SRP full	45	6625	176.41	19.91
SRP adjacent	9	6514	181.48	19.91

Appendix

Table 24: Measured 20-step efficiency pilots with the same eight-GPU allocation and training configuration. Throughput excludes the first five warmup steps and checkpoint I/O. Wall time covers the training run. Peak is the maximum allocated memory over ranks. Pairs counts auxiliary rank constraints per position. CE and LP-SFT have no auxiliary pairs.

Paired check	Pairs	Result
Input and RNG hashes	1,024	All match
Candidate-mask hashes	1,024	All match
Selected-logit-gradient hashes	1,024	All match
Maximum CE loss difference	1,024	$0$
Maximum loss–entropy residual	1,024	$7.15\!\times\!10^{-7}$
Maximum update gradient-norm difference	64	$0$

Appendix

Table 25: LP-SFT/ListNet training identity audit. Microbatch checks cover

8

ranks

\times

8

updates

\times

16

accumulation steps. Gradient norms cover all

64

rank–update pairs. The nine parameter snapshots comprise initialization and eight post-update states on rank 0. The loss residual subtracts the reduced teacher entropy from the ListNet-minus-LP total objective. Its absolute tolerance is

10^{-5}

Method	MATH		HE+		MBPP+		Code mean
	p@1	p@4	p@1	p@4	p@1	p@4
ListMLE	52.21	79.58	62.35	83.85	55.46	73.74	68.85
Margin-MSE	49.19	79.77	61.52	84.66	54.85	74.64	68.92
SRP-SFT	54.56	80.50	62.84	83.87	56.24	75.06	69.50

Appendix

Table 26: One-epoch ranking controls at

\mu=1

(seed 1234, percentages).

Loss	MATH	HE+	MBPP+	Mean	Flip $\downarrow$	Contr. $\downarrow$
CE	27.98	44.18	44.66	38.94	0.1540	0.5995
LP-SFT	49.56	57.90	51.87	53.11	0.0741	0.2681
RankOnly ( $\rho=0$ )	41.89	55.37	49.96	49.07	.0828	.4124
RankNet	46.41	59.09	52.63	52.71	.0853	.1872
ListMLE	52.15	61.71	56.18	56.68	.0691	.1271
Margin-MSE	48.68	61.07	54.72	54.82	.0380	.1522

Appendix

Table 27: Representative ranking-loss screen. MATH, HE+, and MBPP+ are pass@1. Mean averages them. Flip is a fraction and Contr. is in logit units. SRP-full (

\rho=1

) is the headline checkpoint and matches Table 2 . SRP-SFT (

\rho=.5

) is the sensitivity-reference checkpoint in Table 21 . Its HumanEval+ and MBPP+ pass@1 average to that table’s code score of 54.65 before rounding.

Relation and preservation diagnostics
Loss	Relation	Flip	Contr.
CE	None	0.1540	0.5995
LP-SFT	Distribution	0.0741	0.2681
RankOnly ( $\rho=0$ )	Pairwise order	.0828	.4124
RankNet	Pairwise order	.0853	.1872
ListMLE	Listwise order	.0691	.1271

Appendix

Table 28: Relation and directional diagnostics for the ranking-loss screen. Flip is a fraction and Contr. is in logit units. Both are lower-is-better. † ListNet is LP-equivalent. Performance is in Table 4 .

Method	$H$	$\alpha^{*}$	$1/\alpha^{*}$	Contr.	Flip	Viol.
CE	0.906	1.081	0.925	0.600	0.154	0.501
LP-SFT	0.938	1.099	0.910	0.268	0.074	0.513
SRP-SFT	0.802	1	1	0.110	0.056	0.229

Appendix

Table 29: Mechanism diagnostics on the held-out cache (280,908 response positions).

H

is mean full-vocabulary entropy. Contr./Flip are contraction/inversion, and Viol. is the fraction below the

\rho=1

margin. CE and LP-SFT

\alpha^{*}

match the SRP-SFT entropy.

Method	Mean ratio	CV	$R^{2}$	Below $\rho$
CE	0.942	5.829	0.360	0.514
LP-SFT	0.934	1.952	0.833	0.539
LP-TempMatch	1.043	1.952	0.833	0.447
SRP-SFT	1.244	1.346	0.898	0.229

Appendix

Table 30: Teacher-preferred margin geometry for

\Delta^{T}>0.1

. Mean ratio is

\Delta^{S}/\Delta^{T}

. CV is its coefficient of variation.

R^{2}

is the through-origin fit. LP-TempMatch uses

\alpha^{*}=1.116

Epoch	Base	CE	LP-SFT	DFT	ASFT	SRP-SFT
0.0	60.24	–	–	–	–	–
0.5	–	43.22	61.70	50.30	52.27	64.21
1.0	–	40.00	59.61	24.36	37.18	60.32
1.5	–	46.72	60.16	28.13	57.97	58.42
2.0	–	42.50	53.04	31.68	46.70	56.54
2.5	–	48.93	57.10	33.27	46.85	59.30

Appendix

Table 32: MATH mean convergence values underlying Figure 6 . Step 0 is this run’s Base draw under the same MATH-500 manifest and sampling protocol as Table 1 .

Epoch	Base	CE	LP-SFT	DFT	ASFT	SRP-SFT
0.0	59.17	–	–	–	–	–
0.5	–	55.66	66.12	61.03	66.34	66.31
1.0	–	52.87	64.32	60.12	66.66	65.25
1.5	–	58.47	65.75	61.47	65.56	65.46
2.0	–	58.01	65.73	60.98	65.66	66.47
2.5	–	61.91	67.44	60.83	67.99	68.13

Appendix

Table 33: Code mean convergence values underlying Figure 6 .

Supervised Fine-Tuning

Supervised fine-tuning adapts pretrained models by learning from labeled examples, instructions, demonstrations, and task-specific trajectories.

See all 223 papers

A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design
Jun 9, 2026Tong Xie, Yuanhao Ban, Yunqi Hong +3Supervised Fine-Tuning Fine-Tuning
University of California, Los Angeles (UCLA) · Arena
LP-SFT: Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure
Jul 6, 2026Yueyang Wang, Baolong Bi, Shuo Lu +1Supervised Fine-Tuning Fine-Tuning
School of Mathematical Sciences, Peking University · Institute of Computing Technology, Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences +1
PriFT: Guiding Supervised Fine-Tuning with Pretrained Priors
Jun 8, 2026Ke Wang, Shuangqi Li, Mathieu Salzmann +1Supervised Fine-Tuning Fine-Tuning
EPFL, Lausanne, Switzerland

Jun 9, 2026J/K move · Enter open · S save

A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design

Tong Xie, Yuanhao Ban, Yunqi Hong, Sohyun An+2

University of California, Los Angeles (UCLA) · Arena

Supervised Fine-Tuning Fine-Tuning LLM Fine-Tuning

Open paper AskPDF arXiv

Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory. However, an observed token can be non-unique, noisy, or misaligned with the model prior. Strictly fitting toward this one-hot target may be suboptimal, especially when the pretrained model encodes a rich knowledge prior. In this work, we reinterpret SFT as target distribution design: instead of studying only the loss objective, we analyze the token-level target that the loss drives the model to match. We introduce the Q-target framework, which decomposes SFT supervision into two explicit choices: (1) how strongly to rely on the observed token, and (2) how to allocate the remaining probability mass over alternatives. This perspective unifies many existing SFT variants as implicit choices of the target distribution Q. Building on this view, we propose Target-SFT which constructs the training objective directly from the desired target distribution. This method consistently outperforms across the ten reasoning dataset-model settings evaluated, showing the effectiveness of this target-based approach. Overall, our formulation reveals a more fundamental design principle for SFT training and opens a broader search space for SFT objectives.

Abstract

Supervised Fine-Tuning

Explore similar work

A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design

LP-SFT: Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure

PriFT: Guiding Supervised Fine-Tuning with Pretrained Priors