Self-Privileged Fine-Tuning

Yangrong Hu; Xueyu Zhou; Yue Wu; Jian Huang

cs.LGNov 3, 2026

Self-Privileged Fine-Tuning

Authors: Yangrong Hu, Xueyu Zhou, Yue Wu, Jian Huang

Organizations: Department of Data Science and Artificial Intelligence The Hong Kong Polytechnic University · Department of Applied Mathematics The Hong Kong Polytechnic University

Abstract

Supervised fine-tuning (SFT) is a key component of language model post-training, enabling effective task adaptation and serving as an important foundation for subsequent reinforcement learning. Recent methods seek to improve SFT by introducing token-level weights into the training objective, where these weights are typically derived from a model's output distribution at each token position. To better understand how such weighting schemes affect SFT, we analyze their optimization objectives and theoretically characterize the target distributions they induce. Our analysis shows that dynamic weighting methods, such as Dynamic Fine-Tuning (DFT), induce a heavily sharpened distribution, whereas static weighting methods fit a target distribution proportional to the product of the data distribution and a frozen prior. These findings suggest that static weighting provides a more stable training target, while its effectiveness critically depends on the quality of the prior distribution. Building on this observation, we propose Self-Privileged Fine-Tuning (SPFT), which uses a frozen copy of the initial model conditioned on privileged information (such as ground truth for mathematical questions) as a prior to provide guidance for token-level weighting during fine-tuning. Through a Bayesian decomposition, we further show that SPFT reweights the standard static prior according to how informative each target token is for predicting the ground-truth outcome, thereby assigning larger weights to more predictive tokens. Experiments on mathematical reasoning and code generation show that SPFT improves over standard SFT and other baselines across the main settings.

Figures & tables

Panel of Figure 1 — Figure 1: Next-token distribution evolution of SFT, DFT, and SPFT at four training steps.

Panel of Figure 2 — Figure 2: Distributional evolution of SFT, DFT, and SPFT during training, measured by token-level entropy, top-1 confidence, and KL divergence from the initial model. DFT rapidly sharpens the output distribution and moves farthest from the base model, while SPFT produces a more controlled increase in confidence and reduction in entropy with substantially smaller distributional drift.

Model	Method	MATH-500		Minerva Math		OlympiadBench		AIME24		AMC23		Average
	Method	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8
Qwen2.5 Math (7B)	Base	60.38	87.40	16.18	42.28	27.76	54.07	14.58	36.67	49.69	85.00	33.72	61.08
	SFT	67.85	89.40	24.45	47.43	30.96	55.85	5.42	20.00	40.62	75.00	33.86	57.54
	DFT	68.32	81.20	25.46	35.66	33.04	48.00	6.25	16.67	46.56	67.50	35.93	49.81
	ASFT	70.32	88.00	22.75	38.24	34.50	57.04	7.08	16.67	47.81	80.00	36.49	55.99
	TALR	65.88	85.20	22.56	44.85	29.20	53.48	5.83	16.67	35.62	75.00	31.82	55.04

Table 1: Mathematical reasoning results across five benchmarks. Each benchmark reports Avg@8 / Pass@8, and the final pair gives the unweighted five-benchmark average. External baselines include ASFT ( Zhu et al., 2026 ) , TALR ( Lin et al., 2026 ) , and PriFT ( Wang et al., 2026 ) whenever a matched result is available.

Panel of Figure 3 — Figure 3: Response-level sampling scaling on AMC 2023. Increasing $k$ reveals whether a method can obtain a correct solution from multiple samples (Pass@k) and whether correct solutions become sufficiently dominant for majority voting (Maj@k). SPFT combines high discovery probability with strong consensus across the sampling budget.

Method	HumanEval	HumanEval+	MBPP	MBPP+	Code Avg	$\Delta$ vs SFT
Base	43.9	36.6	69.2	58.6	52.08	–
SFT	41.5	37.2	63.2	54.2	49.03	0.00
ASFT	49.4	43.3	67.2	55.8	53.92	+4.89
PriFT	56.1	50.6	67.7	57.4	57.95	+8.92
TALR	59.8	52.4	67.5	56.3	59.00	+9.97
SPFT	56.7	53.0	70.6	58.5	59.70	+10.67

Table 2: Qwen2.5-Coder-1.5B EvalPlus results. All values are greedy Pass@1 percentages;

\Delta

reports the change in four-task Code Avg relative to SFT (49.03%).

Reference	MATH-500		Minerva Math		OlympiadBench		AIME24		AMC23		Average
Reference	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8
Online ( $\theta_{t}$ )	60.30	84.00	21.65	51.10	23.61	47.70	5.42	16.67	38.12	75.00	29.82	54.89
Frozen ( $\theta_{0}$ )	64.75	87.40	20.82	40.44	28.94	51.26	7.08	16.67	43.12	82.50	32.94	55.65
Frozen $-$ Online	+4.45	+3.40	-0.83	-10.66	+5.33	+3.56	+1.66	0.00	+5.00	+7.50	+3.12	+0.76

Table 3: Frozen versus online privileged reference for Qwen2.5-Math-1.5B. Each benchmark reports Avg@8 / Pass@8, and the final pair gives the unweighted five-benchmark average.

Privileged information	HumanEval	HumanEval+	MBPP	MBPP+	Code Avg
SPFT (Clean Rule)	56.7	53.0	70.6	58.5	59.70
SPFT (Full Code)	58.5	54.3	63.9	54.1	57.70
SPFT (Unit Tests)	53.0	48.2	68.2	56.6	56.50
SPFT (Suffix-Half)	57.3	51.8	66.2	56.9	58.05

Table 4: Privileged-information form ablation for Qwen2.5-Coder-1.5B. All rows use the same SPFT training objective; only the privileged prompt content changes.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Objective	MATH-500		Minerva Math		OlympiadBench		AIME24		AMC23		Average
Objective	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8
SPFT	64.75	87.40	20.82	40.44	28.94	51.26	7.08	16.67	43.12	82.50	32.94	55.65
Forward KL	41.23	81.80	9.97	37.50	19.02	46.37	3.75	16.67	20.00	65.00	18.79	49.47
Reverse KL	31.55	76.40	8.36	32.35	15.96	44.59	1.67	6.67	16.88	60.00	14.88	44.00

Appendix

Table 5: Direct token weighting versus full-vocabulary KL for Qwen2.5-Math-1.5B. Each benchmark reports Avg@8 / Pass@8, and the final pair gives the unweighted five-benchmark average.

Configuration Item	Mathematical Reasoning	Code Generation
Training dataset	NuminaMath-CoT 10K (9,637 verified pairs)	OpenCodeInstruct 10K (10,000 verified pairs)
Backbone models	Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen3-8B-Base	Qwen2.5-Coder-1.5B
Privileged prompt style ( $c$ )	opsd_answer (boxed final answer)	code_rule_paradigm (Clean Rule AST summary)
Global batch size	32	16
Peak learning rate	$1.0\times 10^{-5}$	$5.0\times 10^{-5}$
LR schedule & warmup	Cosine decay, 5% linear warmup	Cosine decay, 5% linear warmup

Appendix

Table 6: Summary of training and evaluation hyperparameters for mathematical reasoning and code generation.

Model	Method	MATH-500		Minerva Math		OlympiadBench		AIME24		AMC23		Average
Model	Method	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8	Avg@8	Pass@8
Qwen2.5-Math 1.5B	Base	44.15	82.20	8.04	26.84	23.81	50.37	6.25	13.33	25.00	67.50	21.45	48.05
	SFT	55.88	82.40	15.40	37.87	22.37	49.48	5.00	23.33	31.25	60.00	25.98	50.62
	DFT	62.92	80.00	22.24	38.60	24.41	43.56	4.58	10.00	34.06	60.00	29.64	46.43
	ASFT	59.20	82.40	18.98	40.81	23.28	46.52	4.17	13.33	32.81	70.00	27.69	50.61
	TALR	56.60	82.60	17.97	41.54	22.22	48.00	1.67	6.67	31.56	62.50	26.00	48.26

Appendix

Table 7: Additional mathematical reasoning results in the same format as Table 1 .

Model	Method	MATH-500	Minerva	Olympiad	AIME24	AMC23	Average
Qwen2.5-Math-1.5B	Base	56.20	6.62	31.11	13.33	27.50	26.95
	SFT	68.00	20.59	30.07	6.67	42.50	33.57
	DFT	67.20	25.37	27.85	6.67	35.00	32.42
	ASFT	67.60	24.26	30.07	3.33	42.50	33.55
	TALR	68.20	25.74	31.26	3.33	45.00	34.71
	PriFT	69.20	14.34	33.63	6.67	40.00	32.77

Appendix

Table 8: Majority-vote accuracy (Maj@8) across all five mathematical reasoning benchmarks and their unweighted average for the main comparisons and the additional Qwen3-1.7B-Base model-scale comparison.

Supervised Fine-Tuning

Supervised fine-tuning adapts pretrained models by learning from labeled examples, instructions, demonstrations, and task-specific trajectories.

See all 231 papers

PriFT: Guiding Supervised Fine-Tuning with Pretrained Priors
Jun 8, 2026Ke Wang, Shuangqi Li, Mathieu Salzmann +1Supervised Fine-Tuning Fine-Tuning
EPFL, Lausanne, Switzerland
A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design
Jun 9, 2026Tong Xie, Yuanhao Ban, Yunqi Hong +3Supervised Fine-Tuning Fine-Tuning
University of California, Los Angeles (UCLA) · Arena
Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning
Sep 9, 2026Yaning Jia, Chunhui Zhang, Wenxuan Xu +3Numerical Reasoning in Language Models Supervised Fine-Tuning
Dartmouth College · Carnegie Mellon University

Jun 8, 2026J/K move · Enter open · S save

PriFT: Guiding Supervised Fine-Tuning with Pretrained Priors

Ke Wang, Shuangqi Li, Mathieu Salzmann, Pascal Frossard

EPFL, Lausanne, Switzerland

Supervised Fine-Tuning Fine-Tuning LLM Fine-Tuning

Open paper AskPDF arXiv

Supervised fine-tuning (SFT) is an efficient approach for downstream task adaptation and often serves as the initialization stage for reinforcement learning (RL), but it can show weaker generalization than RL. A key limitation is its off-policy objective: SFT fits fixed demonstrations token by token, including targets poorly aligned with the model's pretrained distribution, which can lead to overfitting. A recent line of work addresses this issue by assigning larger training weights to tokens better aligned with the current model's predictive distribution, with the intuition that fitting these tokens are less distortive to the model's knowledge and representations. However, computing the token weights from the model that is currently fine-tuned entangles token weights with the optimization trajectory, inducing a self-reinforcing loop which concentrates the supervision signal to a narrow set of initially preferred tokens. To address this, we propose PriFT (Prior-support guided Fine-Tuning), which derives token weights from a frozen pretrained reference to obtain a stable reweighting signal unaffected by fine-tuning. This signal estimates prior support: the extent to which each target token is supported by the pretrained distribution. By reweighting tokens according to their prior support, PriFT limits excessive adaptation to targets requiring substantial deviation from the pretrained distribution and improves SFT generalization. We introduce two instantiations: PriFT-prob uses pretrained target-token probability, while PriFT-mass selects tokens by cumulative probability mass under the pretrained distribution. Across mathematical reasoning, code generation, and medical QA, PriFT achieves state-of-the-art results among SFT baselines, while also providing a stronger initialization for subsequent RL.

Abstract

Supervised Fine-Tuning

Explore similar work

PriFT: Guiding Supervised Fine-Tuning with Pretrained Priors

A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design

Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning