What Should SFT Learn? Conditional Information Gain for Generalizable Post-Training

Yuxuan Gu; Kequan Niu; Xiaocheng Feng; Yun Li; Yiqiao Yang; Bing Qin

cs.CLOct 9, 2026

What Should SFT Learn? Conditional Information Gain for Generalizable Post-Training

Authors: Yuxuan Gu, Kequan Niu, Xiaocheng Feng, Yun Li, Yiqiao Yang, Bing Qin

Organizations: Harbin Institute of Technology

Abstract

Supervised fine-tuning (SFT) remains a widely used approach for post-training large language models, because fixed demonstrations can directly introduce desired reasoning strategies and behavioral patterns. Yet forcing a model to imitate externally provided trajectories may also interfere with behaviors already supported by pretraining, limiting generalization beyond the training distribution. Recent dynamic SFT methods mitigate this issue by reweighting token-level supervision according to model-side signals such as predictive probability, prior support, or distributional compatibility. However, model compatibility is not equivalent to learning value: a token that is weakly supported by the current model may either be an undesirable mismatch or precisely the task-specific decision that the demonstration is intended to teach. We propose CIG-SFT, an information-theoretic framework that estimates the task-conditioned information carried by each demonstrated token. Under teacher forcing, we compare the token likelihood from a frozen reference model with and without access to the task input while keeping the gold solution prefix fixed. The resulting conditional information gain quantifies how much the task itself reduces uncertainty about the demonstrated decision. We then convert this information into a KL-regularized projection of the current policy and derive an equivalent scalar-weighted SFT objective whose logit gradient exactly matches that of the projected target. Across mathematical reasoning and code-generation benchmarks, CIG-SFT consistently outperforms standard SFT and recent adaptive fine-tuning baselines, with particularly strong improvements in generalization to challenging evaluation distributions, while requiring neither additional training data nor online rollouts.

Figures & tables

Figure 1: Overview of CIG-SFT. A frozen initial policy estimates token-level conditional information gain from fixed demonstrations. The information score and current policy jointly define an information-proximal target distribution, which is converted into a gradient-equivalent token weight for supervised optimization. During training, we update only the current policy.

Method	Average	MATH500	Minerva	Olympiad	AIME24	AIME25	AMC23
Base	$17.2/55.6$	$39.4/88.2$	$11.7/46.7$	$14.8/57.2$	$\phantom{0}8.5/46.7$	$\phantom{0}2.9/20.0$	$25.8/75.0$
SFT	$19.7/49.4$	$53.0/87.2$	$16.9/48.9$	$19.2/55.1$	$\phantom{0}2.3/13.3$	$\phantom{0}1.7/16.7$	$25.3/75.0$
DFT	$31.7/51.1$	$68.9/86.0$	$23.9/43.4$	$33.5/56.3$	$11.7/23.3$	$\phantom{0}5.8/20.0$	$46.6/77.5$
IDFT	$30.1/49.9$	$69.3/85.0$	$25.7/42.6$	$34.1/53.5$	$\phantom{0}6.7/23.3$	$\phantom{0}3.1/20.0$	$41.7/87.5$
ASFT	$30.3/60.1$	$67.8/\underline{92.2}$	$27.2/\underline{53.7}$	$33.7/\underline{64.6}$	$\phantom{0}8.1/33.3$	$\phantom{0}3.1/26.7$	$41.7/90.0$
PriFT-prob	$35.8/62.5$	$72.3/\underline{92.2}$	$\underline{29.6}/\textbf{54.8}$	$36.8/\textbf{65.3}$	$14.4/43.3$	$\phantom{0}\underline{7.5}/26.7$	$53.9/\underline{92.5}$

Table 1: Mathematical reasoning results with Qwen2.5-Math-7B. We report Avg@16 / Pass@16.

Off-policy Reinforcement Learning
Method	Average	MATH500	Minerva	Olympiad	AIME24	AIME25	AMC23
Base	$16.5/51.6$	$39.0/86.0$	$10.2/41.2$	$19.2/52.1$	$\phantom{0}4.8/26.7$	$1.9/23.3$	$23.6/80.0$
DPO	$17.3/49.7$	$40.0/86.0$	$11.1/44.1$	$19.7/55.0$	$\phantom{0}4.2/33.3$	$0.8/10.0$	$28.0/80.0$
SimPO	$17.0/50.9$	$41.2/86.0$	$10.9/43.8$	$19.6/52.4$	$\phantom{0}4.2/26.7$	$1.7/16.7$	$24.2/80.0$
On-policy Reinforcement Learning
GRPO	$28.0/56.2$	$63.2/90.4$	$22.7/\underline{51.1}$	$29.1/\textbf{62.1}$	$\phantom{0}7.5/30.0$	$4.8/23.3$	$40.9/80.0$

Table 2: Mathematical reasoning results with Qwen2.5-Math-1.5B. We report Avg@16 / Pass@16.

Method	Average	HumanEval+	MBPP+	LCB v5	LCB v6	BigCode	DS-1000
Base	$30.9/49.4$	$40.9/\mathbf{67.1}$	$49.5/74.1$	$18.0/28.3$	$17.1/26.9$	$35.9/57.2$	$23.7/42.9$
SFT	$32.8/49.0$	$44.8/62.2$	$53.2/\underline{74.3}$	$18.6/28.2$	$17.7/26.9$	$38.2/\underline{59.0}$	$24.2/43.2$
DFT	$34.2/49.3$	$43.8/62.2$	$\underline{54.6}/73.3$	$21.1/\underline{29.7}$	$20.0/27.8$	$\underline{39.9}/58.5$	$\underline{26.2}/\underline{44.7}$
ASFT	$\underline{34.3}/\underline{49.7}$	$45.6/\underline{65.2}$	$54.2/73.3$	$\underline{21.1}/29.3$	$\underline{20.1}/\underline{27.9}$	$39.5/59.0$	$25.2/43.7$
PriFT-prob	$33.6/49.3$	$\underline{45.6}/64.0$	$52.6/73.5$	$19.9/29.1$	$19.1/27.8$	$39.0/56.9$	$25.6/44.6$
CIG-SFT	$\textbf{35.4}/\textbf{50.5}$	$\textbf{46.5}/64.0$	$\textbf{55.0}/\textbf{74.1}$	$\textbf{22.0}/\textbf{30.5}$	$\textbf{21.1}/\textbf{29.0}$	$\textbf{41.0}/\textbf{59.3}$	$\textbf{26.9}/\textbf{46.4}$

Table 3: Code-generation results with Qwen2.5-Coder-3B. We report Avg@16 / Pass@16.

Method	Average	MATH500	Minerva	Olympiad	AIME24	AIME25	AMC23
SFT	$13.8/42.3$	$41.5/81.8$	$12.1/40.4$	$12.0/45.6$	$\phantom{0}0.6/10.0$	$0.2/\phantom{0}3.3$	$16.4/72.5$
CIG-SFT	$31.5/58.6$	${68.4}/89.6$	${24.1}/48.5$	${32.1}/59.4$	$\phantom{0}9.4/{40.0}$	$5.2/{26.7}$	${50.0}/{87.5}$
w/o $c_{t}$	$21.9/48.4$	$55.4/88.6$	$19.2/46.0$	$22.1/54.7$	$\phantom{0}4.0/16.7$	$1.4/\phantom{0}6.7$	$29.4/77.5$
w/o $D_{\text{KL}}$	$28.4/55.1$	$62.1/90.4$	$19.3/46.0$	$29.2/59.1$	$\phantom{0}9.6/30.0$	$4.4/20.0$	$45.5/85.0$
Online $c_{t}$	$\phantom{0}6.9/21.9$	$21.1/54.6$	$\phantom{0}6.6/22.4$	$\phantom{0}3.3/12.6$	$\phantom{0}0.2/\phantom{0}3.3$	$0.2/\phantom{0}3.3$	$\phantom{0}9.8/24.4$
Frozen $D_{\text{KL}}$	$\phantom{}30.5/55.4$	$64.7/89.4$	$\phantom{}22.7/47.8$	$\phantom{}31.2/60.1$	$11.9/\phantom{}30.0$	$5.6/\phantom{}20.0$	$\phantom{}46.7/85.0$

Table 4: Ablations on mathematical reasoning with Qwen2.5-Math-1.5B.

Figure 2: Analysis of training scale.

Figure 3: Analysis of learning rate and proximity strength

\lambda

Figure 4: Analysis of the training dynamics.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Token-level allocations for two representative cases. The columns show the demonstrated tokens, and the rows report SFT, DFT, task-conditioned information gain

c_{t}

, and update weights

w_{t}

Method	s $/$ Step	Peak GiB	Wall clock (h)
SFT	$23.92$	$28.93$	$1.58$
DFT	$23.98$	$28.93$	$1.58$
CIG-SFT	$23.92$	$28.93$	$0.48+1.58$
IDFT	$25.61$	$42.84$	$1.68$
ASFT	$31.51$	$40.49$	$2.07$
PriFT-prob	$30.73$	$31.81$	$2.03$

Appendix

Table 5: Measured training cost.

Method	42	43	44	45	46	Mean $\pm$ Std.	95% CI
SFT	$13.8$	$13.9$	$13.9$	$14.1$	$14.3$	$14.01\pm 0.20$	$[13.76,14.26]$
DFT	$26.3$	$26.7$	$26.2$	$26.0$	$26.7$	$26.38\pm 0.34$	$[25.95,26.80]$
IDFT	$25.2$	$26.4$	$26.0$	$25.6$	$27.0$	$26.05\pm 0.69$	$[25.19,26.90]$
PriFT-prob	$\underline{30.3}$	$\underline{29.2}$	$\underline{29.3}$	$\underline{29.3}$	$\underline{29.3}$	$\underline{29.46}\pm 0.46$	$\underline{[28.89,30.03]}$
CIG-SFT	31.5	30.5	30.6	31.0	31.4	$\textbf{31.00}\pm 0.46$	$[\textbf{30.42},\textbf{31.57}]$

Appendix

Table 6: Stability analysis on Qwen2.5-Math-1.5B.

Method	1K	3K	5K	10K	30K	50K	100K
Base	$16.5/51.6$	$16.5/51.6$	$16.5/51.6$	$16.5/\underline{51.6}$	$16.5/51.6$	$16.5/51.6$	$16.5/51.6$
SFT	$9.3/38.4$	$11.0/38.6$	$12.8/41.2$	$13.7/43.4$	$13.8/42.3$	$14.4/42.5$	$15.1/42.7$
DFT	$16.9/45.5$	$22.8/46.2$	$22.9/46.1$	$25.2/\underline{49.8}$	$26.3/49.0$	$26.7/46.9$	$26.1/47.5$
ASFT	$24.0/53.7$	$\underline{24.6}/\underline{53.1}$	$\underline{25.0}/\underline{53.4}$	$\underline{26.0}/51.0$	$\underline{27.6}/\underline{54.8}$	$\underline{27.3}/\underline{53.8}$	$\underline{26.8}/\underline{53.6}$
IDFT	$\underline{26.3}/\textbf{57.0}$	$23.6/47.6$	$24.8/45.9$	$24.4/44.4$	$25.2/48.9$	$25.7/46.1$	$26.3/45.3$
CIG-SFT	$\textbf{28.9}/\underline{55.9}$	$\textbf{29.9}/\textbf{56.4}$	$\textbf{30.2}/\textbf{58.5}$	$\textbf{30.5}/\textbf{55.2}$	$\textbf{31.5}/\textbf{56.7}$	$\textbf{30.8}/\textbf{56.9}$	$\textbf{30.8}/\textbf{57.3}$

Appendix

Table 7: Results of Avg@16 / Pass@16 across training-set sizes.

Learning rate	SFT	DFT	CIG-SFT
$1\times 10^{-5}$	$10.4/41.8$	$23.6/\textbf{49.3}$	$27.8/56.1$
$2\times 10^{-5}$	$12.8/\underline{42.2}$	$24.9/45.8$	$30.3/\textbf{58.6}$
$5\times 10^{-5}$	$\underline{13.8}/\textbf{42.3}$	$\underline{26.3}/\underline{49.0}$	$\textbf{31.5}/\underline{56.7}$
$1\times 10^{-4}$	$\textbf{14.5}/\textbf{42.3}$	$\textbf{26.5}/45.6$	$\underline{30.8}/54.3$
$2\times 10^{-4}$	$13.4/39.1$	$21.8/42.4$	$25.8/51.1$

Appendix

Table 8: Detailed sensitivity results for the learning rate.

$\lambda$	$10^{-8}$	0.1	0.3	0.5	0.7	1.0	2	3	5	6	8	10
Avg@16	$22.4$	$29.7$	$31.5$	$31.1$	$31.5$	$\underline{31.8}$	$31.1$	$31.3$	32.1	$31.1$	$31.4$	$31.0$
Pass@16	$54.1$	$55.4$	$56.7$	$\underline{57.3}$	57.4	$55.4$	$52.9$	$54.1$	$55.7$	$54.7$	$53.9$	$54.6$

Appendix

Table 9: Detailed sensitivity results for the projection coefficient

\lambda

Method	Average	MMLU-General	ARC-Challenge	HellaSwag	PIQA
Base	56.1	$\underline{58.2}$	48.5	45.2	$72.6$
SFT	$\underline{55.9}$	$57.9$	$\underline{48.0}$	$44.6$	73.0
DFT	$54.9$	$57.9$	$47.1$	$42.4$	$72.0$
IDFT	$54.4$	$57.7$	$46.2$	$41.4$	$72.2$
ASFT	$55.7$	58.3	$47.5$	$44.5$	$72.6$
CIG-SFT	$\underline{55.9}$	$58.0$	$\underline{48.0}$	$\underline{44.6}$	$\underline{72.9}$

Appendix

Table 10: General-task performance. Values are raw Acc (%).

Method	Avg@16
Base	$32.2$
SFT	$42.4$
DFT	$\underline{50.8}$
IDFT	$49.7$
ASFT	$50.5$
PriFT-prob	$49.2$

Appendix

Table 11: In domain generalization performance for Qwen2.5-Math-7B.

Explore similar work

InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting
May 14, 2026Mahdi Sabbaghi, George Pappas, Adel Javanmard +1Supervised Fine-Tuning Adaptive Loss Weighting
University of Pennsylvania · University of Southern California
PriFT: Prior-Support Guided Supervised Fine-Tuning
Jun 8, 2026Ke Wang, Shuangqi Li, Mathieu Salzmann +1Supervised Fine-Tuning Fine-Tuning
EPFL, Lausanne, Switzerland
SoFT: Soft Targets for Generalizable LLM Fine-Tuning
Sep 26, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +7LLM Fine-Tuning Fine-Tuning
HKUST, HKUST-GZ

May 14, 2026J/K move · Enter open · S save

InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting

Mahdi Sabbaghi, George Pappas, Adel Javanmard, Hamed Hassani

University of Pennsylvania · University of Southern California

Supervised Fine-Tuning Adaptive Loss Weighting Catastrophic Forgetting

Open paper AskPDF arXiv

Supervised fine-tuning (SFT) provides the standard approach for teaching LLMs new behaviors from offline expert demonstrations. However, standard SFT uniformly fits all samples -- including those with low likelihood under the base model -- which can disproportionately drive training updates toward overfitting specific samples rather than learning the target behavior. Moreover, adapting to these unlikely samples induces substantial policy shifts that degrade prior capabilities. Existing methods mitigate this by filtering, regenerating, or down-weighting low-likelihood data. In doing so, they often suppress precisely the novel behaviors the base model has yet to learn. We propose InfoSFT, a principled weighting scheme for the SFT objective that concentrates learning signals on maximally informative, medium-confidence tokens -- those neither overly familiar to the base model nor too unlikely to cause instability. Requiring only a one-line modification to the standard token-wise loss, InfoSFT demonstrably improves generalization over vanilla SFT and likelihood-weighted baselines across math, code, and chain-of-thought tasks with diverse model families, while better preserving pre-existing capabilities.

Abstract

Explore similar work

InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting

PriFT: Prior-Support Guided Supervised Fine-Tuning

SoFT: Soft Targets for Generalizable LLM Fine-Tuning