Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Features

Cunchun Li; Haonan He; Yifan Gao; Minglei Li; Jingqi Ye; Qingyu Yang; Peng Ye

cs.CLSep 27, 2026

Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Features

Authors: Cunchun Li, Haonan He, Yifan Gao, Minglei Li, Jingqi Ye, Qingyu Yang, Peng Ye

Organizations: Shanghai AI Laboratory · University of Science and Technology of China · Fudan University · KTH Royal Institute of Technology · The Chinese University of Hong Kong

Abstract

Supervised fine-tuning (SFT) learns most aggressively from tokens that the model deems least likely. This helps acquire new behaviors, but also amplifies noisy or conflicting supervision and can overwrite useful pretrained knowledge. Through a unified policy-loss view, we revisit existing token-reweighting methods and show that they assign nonnegative coefficients to demonstrated tokens. Consequently, they can suppress or amplify supervised updates, but cannot reverse harmful features once learned. Moreover, larger training weights do not amount to feature extrapolation, since they change the optimization trajectory rather than scale a fixed SFT direction. We argue that reversal and extrapolation require a stable reference frame defined by a fixed SFT delta. Motivated by this, we propose SCALE (Selective Control of Adaptation via Local Entropy), an entropy-guided adaptation-strength-control method that freezes the pretrained model and the SFT delta and learns bounded token- and module-specific gates by minimizing predictive entropy alone. These gates suppress, reverse, or extrapolate frozen SFT features according to their alignment with entropy reduction. Across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-4B-Base, SCALE achieves mathematical-reasoning averages of 37.84, 43.60, and 36.57, exceeding the strongest corresponding baselines while remaining competitive on general-retention benchmarks. It also attains the best average code-generation performance across HumanEval, HumanEval+, and MBPP for all three models. These results suggest that effective SFT correction can benefit from controlling how already learned residuals are used, rather than only modifying how they are learned.

Figures & tables

	Mathematical reasoning						General Retention
Method	Math500	Minerva	Olympiad	AIME24	AMC23	Avg	MMLU-P	BBH	OBQA	Avg
Qwen2.5-Math-1.5B ( Yang et al., 2024a )
Base	21.87	5.24	9.18	1.66	12.81	10.15	20.24	26.32	30.40	25.65
LoRA ( Hu et al., 2022 )	39.60	9.49	10.80	0.41	17.03	15.47	27.89	31.76	33.80	31.15
DoRA ( Liu et al., 2024 )	40.10	9.39	10.69	1.24	16.88	15.66	27.58	32.20	33.60	31.13
ProFit ( Liu et al., 2026b )	56.12	21.93	22.69	2.29	31.56	26.92	28.02	29.78	33.80	30.53

Table 1: Mathematical reasoning and general-capability retention results (%). Mathematical-reasoning scores are Avg@16. Boldface and underlining mark the best and second-best adapted scores in each column.

Figure 3: Gate-range sensitivity across backbones. We evaluate all endpoint pairs to characterize the intervention space. The selected maxima are reported as oracle sweep results, while fixed-range results are used for controlled comparisons.

	Qwen2.5-Math-1.5B				Qwen2.5-Math-7B				Qwen3-4B-Base
Method	HE	HE+	MBPP	Avg	HE	HE+	MBPP	Avg	HE	HE+	MBPP	Avg
Base	28.66	17.68	32.80	26.38	42.28	39.02	34.20	38.50	43.09	39.02	37.00	39.70
LoRA ( Hu et al., 2022 )	34.15	25.61	30.80	30.19	48.37	43.29	38.00	43.22	51.22	45.73	40.20	45.72
DoRA ( Liu et al., 2024 )	31.91	28.05	31.20	30.39	46.14	43.29	36.20	41.88	50.81	45.12	40.00	45.31
DFT ( Wu et al., 2026 )	34.96	29.88	33.60	32.81	48.58	42.07	36.00	42.22	44.51	38.41	37.20	40.04
ProFit ( Liu et al., 2026b )	35.57	30.49	33.80	33.29	47.56	43.29	35.60	42.15	48.78	43.90	40.00	44.23

Table 2: Code-generation results (%). Boldface and underlining mark the best and second-best adapted scores in each column.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition
$x$	Input prompt
$y=(y_{1},\dots,y_{T})$	Demonstrated response tokens
$c_{t}=(x,y_{<t})$	Prefix context at token position $t$
$a\in\mathcal{V}$	Next-token action from vocabulary $\mathcal{V}$
$\pi_{\theta}(a\mid c_{t})$	Model next-token distribution parameterized by $\theta$
$z_{t}$	Output logits at position $t$

Appendix

Table 3: Notation used throughout the paper.

Method	CE coefficient $w_{t}$	Signal / qualification
SFT	$1$	Uniform demonstration weighting.
DFT	$\operatorname{sg}\!\left[p_{t}\right]$	Probability-weighted CE ( Wu et al., 2026 ) .
ProFit	$\mathbf{1}[\operatorname{sg}\!\left[p_{t}\right]>\tau_{p}]$	Probability mask ( Liu et al., 2026b ) .
EAFT	$H_{t}^{\mathrm{top}\text{-}20}/\log 20$	Normalized top- $20$ entropy; frozen-coefficient CE interpretation ( Diao et al., 2026 ) .
TALR	$\max\{w_{\min},\operatorname{sg}\!\left[e^{-\ell_{t}/\tau_{\ell}}\right]\}$	Loss-dependent weight, dynamic temperature, and floor ( Lin et al., 2026 ) .
Offline token mask	$m_{t}\in\{0,1\}$	Fixed retained positions, e.g., token cleaning ( Pang et al., 2025 ) .

Appendix

Table 4: Nonnegative coefficients of demonstration-action CE components. Entries describe CE components. The EAFT row reports its published entropy weight without specifying a stop-gradient convention.

Method	Math500	Minerva	Olympiad	AIME24	AMC23	$\mathrm{Avg@16}$
Negative	30.85	6.54	8.85	1.24	15.47	12.59
LoRA	39.60	9.49	10.80	0.41	17.03	15.47
Positive	47.99	15.24	20.08	4.16	25.47	22.59
Ours	71.60	31.44	31.34	8.55	46.25	37.84

Appendix

Table 5: Zero-threshold entropy-sign ablation on Qwen2.5-Math -1.5B. Negative and Positive suppress deltas where

\Delta H<0

and

\Delta H>0

, respectively. Scores are Avg@16. Boldface and underlining mark the best and second-best scores in each column.

Method	Math500	Minerva	Olympiad	AIME24	AMC23	$\mathrm{Avg@16}$
LoRA	39.60	9.49	10.80	0.41	17.03	15.47
LoRA +20k	40.12	9.72	11.58	1.04	18.91	16.27
SCALE w/ CE	41.97	10.31	11.36	0.82	16.41	16.17
SCALE w/ Entropy	71.60	31.44	31.34	8.55	46.25	37.84

Appendix

Table 6: Calibration-loss ablation on Qwen2.5-Math -1.5B (%). Scores are Avg@16; boldface and underlining mark the best and second-best values.

Method	Math500	Minerva	Olympiad	AIME24	AMC23	$\mathrm{Avg@16}$
LoRA	$39.52\pm 0.07$	$9.60\pm 0.33$	$10.59\pm 0.39$	$0.62\pm 0.21$	$16.72\pm 1.43$	$15.41\pm 0.44$
DFT	$\underline{65.30\pm 1.03}$	$\underline{21.42\pm 1.89}$	$\underline{27.22\pm 0.37}$	$\underline{7.64\pm 1.78}$	$\underline{39.95\pm 0.65}$	$\underline{32.31\pm 0.92}$
SCALE	$\mathbf{71.73\pm 0.57}$	$\mathbf{27.18\pm 2.47}$	$\mathbf{31.56\pm 0.31}$	$\mathbf{10.21\pm 1.08}$	$\mathbf{48.44\pm 1.18}$	$\mathbf{37.83\pm 0.20}$

Appendix

Table 7: Three-seed mathematical-reasoning results on Qwen2.5-Math -1.5B (%). Scores are Avg@16, reported as mean

\pm

sample standard deviation.

Backbone	$\lambda<0$	$0\leq\lambda\leq 1$	$\lambda>1$
Qwen2.5-Math -1.5B	39.36	17.81	42.82
Qwen2.5-Math -7B	44.34	19.11	36.55

Appendix

Table 8: Distribution of gate multipliers on generated mathematical-reasoning tokens (%).

Gate applied at inference	Qwen2.5-Math -1.5B	Qwen2.5-Math -7B
Original $\lambda$	37.05	42.52
No reversal, $\max(\lambda,0)$	20.10 ( $-16.95$ )	33.37 ( $-9.15$ )
No extrapolation, $\min(\lambda,1)$	30.06 ( $-6.99$ )	30.06 ( $-12.46$ )
Unit interval, $\operatorname{clip}(\lambda,0,1)$	19.44 ( $-17.61$ )	24.07 ( $-18.45$ )

Appendix

Table 9: Mathematical-reasoning gate interventions. Scores are Math Avg@16; parentheses show the change from the original gate in percentage points.

Gate range	Gate applied at inference	pass@1	Change
$[0,6]$	Original $\lambda$	57.11	—
	Unit interval, $\operatorname{clip}(\lambda,0,1)$	47.15	$-9.96$
$[-3,3]$	Original $\lambda$	49.80	—
	No reversal, $\max(\lambda,0)$	52.64	$+2.85$
	No extrapolation, $\min(\lambda,1)$	45.12	$-4.67$
	Unit interval, $\operatorname{clip}(\lambda,0,1)$	47.97	$-1.83$

Appendix

Table 10: Gate interventions for code generation on Qwen2.5-Math -7B. Scores are HumanEval pass@1 (%).

Figure 4: Single-GPU training time (minutes) for mathematical reasoning and code generation on Qwen2.5-Math-1.5B and 7B.

Explore similar work

Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning
Sep 9, 2026Yaning Jia, Chunhui Zhang, Wenxuan Xu +3Supervised Finetuning Mathematical Reasoning Benchmarks
Dartmouth College · Carnegie Mellon University
PriFT: Prior-Support Guided Supervised Fine-Tuning
Jun 8, 2026Ke Wang, Shuangqi Li, Mathieu Salzmann +1Model Fine-Tuning
EPFL, Lausanne, Switzerland
InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting
May 14, 2026Mahdi Sabbaghi, George Pappas, Adel Javanmard +1Supervised Finetuning
University of Pennsylvania · University of Southern California

Sep 9, 2026, cs.AIJ/K move · Enter open · S save

Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning

Yaning Jia, Chunhui Zhang, Wenxuan Xu, Xingjian Diao+2

Dartmouth College · Carnegie Mellon University

Supervised Finetuning Mathematical Reasoning Benchmarks Mathematical Reasoning Visual Token Pruning Cross-Entropy Losses

Open paper AskPDF arXiv

Supervised fine-tuning (SFT) applies a uniform cross-entropy loss to all target tokens, even though different tokens provide unequal learning signals for mathematical reasoning. This uniform treatment can over-sharpen already mastered tokens while amplifying learning pressure on uncertain, low-confidence tokens, leading to suboptimal training dynamics. We propose Trimmed Logit-Gap SFT (TrimSFT), a simple token-level reweighting method that scales the SFT loss according to the logit gap between the gold token and its strongest competitor. TrimSFT trims supervision away from both extremes: tokens already mastered (large logit gap) and tokens weakly supported by the current model (small or negative logit gap), concentrating learning within an intermediate logit-gap region between them. We instantiate this principle with a Gaussian weight centered at margin m with bandwidth τ, requiring no reference model or additional forward pass. We evaluate TrimSFT on six base models from the Llama, Qwen, and DeepMath families across five mathematical reasoning benchmarks. TrimSFT consistently improves over standard SFT, achieving the best average performance on five out of six models, with gains of up to +26.9 points over SFT on MATH500. Further analyses show that the bandwidth τ matters more than the exact margin location, and that half-trim variants that remove supervision pressure from only one side yield inferior trade-offs. A token-level logit-gap distribution analysis suggests that TrimSFT reshapes model confidence in a more balanced way than uniform SFT or monotonic reweighting methods. These results suggest that reasoning SFT can benefit from trimming both extremes rather than treating all tokens uniformly.

Abstract

Explore similar work

Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning

PriFT: Prior-Support Guided Supervised Fine-Tuning

InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting