TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

Sadhika Malladi; Samy Jelassi; Dylan Foster; Jordan T. Ash; Akshay Krishnamurthy

cs.LGAug 26, 2026

TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

Authors: Sadhika Malladi, Samy Jelassi, Dylan Foster, Jordan T. Ash, Akshay Krishnamurthy

Organizations: University of California San Diego · Microsoft Research NYC

Abstract

Reinforcement learning post-training drives reasoning and agentic capabilities in modern AI systems, yet a growing body of work shows that it is most effective when used to fine-tune an already capable base model. We question whether existing pipelines yield models that are most suitable for reinforcement learning. Building on prior work highlighting the role of coverage and pass@K as predictors of post-RL performance, we design a simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution. We justify and validate the design choices in TailSFT, particularly the specific filtering criteria, through a combination of controlled experiments and theoretical analysis. On OLMo-3 7B, TailSFT often improves pass@16 performance on math and coding evaluations, with gains up to 17% absolute, while incurring minimal computational overhead. These higher-coverage checkpoints consistently translate to up to 4% absolute pass@1 gains in subsequent GRPO runs, demonstrating that TailSFT checkpoints serve as better initializations for RL. We further introduce a lightweight diagnostic for identifying settings where TailSFT is most likely to help. More broadly, our results motivate a principled, stage-aware approach to model development, in which intermediate checkpoints are judged by how effectively they support subsequent training.

Figures & tables

Figure 1: TailSFT yields better coverage, possibly at the cost of worse cross-entropy, which translates to stronger post-RL performance. Standard SFT achieves lower cross-entropy, while TailSFT preserves higher pass@ $K$ at large $K$ . Starting from this higher-coverage checkpoint, the same RL procedure yields higher pass@ $1$ . Curves are illustrative.

Panel of Figure 2 — Figure 2: TailSFT trades single-sample accuracy for better coverage under repeated sampling. The SFT distribution contains majority and minority shards with different rewarding paths. Standard SFT over-anchors on the majority shard, achieving high pass@ $1$ but no improvement as $K$ grows. A filtered objective, illustrated here by TailSFT , retains probability on both paths, yielding lower pass@ $1$ but substantially better pass@ $K$ scaling.

Figure 3: Filtering improves pass@

K

at the expense of cross entropy and pass@

1

. Results on the graph-navigation task. Standard SFT achieves the lowest cross-entropy and highest pass@

1

, while all three filtering methods achieve substantially higher pass@

8

. See Appendix C for details.

		pass@1			pass@16
SFT data	Benchmark	Standard	TailSFT	$\Delta$	Standard	TailSFT	$\Delta$
OMI	AIME 2022–2025 ( $n{=}120$ )	$3.65_{\,\pm 0.37}$	$4.03_{\,\pm 0.02}$	$+0.37$	$15.24_{\,\pm 2.02}$	$18.31_{\,\pm 0.57}$	$+3.07$
	MATH-500 Level 5 ( $n{=}134$ )	$24.80_{\,\pm 0.40}$	$25.16_{\,\pm 0.40}$	$+0.36$	$66.42_{\,\pm 0.00}$	$69.15_{\,\pm 0.86}$	$+2.74$
	OMEGA-500 ( $n{=}500$ )	$6.58_{\,\pm 0.50}$	$6.51_{\,\pm 0.26}$	$-0.06$	$32.80_{\,\pm 2.25}$	$32.60_{\,\pm 1.56}$	$-0.20$
BigCode	MBPP+ ( $n{=}378$ )	$53.04_{\,\pm 0.93}$	$51.36_{\,\pm 0.14}$	$-1.68$	$74.69_{\,\pm 1.10}$	$78.84_{\,\pm 1.06}$	$+4.14$
	HumanEval+ ( $n{=}164$ )	$45.06_{\,\pm 0.57}$	$46.33_{\,\pm 1.07}$	$+1.27$	$76.63_{\,\pm 0.70}$	$80.08_{\,\pm 0.93}$	$+3.46$

Table 1: TailSFT improves coverage over standard SFT. Across 18 model–benchmark pairs, TailSFT often improves pass@

16

, while changes in pass@

1

are mixed. Values are percentages reported as mean

\pm

standard deviation over three seeds;

\Delta

denotes TailSFT minus Standard SFT in percentage points, shown in green when positive, red when negative, and yellow when within one standard deviation. We later establish a sufficient condition ( Section 4.2 ) that accurately predicts the failure modes of TailSFT exhibited in this table ( Figure 4 ). MATH-500 Level 5 refers to the subset of the benchmark with the highest difficulty.

Figure 4: When standard SFT loses more coverage than it gains, TailSFT preserves or improves coverage. The vertical line marks $\rho_{16}=1$ , where the estimated coverage lost and gained by standard SFT are equal. Of the 11 evaluated settings with $\rho_{16}>1$ , 10 have positive coverage gains from TailSFT and one is essentially unchanged. Several settings with $\rho_{16}<1$ also improve. Coverage gain and ratio are computed on the base-reachable set ( Section 4.2 ).

		pass@1			pass@16
SFT data	Benchmark	Standard	TailSFT	$\Delta$	Standard	TailSFT	$\Delta$
GRPO on MATH
OMI	MATH-500 Level 5 ( $n{=}134$ )	$57.70_{\,\pm 0.44}$	$60.26_{\,\pm 1.02}$	$+2.56$	$83.83_{\,\pm 1.72}$	$87.06_{\,\pm 0.86}$	$+3.23$
	AIME 2022–2025 ( $n{=}120$ )	$14.40_{\,\pm 0.26}$	$15.61_{\,\pm 0.44}$	$+1.21$	$32.76_{\,\pm 0.88}$	$36.06_{\,\pm 0.98}$	$+3.30$
GRPO on MBPP+
BigCode	MBPP+ ( $n{=}378$ )	$69.57_{\,\pm 0.29}$	$73.50_{\,\pm 0.70}$	$+3.93$	$75.93_{\,\pm 0.00}$	$78.66_{\,\pm 0.15}$	$+2.73$

Table 2: Initializing from the TailSFT model improves post-GRPO performance across math and code. GRPO is initialized from the Standard SFT and TailSFT checkpoints evaluated in Table 1 , and rows are grouped by the corpus used for GRPO training: the MATH train split, excluding MATH-500, for the math rows, and the MBPP+ train split for the code rows. The SFT data column indicates the corpus used to train the initialization for each GRPO run. Values are percentages reported as mean

\pm

standard deviation over three seeds, and

\Delta

is improvement from TailSFT . Experiment details are in Section D.2 .

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Detailed results of hyperparameter sweep for TailSFT with absolute clipping in the synthetic graph navigation experiment.

Figure 6: Detailed results of hyperparameter sweep for TailSFT with quantile clipping in the synthetic graph navigation experiment.

Figure 7: Detailed results of hyperparameter sweep for TailSFT with offset quantile clipping in the synthetic graph navigation experiment.

SFT data	Benchmark	Base	Standard SFT	TailSFT	$\Delta_{\mathrm{Std}-\mathrm{Base}}$	$\Delta_{\mathrm{Tail}-\mathrm{Base}}$
OMI	AIME	$5.16$	$3.65_{\pm 0.37}$	$4.03_{\pm 0.02}$	$-1.51$	$-1.13$
	MATH-500 L5	—	$24.80_{\pm 0.40}$	$25.16_{\pm 0.40}$	—	—
	OMEGA-500	$3.54_{\pm 0.30}$	$6.58_{\pm 0.50}$	$6.51_{\pm 0.26}$	$+3.04$	$+2.97$
BigCode	MBPP+	$36.82_{\pm 0.56}$	$53.04_{\pm 0.93}$	$51.36_{\pm 0.14}$	$+16.22$	$+14.54$
	HumanEval+	$32.94_{\pm 1.11}$	$45.06_{\pm 0.57}$	$46.33_{\pm 1.07}$	$+12.12$	$+13.39$
	CruxEval-I	$25.67_{\pm 0.62}$	$29.79_{\pm 0.48}$	$30.43_{\pm 0.15}$	$+4.12$	$+4.76$

Appendix

Table 3: Per-benchmark pass@1 for the OLMo-3 7B base model (no SFT), Standard SFT (

f{=}0

), and TailSFT , at the settings used in the main results table.

\Delta_{\mathrm{Std}-\mathrm{Base}}

is Standard SFT minus base;

\Delta_{\mathrm{Tail}-\mathrm{Base}}

is TailSFT minus base. Emdashes indicate missing evaluations.

SFT data	Benchmark	Base	Standard SFT	TailSFT	$\Delta_{\mathrm{Std}-\mathrm{Base}}$	$\Delta_{\mathrm{Tail}-\mathrm{Base}}$
OMI	AIME	$20.91$	$15.24_{\pm 2.02}$	$18.31_{\pm 0.57}$	$-5.67$	$-2.60$
	MATH-500 L5	—	$66.42_{\pm 0.00}$	$69.15_{\pm 0.86}$	—	—
	OMEGA-500	$26.13_{\pm 1.42}$	$32.80_{\pm 2.25}$	$32.60_{\pm 1.56}$	$+6.67$	$+6.47$
BigCode	MBPP+	$75.93_{\pm 1.06}$	$74.69_{\pm 1.10}$	$78.84_{\pm 1.06}$	$-1.24$	$+2.91$
	HumanEval+	$79.27_{\pm 0.61}$	$76.63_{\pm 0.70}$	$80.08_{\pm 0.93}$	$-2.64$	$+0.81$
	CruxEval-I	$70.50_{\pm 1.54}$	$61.71_{\pm 2.27}$	$65.79_{\pm 0.26}$	$-8.79$	$-4.71$

Appendix

Table 4: Per-benchmark pass@16 (coverage) for the OLMo-3 7B base model (no SFT), Standard SFT (

f{=}0

), and TailSFT , at the settings used in the main results table. Columns as in Table 3 .

SFT data	Benchmark	#gained	#lost	#unchanged	median $\Delta$ (pp)	mean $\Delta$ (pp)	95% CI
OMI	AIME	31	14	75	+0.00	+3.07	[+0.77, +5.59]
OMI	MATH-500 L5	25	15	94	+0.00	+2.74	[-1.24, +6.47]
OMI	OMEGA-500	75	77	348	+0.00	-0.20	[-2.33, +1.80]
BigCode	MBPP+	39	19	320	+0.00	+4.14	[+2.12, +6.35]
BigCode	HumanEval+	21	11	132	+0.00	+3.46	[+0.00, +6.91]
BigCode	CruxEval-I	157	89	554	+0.00	+4.08	[+2.12, +6.12]

Appendix

Table 5: Per-problem paired changes in pass@

16

for TailSFT versus Standard SFT at the main-table settings. A problem counts as gained or lost only when its paired pass@

16

strictly changes; unchanged includes ceiling and floor ties. The final column is a paired bootstrap

95\%

interval for the mean per-problem change.

SFT data	Source	Train / val
Magicoder	ise-uiuc/Magicoder-OSS-Instruct-75K	33,817 / 342
BigCode	bigcode/self-oss-instruct-sc2-exec-filter-50k	49,260 / 498
OCI	nvidia/OpenCodeInstruct	72,635 / 365

Appendix

Table 6: Code SFT datasets. Row counts are after decontamination, format conversion, and initialization-loss annotation.

Hyperparameter	Values
Base model	OLMo-3 7B base
Learning rate	$\{1,2,3\}\times 10^{-5}$
Filter schedule	none; static; ramp $0\!\to\!\cdot$
Filter fraction $f$	$0$ (none); static $\{0.125,0.25,0.5\}$ ; ramp $0\!\to\!0.5$
Epochs	fixed per dataset ( $4$ / $5$ / $3$ for BigCode / Magicoder / OCI)
Sequence length	$2048$

Appendix

Table 7: Code SFT hyperparameter grid over learning rates and filtering configurations.

Filter	LR	pass@1	$\Delta$	pass@16	$\Delta$
base (no SFT)	—	$36.82\,\pm\,0.56$	—	$75.93\,\pm\,1.06$	—
no filter	$1\times 10^{-5}$	$47.45\,\pm\,0.82$	—	$79.45\,\pm\,0.81$	—
static $f{=}0.125$	$1\times 10^{-5}$	$47.00\,\pm\,0.60$	$-0.45$	$79.81\,\pm\,0.81$	$+0.36$
ramp $0{\to}0.5$	$1\times 10^{-5}$	$45.62\,\pm\,0.57$	$-1.83$	$79.72\,\pm\,0.85$	$+0.27$
no filter	$2\times 10^{-5}$	$52.58\,\pm\,0.11$	—	$76.72\,\pm\,1.06$	—
static $f{=}0.125$	$2\times 10^{-5}$	$52.07\,\pm\,0.70$	$-0.51$	$77.51\,\pm\,0.46$	$+0.79$

Appendix

Table 8: BigCode MBPP+ (

n{=}378

) SFT filtering grid,

4

epochs. Values are percent, mean

\pm

sample standard deviation over three seeds;

\Delta

is the change against the no-filter run at the same learning rate. The base (no SFT) row is the untuned OLMo-3 7B base under the same MBPP+ protocol (format-robust EvalPlus stops).

Filter	LR	pass@1	$\Delta$	pass@16	$\Delta$
base (no SFT)	—	$36.82\,\pm\,0.56$	—	$75.93\,\pm\,1.06$	—
no filter	$1\times 10^{-5}$	$53.14\,\pm\,0.52$	—	$80.69\,\pm\,0.26$	—
static $f{=}0.125$	$1\times 10^{-5}$	$52.19\,\pm\,0.25$	$-0.95$	$79.98\,\pm\,0.61$	$-0.71$
static $f{=}0.25$	$1\times 10^{-5}$	$51.51\,\pm\,0.04$	$-1.63$	$80.42\,\pm\,1.40$	$-0.27$
static $f{=}0.5$	$1\times 10^{-5}$	$50.00\,\pm\,0.59$	$-3.14$	$79.45\,\pm\,1.55$	$-1.24$
no filter	$2\times 10^{-5}$	$55.01\,\pm\,0.36$	—	$75.13\,\pm\,1.15$	—

Appendix

Table 9: Magicoder MBPP+ (

n{=}378

) SFT filtering grid,

5

epochs. Values are percent, mean

\pm

sample standard deviation over three seeds;

\Delta

is the change against the no-filter run at the same learning rate. The base (no SFT) row is the same untuned base as in Table 8 .

Benchmark	Filter	pass@1	$\Delta$	pass@16	$\Delta$
MBPP+ ( $n{=}378$ )	base (no SFT)	$36.82\,\pm\,0.56$	—	$75.93\,\pm\,1.06$	—
MBPP+ ( $n{=}378$ )	no filter	$58.26\,\pm\,0.88$	—	$81.22\,\pm\,0.75$	—
MBPP+ ( $n{=}378$ )	ramp $0{\to}0.5$	$55.81\,\pm\,0.06$	$-2.44$	$82.36\,\pm\,0.81$	$+1.15$
HumanEval+ ( $n{=}164$ )	base (no SFT)	$32.94\,\pm\,1.11$	—	$79.27\,\pm\,0.61$	—
HumanEval+ ( $n{=}164$ )	no filter	$54.76\,\pm\,1.16$	—	$85.98\,\pm\,1.61$	—
HumanEval+ ( $n{=}164$ )	ramp $0{\to}0.5$	$51.94\,\pm\,1.83$	$-2.82$	$83.23\,\pm\,2.16$	$-2.74$

Appendix

Table 10: OCI MBPP+ and HumanEval+ SFT grid,

3

epochs, learning rate

2\times 10^{-5}

; Standard is no filtering and TailSFT is the ramp

0\!\to\!0.5

. Values are percent, mean

\pm

sample standard deviation;

\Delta

is the change against no filtering. The base (no SFT) rows are the untuned base under the same per-benchmark protocol.

Filter	LR	pass@1	$\Delta$	pass@16	$\Delta$
base (no SFT)	—	$32.94\,\pm\,1.11$	—	$79.27\,\pm\,0.61$	—
BigCode
no filter	$3\times 10^{-5}$	$45.06\,\pm\,0.57$	—	$76.63\,\pm\,0.70$	—
static $f{=}0.125$	$2\times 10^{-5}$	$48.08\,\pm\,0.12$	$+3.02$	$79.47\,\pm\,1.76$	$+2.85$
static $f{=}0.25$	$2\times 10^{-5}$	$46.33\,\pm\,1.07$	$+1.27$	$80.08\,\pm\,0.93$	$+3.46$
static $f{=}0.5$	$2\times 10^{-5}$	$44.08\,\pm\,0.48$	$-0.98$	$80.69\,\pm\,1.27$	$+4.07$

Appendix

Table 11: HumanEval+ (

n{=}164

) SFT filtering grid. Epochs are

4

(BigCode) and

5

(Magicoder); OCI HumanEval+ appears in Table 10 . Values are percent, mean

\pm

sample standard deviation over three seeds;

\Delta

is the change against the no-filter run for the same dataset; the base (no SFT) row is the untuned OLMo-3 7B base under the same protocol.

Filter	LR	pass@1	$\Delta$	pass@16	$\Delta$
base (no SFT)	—	$25.67\,\pm\,0.62$	—	$70.50\,\pm\,1.54$	—
BigCode
no filter	$3\times 10^{-5}$	$29.79\,\pm\,0.48$	—	$61.71\,\pm\,2.27$	—
static $f{=}0.125$	$2\times 10^{-5}$	$30.65\,\pm\,0.26$	$+0.86$	$63.54\,\pm\,0.76$	$+1.83$
static $f{=}0.25$	$2\times 10^{-5}$	$30.43\,\pm\,0.15$	$+0.64$	$65.79\,\pm\,0.26$	$+4.08$
static $f{=}0.5$	$2\times 10^{-5}$	$29.76\,\pm\,0.35$	$-0.03$	$69.29\,\pm\,1.92$	$+7.58$

Appendix

Table 12: CruxEval-I (

n{=}800

) SFT filtering grid. Epochs are

4

5

3

for BigCode / Magicoder / OCI. Values are percent, mean

\pm

sample standard deviation over three seeds;

\Delta

is the change against the no-filter run for the same dataset; the base (no SFT) row is the untuned OLMo-3 7B base under the same protocol.

Filter	LR	pass@1	$\Delta$	pass@16	$\Delta$
base (no SFT)	—	$4.22\,\pm\,0.74$	—	$32.08\,\pm\,1.98$	—
BigCode
no filter	$3\times 10^{-5}$	$4.16\,\pm\,0.44$	—	$24.21\,\pm\,2.38$	—
static $f{=}0.125$	$2\times 10^{-5}$	$12.02\,\pm\,0.11$	$+7.86$	$39.75\,\pm\,0.76$	$+15.54$
static $f{=}0.25$	$2\times 10^{-5}$	$13.18\,\pm\,0.30$	$+9.02$	$41.00\,\pm\,1.35$	$+16.79$
static $f{=}0.5$	$2\times 10^{-5}$	$7.35\,\pm\,0.37$	$+3.20$	$36.54\,\pm\,0.14$	$+12.33$

Appendix

Table 13: CruxEval-O (

n{=}800

) SFT filtering grid. Epochs are

4

5

3

for BigCode / Magicoder / OCI. Values are percent, mean

\pm

sample standard deviation over three seeds;

\Delta

is the change against the no-filter run for the same dataset; the base (no SFT) row is the untuned OLMo-3 7B base under the same protocol.

Filter	LR	pass@1	$\Delta$	pass@16	$\Delta$
base (no SFT)	—	$5.50\,\pm\,0.42$	—	$32.30\,\pm\,0.93$	—
BigCode
no filter	$3\times 10^{-5}$	$10.74\,\pm\,0.46$	—	$30.39\,\pm\,0.86$	—
static $f{=}0.125$	$2\times 10^{-5}$	$11.77\,\pm\,0.44$	$+1.03$	$31.54\,\pm\,0.71$	$+1.14$
static $f{=}0.25$	$2\times 10^{-5}$	$11.49\,\pm\,0.38$	$+0.75$	$33.12\,\pm\,0.34$	$+2.72$
static $f{=}0.5$	$2\times 10^{-5}$	$11.45\,\pm\,0.42$	$+0.71$	$33.39\,\pm\,0.25$	$+3.00$

Appendix

Table 14: LiveCodeBench (

n{=}612

) SFT filtering grid. Epochs are

4

5

3

for BigCode / Magicoder / OCI. Values are percent, mean

\pm

sample standard deviation over three seeds;

\Delta

is the change against the no-filter run for the same dataset; the base (no SFT) row is the untuned OLMo-3 7B base under the same protocol.

SFT data	Benchmark	Standard p@1	TailSFT p@1	Standard p@16	TailSFT p@16
BigCode	MBPP+ ( $n=378$ )	$53.04_{\pm 0.93}$	$51.36_{\pm 0.14}$	$74.69_{\pm 1.10}$	$78.84_{\pm 1.06}$
BigCode	HumanEval+ ( $n=164$ )	$45.06_{\pm 0.57}$	$46.33_{\pm 1.07}$	$76.63_{\pm 0.70}$	$80.08_{\pm 0.93}$
BigCode	CruxEval-I ( $n=800$ )	$29.79_{\pm 0.48}$	$30.43_{\pm 0.15}$	$61.71_{\pm 2.27}$	$65.79_{\pm 0.26}$
BigCode	CruxEval-O ( $n=800$ )	$4.16_{\pm 0.44}$	$13.18_{\pm 0.30}$	$24.21_{\pm 2.38}$	$41.00_{\pm 1.35}$
BigCode	LiveCodeBench ( $n=612$ )	$10.74_{\pm 0.46}$	$11.49_{\pm 0.38}$	$30.39_{\pm 0.86}$	$33.12_{\pm 0.34}$
Magicoder	MBPP+ ( $n=378$ )	$55.35_{\pm 0.47}$	$54.60_{\pm 0.55}$	$75.31_{\pm 0.93}$	$78.66_{\pm 1.25}$

Appendix

Table 15: Post-SFT code results, matching Table 1 . Values are percentages, mean

\pm

sample standard deviation over three seeds.

SFT data	Benchmark	$\Delta$ p@1	$\Delta$ p@2	$\Delta$ p@4	$\Delta$ p@8	$\Delta$ p@16
BigCode	MBPP+ ( $n{=}378$ )	$-1.68$	$+0.64$	$+2.19$	$+3.19$	$+4.14$
	HumanEval+ ( $n{=}164$ )	$+1.27$	$+3.37$	$+4.34$	$+4.44$	$+3.46$
	CruxEval-I ( $n{=}800$ )	$+0.64$	$+1.61$	$+2.57$	$+3.25$	$+4.08$
	CruxEval-O ( $n{=}800$ )	$+9.02$	$+12.58$	$+15.45$	$+16.95$	$+16.79$
	LiveCodeBench ( $n{=}612$ )	$+0.75$	$+1.39$	$+1.86$	$+2.21$	$+2.72$
Magicoder	MBPP+ ( $n{=}378$ )	$-0.75$	$+1.73$	$+2.95$	$+3.25$	$+3.35$

Appendix

Table 16: SFT pass@k deltas for code benchmarks, reported as TailSFT minus Standard SFT in percentage points at the main-table settings. Deltas are computed with the unbiased OLMES ( Gu et al., 2025 ) pass@k estimator.

Dataset	Source/filter	Train rows	Validation rows
OMI math subset	OpenMathInstruct-2 math / augmented_math , MATH-500 decontaminated	332,500	17,500

Appendix

Table 17: In-domain math SFT data. The annotated training split contains

326{,}725

augmented_math and

5{,}775

math rows; the validation split contains

17{,}192

and

308

, respectively.

Hyperparameter	Values
Base model	OLMo-3 7B base
Learning rate	fixed at $3\times 10^{-5}$
Filter schedule	none; static; ramp $0\!\to\!\cdot$
Filter fraction $f$	$0$ (none); static $\{0.0625,0.125,0.1875,0.25,0.5\}$ ; ramp $0\!\to\!\{0.0625,0.125,0.1875,0.25,0.5\}$
Epochs	$2$
Sequence length	$4096$

Appendix

Table 18: Math SFT hyperparameter grid.

Filter	pass@1	$\Delta$	pass@16	$\Delta$
no filter	$53.55\,\pm\,0.49$	—	$86.27\,\pm\,0.61$	—
static $f{=}0.0625$	$53.66\,\pm\,0.28$	$+0.11$	$87.00\,\pm\,0.87$	$+0.73$
static $f{=}0.125$	$53.60\,\pm\,0.39$	$+0.05$	$87.20\,\pm\,0.40$	$+0.93$
static $f{=}0.1875$	$53.15\,\pm\,0.33$	$-0.41$	$86.20\,\pm\,0.69$	$-0.07$
static $f{=}0.25$	$53.51\,\pm\,0.41$	$-0.04$	$86.33\,\pm\,1.29$	$+0.07$
static $f{=}0.5$	$52.02\,\pm\,0.13$	$-1.54$	$85.60\,\pm\,0.72$	$-0.67$

Appendix

Table 19: MATH-500 full (

n{=}500

) SFT filtering grid,

4096

-token generations. Values are percent, mean

\pm

sample standard deviation over three generation seeds;

\Delta

is the change against no filtering.

Filter	Level 1 ( $n{=}43$ )	Level 2 ( $n{=}90$ )	Level 3 ( $n{=}105$ )	Level 4 ( $n{=}128$ )	Level 5 ( $n{=}134$ )
no filter	$100.00\,\pm\,0.00$	$97.78\,\pm\,1.11$	$96.51\,\pm\,1.98$	$85.68\,\pm\,1.63$	$66.67\,\pm\,1.88$
static $f{=}0.0625$	$100.00\,\pm\,0.00$	$98.15\,\pm\,0.64$	$96.19\,\pm\,0.95$	$85.68\,\pm\,0.90$	$69.40\,\pm\,3.25$
static $f{=}0.125$	$100.00\,\pm\,0.00$	$97.41\,\pm\,0.64$	$96.83\,\pm\,1.10$	$85.68\,\pm\,2.51$	$70.15\,\pm\,1.29$
static $f{=}0.1875$	$100.00\,\pm\,0.00$	$97.78\,\pm\,0.00$	$95.56\,\pm\,1.46$	$84.38\,\pm\,0.78$	$68.41\,\pm\,1.88$
static $f{=}0.25$	$100.00\,\pm\,0.00$	$97.41\,\pm\,0.64$	$96.51\,\pm\,0.55$	$85.94\,\pm\,3.40$	$66.92\,\pm\,2.62$
static $f{=}0.5$	$100.00\,\pm\,0.00$	$96.30\,\pm\,1.70$	$95.56\,\pm\,1.98$	$85.68\,\pm\,3.25$	$65.92\,\pm\,4.24$

Appendix

Table 20: MATH-500 SFT pass@16 by difficulty level, from the same

4096

-token SFT filtering sweep as Table 19 . Columns are the five MATH difficulty levels, with the number of problems in each header. Values are percent, mean

\pm

sample standard deviation over three seeds. pass@16 is at or near ceiling through Level 3, so the aggregate is dominated by the easy levels; the gains from filtering concentrate at Levels 4–5. The frozen main table reports the selected ramp arm on Level 5, re-evaluated with

8192

-token generations.

Filter	Level 1 ( $n{=}43$ )	Level 2 ( $n{=}90$ )	Level 3 ( $n{=}105$ )	Level 4 ( $n{=}128$ )	Level 5 ( $n{=}134$ )
no filter	$85.27\,\pm\,1.46$	$76.25\,\pm\,0.43$	$65.95\,\pm\,1.59$	$47.02\,\pm\,0.65$	$24.66\,\pm\,0.63$
static $f{=}0.0625$	$85.66\,\pm\,1.05$	$76.44\,\pm\,1.15$	$64.98\,\pm\,1.93$	$46.92\,\pm\,0.21$	$25.67\,\pm\,0.72$
static $f{=}0.125$	$84.98\,\pm\,0.89$	$76.69\,\pm\,0.70$	$64.46\,\pm\,0.77$	$47.28\,\pm\,0.47$	$25.56\,\pm\,0.73$
static $f{=}0.1875$	$84.79\,\pm\,0.75$	$75.83\,\pm\,1.05$	$65.06\,\pm\,0.51$	$46.14\,\pm\,1.65$	$25.11\,\pm\,0.66$
static $f{=}0.25$	$85.08\,\pm\,0.69$	$75.62\,\pm\,0.77$	$65.08\,\pm\,0.34$	$47.18\,\pm\,0.86$	$25.51\,\pm\,0.33$
static $f{=}0.5$	$83.58\,\pm\,0.58$	$74.31\,\pm\,0.90$	$62.38\,\pm\,1.11$	$46.48\,\pm\,0.13$	$24.08\,\pm\,0.03$

Appendix

Table 21: MATH-500 SFT pass@1 by difficulty level, from the same sweep as Table 20 . Values are percent, mean

\pm

sample standard deviation over three seeds.

Eval family	Filter	pass@1	$\Delta$	pass@16	$\Delta$
base (no SFT) †		$5.16$	—	$20.91$	—
ramp sweep	no filter	$3.65\,\pm\,0.37$	—	$15.24\,\pm\,2.02$	—
ramp sweep	ramp $0{\to}0.1875$	$3.82\,\pm\,0.20$	$+0.16$	$16.06\,\pm\,1.42$	$+0.82$
ramp sweep	ramp $0{\to}0.25$	$3.78\,\pm\,0.20$	$+0.13$	$15.37\,\pm\,0.23$	$+0.13$
ramp sweep	ramp $0{\to}0.5$	$4.03\,\pm\,0.02$	$+0.37$	$18.31\,\pm\,0.57$	$+3.07$
static	no filter	$3.65\,\pm\,0.34$	—	$15.52\,\pm\,2.00$	—

Appendix

Table 22: AIME 2022–2025 (

n{=}120

) completed SFT filtering evaluations. The ramp sweep and the static pair were evaluated in separate runs, each against its own matched no-filter baseline. Values are percent, mean

\pm

sample standard deviation over three seeds;

\Delta

is the change against the matching no-filter row. † The base (no SFT) row is the untuned OLMo-3 7B base under the same AIME protocol.

SFT data	Benchmark	Standard p@1	TailSFT p@1	Standard p@16	TailSFT p@16
OMI	AIME 2022–2025 ( $n=120$ )	$3.65_{\pm 0.37}$	$4.03_{\pm 0.02}$	$15.24_{\pm 2.02}$	$18.31_{\pm 0.57}$
OMI	MATH-500 Level 5 ( $n=134$ )	$24.80_{\pm 0.40}$	$25.16_{\pm 0.40}$	$66.42_{\pm 0.00}$	$69.15_{\pm 0.86}$
OMI	OMEGA-500 ( $n=500$ )	$6.58_{\pm 0.50}$	$6.51_{\pm 0.26}$	$32.80_{\pm 2.25}$	$32.60_{\pm 1.56}$

Appendix

Table 23: Post-SFT math results, matching Table 1 . Values are percentages, mean

\pm

sample standard deviation over three seeds.

Benchmark	$\Delta$ p@1	$\Delta$ p@2	$\Delta$ p@4	$\Delta$ p@8	$\Delta$ p@16
AIME 2022–2025 ( $n{=}120$ )	$+0.37$	$+0.72$	$+1.20$	$+1.93$	$+3.07$
MATH-500 Level 5 ( $n{=}134$ )	$+0.36$	$+0.71$	$+0.83$	$+0.87$	$+2.74$
OMEGA-500 ( $n{=}500$ )	$-0.06$	$-0.09$	$-0.12$	$-0.23$	$-0.20$

Appendix

Table 24: SFT pass@k deltas for math benchmarks, reported as TailSFT minus Standard SFT in percentage points at the main-table settings. Deltas are computed with the unbiased OLMES ( Gu et al., 2025 ) pass@k estimator.

Setting	Value
Initialization	matched Standard / TailSFT OMI SFT checkpoints ( Table 1 )
Algorithm	GRPO, group-relative advantage, no KL
Training prompts	MATH-lighteval train, MATH-500 removed ( $11{,}396$ train / $600$ val)
Reward	verifiable boxed-answer exact match ( $1$ / $0$ )
Rollout group size $n$	$4$
Actor learning rate	searched $\{1,2\}\times 10^{-5}$ ; reported $2\times 10^{-5}$

Appendix

Table 25: Math GRPO configuration. All runs share the fixed settings; only the actor learning rate is searched, and Table 26 reports the tuned configuration.

	pass@1			pass@16
Benchmark	Standard	TailSFT	$\Delta$	Standard	TailSFT	$\Delta$
MATH-500 Level 5 ( $n{=}134$ )	$57.70\,\pm\,0.44$	$60.26\,\pm\,1.02$	$+2.56$	$83.83\,\pm\,1.72$	$87.06\,\pm\,0.86$	$+3.23$
AIME 2022–2025 ( $n{=}120$ )	$14.40\,\pm\,0.26$	$15.61\,\pm\,0.44$	$+1.21$	$32.76\,\pm\,0.88$	$36.06\,\pm\,0.98$	$+3.30$

Appendix

Table 26: Math GRPO at the tuned configuration (rollout

n{=}4

, actor learning rate

2\times 10^{-5}

, mean-best checkpoint,

8192

-token generation), evaluated from the matched Standard and TailSFT initializations of Table 1 . Values are percent, mean

\pm

standard deviation over three seeds;

\Delta

is the TailSFT improvement.

Setting	Value
Initialization	matched Standard / TailSFT SFT checkpoints per dataset ( Table 1 )
Algorithm	GRPO, group-relative advantage, no KL
Training prompts	MBPP+ train ( $100$ -problem held-out validation)
Reward	full MBPP+ unit-test pass ( $1$ / $0$ )
Rollout group size $n$	searched $\{2,4\}$ ; reported $4$
Actor learning rate	searched $\{5\times 10^{-6},1\times 10^{-5},2\times 10^{-5}\}$ ; reported $2\times 10^{-5}$

Appendix

Table 27: Code GRPO configuration. All runs share the fixed settings; the rollout group size and actor learning rate are searched, and Table 28 reports the tuned configuration for each dataset.

			pass@1			pass@16
SFT data ( TailSFT filter)	$n$	LR	Standard	TailSFT	$\Delta$	Standard	TailSFT	$\Delta$
BigCode (static $f{=}0.25$ )	$4$	$2\times 10^{-5}$	$69.57\,\pm\,0.29$	$73.50\,\pm\,0.70$	$+3.93$	$75.93\,\pm\,0.00$	$78.66\,\pm\,0.15$	$+2.73$
Magicoder (static $f{=}0.25$ )	$2$	$1\times 10^{-5}$	$64.08\,\pm\,0.32$	$66.02\,\pm\,0.55$	$+1.94$	$75.84\,\pm\,0.31$	$79.28\,\pm\,0.55$	$+3.44$
Magicoder (static $f{=}0.25$ )	$4$	$2\times 10^{-5}$	$70.52\,\pm\,0.25$	$73.24\,\pm\,0.09$	$+2.72$	$78.22\,\pm\,0.61$	$80.60\,\pm\,0.55$	$+2.38$
OCI (ramp $0{\to}0.5$ )	$4$	$2\times 10^{-5}$	$74.67\,\pm\,0.08$	$76.30\,\pm\,0.05$	$+1.62$	$84.22\,\pm\,0.40$	$84.04\,\pm\,0.15$	$-0.18$

Appendix

Table 28: Code GRPO (MBPP+,

n{=}378

2048

-token generation) from the matched Standard and TailSFT initializations of Table 1 , at tuned configurations.

n

is the GRPO rollout group size and LR the actor learning rate. Values are percent, mean

\pm

standard deviation over three seeds;

\Delta

is the TailSFT improvement, with yellow marking a change within one standard deviation.

Figure 8: GRPO training reward from Standard and TailSFT initializations. Mean training reward ( critic/rewards/mean ) versus GRPO step for the four main-table settings. The horizontal dashed line marks the Standard initialization’s first-step reward. TailSFT begins below Standard but recovers Standard’s starting reward within a small fraction of training and rises at least as fast thereafter.

SFT dataset	Benchmark	$\lvert\mathcal{R}_{0}\rvert$	$\rho_{16}$	$\Delta$ (pp)
OMI	AIME	16	$7.60$	$+9.92$
	MATH-500 L5	31	$2.28$	$+3.23$
	OMEGA-500	140	$0.90$	$-0.95$
BigCode	CruxEval-I	199	$1.63$	$+3.02$
	CruxEval-O	237	$2.87$	$+28.69$
	HumanEval+	29	$1.39$	$+9.20$

Appendix

Table 29: Exact values plotted in Figure 4 : the base-reachable set size

\lvert\mathcal{R}_{0}\rvert

, the coverage ratio

\rho_{16}

( Section 4.2 ), and the coverage gain

\Delta

(mean empirical pass@

16

difference of TailSFT over standard SFT on

\mathcal{R}_{0}

, in percentage points), for each (SFT dataset, benchmark) pair.

Explore similar work

Jun 7, 2026cs.LG

When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff

Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become a standard pipeline for Large Language Model (LLM) post-training. SFT is expected to provide a useful behavioral prior for RL to further enhance model capabilities. However, checkpoints with excessive SFT often show limited improvement during RL. We attribute this failure to the loss of model plasticity: the reduced ability of an SFT-initialized policy to be effectively reshaped by subsequent RL. To better understand this phenomenon, we conduct detailed analysis from multiple perspectives, including parameter changes, output spaces, and RL optimization dynamics. Our results show that models from excessive SFT tend to produce over-confident token distributions and exhibit sharp parameter landscapes, which make them harder to optimize in the RL stage. To enable a more robust SFT-to-RL handoff, we propose \texttt{Rejuvenation}, a simple yet effective method that restores plasticity while preserving useful SFT-acquired priors. Rejuvenation leverages base-anchored model fusion to reduce excessive SFT-induced drift with targeted neuron reset to mitigate model rigidity. Experimental results on both math reasoning tasks and agentic tasks demonstrate that our approach consistently improves RL performance on over-trained SFT models, while also enhancing generalization to out-of-distribution tasks.

Runze Liu, Jiashun Liu, Xu Wan +2

Hong Kong University of Science and Technology · Zhejiang University · State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA

Supervised Fine-Tuning RL Post-Training

Ask PDF

Jun 22, 2026cs.LG

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

Recent advances in large language models (LLMs) have demonstrated that reinforcement fine-tuning of pretrained base models can lead to significant gains in reasoning performance at inference time. In this work, we theoretically analyze why reinforcement fine-tuning induces better reasoning ability than purely supervised fine-tuning (SFT) methods. We model chain-of-thought (CoT) reasoning as a pathfinding problem on graphs and compare the popular method of reinforcement learning with verifiable rewards (RLVR) against traditional SFT. We prove that SFT, when trained on golden shortest paths without negative examples, fails to learn how to efficiently backtrack. In contrast, an RLVR-trained model can learn how to efficiently backtrack from dead ends using only outcome reward. This leads to an exponential separation in inference-time compute between the two methods, and demonstrates that RLVR leads the model to learn the location of difficult decisions in a reasoning chain, ultimately allowing for better allocation of inference-time compute. Finally, we show that the reasoning traces of an RLVR model can be distilled to train a base model to backtrack efficiently as well.

Stanley Wei, Juno Kim

1Princeton University · University of California, Berkeley.

RL for Language Model Reasoning Efficient Language Model Reasoning Reinforcement Learning with Verifiable Rewards RL Fine-Tuning LLM Reasoning

Ask PDF

Sep 27, 2026cs.LG

Selecting Diverse SFT Traces Improves Post-RL Generalization

Verified solutions are not equally useful for preparing reasoning models for reinforcement learning (RL). We present a comprehensive study of route diversity, the variation in the sequences of reasoning steps in supervised fine-tuning (SFT) data, and propose a lightweight, rule-based fingerprint to select for it. From one pool at one budget, with matched training recipes and checkpoints, selecting diverse rather than similar routes improves post-RL problem coverage across puzzles and mathematics, including on problems harder than those seen in either training stage. In synthetic experiments, route-diverse SFT improves OLMo3-7B's pass@8 by 16.9 points on environments held out from SFT. In a single-model condition, where one model writes every candidate, diverse selection gains up to 6.2 points of mean pass@8 across 10 mathematics benchmarks. Pre-RL diagnostics suggest why: diverse SFT can produce both successful and failed attempts on more prompts despite slightly lower mean accuracy, giving group-relative RL more prompts with a learning signal. On 3 open-source corpora, our CPU-only selector, without model calls, outperforms more expensive alternatives in every comparison of mean post-RL performance. These results identify reasoning-route diversity as a practical criterion for selecting SFT data that better prepares models for RL.

Dylan Zhang, Mingyuan Wu, Jinning Li

University of Illinois Urbana-Champaign, work done at Google · Google

Reasoning Diversity RL for Language Model Reasoning Training Data Selection RL Post-Training LLM Reasoning

Ask PDF