Large language models trained with Reinforcement Learning (RL) with verifiable rewards exhibit strong reasoning ability and broad generalization, whereas models trained with Supervised Fine-Tuning (SFT) are often viewed as more prone to memorization and limited transfer. This paper rethinks this distinction through the lens of SFT training data on widely used Qwen2.5 and Qwen3 series models. First, we study the role of data source and show that it is critical: a carefully mixed SFT dataset substantially outperforms data generated solely by a larger model. Second, we study the role of data scale and show that matching the number of correct rollouts between SFT and RL greatly improves SFT generalization, while matching the total rollout budget enables SFT to generalize as well as RL. Combining these two factors further enables SFT to generalize even better than RL. Third, by using LLM annotations to characterize the solution methods in training rollouts, we show that larger datasets cover more tail methods and that these tail methods provide generalizable reasoning signals. Finally, we support these empirical findings theoretically by analyzing the training dynamics of shallow transformers under both RL and SFT.
Figures & tables
Figure 3: Performance comparison of sft -trained models using datasets with different mixing ratios. Mix- k consists of (11−2k)×10% data from the rl -tuned dataset and (2k−1)×10% data from the Qwen3-14B dataset. The results report performance on both mathematical reasoning and broader reasoning tasks. The mixture with 70% rl -tuned data and 30% Qwen3-14B data achieves the best performance across both tasks.
Math
General Reasoning
Model
AIME24 (Avg@32)
Minerva Math
GSM8k
MATH500
Olympiad Bench
ACP Bench
BBH
C-Eval
Ground Cocoa
MMLU -Pro
Qwen2.5-7B
4.7
26.5
88.6
63.6
30.2
23.3
10.9
40.1
34.7
9.2
RL-tuned
14.2
34.9
90.8
74.8
39.7
50.1
42.9
43.6
34.8
45.7
SFT w/ RL-tuned
13.8
37.1
91.4
74.8
39.3
58.0
33.8
42.4
35.6
19.4
SFT w/ Mix-1
13.9
37.5
91.3
76.6
39.0
56.6
34.4
36.4
36.8
17.9
SFT w/ Mix-2
13.4
41.2
91.9
76.2
36.9
53.4
37.3
47.8
37.0
21.3
Table 1: Evaluation results on math and selected general reasoning tasks of models trained from mixed data sources. The highest and second highest number within the sft -tuned models are highlighted in bold and underlined , respectively. SFT w/ Mix-2 achieves the best performance among the sft -tuned models.
Figure 6: Performance comparison of sft -trained models on mathematical and broader reasoning benchmarks when different training factors are matched to rl , including the number of prompts, solvable prompts, correct rollouts, and total rollouts. Matching the total rollout budget of sft to that of rl substantially improves general reasoning performance, while preserving strong mathematical reasoning ability.
Figure 9: Performance comparison of sft -trained models using scaled training datasets with different mixing ratios. The datasets are mixtures of rl -tuned data and Qwen3-14B data. After scaling the training data, the mixture with 90% rl -tuned data and 10% Qwen3-14B data achieves the best performance.
Figure 10: Characterization of LLM-annotated method distribution and coverage. (a) plots the occurrence frequencies of solution methods in the rl -tuned dataset, showing that the method distribution is long-tailed. (b) plots the number of covered methods in datasets of different sizes, showing that method coverage increases as the training data scale grows.
Figure 13: Tail-method usage statistics. The figures report the usage of LLM-annotated tail methods by sft -trained models on MMLU-Pro and MMLU-Pro+. Tail methods are used more frequently (light bars) and can correctly solve more questions (dark bars) in general reasoning tasks (e.g., MMLU-Pro, MMLU-Pro+), as the training dataset encompasses a broader range of methods.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Method
General Reasoning Average
RL-tuned
42.5
SFT w/RL-tuned
42.1
Mix-1
41.3
Mix-2
42.0
Mix-3
42.6
SFT w/Qwen3-14B
39.7
Appendix
Table 2: General reasoning performance comparison with different mixing ratios on Qwen3-8B-Base.
Hyperparameter
Value
Training batch size
128
PPO mini-batch size
64
PPO micro-batch size per GPU
1
Prompt length
1024
Response length
8192
Rollout sample number ( n )
8
Appendix
Table 3: Hyperparameters for RL training with verl.trainer.main_ppo .
Hyperparameter
Value
DeepSpeed config
deepspeed_z3
Template
qwen
Cutoff length
16000
Per-device train batch size
1
Gradient accumulation steps
8
Learning rate
1.0×10−5
Appendix
Table 4: Hyperparameters for supervised fine-tuning (SFT).
Benchmark
Qwen2.5-7B
RL
SFT w/ RL-tuned
Mix model 1
Mix model 2
Mix model 3
Mix model 4
Mix model 5
SFT w/ Qwen3-14B
GPQA-Diamond
22.7
30.3
21.7
23.7
25.3
21.7
28.8
27.3
27.3
HEAD-QA
33.6
33.6
32.9
33.3
33.1
33.0
32.1
31.6
31.4
ARC
42.1
42.0
40.3
40.8
41.6
42.9
40.3
40.0
37.8
ACPBench
23.3
50.1
58.0
56.6
53.4
50.7
43.7
46.6
45.3
BIG-Bench Hard
10.9
42.9
33.8
34.4
37.3
35.6
34.1
37.7
42.1
C-Eval
40.1
43.6
42.4
36.4
47.8
28.2
52.9
53.8
36.5
Appendix
Table 5: Performance Comparison Across General Reasoning Benchmarks
Method
Average score ± stderr
Base
33.8±0.41
RL
43.8±0.43
SFT w/ RL-tuned
43.7±0.41
Mix 1
44.4±0.42
Mix 3
42.9±0.41
Mix 5
41.3±0.42
Appendix
Table 6: Evaluation results with standard error.
Figure 16: Ablation study on the components of the training dataset with the training compute budget fixed. When the training dataset shrinks, the performance also downgrades.
Method
Method
Method
3D geometry & volumes
equation solving
partial fractions
absolute value methods
estimation & bounding
partitions
algebraic manipulation
Euler’s formula
pattern recognition
angle properties
expected value
percentages
area & perimeter
exponents & powers
perfect powers
arithmetic operations
extremal principle
periodicity
Appendix
Table 7: Alphabetical Classification of Mathematical Methods
Benchmark
SFT w/ less tail
SFT w/ RL-tuned data
All rollouts
MMLU-Pro
455
550
MMLU-Pro+
431
589
Correct rollouts
MMLU-Pro
40
108
MMLU-Pro+
21
84
Appendix
Table 8: Tail method usage count. With more tail methods in the training dataset, more tail methods are used.
Notations
Descriptions
Pin=(X1⊤,X2⊤,…,XN⊤)⊤
The input informative and context tokens.
Sn=soft(XqWPin⊤)n
The attention score from query Xq to token Xn at position n .
Sl,k=∑n=1N\mathbbm1{Xn=il,k}Sn
The attention score from query ul to informative token il,k .
Sm=∑n=1N\mathbbm1{Xn=cj}Sn
The attention score from query Xq to context token cj .
S=∑j=1JSm
The attention score from query Xq to all context token.
Bl,k=ulWil,k⊤
The attention logit from query ul to informative token il,k .
Appendix
Table 9: Summary of frequently used notations in proof. We omit (t) here for simplicity.
Model
General Reasoning Average
RL-tuned
43.8
SFT w/RL-tuned
43.7
SFT w/RL-tuned (more unique rollouts)
43.4
Appendix
Table 10: SFT with more unique correct rollouts share similar performance with SFT w/ RL-tuned with total number of rollouts matched.
Setting
Method coverage
Rollout during RL
137
RL-tuned dataset (rollout=8)
132
RL-tuned dataset (rollout=1)
101
Appendix
Table 11: Method coverage of the training datasets. RL produces a more balanced rollout dataset during training.
Reinforcement learning (RL)-based post-training often improves the reasoning performance of large language models (LLMs) beyond the training domain, while supervised fine-tuning (SFT) frequently leads to general capabilities forgetting. However, the mechanisms underlying this contrast remain unclear. To bridge this gap, we present a feature-level mechanistic analysis methodology to probe RL generalization using a controlled experimental setup, where RL- and SFT-tuned models are trained from the same base model on identical data. Leveraging our interpretability framework, we align internal activations across models within a shared feature space and analyze how features evolve during post-training. We find that SFT rapidly introduces many highly specialized features that stabilize early in training, whereas RL induces more restrained and continually evolving feature changes that largely preserve base models' representations. Focusing on samples where RL succeeds but the base model fails, we identify a compact, task-agnostic set of features that directly mediate generalization across diverse tasks. Feature-level interventions confirm their causal role: disabling these features significantly degrades RL models' generalization performance, while amplifying them improves base models' performance. The code is available at https://github.com/danshi777/RL-generalization.
Dan Shi, Zhuowen Han, Simon Ostermann +3
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany · Saarland University, Saarbrücken, Germany
A prevailing narrative in LLM post-training holds that supervised finetuning (SFT) memorizes while reinforcement learning (RL) generalizes. We revisit this claim for reasoning SFT with long chain-of-thought (CoT) supervision and find that cross-domain generalization is not absent but conditional, jointly shaped by optimization dynamics, training data, and base-model capability. Some reported failures are under-optimization artifacts: cross-domain performance first degrades before recovering and improving with extended training (a dip-and-recovery pattern), so shorttraining checkpoints can underestimate generalization. Data quality and structure both matter: low-quality solutions broadly hurt generalization,while verified long-CoT traces yield consistent cross-domain gains. Model capability is essential: stronger models internalize transferable procedural patterns (e.g., backtracking) even from a toy arithmetic game, while weaker ones imitate surface verbosity. This generalization is asymmetric, however: reasoning improves while safety degrades, reframing the question from whether reasoning SFT generalizes to under what conditions and at what cost.
Qihan Ren, Peng Wang, Ruikun Cai +8
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · University of Science and Technology of China
Recent advances in large language models (LLMs) have demonstrated that reinforcement fine-tuning of pretrained base models can lead to significant gains in reasoning performance at inference time. In this work, we theoretically analyze why reinforcement fine-tuning induces better reasoning ability than purely supervised fine-tuning (SFT) methods. We model chain-of-thought (CoT) reasoning as a pathfinding problem on graphs and compare the popular method of reinforcement learning with verifiable rewards (RLVR) against traditional SFT. We prove that SFT, when trained on golden shortest paths without negative examples, fails to learn how to efficiently backtrack. In contrast, an RLVR-trained model can learn how to efficiently backtrack from dead ends using only outcome reward. This leads to an exponential separation in inference-time compute between the two methods, and demonstrates that RLVR leads the model to learn the location of difficult decisions in a reasoning chain, ultimately allowing for better allocation of inference-time compute. Finally, we show that the reasoning traces of an RLVR model can be distilled to train a base model to backtrack efficiently as well.
Stanley Wei, Juno Kim
1Princeton University · University of California, Berkeley.