Rethinking "RL Generalizes, SFT Memorizes": The Role of SFT Data
Organizations: National University of Singapore, Singapore · Yale University, USA · Cisco Foundation AI, USA
Abstract
Large language models trained with Reinforcement Learning (RL) with verifiable rewards exhibit strong reasoning ability and broad generalization, whereas models trained with Supervised Fine-Tuning (SFT) are often viewed as more prone to memorization and limited transfer. This paper rethinks this distinction through the lens of SFT training data on widely used Qwen2.5 and Qwen3 series models. First, we study the role of data source and show that it is critical: a carefully mixed SFT dataset substantially outperforms data generated solely by a larger model. Second, we study the role of data scale and show that matching the number of correct rollouts between SFT and RL greatly improves SFT generalization, while matching the total rollout budget enables SFT to generalize as well as RL. Combining these two factors further enables SFT to generalize even better than RL. Third, by using LLM annotations to characterize the solution methods in training rollouts, we show that larger datasets cover more tail methods and that these tail methods provide generalizable reasoning signals. Finally, we support these empirical findings theoretically by analyzing the training dynamics of shallow transformers under both RL and SFT.
Figures & tables
| Math | General Reasoning | |||||||||
| Model | AIME24 (Avg@32) | Minerva Math | GSM8k | MATH500 | Olympiad Bench | ACP Bench | BBH | C-Eval | Ground Cocoa | MMLU -Pro |
| Qwen2.5-7B | 4.7 | 26.5 | 88.6 | 63.6 | 30.2 | 23.3 | 10.9 | 40.1 | 34.7 | 9.2 |
| RL-tuned | 14.2 | 34.9 | 90.8 | 74.8 | 39.7 | 50.1 | 42.9 | 43.6 | 34.8 | 45.7 |
| SFT w/ RL-tuned | 13.8 | 37.1 | 91.4 | 74.8 | 39.3 | 58.0 | 33.8 | 42.4 | 35.6 | 19.4 |
| SFT w/ Mix-1 | 13.9 | 37.5 | 91.3 | 76.6 | 39.0 | 56.6 | 34.4 | 36.4 | 36.8 | 17.9 |
| SFT w/ Mix-2 | 13.4 | 41.2 | 91.9 | 76.2 | 36.9 | 53.4 | 37.3 | 47.8 | 37.0 | 21.3 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | General Reasoning Average |
| RL-tuned | 42.5 |
| SFT w/RL-tuned | 42.1 |
| Mix-1 | 41.3 |
| Mix-2 | 42.0 |
| Mix-3 | 42.6 |
| SFT w/Qwen3-14B | 39.7 |
| Hyperparameter | Value |
| Training batch size | 128 |
| PPO mini-batch size | 64 |
| PPO micro-batch size per GPU | 1 |
| Prompt length | 1024 |
| Response length | 8192 |
| Rollout sample number ( ) | 8 |
| Hyperparameter | Value |
| DeepSpeed config | deepspeed_z3 |
| Template | qwen |
| Cutoff length | 16000 |
| Per-device train batch size | 1 |
| Gradient accumulation steps | 8 |
| Learning rate |
| Benchmark | Qwen2.5-7B | RL | SFT w/ RL-tuned | Mix model 1 | Mix model 2 | Mix model 3 | Mix model 4 | Mix model 5 | SFT w/ Qwen3-14B |
| GPQA-Diamond | 22.7 | 30.3 | 21.7 | 23.7 | 25.3 | 21.7 | 28.8 | 27.3 | 27.3 |
| HEAD-QA | 33.6 | 33.6 | 32.9 | 33.3 | 33.1 | 33.0 | 32.1 | 31.6 | 31.4 |
| ARC | 42.1 | 42.0 | 40.3 | 40.8 | 41.6 | 42.9 | 40.3 | 40.0 | 37.8 |
| ACPBench | 23.3 | 50.1 | 58.0 | 56.6 | 53.4 | 50.7 | 43.7 | 46.6 | 45.3 |
| BIG-Bench Hard | 10.9 | 42.9 | 33.8 | 34.4 | 37.3 | 35.6 | 34.1 | 37.7 | 42.1 |
| C-Eval | 40.1 | 43.6 | 42.4 | 36.4 | 47.8 | 28.2 | 52.9 | 53.8 | 36.5 |
| Method | Average score stderr |
| Base | |
| RL | |
| SFT w/ RL-tuned | |
| Mix 1 | |
| Mix 3 | |
| Mix 5 |
| Method | Method | Method |
| 3D geometry & volumes | equation solving | partial fractions |
| absolute value methods | estimation & bounding | partitions |
| algebraic manipulation | Euler’s formula | pattern recognition |
| angle properties | expected value | percentages |
| area & perimeter | exponents & powers | perfect powers |
| arithmetic operations | extremal principle | periodicity |
| Benchmark | SFT w/ less tail | SFT w/ RL-tuned data |
| All rollouts | ||
| MMLU-Pro | 455 | 550 |
| MMLU-Pro+ | 431 | 589 |
| Correct rollouts | ||
| MMLU-Pro | 40 | 108 |
| MMLU-Pro+ | 21 | 84 |
| Notations | Descriptions |
| The input informative and context tokens. | |
| The attention score from query to token at position . | |
| The attention score from query to informative token . | |
| The attention score from query to context token . | |
| The attention score from query to all context token. | |
| The attention logit from query to informative token . |
| Model | General Reasoning Average |
| RL-tuned | 43.8 |
| SFT w/RL-tuned | 43.7 |
| SFT w/RL-tuned (more unique rollouts) | 43.4 |
| Setting | Method coverage |
| Rollout during RL | 137 |
| RL-tuned dataset (rollout=8) | 132 |
| RL-tuned dataset (rollout=1) | 101 |