Reinforcement learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm for eliciting reasoning capabilities in large language models, particularly in mathematics and coding. While recent efforts have extended this paradigm to broader general scientific (STEM) domains, the complex interplay between supervised fine-tuning (SFT) and RL in these contexts remains underexplored. In this paper, we conduct controlled experiments revealing a critical challenge: for general STEM domains, RL applied directly to base models is highly sample-inefficient and is consistently surpassed by supervised fine-tuning (SFT) on moderate-quality responses. Yet sequential SFT followed by RL can further improve performance, suggesting that the two stages play complementary roles, and that how training data is allocated between them matters. Therefore, we propose DeReason, a difficulty-based data decoupling strategy for general reasoning. DeReason partitions training data by reasoning intensity estimated via LLM-based scoring into reasoning-intensive and non-reasoning-intensive subsets. It allocates broad-coverage, non-reasoning-intensive problems to SFT to establish foundational domain knowledge, and reserves a focused subset of difficult problems for RL to cultivate complex reasoning. We demonstrate that this principled decoupling yields better performance than randomly splitting the data for sequential SFT and RL. Extensive experiments on general STEM and mathematical benchmarks demonstrate that our decoupled curriculum training significantly outperforms SFT-only, RL-only, and random-split baselines. Our work provides a systematic study of the interplay between SFT and RL for general reasoning, offering a highly effective and generalized post-training recipe.
Figures & tables
Figure 1: Scaling behavior of RL and SFT on both general STEM and math domains, trained on the same set of problems. In the general domain, we report pass@1 averaged across 8 runs on GPQA-Diamond. In the math domain, we report the same metric averaged across AIME24, AIME25, and MATH500. In both settings, SFT with moderate-quality model responses outperforms RL as training data increases.
Models
MMLU-Pro
GPQA-D
SuperGPQA
BBEH
AVG
Baselines
GPT-4o
74.6
50.0
46.3
22.3
48.3
QwQ-32B
52.0
54.5
43.6
22.6
43.2
DeepSeek-R1
84.0
71.5
59.9
34.9
62.6
Qwen2.5-7B-Base
47.7
29.3
26.7
8.0
27.9
Qwen2.5-7B-Instruct
57.0
33.8
30.7
12.2
33.4
Table 1: Main results on reasoning benchmarks. Bold indicates best performance within each model group.
Figure 2: The category distribution across difficulty scores.
Models
AIME24
AIME25
MATH500
4B Models (Webinstruct-verified)
WebIns-V (RL only)
20.0
15.4
80.6
WebIns-V (SFT only)
22.0
17.6
82.6
WebIns-V (Ours, SFT then selected RL)
22.1
18.0
84.1
Webscale (RL only)
21.3
14.0
81.6
Webscale (SFT only)
26.3
23.3
87.5
Table 2: Math reasoning results on AIME24, AIME25, and MATH500.
Figure 3: Reward score during RL training with data in different difficulty score (from 1 to 5) from Webinstruct-verified. The left sub figure is starting from SFT checkpoint, right sub figure is starting from base checkpoint
Figure 4: Response length during RL training with data in different difficulty scores.
Figure 5: Actor’s entropy during RL training with different initial checkpoint.
Figure 6: GPQA-diamond score during RL training from SFT checkpoint and base model respectively. Pass@1 averaged across 8 runs is computed for each checkpoints.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Table 3: Prompt template for rating the reasoning complexity of questions. The {question} placeholder is replaced with the target question at inference time.
Post-training Small Language Models (SLMs) for reasoning typically follows an SFT-then-RL pipeline, yet existing work rarely considers what data should be learned at each stage. We argue that data strategy should be aligned with the distinct roles of SFT and RL: SFT is better suited for acquiring not-yet-mastered reasoning skills, while RL is better suited for consolidating skills that the model can already partially access. Based on this principle, we propose a difficulty-aware SFT-then-RL framework that organizes training data into stage-specific sets. For hard samples in the SFT stage, we introduce a Bridge mechanism that transforms raw teacher-generated reasoning traces into more learnable supervision for SLMs. For hard samples that remain unsolved during RL, we apply Critique Fine-Tuning by converting all-zero-reward failures into diagnostic, repair, and new reasoning trace supervision for the next SFT stage. Experiments on two SLMs across five reasoning benchmarks show that our method consistently improves over representative SFT, distillation, and RL baselines. Our results highlight the importance of coordinating data difficulty across SFT and RL for effective SLM reasoning post-training.
Chongyang He, Rui Zhang, Zixuan Wang +1
1Tsinghua University · 2National University of Singapore · 3DiDi +1
Recent advances in large language models (LLMs) have demonstrated that reinforcement fine-tuning of pretrained base models can lead to significant gains in reasoning performance at inference time. In this work, we theoretically analyze why reinforcement fine-tuning induces better reasoning ability than purely supervised fine-tuning (SFT) methods. We model chain-of-thought (CoT) reasoning as a pathfinding problem on graphs and compare the popular method of reinforcement learning with verifiable rewards (RLVR) against traditional SFT. We prove that SFT, when trained on golden shortest paths without negative examples, fails to learn how to efficiently backtrack. In contrast, an RLVR-trained model can learn how to efficiently backtrack from dead ends using only outcome reward. This leads to an exponential separation in inference-time compute between the two methods, and demonstrates that RLVR leads the model to learn the location of difficult decisions in a reasoning chain, ultimately allowing for better allocation of inference-time compute. Finally, we show that the reasoning traces of an RLVR model can be distilled to train a base model to backtrack efficiently as well.
Stanley Wei, Juno Kim
1Princeton University · University of California, Berkeley.
Large reasoning models trained via reinforcement learning (RL) have been increasingly shown to outperform their supervised fine-tuned (SFT) counterparts on mathematical reasoning tasks; Yet the mechanistic basis for this advantage remains unclear. We therefore ask, what internal representational differences enable RL models' superior performance? Our work presents two converging lines of evidence: First, linear probes trained on layer-wise hidden states reveal that RL models tend to achieve higher accuracy in predicting answer correctness compared to SFT models, indicating more linearly separable and structured representations. Second, mean ablation studies show that RL models develop a hierarchical architecture where deeper layers become progressively more critical, whereas SFT models distribute importance uniformly across layers. Together, these findings demonstrate that RL training fundamentally restructures how models represent and process reasoning problems. Finally, we analyze token-count variability under repeated sampling across problems to assess adaptive compute allocation. While we observe higher variability in some RL-tuned models than in their SFT counterparts, we see strong consistency in others, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone. We believe this token-allocation variability reveals the spread of plausible on-policy reasoning, highlighting which models exhibit stable policies versus those that are under-determined, potentially non-identifiable solution behaviour.