Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
Authors: Taojie Zhu, Dongyang Xu, Ding Zou, Sen Zhao, Qiaobo Hao, Zhiguo Yang, Yonghong He
Organizations: Shenzhen International Graduate School, Tsinghua University · Intelligent System Department, Zhongxing Telecom Equipment (ZTE) · Institute of Advanced Interdisciplinary Studies, Chongqing University of Posts and Telecommunications
Post-training paradigms for Large Language Models (LLMs), primarily Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), face a fundamental dilemma: SFT provides stability (low variance) but suffers from high fitting bias, while RL enables exploration (low bias) but grapples with high gradient variance. Existing unified optimization strategies often employ naive loss weighting, overlooking the statistical conflict between these distinct gradient signals. In this paper, we provide a rigorous theoretical analysis of this bias-variance trade-off and propose DYPO (Dynamic Policy Optimization), a unified framework designed to structurally mitigate this conflict. DYPO integrates three core components: (1) a Group Alignment Loss (GAL) that leverages intrinsic group dynamics to significantly reduce RL gradient variance; (2) a Multi-Teacher Distillation mechanism that corrects SFT fitting bias via diverse reasoning paths; and (3) a Dynamic Exploitation-Exploration Gating mechanism that adaptively arbitrates between stable SFT and exploratory RL based on reward feedback. Theoretical analysis confirms that DYPO linearly reduces fitting bias and minimizes overall variance. Extensive experiments demonstrate that DYPO significantly outperforms traditional sequential pipelines, achieving an average improvement of 4.8% on complex reasoning benchmarks and 13.3% on out-of-distribution tasks. Our code is publicly available at https://github.com/Tocci-Zhu/DYPO.
Figures & tables
Figure 1: The SFT-RL Dilemma: Balancing the high-bias stability of SFT against the high-variance exploration of RL.
Figure 2: The overall framework of DYPO. The system employs a Dynamic Difficulty Grading mechanism to categorize queries into Easy, Hard, and Mid tiers based on group rollout outcomes, dispatching them to the most effective optimization pathway.
Figure 3: The Contrastive Mechanism in GAL.
Model
In-Distribution
Out-of-Distribution
AIME 24
AIME 25
AMC
MATH-500
Minerva
Avg
ARC-c
GPQA-D
Avg
Qwen2.5-Math-7B
11.5
4.9
31.3
43.6
7.4
19.7
18.2
11.1
14.6
Supervised Fine-Tuning
SFT
22.2
22.3
52.8
82.6
40.8
44.1
75.2
24.7
50.0
Reinforcement Learning
RL
25.1
15.3
62.0
84.4
39.3
45.2
82.3
40.4
61.4
Table 1: Overall performance on five competition-level mathematical reasoning benchmarks and two out-of-distribution benchmarks(Qwen2.5-Math-7B). Best results are bolded and second-best are underlined .
Model
In-Distribution
Out-of-Distribution
AIME 24/25
AMC
MATH-500
Minerva
Avg
ARC-c
GPQA-D
Avg
Qwen3-4B-Base
9.3/5.3
40.0
66.8
27.9
29.9
49.4
14.1
31.8
SFT
33.3/27.3
62.9
73.8
43.0
48.1
73.8
28.8
51.3
RL
40.6/37.3
71.8
91.0
46.3
57.4
76.7
29.3
53.0
SFT → RL
43.3/39.3
75.4
77.4
44.9
56.1
77.4
27.8
52.6
DYPO
59.3/44.0
86.0
94.6
50.4
66.9
92.5
44.4
68.5
Table 2: Overall performance on mathematical reasoning benchmarks (Qwen3-4B-Base). Best results are bolded and second-best are underlined .
Model
AIME 24
AIME 25
AMC
GPQA-D
Qwen2.5-Math-7B
11.5
4.9
31.3
11.1
+ SFT
22.2
22.3
52.8
24.7
+ Multi-Teacher (235B / 32B / 8B)
26.6 / 26.8 / 24.5
23.3 / 23.5 / 22.0
61.4 / 61.8 / 59.5
33.3 / 32.3 / 30.8
+ RL (235B / 32B / 8B)
27.3 / 26.9 / 25.0
26.6 / 26.0 / 25.5
64.1 / 63.5 / 61.0
34.8 / 35.4 / 31.8
+ Dynamic Grading (235B / 32B / 8B)
33.3 / 31.5 / 31.8
28.7 / 27.9 / 28.1
63.6 / 62.0 / 62.4
36.4 / 35.4 / 34.3
+ GAL (DYPO) (235B / 32B / 8B)
36.0 / 35.2 / 33.5
28.7 / 28.5 / 27.8
67.0 / 66.5 / 65.2
41.4 / 41.4 / 39.4
Table 3: Ablation study under different teacher strengths.
Figure 4: Left: Offline data ratio over steps. Mid: Training reward; Right: Policy entropy.
Figure 5: Gradient Norm Comparison.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Method
AIME 24
AMC
GPQA-D
DYPO (Single-Teacher)
32.8
64.2
37.5
DYPO (2-Teacher)
36.0
67.0
41.4
Appendix
Table 4: Comparison between single-teacher and multi-teacher supervision in DYPO.
α
βGAL
AIME 24
MATH-500
0.2
0.1
33.5 ± 1.4
87.6 ± 0.7
0.2
1
34.8 ± 0.9
88.5 ± 0.5
0.2
2
33.9 ± 1.2
87.9 ± 0.6
0.5
0.1
35.2 ± 0.8
88.9 ± 0.4
0.5
1
36.0 ± 1.1
89.2 ± 0.3
0.5
2
35.5 ± 1.0
88.7 ± 0.5
Appendix
Table 5: Sensitivity analysis of α and βGAL . Results are reported as mean ± standard deviation over multiple random seeds.
Reinforcement Learning (RL) has significantly improved large language model reasoning, but existing RL fine-tuning methods rely heavily on heuristic techniques such as entropy regularization and reweighting to maintain stability. In practice, they often suffer from late-stage performance collapse, leading to degraded reasoning quality and unstable training. We identify a key factor behind this instability: a small fraction of tokens, termed spurious tokens (around 0.01%), which contribute little to the reasoning outcome but receive disproportionately amplified gradient updates due to inheriting the full sequence-level reward. We present a unified framework for evaluating token-level optimization impacts across spurious risk, gradient norms, and entropy changes. Building on the analysis of token characteristics that severely disrupt optimization, we propose the Silencing Spurious Tokens (S2T) mechanism to efficiently suppress their gradient perturbations. Incorporating this mechanism into a group-based objective, we propose Spurious-Token-Aware Policy Optimization (STAPO), which promotes stable and effective large-scale model refinement. Across six mathematical reasoning benchmarks using Qwen 1.7B, 8B, and 14B base models, STAPO consistently demonstrates superior entropy stability and achieves an average performance improvement of 11.49% (ρT=1.0, top-p=1.0) and 3.73% (ρT=0.7, top-p=0.9) over GRPO, 20-Entropy, and JustRL.
Shiqi Liu, Zeyu He, Guojian Zhan +10
School of Vehicle and Mobility & College of AI, Tsinghua University · Didi Voyager Labs, DiDi Autonomous Driving
Recent mixed-policy optimization methods for LLM reasoning that interleave or blend supervised and reinforcement learning signals report improvements over the standard SFT-then-RL pipeline. We show that numerous recently published research papers rely on a faulty baseline caused by two distinct bugs: a CPU-offloaded optimizer bug in DeepSpeed that silently drops intermediate micro-batches during gradient accumulation (affecting multiple downstream frameworks including TRL, OpenRLHF and Llama-Factory), and a loss aggregation bug in OpenRLHF that incorrectly weights per-mini-batch losses. Together they suppress SFT performance, with the optimizer bug accounting for most of the gap and the loss aggregation bug contributing a smaller additional effect. Once corrected, the standard SFT-then-RL pipeline surpasses every published mixed-policy method we evaluate by +3.8 points on math benchmarks with Qwen2.5-Math-7B and by +22.2 points with Llama-3.1-8B. Even a truncated variant with just 50 RL steps outperforms mixed-policy methods on math benchmarks while using fewer FLOPs.
Alexis Limozin, Eduard Durech, Torsten Hoefler +2
ETH AI Center, ETH Zürich · EPFL · Allen Institute for AI
RL with verifiable rewards can substantially improve LLM reasoning, yet standard GRPO-style training often treats easy, hard, and learnable questions alike through uniform sampling and weighting, leading to inefficient compute allocation. We study GRPO by tracking token log-probabilities, group-normalized advantages, and the induced token-level update weights. This reveals three recurring dynamics as training proceeds: (1) confidence inflation, (2) advantage contraction, and (3) hierarchical convergence. These findings suggest that the utility of each update depends strongly on both question difficulty and the model's current competence. Motivated by this, we propose Confidence and Difficulty-adaptive Policy Optimization (CoDaPO), which assigns each question a bounded value from rollout confidence and empirical difficulty. CoDaPO then uses this value to reweight policy updates and resample high-value learnable questions within mini-batches, thereby increasing discovery within the learnable band under a fixed compute budget. Across twelve benchmarks, CoDaPO consistently improves accuracy over existing RL methods. Our code is publicly available at https://github.com/tmlr-group/CoDaPO.
Zhanke Zhou, Xiangyu Lu, Chentao Cao +4
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney