Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning

Taojie Zhu; Dongyang Xu; Ding Zou; Sen Zhao; Qiaobo Hao; Zhiguo Yang; Yonghong He

cs.LGApr 10, 2026

Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning

Authors: Taojie Zhu, Dongyang Xu, Ding Zou, Sen Zhao, Qiaobo Hao, Zhiguo Yang, Yonghong He

Organizations: Shenzhen International Graduate School, Tsinghua University · Intelligent System Department, Zhongxing Telecom Equipment (ZTE) · Institute of Advanced Interdisciplinary Studies, Chongqing University of Posts and Telecommunications

Abstract

Post-training paradigms for Large Language Models (LLMs), primarily Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), face a fundamental dilemma: SFT provides stability (low variance) but suffers from high fitting bias, while RL enables exploration (low bias) but grapples with high gradient variance. Existing unified optimization strategies often employ naive loss weighting, overlooking the statistical conflict between these distinct gradient signals. In this paper, we provide a rigorous theoretical analysis of this bias-variance trade-off and propose DYPO (Dynamic Policy Optimization), a unified framework designed to structurally mitigate this conflict. DYPO integrates three core components: (1) a Group Alignment Loss (GAL) that leverages intrinsic group dynamics to significantly reduce RL gradient variance; (2) a Multi-Teacher Distillation mechanism that corrects SFT fitting bias via diverse reasoning paths; and (3) a Dynamic Exploitation-Exploration Gating mechanism that adaptively arbitrates between stable SFT and exploratory RL based on reward feedback. Theoretical analysis confirms that DYPO linearly reduces fitting bias and minimizes overall variance. Extensive experiments demonstrate that DYPO significantly outperforms traditional sequential pipelines, achieving an average improvement of 4.8% on complex reasoning benchmarks and 13.3% on out-of-distribution tasks. Our code is publicly available at https://github.com/Tocci-Zhu/DYPO.

Figures & tables

Panel of Figure 1 — Figure 1: The SFT-RL Dilemma: Balancing the high-bias stability of SFT against the high-variance exploration of RL.

Figure 2: The overall framework of DYPO. The system employs a Dynamic Difficulty Grading mechanism to categorize queries into Easy, Hard, and Mid tiers based on group rollout outcomes, dispatching them to the most effective optimization pathway.

Figure 3: The Contrastive Mechanism in GAL.

Model	In-Distribution						Out-of-Distribution
Model	AIME 24	AIME 25	AMC	MATH-500	Minerva	Avg	ARC-c	GPQA-D	Avg
Qwen2.5-Math-7B	11.5	4.9	31.3	43.6	7.4	19.7	18.2	11.1	14.6
Supervised Fine-Tuning
SFT	22.2	22.3	52.8	82.6	40.8	44.1	75.2	24.7	50.0
Reinforcement Learning
RL	25.1	15.3	62.0	84.4	39.3	45.2	82.3	40.4	61.4

Table 1: Overall performance on five competition-level mathematical reasoning benchmarks and two out-of-distribution benchmarks(Qwen2.5-Math-7B). Best results are bolded and second-best are underlined .

Model	In-Distribution					Out-of-Distribution
Model	AIME 24/25	AMC	MATH-500	Minerva	Avg	ARC-c	GPQA-D	Avg
Qwen3-4B-Base	9.3/5.3	40.0	66.8	27.9	29.9	49.4	14.1	31.8
SFT	33.3/27.3	62.9	73.8	43.0	48.1	73.8	28.8	51.3
RL	40.6/37.3	71.8	91.0	46.3	57.4	76.7	29.3	53.0
SFT $\to$ RL	43.3/39.3	75.4	77.4	44.9	56.1	77.4	27.8	52.6
DYPO	59.3/44.0	86.0	94.6	50.4	66.9	92.5	44.4	68.5

Table 2: Overall performance on mathematical reasoning benchmarks (Qwen3-4B-Base). Best results are bolded and second-best are underlined .

Model	AIME 24	AIME 25	AMC	GPQA-D
Qwen2.5-Math-7B	11.5	4.9	31.3	11.1
+ SFT	22.2	22.3	52.8	24.7
+ Multi-Teacher (235B / 32B / 8B)	26.6 / 26.8 / 24.5	23.3 / 23.5 / 22.0	61.4 / 61.8 / 59.5	33.3 / 32.3 / 30.8
+ RL (235B / 32B / 8B)	27.3 / 26.9 / 25.0	26.6 / 26.0 / 25.5	64.1 / 63.5 / 61.0	34.8 / 35.4 / 31.8
+ Dynamic Grading (235B / 32B / 8B)	33.3 / 31.5 / 31.8	28.7 / 27.9 / 28.1	63.6 / 62.0 / 62.4	36.4 / 35.4 / 34.3
+ GAL (DYPO) (235B / 32B / 8B)	36.0 / 35.2 / 33.5	28.7 / 28.5 / 27.8	67.0 / 66.5 / 65.2	41.4 / 41.4 / 39.4

Table 3: Ablation study under different teacher strengths.

Figure 4: Left: Offline data ratio over steps. Mid: Training reward; Right: Policy entropy.

Figure 5: Gradient Norm Comparison.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	AIME 24	AMC	GPQA-D
DYPO (Single-Teacher)	32.8	64.2	37.5
DYPO (2-Teacher)	36.0	67.0	41.4

Appendix

Table 4: Comparison between single-teacher and multi-teacher supervision in DYPO.

$\alpha$	$\beta_{\text{GAL}}$	AIME 24	MATH-500
0.2	0.1	33.5 $\pm$ 1.4	87.6 $\pm$ 0.7
0.2	1	34.8 $\pm$ 0.9	88.5 $\pm$ 0.5
0.2	2	33.9 $\pm$ 1.2	87.9 $\pm$ 0.6
0.5	0.1	35.2 $\pm$ 0.8	88.9 $\pm$ 0.4
0.5	1	36.0 $\pm$ 1.1	89.2 $\pm$ 0.3
0.5	2	35.5 $\pm$ 1.0	88.7 $\pm$ 0.5

Appendix

Table 5: Sensitivity analysis of

\alpha

and

\beta_{\text{GAL}}

. Results are reported as mean

\pm

standard deviation over multiple random seeds.

Explore similar work

STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens
Feb 17, 2026Shiqi Liu, Zeyu He, Guojian Zhan +10Reinforcement Learning RL for Language Model Reasoning
School of Vehicle and Mobility & College of AI, Tsinghua University · Didi Voyager Labs, DiDi Autonomous Driving
SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
Apr 26, 2026Alexis Limozin, Eduard Durech, Torsten Hoefler +2RL for Language Model Reasoning Policy Optimization
ETH AI Center, ETH Zürich · EPFL · Allen Institute for AI
The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning
Jun 6, 2026Zhanke Zhou, Xiangyu Lu, Chentao Cao +4RL for Language Model Reasoning Policy Optimization
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney

Feb 17, 2026J/K move · Enter open · S save

STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens

Shiqi Liu, Zeyu He, Guojian Zhan, Letian Tao+9

School of Vehicle and Mobility & College of AI, Tsinghua University · Didi Voyager Labs, DiDi Autonomous Driving

Reinforcement Learning RL for Language Model Reasoning RL Fine-Tuning Token-Level Credit Assignment

Open paper AskPDF arXiv

Reinforcement Learning (RL) has significantly improved large language model reasoning, but existing RL fine-tuning methods rely heavily on heuristic techniques such as entropy regularization and reweighting to maintain stability. In practice, they often suffer from late-stage performance collapse, leading to degraded reasoning quality and unstable training. We identify a key factor behind this instability: a small fraction of tokens, termed spurious tokens (around 0.01%), which contribute little to the reasoning outcome but receive disproportionately amplified gradient updates due to inheriting the full sequence-level reward. We present a unified framework for evaluating token-level optimization impacts across spurious risk, gradient norms, and entropy changes. Building on the analysis of token characteristics that severely disrupt optimization, we propose the Silencing Spurious Tokens (S2T) mechanism to efficiently suppress their gradient perturbations. Incorporating this mechanism into a group-based objective, we propose Spurious-Token-Aware Policy Optimization (STAPO), which promotes stable and effective large-scale model refinement. Across six mathematical reasoning benchmarks using Qwen 1.7B, 8B, and 14B base models, STAPO consistently demonstrates superior entropy stability and achieves an average performance improvement of 11.49% (

ρ_{\mathrm{T}}

=1.0, top-p=1.0) and 3.73% (

ρ_{\mathrm{T}}

=0.7, top-p=0.9) over GRPO, 20-Entropy, and JustRL.

Abstract

Explore similar work

STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens

SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning

The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning