Reinforcement learning with verifiable rewards has substantially improved mathematical reasoning. However, terminal correctness alone provides limited insight into the quality of high-level strategies, such as theorem selection and subgoal decomposition, when considered separately from their subsequent execution. This paper studies strategy utility, which is defined as the likelihood that a strategy supports a correct downstream solution under a given executor. We introduce SURE, a framework for learning and leveraging relative strategy utility. In this framework, high-level strategies are separated from their detailed reasoning. Based on the pairwise preferences constructed from strategy-conditioned rollouts and teacher-generated contrasts, a Strategy Reward Model is learned to estimate relative strategy utility. During reinforcement learning, the frozen reward model reads only the extracted strategy, whose score is combined with the correctness and format rewards in a sequence-level GRPO objective. Compared with outcome-and-format GRPO baselines, experiments show that SURE improves average pass@1 by 1.87%, 2.64%, and 2.93% across three policy backbones. Our method also achieves competitive or better accuracy than stronger reward baselines while requiring substantially lower GRPO-stage compute.
Figures & tables
Figure 1: Outcome rewards and strategy utility signals. An outcome reward evaluates the final answer. SURE extracts the strategy and uses the SRM to estimate its relative utility before forming the sequence-level training reward.
Figure 2: Overview of SURE. 1 SFT teaches the policy to separate strategy from execution. 2 SRM training constructs pairwise preferences and learns relative strategy utility. 3 Reinforcement learning combines the frozen SRM score with correctness and format rewards. We use GRPO for policy optimization in our experiments.
Backbone
Method
AIME24 ↑
AIME25 ↑
AMC23 ↑
MATH500 ↑
Avg ↑
Qwen3-4B
Base
23.00
22.67
64.00
76.76
46.61
CoT-SFT (token-matched)
36.67
24.33
76.50
77.88
53.85
SFT (strategy-guided)
42.00
29.67
80.00
81.82
58.37
GRPO from CoT-SFT
38.33
26.67
78.00
78.96
55.49
GRPO from Strategy-SFT
40.67
30.00
80.50
82.32
58.37
PTA-GRPO ( Dou et al., 2025 )
40.33
31.00
82.25
81.54
58.78
Table 1: Main results on pass@1 (%) across four mathematical reasoning benchmarks. Bold marks the best result within each backbone block. Both GRPO initialization controls use outcome and format rewards.
SFT strategy teacher
AIME24
AIME25
AMC23
MATH500
Avg
Strong teacher
42.67
31.00
84.25
83.02
60.24
Qwen3-4B
42.67
31.00
82.25
82.70
59.66
Δ
0.00
0.00
−2.00
−0.32
−0.58
Table 2: SFT strategy teacher ablation on Qwen3-4B. The weak-teacher setting changes only the teacher used for SFT strategy construction.
Backbone
Method
AIME24
AIME25
AMC23
MATH500
Avg
Qwen3-4B
SURE (ours)
42.67
31.00
84.25
83.02
60.24
GRPO + PRM
43.00
34.33
82.00
82.34
60.42
GRPO + PRM + SRM (ours)
45.67
33.33
86.00
82.70
61.93
Qwen3-8B
SURE (ours)
42.33
32.67
84.50
82.72
60.56
GRPO + PRM
44.33
30.33
82.50
82.24
59.85
GRPO + PRM + SRM (ours)
45.33
33.33
83.50
83.08
61.31
Table 3: PRM and SRM results on Qwen3 backbones with Qwen2.5-Math-PRM-7B. The table reports each reward model separately and their combination.
SRM quantile
Strategies
Accuracy (%)
80–100%
2,400
80.88
60–80%
2,400
78.76
40–60%
2,400
78.01
20–40%
2,400
77.63
0–20%
2,400
77.14
Table 4: Downstream answer accuracy by within-problem SRM score quantile with final SURE-trained Qwen3-4B as the fixed executor.
Variant
AIME24
AIME25
AMC23
MATH500
Avg
SURE (full)
42.67
31.00
84.25
83.02
60.24
−Drollout
42.00
30.33
83.75
82.68
59.69
−Dteacher
42.33
30.33
84.00
82.94
59.90
− correctness gating
40.67
30.33
83.75
82.24
59.25
− group normalization
39.33
30.67
84.50
82.54
59.26
Table 5: SRM data and reward-design ablations on Qwen3-4B.
Figure 3: Correctness by epoch on Qwen3-4B. Removing either stabilizer causes a plateau or regression in epoch 3.
Method
GRPO-stage GPU-h
AMC23
MATH500
GRPO (outcome+format)
168.06
80.50
82.32
PTA-GRPO
235.29
82.25
81.54
GRPO + PRM
318.30
82.00
82.34
SURE
174.05
84.25
83.02
Table 6: GRPO-stage cost on Qwen3-4B. SURE adds about 6 GPU-h over outcome-only GRPO and uses 45% less compute than GRPO+PRM.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Raw pairs
Retained
Rollout-induced (post confidence filter)
53,464
—
+ LLM judge consistency filter
—
48,413
Teacher-constructed
17,197
17,197
Total DSRM
—
65,610
Appendix
Table 7: SRM preference data statistics. Rollout-induced pairs are constructed from 32,000 problems (DeepMath + OpenThoughts) with K=8 strategies and M=24 rollouts each.
Hyperparameter
Stage 1 SFT
Stage 2 SRM
Stage 3 SURE
Model
Base / policy model
Qwen3-4B
Qwen2.5-7B-Instruct
Stage-1 checkpoint
Reward model
–
–
Stage-2 SRM (frozen)
Training type
Full fine-tuning
LoRA
Full fine-tuning
LoRA rank / α / target
–
64 / 128 / all-linear
–
Reward / value head
–
Scalar value head
–
Appendix
Table 8: Training hyperparameters for the three-stage SURE pipeline on the Qwen3-4B backbone. The same configuration is applied to the Qwen3-8B and Llama-3.2-3B-Instruct backbones, except for the policy model and the matching Stage 1 SFT checkpoint used to initialize Stage 3.
SFT strategy teacher
AIME24
AIME25
AMC23
MATH500
Avg
Qwen3.5-397B-A17B
42.67
31.00
84.25
83.02
60.24
Qwen3-4B
42.67
31.00
82.25
82.70
59.66
Difference
0.00
0.00
−2.00
−0.32
−0.58
Appendix
Table 9: SFT teacher strength ablation details on Qwen3-4B. Only the teacher used for Stage 1 strategy construction is changed.
SRM quantile
Epoch 1
Epoch 2
Epoch 3
80–100%
80.20
80.78
80.88
60–80%
78.70
78.80
78.76
40–60%
77.55
77.85
78.01
20–40%
77.33
77.60
77.63
0–20%
76.96
77.21
77.14
Appendix
Table 10: Downstream answer accuracy by SRM score quantile across SURE checkpoints. Candidate strategies are regenerated by the corresponding checkpoint at each epoch, and each strategy receives one downstream execution.
Executor
Pairwise Acc.
Bottom
Top
Top − Bottom
Qwen3-4B Strategy-SFT
68.1
55.5
59.0
+3.5
Qwen3-8B Strategy-SFT
64.8
56.4
59.4
+3.0
Llama-3.2-3B Strategy-SFT
64.6
17.7
20.8
+3.1
Appendix
Table 11: Cross-executor strategy utility validation with a shared strategy pool generated by the Qwen3-4B Stage-1 Strategy-SFT policy. Pairwise Acc. measures agreement with executor-specific empirical utility rankings within each problem. All values are percentages.
Figure 4: A complete (question, strategy, execution) example. The <strategy> block specifies the high-level route, while the <think> block performs the detailed derivation that produces the final answer.
Figure 5: SFT strategy generation prompt used to distill high-level strategies from teacher chain-of-thought solutions.
Figure 6: LLM-assisted preference reasonableness verification prompt. Given a problem and a labeled preference (s+,s−) , the judge decides whether the proposed preference is justified.
Figure 7: Teacher-constructed contrastive prompt. The teacher LLM generates a paired positive and negative strategy for each problem, exposing recognizable structural errors that may be underrepresented in rollout-induced data.