Reinforcement learning with verifiable rewards has substantially improved mathematical reasoning. However, terminal correctness alone provides limited insight into the quality of high-level strategies, such as theorem selection and subgoal decomposition, when considered separately from their subsequent execution. This paper studies strategy utility, which is defined as the likelihood that a strategy supports a correct downstream solution under a given executor. We introduce SURE, a framework for learning and leveraging relative strategy utility. In this framework, high-level strategies are separated from their detailed reasoning. Based on the pairwise preferences constructed from strategy-conditioned rollouts and teacher-generated contrasts, a Strategy Reward Model is learned to estimate relative strategy utility. During reinforcement learning, the frozen reward model reads only the extracted strategy, whose score is combined with the correctness and format rewards in a sequence-level GRPO objective. Compared with outcome-and-format GRPO baselines, experiments show that SURE improves average pass@1 by 1.87%, 2.64%, and 2.93% across three policy backbones. Our method also achieves competitive or better accuracy than stronger reward baselines while requiring substantially lower GRPO-stage compute.
Figures & tables
Figure 1: Outcome rewards and strategy utility signals. An outcome reward evaluates the final answer. SURE extracts the strategy and uses the SRM to estimate its relative utility before forming the sequence-level training reward.
Figure 2: Overview of SURE. 1 SFT teaches the policy to separate strategy from execution. 2 SRM training constructs pairwise preferences and learns relative strategy utility. 3 Reinforcement learning combines the frozen SRM score with correctness and format rewards. We use GRPO for policy optimization in our experiments.
Backbone
Method
AIME24 ↑
AIME25 ↑
AMC23 ↑
MATH500 ↑
Avg ↑
Qwen3-4B
Base
23.00
22.67
64.00
76.76
46.61
CoT-SFT (token-matched)
36.67
24.33
76.50
77.88
53.85
SFT (strategy-guided)
42.00
29.67
80.00
81.82
58.37
GRPO from CoT-SFT
38.33
26.67
78.00
78.96
55.49
GRPO from Strategy-SFT
40.67
30.00
80.50
82.32
58.37
PTA-GRPO ( Dou et al., 2025 )
40.33
31.00
82.25
81.54
58.78
Table 1: Main results on pass@1 (%) across four mathematical reasoning benchmarks. Bold marks the best result within each backbone block. Both GRPO initialization controls use outcome and format rewards.
SFT strategy teacher
AIME24
AIME25
AMC23
MATH500
Avg
Strong teacher
42.67
31.00
84.25
83.02
60.24
Qwen3-4B
42.67
31.00
82.25
82.70
59.66
Δ
0.00
0.00
−2.00
−0.32
−0.58
Table 2: SFT strategy teacher ablation on Qwen3-4B. The weak-teacher setting changes only the teacher used for SFT strategy construction.
Backbone
Method
AIME24
AIME25
AMC23
MATH500
Avg
Qwen3-4B
SURE (ours)
42.67
31.00
84.25
83.02
60.24
GRPO + PRM
43.00
34.33
82.00
82.34
60.42
GRPO + PRM + SRM (ours)
45.67
33.33
86.00
82.70
61.93
Qwen3-8B
SURE (ours)
42.33
32.67
84.50
82.72
60.56
GRPO + PRM
44.33
30.33
82.50
82.24
59.85
GRPO + PRM + SRM (ours)
45.33
33.33
83.50
83.08
61.31
Table 3: PRM and SRM results on Qwen3 backbones with Qwen2.5-Math-PRM-7B. The table reports each reward model separately and their combination.
SRM quantile
Strategies
Accuracy (%)
80–100%
2,400
80.88
60–80%
2,400
78.76
40–60%
2,400
78.01
20–40%
2,400
77.63
0–20%
2,400
77.14
Table 4: Downstream answer accuracy by within-problem SRM score quantile with final SURE-trained Qwen3-4B as the fixed executor.
Variant
AIME24
AIME25
AMC23
MATH500
Avg
SURE (full)
42.67
31.00
84.25
83.02
60.24
−Drollout
42.00
30.33
83.75
82.68
59.69
−Dteacher
42.33
30.33
84.00
82.94
59.90
− correctness gating
40.67
30.33
83.75
82.24
59.25
− group normalization
39.33
30.67
84.50
82.54
59.26
Table 5: SRM data and reward-design ablations on Qwen3-4B.
Figure 3: Correctness by epoch on Qwen3-4B. Removing either stabilizer causes a plateau or regression in epoch 3.
Method
GRPO-stage GPU-h
AMC23
MATH500
GRPO (outcome+format)
168.06
80.50
82.32
PTA-GRPO
235.29
82.25
81.54
GRPO + PRM
318.30
82.00
82.34
SURE
174.05
84.25
83.02
Table 6: GRPO-stage cost on Qwen3-4B. SURE adds about 6 GPU-h over outcome-only GRPO and uses 45% less compute than GRPO+PRM.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Raw pairs
Retained
Rollout-induced (post confidence filter)
53,464
—
+ LLM judge consistency filter
—
48,413
Teacher-constructed
17,197
17,197
Total DSRM
—
65,610
Appendix
Table 7: SRM preference data statistics. Rollout-induced pairs are constructed from 32,000 problems (DeepMath + OpenThoughts) with K=8 strategies and M=24 rollouts each.
Hyperparameter
Stage 1 SFT
Stage 2 SRM
Stage 3 SURE
Model
Base / policy model
Qwen3-4B
Qwen2.5-7B-Instruct
Stage-1 checkpoint
Reward model
–
–
Stage-2 SRM (frozen)
Training type
Full fine-tuning
LoRA
Full fine-tuning
LoRA rank / α / target
–
64 / 128 / all-linear
–
Reward / value head
–
Scalar value head
–
Appendix
Table 8: Training hyperparameters for the three-stage SURE pipeline on the Qwen3-4B backbone. The same configuration is applied to the Qwen3-8B and Llama-3.2-3B-Instruct backbones, except for the policy model and the matching Stage 1 SFT checkpoint used to initialize Stage 3.
SFT strategy teacher
AIME24
AIME25
AMC23
MATH500
Avg
Qwen3.5-397B-A17B
42.67
31.00
84.25
83.02
60.24
Qwen3-4B
42.67
31.00
82.25
82.70
59.66
Difference
0.00
0.00
−2.00
−0.32
−0.58
Appendix
Table 9: SFT teacher strength ablation details on Qwen3-4B. Only the teacher used for Stage 1 strategy construction is changed.
SRM quantile
Epoch 1
Epoch 2
Epoch 3
80–100%
80.20
80.78
80.88
60–80%
78.70
78.80
78.76
40–60%
77.55
77.85
78.01
20–40%
77.33
77.60
77.63
0–20%
76.96
77.21
77.14
Appendix
Table 10: Downstream answer accuracy by SRM score quantile across SURE checkpoints. Candidate strategies are regenerated by the corresponding checkpoint at each epoch, and each strategy receives one downstream execution.
Executor
Pairwise Acc.
Bottom
Top
Top − Bottom
Qwen3-4B Strategy-SFT
68.1
55.5
59.0
+3.5
Qwen3-8B Strategy-SFT
64.8
56.4
59.4
+3.0
Llama-3.2-3B Strategy-SFT
64.6
17.7
20.8
+3.1
Appendix
Table 11: Cross-executor strategy utility validation with a shared strategy pool generated by the Qwen3-4B Stage-1 Strategy-SFT policy. Pairwise Acc. measures agreement with executor-specific empirical utility rankings within each problem. All values are percentages.
Figure 4: A complete (question, strategy, execution) example. The <strategy> block specifies the high-level route, while the <think> block performs the detailed derivation that produces the final answer.
Figure 5: SFT strategy generation prompt used to distill high-level strategies from teacher chain-of-thought solutions.
Figure 6: LLM-assisted preference reasonableness verification prompt. Given a problem and a labeled preference (s+,s−) , the judge decides whether the proposed preference is justified.
Figure 7: Teacher-constructed contrastive prompt. The teacher LLM generates a paired positive and negative strategy for each problem, exposing recognizable structural errors that may be underrepresented in rollout-induced data.
Example-based guidance is widely used to improve mathematical reasoning at inference time, yet its effectiveness is highly unstable across problems and models-even when the guidance is correct and problem-relevant. We show that this instability arises from a previously underexplored gap between strategy usage-whether a reasoning strategy appears in successful solutions-and strategy executability-whether the strategy remains effective when instantiated as guidance for a target model. Through a controlled analysis of paired human-written and model-generated solutions, we identify a systematic dissociation between usage and executability: human- and model-derived strategies differ in structured, domain-dependent ways, leading to complementary strengths and consistent source-dependent reversals under guidance. Building on this diagnosis, we propose Selective Strategy Retrieval (SSR), a test-time framework that explicitly models executability by selectively retrieving and combining strategies using empirical, multi-route, source-aware signals. Across multiple mathematical reasoning benchmarks, SSR yields reliable and consistent improvements over direct solving, in-context learning, and single-source guidance, improving accuracy by up to +13 points on AIME25 and +5 points on Apex for compact reasoning models. Code and benchmark are publicly available at: https://github.com/lwd17/strategy-execute-pipeline.
Weida Liang, Yiyou Sun, Shuyuan Nan +3
National University of Singapore · University of California, Berkeley
Mathematical reasoning in large language models has improved substantially with reinforcement learning using verifiable rewards, where final answers can be checked automatically and converted into reliable training signals. Most such pipelines optimize outcome correctness only, which yields sparse feedback for long, multi-step solutions and offers limited guidance on intermediate reasoning errors. Recent work therefore introduces process reward models (PRMs) to score intermediate steps and provide denser supervision. In practice, PRM scores are often imperfectly aligned with final correctness and can reward locally fluent reasoning that still ends in an incorrect answer. When optimized as absolute rewards, such signals can amplify fluent failure modes and induce reward hacking. We propose PROGRS, a framework that leverages PRMs while keeping outcome correctness dominant. PROGRS treats process rewards as relative preferences within outcome groups rather than absolute targets. We introduce outcome-conditioned centering, which shifts PRM scores of incorrect trajectories to have zero mean within each prompt group. It removes systematic bias while preserving informative rankings. PROGRS combines a frozen quantile-regression PRM with a multi-scale coherence evaluator. We integrate the resulting centered process bonus into Group Relative Policy Optimization (GRPO) without auxiliary objectives or additional trainable components. Across MATH-500, AMC, AIME, MinervaMath, and OlympiadBench, PROGRS consistently improves Pass@1 over outcome-only baselines and achieves stronger performance with fewer rollouts. These results show that outcome-conditioned centering enables safe and effective use of process rewards for mathematical reasoning.
Mohammad Rezaei, Jens Lehmann, Sahar Vahdati
Faculty of Computer Science, Technical University of Dresden, Dresden, Germany · Amazon Science · Technical University of Dresden, Dresden, Germany +1
Algorithmic mathematical reasoning requires reliable decomposition, computation, and aggregation. Final-answer rewards provide limited guidance on intermediate errors, while successful execution does not guarantee mathematical correctness. This work proposes Function-Structured Graph Reinforcement Learning (FSG-RL), connecting subproblem graphs and Python implementations with multi-verifier feedback. The policy first learns to generate code from function graphs through supervised fine-tuning (SFT). Group Relative Policy Optimization (GRPO) then optimizes the policy using answer-gated rewards and span-level credit assignment. The framework also supports teacher supervision and structured memory. A benchmark curated from Grade School Math 8K (GSM8K), MathQA, MATH, and Omni-MATH pairs public function graphs with private verification specifications. Under a unified evaluation protocol, GRPO improves final-answer accuracy from 43.25% to 67.50% and full solution success from 32.25% to 52.25% over SFT. Continued reinforcement learning (RL) with teacher supervision yields additional gains. The gains extend beyond producing correctly formatted code, supporting verifier-guided reinforcement learning for mathematical reasoning. Code is available at https://github.com/ZihanLiummyycc/FSG-RL.
Zihan Liu, Xurong Xie
Department of Computer Science, University College London, UK · Institute of Software, Chinese Academy of Sciences, China