Reinforcement learning is critical to improving large reasoning models, but its success relies heavily on verifiable rewards (RLVR), making it hard to use in open-ended domains where correctness is ambiguous and cannot be verified. Moreover, reasoning trajectories remain largely unconstrained, and optimizing solely toward the final answer can favor early exploitation over generalization. In this work, we ask whether general reasoning ability can be improved by teaching models how to think (the structure of reasoning) rather than what to produce (the outcome of reasoning), and we extend traditional RLVR to open-ended settings. We introduce Structure-Aware Reinforcement Learning (SARL), a label-free framework that constructs per-response reasoning maps from intermediate thinking steps and rewards their reasoning topology. SARL shifts supervision from destination to path, encouraging reasoning trajectories that are both locally coherent and globally efficient. On verifiable math tasks, SARL outperforms prior label-free RL baselines and even exceeds RL methods with ground truth supervision, with average gains of +9.1% under PPO and +11.6% under GRPO across four math benchmarks, with particularly large improvements on AIME25 (+35.5% with PPO and +44.7% with GRPO). On non-verifiable open-ended tasks, SARL achieves average gains of +34.6% under PPO and +30.4% under GRPO on WildBench across five task categories, outperforming prior label-free RL methods and DPO, which relies on additional preference labels. Beyond strong performance, SARL exhibits substantially lower KL divergence and higher policy entropy, indicating more stable and exploratory training dynamics. Code and data are available at Code Link.
Figures & tables
Figure 1: Overview of Structure Aware Reinforcement Learning (SARL). Left: SARL replaces outcome based supervision with a label free structure reward computed from the model’s reasoning process. Middle: construction of the SARL Reasoning Map and Structure Reward. Right: Performance improvements on verifiable and open-ended tasks.
Method
Reward Signal
Algorithm-agnostic
Open-ended
Verifiable
EMPO ( Zhang et al., 2025 )
Entropy minimization
✗
✓
✓
TTRL ( Zuo et al., 2025 )
Majority voting
✗
✗
✓
Format & Length ( Xin et al., 2025 )
Format + length
✓
✗
✓
SARL (Ours)
Reasoning Structure
✓
✓
✓
Table 1: Comparison of label-free RL methods. “Algorithm-agnostic” indicates whether the reward signal can be used with any RL algorithm (e.g., PPO) beyond group-based methods (e.g., GRPO).
Method
AIME25
AMC23
MATH500
Minerva
Avg ( Δ %)
Base
31.67
82.81
90.10
53.45
64.51
PPO
w/ Ground-Truth †
41.67
86.56
92.75
59.74
70.18 (+8.8%)
w/ SARL ( Ours )
42.92
85.00
92.53
61.08
70.38 (+9.1%)
GRPO
w/ Ground-Truth †
46.67
84.38
93.15
62.45
71.66 (+11.1%)
Table 2: Results on mathematical reasoning benchmarks. Bold marks the best score in each column and underline marks the second best. † indicates methods that train with ground-truth correctness labels (oracle); all other methods are fully label-free. SARL achieves the best overall average under both algorithms and matches or surpasses the ground-truth supervision.
Method
Creative
Planning
Math
Info
Code
WB Score ( Δ %)
Base
51.01
36.23
16.35
48.71
14.72
29.91
DPO
w/ preference labels †
51.16
37.82
17.06
48.37
14.34
30.34 (+1.4%)
PPO
w/ SARL ( Ours )
57.05
45.87
27.70
53.47
30.05
40.26 (+34.6%)
GRPO
Table 3: WildBench evaluation results for open-ended reasoning. Methods are grouped by training algorithm (DPO, PPO, GRPO). Bold marks the best score in each column and underline marks the second best. † indicates methods that train with explicit human supervision (preference labels for DPO); SARL and EMPO are fully label-free. TTRL is omitted because it requires binary correctness labels and does not generalize to open-ended tasks.
Figure 2: (a) Structure reward change across all benchmarks after SARL training. (b) Case study on a Minerva Math problem: reasoning topology before and after SARL training.
Figure 3: Training dynamics of different methods under GRPO.
Method
Avg Tokens
Δ vs. Base
Verifiable reasoning (Math).
Base
5736.6
0.0
PPO w/ SARL
5075.6
-661.0
GRPO w/ SARL
4587.8
-1148.8
Non-verifiable reasoning (Open-ended).
Base
2677.91
0.00
Table 4: Response length comparison.
Reward variant
Avg ( Δ %)
Base
64.51
SARL w/ L only
70.96 (+10.0%)
SARL w/ C only
71.97 (+11.6%)
SARL w/ C+L
72.16 (+11.9%)
Table 5: Ablation of the two structure reward terms ( C : clustering coefficient; L : average shortest path length), averaged over MATH500, AIME25, AMC23, and Minerva.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Math Setting
Open-Ended Setting
RL algorithm
PPO
PPO
Advantage estimator
GAE
GAE
Training data
AIME 1983–2024
OpenRubrics-v2
Base model
Qwen/Qwen3-4B
Qwen/Qwen3-4B
Rollout engine
vLLM
vLLM
Rollouts per prompt ( N )
8
8
Appendix
Table 6: PPO training hyperparameters for the Math and Open-Ended settings.
Hyperparameter
Math Setting
Open-Ended Setting
RL algorithm
GRPO
GRPO
Training data
AIME 1983–2024
OpenRubrics-v2
Base model
Qwen/Qwen3-4B
Qwen/Qwen3-4B
Rollout engine
vLLM
vLLM
Rollouts per prompt ( N )
8
8
Train batch size
256
256
Appendix
Table 7: GRPO training hyperparameters for the Math and Open-Ended settings.
Method
AIME25
AMC23
MATH500
Minerva
Avg
Δ vs. Base
Base
7728.0
5884.1
4182.6
5151.7
5736.6
0.0
PPO w/ Ground-Truth
7047.7
4634.7
3124.9
4021.7
4707.3
-1029.3
PPO w/ SARL
7296.6
5069.4
3451.1
4485.4
5075.6
-661.0
GRPO w/ Ground-Truth
6477.7
3956.6
2559.0
3143.8
4034.3
-1702.3
GRPO w/ EMPO
6651.5
4058.6
2702.4
3327.9
4185.1
-1551.5
GRPO w/ TTRL
6635.0
4057.8
2663.5
3258.5
4153.7
-1582.9
Appendix
Table 8: Full average response length comparison on mathematical reasoning benchmarks.
Method
Avg Tokens
Δ vs. Base
Base
2677.91
0.00
DPO
2661.46
-16.45
EMPO
2731.92
+54.01
PPO w/ SARL
2303.17
-374.74
GRPO w/ SARL
2285.87
-392.04
Appendix
Table 9: Full average response length comparison on WildBench.
Reward variant
AIME25
AMC23
MATH500
Minerva
Avg ( Δ %)
Base
31.67
82.81
90.10
53.45
64.51
SARL w/ L only
40.42
87.81
92.92
62.68
70.96 (+10.0%)
SARL w/ C only
45.83
87.19
93.00
61.86
71.97 (+11.6%)
SARL w/ C+L
45.83
87.50
93.30
61.99
72.16 (+11.9%)
Appendix
Table 10: Per-benchmark ablation of the two structure reward terms under GRPO on the math setting. C denotes the average clustering coefficient and L denotes the average shortest path length, with the reward components C(G)/2 and 1/(1+L(G)) defined in Eq. ( 6 ).
Reinforcement learning with verifiable rewards (RLVR) enhances the reasoning of large language models (LLMs), but standard RLVR often depends on human-annotated answers or carefully curated reward specifications. In machine-checkable domains, label-free alternatives such as majority voting or LLM-as-a-judge remove annotation cost but can introduce false positives that destabilize training. We introduce JURY-RL, a label-free RLVR framework that decouples answer proposal from reward disposal: votes from model rollouts propose a candidate answer, and a formal verifier determines whether that candidate can receive positive reward. Concretely, only rollouts matching the plurality-voted answer are rewarded when that answer is successfully verified in Lean. When verification is inconclusive, we invoke ResZero (Residual-Zero), a fallback reward that discards the unverified plurality proposal and redistributes a zero-mean, variance-preserving signal over the residual answers. This design maintains a stable optimization gradient without reinforcing unverifiable consensus. Across three backbone models trained on mathematical data, JURY-RL consistently outperforms other label-free baselines on mathematical reasoning benchmarks and transfers competitively to code generation and general benchmarks. It attains pass@1 performance comparable to supervised ground-truth training, with superior generalization demonstrated by higher pass@k and response diversity.
Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards. This sparse anchor reliably verifies whether a trajectory succeeds but provides no direct feedback on the reasoning path that produced it. Before success, prerequisite progress on hard problems receives no reward signal; after success, outcome rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. We introduce SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage framework that densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine correct trajectories after success. An expert-validated Step-Quality Evaluation Protocol evaluates useful-step density, error localization, and token efficiency beyond final-answer accuracy. On Qwen3-8B-Instruct trained on DAPO-Math and Big-Math, SCOPE-RL improves average accuracy by up to 11.2 pp and reduces reasoning tokens by up to 27.1% over outcome-only GRPO; the gains hold under GSPO and on Qwen3-0.6B-Instruct, indicating that reward-signal densification is complementary to policy-update-level RLVR advances. Code and data are available at https://github.com/tokencraft-lab/SCOPE-RL.
Recent advances in large language model (LLM) reasoning through reinforcement learning rely on annotated datasets for verifiable rewards, which may limit models' ability to surpass human-level performance. While self-play offers a promising alternative, prior methods yield only marginal or even negative gains on post-trained models because they generate problems that cluster around familiar concepts rather than discovering novel ones. We introduce Open-Ended Self-Improving Reasoner (OpenSIR), a self-play framework in which a single LLM alternates teacher and student roles to generate and solve novel problems without external verifiers or annotated data. Starting from a single seed problem, OpenSIR sustains open-ended exploration through diversity rewards that push the model toward unfamiliar concepts and difficulty calibration that keeps problems learnable. Across seven math benchmarks, OpenSIR consistently improves all models, averaging +3.6 points on instruction models and +3.1 on reasoning models, while recent self-play baselines yield marginal or even negative gains; starting from a single trivial seed, it also surpasses GRPO baselines trained on over 7K annotated examples. Despite training only on self-generated math, OpenSIR is the only self-play method that transfers to general reasoning, improving by at least +4.4 points on reasoning models.
Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis +3
University of Edinburgh · Imperial College London · Huawei Technologies Research & Development (UK) Limited +1