Recent methods for Reinforcement Learning with Verifiable Rewards (RLVR) use rollout diversity to improve reasoning performance or training efficiency. Despite these gains, the extent to which differences among correct-answer rollouts reflect useful alternative reasoning or unresolved errors remains poorly understood. We investigate this question through a blind paired analysis of a model trained with Group Relative Policy Optimization (GRPO), comparing the highest- and lowest-agreement correct-answer traces for the same problem. Across 126 pairs, low-agreement traces have more spurious reasoning than high-agreement traces (35% vs. 22%), including internal inconsistencies and misuse of problem conditions. Based on these findings, we propose RoAR (Rollout-Agreement Reward), which adds reasoning-level agreement to GRPO's rewards for verified-correct rollouts. RoAR computes this signal from existing on-policy rollout groups, requiring no human-written rationales, additional sampling, or separately trained process evaluator. On five mathematical reasoning benchmarks, the 1.5B GRPO-RoAR model reaches 55.5% mean pass@1 under zero-shot evaluation using 7K training examples, exceeding GRPO and GRPO-CARE by 6.0 and 5.5 percentage points. A separate blind audit of 100 problems answered correctly by all four 1.5B models finds spurious rates of 18% for GRPO-RoAR, 32% for GRPO, 27% for GRPO-CARE, and 37% for DRA-GRPO. Our findings identify reasoning validity as an important consideration when using rollout differences to guide learning. Our code is available at https://github.com/ldi-kyunghee/RoAR.
Figures & tables
Figure 1: Different traces can reach the same correct answer. Blind paired analysis of the highest- and lowest-agreement verified-correct traces for each of 126 problems from a GRPO-trained 1.5B model. Low-agreement traces have a higher spurious rate (35% vs. 22%) and a lower valid rate (31% vs. 47%).
Figure 2: Internal inconsistency shows the largest error-rate gap between agreement groups. Error-rate differences in the 126 paired comparisons are computed as lowest agreement minus highest agreement. Positive values indicate higher error rates in the lowest-agreement verified-correct traces.
Figure 3: Overview of RoAR within group-based policy optimization. RoAR measures reasoning-level agreement over the sampled rollout group and assigns the agreement reward only to verified-correct rollouts before the group-relative policy update. R and A denote the reasoning trace and final answer ; the figure uses K=4 .
Model
AIME26
MATH-500
AMC23
Minerva
OlympiadBench
Avg.
DeepSeek-R1-Distill-Qwen-1.5B
20.0
82.8
62.9
26.5
43.3
47.1
OpenRS3 ( Dang and Ngo, 2026 )
26.7
84.4
72.5
26.8
51.3
52.3
DRA-GRPO ( Chen et al., 2026a )
23.3
86.2
75.0
32.4
53.0
54.0
GRPO †
16.7
84.4
65.0
27.9
53.5
49.5
GRPO-CARE † ( Chen et al., 2026b )
20.0
82.0
65.0
29.0
54.2
50.1
GRPO w/ LLM reward † ( Corbitt et al., 2025 )
20.0
83.2
75.0
29.4
51.6
51.8
Table 1: Pass@1 comparison across mathematical reasoning benchmarks. Numbers are in %. Bold numbers indicate the highest value in each column. † denotes models trained by us. All reported AIME26 scores are evaluated by us; external-baseline scores on the other four benchmarks are taken from Dang and Ngo (2026) and Chen et al. (2026a) . Indented rows are ablations of RoAR .
Method
Pass@4
GRPO
66.2
GRPO-CARE
65.1 -1.1
DRA-GRPO
63.7 -2.5
GRPO- RoAR ( Ours )
68.3 +2.1
Table 2: Macro-average pass@4 across five benchmarks. Numbers are in %. The small values denote percentage-point differences from GRPO. DRA-GRPO results are obtained from our evaluation of its released checkpoint.
1.5B
4B
Task
GRPO
GRPO- RoAR
Δ
GRPO
GRPO- RoAR
Δ
AIME26
22.2±6.9
24.4±3.8
+2.2
61.1±5.1
66.7±5.8
+5.6
MATH-500
83.8±0.7
83.9±1.0
+0.1
95.7±0.4
96.4±0.4
+0.7
AMC23
70.8±8.0
74.2±5.2
+3.3
96.7±2.9
95.8±5.2
−0.8
Minerva
29.4±1.3
30.5±2.0
+1.1
46.2±0.8
46.4±0.4
+0.2
OlympiadBench
51.9±1.4
53.8±1.3
+1.9
73.8±0.6
75.4±0.6
+1.5
Table 3: Pass@1 (%), reported as mean ± sample SD over three training seeds for DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-4B. Within each model size, both methods use the same 7K examples, training budget, and evaluation protocol. Average is the five-benchmark macro-average; its SD is computed across the per-seed macro-averages. Δ is the GRPO- RoAR improvement in percentage points, computed from unrounded scores.
Figure 4: Reasoning validity after training. Blind human audit on 100 shared problems answered correctly by GRPO, GRPO-CARE, DRA-GRPO, and GRPO- RoAR (1.5B). Bars show the percentages of valid, self-corrected, and spurious traces.
Model
Strong
Weak Pass@1
Weak Pass@4
GRPO
48.7
21.2
47.9
GRPO-CARE
52.5
14.8
43.0
DRA-GRPO
48.4
20.9
46.9
GRPO- RoAR
48.8
28.1
52.4
Table 4: Answer-consensus diagnostic over four sampled rollouts. Strong denotes the percentage of problems where at least three rollouts produce the same extracted final answer. Weak Pass@1 and Weak Pass@4 are computed only on the weak-consensus subset. Numbers are in % and macro-averaged across five benchmarks. DRA-GRPO results are obtained from our evaluation of its released checkpoint.
Figure 5: Ablation of the mixing coefficient α in the pairwise similarity score (Eq. 1 ). We report mean pass@1 across the five benchmarks. Numbers are in %. α=0 denotes lexical-only agreement, and α=1 denotes semantic-only agreement.
Figure 6: Sensitivity to the rollout-agreement reward weight w\textscRoAR . We report mean pass@1 across the five benchmarks. Numbers are in %.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Artifact
Use
Repository
License / terms source
# Examples
Open-RS dataset
Train
knoveleng/open-rs
MIT
7,000
Open-RS codebase
Training pipeline
knoveleng/open-rs
MIT (official repo)
–
AIME26
Evaluation
math-ai/aime26
Apache-2.0
30
MATH-500
Evaluation
HuggingFaceH4/MATH-500
MIT (source repo)
500
AMC23
Evaluation
knoveleng/AMC-23
not specified
40
Minerva
Evaluation
knoveleng/Minerva-Math
MIT (equivalent mirror)
272
Appendix
Table 5: External data, code, and model artifacts used in this work. The table reports each artifact’s role in our experiments, repository, license or terms information available from the relevant repository or source, and dataset size when applicable.
Parameter
Value
General Settings
Precision
bf16
vLLM
enabled, colocate mode
vLLM memory ratio
0.1
vLLM max length
4608
Evaluation during training
false
Appendix
Table 6: Hyperparameter setup for GRPO training. The multi-seed experiments use three training seeds. For RoAR , the default configuration uses the full rollout group as the agreement reference and assigns the rollout-agreement reward only to verified-correct rollouts.
Method
Runtime
GPU-hours
Step time
Peak memory
Reward comp.
RoAR comp.
(h)
(s)
(GiB/GPU)
(ms)
(ms)
GRPO
11.34
90.75
80.67
178.3
27.3
–
GRPO- RoAR
11.27
90.14
81.35
178.3
47.7
23.6
Appendix
Table 7: Resource-overhead check for outcome-only GRPO and GRPO- RoAR under matched training runs. GPU-hours are computed from wall-clock time and 8 GPUs. Reward-computation times are averages from logged profiler metrics.
Baseline
Metric
Δ
95% CI
GRPO
Pass@1
+6.0
[1.6, 10.6]
GRPO-CARE
Pass@1
+5.5
[0.9, 10.0]
GRPO
Pass@4
+2.1
[-1.2, 5.6]
GRPO-CARE
Pass@4
+3.2
[0.7, 6.2]
DRA-GRPO
Pass@4
+4.7
[1.5, 8.2]
Appendix
Table 8: Paired bootstrap confidence intervals over evaluation problems. Deltas compare GRPO- RoAR against each baseline and are macro-average percentage-point differences across the five evaluation benchmarks.
1.5B
4B
Training seed
GRPO
GRPO- RoAR
Δ
GRPO
GRPO- RoAR
Δ
0
53.21
53.31
+0.10
75.41
74.35
−1.06
7
52.19
51.27
−0.92
73.16
75.93
+2.77
2025
49.50
55.52
+6.02
75.52
78.15
+2.64
Appendix
Table 9: Five-benchmark mean pass@1 (%) for individual training runs. Each score is the macro-average over AIME26, MATH-500, AMC23, Minerva, and OlympiadBench. Δ is the GRPO- RoAR improvement in percentage points, computed from unrounded scores. Bold indicates the higher score within each model size and training seed.
α
AIME26
MATH-500
AMC23
Minerva
OlympiadBench
Avg.
0.00 (Lexical only)
23.3
84.0
70.0
34.2
52.7
52.8
0.25
26.7
85.2
77.5
30.9
50.7
54.2
0.50
26.7
84.8
80.0
30.9
55.3
55.5
0.75
30.0
85.2
72.5
30.9
52.9
54.3
1.00 (Semantic only)
23.3
83.6
70.0
32.7
50.5
52.0
Appendix
Table 10: Per-benchmark ablation of the semantic–lexical mixing coefficient α in the mixed similarity score. Numbers are in %. α=0 denotes lexical-only agreement, and α=1 denotes semantic-only agreement.
w\textscRoAR
AIME26
MATH-500
AMC23
Minerva
OlympiadBench
Avg.
2.0
30.0
82.8
72.5
32.7
50.4
53.7
1.5
30.0
82.8
80.0
30.9
54.2
55.6
1.0
26.7
84.8
80.0
30.9
55.3
55.5
0.5
20.0
83.2
72.5
30.9
52.6
51.8
Appendix
Table 11: Per-benchmark ablation of the rollout-agreement reward weight w\textscRoAR . Numbers are in %. The shaded row denotes the default setting used in the main experiments.
Dataset
Model
Strong Cons.
Weak Pass@1
Weak Pass@4
AIME26
GRPO
13.3
3.8
26.9
GRPO-CARE
23.3
8.7
17.4
DRA-GRPO
13.3
11.5
15.4
GRPO- RoAR
20.0
16.7
33.3
MATH-500
GRPO
82.6
32.2
66.7
GRPO-CARE
82.0
23.3
63.3
Appendix
Table 12: Per-benchmark consensus-conditioned evaluation over four sampled rollouts. Strong consensus denotes the fraction of problems where at least three rollouts produce the same extracted final answer. Weak Pass@1 and Weak Pass@4 are computed only over the remaining weak-consensus problems. Numbers are in %. DRA-GRPO is evaluated in our environment using its released checkpoint.
Model
Valid ( ↑ )
Self-corrected ( − )
Spurious ( ↓ )
GRPO
49
15
36
GRPO- RoAR
56
15
29
Appendix
Table 13: Supplementary blind human evaluation of the Qwen3-4B models trained under matched settings on 100 sampled problems where both models produce a correct first rollout. Entries are trace counts; arrows indicate the preferred direction.
Model
Valid ( ↑ )
Self-corr. ( − )
Spurious ( ↓ )
GRPO
45
13
42
GRPO- RoAR
47
18
35
GRPO-CARE
48
15
37
DRA-GRPO
42
18
40
Appendix
Table 14: GPT-6.1 Sol (low reasoning effort) reasoning-validity labels for all four 1.5B models on the same 100 shared correct-answer problems. Entries are percentages; arrows indicate the preferred direction.
Human \ LLM
Valid
Self-corr.
Spurious
Valid
177
12
23
Self-corrected
1
41
32
Spurious
4
11
99
Appendix
Table 15: Human–LLM label agreement on the shared correct-answer traces. Rows are human labels; columns are GPT-6.1 Sol labels.
Figure 7: Qualitative case study comparing reasoning validity across methods. All responses reach the same correct final answer, but exhibit different reasoning traces: spurious, valid, and self-corrected.
Higher agreement: 0.788 (Spurious)
Lower agreement: 0.671 (Valid)
“If the world price is $1, and the country’s supply is Qs = P, then at P = 1, Qs = 1. The demand is Qd = 4 - 1 = 3. So, the country will export the difference, which is 2 bushels.”
“The world price is 1. So the demand in Moldavia is 3 and supply is 1, so 2 bushels will be imported. The price is still 1 because the world price is lower than the local equilibrium.”
Verified final answer: \1$
Verified final answer: \1$
Appendix
Table 16: A paired counterexample from the 126-pair GRPO diagnostic. Excerpts are taken from the reasoning traces. Agreement scores use the other verified-correct rollouts in the same 32-rollout group, following Section 2 ; labels are from the original blind annotation. The higher-agreement trace retains an incorrect trade direction despite reaching the correct final price.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Reinforcement learning with verifiable rewards (RLVR) has become a powerful paradigm for improving the reasoning abilities of large language models, yet group-based methods lose their learning signal when every sampled response receives zero reward. We propose Group Contrastive Policy Optimization (GCPO), a hybrid objective that preserves standard on-policy optimization for informative rollout groups and activates a proximal contrastive objective only when exploration completely fails. The auxiliary objective pairs an offline golden reasoning trajectory with the current policy's failed responses, thereby providing a reliable positive reference and prompt-specific hard negatives without treating the golden trajectory as an on-policy rollout. Its old-policy token probabilities are evaluated through teacher forcing and used solely to constrain the update, rather than as an importance-sampling correction for the teacher distribution. A first-order analysis shows that the resulting update increases a length-normalized likelihood margin between reliable reasoning and observed failures. Experiments across two model scales and 10 core mathematical reasoning benchmarks demonstrate that GCPO improves training efficiency and final reasoning performance over competitive supervised and RLVR baselines.
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Language Models (LLMs) by leveraging direct outcome verification instead of learned reward models. Building on this paradigm, Group Relative Policy Optimization (GRPO) eliminates the need for critic models but suffers from indiscriminate credit assignment for intermediate steps, which limits its ability to identify effective reasoning strategies and incurs overthinking. In this work, we introduce a model-free and verifiable process supervision via probing the model's belief in the correct answer throughout its reasoning trajectory. By segmenting the generation into discrete steps and tracking the conditional probability of the correct answer appended at each segment boundary, we efficiently compute interpretable segment-wise progress measurements to refine GRPO's trajectory-level feedback. This approach enables more targeted and sample-efficient policy updates, while avoiding the need for intermediate supervision derived from costly Monte Carlo rollouts or auxiliary models. Experiments on mathematical and general-domain benchmarks show consistent gains over GRPO across diverse models: up to 2.6-point accuracy improvements and 13.7% reasoning-length reductions on math tasks, and up to 2.4 points and 4% on general-domain tasks, demonstrating strong generalization.
Jingyi Wang, Lei Zhu, Tengjin Weng +8
Tsinghua University · Huawei Noah’s Ark Lab · Shenzhen University
Reinforcement learning with verifiable rewards (RLVR) has emerged as a highly effective framework for improving LLM reasoning, with methods such as GRPO among its most successful instantiations. However, GRPO relies on repeated generation of long chain-of-thought rollouts. Training time scales with the number of rollouts, a large fraction of which are uninformative. Thus, GRPO is computationally expensive and unstable. To mitigate this, existing approaches either generate a larger pool of rollouts and filter the most informative prompts, or leverage historical signals for filtering at later stages of training. These strategies offer modest performance gains, but slow down the overall process. To address this, we propose VarIance Guided Online Rollout allocation (VIGOR) which instead of allocating a fixed rollout budget per example, begins with a small number of rollouts for all examples in a batch and iteratively allocates additional rollouts to those with the highest group reward variance until a fixed total rollout budget is reached. Theoretically, we show that under RLVR, reward variance controls the gradient magnitude, and derive VIGOR's closed-form speedup ratio over GRPO, which grows with refinement rounds under Pareto-distributed reward variance. Experiments on mathematical reasoning and coding tasks show that VIGOR reaches target accuracy with up to 2.3× fewer rollouts on math, reaches GRPO's final coding full pass rate with 1.49× fewer rollouts, and improves the coding average test pass rate by 3.4 points.