Group relative policy optimization (GRPO) learns only from prompts whose sampled responses disagree: a group that is entirely correct or entirely incorrect has zero reward variance, contributes no gradient, and still consumes its rollouts. Prompt-selection methods reduce this waste by steering sampling toward intermediate pass rates, but they choose the target, its width, and the uncertainty model heuristically, in raw pass-rate or logit coordinates. We show that GRPO comes with a natural coordinate for pass rates: the arc length ψ=arcsinp on the Bernoulli Fisher--Rao manifold. In arc length, the expected GRPO update is uniform up to two boundary ramps; the probability of a zero-variance group is bounded by two Gaussian boundary layers of width 1/2G; pass-rate evidence has constant noise; and the gradients of the pass@k and passk objectives are Gaussians whose center and width follow from k in closed form. A prompt curriculum for GRPO is therefore a Gaussian in arc length, and choosing its center amounts to choosing the objective. We turn this observation into ARCUS, a drop-in sampler that tracks every prompt with a Kalman filter in arc length, scores prompts by an objective-matched Gaussian kernel times the predicted probability of an informative group, keeps only informative groups for the unchanged GRPO update, and paces the target toward the hardest objective whose predicted yield stays within a small slack of the best. Across six mathematical reasoning benchmarks and three backbones, ARCUS improves the average accuracy of GRPO by 2.8--2.9 points and that of dynamic sampling by 1.1--1.2 points, while generating 48--57% fewer rollouts than dynamic sampling.
Figures & tables
Figure 1: Prompts live on an arc (exact curves). (a) GRPO’s expected update as a weight on ∇p is U-shaped (blue); as a weight on ∇ψ , ψ=arcsinp , it is flat up to two ramps (red; Prop. 1 ). (b) Zero-variance probability (solid) and its Gaussian bound (dotted; Prop. 2 ). (c) After one group of n=8 , arc-length posteriors are calibrated while logit posteriors are over-confident by 20× where zero-variance prompts live (Prop. 3 ). (d) Arc-length gradients of pass@ k and pass 4 (solid) and their Gaussian forms (dotted; Prop. 4 ).
Figure 2: Arcus within one GRPO step. Beliefs live on the arc ψ=arcsinp . An objective-matched Gaussian kernel at the paced target ψt⋆ scores prompts; Gumbel-top- M candidates are rolled out and update the beliefs; only informative groups enter the unchanged GRPO update; pacing moves ψt⋆ to the hardest objective whose predicted yield stays within ε of the best.
Method
Roll.
AIME24
AIME25
AMC23
MATH500
Minerva
Olymp.
Avg.
Δ
Qwen2.5-Math-1.5B (DAPO-Math-17k, 300 updates)
Base model
–
7.4
3.9
28.6
42.3
12.1
20.8
19.2
–
GRPO
× 1.00
15.1
9.4
47.9
73.2
28.7
34.6
34.8
–
GCS
× 1.00
15.4
9.8
48.3
73.4
28.9
34.9
35.1
+0.3
AdaRFT
× 1.00
15.8
9.6
48.6
73.9
28.6
35.2
35.3
+0.5
CurES
× 1.00
16.1
10.2
49.0
73.6
29.4
35.3
35.6
+0.8
Table 1: Main results (accuracy, %, mean of three seeds; AIME mean@32, AMC23 mean@16, others mean@4). Groups: uniform, predict-then-select, evaluate-then-filter, ours. Roll. : generated rollouts relative to GRPO at equal policy updates; Δ : average gain over GRPO. Bold / underline : best/second best among trained models.
Method
Roll.
Tok.
Yield
Inf.
Hours
Avg.
(M)
(B)
(%)
(%)
GRPO
0.61
0.50
47
47
19.8
44.1
MoPPS
0.61
0.52
69
69
19.9
45.0
KGPS
0.61
0.52
75
75
20.0
45.7
GRESO
1.05
0.93
58
100
30.7
45.2
DS
1.48
1.33
41
100
41.6
45.7
Table 2: Cost on Qwen2.5-Math-7B (300 updates, 8 × H100): rollouts (M), tokens (B), yield (informative share of generated groups), Inf. (informative share of the update batch), wall-clock hours, and six-benchmark average.
Figure 3: Accuracy, efficiency, and objective control (Qwen2.5-Math-7B). (a) Six-benchmark average vs. policy updates ( ±1 s.d.). (b) The same runs vs. cumulative rollouts. (c) Fixed targets along the objective family (gray; p k = pass k ) trade pass@1 against pass@16; the paced target dominates. (d) Cost–accuracy plane; Arcus ’s curve varies ρ .
Configuration
Avg.
Δ
Inf. (%)
Arcus (full)
46.90
–
97
Geometry and belief
Gaussian over raw p
46.21
−0.69
93
Logit-space Kalman belief
45.94
−0.96
90
Discounted Beta belief
46.02
−0.88
91
Objective kernel
Table 3: Ablations on Qwen2.5-Math-7B (six-benchmark average; Inf.: informative share of the update batch).
Figure 4: How Arcus works (Qwen2.5-Math-7B). (a) Paced target pt⋆=sin2ψt⋆ for three group sizes (dotted: sharp-belief frontier). (b) Informative share of generated groups (dotted red: Arcus ’s update batch). (c) Pass-rate error of the selected prompts. (d) Median and inter-quartile band of the true pass rate of selected prompts.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Check
Result
Prop. 1 vs. the finite-group weight of Davis and Recht (2025)
max. relative difference <10−14 for G∈{4,8,16,32}
maxpωG(p) vs. the bound 1−1/G
0.928 vs. 0.935 ( G=8 ); 0.967 vs. 0.968 ( G=16 )
minp∈[0.1,0.9]ωG(p)
0.51 , 0.70 , 0.85 , 0.94 for G=4,8,16,32
Ramp slope ωG/ψ as ψ→0
2.61 vs. G−1=2.65 ( G=8 )
Prop. 2 boundary-layer bound
holds on [0,2π] for G∈{4,8,16}
Closed-form E[e−Gψ2] under a Gaussian belief
matches Monte Carlo to 5×10−4
Appendix
Table 4: Exact numerical checks of the analytic results (script code/theory_checks.py ).
Hyperparameter
Value
Framework
verl + vLLM, bf16
Policy updates
300
Prompts per update B
256
Group size G
8
Mini-batch (prompts)
64 (4 steps per update)
Optimizer
AdamW, lr 10−6 , wd 0.01
Appendix
Table 5: Training and Arcus hyperparameters.
Method
Gen.
Reward
Update
Select
Total
GRPO
13.9
0.4
5.5
< 0.01
19.8
KGPS
14.0
0.4
5.6
0.01
20.0
DS
34.9
1.2
5.5
< 0.01
41.6
Arcus ρ=0
13.9
0.4
5.7
0.01
20.0
Arcus
18.0
0.5
5.6
0.01
24.1
Appendix
Table 6: Wall-clock breakdown (hours) on Qwen2.5-Math-7B, 8 × H100. Selection is Arcus ’s belief update, scoring, and pacing, which is negligible; its extra time is generating the candidate margin.
Method
Q2.5-M-1.5B
Q2.5-M-7B
Q3-4B-Base
GRPO
34.82±0.41
44.08±0.37
46.27±0.39
MoPPS
35.73±0.38
45.03±0.35
47.27±0.36
KGPS
36.32±0.36
45.67±0.34
47.82±0.33
DS
36.50±0.33
45.70±0.30
48.00±0.31
Arcus ρ=0
36.97±0.31
46.25±0.30
48.50±0.29
Arcus
37.65±0.29
46.90±0.27
49.20±0.28
Appendix
Table 7: Six-benchmark average, mean ± s.d. over three seeds.
Figure 5: Per-benchmark accuracy over training on Qwen2.5-Math-7B.
Table 9: Fixed targets along the objective family vs. paced Arcus (Qwen2.5-Math-7B; six-benchmark averages).
G
GRPO
DS (roll.)
Arcus
final pT⋆
frontier
4
42.86
44.31 ( × 3.37)
45.52
0.43
0.40
8
44.08
45.70 ( × 2.41)
46.90
0.29
0.34
16
44.93
46.12 ( × 2.02)
47.64
0.18
0.20
Appendix
Table 10: Effect of group size on Qwen2.5-Math-7B. Frontier: closed-form sharp-belief target for ε=0.03 .
Figure 6: Sensitivity on Qwen2.5-Math-7B: (a) yield slack, (b) Gumbel temperature, (c) group size, and (d) pass@ k on AIME24.
Setting
GRPO
KGPS
DS
Arcus
DS roll.
R1-Distill-1.5B (8k)
47.86
48.95
49.12
50.21
× 2.12
Llama-3.2-3B-Inst.
21.37
22.31
22.48
23.26
× 3.46
Q2.5-Math-7B, MATH pool
43.15
44.30
44.46
45.38
× 2.78
Appendix
Table 11: Additional backbones and training pools (six-benchmark average; DS roll.: DS rollouts relative to GRPO; Arcus uses ×1.25 ).
Method
AIME24
AIME25
AMC23
MATH500
Minerva
Olymp.
Avg.
R1-Distill-1.5B (8k)
GRPO
33.4
25.1
71.0
85.2
30.4
42.1
47.87
KGPS
35.2
26.0
72.4
85.9
30.9
43.3
48.95
DS
35.0
26.4
72.9
86.4
30.8
43.2
49.12
Arcus
37.4
27.6
74.1
86.3
31.5
44.4
50.22
Llama-3.2-3B-Inst.
Appendix
Table 12: Per-benchmark results for the additional backbones and the MATH-train pool (accuracy, %; same protocol as Table 1 ).
Setting
GRPO
DS
Arcus
B=128
43.52
45.11
46.33
B=256 (default)
44.08
45.70
46.90
B=512
44.61
46.02
47.21
lr 5×10−7
43.12
44.71
45.98
lr 2×10−6
43.89
45.38
46.64
pool 4k
43.61
44.95
45.87
Appendix
Table 13: Robustness to batch size B , learning rate, and pool size on Qwen2.5-Math-7B (six-benchmark average).
Initial belief
Avg.
Warm-up yield
Extra roll.
Jeffreys moments only
46.71
47%
0.0%
Empirical Bayes (default)
46.90
51%
0.0%
Reference-model prior
46.98
62%
0.0%
Offline probe, 2 samples/prompt
47.03
66%
3.4%
Appendix
Table 14: Initial beliefs on Qwen2.5-Math-7B. Warm-up yield: informative share of generated groups during the first ⌈N/M⌉ updates.
Policy loss
Uniform
DS
Arcus
Δ vs. DS
GRPO
44.08
45.70
46.90
+1.20
Dr. GRPO
43.92
45.41
46.47
+1.06
DAPO loss
44.61
46.03
47.18
+1.15
RLOO
43.54
45.02
46.11
+1.09
Appendix
Table 15: Compatibility with other group-based losses (Qwen2.5-Math-7B, six-benchmark average).
Figure 7: Training dynamics on Qwen2.5-Math-7B: (a) mean response length, (b) token entropy, and (c) normalized innovation squared of Arcus ’s filter (1 = consistent).
Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled trajectories, extended deliberation can create more chances to realize relative gains, leading to inefficient reasoning and overthinking, and complicating the trade-off between correctness and rollout efficiency. Controlling this behavior is difficult in practice, considering (i) Length penalties are hard to calibrate because longer rollouts may reflect harder problems that require longer reasoning, penalizing tokens risks truncating useful reasoning along with redundant continuation; and (ii) supervision that directly indicates when to continue or stop is typically unavailable beyond final answer correctness. We propose Weakly Supervised GRPO (WS-GRPO), which improves rollout efficiency by converting terminal rewards into correctness-aware guidance over partial trajectories. Unlike global length penalties that are hard to calibrate, WS-GRPO trains a preference model from outcome-only correctness to produce prefix-level signals that indicate when additional continuation is beneficial. Thus, WS-GRPO supplies outcome-derived continue/stop guidance, reducing redundant deliberation while maintaining accuracy. We provide theoretical results and empirically show on reasoning benchmarks that WS-GRPO substantially reduces rollout length while remaining competitive with GRPO baselines.
Gagan Mundada, Zihan Huang, Rohan Surana +8
University of California, San Diego · University of Toronto · 3Adobe Research +2
Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop Best Practice Critic Optimization (BPCO), a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic.
Penghui Qi, Xiangxin Zhou, Wee Sun Lee
National University of Singapore · Tencent Hunyuan
Group Relative Policy Optimization (GRPO) is widely used for training reasoning models, but updating all sampled completions in each group incurs substantial cost and can reinforce verbose reasoning trajectories. In this paper, we study whether all completions provide equally useful update signals in GRPO-style reasoning RL. Our gradient-similarity analysis shows that, within the same prompt group, same-class completions often induce highly similar update directions, whereas correct-incorrect pairs provide more distinct contrastive signals. Motivated by this observation, we propose Binary Prefix Policy Optimization (BPPO), which uses the shortest correct completion and the shortest incorrect completion as a compact update unit while preserving full-group advantage normalization. BPPO further improves efficiency with adaptive completion scheduling and prefix-focused optimization; by updating only response prefixes, it avoids reinforcing redundant suffixes and encourages more concise responses. Experiments on GSM8K, MATH, and Geo3K show that BPPO achieves up to 6.08x speedup over GRPO while maintaining competitive accuracy, and reduces mean response length by approximately 30-50% without modifying the reward with an explicit length penalty.