Reasoning-oriented language model training increasingly relies on multiple on-policy rollouts, but standard Group Relative Policy Optimization (GRPO) uses a fixed group size, wasting compute on easy prompts and under-exploring hard ones. Existing adaptive-budget methods alleviate this inefficiency, yet their stopping rules are heuristic and do not provide a finite-sample calibration guarantee. We propose CGRPO, which replaces the fixed rollout budget of standard GPRO with calibrated conformal thresholds over task-specific nonconformity scores, combining Adaptive Prediction Set (APS)-style calibration for closed-form reasoning, a first-success score for execution-based code, automatic miscoverage selection, and periodic re-calibration. We prove finite-sample coverage for the resulting stopping rule and show that, under monotone policy improvement, recalibration induces a non-increasing expected training budget. Across six benchmarks spanning mathematics, graduate-level science, and code generation, CGRPO reduces mean training rollouts by 49-70% relative to fixed-budget GRPO while retaining strong fixed-budget accuracy, achieves the strongest pass@1 among evaluated static and dynamic baselines on different datasets, and transfers across model families with up to 82% rollout savings.
Figures & tables
Figure 1: Overview of C-GRPO workflow.
Figure 2: Training dynamics (GSM8K, Qwen2.5-7B). C-GRPO sustains the highest reward and lowest KL divergence throughout training. Periodic dips align with Δcal=50 re-calibration events; the policy recovers to a higher reward peak within ∼5 steps each time.
Dataset
Accuracy (%)
Training Compute
Eval ( k=16 , 1 × H100)
p@1
p@8
p@16
p@32
kˉtrain
Rollout sav.
GPU-h
VRAM (GB)
Tok/s
Mathematical reasoning
GSM8K
57.2
70.5
73.8
75.0
10.4
68%
4.04
15.6
450
MATH-500
52.5
66.8
71.2
71.5
15.9
50%
1.61
21.6
1,880
MATH-Algebra
61.2
69.5
72.0
74.0
12.2
62%
8.24
16.6
442
Graduate-level science
Table 1: C-GRPO on six benchmarks (Qwen2.5-7B-Instruct, step 300). Fixed-budget accuracy p@ k , mean training rollouts kˉtrain , and rollout savings 1−kˉtrain/32 vs. GRPO G=32 (training rollouts only; calibration cost in Table 10 ).
Model
Algorithm
Pass@1 (%)
Δ pass@1
kˉtrain
Type
Training Efficiency (1 × H100)
GSM8K
HumanEval
MBPP
MATH-500
GPU-h
VRAM (GB)
Tok/s
Qwen2.5 7B-Instruct
GRPO ( Shao et al., 2024 )
54.8
33.3
56.0
48.5
—
32.0
static
11.7
26.8
3,180
GDRO ( Panaganti et al., 2026 )
56.8
34.5
58.2
51.0
+2.0
32.0 ∗
dynamic
11.7
26.8
3,180
AERO ( Zhang et al., 2026 )
56.1
34.0
57.5
50.2
+1.3
16.6
dynamic
6.1
22.4
4,270
GRESO ( Zheng et al., 2025 )
54.7
33.1
55.8
48.3
−0.1
15.6
dynamic
5.7
21.9
4,410
Reinforce-Ada ( Xiong et al., 2025 )
55.4
33.8
56.5
49.5
+0.6
14.8
dynamic
5.4
21.4
4,520
Table 2: Pass@1 (%) and training compute (Qwen2.5-7B-Instruct, step 300). kˉtrain : mean rollouts/prompt (lower is better). Δ : pass@1 gap vs. GRPO on GSM8K. GPU-h / VRAM / Tok/s: total training wall-clock (including all calibration generation for C-GRPO), peak GPU memory, and effective rollout generation throughput on 1 × H100 (all methods, 300 steps). ∗ GDRO uses a compute-neutral global budget ( kˉ=32 ).
Dataset
Model
δ(0)
p@1
p@8
p@16
p@32
kˉtrain
Savings
GSM8K
Qwen2.5-7B-Inst.
0.12
57.2
70.5
73.8
75.0
10.4
68%
Llama-3.1-8B-Inst.
0.05
72.5
85.0
91.0
91.5
5.7
82%
Gemma-3-4B-IT
0.18
48.5
62.0
65.5
66.0
14.2
56%
MBPP
Qwen2.5-7B-Inst.
0.10
60.5
78.5
83.0
83.5
10.0
69%
Llama-3.1-8B-Inst.
0.08
65.5
81.0
86.5
87.0
8.5
73%
Gemma-3-4B-IT
0.15
50.2
65.0
70.0
71.0
12.5
61%
Table 3: Cross-model generalisation at step 300 on held-out test sets. δ(0) is auto-selected per model–dataset pair without manual tuning.
Figure 3: Rollout efficiency during training (GSM8K, Qwen2.5-7B, 300 steps). (a) Mean kˉ per step falls from Kmax=32 to ≈10 ; drops at each Δcal=50 checkpoint mark auto- δ tightening. (b) Cumulative rollouts: 59.8% fewer than the fixed- k=32 ceiling (dashed; shaded: 5 738 rollouts saved).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
δ(0)
Target
Empirical
Slack
GSM8K
0.12
84.0%
83.8%
−0.2 pp
MATH-500
0.25
74.0%
74.7%
+0.7 pp
GPQA-Diamond
0.48
44.0%
43.9%
−0.1 pp
MATH-Algebra
0.21
80.0%
80.0%
±0 pp
HumanEval
0.90
90.0%
90.9%
+0.9 pp
MBPP
0.23
76.0%
76.3%
+0.3 pp
Appendix
Table 4: Coverage validation across six benchmarks (Qwen2.5-7B, held-out test, step 300). δ(0) : auto-selected initial miscoverage. Empirical coverage matches or exceeds the nominal target on all benchmarks within the finite-sample tolerance ±1/(ncal+1)≈0.5% .
Figure 4: Per- (k,t) coverage scatter on GSM8K (Qwen2.5-7B). Each point is one (k,step) pair. Every point lies on or above the diagonal (empirical ≥ target), jointly validating Theorem 2 for all k∈K and all training steps simultaneously. The densest cluster at 80–90% corresponds to k=8 – 16 during mid-training; points move rightward as auto- δ tightens targets with policy improvement.
(A) Auto- δ initialisation / GSM8K
(B) Re-calibration freq. / GSM8K
(C) Budget grid / MATH-500
Strategy
p@1
p@16
kˉ
Cov.
Triv.
Δ
p@1
p@16
kˉ
OH
K
p@1
p@16
kˉ
#
Fixed .05
54.5
70.2
29.5
94.2
∼200
25
57.3
73.9
10.3
+12
{2,8,32}
51.0
69.5
18.4
3
Fixed .20
56.0
72.5
18.4
81.5
∼30
50
57.2
73.8
10.4
base
{2,4,8,16,32}
52.5
71.2
15.9
5
Auto
57.2
73.8
10.4
83.8
0
100
55.8
72.1
13.5
−6
{1,2,4,8,16,32}
52.3
70.8
16.2
6
∞
54.5
70.0
18.2
−12
Appendix
Table 5: Ablation studies for C-GRPO. Bold = default configuration. “Triv.” = training steps where all thresholds are inactive (curriculum reduces to vanilla GRPO). “OH” = overhead steps per checkpoint relative to Δcal=50 baseline. “#” = number of distinct budget levels. All rows use the original calibration procedure.
Test Domain
Shift
dTV
Theo. LB
Empirical
Margin
GSM8K (in-dist.)
none
0.00
87.5%
87.9%
+0.4 pp
MATH-Algebra
mild
0.06
81.5%
83.6%
+2.1 pp
MATH-500
moderate
0.15
72.5%
76.8%
+4.3 pp
GPQA-Diamond
severe
0.29
58.5%
70.4%
+11.9 pp
Appendix
Table 6: Coverage under distribution shift (Qwen2.5-7B, trained and calibrated on GSM8K, step 300, δ(0)=0.12 , nominal target 88% ). dTV : TV distance between GSM8K calibration and test distribution. Theoretical bound: 1−δ−dTV−1/(ncal+1) . Empirical coverage consistently exceeds the theoretical lower bound.
Benchmark
Method
G / kˉ
p@1
p@8
p@16
p@32
GPU-h
GSM8K
GRPO
2
48.2
64.0
67.5
68.0
1.8
GRPO
4
51.0
68.2
70.1
71.2
2.4
GRPO
8
52.8
70.1
72.0
73.1
3.6
GRPO
10 (matched)
53.1
70.5
72.4
73.5
4.2
GRPO
16
54.2
71.8
73.2
74.1
6.2
GRPO
32
54.8
72.0
73.5
74.6
11.7
Appendix
Table 7: Full static- G sweep (Qwen2.5-7B-Instruct, 300 steps, 1 × H100). Every static row is trained at fixed G .
Table 9: Stronger base, full-parameter fine-tuning (GSM8K, Qwen2.5-Math-7B-Instruct, base p@1 =82.4 , 300 steps; mean ± std over three seeds).
Phase (hours)
GRPO G=32
GRPO G=10
C-GRPO ( kˉ=10.4 )
Training generation
8.10
2.55
2.80
Forward + backward
3.60
1.65
0.75
Re-calibration (6 events)
—
—
0.25
Micro-batch scheduling
—
—
0.008
Total wall-clock
11.70
4.20
3.81
Speed-up vs. G=32
1.00 ×
2.79 ×
3.07 ×
Appendix
Table 10: Wall-clock breakdown (GSM8K, Qwen2.5-7B-Instruct, 300 steps, 1 × H100; original calibration procedure). C-GRPO includes all calibration generation, batched as large offline batches; with the released code’s default (training-batch-size) calibration the re-calibration share is larger (36–47% of training time in a 100-step check).
Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled trajectories, extended deliberation can create more chances to realize relative gains, leading to inefficient reasoning and overthinking, and complicating the trade-off between correctness and rollout efficiency. Controlling this behavior is difficult in practice, considering (i) Length penalties are hard to calibrate because longer rollouts may reflect harder problems that require longer reasoning, penalizing tokens risks truncating useful reasoning along with redundant continuation; and (ii) supervision that directly indicates when to continue or stop is typically unavailable beyond final answer correctness. We propose Weakly Supervised GRPO (WS-GRPO), which improves rollout efficiency by converting terminal rewards into correctness-aware guidance over partial trajectories. Unlike global length penalties that are hard to calibrate, WS-GRPO trains a preference model from outcome-only correctness to produce prefix-level signals that indicate when additional continuation is beneficial. Thus, WS-GRPO supplies outcome-derived continue/stop guidance, reducing redundant deliberation while maintaining accuracy. We provide theoretical results and empirically show on reasoning benchmarks that WS-GRPO substantially reduces rollout length while remaining competitive with GRPO baselines.
Gagan Mundada, Zihan Huang, Rohan Surana +8
University of California, San Diego · University of Toronto · 3Adobe Research +2
Group Relative Policy Optimization (GRPO) is one of the most widely adopted RLVR algorithms for post-training large language models on reasoning tasks. We first show that GRPO admits an equivalent discriminative reformulation, in which policy optimization maximizes the expected score gap between verified positive and negative rollouts. This reformulation reveals two objective-level limitations: likelihood-misaligned surrogate scores, in which clipped ratio-based scores are optimized rather than the sequence likelihoods that govern generation, and score-insensitive credit assignment, in which rollout-level credit does not reflect the current score gaps between positive and negative rollouts. To address these limitations, we propose ConSPO, a Contrastive Sequence-level Policy Optimization method that uses length-normalized sequence log-probabilities as rollout scores and contrasts verified positive rollouts against negative distractors within the same group. ConSPO optimizes a group-wise InfoNCE-style objective to adaptively strengthen updates for poorly separated positives and high-scoring negatives, together with a curriculum-scheduled margin that preserves separation pressure as training progresses. Experiments across diverse settings show that ConSPO outperforms strong baselines on challenging reasoning benchmarks. Code will be released upon paper acceptance.
Feng Zhang, Xinhong Ma, Ziqiang Dong +5
Beijing Institute of Technology · Qwen Business Unit of Alibaba · The Chinese University of Hong Kong, Shenzhen +1
Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models. While it computes advantages based on group mean, GRPO treats each output as an independent sample during the optimization and overlooks a vital structural signal: the natural contrast between correct and incorrect solutions within the same group, thus ignoring the rich, comparative data that could be leveraged by explicitly pitting successful reasoning traces against failed ones. To capitalize on this, we present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples. Building on this insight, we propose Bilateral Context Conditioning (BICC), a mechanism that allows the model to cross-reference successful and failed reasoning traces during the optimization, enabling a direct information flow across samples. We further introduce Reward-Confidence Correction (RCC) to stabilize training by dynamically adjusts the advantage baseline in GRPO using reward-confidence covariance derived from the first-order approximation of the variance-minimizing estimator. Both mechanisms require no additional sampling or auxiliary models and can be adapted to all GRPO variants. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements across comprehensive models and algorithms. Code is available at https://github.com/Skylanding/BiCC.
Yu Li, Tian Lan, Zhengling Qi
Department of Electrical and Computer Engineering, George Washington University · School of Business, George Washington University