Reasoning-oriented language model training increasingly relies on multiple on-policy rollouts, but standard Group Relative Policy Optimization (GRPO) uses a fixed group size, wasting compute on easy prompts and under-exploring hard ones. Existing adaptive-budget methods alleviate this inefficiency, yet their stopping rules are heuristic and do not provide a finite-sample calibration guarantee. We propose CGRPO, which replaces the fixed rollout budget of standard GPRO with calibrated conformal thresholds over task-specific nonconformity scores, combining Adaptive Prediction Set (APS)-style calibration for closed-form reasoning, a first-success score for execution-based code, automatic miscoverage selection, and periodic re-calibration. We prove finite-sample coverage for the resulting stopping rule and show that, under monotone policy improvement, recalibration induces a non-increasing expected training budget. Across six benchmarks spanning mathematics, graduate-level science, and code generation, CGRPO reduces mean training rollouts by 49-70% relative to fixed-budget GRPO while retaining strong fixed-budget accuracy, achieves the strongest pass@1 among evaluated static and dynamic baselines on different datasets, and transfers across model families with up to 82% rollout savings.
Figures & tables
Figure 1: Overview of C-GRPO workflow.
Figure 2: Training dynamics (GSM8K, Qwen2.5-7B). C-GRPO sustains the highest reward and lowest KL divergence throughout training. Periodic dips align with Δcal=50 re-calibration events; the policy recovers to a higher reward peak within ∼5 steps each time.
Dataset
Accuracy (%)
Training Compute
Eval ( k=16 , 1 × H100)
p@1
p@8
p@16
p@32
kˉtrain
Rollout sav.
GPU-h
VRAM (GB)
Tok/s
Mathematical reasoning
GSM8K
57.2
70.5
73.8
75.0
10.4
68%
4.04
15.6
450
MATH-500
52.5
66.8
71.2
71.5
15.9
50%
1.61
21.6
1,880
MATH-Algebra
61.2
69.5
72.0
74.0
12.2
62%
8.24
16.6
442
Graduate-level science
Table 1: C-GRPO on six benchmarks (Qwen2.5-7B-Instruct, step 300). Fixed-budget accuracy p@ k , mean training rollouts kˉtrain , and rollout savings 1−kˉtrain/32 vs. GRPO G=32 (training rollouts only; calibration cost in Table 10 ).
Model
Algorithm
Pass@1 (%)
Δ pass@1
kˉtrain
Type
Training Efficiency (1 × H100)
GSM8K
HumanEval
MBPP
MATH-500
GPU-h
VRAM (GB)
Tok/s
Qwen2.5 7B-Instruct
GRPO ( Shao et al., 2024 )
54.8
33.3
56.0
48.5
—
32.0
static
11.7
26.8
3,180
GDRO ( Panaganti et al., 2026 )
56.8
34.5
58.2
51.0
+2.0
32.0 ∗
dynamic
11.7
26.8
3,180
AERO ( Zhang et al., 2026 )
56.1
34.0
57.5
50.2
+1.3
16.6
dynamic
6.1
22.4
4,270
GRESO ( Zheng et al., 2025 )
54.7
33.1
55.8
48.3
−0.1
15.6
dynamic
5.7
21.9
4,410
Reinforce-Ada ( Xiong et al., 2025 )
55.4
33.8
56.5
49.5
+0.6
14.8
dynamic
5.4
21.4
4,520
Table 2: Pass@1 (%) and training compute (Qwen2.5-7B-Instruct, step 300). kˉtrain : mean rollouts/prompt (lower is better). Δ : pass@1 gap vs. GRPO on GSM8K. GPU-h / VRAM / Tok/s: total training wall-clock (including all calibration generation for C-GRPO), peak GPU memory, and effective rollout generation throughput on 1 × H100 (all methods, 300 steps). ∗ GDRO uses a compute-neutral global budget ( kˉ=32 ).
Dataset
Model
δ(0)
p@1
p@8
p@16
p@32
kˉtrain
Savings
GSM8K
Qwen2.5-7B-Inst.
0.12
57.2
70.5
73.8
75.0
10.4
68%
Llama-3.1-8B-Inst.
0.05
72.5
85.0
91.0
91.5
5.7
82%
Gemma-3-4B-IT
0.18
48.5
62.0
65.5
66.0
14.2
56%
MBPP
Qwen2.5-7B-Inst.
0.10
60.5
78.5
83.0
83.5
10.0
69%
Llama-3.1-8B-Inst.
0.08
65.5
81.0
86.5
87.0
8.5
73%
Gemma-3-4B-IT
0.15
50.2
65.0
70.0
71.0
12.5
61%
Table 3: Cross-model generalisation at step 300 on held-out test sets. δ(0) is auto-selected per model–dataset pair without manual tuning.
Figure 3: Rollout efficiency during training (GSM8K, Qwen2.5-7B, 300 steps). (a) Mean kˉ per step falls from Kmax=32 to ≈10 ; drops at each Δcal=50 checkpoint mark auto- δ tightening. (b) Cumulative rollouts: 59.8% fewer than the fixed- k=32 ceiling (dashed; shaded: 5 738 rollouts saved).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
δ(0)
Target
Empirical
Slack
GSM8K
0.12
84.0%
83.8%
−0.2 pp
MATH-500
0.25
74.0%
74.7%
+0.7 pp
GPQA-Diamond
0.48
44.0%
43.9%
−0.1 pp
MATH-Algebra
0.21
80.0%
80.0%
±0 pp
HumanEval
0.90
90.0%
90.9%
+0.9 pp
MBPP
0.23
76.0%
76.3%
+0.3 pp
Appendix
Table 4: Coverage validation across six benchmarks (Qwen2.5-7B, held-out test, step 300). δ(0) : auto-selected initial miscoverage. Empirical coverage matches or exceeds the nominal target on all benchmarks within the finite-sample tolerance ±1/(ncal+1)≈0.5% .
Figure 4: Per- (k,t) coverage scatter on GSM8K (Qwen2.5-7B). Each point is one (k,step) pair. Every point lies on or above the diagonal (empirical ≥ target), jointly validating Theorem 2 for all k∈K and all training steps simultaneously. The densest cluster at 80–90% corresponds to k=8 – 16 during mid-training; points move rightward as auto- δ tightens targets with policy improvement.
(A) Auto- δ initialisation / GSM8K
(B) Re-calibration freq. / GSM8K
(C) Budget grid / MATH-500
Strategy
p@1
p@16
kˉ
Cov.
Triv.
Δ
p@1
p@16
kˉ
OH
K
p@1
p@16
kˉ
#
Fixed .05
54.5
70.2
29.5
94.2
∼200
25
57.3
73.9
10.3
+12
{2,8,32}
51.0
69.5
18.4
3
Fixed .20
56.0
72.5
18.4
81.5
∼30
50
57.2
73.8
10.4
base
{2,4,8,16,32}
52.5
71.2
15.9
5
Auto
57.2
73.8
10.4
83.8
0
100
55.8
72.1
13.5
−6
{1,2,4,8,16,32}
52.3
70.8
16.2
6
∞
54.5
70.0
18.2
−12
Appendix
Table 5: Ablation studies for C-GRPO. Bold = default configuration. “Triv.” = training steps where all thresholds are inactive (curriculum reduces to vanilla GRPO). “OH” = overhead steps per checkpoint relative to Δcal=50 baseline. “#” = number of distinct budget levels. All rows use the original calibration procedure.
Test Domain
Shift
dTV
Theo. LB
Empirical
Margin
GSM8K (in-dist.)
none
0.00
87.5%
87.9%
+0.4 pp
MATH-Algebra
mild
0.06
81.5%
83.6%
+2.1 pp
MATH-500
moderate
0.15
72.5%
76.8%
+4.3 pp
GPQA-Diamond
severe
0.29
58.5%
70.4%
+11.9 pp
Appendix
Table 6: Coverage under distribution shift (Qwen2.5-7B, trained and calibrated on GSM8K, step 300, δ(0)=0.12 , nominal target 88% ). dTV : TV distance between GSM8K calibration and test distribution. Theoretical bound: 1−δ−dTV−1/(ncal+1) . Empirical coverage consistently exceeds the theoretical lower bound.
Benchmark
Method
G / kˉ
p@1
p@8
p@16
p@32
GPU-h
GSM8K
GRPO
2
48.2
64.0
67.5
68.0
1.8
GRPO
4
51.0
68.2
70.1
71.2
2.4
GRPO
8
52.8
70.1
72.0
73.1
3.6
GRPO
10 (matched)
53.1
70.5
72.4
73.5
4.2
GRPO
16
54.2
71.8
73.2
74.1
6.2
GRPO
32
54.8
72.0
73.5
74.6
11.7
Appendix
Table 7: Full static- G sweep (Qwen2.5-7B-Instruct, 300 steps, 1 × H100). Every static row is trained at fixed G .
Table 9: Stronger base, full-parameter fine-tuning (GSM8K, Qwen2.5-Math-7B-Instruct, base p@1 =82.4 , 300 steps; mean ± std over three seeds).
Phase (hours)
GRPO G=32
GRPO G=10
C-GRPO ( kˉ=10.4 )
Training generation
8.10
2.55
2.80
Forward + backward
3.60
1.65
0.75
Re-calibration (6 events)
—
—
0.25
Micro-batch scheduling
—
—
0.008
Total wall-clock
11.70
4.20
3.81
Speed-up vs. G=32
1.00 ×
2.79 ×
3.07 ×
Appendix
Table 10: Wall-clock breakdown (GSM8K, Qwen2.5-7B-Instruct, 300 steps, 1 × H100; original calibration procedure). C-GRPO includes all calibration generation, batched as large offline batches; with the released code’s default (training-batch-size) calibration the re-calibration share is larger (36–47% of training time in a 100-step check).