cs.LGNov 3, 2026

C-GRPO: Conformal Group Relative Policy Optimization

Authors: Arya Fayyazi, Seyedarmin Azizi, Mehdi Kamal, Massoud Pedram

Organizations: University of Southern California Los Angeles, CA, USA

Abstract

Reasoning-oriented language model training increasingly relies on multiple on-policy rollouts, but standard Group Relative Policy Optimization (GRPO) uses a fixed group size, wasting compute on easy prompts and under-exploring hard ones. Existing adaptive-budget methods alleviate this inefficiency, yet their stopping rules are heuristic and do not provide a finite-sample calibration guarantee. We propose CGRPO, which replaces the fixed rollout budget of standard GPRO with calibrated conformal thresholds over task-specific nonconformity scores, combining Adaptive Prediction Set (APS)-style calibration for closed-form reasoning, a first-success score for execution-based code, automatic miscoverage selection, and periodic re-calibration. We prove finite-sample coverage for the resulting stopping rule and show that, under monotone policy improvement, recalibration induces a non-increasing expected training budget. Across six benchmarks spanning mathematics, graduate-level science, and code generation, CGRPO reduces mean training rollouts by 49-70% relative to fixed-budget GRPO while retaining strong fixed-budget accuracy, achieves the strongest pass@1 among evaluated static and dynamic baselines on different datasets, and transfers across model families with up to 82% rollout savings.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning

    Feb 19, 2026Gagan Mundada, Zihan Huang, Rohan Surana +8Efficient Language Model ReasoningRL for Language Model Reasoning

  2. Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective

    May 13, 2026Feng Zhang, Xinhong Ma, Ziqiang Dong +5RL for Language Model ReasoningGroup Relative Policy Optimization

  3. When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO

    Mar 13, 2026Yu Li, Tian Lan, Zhengling QiRL for Language Model ReasoningGroup Relative Policy Optimization