SetGRPO: Calibrating Grader-Induced Policy Optimization

Jiale Huang; Zixu Li; Zhiwei Chen; Zhiheng Fu; Jinhe Bi; Yupeng Hu

cs.LGOct 27, 2026

SetGRPO: Calibrating Grader-Induced Policy Optimization

Authors: Jiale Huang, Zixu Li, Zhiwei Chen, Zhiheng Fu, Jinhe Bi, Yupeng Hu

Organizations: Shandong University · National University of Singapore

Abstract

Rubric-based reinforcement learning uses fine-grained grader feedback to train large language models on open-ended tasks, but grader randomness can alter GRPO updates in ways that score discrepancies alone cannot characterize. Because GRPO jointly normalizes rewards within each response group and combines the resulting training coefficients with response-level gradients, a scoring discrepancy can shift the coefficients of the entire group and induce policy-dependent update changes. Selective execution further affects the direction-reversal risk of retained updates. To address this, we propose Set-Valued Group Relative Policy Optimization (SetGRPO). It comprises three modules: (a) Joint Reward Set Calibration constructs a joint uncertainty set for the training coefficients; (b) Geometry-Guided Safe Recovery maps candidate coefficients into the current policy's local update geometry and determines the largest safe recovery ratio for each response group; and (c) Selective Risk-Gated Optimization selects a global threshold through independent risk calibration to control the conditional direction-reversal risk of accepted updates while preserving the original GRPO objective. Under exchangeability, SetGRPO provides finite-sample marginal coverage for the joint coefficient set and simultaneous risk control across candidate thresholds. On ten open-ended benchmarks spanning three domains, SetGRPO achieves the highest macro average among all compared training methods on both Qwen3.5 and Llama3.2 model families.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of score disagreement and its effect on local policy-update geometry.

Figure 2: Overview of SetGRPO. (a) Joint reward-set calibration uses two grader scores on the same response group to construct a joint coefficient uncertainty set. (b) Geometry-guided safe recovery maps candidate coefficients through the current policy geometry and identifies the largest direction-preserving recovery ratio. (c) Selective risk-gated optimization selects a global threshold and applies the recovered coefficients only to updates satisfying the calibrated conditional-risk constraint.

Model	Medical				Writing		Instruction Following				Macro Avg.
Model	HB	LEM	MQA	MCQA	WB	CW	IFE	IFB	IB	MIF	Macro Avg.
Closed-source Models
OpenAI-o3	59.80	78.90	96.00	84.70	77.70	81.40	91.60	67.80	89.40	64.00	79.13
GPT-4.1	47.90	71.20	92.40	80.00	69.00	79.00	87.00	37.20	89.40	64.00	71.71
Gemini-3 Pro Preview	49.30	72.70	67.00	62.20	78.50	81.50	94.20	61.20	85.60	66.83	71.90
Open-sourced Models

Table 1: Main results on ten downstream benchmarks. HB: HealthBench; LEM: LLMEval-Med; MQA: MedQA; MCQA: MedMCQA; WB: WritingBench; CW: Creative Writing v3; IFE: IFEval; IFB: IFBench; IB: InfoBench; and MIF: MulDimIF. Our results report the mean across repeated runs, with absolute improvements over the corresponding Base model shown in parentheses.

Figure 3: Training dynamics of SetGRPO, including the reward obtained from the first score, policy entropy, accepted-group ratio, and mean recovery ratio over the first 500 training steps.

Backbone	Method	Medical				Writing		Instruction Following				Macro Avg.
Backbone	Method	HB	LEM	MQA	MCQA	WB	CW	IFE	IFB	IB	MIF	Macro Avg.
Qwen3-4B	RubricHub	64.34	45.12	78.38	61.88	66.51	28.47	64.21	24.08	80.21	47.35	56.06
Qwen3-4B	SetGRPO (Ours)	66.82	47.57	77.91	61.42	68.97	30.18	66.83	26.49	79.66	49.72	57.56
Qwen3-14B	Rubrichub	68.44	53.36	77.53	64.76	70.42	32.71	69.74	25.61	82.88	46.79	59.12
Qwen3-14B	SetGRPO (Ours)	70.91	54.88	76.94	64.18	72.83	34.62	72.21	27.94	82.17	49.18	60.59

Table 2: Same-backbone comparison between RubricHub and SetGRPO on Qwen3-4B and Qwen3-14B. RubricHub results were reproduced by our implementation using its publicly available training procedure. HB: HealthBench; LEM: LLMEval-Med; MQA: MedQA; MCQA: MedMCQA; WB: WritingBench; CW: Creative Writing v3; IFE: IFEval; IFB: IFBench; IB: InfoBench; and MIF: MulDimIF.

Configuration	Qwen3-4B	Qwen3-14B
Rollout group size $K$	8	8
Max prompt length	2048	2048
Max response length	4096	4096
Batch size	64	64
Mini-batch size	32	32
Optimizer	AdamW	AdamW

Table 3: Training and rollout configuration for Qwen3-4B and Qwen3-14B.

Backbone	Cov.	Risk	$Q_{\alpha}$	$\tau_{\mathrm{sel}}$	$N$	$F$	ECR	$U$
Qwen3-4B	27,229	27,229	1.32	0.25	19113	1047	0.0548	0.575
Qwen3-14B	27,229	27,229	1.45	0.25	19137	1058	0.0554	0.580

Table 4: Frozen coverage and risk calibration records for Qwen3-4B and Qwen3-14B. ECR denotes empirical conditional risk.

Figure 4: Risk calibration and retained updates at the selected threshold

\tau_{\mathrm{sel}}

D#	Derivative	Macro Avg.		Reversal risk		CR		NUR
D#	Derivative	$\uparrow$	$\Delta$	$\downarrow$	$\Delta$	$\uparrow$	$\Delta$	$\uparrow$	$\Delta$
(a) Coefficient-set construction
1	Spherical set	56.62	-3.64	0.096	+0.049	0.61	-0.19	0.512	-0.190
(b) Local geometry and recovery
2	Identity geometry	55.08	-5.18	0.118	+0.071	0.51	-0.29	0.438	-0.264
3	Safe-anchor-only	56.94	-3.32	0.061	+0.014	0.36	-0.44	0.318	-0.384

Table 5: Ablation study of SetGRPO on Qwen3.5-4B.

\Delta

denotes the change from Full SetGRPO in each metric’s native units; upward arrows indicate the preferred direction. CR is normalized cumulative recovery and NUR is the nonzero update rate. Reversal risk, NUR, and CR lie in

[0,1]

Method	Training-time breakdown					Resource usage		10-Benchmark Macro Avg.
Method	Single-step time (s)	Rollout time (s)	Grader time (s)	Jacobian time (s)	Coefficient recovery time (s)	Peak VRAM (GB)	Cumulative GPU-hours	10-Benchmark Macro Avg.
GRPO	82	48	18	0	0	61	22.8	55.41
SetGRPO	103	48	18	13	7	67	28.6	60.26

Table 6: Efficiency comparison between GRPO and SetGRPO.

Panel of Figure 5 — Figure 5: Coefficient-norm dynamics during training. The left panel shows the coefficient-norm trajectories of the two backbones, while the right panel summarizes the observed norm range between training steps 0 and 500. The bounded ranges indicate that coefficient recovery does not lead to sustained numerical growth.

Panel of Figure 6 — Figure 6: Projection diagnostics for accepted and rejected updates. The left panel shows the distributions of full-geometry and reward-space minimum projections. The right panel shows their joint constraints; the shaded upper-right region satisfies both the full-geometry threshold $\tau_{\mathrm{sel}}$ and the positive reward-space projection condition.

Figure 7: Cumulative recovery over the first 500 training steps for Qwen3.5-4B and Qwen3.5-9B. Shaded regions distinguish the early, middle, and late stages; the bracket reports the final cumulative-recovery gap.

Figure 8: Positive-affine invariance under frozen-policy control. The left plot reports coefficient differences between the original and transformed standardized coefficients. The right table summarizes coefficient cosine, update cosine, decision flips, and the absolute recovery-ratio difference for each transform

\mathbf{r}^{\prime}_{x}=a\mathbf{r}_{x}+b\mathbf{1}_{K}

with

a>0

Panel of Figure 9 — Figure 9: Empirical coverage of the joint coefficient set and direction-reversal risk separation at the selected threshold.

Transform	Coeff. cosine	Update cosine	Flip (%)	$\|\Delta\eta^{\star}\|$ ( $10^{-3}$ )
$a=0.5,\ b=-1$	0.9998	0.9996	0.00	0.6
$a=1,\ b=0$	1.0000	1.0000	0.00	0.0
$a=2,\ b=1$	0.9999	0.9998	0.00	0.4
$a=3,\ b=-2$	0.9997	0.9995	0.00	0.8

Table 7: Positive-affine invariance diagnostics.

Figure 10: Checkpoint-wise validation of the fixed SRGO rule under policy drift on Qwen3.5-4B. Curves show the accepted-group rate, mean recovery ratio, and accepted-group upper confidence bound; bars show empirical direction-reversal rates for accepted and rejected groups. The red dashed line denotes

\beta=0.08

Figure 11: Representative accepted cases. Each case shows the anonymized query–response record, point and reference scores, full-geometry and reward-space projection checks, the recovery path from $\eta=0$ to $\eta=1$ , the recovered training coefficient $\hat{\mathbf{a}}_{x}$ , and the final decision.

Figure 12: Representative rejected cases. Each case shows the anonymized query–response record, point and reference scores, full-geometry and reward-space projection checks, the recovery path from $\eta=0$ to $\eta=1$ , the recovered training coefficient $\hat{\mathbf{a}}_{x}$ , and the final decision.

Figure 13: Representative recovery-infeasible cases. Each case shows the anonymized query–response record, point and reference scores, full-geometry and reward-space projection checks, the recovery path from $\eta=0$ to $\eta=1$ , the recovered training coefficient $\hat{\mathbf{a}}_{x}$ , and the final decision.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
May 17, 2026Minghao Tian, Yunfei Xie, Chen WeiRL for Language Model Reasoning Group Relative Policy Optimization
Rice University
TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback
Aug 4, 2025Lei Pang, Jun Luo, Ruinan JinImportance Sampling for RL RL for Language Models
Peking University Beijing, China · The Ohio State University Columbus, USA · Shugu Yuntai Technology Co., Ltd. +1
ReCo: Reweighting GRPO Against Distributional Concentration
Jul 29, 2026Junoh Park, Junseo Hwang, Wonguk Cho +1Reasoning Diversity RL for Language Model Reasoning
Graduate School of Data Science, Seoul National University

May 17, 2026J/K move · Enter open · S save

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning

Minghao Tian, Yunfei Xie, Chen Wei

Rice University

RL for Language Model Reasoning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards RL Post-Training Policy Learning Off-Policy RL

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose Mu-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation-optimization stages. This design induces high rollout staleness while greatly reducing rollout-optimization switching overhead. To stabilize learning under stale data, Mu-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds the performance of standard GRPO while achieving around 2x speedup in wall-clock training time, establishing a substantially improved performance-efficiency trade-off for LLM reinforcement learning.

Abstract

Reinforcement Learning

Explore similar work

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning

TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback

ReCo: Reweighting GRPO Against Distributional Concentration