Rubric-based reinforcement learning uses fine-grained grader feedback to train large language models on open-ended tasks, but grader randomness can alter GRPO updates in ways that score discrepancies alone cannot characterize. Because GRPO jointly normalizes rewards within each response group and combines the resulting training coefficients with response-level gradients, a scoring discrepancy can shift the coefficients of the entire group and induce policy-dependent update changes. Selective execution further affects the direction-reversal risk of retained updates. To address this, we propose Set-Valued Group Relative Policy Optimization (SetGRPO). It comprises three modules: (a) Joint Reward Set Calibration constructs a joint uncertainty set for the training coefficients; (b) Geometry-Guided Safe Recovery maps candidate coefficients into the current policy's local update geometry and determines the largest safe recovery ratio for each response group; and (c) Selective Risk-Gated Optimization selects a global threshold through independent risk calibration to control the conditional direction-reversal risk of accepted updates while preserving the original GRPO objective. Under exchangeability, SetGRPO provides finite-sample marginal coverage for the joint coefficient set and simultaneous risk control across candidate thresholds. On ten open-ended benchmarks spanning three domains, SetGRPO achieves the highest macro average among all compared training methods on both Qwen3.5 and Llama3.2 model families.
Figures & tables
Figure 1: Illustration of score disagreement and its effect on local policy-update geometry.
Figure 2: Overview of SetGRPO. (a) Joint reward-set calibration uses two grader scores on the same response group to construct a joint coefficient uncertainty set. (b) Geometry-guided safe recovery maps candidate coefficients through the current policy geometry and identifies the largest direction-preserving recovery ratio. (c) Selective risk-gated optimization selects a global threshold and applies the recovered coefficients only to updates satisfying the calibrated conditional-risk constraint.
Model
Medical
Writing
Instruction Following
Macro Avg.
HB
LEM
MQA
MCQA
WB
CW
IFE
IFB
IB
MIF
Closed-source Models
OpenAI-o3
59.80
78.90
96.00
84.70
77.70
81.40
91.60
67.80
89.40
64.00
79.13
GPT-4.1
47.90
71.20
92.40
80.00
69.00
79.00
87.00
37.20
89.40
64.00
71.71
Gemini-3 Pro Preview
49.30
72.70
67.00
62.20
78.50
81.50
94.20
61.20
85.60
66.83
71.90
Open-sourced Models
Table 1: Main results on ten downstream benchmarks. HB: HealthBench; LEM: LLMEval-Med; MQA: MedQA; MCQA: MedMCQA; WB: WritingBench; CW: Creative Writing v3; IFE: IFEval; IFB: IFBench; IB: InfoBench; and MIF: MulDimIF. Our results report the mean across repeated runs, with absolute improvements over the corresponding Base model shown in parentheses.
Figure 3: Training dynamics of SetGRPO, including the reward obtained from the first score, policy entropy, accepted-group ratio, and mean recovery ratio over the first 500 training steps.
Backbone
Method
Medical
Writing
Instruction Following
Macro Avg.
HB
LEM
MQA
MCQA
WB
CW
IFE
IFB
IB
MIF
Qwen3-4B
RubricHub
64.34
45.12
78.38
61.88
66.51
28.47
64.21
24.08
80.21
47.35
56.06
SetGRPO (Ours)
66.82
47.57
77.91
61.42
68.97
30.18
66.83
26.49
79.66
49.72
57.56
Qwen3-14B
Rubrichub
68.44
53.36
77.53
64.76
70.42
32.71
69.74
25.61
82.88
46.79
59.12
SetGRPO (Ours)
70.91
54.88
76.94
64.18
72.83
34.62
72.21
27.94
82.17
49.18
60.59
Table 2: Same-backbone comparison between RubricHub and SetGRPO on Qwen3-4B and Qwen3-14B. RubricHub results were reproduced by our implementation using its publicly available training procedure. HB: HealthBench; LEM: LLMEval-Med; MQA: MedQA; MCQA: MedMCQA; WB: WritingBench; CW: Creative Writing v3; IFE: IFEval; IFB: IFBench; IB: InfoBench; and MIF: MulDimIF.
Configuration
Qwen3-4B
Qwen3-14B
Rollout group size K
8
8
Max prompt length
2048
2048
Max response length
4096
4096
Batch size
64
64
Mini-batch size
32
32
Optimizer
AdamW
AdamW
Table 3: Training and rollout configuration for Qwen3-4B and Qwen3-14B.
Backbone
Cov.
Risk
Qα
τsel
N
F
ECR
U
Qwen3-4B
27,229
27,229
1.32
0.25
19113
1047
0.0548
0.575
Qwen3-14B
27,229
27,229
1.45
0.25
19137
1058
0.0554
0.580
Table 4: Frozen coverage and risk calibration records for Qwen3-4B and Qwen3-14B. ECR denotes empirical conditional risk.
Figure 4: Risk calibration and retained updates at the selected threshold τsel .
D#
Derivative
Macro Avg.
Reversal risk
CR
NUR
↑
Δ
↓
Δ
↑
Δ
↑
Δ
(a) Coefficient-set construction
1
Spherical set
56.62
-3.64
0.096
+0.049
0.61
-0.19
0.512
-0.190
(b) Local geometry and recovery
2
Identity geometry
55.08
-5.18
0.118
+0.071
0.51
-0.29
0.438
-0.264
3
Safe-anchor-only
56.94
-3.32
0.061
+0.014
0.36
-0.44
0.318
-0.384
Table 5: Ablation study of SetGRPO on Qwen3.5-4B. Δ denotes the change from Full SetGRPO in each metric’s native units; upward arrows indicate the preferred direction. CR is normalized cumulative recovery and NUR is the nonzero update rate. Reversal risk, NUR, and CR lie in [0,1] .
Method
Training-time breakdown
Resource usage
10-Benchmark Macro Avg.
Single-step time (s)
Rollout time (s)
Grader time (s)
Jacobian time (s)
Coefficient recovery time (s)
Peak VRAM (GB)
Cumulative GPU-hours
GRPO
82
48
18
0
0
61
22.8
55.41
SetGRPO
103
48
18
13
7
67
28.6
60.26
Table 6: Efficiency comparison between GRPO and SetGRPO.
Figure 5: Coefficient-norm dynamics during training. The left panel shows the coefficient-norm trajectories of the two backbones, while the right panel summarizes the observed norm range between training steps 0 and 500. The bounded ranges indicate that coefficient recovery does not lead to sustained numerical growth.
Figure 6: Projection diagnostics for accepted and rejected updates. The left panel shows the distributions of full-geometry and reward-space minimum projections. The right panel shows their joint constraints; the shaded upper-right region satisfies both the full-geometry threshold τsel and the positive reward-space projection condition.
Figure 7: Cumulative recovery over the first 500 training steps for Qwen3.5-4B and Qwen3.5-9B. Shaded regions distinguish the early, middle, and late stages; the bracket reports the final cumulative-recovery gap.
Figure 8: Positive-affine invariance under frozen-policy control. The left plot reports coefficient differences between the original and transformed standardized coefficients. The right table summarizes coefficient cosine, update cosine, decision flips, and the absolute recovery-ratio difference for each transform rx′=arx+b1K with a>0 .
Figure 9: Empirical coverage of the joint coefficient set and direction-reversal risk separation at the selected threshold.
Transform
Coeff. cosine
Update cosine
Flip (%)
∣Δη⋆∣ ( 10−3 )
a=0.5,b=−1
0.9998
0.9996
0.00
0.6
a=1,b=0
1.0000
1.0000
0.00
0.0
a=2,b=1
0.9999
0.9998
0.00
0.4
a=3,b=−2
0.9997
0.9995
0.00
0.8
Table 7: Positive-affine invariance diagnostics.
Figure 10: Checkpoint-wise validation of the fixed SRGO rule under policy drift on Qwen3.5-4B. Curves show the accepted-group rate, mean recovery ratio, and accepted-group upper confidence bound; bars show empirical direction-reversal rates for accepted and rejected groups. The red dashed line denotes β=0.08 .
Figure 11: Representative accepted cases. Each case shows the anonymized query–response record, point and reference scores, full-geometry and reward-space projection checks, the recovery path from η=0 to η=1 , the recovered training coefficient a^x , and the final decision.
Figure 12: Representative rejected cases. Each case shows the anonymized query–response record, point and reference scores, full-geometry and reward-space projection checks, the recovery path from η=0 to η=1 , the recovered training coefficient a^x , and the final decision.
Figure 13: Representative recovery-infeasible cases. Each case shows the anonymized query–response record, point and reference scores, full-geometry and reward-space projection checks, the recovery path from η=0 to η=1 , the recovered training coefficient a^x , and the final decision.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose Mu-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation-optimization stages. This design induces high rollout staleness while greatly reducing rollout-optimization switching overhead. To stabilize learning under stale data, Mu-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds the performance of standard GRPO while achieving around 2x speedup in wall-clock training time, establishing a substantially improved performance-efficiency trade-off for LLM reinforcement learning.
Group Relative Policy Optimization (GRPO) trains language models without a value critic using rewards centered within sampled response groups. We study how importance weighting, clipping, and length normalization affect its stochastic updates and propose Trajectory-level Importance-Corrected GRPO (TIC-GRPO), combining an upper importance-ratio cap with a full-trajectory likelihood ratio. Under common maximum-length normalization, a trajectory change of measure gives a score second moment proportional to the response horizon. We propagate this estimate through the actual empirical baseline and all inner updates. With separately optimized constant steps, the sampling term in TIC's stationarity bound has a linear horizon coefficient, versus order three halves for token-level GRPO2. Matching upper and lower bounds separate the capped methods' worst-case response-level variances by a factor T. The bounds track prompt count, response-group size, vocabulary size, reward scale, and score regularity; per-response-normalized GRPO also retains a length-covariance term. Experiments at two Qwen3 scales on four reasoning and coding benchmarks evaluate both mechanisms in sampled-token-normalized training.
Lei Pang, Jun Luo, Ruinan Jin
Peking University Beijing, China · The Ohio State University Columbus, USA · Shugu Yuntai Technology Co., Ltd. +1
Group Relative Policy Optimization (GRPO) has become a standard reinforcement learning method for post-training language models. Recent work shows that GRPO can reduce the base model's reasoning capacity and underperform it in Pass@k when k is large, indicating reduced coverage of reasoning paths. We find that this reduction is associated with GRPO concentrating on responses that the base model already generates with high probability. We trace this concentration to two mechanisms in the GRPO update. At the response level, high-probability responses dominate the group gradient through repeated occurrence. At the token level, GRPO's importance ratio scales gradients, further reinforcing tokens that become more likely under the current policy. We propose ReCo, a reweighting method that addresses both effects. Response contributions are normalized by their expected occurrence within the rollout group, and the token-level importance ratio is replaced with a variance-based ratio that gives larger update scale to non-saturated decision points where alternative token choices remain plausible. Across Qwen2.5-Math-1.5B/7B and Llama-3.1-8B-Instruct on five mathematical reasoning benchmarks, ReCo improves Pass@k for large values of k and is comparable to GRPO for small values of k.
Junoh Park, Junseo Hwang, Wonguk Cho +1
Graduate School of Data Science, Seoul National University