When Rewards Depend on the Group, GRPO Plays a Tournament

Junyi Gao; Yu Shi; Pingzhao Hu; Ewen M Harrison

cs.LGNov 2, 2026

When Rewards Depend on the Group, GRPO Plays a Tournament

Authors: Junyi Gao, Yu Shi, Pingzhao Hu, Ewen M Harrison

Organizations: Centre for Medical Informatics, University of Edinburgh · Health Data Research UK · Biostatistics Division, Dalla Lana School of Public Health, University of Toronto · Department of Biochemistry, Western University

Abstract

Rewards for diversity, coverage and distribution matching make the reward of each sampled response depend on the rest of its group, and they are now routinely optimized with GRPO and RLOO. We prove that in this setting the group-relative baseline changes the objective. The expected update equals each response's own reward gradient plus $α$ times its effect on its groupmates' rewards, with $α=-1/(K-1)$ for groups of size $K$ , so GRPO trains every response to beat its groupmates in a zero-sum tournament. It cannot see a pairwise diversity bonus when K=2, sees less than half of it at any K, and often does worse than using no baseline at all. Setting $α=+1$ gives Team-GRPO, an unbiased estimator for any group-dependent reward that works with two samples per prompt and requires no extra model calls for pairwise rewards. The theory predicts training quantitatively. It locates the fixed points of a 9B policy to within 0.02 in $\ell_1$ and predicts how concentrated trained policies become with Pearson r=0.995. On GlobalOpinionQA, GRPO collapses onto the majority answer and ends further from human opinion than the untrained model, while Team-GRPO cuts the error from $0.20$ to $0.08$ . On NoveltyBench, Team-GRPO avoids the diversity collapse of GRPO and significantly improves set utility.

Figures & tables

Figure 1: The problem. An answer helps its group by being good and by being different. GRPO credits each sample with how far it beats the group mean, so groupmates become rivals and converge on one answer. Team-GRPO credits each sample with what it adds to the group, so the group stays diverse.

Figure 2: GRPO plays a tournament. (a) Team-GRPO maximizes the group’s total reward, REINFORCE without a baseline maximizes each response’s own reward (a Nash equilibrium), and GRPO maximizes each response’s reward minus the mean of its groupmates’ (a zero-sum tournament). Bars give the weight of a pairwise diversity bonus in the expected update relative to the true gradient ( Corollary 2 ). (b) Two responses of quality

q=0.8

under the reward

q\,(1+\lambda d)

\lambda=2

. GRPO assigns zero advantage whether the answers differ (

d=1

) or coincide (

d=0

). Team-GRPO separates the two cases. (c) Answer shares on a held-out GlobalOpinionQA question (whether the gap between rich and poor has grown, Venezuela) after 200 steps with two samples per question.

Figure 3: The theory is exact. (a) Weight GRPO gives an

m

-body symmetric interaction, relative to the true gradient. Lines are Equation 3 , dots are exact enumeration over random kernels, and dashed lines mark the limit

1/m

. (b) Probability of the best option at the fixed point of the diversity bonus (

\lambda=1

). Lines are closed forms and dots are Qwen3.5-9B after 500 steps of LoRA training. (c) Full policies at

K=2

, trained (bars) against closed form (black ticks). GRPO collapses onto the best option, REINFORCE reaches the Nash solution, and Team-GRPO reaches the optimum of the group reward.

Group reward (used by)	$K{=}2$	$4$	$8$	$2$	$4$	$8$
	GRPO			REINFORCE
Distribution matching (SWYS)	$-31$	67	75	82	82	82
Quality $\times$ distance (DARLING)	83	90	96	98	98	98
Correct $\times$ #strategies (Poly-EPO)	67	53	44	100	98	83
Unique valid answers (coverage)	3	94	40	100	94	40
Split credit (distinct@ $K$ )	61	100	100	100	100	100

Table 1: Group objective at each estimator’s fixed point , as a percentage of Team-GRPO (exact computation). Split credit divides a unit of reward among the members that earn it, so the group total is the set metric in parentheses. Shading marks the shortfall.

Estimator	Fixed point	$K{=}2$	$K{=}4$	$K{=}8$
GRPO	$\mathrm{Proj}_{\Delta}(2w_{K}t)$	0.491	0.181	0.179
REINFORCE	$\mathrm{Proj}_{\Delta}(2t)$	0.131	same update as $K{=}2$
Centred witness	$\mathrm{Proj}_{\Delta}(w_{K}t)$	$=$ GRPO	0.096	0.073
Poly-EPO (pairs)	$t$	needs $K{>}2$	0.078	0.074
Team-GRPO	$t$	0.081	0.074	0.077

Table 2: GlobalOpinionQA test error

\|\pi-t\|^{2}

after 200 steps (510 unseen questions, mean over seeds, base model

0.197

), with

K

samples per question at 256 rollouts per step. Fixed points follow Proposition 4 (b) with

w_{K}=\tfrac{K-1}{K-2}

. Grey cells need no run, since at

K=2

the centred witness gives exactly GRPO’s advantages, REINFORCE has the same expected update at every

K

, and Poly-EPO needs more samples than its set size. Here DARLING’s estimator is GRPO, GAPO’s is GRPO (std), and SetPO’s coincides with Team-GRPO. Bold is best per column.

Figure 4: Distributional alignment on GlobalOpinionQA. (a) Squared error to the human distribution on unseen questions after 200 steps (log scale, means over seeds). Lines start or stop where the theory makes a run redundant ( Table 2 ). The bias of the centred witness shrinks with

K

as predicted. (b) Test error during training at

K=2

(means over seeds). GRPO collapses within 50 steps, with or without standard-deviation normalization. (c) Predicted against measured mass on the most-chosen answer for all 13 configurations with a closed-form fixed point, with Poly-EPO shown separately for 4 and 8 samples (dashed line is equality).

Figure 5: Open-ended generation on NoveltyBench (

K=2

\lambda=2

unless noted). (a,b) Set utility and number of distinct answers among 10 samples. Lines are means over three seeds and shading spans the seeds. (c) Utility against sequence-level KL divergence from the base model, per seed. (d) Mean pairwise embedding distance within training groups (smoothed). GRPO collapses faster as the diversity weight

\lambda

grows, while Team-GRPO becomes more diverse.

$K=2$
	Utility $\uparrow$	Distinct $\uparrow$	1st sample
Base model	4.01	4.33	7.58
GRPO ( $\lambda{=}2$ )	3.44	2.65	9.04
GRPO (best $\lambda{=}1$ )	3.83	3.19	9.07
Team-GRPO ( $\lambda{=}2$ )	4.61	4.56	8.51
$\Delta$ , same step	$+1.16$ $[0.52,1.80]$	$+1.91$ $[1.01,2.81]$

Table 3: NoveltyBench performance (benchmark scorer, 400 prompts, three seeds).

\Delta

is Team-GRPO minus GRPO with 95% CIs, and at

K=8

matched KL compares checkpoints within the common observed KL range. Bold is the best trained model on set metrics, and 1st sample scores one response. LLM-judge results are in Table 13 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Identity	$K=2$	$3$	$4$	$6$	$10$
$\mathbb{E}\hat{g}_{\textup{{reinforce}}}=Kg_{0}$	$3.1\cdot 10^{-16}$	$3.7\cdot 10^{-16}$	$1.0\cdot 10^{-15}$	$4.4\cdot 10^{-16}$	$2.6\cdot 10^{-15}$
$\mathbb{E}\hat{g}_{\textup{{rloo}}}=Kg_{-1/(K-1)}$	$3.2\cdot 10^{-16}$	$2.2\cdot 10^{-16}$	$2.6\cdot 10^{-16}$	$4.9\cdot 10^{-16}$	$2.1\cdot 10^{-15}$
$\mathbb{E}\hat{g}_{\textup{{grpo}}}=\tfrac{K-1}{K}\mathbb{E}\hat{g}_{\textup{{rloo}}}$	$1.4\cdot 10^{-16}$	$1.9\cdot 10^{-16}$	$2.7\cdot 10^{-16}$	$3.3\cdot 10^{-16}$	$7.0\cdot 10^{-16}$
$g_{1}=\nabla J$ (finite differences)	$6.1\cdot 10^{-11}$	$2.0\cdot 10^{-10}$	$1.0\cdot 10^{-10}$	$1.1\cdot 10^{-10}$	$9.3\cdot 10^{-11}$

Appendix

Table 4: Largest relative error of Theorem 1 in exact enumeration over random exchangeable rewards.

$K$	error	variance	error	variance	error	variance
	Team-GRPO		REINFORCE		GRPO / RLOO
3	$1\cdot 10^{-10}$	5.33	0.49	6.45	0.78	1
4	$1\cdot 10^{-10}$	1.69	0.43	2.26	0.64	1
8	$4\cdot 10^{-10}$	2.11	0.31	4.44	0.39	1
16	$1\cdot 10^{-9}$	1.07	0.80	6.55	0.86	1

Appendix

Table 5: Relative error of the expected update with respect to

\nabla J

, and variance at matched scale relative to RLOO, at a random policy (exact).

Reward family	$K=2$	$K=4$	$K=8$
Quality $\times$ same-class indicator (DARLING, binary kernel)	0 / 0	88 / 100	93 / 99.8
Coverage with shared credit	73 / 73	91 / 99	91 / 99.8
Graded quality $\times$ continuous cosine distance	0.3 / 11	61 / 89	70 / 96

Appendix

Table 6: Group objective reached by GRPO with its default coefficient and with the best coefficient on a grid, as a percentage of Team-GRPO (exact computation). The continuous-distance row averages three random embedding draws.

	$K=3$	$K=4$	$K=6$	$K=8$
GRPO, closed form	1.000	0.875	0.813	0.792
GRPO (std), numerical	1.000	0.686	0.814	0.713
GRPO (std), Qwen3.5-9B	–	0.695	–	0.717

Appendix

Table 7: Mass on the best answer at the fixed point of the diversity bonus (

\lambda=1

): closed form, numerical ascent using exact expectations, and the 9B sandbox.

Reward	Estimator	$K$	$\lambda$	Fixed point	$\ell_{1}$	Clipped	$\mu=1$	$\mu=4$	$\mu=8$	Clipped
					Exact, $\mu=4$		Sampled
Diversity bonus	GRPO	2	0	(1, 0, 0, 0)	0.0001	92	0.000	0.000	0.000	0.0
	GRPO	2	1	(1, 0, 0, 0)	0.0001	92	0.000	0.000	0.000	0.0
	GRPO	2	5	(1, 0, 0, 0)	0.0001	92	0.000	0.000	0.000	0.0
	GRPO	4	1	(.875, .125, 0, 0)	0.0002	80	0.005	0.013	0.012	2.4
	GRPO	4	3	(.537, .287, .138, .037)	0.0000	79	0.005	0.008	0.011	2.5

Appendix

Table 8: Clipped surrogate (

\epsilon=0.2

) against the fixed point of the plain update. The exact columns give the

\ell_{1}

error at convergence and the largest share of clipped samples during training (%). The sampled columns give the

\ell_{1}

error of the time-averaged policy for

\mu

epochs per batch and the share of clipped samples at

\mu=8

(%). ∗ Numerical fixed point of the plain update.

Method	Group-dependent term	$K$	Structure	Evaluated by	Under GRPO
DARLING ( Li et al., 2025 )	quality $\times$ mean semantic distinctness	8	asymmetric pairwise	quality and novelty, pass@ $k$ on math	not repairable by retuning ( Proposition 5 )
Song et al. ( Song et al., 2025 )	$-$ mean indicator of equal answers	8, 16	symmetric pairwise	pass@ $k$	keeps $\tfrac{3}{7}$ , $\tfrac{7}{15}$ of the weight
GAPO ( Anschel et al., 2025 )	penalty on the answer’s frequency in the group	32	symmetric pairwise	diversity and coverage of valid answers	keeps $\tfrac{15}{31}$ of the weight
DSDR ( Wan et al., 2026 )	embedding distance and formula uniqueness, gated by correctness	8	pairwise, gated by own correctness	pass@ $k$	diversity term attenuated
SWYS ( Arabi et al., 2026 )	collision with groupmates	64	symmetric pairwise	distance to the target distribution	fixed by their witness estimator, centred variant biased ( Section A.9 )
Pref-GRPO ( Wang et al., 2025 )	win rate within the group	–	constant-sum	per-image quality	amplified by $\tfrac{K}{K-1}$ , harmless alone

Appendix

Table 9: Group-dependent rewards in recent work and the consequence of optimizing them with group-relative baselines. Evaluated by is the metric each paper reports for the property its reward targets.

Estimator	$K$	$\lambda$	Trained (train prompts)	Trained (held-out)	Closed form	$\ell_{1}$
GRPO	2	0	(1, 0, 0, 0)	(1, 0, 0, 0)	(1, 0, 0, 0)	0.000
GRPO	2	1	(1, 0, 0, 0)	(1, 0, 0, 0)	(1, 0, 0, 0)	0.000
GRPO	2	5	(1, 0, 0, 0)	(1, 0, 0, 0)	(1, 0, 0, 0)	0.000
GRPO	4	1	(.878, .122, 0, 0)	(.876, .124, 0, 0)	(.875, .125, 0, 0)	0.006
GRPO	4	3	(.538, .286, .130, .046)	(.538, .286, .130, .046)	(.537, .287, .138, .037)	0.018
GRPO	8	1	(.797, .203, 0, 0)	(.795, .205, 0, 0)	(.792, .208, 0, 0)	0.011

Appendix

Table 10: All sandbox runs. Policies are averaged over prompts at the end of training.

$K$	Estimator	50	100	150	200	TV	Coll. (%)
2	GRPO	0.528	0.514	0.505	0.491	0.515	96.5
2	GRPO (std)	0.539	0.534	0.510	0.508	0.523	96.1
2	REINFORCE	0.130	0.158	0.131	0.131	0.268	13.3
2	Team-GRPO	0.097	0.087	0.082	0.081	0.239	0.5
4	GRPO	0.194	0.213	0.196	0.181	0.311	28.8
4	GRPO (std)	0.204	0.224	0.200	0.207	0.329	37.8

Appendix

Table 11: Test error

\|\pi-t\|^{2}

at each checkpoint, and TV distance and collapse rate after 200 steps, averaged over seeds for each estimator and

K

. The base model has error

0.197

, TV

0.330

and collapse rate

16.6\%

Comparison	half distance	full distance
Team-GRPO $-$ GRPO, $K=2$ , seed 0	$-0.115$ $[-0.122,-0.107]$	$-0.207$ $[-0.219,-0.195]$
Team-GRPO $-$ GRPO, $K=2$ , seed 1	$-0.110$ $[-0.117,-0.104]$	$-0.217$ $[-0.229,-0.204]$
Team-GRPO (general form) $-$ GRPO, $K=4$	$-0.076$ $[-0.081,-0.070]$	$-0.121$ $[-0.131,-0.111]$
Team-GRPO $-$ GRPO, $K=8$	$-0.071$ $[-0.078,-0.064]$	$-0.088$ $[-0.100,-0.077]$

Appendix

Table 12: Difference in test error at equal distance from the base model (95% CIs over questions).

	$K=2$			$K=8$
	Utility $\uparrow$	Distinct $\uparrow$	1st sample	Utility $\uparrow$	Distinct $\uparrow$	1st sample
Base model	2.27	4.09	5.11	2.27	4.09	5.11
GRPO ( $\lambda{=}2$ )	2.10	2.27	5.82	2.35	2.81	5.77
Team-GRPO ( $\lambda{=}2$ )	2.71	4.36	5.50	2.61	4.15	5.55
$\Delta$ , same step	$+0.61$ $[0.36,0.85]$	$+2.09$ $[1.67,2.52]$		$+0.26$ $[0.07,0.45]$	$+1.34$ $[1.05,1.63]$
$\Delta$ , matched KL	$+0.41$ $[0.24,0.58]$	$+1.41$ $[1.13,1.70]$		$+0.17$ $[0.03,0.31]$	$+0.79$ $[0.49,1.08]$

Appendix

Table 13: NoveltyBench under the LLM judge (NoveltyBench v1.1 protocol, 100 prompts).

\Delta

is Team-GRPO minus GRPO with 95% CIs. Bold is the best trained model on set metrics.

Baseline $b_{k}$	$\mathbb{E}[A^{2}]$
Missing response’s diversity set to zero	0.145
Batch means of quality and distance	0.078
Batch-mean distance, quality shrunk toward the teammate (used)	0.056
GRPO, rescaled to the same scale (reference)	0.035

Appendix

Table 14: Second moment of the Team-GRPO advantage at

K=2

for different baselines (

\lambda=1

, real batches).

Estimator	$\lambda$	Training diversity	Utility	Distinct	First sample
GRPO	0.25	0.193	3.80	3.10	9.12
GRPO	1	0.186	3.83	3.19	9.07
GRPO	2	0.147	3.44	2.65	9.04
Team-GRPO	0.25	0.166	3.84	3.18	9.09
Team-GRPO	1	0.231	4.25	4.01	8.72
Team-GRPO	2	0.249	4.61	4.56	8.51

Appendix

Table 15: Diversity coefficient sweep with the benchmark scorer at

K=2

, step 150. Training diversity is the mean pairwise embedding distance over the last 25 training updates.

Prompt	Base model	GRPO	Team-GRPO
“Randomly pick a prime number between 1 and 50.”	29 $\times 4$ , 17, 23, 37, 43, 47, and one sample listing four primes. distinct 7, utility 6.42	23 $\times 7$ , 29 $\times 2$ , 17. All ten use the same template, and two garble the list of primes they append. distinct 3, utility 3.70	29 $\times 3$ , 19 $\times 2$ , 37 $\times 2$ , 11, 23, 31, mostly as a bare number. distinct 6, utility 6.45
“I need you to give me the single best ice breaking question.”	“If you could instantly master any skill, what would it be?” $\times 4$ , and six other questions. distinct 6, utility 4.64	“What’s something you’re really passionate about right now?” $\times 8$ , and the same question with “looking forward to” and “excited about”. distinct 3, utility 2.78	A small win you had recently $\times 2$ , something you’re quietly excited about this week $\times 2$ , a skill you would master instantly $\times 2$ , something you’re learning this year, something you’re quietly proud of, something that surprised even you, a simple joy. distinct 7, utility 6.29
“Suggest a name for a dappled-gray filly living in the mountains.”	A list of names grouped by theme in every sample, six of them headed by Moss. distinct 3, utility 2.27	A short bulleted list in every sample, six of them headed by Mist, often followed by Fawn and Stone. distinct 2, utility 1.61	One recommended name with a rationale per sample, Cloudstrider $\times 2$ , Ashen Peak, Mistweave, Mistynose, Hemlock Storm, Summit Sprout, Mistborn, Moss Run, Frostbit. Half add a few alternatives. distinct 9, utility 5.69

Appendix

Table 16: All ten samples for three NoveltyBench prompts from the LLM-judge evaluation (

K=2

\lambda=2

, step 150), summarized by the answer each sample gives, with the judge’s number of distinct answers and set utility.

\times n

marks an answer given in

n

samples.

Group Relative Policy Optimization

Group Relative Policy Optimization (GRPO) trains policies by comparing sampled outputs within groups, without a separate learned value critic.

See all 341 papers

Oct 18, 2026

Where Groups Split: A Probe Prior for Nonzero Within-Group Variance in RL Post-Training

Group relative policy optimization learns from a prompt only when the rewards in the sampled group disagree. With a binary verifier, a group that is entirely correct or entirely incorrect has within-group variance zero and a zero advantage. Uniform sampling continues to spend rollouts on prompts that are unanimous now and stay unanimous as training goes on. Pass rate is a weak forecast of that future. Two prompts at the same pass rate can move apart: one into the range where a finite group splits, and one toward a ceiling where the group becomes unanimous again. Aperture estimates the later pass rate from a short probe. A small subset of the pool is trained with the same recipe, every prompt is scored on the base model and on the probe model, and the two pass rates are plugged into the probability that a group of the training size is all wrong or split. The resulting weight is a static prior. It can be used alone or multiplied by an online scheduler, and the GRPO update is left as it is. On mathematical and logical reasoning with 1.7B and 4B models, the prior improves greedy accuracy over uniform GRPO and raises the fraction of groups with a nonzero advantage. The same prior still helps when it is combined with dynamic sampling or with selective rollouts.

Jonah Ellison, Mateo Ruiz, Owen Pell +4

University of Michigan · University of Edinburgh

RL for Language Model Reasoning RL Post-Training Group Relative Policy Optimization

Ask PDF

May 8, 2026

Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works

Group Relative Policy Optimization (GRPO) is a standard algorithm for reinforcement learning from verifiable rewards, but its group-mean-centered advantage can fail under binary rewards. The failure mode is gradient starvation: when every response in a group is correct or every response is wrong, the centered advantage is exactly zero and the policy receives no learning signal. We prove that the true degeneracy rate always exceeds the i.i.d. Bernoulli prediction by Jensen's inequality, and observe a 0.69 degeneracy rate at group size four in logged Qwen3.5-9B GSM8K training. We then show that the fixed-reference Sign advantage,

A=2r-1

, performs pass@

G

failure descent by increasing the probability that at least one sample in the group succeeds. On the full GSM8K test set across seven seeds, Sign reaches 73.8% accuracy versus 28.4% for standard normalized group-mean DrGRPO at group size four, a 45.4 point gain with

p<0.0001

. The effect is directionally consistent on Llama-3.1-8B and positive but underpowered on a MATH-500 transfer check. Pass@

k

analysis indicates that the main benefit is search compression rather than large capacity expansion, aligning the empirical gains with recent RLVR ceiling observations.

Wenhua Nie, Jianan Wu, Junlin Liu +6

National Taiwan University

Reinforcement Learning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards

Ask PDF

Aug 4, 2025

TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback

Group Relative Policy Optimization (GRPO) trains language models without a value critic using rewards centered within sampled response groups. We study how importance weighting, clipping, and length normalization affect its stochastic updates and propose Trajectory-level Importance-Corrected GRPO (TIC-GRPO), combining an upper importance-ratio cap with a full-trajectory likelihood ratio. Under common maximum-length normalization, a trajectory change of measure gives a score second moment proportional to the response horizon. We propagate this estimate through the actual empirical baseline and all inner updates. With separately optimized constant steps, the sampling term in TIC's stationarity bound has a linear horizon coefficient, versus order three halves for token-level GRPO

_2

. Matching upper and lower bounds separate the capped methods' worst-case response-level variances by a factor

T

. The bounds track prompt count, response-group size, vocabulary size, reward scale, and score regularity; per-response-normalized GRPO also retains a length-covariance term. Experiments at two Qwen3 scales on four reasoning and coding benchmarks evaluate both mechanisms in sampled-token-normalized training.

Lei Pang, Jun Luo, Ruinan Jin

Peking University Beijing, China · The Ohio State University Columbus, USA · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE

Importance Sampling for RL RL for Language Models Group Relative Policy Optimization Policy Optimization RL from Human Feedback

Ask PDF

Abstract

Group Relative Policy Optimization

Explore similar work

Where Groups Split: A Probe Prior for Nonzero Within-Group Variance in RL Post-Training

Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works

TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback