Your Group-Relative Advantage Is Biased

Fengkai Yang; Zherui Chen; Xiaohan Wang; Xiaodong Lu; Jiajun Chai; Guojun Yin; Wei Lin; Shuai Ma

cs.LGJan 13, 2026

Your Group-Relative Advantage Is Biased

Authors: Fengkai Yang, Zherui Chen, Xiaohan Wang, Xiaodong Lu, Jiajun Chai, Guojun Yin, Wei Lin, Shuai Ma,

Organizations: Beihang University · Peking University · Meituan · University of California, Berkeley

Abstract

Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood. In this work, we characterize conditional estimation bias in group-relative advantages among non-degenerate groups. For mean-centered advantages under binary rewards, our analysis shows negative conditional mean estimation errors for hard prompts and positive errors for easy prompts. At the response level, correct answers to hard prompts are underweighted, while incorrect answers to easy prompts receive insufficient penalties. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.

Figures & tables

Figure 1: (a) Comparison of the performance of RL algorithms with and without HA-DW on Qwen3-4B-Base across five mathematical reasoning benchmarks. (b) Significant biased advantage estimation on the MATH dataset under 8 and 128 rollouts. (c) Performance gain by GRPO+HA-DW on MATH500 stratified by difficulty levels.

Figure 2: HA-DW consists of two collaborative phases. In the first phase, an evolving difficulty anchor incorporates cross-batch historical information by propagating the model’s prior through a history buffer, capturing long-term reward trends. In the second phase, prompt weights are adaptively adjusted based on their estimated difficulty under the model’s evolving state, compensating for biased advantage estimates.

Algorithm	Math						Code
Algorithm	MATH500	AIME25	AMC23	Minerva	OlympiadBench	Avg.	HumanEval	MBPP	Avg.
Qwen-3-4B-Base
GRPO	75.4	19.6	60.3	33.8	43.5	46.5	70.1	58.2	64.2
$\hookrightarrow$ + HA-DW	78.0	20.4	63.4	36.8	44.7	48.7	73.2	60.4	66.8
GSPO	75.8	20.0	62.2	35.3	42.3	47.1	69.5	58.6	64.1
$\hookrightarrow$ + HA-DW	77.6	19.6	68.6	37.1	43.2	49.2	72.0	59.4	65.7

Table 1: Overall results across models (Qwen, LLaMA) and different group-relative RL algorithms (GRPO, GSPO, DAPO). We report the performance of different base RL algorithms, and the corresponding accuracy when applied HA-DW for each model scale and family.

Threshold	MATH500	AIME25	AMC23	Minerva	OlympiadBench	AVG
Base	75.4	19.6	60.3	33.8	43.5	46.5
0.4 (fixed)	77.0	18.5	63.1	37.5	44.3	48.1
0.5 (fixed)	76.6	20.0	62.7	35.7	44.0	47.8
0.6 (fixed)	76.8	21.3	61.1	36.4	44.3	48.0
$C_{t}$	78.0	20.4	63.4	36.8	44.7	48.7

Table 2: Ablation on the effectiveness of dynamic threshold for RL training using Qwen3-4B-Base.

C_{t}

denotes the dynamic threshold.

Figure 3: Comparison of training dynamics under different training strategies. Average accuracy across five benchmarks, training reward and response length of Qwen3-4B-Base and Qwen3-8B-Base on different training methods.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

$G$	$\mathbb{P}(G,0,0.25)$
2	0.999997499987
4	0.999995948256
6	0.827761785622
8	0.781129955681

Appendix

Table 3:

\mathbb{P}(G,0,0.25)

as a function of

G\in[2,8]

$G$	$\mathbb{P}(G,0,0.5)$
2	0.999994999975
4	0.776965795853
6	0.689721502158
8	0.640944744224

Appendix

Table 4:

\mathbb{P}(G,0,0.5)

as a function of

G\in[2,8]

$G$	$\frac{G}{2}\int_{0}^{2/G}f(p_{t})\,dp_{t}$
2	0.499997499987
3	0.749995833315
4	0.776965795853
5	0.780787089465
6	0.781154327380

Appendix

Table 5:

\frac{G}{2}\int_{0}^{2/G}f(p_{t})\,dp_{t}

as a function of

G\in[2,6]

Figure 4: Curve of the normalization term $1/\sqrt{p_{t}(1-p_{t})}$ with respect to $p_{t}$ .

Figure 5: Illustration of advantage bias under truncated Gaussian rewards for different group sizes.

Dataset	8	16	8+HA-DW
MATH500	75.4	76.2	78.0
AIME25	19.6	19.2	20.4
AMC23	60.3	61.6	63.4
Minerva	33.8	34.2	36.8
OlympiadBench	43.5	43.9	44.7

Appendix

Table 6: Performance of Qwen3-4B-Base trained with: Rollout=8 with GRPO, Rollout=16 with GRPO and Rollout=8 with GRPO+HA-DW. Rollout=32 with GRPO is out of memory .

Method	MATH500	AIME25	AMC23	Minerva	OlympiadBench	AVG
GRPO	75.4	19.6	60.3	33.8	43.5	46.5
GRPO + HA-DW (Hard Prompts Only)	77.4	20.6	62.8	35.7	44.3	48.2
GRPO + HA-DW	78.0	20.4	63.4	36.8	44.7	48.7

Appendix

Table 7: Ablation of HA-DW on easy prompts using Qwen3-4B-Base. The hard-prompt-only variant applies HA-DW reweighting only to hard prompts, while leaving easy-prompt updates unchanged.

Method	Rollout Token Budget	MATH500	AIME25	AMC23	Minerva	OlympiadBench	AVG
GRPO	120M	75.4	19.6	60.3	33.8	43.5	46.5
GRPO + HA-DW	120M	77.4	19.8	62.7	35.9	44.3	48.0

Appendix

Table 8: Performance under a matched rollout-token budget. GRPO and GRPO+HA-DW use the same group size, maximum generation length, and total budget of 120M generated response tokens.

Method	MATH500	AIME25	AMC23	Minerva	OlympiadBench	AVG
GRPO	75.4	19.6	60.3	33.8	43.5	46.5
GRPO + Reward Noise	72.6	16.7	56.9	30.9	40.6	43.5
GRPO + HA-DW	78.0	20.4	63.4	36.8	44.7	48.7
GRPO + HA-DW + Reward Noise	73.4	16.3	58.1	31.8	40.9	44.1

Appendix

Table 9: Robustness of HA-DW under injected reward noise on hard prompts using Qwen3-4B-Base.

$\lambda_{\mathrm{scale}}$	MATH500	AIME25	AMC23	Minerva	OlympiadBench	AVG
0.5	75.4	18.1	61.1	34.2	43.7	46.5
0.8	76.8	19.2	61.3	34.9	43.7	47.2
1.0	76.8	18.5	61.6	36.0	44.3	47.4
1.3	78.0	20.4	63.4	36.8	44.7	48.7
1.5	77.8	20.8	63.1	37.1	44.0	48.6
1.7	76.4	20.0	63.4	36.4	44.3	48.1

Appendix

Table 10: Performance of Qwen3-4B-Base trained with GRPO+HA-DW on different

\lambda_{\mathrm{scale}}

Algorithm	IFEval	GPQA	API-Bank	AVG
Qwen3-4B-Base
GRPO	32.3	26.2	62.1	40.2
GRPO+HA-DW	34.0	27.8	63.6	41.8
GSPO	32.0	26.8	61.2	40.0
GSPO+HA-DW	33.1	26.3	60.7	40.0
DAPO	31.1	27.2	62.6	40.3

Appendix

Table 11: Performance comparison on instruction following and tool-use benchmarks.

Benchmark	GRPO (mean $\pm$ std, N=5)	HA-DW (mean $\pm$ std, N=5)	Average Improvement	P-Value
MATH500	75.44 $\pm$ 0.30	77.88 $\pm$ 0.23	+2.44	$3.08\times 10^{-5}$
AIME25	19.04 $\pm$ 0.61	20.44 $\pm$ 0.30	+1.40	$7.59\times 10^{-3}$
AMC23	60.42 $\pm$ 0.34	63.58 $\pm$ 0.41	+3.16	$6.71\times 10^{-6}$
Minerva	33.68 $\pm$ 0.41	36.78 $\pm$ 0.25	+3.10	$4.33\times 10^{-4}$
OlympiadBench	43.42 $\pm$ 0.44	44.88 $\pm$ 0.20	+1.46	$1.20\times 10^{-3}$

Appendix

Table 12: Significance analysis of HA-DW against GRPO on Qwen3-4B-Base over five independent runs.

Figure 6: The distribution of prompts by the number of correct and incorrect responses on the MATH dataset and DAPO-Math-17k under 8 and 128 rollouts of Qwen3-4B-Base.

	Hyperparameter	GRPO	GRPO+HA-DW	GSPO	GSPO+HA-DW	DAPO	DAPO+HA-DW
General	nnode	1	1	1	1	1	1
	gpus per node	8	8	8	8	8	8
	use kl in reward	False	False	False	False	False	False
	use kl loss	False	False	False	False	False	False
	tensor parallel size	1	1	1	1	1	1
	test frequency	5	5	5	5	5	5

Appendix

Table 13: Hyperparameter settings for group-relative methods.

Panel of Figure 7 — Figure 7: An example of GRPO

Panel of Figure 8 — Figure 8: An example of GRPO+HA-DW

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,120 papers

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
Feb 6, 2026Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4Reinforcement Learning RL for Language Model Reasoning
T-Tech · Saint Petersburg Electrotechnical University “LETI”
Leveraging Error Diversity in Group Rollouts for Reinforcement Learning
May 17, 2026Wenpu Liu, Yuqi Xu, Weichu Xie +8Reward Shaping RL for Language Model Reasoning
Peking University · Shanghai Innovation Institute
Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation
Feb 5, 2026Zhiqi Yu, Zhangquan Chen, Mengting Liu +2RL for Language Model Reasoning Group Relative Policy Optimization
University of Hong Kong · Tsinghua University · Sun Yat-sen University

Feb 6, 2026J/K move · Enter open · S save

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii+3

T-Tech · Saint Petersburg Electrotechnical University “LETI”

Reinforcement Learning RL for Language Model Reasoning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, computational limits often rule out very large groups, so training proceeds with finite rollout sets that can reinforce only the correct behavior they expose. At practical group sizes, updates can miss rare-correct trajectories while still containing mixed rewards, concentrating probability on more common sampled solutions. We derive the probability of such prompt-local tail-miss events as a function of group size, showing non-monotonic behavior, and in the categorical abstraction characterize how unsampled-correct mass can shrink even as total correct mass grows. Motivated by this analysis, we propose a difficulty-aware scaling coefficient, inspired by Focal loss, that down-weights updates on high-success sampled groups. Empirically, categorical simulation illustrates the same effect in the categorical setting, Maze provides a single-solution test, and LLM experiments include a representative GRPO group-size sweep together with fixed-

N

transfer across GRPO, DAPO, and CISPO. On Qwen2.5-7B at

N{=}8

, our method improves average math pass@256 from 64.1

\rightarrow

70.3 (GRPO), 69.3

\rightarrow

72.5 (DAPO), and 73.2

\rightarrow

76.8 (CISPO); OOD pass@256 also improves in all three cases, without increasing group size or computational cost.

Abstract

RL for Language Model Reasoning

Explore similar work

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Leveraging Error Diversity in Group Rollouts for Reinforcement Learning

Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation