Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation

Zhiqi Yu; Zhangquan Chen; Mengting Liu; Heye Zhang; Liangqiong Qu

cs.LGFeb 5, 2026

Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation

Authors: Zhiqi Yu, Zhangquan Chen, Mengting Liu, Heye Zhang, Liangqiong Qu

Organizations: University of Hong Kong · Tsinghua University · Sun Yat-sen University

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR), particularly GRPO, has become the standard for eliciting LLM reasoning. However, its efficiency in exploration and difficulty adaptation remains an open challenge. In this work, we identify an implicit advantage symmetry inherent in Group Relative Advantage Estimation (GRAE) as a structural property that provides a new perspective for understanding these bottlenecks. This symmetry induces two critical limitations: (i) at the group level, strict symmetry in weights between correct and incorrect trajectories leaves unsampled action logits unchanged, thereby hindering the exploration of novel correct solution. (ii) at the sample level, the algorithm implicitly prioritizes medium-difficulty samples, remaining agnostic to the non-stationary demands of difficulty focus. Through controlled experiments, we reveal that this symmetric property is sub-optimal, yielding two pivotal insights: (i) asymmetrically down-weighting the advantages of correct trajectories encourages essential exploration but risks instability; (ii) learning efficiency can be boosted by a curriculum-like transition-prioritizing simpler samples initially before gradually shifting to complex ones. Motivated by these findings, we propose Asymmetric GRAE (A-GRAE), which dynamically modulates exploration incentives and sample-difficulty focus. Experiments across seven benchmarks demonstrate that A-GRAE consistently improves GRPO and its variants across both LLMs and MLLMs. Code is available at https://github.com/HKU-HealthAI/A-GRAE

Figures & tables

Figure 1 : The two-fold implicit advantage symmetry problem of GRAE in GRPO. At the group level, the advantage weights for correct trajectories equal those of incorrect trajectories. This symmetry leads to the logits of low-probability correct paths unchanged under the advantage-driven update, thereby hindering the model’s exploration. At the sample level, medium-difficulty samples exhibit the largest sum of absolute advantage values, leading to insufficient training on harder data.

Table 1 : Summary of Controlled Experiments. We design two experiments to investigate the impact of symmetry at the group and sample levels.

\beta=10

and

\gamma=0.5

are scaling constants.

Panel of Figure 2 — Figure 2 : Experimental results on breaking group-level symmetry using Qwen2.5-Math-7B . We amplify (Positive-Dominant) or suppress (Negative-Dominant) the advantages of correct trajectories to compare their performance with those of GRPO and the base model.The performance is evaluated using Pass@ $k$ . The last figure illustrates the entropy dynamics on the training set.

Panel of Figure 3 — Figure 3 : Experimental results on breaking sample-level symmetry using Qwen2.5-Math-7B . We rescale the advantages to shift the learning focus toward harder queries (Hard-Focused) or easier queries (Easy-Focused) to compare their performance with those of GRPO and the base model.The performance is evaluated using Pass@ $k$ . The last figure illustrates the within-batch count of correct sampled responses on the training set.

Method	Pass@ $k$
$k$	1	2	4	8	16	32	64	128	256	Avg.
MATH
Base Model	63.4 $\pm 0.4$	74.8 $\pm 0.5$	83.2 $\pm 0.6$	88.6 $\pm 0.7$	91.2 $\pm 0.7$	93.4 $\pm 0.6$	94.1 $\pm 0.5$	95.0 $\pm 0.4$	96.3 $\pm 0.3$	86.7
SEC	77.6 $\pm 0.5$	82.5 $\pm 0.6$	86.0 $\pm 0.6$	88.9 $\pm 0.4$	90.2 $\pm 0.5$	92.2 $\pm 0.6$	92.6 $\pm 0.5$	93.3 $\pm 0.3$	94.7 $\pm 0.3$	88.7
GRPO-LEAD	77.8 $\pm 0.3$	83.0 $\pm 0.4$	86.5 $\pm 0.5$	89.2 $\pm 0.5$	90.5 $\pm 0.5$	92.3 $\pm 0.5$	92.8 $\pm 0.4$	93.6 $\pm 0.4$	95.0 $\pm 0.3$	89.0
P@kT	76.2 $\pm 0.6$	82.2 $\pm 0.8$	86.9 $\pm 0.6$	89.6 $\pm 0.4$	91.4 $\pm 0.3$	93.3 $\pm 0.4$	94.0 $\pm 0.5$	96.2 $\pm 0.4$	96.2 $\pm 0.4$	89.3

Table 2 : Pass@

k

results on MATH, AIME 2025 and AMC23 with Qwen2.5-Math-7B . Bold and underlined numbers denote the best and second-best results for each

k

. Specifically, We conduct 16 runs on AIME2025 and AMC23, and 4 runs on the remaining benchmarks.

	ID Domain	OOD Domain
Method	Geo3K	MathVision	Mathverse
Task A: General Mathematical Reasoning
Base Model	27.8 $\pm 1.0$	20.8 $\pm 0.8$	31.6 $\pm 0.7$
GRPO	43.5 $\pm 0.8$	23.4 $\pm 0.7$	35.2 $\pm 0.9$
GRPO + A-GRAE	45.7 $\pm 0.6$	24.0 $\pm 0.4$	36.8 $\pm 0.5$
DAPO	44.7 $\pm 0.9$	23.8 $\pm 0.6$	35.9 $\pm 0.8$

Table 3 : Results on multi-modal benchmarks with Qwen2.5-VL-3B-Instruct . We conduct 4 runs on each benchmark. Bold and underlined numbers denote the best and second-best results.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4 : Visualization of Update Magnitude with Respect to Sample Difficulty. The update magnitude reaches its peak at $p=0.5$ , simple samples (e.g., $p=0.75$ ) and hard samples (e.g., $p=0.25$ ) exhibiting the same deviation from $p=0.5$ are assigned identical importance weights.

Panel of Figure 5 — Figure 5 : Experimental results on breaking group-level symmetry using Llama-3.2-3B-Instruct . The performance is evaluated using Pass@ $k$ ( $k=\{1,2,4,8,16,32,64,128,256\}$ ) to provide a comprehensive analysis.

Panel of Figure 6 — Figure 6 : Experimental results on breaking sample-level symmetry using Llama-3.2-3B-Instruct . The performance is evaluated using Pass@ $k$ ( $k=\{1,2,4,8,16,32,64,128,256\}$ ) to provide a comprehensive analysis.

Method	Pass@ $k$
$k$	1	2	4	8	16	32	64	128	256	Avg.
MATH
Base Model	84.3 $\pm 0.5$	89.6 $\pm 0.6$	92.8 $\pm 0.5$	94.8 $\pm 0.4$	96.0 $\pm 0.4$	96.8 $\pm 0.3$	97.4 $\pm 0.3$	97.8 $\pm 0.3$	98.0 $\pm 0.2$	94.2
GRPO	92.4 $\pm 0.3$	95.3 $\pm 0.4$	96.5 $\pm 0.4$	97.0 $\pm 0.3$	97.3 $\pm 0.3$	97.7 $\pm 0.2$	98.0 $\pm 0.2$	98.3 $\pm 0.2$	98.6 $\pm 0.2$	96.8
GRPO + A-GRAE	92.9 $\pm 0.2$	95.7 $\pm 0.3$	96.9 $\pm 0.3$	97.5 $\pm 0.3$	97.9 $\pm 0.3$	98.3 $\pm 0.2$	98.6 $\pm 0.2$	99.0 $\pm 0.2$	99.3 $\pm 0.1$	97.3
DAPO	92.6 $\pm 0.3$	95.3 $\pm 0.4$	96.4 $\pm 0.4$	97.2 $\pm 0.3$	97.5 $\pm 0.3$	97.8 $\pm 0.3$	98.0 $\pm 0.2$	98.2 $\pm 0.2$	98.6 $\pm 0.2$	96.8

Appendix

Table 4 : Pass@

k

results on MATH, AIME 2025 and AMC23 with DeepSeek-R1-7B . Bold and underlined numbers denote the best and second-best results for each

k

. Specifically, We conduct 16 runs on AIME2025 and AMC23, and 4 runs on the remaining benchmarks.

Panel of Figure 7 — Figure 7 : Training collapse of Negative-Dominant group during training with Qwen2.5-Math-7B . Left Part: The unsolved question count suddenly increase in the later stage. Right Part: The collapse leads to the significant performance decrease.

Method	Pass@ $k$
$k$	1	2	4	8	16	32	64	128	256
MATH
Base Model	63.4	74.8	83.2	88.6	91.2	93.4	94.1	95.0	96.3
GRPO	76.5	82.3	86.1	88.8	90.3	92.6	93.5	93.9	95.0
GRPO + A-GRAE (sample level)	77.8	84.2	88.4	90.5	92.1	94.0	94.3	94.8	96.0
GRPO + A-GRAE (group level)	77.6	84.0	88.1	91.2	93.0	94.3	95.3	95.6	97.0

Appendix

Table 5 : Ablation studies on MATH, AIME 2025 and AMC23 with Qwen2.5-Math-7B .

Panel of Figure 8 — Figure 8 : Training dynamics using Qwen2.5-Math-7B . Left Part: Model’s entropy on the Math training set. Center Part: Model’s entropy (actor entropy loss) on the Math test set. Right Part: The greedy decoding accuracy on the Math test set.

Method	$k=1$	$k=2$	$k=4$	$k=8$	$k=16$	$k=256$
GRPO (Baseline)	76.5	82.3	86.1	88.8	90.3	92.4
Negative-Dominant	77.2	83.8	87.6	90.4	92.8	94.5
GRPO + Entropy-Promoting	75.8	80.3	83.6	86.5	88.4	89.0

Appendix

Table 6 : Comparison of MATH Pass@

k

(%) between Negative-Dominant group and entropy-promoting techniques.

Comparison Pair	$k=1$	$k=2$	$k=4$	$k=8$	$k=16$	$k=32$	$k=64$	$k=128$	$k=256$
MATH Benchmark
GRPO + A-GRAE vs. GRPO	.0032	.0018	.0025	.0041	.0084	.0115	.0128	.0155	.0121
DAPO + A-GRAE vs. DAPO	.0015	.0022	.0048	.0055	.0072	.0098	.0105	.0112	.0108
Dr.GRPO + A-GRAE vs. Dr.GRPO	.0024	.0009	.0011	.0022	.0015	.0028	.0035	.0042	.0038
AIME 2025 Benchmark
GRPO + A-GRAE vs. GRPO	.0125	.0098	.0112	.0105	.0142	.0128	.0115	.0092	.0085

Appendix

Table 7 : Detailed

p

-values across scaling intervals (

k=1

k=256

p

-values are calculated comparing the A-GRAE enhanced version against its respective vanilla baseline.

Method / Setting	MATH	AMC23	AIME2025
GRPO (Baseline)	76.5	59.2	10.3
$\alpha=0.25$	77.8	61.8	10.9
$\alpha=0.5$	77.8	61.8	11.0
$\alpha=1.0$	78.3	62.6	11.3
$\alpha=2.0$	78.0	62.8	11.3
$\alpha=5.0$	76.8	62.3	10.8

Appendix

Table 8 : Sensitivity analysis of

\alpha

on Qwen2.5-Math-7B. Performance is measured by Pass@1 (%).

Explore similar work

EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance
May 6, 2026Song Yu, Li Li, Wenwen Zhao +1Reinforcement Learning With Verifiable Reward Flow-Grpo
School of Computer and Information Science, Southwest University, Chongqing 400715, China
How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
May 17, 2026Minghao Tian, Yunfei Xie, Chen WeiLarge Language Model Reinforcement Learning Flow-Grpo
Rice University
F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
Feb 6, 2026Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4Reinforcement Learning With Verifiable Reward Flow-Grpo
T-Tech · Saint Petersburg Electrotechnical University “LETI”

May 6, 2026, cs.LGJ/K move · Enter open · S save

EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance

Song Yu, Li Li, Wenwen Zhao, Zhisheng Yang

School of Computer and Information Science, Southwest University, Chongqing 400715, China

Reinforcement Learning With Verifiable Reward Flow-Grpo Credit Assignment

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), has advanced LLM reasoning. However, GRPO suffers from three credit assignment failures: uniform token-level granularity that ignores heterogeneous informational value, uniform polarity that penalizes correct steps and rewards incorrect ones, and zero-variance collapse that erases outcome-driven gradients. We systematically quantify these failures, revealing highly non-uniform token informativeness, widespread step-level polarity misalignment, and substantial training waste. To address these limitations, we propose Entropy-Progress Aligned GRPO (EP-GRPO), a framework that mines the model's intrinsic information flow for dense, self-supervised guidance. EP-GRPO integrates entropy-gated modulation to prioritize high entropy decision pivots, implicit process signals from policy divergence anchored to outcome advantages for directional token-level feedback without external reward models, and cumulative entropy mapping that enables progress-aligned advantage normalization, naturally maintaining gradient flow under zero reward variance. Extensive experiments on mathematical reasoning benchmarks demonstrate that EP-GRPO achieves superior accuracy and efficiency compared to GRPO and its variants. The code will be available.

Abstract

Explore similar work

EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare