Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation

Zhiqi Yu; Zhangquan Chen; Mengting Liu; Heye Zhang; Liangqiong Qu

cs.LGFeb 5, 2026

Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation

Authors: Zhiqi Yu, Zhangquan Chen, Mengting Liu, Heye Zhang, Liangqiong Qu

Organizations: University of Hong Kong · Tsinghua University · Sun Yat-sen University

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR), particularly GRPO, has become the standard for eliciting LLM reasoning. However, its efficiency in exploration and difficulty adaptation remains an open challenge. In this work, we identify an implicit advantage symmetry inherent in Group Relative Advantage Estimation (GRAE) as a structural property that provides a new perspective for understanding these bottlenecks. This symmetry induces two critical limitations: (i) at the group level, strict symmetry in weights between correct and incorrect trajectories leaves unsampled action logits unchanged, thereby hindering the exploration of novel correct solution. (ii) at the sample level, the algorithm implicitly prioritizes medium-difficulty samples, remaining agnostic to the non-stationary demands of difficulty focus. Through controlled experiments, we reveal that this symmetric property is sub-optimal, yielding two pivotal insights: (i) asymmetrically down-weighting the advantages of correct trajectories encourages essential exploration but risks instability; (ii) learning efficiency can be boosted by a curriculum-like transition-prioritizing simpler samples initially before gradually shifting to complex ones. Motivated by these findings, we propose Asymmetric GRAE (A-GRAE), which dynamically modulates exploration incentives and sample-difficulty focus. Experiments across seven benchmarks demonstrate that A-GRAE consistently improves GRPO and its variants across both LLMs and MLLMs. Code is available at https://github.com/HKU-HealthAI/A-GRAE

Figures & tables

Figure 1 : The two-fold implicit advantage symmetry problem of GRAE in GRPO. At the group level, the advantage weights for correct trajectories equal those of incorrect trajectories. This symmetry leads to the logits of low-probability correct paths unchanged under the advantage-driven update, thereby hindering the model’s exploration. At the sample level, medium-difficulty samples exhibit the largest sum of absolute advantage values, leading to insufficient training on harder data.

Table 1 : Summary of Controlled Experiments. We design two experiments to investigate the impact of symmetry at the group and sample levels.

\beta=10

and

\gamma=0.5

are scaling constants.

Panel of Figure 2 — Figure 2 : Experimental results on breaking group-level symmetry using Qwen2.5-Math-7B . We amplify (Positive-Dominant) or suppress (Negative-Dominant) the advantages of correct trajectories to compare their performance with those of GRPO and the base model.The performance is evaluated using Pass@ $k$ . The last figure illustrates the entropy dynamics on the training set.

Panel of Figure 3 — Figure 3 : Experimental results on breaking sample-level symmetry using Qwen2.5-Math-7B . We rescale the advantages to shift the learning focus toward harder queries (Hard-Focused) or easier queries (Easy-Focused) to compare their performance with those of GRPO and the base model.The performance is evaluated using Pass@ $k$ . The last figure illustrates the within-batch count of correct sampled responses on the training set.

Method	Pass@ $k$
$k$	1	2	4	8	16	32	64	128	256	Avg.
MATH
Base Model	63.4 $\pm 0.4$	74.8 $\pm 0.5$	83.2 $\pm 0.6$	88.6 $\pm 0.7$	91.2 $\pm 0.7$	93.4 $\pm 0.6$	94.1 $\pm 0.5$	95.0 $\pm 0.4$	96.3 $\pm 0.3$	86.7
SEC	77.6 $\pm 0.5$	82.5 $\pm 0.6$	86.0 $\pm 0.6$	88.9 $\pm 0.4$	90.2 $\pm 0.5$	92.2 $\pm 0.6$	92.6 $\pm 0.5$	93.3 $\pm 0.3$	94.7 $\pm 0.3$	88.7
GRPO-LEAD	77.8 $\pm 0.3$	83.0 $\pm 0.4$	86.5 $\pm 0.5$	89.2 $\pm 0.5$	90.5 $\pm 0.5$	92.3 $\pm 0.5$	92.8 $\pm 0.4$	93.6 $\pm 0.4$	95.0 $\pm 0.3$	89.0
P@kT	76.2 $\pm 0.6$	82.2 $\pm 0.8$	86.9 $\pm 0.6$	89.6 $\pm 0.4$	91.4 $\pm 0.3$	93.3 $\pm 0.4$	94.0 $\pm 0.5$	96.2 $\pm 0.4$	96.2 $\pm 0.4$	89.3

Table 2 : Pass@

k

results on MATH, AIME 2025 and AMC23 with Qwen2.5-Math-7B . Bold and underlined numbers denote the best and second-best results for each

k

. Specifically, We conduct 16 runs on AIME2025 and AMC23, and 4 runs on the remaining benchmarks.

	ID Domain	OOD Domain
Method	Geo3K	MathVision	Mathverse
Task A: General Mathematical Reasoning
Base Model	27.8 $\pm 1.0$	20.8 $\pm 0.8$	31.6 $\pm 0.7$
GRPO	43.5 $\pm 0.8$	23.4 $\pm 0.7$	35.2 $\pm 0.9$
GRPO + A-GRAE	45.7 $\pm 0.6$	24.0 $\pm 0.4$	36.8 $\pm 0.5$
DAPO	44.7 $\pm 0.9$	23.8 $\pm 0.6$	35.9 $\pm 0.8$

Table 3 : Results on multi-modal benchmarks with Qwen2.5-VL-3B-Instruct . We conduct 4 runs on each benchmark. Bold and underlined numbers denote the best and second-best results.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4 : Visualization of Update Magnitude with Respect to Sample Difficulty. The update magnitude reaches its peak at $p=0.5$ , simple samples (e.g., $p=0.75$ ) and hard samples (e.g., $p=0.25$ ) exhibiting the same deviation from $p=0.5$ are assigned identical importance weights.

Panel of Figure 5 — Figure 5 : Experimental results on breaking group-level symmetry using Llama-3.2-3B-Instruct . The performance is evaluated using Pass@ $k$ ( $k=\{1,2,4,8,16,32,64,128,256\}$ ) to provide a comprehensive analysis.

Panel of Figure 6 — Figure 6 : Experimental results on breaking sample-level symmetry using Llama-3.2-3B-Instruct . The performance is evaluated using Pass@ $k$ ( $k=\{1,2,4,8,16,32,64,128,256\}$ ) to provide a comprehensive analysis.

Method	Pass@ $k$
$k$	1	2	4	8	16	32	64	128	256	Avg.
MATH
Base Model	84.3 $\pm 0.5$	89.6 $\pm 0.6$	92.8 $\pm 0.5$	94.8 $\pm 0.4$	96.0 $\pm 0.4$	96.8 $\pm 0.3$	97.4 $\pm 0.3$	97.8 $\pm 0.3$	98.0 $\pm 0.2$	94.2
GRPO	92.4 $\pm 0.3$	95.3 $\pm 0.4$	96.5 $\pm 0.4$	97.0 $\pm 0.3$	97.3 $\pm 0.3$	97.7 $\pm 0.2$	98.0 $\pm 0.2$	98.3 $\pm 0.2$	98.6 $\pm 0.2$	96.8
GRPO + A-GRAE	92.9 $\pm 0.2$	95.7 $\pm 0.3$	96.9 $\pm 0.3$	97.5 $\pm 0.3$	97.9 $\pm 0.3$	98.3 $\pm 0.2$	98.6 $\pm 0.2$	99.0 $\pm 0.2$	99.3 $\pm 0.1$	97.3
DAPO	92.6 $\pm 0.3$	95.3 $\pm 0.4$	96.4 $\pm 0.4$	97.2 $\pm 0.3$	97.5 $\pm 0.3$	97.8 $\pm 0.3$	98.0 $\pm 0.2$	98.2 $\pm 0.2$	98.6 $\pm 0.2$	96.8

Appendix

Table 4 : Pass@

k

results on MATH, AIME 2025 and AMC23 with DeepSeek-R1-7B . Bold and underlined numbers denote the best and second-best results for each

k

. Specifically, We conduct 16 runs on AIME2025 and AMC23, and 4 runs on the remaining benchmarks.

Panel of Figure 7 — Figure 7 : Training collapse of Negative-Dominant group during training with Qwen2.5-Math-7B . Left Part: The unsolved question count suddenly increase in the later stage. Right Part: The collapse leads to the significant performance decrease.

Method	Pass@ $k$
$k$	1	2	4	8	16	32	64	128	256
MATH
Base Model	63.4	74.8	83.2	88.6	91.2	93.4	94.1	95.0	96.3
GRPO	76.5	82.3	86.1	88.8	90.3	92.6	93.5	93.9	95.0
GRPO + A-GRAE (sample level)	77.8	84.2	88.4	90.5	92.1	94.0	94.3	94.8	96.0
GRPO + A-GRAE (group level)	77.6	84.0	88.1	91.2	93.0	94.3	95.3	95.6	97.0

Appendix

Table 5 : Ablation studies on MATH, AIME 2025 and AMC23 with Qwen2.5-Math-7B .

Panel of Figure 8 — Figure 8 : Training dynamics using Qwen2.5-Math-7B . Left Part: Model’s entropy on the Math training set. Center Part: Model’s entropy (actor entropy loss) on the Math test set. Right Part: The greedy decoding accuracy on the Math test set.

Method	$k=1$	$k=2$	$k=4$	$k=8$	$k=16$	$k=256$
GRPO (Baseline)	76.5	82.3	86.1	88.8	90.3	92.4
Negative-Dominant	77.2	83.8	87.6	90.4	92.8	94.5
GRPO + Entropy-Promoting	75.8	80.3	83.6	86.5	88.4	89.0

Appendix

Table 6 : Comparison of MATH Pass@

k

(%) between Negative-Dominant group and entropy-promoting techniques.

Comparison Pair	$k=1$	$k=2$	$k=4$	$k=8$	$k=16$	$k=32$	$k=64$	$k=128$	$k=256$
MATH Benchmark
GRPO + A-GRAE vs. GRPO	.0032	.0018	.0025	.0041	.0084	.0115	.0128	.0155	.0121
DAPO + A-GRAE vs. DAPO	.0015	.0022	.0048	.0055	.0072	.0098	.0105	.0112	.0108
Dr.GRPO + A-GRAE vs. Dr.GRPO	.0024	.0009	.0011	.0022	.0015	.0028	.0035	.0042	.0038
AIME 2025 Benchmark
GRPO + A-GRAE vs. GRPO	.0125	.0098	.0112	.0105	.0142	.0128	.0115	.0092	.0085

Appendix

Table 7 : Detailed

p

-values across scaling intervals (

k=1

k=256

p

-values are calculated comparing the A-GRAE enhanced version against its respective vanilla baseline.

Method / Setting	MATH	AMC23	AIME2025
GRPO (Baseline)	76.5	59.2	10.3
$\alpha=0.25$	77.8	61.8	10.9
$\alpha=0.5$	77.8	61.8	11.0
$\alpha=1.0$	78.3	62.6	11.3
$\alpha=2.0$	78.0	62.8	11.3
$\alpha=5.0$	76.8	62.3	10.8

Appendix

Table 8 : Sensitivity analysis of

\alpha

on Qwen2.5-Math-7B. Performance is measured by Pass@1 (%).

Explore similar work

May 6, 2026cs.LG

EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance

Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), has advanced LLM reasoning. However, GRPO suffers from three credit assignment failures: uniform token-level granularity that ignores heterogeneous informational value, uniform polarity that penalizes correct steps and rewards incorrect ones, and zero-variance collapse that erases outcome-driven gradients. We systematically quantify these failures, revealing highly non-uniform token informativeness, widespread step-level polarity misalignment, and substantial training waste. To address these limitations, we propose Entropy-Progress Aligned GRPO (EP-GRPO), a framework that mines the model's intrinsic information flow for dense, self-supervised guidance. EP-GRPO integrates entropy-gated modulation to prioritize high entropy decision pivots, implicit process signals from policy divergence anchored to outcome advantages for directional token-level feedback without external reward models, and cumulative entropy mapping that enables progress-aligned advantage normalization, naturally maintaining gradient flow under zero reward variance. Extensive experiments on mathematical reasoning benchmarks demonstrate that EP-GRPO achieves superior accuracy and efficiency compared to GRPO and its variants. The code will be available.

Song Yu, Li Li, Wenwen Zhao +1

School of Computer and Information Science, Southwest University, Chongqing 400715, China

Reinforcement Learning With Verifiable Reward Flow-Grpo Credit Assignment

Ask PDF

May 17, 2026cs.LG

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning

Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose Mu-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation-optimization stages. This design induces high rollout staleness while greatly reducing rollout-optimization switching overhead. To stabilize learning under stale data, Mu-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds the performance of standard GRPO while achieving around 2x speedup in wall-clock training time, establishing a substantially improved performance-efficiency trade-off for LLM reinforcement learning.

Minghao Tian, Yunfei Xie, Chen Wei

Rice University

Large Language Model Reinforcement Learning Flow-Grpo Offline Reinforcement Learning Off-Policy Learning

Ask PDF

Feb 6, 2026cs.LG

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, computational limits often rule out very large groups, so training proceeds with finite rollout sets that can reinforce only the correct behavior they expose. At practical group sizes, updates can miss rare-correct trajectories while still containing mixed rewards, concentrating probability on more common sampled solutions. We derive the probability of such prompt-local tail-miss events as a function of group size, showing non-monotonic behavior, and in the categorical abstraction characterize how unsampled-correct mass can shrink even as total correct mass grows. Motivated by this analysis, we propose a difficulty-aware scaling coefficient, inspired by Focal loss, that down-weights updates on high-success sampled groups. Empirically, categorical simulation illustrates the same effect in the categorical setting, Maze provides a single-solution test, and LLM experiments include a representative GRPO group-size sweep together with fixed-

N

transfer across GRPO, DAPO, and CISPO. On Qwen2.5-7B at

N{=}8

, our method improves average math pass@256 from 64.1

\rightarrow

70.3 (GRPO), 69.3

\rightarrow

72.5 (DAPO), and 73.2

\rightarrow

76.8 (CISPO); OOD pass@256 also improves in all three cases, without increasing group size or computational cost.

Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4

T-Tech · Saint Petersburg Electrotechnical University “LETI”

Reinforcement Learning With Verifiable Reward Flow-Grpo Verifiable Rewards Single Trajectory

Ask PDF