Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning ability of large language models (LLMs), but it often suffers from restricted exploration, where the policy rapidly concentrates on a narrow set of solutions. A common remedy is entropy regularization, which attempts to preserve exploration by increasing policy entropy. However, for LLM-RL, this intervention is highly sensitive to its coefficient, can introduce semantically weak uncertainty, and often yields limited accuracy gains. This motivates a more precise question: which entropy helps reasoning, and which entropy should be reduced? To study this, we parameterize the advantage estimator in Group Relative Policy Optimization (GRPO) into positive and negative outcome-conditioned channels and analyze their entropy dynamics. Our results show that positive-channel modulation raises productive entropy associated with successful reasoning trajectories, while negative-channel modulation removes noisy entropy associated with failed rollouts and reduces interference with correct paths. Guided by this channel-wise view, we propose AsymGRPO, which decouples the modulation strengths of positive and negative advantages. This enables flexible control over how the model updates across prompt difficulty levels, allowing stronger reinforcement of rare successes on harder prompts or stronger suppression of residual failures on easier prompts without forcing the two channels to share the same modulation strength. Experiments on five mathematical reasoning benchmarks show that AsymGRPO outperforms strong RLVR baselines, with consistent gains across model backbones.
Figures & tables
Figure 1: Group-relative advantage magnitudes as functions of group accuracy under the parametric family in Eq. ( 5 ): (a) successful rollouts and (b) failed rollouts.
Model
κB
Qwen2.5-Math-1.5B
16.26
Qwen3-4B
1.91
Qwen2.5-7B-Base
6.95
Table 1: Estimated κB over the first 20 training steps (scaled by 103 ).
Figure 2: Training dynamics and channel-wise diagnostics. The top row reports results on Qwen2.5-Math-1.5B , while the bottom row reports results on Qwen3-4B . (a, e) Policy entropy over training steps. (b, f) Average test accuracy. (c, g) Proportions of prompt groups categorized as “all-solved” and “none-solved”. (d, h) Average log-probability gain of successful rollouts after each policy update.
Figure 3: Advantage Flip Test. (a, c) Policy entropy. (b, d) Average test accuracy.
Method
MATH-500
AIME24
AIME25
AMC23
Olympiad
Avg.
Qwen3-4B
81.60
21.67
20.00
63.75
47.52
46.91
REINFORCE
86.60
28.67
24.67
73.75
54.86
53.71
GRPO ( Guo et al., 2025 )
88.20
31.00
27.33
78.25
57.74
56.50
GRPO w/ Entro.Regularization
88.20
38.33
28.33
75.50
57.24
57.52
GRPO w/ Clip-higher ( Yu et al., 2025 )
90.07
34.67
32.33
78.50
58.18
58.75
Dr.GRPO ( Liu et al., 2025 )
88.87
36.33
30.00
78.25
57.24
58.14
Table 2: Main experimental results on mathematical reasoning benchmarks (top: full comparison on Qwen3-4B; bottom: generalization across model backbones). The best result in each column is shown in bold , and the second-best is underlined .
Figure 4: Training dynamics and hyperparameter landscape on Qwen3-4B. (a) Average training reward. (b) Policy entropy. (c) Average test accuracy over training steps. (d) Average benchmark accuracy across the (βpos,βneg) grid. All panels use Qwen3-4B as the backbone.
Reinforcement learning with verifiable rewards (RLVR) improves large language model reasoning but often suffers from rapid policy-entropy collapse, where the policy prematurely concentrates on narrow high-probability reasoning paths. While global entropy regularization can encourage exploration, uniformly increasing entropy across all token positions is inefficient for long reasoning trajectories, where many tokens are not decision-relevant. We propose Position-Aware Entropy Calibration (PAEC), a token-level entropy-management framework that constructs a soft mask from local top-p entropy and top-two candidate competition, and applies an anchor-based lower-bound penalty to prevent selected-position entropy collapse. Experiments on five mathematical reasoning benchmarks show that PAEC improves macro-average majority-vote performance over strong RLVR baselines, with clear gains on AIME-style tasks. Our results suggest that entropy management in reasoning RL should be formulated as selective exploration allocation over decision-sensitive positions rather than uniform randomness injection.
Shumeng Yang, Yisu Liu, Jiayi Zheng +2
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · School of Artificial Intelligence, Beijing University of Posts and Telecommunications +2
Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), has advanced LLM reasoning. However, GRPO suffers from three credit assignment failures: uniform token-level granularity that ignores heterogeneous informational value, uniform polarity that penalizes correct steps and rewards incorrect ones, and zero-variance collapse that erases outcome-driven gradients. We systematically quantify these failures, revealing highly non-uniform token informativeness, widespread step-level polarity misalignment, and substantial training waste. To address these limitations, we propose Entropy-Progress Aligned GRPO (EP-GRPO), a framework that mines the model's intrinsic information flow for dense, self-supervised guidance. EP-GRPO integrates entropy-gated modulation to prioritize high entropy decision pivots, implicit process signals from policy divergence anchored to outcome advantages for directional token-level feedback without external reward models, and cumulative entropy mapping that enables progress-aligned advantage normalization, naturally maintaining gradient flow under zero reward variance. Extensive experiments on mathematical reasoning benchmarks demonstrate that EP-GRPO achieves superior accuracy and efficiency compared to GRPO and its variants. The code will be available.
Song Yu, Li Li, Wenwen Zhao +1
School of Computer and Information Science, Southwest University, Chongqing 400715, China
Reinforcement Learning with Verifiable Rewards (RLVR), particularly GRPO, has become the standard for eliciting LLM reasoning. However, its efficiency in exploration and difficulty adaptation remains an open challenge. In this work, we identify an implicit advantage symmetry inherent in Group Relative Advantage Estimation (GRAE) as a structural property that provides a new perspective for understanding these bottlenecks. This symmetry induces two critical limitations: (i) at the group level, strict symmetry in weights between correct and incorrect trajectories leaves unsampled action logits unchanged, thereby hindering the exploration of novel correct solution. (ii) at the sample level, the algorithm implicitly prioritizes medium-difficulty samples, remaining agnostic to the non-stationary demands of difficulty focus. Through controlled experiments, we reveal that this symmetric property is sub-optimal, yielding two pivotal insights: (i) asymmetrically down-weighting the advantages of correct trajectories encourages essential exploration but risks instability; (ii) learning efficiency can be boosted by a curriculum-like transition-prioritizing simpler samples initially before gradually shifting to complex ones. Motivated by these findings, we propose Asymmetric GRAE (A-GRAE), which dynamically modulates exploration incentives and sample-difficulty focus. Experiments across seven benchmarks demonstrate that A-GRAE consistently improves GRPO and its variants across both LLMs and MLLMs. Code is available at https://github.com/HKU-HealthAI/A-GRAE
Zhiqi Yu, Zhangquan Chen, Mengting Liu +2
University of Hong Kong · Tsinghua University · Sun Yat-sen University