Reinforcement learning from verifiable rewards has significantly advanced the reasoning capabilities of large language models. However, Group Relative Policy Optimization (GRPO) typically assigns a uniform, sequence-level advantage to all tokens, thereby overlooking the intrinsic information heterogeneity along reasoning chains. We show that this coarse-grained credit assignment leads to premature entropy collapse and encourages the model to generate redundant, low-quality reasoning paths. Through systematic empirical analysis, we identify Critical Decision Pivots (CDPs): transient high-entropy states where the policy's trajectory is most sensitive to perturbations. These pivots represent the "forks in the road" where effective multi-path exploration is most crucial yet often suppressed by uniform advantage signals. Building on these insights, we propose Entropy-Regulated Policy Optimization (ERPO), which transitions the optimization focus from coarse sequences to fine-grained token dynamics. ERPO introduces three synergistic components: (i) Entropy-aware Gating, which adaptively amplifies exploration at CDPs to facilitate diverse path discovery; (ii) Bucket-based Implicit Normalization, which mitigates difficulty bias by aligning token progress windows; and (iii) Result-anchored Advantage Synthesis, which re-weights token-level signals via outcome-driven anchors. Extensive experiments on competitive mathematical benchmarks demonstrate that ERPO significantly outperforms GRPO. Notably, ERPO not only boosts reasoning accuracy but also yields significantly more concise and robust derivation paths, while achieving performance comparable to large models with orders of magnitude more parameters.
Figures & tables
Figure 1: Token-level entropy distribution and its impact on performance based on Qwen2.5-3B with AMC23 dataset. (a) The LLM exhibits high entropy at high-entropy states (red), indicating that the model is making logical branch decisions, and low entropy at the inference process (yellow), indicating that the model is performing deterministic execution steps. (b) We sampled 50 questions that the model could definitely answer correctly and randomly perturbed the top 5% of high-entropy tokens and the bottom 5% of low-entropy tokens for each sequence. Perturbing these high-entropy hubs resulted in a significant decrease in final accuracy (p<0.001) , confirming their crucial role in inference.
Figure 2: The overall architecture of the ERPO. The process begins with sampling G sequences and computing the sequence-level baseline advantage A via a reward model. Our approach refines this by deriving a token-level entropy weight matrix H through intra-group normalization and an implicit progress matrix S via temporal bucketing and localized normalization of log-probability ratios. These components are integrated using the element-wise product A^=H⊙S⊙sgn(A) , where the outcome sign sgn(A) anchors the fine-grained signal to the final result. The final token-level advantage matrix A is synthesized by merging the baseline and process-oriented advantages, enabling precise credit assignment for complex reasoning tasks.
AMC 23
Minerva
AIME 24
AIME 25
Average
Model
Acc
Fmt
Acc
Fmt
Acc
Fmt
Acc
Fmt
Acc
Fmt
Commercial Baselines
DeepSeek-R1-671B-0528
33.91
33.91
11.41
34.38
13.54
13.54
11.04
11.04
17.48
23.22
Qwen3-235B-A22B-Instr.
47.81
55.00
17.66
92.34
24.58
29.58
16.88
20.21
26.73
49.28
1.5B Scale
Qwen2.5-1.5B Base
0.78
27.97
0.31
27.97
0.21
28.33
0.00
26.67
0.33
27.74
Table 1: Performance evaluation of Qwen2.5 models across reasoning benchmarks. Acc (%) and Fmt (%) denote sample accuracy and boxed rate. Benchmarks are ordered by difficulty: AMC 23, Minerva, AIME 24, and AIME 25. Bold values indicate the best performance within each parameter scale, and underlined values indicate the second best.
AMC 23
Minerva
AIME 24
AIME 25
Model
@2
@4
@8
@16
@2
@4
@8
@16
@2
@4
@8
@16
@2
@4
@8
@16
Commercial Baselines
DeepSeek-R1-671B-0528
47.62
58.72
66.19
72.50
16.35
20.94
23.78
25.00
21.03
28.07
32.14
33.33
17.33
23.68
28.82
33.33
Qwen3-235B-A22B-Instr.
54.35
59.66
63.00
65.00
19.50
21.44
23.40
25.00
27.03
28.32
30.00
33.33
20.03
22.21
24.22
26.67
1.5B Scale
Qwen2.5-1.5B Base
1.56
3.12
6.25
12.50
0.62
1.25
2.50
5.00
0.42
0.83
1.67
3.33
0.00
0.00
0.00
0.00
Table 2: Comprehensive pass@k ( k∈{2,4,8,16} ) performance evaluation. Bold values indicate the best performance within each parameter scale, and underlined values indicate the second best.
Figure 3: Comparison of training efficiency and generalization performance between GRPO (Baseline) and ERPO (Ours) across three model scales (1.5B, 3B, 7B). Each row presents the sample accuracy (%) on AMC23, Minerva, AIME24, and AIME25 benchmarks, smoothed with EMA ( α=0.2 ).
Figure 4: Training dynamics of ERPO vs. GRPO. We visualize the (a) Reward, (b) Entropy, (c) Grad Norm, and (d) KL Divergence. Note that the Entropy axis uses a symlog scale to highlight the significant difference in the late training stage ( 0.4 vs. 0.05 ), demonstrating that ERPO effectively prevents mode collapse. All curves are smoothed with EMA ( α=0.12 ) while raw data is shown in light colors.
Figure 5: Comprehensive efficiency and training dynamics analysis for Qwen2.5-7B. Top row: (a) compares reasoning conciseness at the best checkpoints; (b) evaluates computational overhead. Bottom row: (c) displays the stability of token generation length across four benchmarks. ERPO achieves superior performance with significantly more concise reasoning paths and comparable training time to GRPO.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Configuration
Value
Max Training Length
2,048
Epochs
1
Global Batch Size
16
Gradient Accumulation Steps
8
Generations per Prompt ( G )
8
Temperature (Training)
1.0
Appendix
Table 3: Hyperparameters for GRPO and ERPO training.
Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), has advanced LLM reasoning. However, GRPO suffers from three credit assignment failures: uniform token-level granularity that ignores heterogeneous informational value, uniform polarity that penalizes correct steps and rewards incorrect ones, and zero-variance collapse that erases outcome-driven gradients. We systematically quantify these failures, revealing highly non-uniform token informativeness, widespread step-level polarity misalignment, and substantial training waste. To address these limitations, we propose Entropy-Progress Aligned GRPO (EP-GRPO), a framework that mines the model's intrinsic information flow for dense, self-supervised guidance. EP-GRPO integrates entropy-gated modulation to prioritize high entropy decision pivots, implicit process signals from policy divergence anchored to outcome advantages for directional token-level feedback without external reward models, and cumulative entropy mapping that enables progress-aligned advantage normalization, naturally maintaining gradient flow under zero reward variance. Extensive experiments on mathematical reasoning benchmarks demonstrate that EP-GRPO achieves superior accuracy and efficiency compared to GRPO and its variants. The code will be available.
Song Yu, Li Li, Wenwen Zhao +1
School of Computer and Information Science, Southwest University, Chongqing 400715, China
Group Relative Policy Optimization (GRPO) has significantly advanced the reasoning ability of large language models (LLMs), particularly in their mathemat ical reasoning performance. However, GRPO and related entropy regularization methods still struggle with token-level sparse-rewards, which is an inherent chal lenge in chain-of-thought (CoT) reasoning. These approaches often rely on undifferen tiated token-level entropy regularization, which easily leads to entropy collapse or model degradation under sparse token rewards. In this work, we propose TEPO, a novel token-level framework that (1) leverages sequence-level likelihood to link group-level rewards with individual tokens via token-level aggregation, and (2) introduces a token-level KL-Divergence mask constraint that targets tokens with positive advantages and decreasing entropy to mitigate abrupt policy updates. Experiments demonstrate that TEPO not only achieves state-of-the-art performance on mathematical reasoning benchmarks but also markedly enhances training stability, reducing convergence time by 50% compared with GRPO/DAPO.
Xingyu Lin, Yilin Wen, Du Su +5
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University · Baidu Inc. +2
Group Relative Policy Optimization (GRPO) has emerged as a promising approach for improving the reasoning capabilities of large language models. However, it struggles to effectively balance the tradeoff between exploration and exploitation during training, often resulting in suboptimal performance. Motivated by the theoretical insight that changes in entropy are governed by the covariance between token probabilities and their corresponding advantages, we propose a hyperparameter-free, covariance-weighted optimization method that dynamically down-weights extreme token-level updates via a Gaussian kernel. This approach automatically reduces the instability caused by exploration-exploitation trade-off while preserving informative learning signals. Extensive empirical evaluations show that our approach improves downstream performance across reasoning benchmarks compared with GRPO, and effectively stablizes entropy as training progresses.
Cheng Wang, Qin Liu, Wenxuan Zhou +1
National University of Singapore · University of California, Davis · University of Southern California