ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models

Song Yu; Li Li; Wenwen Zhao; Zhisheng Yang

cs.LGMar 30, 2026

ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models

Authors: Song Yu, Li Li, Wenwen Zhao, Zhisheng Yang

Organizations: School of Computer and Information Science, Southwest University, Chongqing, 400715, China

Abstract

Reinforcement learning from verifiable rewards has significantly advanced the reasoning capabilities of large language models. However, Group Relative Policy Optimization (GRPO) typically assigns a uniform, sequence-level advantage to all tokens, thereby overlooking the intrinsic information heterogeneity along reasoning chains. We show that this coarse-grained credit assignment leads to premature entropy collapse and encourages the model to generate redundant, low-quality reasoning paths. Through systematic empirical analysis, we identify Critical Decision Pivots (CDPs): transient high-entropy states where the policy's trajectory is most sensitive to perturbations. These pivots represent the "forks in the road" where effective multi-path exploration is most crucial yet often suppressed by uniform advantage signals. Building on these insights, we propose Entropy-Regulated Policy Optimization (ERPO), which transitions the optimization focus from coarse sequences to fine-grained token dynamics. ERPO introduces three synergistic components: (i) Entropy-aware Gating, which adaptively amplifies exploration at CDPs to facilitate diverse path discovery; (ii) Bucket-based Implicit Normalization, which mitigates difficulty bias by aligning token progress windows; and (iii) Result-anchored Advantage Synthesis, which re-weights token-level signals via outcome-driven anchors. Extensive experiments on competitive mathematical benchmarks demonstrate that ERPO significantly outperforms GRPO. Notably, ERPO not only boosts reasoning accuracy but also yields significantly more concise and robust derivation paths, while achieving performance comparable to large models with orders of magnitude more parameters.

Figures & tables

Panel of Figure 1 — Figure 1: Token-level entropy distribution and its impact on performance based on Qwen2.5-3B with AMC23 dataset. (a) The LLM exhibits high entropy at high-entropy states (red), indicating that the model is making logical branch decisions, and low entropy at the inference process (yellow), indicating that the model is performing deterministic execution steps. (b) We sampled 50 questions that the model could definitely answer correctly and randomly perturbed the top 5% of high-entropy tokens and the bottom 5% of low-entropy tokens for each sequence. Perturbing these high-entropy hubs resulted in a significant decrease in final accuracy $(p<0.001)$ , confirming their crucial role in inference.

Figure 2: The overall architecture of the ERPO. The process begins with sampling

G

sequences and computing the sequence-level baseline advantage

A

via a reward model. Our approach refines this by deriving a token-level entropy weight matrix

\mathcal{H}

through intra-group normalization and an implicit progress matrix

S

via temporal bucketing and localized normalization of log-probability ratios. These components are integrated using the element-wise product

\hat{A}=\mathcal{H}\odot S\odot\text{sgn}(A)

, where the outcome sign

\text{sgn}(A)

anchors the fine-grained signal to the final result. The final token-level advantage matrix

\mathcal{A}

is synthesized by merging the baseline and process-oriented advantages, enabling precise credit assignment for complex reasoning tasks.

	AMC 23		Minerva		AIME 24		AIME 25		Average
Model	Acc	Fmt	Acc	Fmt	Acc	Fmt	Acc	Fmt	Acc	Fmt
Commercial Baselines
DeepSeek-R1-671B-0528	33.91	33.91	11.41	34.38	13.54	13.54	11.04	11.04	17.48	23.22
Qwen3-235B-A22B-Instr.	47.81	55.00	17.66	92.34	24.58	29.58	16.88	20.21	26.73	49.28
1.5B Scale
Qwen2.5-1.5B ${}_{\text{Base}}$	0.78	27.97	0.31	27.97	0.21	28.33	0.00	26.67	0.33	27.74

Table 1: Performance evaluation of Qwen2.5 models across reasoning benchmarks. Acc (%) and Fmt (%) denote sample accuracy and boxed rate. Benchmarks are ordered by difficulty: AMC 23, Minerva, AIME 24, and AIME 25. Bold values indicate the best performance within each parameter scale, and underlined values indicate the second best.

	AMC 23				Minerva				AIME 24				AIME 25
Model	@2	@4	@8	@16	@2	@4	@8	@16	@2	@4	@8	@16	@2	@4	@8	@16
Commercial Baselines
DeepSeek-R1-671B-0528	47.62	58.72	66.19	72.50	16.35	20.94	23.78	25.00	21.03	28.07	32.14	33.33	17.33	23.68	28.82	33.33
Qwen3-235B-A22B-Instr.	54.35	59.66	63.00	65.00	19.50	21.44	23.40	25.00	27.03	28.32	30.00	33.33	20.03	22.21	24.22	26.67
1.5B Scale
Qwen2.5-1.5B ${}_{\text{Base}}$	1.56	3.12	6.25	12.50	0.62	1.25	2.50	5.00	0.42	0.83	1.67	3.33	0.00	0.00	0.00	0.00

Table 2: Comprehensive

pass@k

(

k\in\{2,4,8,16\}

) performance evaluation. Bold values indicate the best performance within each parameter scale, and underlined values indicate the second best.

Panel of Figure 3 — Figure 3: Comparison of training efficiency and generalization performance between GRPO (Baseline) and ERPO (Ours) across three model scales (1.5B, 3B, 7B). Each row presents the sample accuracy (%) on AMC23, Minerva, AIME24, and AIME25 benchmarks, smoothed with EMA ( $\alpha=0.2$ ).

Figure 4: Training dynamics of ERPO vs. GRPO. We visualize the (a) Reward, (b) Entropy, (c) Grad Norm, and (d) KL Divergence. Note that the Entropy axis uses a symlog scale to highlight the significant difference in the late training stage (

0.4

vs.

0.05

), demonstrating that ERPO effectively prevents mode collapse. All curves are smoothed with EMA (

\alpha=0.12

) while raw data is shown in light colors.

Panel of Figure 5 — Figure 5: Comprehensive efficiency and training dynamics analysis for Qwen2.5-7B. Top row: (a) compares reasoning conciseness at the best checkpoints; (b) evaluates computational overhead. Bottom row: (c) displays the stability of token generation length across four benchmarks. ERPO achieves superior performance with significantly more concise reasoning paths and comparable training time to GRPO.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Configuration	Value
Max Training Length	2,048
Epochs	1
Global Batch Size	16
Gradient Accumulation Steps	8
Generations per Prompt ( $G$ )	8
Temperature (Training)	1.0

Appendix

Table 3: Hyperparameters for GRPO and ERPO training.

Explore similar work

EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance
May 6, 2026Song Yu, Li Li, Wenwen Zhao +1RL for Language Model Reasoning Group Relative Policy Optimization
School of Computer and Information Science, Southwest University, Chongqing 400715, China
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
Apr 14, 2026Xingyu Lin, Yilin Wen, Du Su +5RL for Language Model Reasoning Policy Optimization
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University · Baidu Inc. +2
Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting
May 12, 2026Cheng Wang, Qin Liu, Wenxuan Zhou +1Reinforcement Learning RL for Language Model Reasoning
National University of Singapore · University of California, Davis · University of Southern California

May 6, 2026J/K move · Enter open · S save

EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance

Song Yu, Li Li, Wenwen Zhao, Zhisheng Yang

School of Computer and Information Science, Southwest University, Chongqing 400715, China

RL for Language Model Reasoning Group Relative Policy Optimization Credit Assignment in RL Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), has advanced LLM reasoning. However, GRPO suffers from three credit assignment failures: uniform token-level granularity that ignores heterogeneous informational value, uniform polarity that penalizes correct steps and rewards incorrect ones, and zero-variance collapse that erases outcome-driven gradients. We systematically quantify these failures, revealing highly non-uniform token informativeness, widespread step-level polarity misalignment, and substantial training waste. To address these limitations, we propose Entropy-Progress Aligned GRPO (EP-GRPO), a framework that mines the model's intrinsic information flow for dense, self-supervised guidance. EP-GRPO integrates entropy-gated modulation to prioritize high entropy decision pivots, implicit process signals from policy divergence anchored to outcome advantages for directional token-level feedback without external reward models, and cumulative entropy mapping that enables progress-aligned advantage normalization, naturally maintaining gradient flow under zero reward variance. Extensive experiments on mathematical reasoning benchmarks demonstrate that EP-GRPO achieves superior accuracy and efficiency compared to GRPO and its variants. The code will be available.

Abstract

Explore similar work

EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance

Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood

Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting