Reinforcement learning with verifiable rewards has shifted the alignment paradigm of large language models from subjective preference to objective correctness, yet a fundamental disconnect persists between exploration diversity and reasoning validity. Existing strategies predominantly rely on "blind" entropy metrics, e.g., token-level or global sequence-level entropy, which structurally fail to distinguish between effective reasoning and potential hallucinations. This misalignment creates an illusion of diversity, where high entropy tends to signal chaotic degeneration rather than valuable exploration. To bridge this gap, we introduce Effective Entropy, a validity-aware entropy concept that quantifies diversity exclusively within the valid subspace and systematically filters out the noise of incorrect trajectories. Built on this concept, we instantiate VEPO, a simple Validity-aware Entropy Policy Optimization method that directly incorporates effective entropy into RLVR training. VEPO induces an adaptive attraction dynamic that drives up the probability of under-explored valid trajectories, a mechanism that matches the practical regime of large-scale autoregressive models where sequence probabilities remain extremely small. Experiments on six mathematical reasoning benchmarks with 1.5B and 7B DeepSeek-R1-Distill-Qwen models show consistent aggregate accuracy gains over GRPO and competitive repeated-sampling success. A direct evaluation with matched correct counts finds more distinct strategies on commonly solved MATH500 problems, supported by paired statistical tests and three-seed training comparisons.
Figures & tables
Figure 1: Exploration in RLVR. (a) Standard entropy: Conventional entropy operates on the whole solution space. Since the invalid subspace is combinatorially dominant, blind maximization tends to disperse probability toward diverse errors. (b) Effective entropy: We measure diversity only within the valid subspace. Filtering out invalid trajectories provides a validity-aware view of exploration and motivates a new optimization direction for reasoning alignment.
Figure 2: Overview of VEPO. Effective entropy filters exploration through the reward-defined valid subspace. VEPO directly adds this signal to GRPO, inducing an adaptive attraction mechanism that increases the probability of under-explored valid trajectories while ignoring invalid ones.
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Average
Method
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
Backbone: DeepSeek-R1-Distill-Qwen-1.5B
GRPO
33.2
71.8
27.7
49.9
79.5
94.8
77.6
90.8
26.1
48.8
46.3
64.7
48.4
70.1
Entropy-Reg
32.4
63.3
28.4
56.2
81.3
95.0
78.4
90.5
25.2
47.0
46.3
65.0
48.7
69.5
Entropy-Adv
29.6
67.8
23.9
44.2
77.1
94.1
78.5
89.7
25.2
48.6
46.4
64.3
46.8
68.1
Adv-Reweighting
25.0
59.1
22.3
40.7
70.6
94.6
75.6
90.1
25.1
51.1
42.3
63.2
43.5
66.5
Table 1: Main results on mathematical reasoning benchmarks. We report Avg@32 (A@32) and Pass@32 (P@32). Bold indicates the best performance within each backbone.
Figure 3: Main empirical results. Subfigures (a) and (d) show Pass@K on AIME 2025 for the 1.5B and 7B models. Subfigures (b) and (e) show Avg@32 and Pass@32 training dynamics on the 7B model. Subfigures (c) and (f) compare token entropy and effective entropy during training. Together, these results show that VEPO improves performance while sustaining stronger validity-aware exploration across different model scales.
Metric
GRPO
VEPO
Difference
Distinct clusters/problem
2.31
2.68
+0.37
Hill-1 effective modes
1.97
2.24
+0.27
Largest-cluster share
0.68
0.61
−0.07
Table 2: Direct strategy diversity on the 441 commonly eligible MATH500-7B problems. Each method contributes exactly eight correct trajectories per problem. Hill-1 is the effective number of strategy modes ( Hill, 1973 ) .
Figure 4: Impact of the effective entropy coefficient β . Larger β maintains stronger exploration but can delay convergence. A moderate coefficient provides the best balance between valid diversity and optimization efficiency across training stages.
Figure 5: Attribution analysis. Entropy-Reg increases raw entropy but does not reliably improve the structural quality of exploration. VEPO allocates more exploration to the valid subspace, leading to higher effective ratio and stronger conversion efficiency.
Table 3: Classification of entropy-aware algorithms based on their modification of update magnitudes relative to GRPO. The regions are defined by sample type, entropy level, and update action.
Targets the robustness of long-range reasoning chains; the 2025 version serves as a temporal hold-out to reduce the risk of data leakage.
AMC 2023
Benchmarks the model against standard competitive mathematics at the entry olympiad level.
Broad-Spectrum Knowledge (Domain Versatility)
MATH-500
Provides a curated cross-section of scholastic math to evaluate whether the policy maintains balanced performance across major mathematical sub-fields.
Minerva
Focuses on the transition from standard mathematical problems to complex, scientific-grade symbolic reasoning.
Appendix
Table 4: Detailed profile of the reasoning benchmarks. Datasets are grouped by their strategic role in evaluating the model’s latent reasoning capabilities.
Model Scale
Algorithm
Learning Rate
Key Hyperparameters
Common Settings: Global Batch Size = 512, Mini-batch Size = 32, Max Length = 8192, βKL=0
1.5B
GRPO
1.0×10−6
ε=0.2
Entropy-Reg
βreg=0.001
Entropy-Adv
αE=0.4,κ=2.0
Adv-Reweighting
αA=0.1
GTPO
α1=1.0,α2=0.1
Appendix
Table 5: Detailed hyperparameter configurations for VEPO and all baselines. For baselines, we utilized the values recommended in their respective original papers or tuned them for optimal performance on our validation set.
Figure 6: Comparative dynamics between Entropy-Reg and Adv-Reweighting on 7B model. While Adv-Reweighting maintains or even surges in total entropy, its effective entropy remains consistently below Entropy-Reg, resulting in inferior reasoning performance.
Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@k for large k. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as k scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.
Xingyu Shen, Huishuai Zhang, Peng Li +2
Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies
Reinforcement learning with verifiers (RLVR) is a central paradigm for improving large language model (LLM) reasoning, yet existing methods often suffer from limited exploration. Policies tend to collapse onto a few reasoning patterns and prematurely stop deep exploration, while conventional entropy regularization introduces only local stochasticity and fails to induce meaningful path-level diversity, leading to weak and unstable learning signals in group-based policy optimization. We propose DSDR, a Dual-Scale Diversity Regularization reinforcement learning framework that decomposes diversity in LLM reasoning into global and coupling components. Globally, DSDR promotes diversity among correct reasoning trajectories to explore distinct solution modes. Locally, it applies a length-invariant, token-level entropy regularization restricted to correct trajectories, preventing entropy collapse within each mode while preserving correctness. The two scales are coupled through a global-to-local allocation mechanism that emphasizes local regularization for more distinctive correct trajectories. We provide theoretical support showing that DSDR preserves optimal correctness under bounded regularization, sustains informative learning signals in group-based optimization, and yields a principled global-to-local coupling rule. Experiments on multiple reasoning benchmarks demonstrate consistent improvements in accuracy and pass@k, highlighting the importance of dual-scale diversity for deep exploration in RLVR. Code is available at https://github.com/SUSTechBruce/DSDR.
Zhongwei Wan, Yun Shen, Zhihao Dou +9
The Ohio State University, USA · Case Western Reserve University, USA · The Chinese University of Hong Kong, Hong Kong +4
Recent advances in reinforcement learning (RL) have improved the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs). However, the widely used Group Relative Policy Optimization (GRPO) consistently suffers from entropy collapse, causing the policy to converge prematurely and lose diversity. Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain optimization stability. We propose Unified Entropy Control for Reinforcement Learning (UEC-RL), a framework that provides targeted mechanisms for exploration and stabilization. UEC-RL activates more exploration on difficult prompts to search for potential and valuable reasoning trajectories. In parallel, a stabilizer prevents entropy from growing uncontrollably, thereby keeping training stable as the model consolidates reliable behaviors. Together, these components expand the search space when needed while maintaining robust optimization throughout training. Experiments on both LLM and VLM reasoning tasks show consistent gains over RL baselines on both Pass@1 and Pass@k. On Geometry3K, UEC-RL achieves a 37.9% relative improvement over GRPO, indicating that it sustains effective exploration without compromising convergence and underscoring UEC-RL as a key for scaling RL-based reasoning in large models. Our code is available at https://github.com/597358816/UEC-RL.
Chen Wang, Lai Wei, Yanzhi Zhang +5
College of Software, Nankai University · Zhongguancun Academy · Shanghai Jiao Tong University +3