Reinforcement learning with verifiable rewards has shifted the alignment paradigm of large language models from subjective preference to objective correctness, yet a fundamental disconnect persists between exploration diversity and reasoning validity. Existing strategies predominantly rely on "blind" entropy metrics, e.g., token-level or global sequence-level entropy, which structurally fail to distinguish between effective reasoning and potential hallucinations. This misalignment creates an illusion of diversity, where high entropy tends to signal chaotic degeneration rather than valuable exploration. To bridge this gap, we introduce Effective Entropy, a validity-aware entropy concept that quantifies diversity exclusively within the valid subspace and systematically filters out the noise of incorrect trajectories. Built on this concept, we instantiate VEPO, a simple Validity-aware Entropy Policy Optimization method that directly incorporates effective entropy into RLVR training. VEPO induces an adaptive attraction dynamic that drives up the probability of under-explored valid trajectories, a mechanism that matches the practical regime of large-scale autoregressive models where sequence probabilities remain extremely small. Experiments on six mathematical reasoning benchmarks with 1.5B and 7B DeepSeek-R1-Distill-Qwen models show consistent aggregate accuracy gains over GRPO and competitive repeated-sampling success. A direct evaluation with matched correct counts finds more distinct strategies on commonly solved MATH500 problems, supported by paired statistical tests and three-seed training comparisons.
Figures & tables
Figure 1: Exploration in RLVR. (a) Standard entropy: Conventional entropy operates on the whole solution space. Since the invalid subspace is combinatorially dominant, blind maximization tends to disperse probability toward diverse errors. (b) Effective entropy: We measure diversity only within the valid subspace. Filtering out invalid trajectories provides a validity-aware view of exploration and motivates a new optimization direction for reasoning alignment.
Figure 2: Overview of VEPO. Effective entropy filters exploration through the reward-defined valid subspace. VEPO directly adds this signal to GRPO, inducing an adaptive attraction mechanism that increases the probability of under-explored valid trajectories while ignoring invalid ones.
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Average
Method
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
A@32
P@32
Backbone: DeepSeek-R1-Distill-Qwen-1.5B
GRPO
33.2
71.8
27.7
49.9
79.5
94.8
77.6
90.8
26.1
48.8
46.3
64.7
48.4
70.1
Entropy-Reg
32.4
63.3
28.4
56.2
81.3
95.0
78.4
90.5
25.2
47.0
46.3
65.0
48.7
69.5
Entropy-Adv
29.6
67.8
23.9
44.2
77.1
94.1
78.5
89.7
25.2
48.6
46.4
64.3
46.8
68.1
Adv-Reweighting
25.0
59.1
22.3
40.7
70.6
94.6
75.6
90.1
25.1
51.1
42.3
63.2
43.5
66.5
Table 1: Main results on mathematical reasoning benchmarks. We report Avg@32 (A@32) and Pass@32 (P@32). Bold indicates the best performance within each backbone.
Figure 3: Main empirical results. Subfigures (a) and (d) show Pass@K on AIME 2025 for the 1.5B and 7B models. Subfigures (b) and (e) show Avg@32 and Pass@32 training dynamics on the 7B model. Subfigures (c) and (f) compare token entropy and effective entropy during training. Together, these results show that VEPO improves performance while sustaining stronger validity-aware exploration across different model scales.
Metric
GRPO
VEPO
Difference
Distinct clusters/problem
2.31
2.68
+0.37
Hill-1 effective modes
1.97
2.24
+0.27
Largest-cluster share
0.68
0.61
−0.07
Table 2: Direct strategy diversity on the 441 commonly eligible MATH500-7B problems. Each method contributes exactly eight correct trajectories per problem. Hill-1 is the effective number of strategy modes ( Hill, 1973 ) .
Figure 4: Impact of the effective entropy coefficient β . Larger β maintains stronger exploration but can delay convergence. A moderate coefficient provides the best balance between valid diversity and optimization efficiency across training stages.
Figure 5: Attribution analysis. Entropy-Reg increases raw entropy but does not reliably improve the structural quality of exploration. VEPO allocates more exploration to the valid subspace, leading to higher effective ratio and stronger conversion efficiency.
Table 3: Classification of entropy-aware algorithms based on their modification of update magnitudes relative to GRPO. The regions are defined by sample type, entropy level, and update action.
Targets the robustness of long-range reasoning chains; the 2025 version serves as a temporal hold-out to reduce the risk of data leakage.
AMC 2023
Benchmarks the model against standard competitive mathematics at the entry olympiad level.
Broad-Spectrum Knowledge (Domain Versatility)
MATH-500
Provides a curated cross-section of scholastic math to evaluate whether the policy maintains balanced performance across major mathematical sub-fields.
Minerva
Focuses on the transition from standard mathematical problems to complex, scientific-grade symbolic reasoning.
Appendix
Table 4: Detailed profile of the reasoning benchmarks. Datasets are grouped by their strategic role in evaluating the model’s latent reasoning capabilities.
Model Scale
Algorithm
Learning Rate
Key Hyperparameters
Common Settings: Global Batch Size = 512, Mini-batch Size = 32, Max Length = 8192, βKL=0
1.5B
GRPO
1.0×10−6
ε=0.2
Entropy-Reg
βreg=0.001
Entropy-Adv
αE=0.4,κ=2.0
Adv-Reweighting
αA=0.1
GTPO
α1=1.0,α2=0.1
Appendix
Table 5: Detailed hyperparameter configurations for VEPO and all baselines. For baselines, we utilized the values recommended in their respective original papers or tuned them for optimal performance on our validation set.
Figure 6: Comparative dynamics between Entropy-Reg and Adv-Reweighting on 7B model. While Adv-Reweighting maintains or even surges in total entropy, its effective entropy remains consistently below Entropy-Reg, resulting in inferior reasoning performance.