Correlation-Aware Exploration for Long-Trace Reasoning

Junxing Hu; Zicheng Zhang; Zhiqian Zhang; Ai Han; Yifeng Zhang; Pengzhang Liu; Ling Li

cs.LGOct 26, 2026

Correlation-Aware Exploration for Long-Trace Reasoning

Authors: Junxing Hu, Zicheng Zhang, Zhiqian Zhang, Ai Han, Yifeng Zhang, Pengzhang Liu, Ling Li

Organizations: Microsoft

Abstract

Reasoning models trained with Group Relative Policy Optimization (GRPO) now produce chains of thought thousands of tokens long, and recent studies reveal that such training is prone to policy collapse. Entropy regularization is the standard remedy, but it works by pushing probability mass away from the few tokens that carry it and toward the vast low-probability tail of the vocabulary, and it applies this force once for every token of the reasoning trace. This puts the bonus in direct conflict with the entropy decay that convergence requires, and makes its coefficient difficult to set in the long-trace regime. To resolve this tension, we propose Info-GRPO, an information-theoretic framework that cultivates correlation between the policy and a latent prior. Rather than raising entropy everywhere, Info-GRPO builds on the mutual information between a latent variable and the trajectory it induces: augmenting prompts with latent seeds during training lets the model explore diverse policies correlated with the prior, while the seed-conditioned entropy is guided toward convergence. Diversity is therefore carried across whole reasoning traces rather than by per-token randomness, and exploration no longer fights entropy reduction. Experiments show that Info-GRPO outperforms vanilla GRPO and entropy-regularized GRPO across diverse reasoning benchmarks. On AIME24, it improves $Avg@8$ over GRPO by 3.75%, 1.66%, 4.16%, and 4.17% on four backbones, and it reaches the same majority-vote accuracy with $8\times$ fewer samples. Code and models will be publicly released.

Figures & tables

Figure 1: (a) Validation results on Qwen2.5-Math-7B show that Info-GRPO significantly outperforms the GRPO baseline. (b) Info-GRPO encourages mild high-entropy policies from original prompts and low-entropy policies from seed-augmented prompts. (c) The high correlation between random seeds and their respective outputs shows that generation is reliably steered by the seed it is conditioned on.

Figure 2: (a) An entropy bonus can degrade the policy into a uniform distribution, reaching its entropy upper bound. (b) This instability is inherent to LLMs, whose vocabularies are dominated by a vast tail of low-probability tokens, on which the entropy objective’s logarithmic factor is largest (see Proposition 1 ). Statistically, in Qwen2.5-7B’s 152,064 tokens, fewer than five exceed a $10^{-6}$ probability. This makes direct entropy maximization sensitive to its coefficient at this scale.

Figure 3: An intuitive comparison of exploration methods. Plus (+) and minus (-) signs represent forces that increase or decrease policy entropy. (a) Vanilla GRPO focuses on exploitation (-), often causing policy collapse. (b) Entropy regularization promotes diversity by increasing all entropy (+), but risks collapsing towards a uniform distribution. (c) Info-GRPO balances a mild entropy increase (+) with a strong, seed-conditioned entropy decrease (-). This follows the mutual-information decomposition between seeds and outcomes, using correlation to drive stable exploration.

Backbone	Method	AIME24	AIME25	AMC	MATH500	Minerva	Average
Qwen2.5-7B	Base Model	12.08	7.50	42.77	75.50	35.06	34.58
	GRPO	20.42	10.83	56.48	80.53	37.82	41.22
	DAPO	18.75	11.25	48.95	78.13	36.53	38.72
	CLIP-Cov	15.83	12.08	51.51	79.10	38.19	39.34
	KL-Cov	15.40	11.25	59.94	74.05	35.94	39.32
	Skywork-OR1	20.83	7.50	57.38	78.38	38.05	40.43

Table 1: Comparisons on multiple backbones and benchmarks (

Avg@8

, %). ‘

{{\dagger}}

’ uses the official model. ( bold = best, underline = second best). Additional comparisons are provided in Appendix A.4 .

Method	$Pass@1$	$Pass@8$	$Avg@8$
Base Model	36.92	48.75	36.74
GRPO	37.63	50.18	38.13
Info-GRPO (Ours)	40.86	51.97	40.14

Table 2: Code benchmark (LiveCodeBench) comparisons using DeepSeek-R1-Distill-Qwen-7B (%).

Backbone	Samples (N)	4	8	16	32	64	128
Qwen2.5-Math-7B	GRPO	23.33	16.67	16.67	23.33	23.33	26.67
Qwen2.5-Math-7B	Info-GRPO (Ours)	23.33	30.00	26.67	26.67	26.67	26.67
DeepSeek-R1-Distill-Qwen-7B	GRPO	56.67	76.67	73.33	76.67	80.00	80.00
DeepSeek-R1-Distill-Qwen-7B	Info-GRPO (Ours)	63.33	80.00	80.00	80.00	80.00	80.00

Table 3: Comparison with baselines using multiple backbones on AIME24 (

MV@N

, %).

Table 7

Figure 4: Comprehensive analysis of exploration behavior, policy diversity, and training dynamics. (a) Validation performance on AIME24 shows that Info-GRPO consistently outperforms GRPO across different numbers of latent seeds, with the performance gain appearing to stabilize around 10 seeds. (b) Analysis of policy semantic diversity reveals that Info-GRPO (with 10 seeds) maintains high inter-group diversity across conditional seeds and low intra-group diversity within a seed, effectively balancing exploration and exploitation. (c–f) Info-GRPO exhibits entropy reduction while achieving higher training rewards compared to the GRPO baseline.

Table 9

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Info-GRPO	DAPO	Clip-Cov	KL-Cov
Training batch size	128	128	256	256
Responses per prompt ( $G$ )	16	16	8	8
Max prompt length	2048	2048	2048	2048
Max response length	8192	8192	8192	8192
Learning rate ( $\eta$ )	$1\times 10^{-6}$	$1\times 10^{-6}$	$5\times 10^{-7}$	$5\times 10^{-7}$
Sampling temperature	1.0	1.0	1.0	1.0

Appendix

Table 10: Training hyperparameters for Info-GRPO and baseline methods. Unless otherwise noted, models are trained by us using the same infrastructure and default settings. DAPO, Clip-Cov, and KL-Cov follow the official codebases and configurations.

Backbone	Method	AIME24	AIME25	AMC	MATH500	Minerva	Average
Qwen2.5-7B	Base Model	13.33	6.67	40.96	75.40	35.66	34.40
	GRPO	16.67	13.33	57.83	77.60	37.50	40.59
	DAPO	13.33	16.67	51.81	78.60	35.66	39.21
	CLIP-Cov	20.00	13.33	51.81	79.60	38.24	40.60
	KL-Cov	16.67	10.00	57.83	74.20	34.56	38.65
	Skywork-OR1	16.67	10.00	57.83	77.80	38.24	40.11

Appendix

Table 11: Comparison of methods on multiple backbones and benchmarks (

Pass@1

, %). ‘

{{\dagger}}

’ indicates using the official model.

Backbone	Method	AIME24	AIME25	AMC	MATH500	Minerva	Average
Qwen2.5-7B	Base Model	30.00	20.00	67.47	88.80	47.79	50.81
	GRPO	30.00	23.33	73.49	88.80	47.79	52.68
	DAPO	33.33	23.33	67.47	90.60	50.74	53.09
	CLIP-Cov	30.00	20.00	71.08	90.20	51.10	52.48
	KL-Cov	30.00	23.33	74.70	83.60	48.53	52.03
	Skywork-OR1	33.33	20.00	74.70	87.60	48.90	52.91

Appendix

Table 12: Comparison of methods on multiple backbones and benchmarks (

Pass@8

, %). ‘

{{\dagger}}

’ indicates using the official model.

Method	AIME24	AMC23	MATH500	Avg.
ETR	56.7	87.5	90.6	78.3
Info-GRPO (Ours)	56.7	84.3	94.8	78.6

Appendix

Table 13: Comparison with ETR [ 39 ] on DeepSeek-R1-Distill-Qwen-7B (

Pass@1

, %), under an identical decoding protocol.

Method	$Avg@8$	Diversity (Euclidean distance)
GRPO	10.83	0.206
DAPO	11.25	0.190
Clip-Cov	12.08	0.217
KL-Cov	11.25	0.205
Skywork-OR1	7.50	0.239
Info-GRPO (Ours)	8.75	0.287

Appendix

Table 14: Performance and diversity comparison on AIME25 using the Qwen2.5-7B base model.

Coef $\lambda$	AMC	Minerva
0.5	79.52	39.71
0.05	80.72	41.91
0.005 (Used)	83.13	44.94

Appendix

Table 15: Cross-benchmark ablation on

\lambda

using DeepSeek-R1-Distill-Qwen-7B (

Avg@8

, %), on benchmarks disjoint from the headline AIME sets.

Coef $\lambda$	AIME24			AIME25
Coef $\lambda$	$Pass@1$	$Avg@8$	$Pass@8$	$Pass@1$	$Avg@8$	$Pass@8$
0.5	46.67	52.92	76.67	30.00	33.75	53.33
0.05	53.33	55.83	83.33	33.33	34.58	53.33
0.005	60.00	57.92	86.67	46.67	37.50	60.00

Appendix

Table 16: Ablation on coefficient

\lambda

using DeepSeek-R1-Distill-Llama-8B (%).

Number of Latent Seeds	$Pass@1$	$Avg@8$	$Pass@8$	Diversity (Euclidean distance)
3	30.00	25.00	40.00	0.28
30	26.67	23.33	30.00	0.31
10 (Used)	30.00	25.00	36.67	0.30

Appendix

Table 17: Ablation on the number of latent seeds (%).

Figure 5: Per-seed intra-group diversity dynamics of Info-GRPO (Qwen2.5-Math-7B) on AIME24 over an extended training horizon (up to 1600 steps). Each thin curve represents the intra-group diversity for a single latent seed, while the thick curve shows their average, reported as Info-GRPO (Intra-group) in Figure 4 (b). Although some seeds exhibit mild early fluctuations, all curves follow a generally downward trend and gradually stabilize, confirming that the averaged curve accurately reflects the typical training behavior across seeds.

Figure 6: The correlation heatmap for a case. It is generated by our Info-GRPO based on the DeepSeek-R1-Distill-Qwen-7B models.

Response	No seed	Seed1	Seed2	Seed3	Seed4	Seed5	Seed6	Seed7	Seed8	Seed9	Seed10
R1	1	1	0	1	1	1	0	1	1	1	1
R2	0	1	1	0	0	0	1	1	1	0	1
R3	1	1	1	1	0	0	1	1	1	0	1
R4	0	0	1	0	1	0	0	1	1	1	1
R5	1	1	1	1	1	1	1	1	0	1	1
R6	1	0	1	1	1	0	1	1	0	1	1

Appendix

Table 18: The scores for 8 responses under different prompt seeds.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,103 papers

ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models
Mar 30, 2026Song Yu, Li Li, Wenwen Zhao +1RL for Language Model Reasoning RL for Language Models
School of Computer and Information Science, Southwest University, Chongqing, 400715, China
Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models
May 18, 2026Junyao Yang, Chen Qian, Kun Wang +4LLM Interpretability RL for Language Model Reasoning
National University of Singapore · Renmin University of China · Nanyang Technological University +1
Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting
May 12, 2026Cheng Wang, Qin Liu, Wenxuan Zhou +1Reinforcement Learning RL for Language Model Reasoning
National University of Singapore · University of California, Davis · University of Southern California

Mar 30, 2026J/K move · Enter open · S save

ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models

Song Yu, Li Li, Wenwen Zhao, Zhisheng Yang

School of Computer and Information Science, Southwest University, Chongqing, 400715, China

RL for Language Model Reasoning RL for Language Models LLM Reasoning Large Reasoning Models

Open paper AskPDF arXiv

Reinforcement learning from verifiable rewards has significantly advanced the reasoning capabilities of large language models. However, Group Relative Policy Optimization (GRPO) typically assigns a uniform, sequence-level advantage to all tokens, thereby overlooking the intrinsic information heterogeneity along reasoning chains. We show that this coarse-grained credit assignment leads to premature entropy collapse and encourages the model to generate redundant, low-quality reasoning paths. Through systematic empirical analysis, we identify Critical Decision Pivots (CDPs): transient high-entropy states where the policy's trajectory is most sensitive to perturbations. These pivots represent the "forks in the road" where effective multi-path exploration is most crucial yet often suppressed by uniform advantage signals. Building on these insights, we propose Entropy-Regulated Policy Optimization (ERPO), which transitions the optimization focus from coarse sequences to fine-grained token dynamics. ERPO introduces three synergistic components: (i) Entropy-aware Gating, which adaptively amplifies exploration at CDPs to facilitate diverse path discovery; (ii) Bucket-based Implicit Normalization, which mitigates difficulty bias by aligning token progress windows; and (iii) Result-anchored Advantage Synthesis, which re-weights token-level signals via outcome-driven anchors. Extensive experiments on competitive mathematical benchmarks demonstrate that ERPO significantly outperforms GRPO. Notably, ERPO not only boosts reasoning accuracy but also yields significantly more concise and robust derivation paths, while achieving performance comparable to large models with orders of magnitude more parameters.

Abstract

RL for Language Model Reasoning

Explore similar work

ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models

Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models

Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting