SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training

Chen Wang; Zhaochun Li; Jionghao Bai; Hexuan Deng; Guanting Dong; Ge Lan; Yue Wang

cs.LGOct 9, 2025

SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training

Authors: Chen Wang, Zhaochun Li, Jionghao Bai, Hexuan Deng, Guanting Dong, Ge Lan, Yue Wang

Organizations: College of Software, Nankai University · Zhongguancun Academy · Beijing Institute of Technology · Zhejiang University ·

Abstract

Reinforcement learning (RL) is a key paradigm for post-training large language models (LLMs), but the widely used Group Relative Policy Optimization (GRPO) often suffers from entropy collapse: exploration quickly disappears, policies converge prematurely, and sample diversity declines, ultimately harming training effectiveness. Existing remedies, including entropy bonuses and clip-based methods, rarely keep entropy within a stable exploration regime and often introduce oscillatory entropy or reward degradation. In this work, we identify a previously overlooked asymmetry in entropy dynamics: under high-temperature sampling, positive and negative samples have opposite effects on policy entropy. Specifically, high-temperature positive samples promote entropy growth, whereas negative samples suppress it. We provide a qualitative theoretical explanation for this phenomenon: when entropy decreases during policy updates, its derivative with respect to temperature is strictly positive under positive sample updates, indicating that high-temperature positive samples can counteract entropy decay, thereby slowing entropy collapse and potentially reversing it. Motivated by this insight, we propose SCOPE-RL, a stable entropy control framework through a regularization term constructed from temperature-adaptive positive samples. Extensive experiments show that SCOPE-RL consistently outperforms strong RL baselines on both Pass@1 and Pass@k. Our results provide evidence that escaping entropy collapse can improve reasoning performance, while our quantitative empirical analysis further reveals a non-monotonic entropy-performance relationship, with an optimal level of exploration for RL post-training in reasoning LLMs. Our code is available at https://github.com/597358816/SCOPE-RL.

Figures & tables

Panel of Figure 2 — Figure 2: Training dynamics of Qwen2.5-Math-7B on the DAPO-17k dataset under different settings. Although both methods can induce entropy growth, the increase is often accompanied by unstable training dynamics and noticeable reward degradation.

Panel of Figure 3 — Figure 3: Observation experiments on Qwen2.5-Math-7B trained on DAPO-17k. Starting from a rollout batch size of $512\times 8$ , we additionally mix in 64 temperature-scaled samples per update. The left column varies the global sampling temperature in vanilla GRPO without separating positive and negative samples, while the middle and right columns isolate their effects under high- and low-temperature regimes. Changing temperature alone does not prevent entropy collapse; the entropy effect depends critically on both sample sign and temperature.

Figure 4: SCOPE-RL can not only alleviate entropy collapse, but also stably control entropy at different levels

Methods	AIME24	AIME25	AMC23	GSM8K	MATH	Minerva	Olympiad	Average
Qwen2.5-Math-7B	15.5	7.81	42.1	65.4	59.4	11.0	26.7	32.56
+GRPO	32.1	11.0	72.4	88.7	80.6	34.6	41.8	51.60
+Entropy-Reg $(\lambda=0.01)$	34.2	10.1	73.1	88.7	79.8	35.3	40.8	51.71
+Entropy-Reg $(\lambda=0.03)$	31.4	10.1	74.3	87.0	78.8	35.7	40.4	51.10
+Entropy-Reg $(\lambda=0.05)$	21.6	10.8	63.6	86.5	77.0	31.9	39.2	47.23
+DAPO	33.3	13.0	73.9	87.9	79.6	34.6	40.8	51.87

Table 1: Main results on mathematical reasoning benchmarks. SCOPE-RL consistently improves performance across different base models.

Methods	Qwen2.5-Math-7B			Qwen2.5-7B			Qwen3-4B
	ARC	MMLU	S-GPQA	ARC	MMLU	S-GPQA	ARC	MMLU	S-GPQA
Base	69.9	34.3	19.4	86.3	50.0	18.6	92.9	72.6	32.8
+GRPO	78.9	44.8	25.8	89.3	58.2	27.0	94.6	74.1	38.4
+DAPO	80.9	48.6	26.0	90.9	59.8	24.8	92.6	75.0	37.0
+Entropy-Reg	79.6	42.7	26.2	89.6	58.2	26.8	93.9	73.7	38.2
+Entropy-Adv	80.3	47.3	26.2	90.6	57.7	26.6	93.6	74.8	37.8

Table 2: Generalization results on knowledge-intensive benchmarks, including ARC-Challenge, MMLU-Pro, and SuperGPQA, where SCOPE-RL demonstrates leading performance across RL methods.

Panel of Table 3 — Table 3: Comparison of Pass@ $k$ performance between the base model, GRPO, and SCOPE-RL across AIME24 and AIME-25. The accompanying figures show the evolution of Pass@1024 of SCOPE-RL and GRPO on AIME24 and AIME25 during training.

Benchmarks	AIME24	AIME25	AMC	GSM8K	MATH	Minerva	Olympiad	Average
Qwen2.5-math-7B	15.5	7.81	42.1	65.4	59.4	11.0	26.7	32.56
+SCOPE-RL $\mathcal{H}_{0}=0.25$	37.9	11.3	74.4	89.4	79.8	36.0	42.2	53.00
+SCOPE-RL $\mathcal{H}_{0}=0.50$	36.4	18.9	75.3	89.5	82.6	38.2	43.0	54.84
+SCOPE-RL $\mathcal{H}_{0}=0.75$	33.6	15.1	77.0	89.4	79.2	37.5	42.4	53.45
+SCOPE-RL $\mathcal{H}_{0}=1.00$	33.2	15.6	74.8	88.7	79.6	37.9	42.1	53.13

Table 4: SCOPE-RL under different entropy targets

\mathcal{H}_{0}

, which correspond to different degrees of exploration. The overall performance is maximized at a moderate entropy level.

Figure 5: Entropy and reward curves of SCOPE-RL under different auxiliary sample ratios $\alpha$ on Qwen2.5-Math-7B, trained on DAPO-17K with a rollout batch size of $512\times 8$ and $\mathcal{H}_{0}=0.5$ .

Setting	AIME24	AIME25	AMC	GSM8K	MATH	Minerva	Olympiad	Average
Entropy step	50	100	150	200	250	300	350	Entropy trends
	31.4	11.8	74.5	88.9	79.0	33.5	40.1	51.31 (-3.53)
w/o temperature	0.080	0.036	0.026	0.024	0.022	0.019	0.018	Entropy collapse
	32.7	11.0	73.2	88.1	79.0	35.7	41.2	51.56 (-3.28)
positive+negative	0.114	0.041	0.027	0.024	0.024	0.022	0.020	Entropy collapse
	28.9	12.7	65.1	89.5	77.6	33.8	40.4	49.71 (-5.12)

Table 5: Component ablations of SCOPE-RL on Qwen2.5-Math-7B. Removing either temperature-adjusted or positive sampling degrades performance and entropy control.

Figure 6: Importance-sampling ablations on entropy control and clipping behavior.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	AIME24	AIME25	AMC23	ARC	CMIMC25	GSM8K
Method	HMMT25	MATH500	Minerva	MMLU-Pro	Olympiad	SuperGPQA
SCOPE-RL	$\mathbf{11.25\pm 1.00}$	$\mathbf{1.53\pm 0.33}$	$\mathbf{28.62\pm 1.63}$	$84.95\pm 0.00$	$\mathbf{4.87\pm 1.34}$	$\mathbf{88.32\pm 0.40}$
SCOPE-RL	$\mathbf{1.81\pm 1.15}$	$\mathbf{54.33\pm 1.21}$	$\mathbf{29.17\pm 0.56}$	$\mathbf{53.63\pm 1.19}$	$\mathbf{19.95\pm 0.31}$	$\mathbf{19.47\pm 0.64}$
GRPO	$7.74\pm 0.34$	$0.24\pm 0.16$	$26.72\pm 0.68$	$83.72\pm 1.27$	$3.88\pm 0.16$	$87.09\pm 0.32$
GRPO	$0.90\pm 0.24$	$52.93\pm 1.86$	$25.73\pm 1.33$	$50.83\pm 1.76$	$19.16\pm 2.23$	$17.07\pm 0.46$
DAPO	$6.73\pm 0.95$	$1.04\pm 0.21$	$25.21\pm 1.52$	$81.60\pm 1.74$	$4.20\pm 0.32$	$87.77\pm 0.50$

Appendix

Table 6: Additional results on Llama-3.1-8B-Instruct over three independent random seeds. All methods use identical rollout budgets. Results are reported as mean

\pm

standard deviation.

Base Model	Method	AIME24	AIME25	AMC23	CMIMC25	HMMT25	Average
Qwen2.5- Math-7B	SCOPE-RL	60.00	40.00	87.50	20.00	13.30	44.16
	GRPO	53.30	23.30	82.50	15.00	10.00	36.82
	DAPO	50.00	33.33	82.50	20.00	10.00	39.17
	KL-Cov	53.33	26.67	82.50	20.00	10.00	38.50
	CISPO	50.00	26.67	80.00	15.00	6.67	35.67
Qwen2.5-7B	SCOPE-RL	40.00	36.67	90.00	32.50	13.33	42.50

Appendix

Table 7: Expanded Pass@32 evaluation across four base models spanning three model families. SCOPE-RL achieves the highest average Pass@32 performance on all evaluated base models.

Method	Epoch	Entropy	All-correct	Mixed	All-wrong
GRPO	1	0.192	0.105	0.484	0.411
SCOPE-RL	1	0.310	0.093	0.484	0.423
GRPO	2	0.151	0.163	0.442	0.395
SCOPE-RL	2	0.418	0.161	0.443	0.396
GRPO	3	0.135	0.184	0.408	0.408
SCOPE-RL	3	0.403	0.225	0.407	0.368

Appendix

Table 8: Prompt-level training dynamics of GRPO and SCOPE-RL on Qwen2.5-Math-7B over four epochs.

Figure 7: Entropy trajectories of SCOPE-RL under different clipping settings on Qwen2.5-Math-7B.

Setting	Value
Qwen main experiments
Training data	DAPO-17K
Hardware	$8\times$ A800 GPUs (40 GB each)
Optimizer; learning rate	AdamW; $1\times 10^{-6}$
Training seed	1
Training steps	340 (Qwen2.5-Math-7B); 204 (Qwen2.5-7B); 170 (Qwen3-4B)

Appendix

Table 9: Implementation settings. Unless stated otherwise, the RL and SCOPE-RL settings apply to the three Qwen main experiments.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,704 papers

Apr 16, 2026

Targeted Exploration via Unified Entropy Control for Reinforcement Learning

Recent advances in reinforcement learning (RL) have improved the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs). However, the widely used Group Relative Policy Optimization (GRPO) consistently suffers from entropy collapse, causing the policy to converge prematurely and lose diversity. Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain optimization stability. We propose Unified Entropy Control for Reinforcement Learning (UEC-RL), a framework that provides targeted mechanisms for exploration and stabilization. UEC-RL activates more exploration on difficult prompts to search for potential and valuable reasoning trajectories. In parallel, a stabilizer prevents entropy from growing uncontrollably, thereby keeping training stable as the model consolidates reliable behaviors. Together, these components expand the search space when needed while maintaining robust optimization throughout training. Experiments on both LLM and VLM reasoning tasks show consistent gains over RL baselines on both Pass@1 and Pass@

k

. On Geometry3K, UEC-RL achieves a 37.9% relative improvement over GRPO, indicating that it sustains effective exploration without compromising convergence and underscoring UEC-RL as a key for scaling RL-based reasoning in large models. Our code is available at https://github.com/597358816/UEC-RL.

Chen Wang, Lai Wei, Yanzhi Zhang +5

College of Software, Nankai University · Zhongguancun Academy · Shanghai Jiao Tong University +3

Reinforcement Learning RL for Language Model Reasoning RL Exploration Maximum Entropy RL RL for VLMs

Ask PDF

Apr 2, 2026

A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning

Reinforcement learning (RL) has become a key approach for enhancing reasoning in large language models (LLMs), yet scalable training is often hindered by the rapid collapse of policy entropy, which leads to premature convergence and performance saturation. This paper provides a comparative theoretical analysis of two entropy control strategies: traditional entropy regularization and the recently proposed covariance-based mechanism. We establish a unified framework for entropy dynamics under softmax parameterization, showing that entropy change is governed by the covariance between log-probabilities and logit updates. Our analysis reveals that traditional entropy regularization introduces a dense, persistent bias that modifies the stationary condition, leading to suboptimal policies, while covariance-based methods selectively regularize a sparse subset of high-covariance tokens and achieve asymptotic unbiasedness when the regularization coefficient is annealed. These results provide principled guidelines for entropy control in LLM posttraining, with implications for scaling RL to larger models and more complex reasoning tasks.

Ming Lei, Christophe Baehr

Shanghai Jiao Tong University, Shanghai · M´et´eo France/CNRS, Universit´e de Toulouse, CNRM UMR 3589, Toulouse

Entropy Regularization RL for Language Model Reasoning RL Post-Training Maximum Entropy RL

Ask PDF

Sep 30, 2025

Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models

Reinforcement learning with verifiable rewards (RLVR) has recently emerged as the leading approach for enhancing the reasoning capabilities of large language models (LLMs). However, RLVR is prone to entropy collapse, where the LLM quickly converges to a near-deterministic form, hindering exploration and progress during prolonged RL training. In this work, we reveal that the clipping mechanism in PPO and GRPO induces biases on entropy. Through theoretical and empirical analyses, we show that clip-low increases entropy, while clip-high decreases it. Further, under standard clipping parameters, the effect of clip-high dominates, resulting in an overall entropy reduction even when purely random rewards are provided to the RL algorithm. Our findings highlight an overlooked confounding factor in RLVR: independent of the reward signal, the clipping mechanism influences entropy, which in turn affects the reasoning behavior. Furthermore, our analysis demonstrates that clipping can be deliberately used to control entropy. Specifically, with a more aggressive clip-low value, one can increase entropy, promote exploration, and ultimately prevent entropy collapse in RLVR training.

Jaesung R. Park, Junsu Kim, Gyeongman Kim +4

Department of Mathematics, UCLA · Department of Mathematical Sciences, Seoul National University · KRAFTON +2

RL for Language Models Reinforcement Learning Reinforcement Learning with Verifiable Rewards

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Targeted Exploration via Unified Entropy Control for Reinforcement Learning

A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning

Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models