Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models

Jaesung R. Park; Junsu Kim; Gyeongman Kim; Jinyoung Jo; Sean Choi; Jaewoong Cho; Ernest K. Ryu

cs.LGSep 30, 2025

Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models

Authors: Jaesung R. Park, Junsu Kim, Gyeongman Kim, Jinyoung Jo, Sean Choi, Jaewoong Cho, Ernest K. Ryu

Organizations: Department of Mathematics, UCLA · Department of Mathematical Sciences, Seoul National University · KRAFTON · Department of Linguistics, Stanford University ·

Abstract

Reinforcement learning with verifiable rewards (RLVR) has recently emerged as the leading approach for enhancing the reasoning capabilities of large language models (LLMs). However, RLVR is prone to entropy collapse, where the LLM quickly converges to a near-deterministic form, hindering exploration and progress during prolonged RL training. In this work, we reveal that the clipping mechanism in PPO and GRPO induces biases on entropy. Through theoretical and empirical analyses, we show that clip-low increases entropy, while clip-high decreases it. Further, under standard clipping parameters, the effect of clip-high dominates, resulting in an overall entropy reduction even when purely random rewards are provided to the RL algorithm. Our findings highlight an overlooked confounding factor in RLVR: independent of the reward signal, the clipping mechanism influences entropy, which in turn affects the reasoning behavior. Furthermore, our analysis demonstrates that clipping can be deliberately used to control entropy. Specifically, with a more aggressive clip-low value, one can increase entropy, promote exploration, and ultimately prevent entropy collapse in RLVR training.

Figures & tables

Train steps	$\delta_{0}=0.3$	$\delta_{0}=0.1$	$\delta_{0}=0.01$
0	0.88	0.78	0.63
120	0.95	0.89	0.80
240	0.96	0.91	0.84
360	0.97	0.92	0.86

Table 1: Proportion of visited states that satisfy the heavy head assumption

\max_{a\in\mathcal{A}}\pi_{\theta}(a\mid s)\geq 1-\delta_{0}

throughout RL training.

Panel of Figure 1 — Figure 1: Change of policy entropy during RL training with (Left) random rewards on the Qwen2.5-1.5B-Instruct model and (Right) nonrandom rewards on the Qwen2.5-3B-Instruct model with different clipping settings. We observe that both clip-high and clip-low influence the entropy, consistent with our theoretical predictions. We average over three runs with different random seeds; shades denote pointwise uncertainty bands across seeds.

Update	Clip-low effect	Clip-high effect
Regular PG	$+0.0725$	$-0.1650$
NPG	$+0.6170$	$-1.0591$
AdamW, other clipping mechanism disabled	$+0.5409$	$-1.5727$
AdamW, other clipping mechanism enabled	$+0.6360$	$-1.4776$

Table 2: Clipping effects under PG, NPG, and Transformer–AdamW. All values are in units of

10^{-3}

. PG/NPG entries are learning-rate-free first-order coefficients; AdamW entries are post-update entropy differences evaluated on the same fixed rollout prefixes.

pass@32 (%)
Qwen2.5-7B-Instruct	AIME24	AIME25	MATH500	AMC23	Avg.
$\varepsilon_{\mathrm{high}}=0.2,\varepsilon_{\mathrm{low}}=0.2$	$26.3\pm 4.4$	$22.8\pm 4.8$	$\mathbf{88.3\pm 1.2}$	$76.4\pm 1.0$	$53.5\pm 2.7$
+ $\varepsilon_{\mathrm{low}}=1.0$	$24.9\pm 7.0$	$20.6\pm 2.3$	$80.0\pm 14.4$	$75.2\pm 2.8$	$50.2\pm 4.7$
+ $\varepsilon_{\mathrm{high}}=\infty$	$\mathbf{38.2\pm 2.5}$	$\mathbf{29.1\pm 4.6}$	$85.7\pm 11.3$	$\mathbf{81.2\pm 1.7}$	$\mathbf{58.5\pm 2.6}$
mean@32 (%)
$\varepsilon_{\mathrm{high}}=0.2,\varepsilon_{\mathrm{low}}=0.2$	$14.3\pm 1.3$	$7.7\pm 1.8$	$\mathbf{75.6\pm 2.0}$	$54.2\pm 0.7$	$38.0\pm 1.0$

Table 3: Clipping ablation results on math benchmarks. For the Qwen2.5-7B-Instruct model, we perform RL training with clip-high off (

\varepsilon_{\mathrm{high}}=\infty

) and clip-low off (

\varepsilon_{\mathrm{low}}=1.0

), with the symmetric clipping baseline (

\varepsilon_{\mathrm{high}}=\varepsilon_{\mathrm{low}}=0.2

). Reported numbers are pass@32 (%) and mean@32 (%) scores. Within each run, we average the common checkpoints at steps 190, 200, and 210, and report the mean and sample standard deviation across three independent matched training runs. Bold indicates the best result per metric.

Panel of Figure 2 — Figure 2: (Left) Averaged pass@32 score and entropy for various $\varepsilon_{\mathrm{high}}$ and $\varepsilon_{\mathrm{low}}$ settings. We color entropy in log scale for visibility. (Right) Correlation between entropy and pass@32 compared with correlation between entropy and mean@32 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$5\times 10^{-7}$ / $1\times 10^{-6}$
GRPO batch size	512
Optimizer batch size	256
Policy updates per rollout	16
Group Size	8

Appendix

Table 4: Training configurations used for GSM8K dataset / DAPO-Math-17k dataset.

$\varepsilon_{\mathrm{high}}$ / $\varepsilon_{\mathrm{low}}$	$\{\max_{k}\mathcal{H}(\pi_{\theta_{k}})$ - $\min_{k}\mathcal{H}(\pi_{\theta_{k}})\}$ $\times 10^{5}$
$0.02\;$ / $\;0.02$	4.20
$\infty\;$ / $\;0.02$	4.96
$0.02\;$ / $\;1.0$	5.63

Appendix

Table 5: Difference of maximum entropy and minimum entropy throughout random reward RL training of Qwen2.5-1.5B model with randomly initialized weights.

Dataset	$\delta_{0}=0.3$	$\delta_{0}=0.1$	$\delta_{0}=0.01$
GSM8K	0.96	0.91	0.84
MATH500	0.96	0.92	0.86
AIME2024	0.94	0.86	0.79
AMC2023	0.95	0.90	0.83
GPQA diamond	0.92	0.85	0.76

Appendix

Table 6: Heavy-head assumption validation on Qwen2.5-72B-Instruct model.

Dataset	$\delta_{0}=0.3$	$\delta_{0}=0.1$	$\delta_{0}=0.01$
GSM8K	0.80	0.66	0.50
MATH500	0.87	0.78	0.62
AIME2024	0.86	0.76	0.64
AMC2023	0.85	0.76	0.64
GPQA diamond	0.90	0.82	0.70

Appendix

Table 7: Heavy-head assumption validation on Llama3-70B-Instruct model.

Panel of Figure 3 — Figure 3: Change of policy entropy during RL training with (Left) random rewards on the Qwen2.5-1.5B-Instruct model and (Right) nonrandom rewards on the Qwen2.5-3B-Instruct model with KL divergence loss term during training.

Figure 4: Entropy change across different model families when trained with random rewards under symmetric clipping $\varepsilon_{\mathrm{low}}=\varepsilon_{\mathrm{high}}$ . The entropy is normalized with the corresponding base model’s entropy. This result substantiates that the clipping bias is not a model-specific phenomenon.

Figure 5: Entropy change of Qwen2.5-1.5B-Instruct model with random rewards sampled from various probability distributions.

Panel of Figure 6 — Figure 6: Plotting the entropy, mean@32 , and pass@32 metric throughout the RLVR training for the Qwen2.5-3B-Instruct model with various $\varepsilon_{\mathrm{high}}$ and $\varepsilon_{\mathrm{low}}$ parameters. The mean@32 and pass@32 metrics were evaluated on the GSM8K test dataset. We observe that the clipping hyperparameters continuously control the entropy, while the pass@32 shows high correlation with it.

Panel of Figure 7 — Figure 7: Performance measured by the mean@32 metric (Left) and pass@32 metric (Right) metric during RLVR for the Qwen2.5-7B-Instruct model trained with DAPO-Math-17k dataset, evaluated on the AIME2024 , AIME2025 , AMC , and MATH-500 datasets.

Panel of Figure 8 — Figure 8: Clipping ablation results reproduced with Llama base models. (a) We plot the policy entropy change during RL training trained with (Left) random rewards and (Right) true rewards. We observe that the clipping bias has the same qualitative trend for Llama -family base models as well. (b) We measure the (Left) mean@8 metric and (Right) pass@8 metric throughout the training. We note that the high-entropy settings improve exploration while preserving the mean accuracy.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 430 papers

SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
Oct 9, 2025Chen Wang, Zhaochun Li, Jionghao Bai +4Reinforcement Learning RL Post-Training
College of Software, Nankai University · Zhongguancun Academy · Beijing Institute of Technology +2
Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
May 12, 2026Huimin Xu, Shuai Zhao, Xiaobao Wu +1Reinforcement Learning RL for Language Model Reasoning
Nanyang Technological University, Singapore · Shanghai Jiao Tong University, China · 3VinUniversity, Vietnam
BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
Mar 5, 2026Yuan Li, Bo Wang, Yufei Gao +4RL for Language Models Proximal Policy Optimization
Fudan University · Shanghai Innovation Institute

Oct 9, 2025J/K move · Enter open · S save

SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training

Chen Wang, Zhaochun Li, Jionghao Bai, Hexuan Deng+3

College of Software, Nankai University · Zhongguancun Academy · Beijing Institute of Technology+2

Reinforcement Learning RL Post-Training RL for Language Model Reasoning Group Relative Policy Optimization Maximum Entropy RL

Open paper AskPDF arXiv

Reinforcement learning (RL) is a key paradigm for post-training large language models (LLMs), but the widely used Group Relative Policy Optimization (GRPO) often suffers from entropy collapse: exploration quickly disappears, policies converge prematurely, and sample diversity declines, ultimately harming training effectiveness. Existing remedies, including entropy bonuses and clip-based methods, rarely keep entropy within a stable exploration regime and often introduce oscillatory entropy or reward degradation. In this work, we identify a previously overlooked asymmetry in entropy dynamics: under high-temperature sampling, positive and negative samples have opposite effects on policy entropy. Specifically, high-temperature positive samples promote entropy growth, whereas negative samples suppress it. We provide a qualitative theoretical explanation for this phenomenon: when entropy decreases during policy updates, its derivative with respect to temperature is strictly positive under positive sample updates, indicating that high-temperature positive samples can counteract entropy decay, thereby slowing entropy collapse and potentially reversing it. Motivated by this insight, we propose SCOPE-RL, a stable entropy control framework through a regularization term constructed from temperature-adaptive positive samples. Extensive experiments show that SCOPE-RL consistently outperforms strong RL baselines on both Pass@1 and Pass@k. Our results provide evidence that escaping entropy collapse can improve reasoning performance, while our quantitative empirical analysis further reveals a non-monotonic entropy-performance relationship, with an optimal level of exploration for RL post-training in reasoning LLMs. Our code is available at https://github.com/597358816/SCOPE-RL.

Abstract

RL for Language Models

Explore similar work

SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training

Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization

BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning