Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
Authors: Jaesung R. Park, Junsu Kim, Gyeongman Kim, Jinyoung Jo, Sean Choi, Jaewoong Cho, Ernest K. Ryu
Organizations: Department of Mathematics, UCLA · Department of Mathematical Sciences, Seoul National University · KRAFTON · Department of Linguistics, Stanford University · Department of Computer Science and Engineering, Santa Clara University
Reinforcement learning with verifiable rewards (RLVR) has recently emerged as the leading approach for enhancing the reasoning capabilities of large language models (LLMs). However, RLVR is prone to entropy collapse, where the LLM quickly converges to a near-deterministic form, hindering exploration and progress during prolonged RL training. In this work, we reveal that the clipping mechanism in PPO and GRPO induces biases on entropy. Through theoretical and empirical analyses, we show that clip-low increases entropy, while clip-high decreases it. Further, under standard clipping parameters, the effect of clip-high dominates, resulting in an overall entropy reduction even when purely random rewards are provided to the RL algorithm. Our findings highlight an overlooked confounding factor in RLVR: independent of the reward signal, the clipping mechanism influences entropy, which in turn affects the reasoning behavior. Furthermore, our analysis demonstrates that clipping can be deliberately used to control entropy. Specifically, with a more aggressive clip-low value, one can increase entropy, promote exploration, and ultimately prevent entropy collapse in RLVR training.
Figures & tables
Train steps
δ0=0.3
δ0=0.1
δ0=0.01
0
0.88
0.78
0.63
120
0.95
0.89
0.80
240
0.96
0.91
0.84
360
0.97
0.92
0.86
Table 1: Proportion of visited states that satisfy the heavy head assumption maxa∈Aπθ(a∣s)≥1−δ0 throughout RL training.
Figure 1: Change of policy entropy during RL training with (Left) random rewards on the Qwen2.5-1.5B-Instruct model and (Right) nonrandom rewards on the Qwen2.5-3B-Instruct model with different clipping settings. We observe that both clip-high and clip-low influence the entropy, consistent with our theoretical predictions. We average over three runs with different random seeds; shades denote pointwise uncertainty bands across seeds.
Update
Clip-low effect
Clip-high effect
Regular PG
+0.0725
−0.1650
NPG
+0.6170
−1.0591
AdamW, other clipping mechanism disabled
+0.5409
−1.5727
AdamW, other clipping mechanism enabled
+0.6360
−1.4776
Table 2: Clipping effects under PG, NPG, and Transformer–AdamW. All values are in units of 10−3 . PG/NPG entries are learning-rate-free first-order coefficients; AdamW entries are post-update entropy differences evaluated on the same fixed rollout prefixes.
Qwen2.5-7B-Instruct
AIME24
AIME25
MATH500
AMC23
Avg.
pass@32 (%)
εhigh=0.2,εlow=0.2
26.3±4.4
22.8±4.8
88.3±1.2
76.4±1.0
53.5±2.7
+ εlow=1.0
24.9±7.0
20.6±2.3
80.0±14.4
75.2±2.8
50.2±4.7
+ εhigh=∞
38.2±2.5
29.1±4.6
85.7±11.3
81.2±1.7
58.5±2.6
mean@32 (%)
εhigh=0.2,εlow=0.2
14.3±1.3
7.7±1.8
75.6±2.0
54.2±0.7
38.0±1.0
Table 3: Clipping ablation results on math benchmarks. For the Qwen2.5-7B-Instruct model, we perform RL training with clip-high off ( εhigh=∞ ) and clip-low off ( εlow=1.0 ), with the symmetric clipping baseline ( εhigh=εlow=0.2 ). Reported numbers are pass@32 (%) and mean@32 (%) scores. Within each run, we average the common checkpoints at steps 190, 200, and 210, and report the mean and sample standard deviation across three independent matched training runs. Bold indicates the best result per metric.
Figure 2: (Left) Averaged pass@32 score and entropy for various εhigh and εlow settings. We color entropy in log scale for visibility. (Right) Correlation between entropy and pass@32 compared with correlation between entropy and mean@32 .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Optimizer
AdamW
Learning rate
5×10−7 / 1×10−6
GRPO batch size
512
Optimizer batch size
256
Policy updates per rollout
16
Group Size
8
Appendix
Table 4: Training configurations used for GSM8K dataset / DAPO-Math-17k dataset.
εhigh / εlow
{maxkH(πθk) - minkH(πθk)}×105
0.02 / 0.02
4.20
∞ / 0.02
4.96
0.02 / 1.0
5.63
Appendix
Table 5: Difference of maximum entropy and minimum entropy throughout random reward RL training of Qwen2.5-1.5B model with randomly initialized weights.
Dataset
δ0=0.3
δ0=0.1
δ0=0.01
GSM8K
0.96
0.91
0.84
MATH500
0.96
0.92
0.86
AIME2024
0.94
0.86
0.79
AMC2023
0.95
0.90
0.83
GPQA diamond
0.92
0.85
0.76
Appendix
Table 6: Heavy-head assumption validation on Qwen2.5-72B-Instruct model.
Dataset
δ0=0.3
δ0=0.1
δ0=0.01
GSM8K
0.80
0.66
0.50
MATH500
0.87
0.78
0.62
AIME2024
0.86
0.76
0.64
AMC2023
0.85
0.76
0.64
GPQA diamond
0.90
0.82
0.70
Appendix
Table 7: Heavy-head assumption validation on Llama3-70B-Instruct model.
Figure 3: Change of policy entropy during RL training with (Left) random rewards on the Qwen2.5-1.5B-Instruct model and (Right) nonrandom rewards on the Qwen2.5-3B-Instruct model with KL divergence loss term during training.
Figure 4: Entropy change across different model families when trained with random rewards under symmetric clipping εlow=εhigh . The entropy is normalized with the corresponding base model’s entropy. This result substantiates that the clipping bias is not a model-specific phenomenon.
Figure 5: Entropy change of Qwen2.5-1.5B-Instruct model with random rewards sampled from various probability distributions.
Figure 6: Plotting the entropy, mean@32 , and pass@32 metric throughout the RLVR training for the Qwen2.5-3B-Instruct model with various εhigh and εlow parameters. The mean@32 and pass@32 metrics were evaluated on the GSM8K test dataset. We observe that the clipping hyperparameters continuously control the entropy, while the pass@32 shows high correlation with it.
Figure 7: Performance measured by the mean@32 metric (Left) and pass@32 metric (Right) metric during RLVR for the Qwen2.5-7B-Instruct model trained with DAPO-Math-17k dataset, evaluated on the AIME2024 , AIME2025 , AMC , and MATH-500 datasets.
Figure 8: Clipping ablation results reproduced with Llama base models. (a) We plot the policy entropy change during RL training trained with (Left) random rewards and (Right) true rewards. We observe that the clipping bias has the same qualitative trend for Llama -family base models as well. (b) We measure the (Left) mean@8 metric and (Right) pass@8 metric throughout the training. We note that the high-entropy settings improve exploration while preserving the mean accuracy.
Reinforcement learning (RL) is a key paradigm for post-training large language models (LLMs), but the widely used Group Relative Policy Optimization (GRPO) often suffers from entropy collapse: exploration quickly disappears, policies converge prematurely, and sample diversity declines, ultimately harming training effectiveness. Existing remedies, including entropy bonuses and clip-based methods, rarely keep entropy within a stable exploration regime and often introduce oscillatory entropy or reward degradation. In this work, we identify a previously overlooked asymmetry in entropy dynamics: under high-temperature sampling, positive and negative samples have opposite effects on policy entropy. Specifically, high-temperature positive samples promote entropy growth, whereas negative samples suppress it. We provide a qualitative theoretical explanation for this phenomenon: when entropy decreases during policy updates, its derivative with respect to temperature is strictly positive under positive sample updates, indicating that high-temperature positive samples can counteract entropy decay, thereby slowing entropy collapse and potentially reversing it. Motivated by this insight, we propose SCOPE-RL, a stable entropy control framework through a regularization term constructed from temperature-adaptive positive samples. Extensive experiments show that SCOPE-RL consistently outperforms strong RL baselines on both Pass@1 and Pass@k. Our results provide evidence that escaping entropy collapse can improve reasoning performance, while our quantitative empirical analysis further reveals a non-monotonic entropy-performance relationship, with an optimal level of exploration for RL post-training in reasoning LLMs. Our code is available at https://github.com/597358816/SCOPE-RL.
Chen Wang, Zhaochun Li, Jionghao Bai +4
College of Software, Nankai University · Zhongguancun Academy · Beijing Institute of Technology +2
Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning ability of large language models. However, widely used RLVR algorithms, such as GRPO, often suffer from entropy collapse, leading to premature determinism and unstable optimization. Existing remedies, including entropy regularization and ratio-based clipping heuristics, either control entropy in a coarse-grained manner or rely on approximate on-policy training. In this paper, we revisit entropy collapse from a token-level entropy flow perspective. Our analysis reveals that entropy-decreasing tokens consistently outweigh entropy-increasing ones, resulting in a severely imbalanced entropy flow. This perspective provides a unified explanation of entropy collapse in existing RLVR algorithms and highlights the importance of balancing entropy dynamics. Motivated by this analysis, we propose On-Policy Entropy Flow Optimization (OPEFO), an adaptive entropy flow balancing mechanism that rescales entropy-increasing and entropy-decreasing updates according to their contributions to entropy change, while remaining strict on-policy. Experiments on six mathematical reasoning benchmarks demonstrate that OPEFO improves training stability and final performance. We will release the code and models upon publication.
Proximal constraints are fundamental to the stability of the Large Language Model reinforcement learning. While the canonical clipping mechanism in PPO serves as an efficient surrogate for trust regions, we identify a critical bottleneck: fixed bounds strictly constrain the upward update margin of low-probability actions, disproportionately suppressing high-advantage tail strategies and inducing rapid entropy collapse. To address this, we introduce Band-constrained Policy Optimization (BandPO). BandPO replaces canonical clipping with Band, a unified theoretical operator that projects trust regions defined by f-divergences into dynamic, probability-aware clipping intervals. Theoretical analysis confirms that Band effectively resolves this exploration bottleneck. We formulate this mapping as a convex optimization problem, guaranteeing a globally optimal numerical solution while deriving closed-form solutions for specific divergences. Extensive experiments across diverse models and datasets demonstrate that BandPO consistently outperforms canonical clipping and Clip-Higher, while robustly mitigating entropy collapse.