Entropy-Preserving Reinforcement Learning

Aleksei Petrenko; Ben Lipkin; Kevin Chen; Erik Wijmans; Marco Cusumano-Towner; Raja Giryes; Philipp Krähenbühl

cs.LGMar 12, 2026

Entropy-Preserving Reinforcement Learning

Authors: Aleksei Petrenko, Ben Lipkin, Kevin Chen, Erik Wijmans, Marco Cusumano-Towner, Raja Giryes, Philipp Krähenbühl

Organizations: Apple · MIT

Abstract

Policy gradient algorithms have driven many recent advancements in language model reasoning. An appealing property is their ability to learn from exploration on their own trajectories, a process crucial for fostering diverse and creative solutions. As we show in this paper, many policy gradient algorithms naturally reduce the entropy -- and thus the diversity of explored trajectories -- as part of training, yielding a policy increasingly limited in its ability to explore. In this paper, we argue that entropy should be actively monitored and controlled throughout training. We formally analyze the contributions of leading policy gradient objectives on entropy dynamics, identify empirical factors (such as numerical precision) that significantly impact entropy behavior, and propose explicit mechanisms for entropy control. These include REPO, a family of algorithms that modify the advantage function to regulate entropy, and ADAPO, an adaptive asymmetric clipping approach. Models trained with our entropy-preserving methods maintain diversity throughout training, yielding final policies that are more performant and retain their trainability for sequential learning in new environments.

Figures & tables

Panel of Figure 1 — Figure 1: Top: Evolution of the average per-token entropy and test accuracy during training for several baselines (GRPO, LOOP, DAPO, GSPO) and their entropy regularized versions (REPO). Each curve shows the average trajectory over several training runs with different seeds. Bottom: Cumulative entropy experienced during training up to a given checkpoint is positively correlated with the test accuracy. Each point is a checkpoint of a single training run (best-performing checkpoint per run highlighted). Algorithms that collapse the entropy early (see e.g. Qwen-3-8B on AppWorld; middle column) perform significantly worse than algorithms that maintain a steady entropy during training.

Panel of Figure 2 — Figure 2: (a) Fraction of tokens hitting clip bounds with 16-bit rounding: ${\epsilon}_{\text{high}}$ is reached more often and ${\epsilon}_{\text{low}}$ less often, hindering the promotion of low-probability actions. (b) Average probability difference and max importance weight ratio between vLLM inference and training forward pass.

Figure 3: FP16 training on Qwen-3-8B AppWorld and clipping fix lead to a qualitative change: DAPO entropy collapse transitions to entropy increase.

Figure 4: Entropy-preserving methods compared to baselines with Qwen-3-8B on AppWorld.

Figure 5: Entropy-preserving RL training with Qwen-3-32B on AppWorld.

Figure 6: Entropy-preserving RL training with Qwen-3-8B on AIME.

Figure 7: Sequential learning experiment. Top row: We use an AIME-trained model for GRPO, DAPO, REPO-R, and continue training the model on AppWorld. The left and middle plots show Task Goal Completion (TGC) on the normal (TN) and challenging (TC) test sets. A collapsed model (GRPO) does significantly worse than one in which entropy is preserved. Bottom row: We use an AppWorld-trained model and continue training on AIME. The same trends hold. All curves reflect the mean across three independent seeds.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Cumulative effect of the MixedPrecisionPolicy (MPP) fix and FP16 training when applied to DAPO algorithm with Qwen-3-8B . Each curve represents the mean of three independent runs (seeds). This is the expanded version of Fig. 3 .

Algorithm	Test Normal	Best TN	Test Challenge	Best TC	$\Delta\mathcal{H}$
RLOO	0.53 ± 0.05	0.64	0.32 ± 0.05	0.40	+0.01 (+4%)
GRPO	0.46 ± 0.01	0.46	0.21 ± 0.02	0.23	-0.09 (-64%)
LOOP	0.40 ± 0.02	0.42	0.19 ± 0.02	0.22	-0.08 (-53%)
GSPO	0.56 ± 0.04	0.60	0.32 ± 0.03	0.36	+0.00 (+2%)
DAPO	0.51 ± 0.03	0.55	0.28 ± 0.01	0.29	+0.06 (+40%)
GRPO + $\mathcal{H}$ bonus	0.48 ± 0.03	0.52	0.26 ± 0.02	0.28	-0.01 (-6%)

Appendix

Table 1: Task goal completion scores for AppWorld with Qwen-3-8B by training algorithm.

Algorithm	Test Normal	Best TN	Test Challenge	Best TC	$\Delta\mathcal{H}$
RLOO	0.77 ± 0.02	0.79	0.61 ± 0.06	0.71	-0.18 (-36%)
GRPO	0.67 ± 0.03	0.71	0.46 ± 0.03	0.49	-0.29 (-57%)
LOOP	0.66 ± 0.02	0.68	0.45 ± 0.03	0.47	-0.31 (-60%)
GSPO	0.69 ± 0.01	0.70	0.51 ± 0.00	0.51	-0.07 (-14%)
DAPO	0.73 ± 0.04	0.77	0.52 ± 0.02	0.55	+1.52 (+298%)
GRPO + $\mathcal{H}$ bonus	0.71 ± 0.02	0.74	0.49 ± 0.02	0.51	+0.14 (+27%)

Appendix

Table 2: Task goal completion scores for AppWorld with Qwen-3-32B by training algorithm.

Algorithm	AIME 2024	Best ’24	AIME 2025	Best ’25	$\Delta\mathcal{H}$
RLOO	0.47 ± 0.01	0.48	0.33 ± 0.01	0.34	-0.04 (-29%)
GRPO	0.45 ± 0.02	0.47	0.32 ± 0.02	0.35	-0.07 (-47%)
GSPO	0.43 ± 0.03	0.47	0.34 ± 0.01	0.34	+0.05 (+37%)
DAPO	0.43 ± 0.01	0.44	0.32 ± 0.02	0.33	+0.23 (+158%)
ADAPO	0.43 ± 0.00	0.43	0.34 ± 0.02	0.36	+0.13 (+93%)
REPO-R	0.45 ± 0.01	0.46	0.34 ± 0.01	0.35	+0.13 (+91%)

Appendix

Table 3: Scores for AIME with Qwen-3-8B by training algorithm. The model is limited to 4K context length.

Algorithm	AIME 2024	Best ’24	AIME 2025	Best ’25	$\Delta\mathcal{H}$
RLOO	0.53 ± 0.01	0.55	0.39 ± 0.01	0.40	+0.04 (+28%)
GRPO	0.51 ± 0.01	0.52	0.36 ± 0.02	0.39	+0.06 (+40%)
DAPO	0.43 ± 0.01	0.44	0.31 ± 0.03	0.35	+0.75 (+519%)
ADAPO	0.46 ± 0.03	0.50	0.34 ± 0.01	0.35	+0.29 (+198%)
REPO-R	0.49 ± 0.02	0.51	0.35 ± 0.02	0.38	+0.17 (+117%)

Appendix

Table 4: Scores for AIME with Qwen-3-32B by training algorithm. The model is limited to 4K context length.

Figure 9: The REPO transformation rotates

(A,\log\pi)

pairs, promoting low probability actions. Original unmodified advantages shown in blue, advantages

A_{\text{REPO-D}}

are shown in green and

A_{\text{REPO-R}}

in red. This plot demonstrates the transformation in a Qwen-3-32B AIME experiment, showcasing the real distribution of the data:

(A,\log\pi)

pairs are highly concentrated near

\log\pi\approx 0.0

Figure 10: Qwen 2.5 32B test performance and token entropy on AppWorld vs. training iterations. Curves show mean across three independent seeds for each algorithm.

Algorithm	Test Normal	Best TN	Test Challenge	Best TC
RLOO	0.72	0.78	0.47	0.50
LOOP	0.75	0.78	0.50	0.54
DAPO	0.74	0.77	0.50	0.56
ADAPO	0.73	0.78	0.51	0.59

Appendix

Table 5: Task-goal completion scores for AppWorld Qwen-2.5-32B by training algorithm. For each test split, we report the best average score across three seeds and the highest score among all seeds and training iterations.

Policy Gradient Methods

Policy gradient methods optimize stochastic policies by estimating how parameter changes affect expected reward.

See all 158 papers

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
Jun 17, 2026Haipeng Luo, Qingfeng Sun, Songli Wu +4RL for Language Model Reasoning Maximum Entropy RL
Shenzhen International Graduate School, Tsinghua University · Tencent Hunyuan
Group Entropy-Controlled Policy Optimization
Jul 18, 2026Guangran Cheng, Chengqi Lyu, Songyang Gao +2RL for Language Models Group Relative Policy Optimization
Shanghai AI Laboratory
A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
Apr 2, 2026Ming Lei, Christophe BaehrEntropy Regularization RL for Language Model Reasoning
Shanghai Jiao Tong University, Shanghai · M´et´eo France/CNRS, Universit´e de Toulouse, CNRM UMR 3589, Toulouse

Jun 17, 2026J/K move · Enter open · S save

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

Haipeng Luo, Qingfeng Sun, Songli Wu, Can Xu+3

Shenzhen International Graduate School, Tsinghua University · Tencent Hunyuan

RL for Language Model Reasoning Maximum Entropy RL Reinforcement Learning with Verifiable Rewards Policy Gradient Methods

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: the per-token entropy variation decomposes into the product of the trajectory-level advantage and an entropy sensitivity function over the next-token distribution, yielding an advantage-surprisal four-quadrant structure and a near-criticality property. Motivated by it, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability), which identifies entropy-critical token subsets via batch-internal surprisal quantiles, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate for stable entropy regulation. Across model scales from 1.5B to 32B and three task families (Short CoT, Long CoT, and Multi-Turn Tool Use), STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy, with reflection tokens and response length growing in tandem, indicating sustained exploration-exploitation balance that further unlocks RL training potential.Code is available at https://github.com/hp-luo/STARE.

Abstract

Policy Gradient Methods

Explore similar work

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

Group Entropy-Controlled Policy Optimization

A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning