Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO

Yiyang He; Zhichun Zhou; Ziwei Wang; Tao Xue; Haolin Fei

cs.LGAug 25, 2026

Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO

Authors: Yiyang He, Zhichun Zhou, Ziwei Wang, Tao Xue, Haolin Fei

Organizations: Department of Engineering, Beihang University, Beijing, China · Department of Automation, Tsinghua University, Beijing, China

Abstract

Many continuous-control policies are optimized as unbounded Gaussians and then mapped into bounded actions. We show that where entropy is measured changes the policy geometry learned by proximal policy optimization (PPO). In an 80-muscle MyoLeg task, a clipped Gaussian executes 89.07% of actions within 5% of a bound. A same-state decomposition shows that this is not due to variance alone: setting variance to zero still leaves 83.83% of actions near a bound, while 82.12% of state-conditioned means lie outside the executable interval. Replacing clipping with a tanh map does not remove the high-variance regime. For latent Gaussian entropy H(u), the entropy loss has zero gradient with respect to the mean and a constant variance-increasing gradient. For executed-action entropy H(a), the transform Jacobian adds an inward gradient on the mean. Across three matched MyoLeg seeds, near-boundary occupancy is 71.42%, 29.76%, and 18.83% under latent entropy, no entropy, and executed-action entropy. A 38-dimensional Dog-Stand replication with an independent CleanRL-based PPO implementation reproduces the ordering in mean geometry, which also survives shared-state evaluation and boundary margins from 1% to 10%. Direct mean penalties can match or exceed the centering produced by H(a), showing that interior means are not unique to executed entropy. However, matched mean geometry can coexist with substantially different variance and return. Entropy measurement space is therefore a coupled mean-variance design choice, and task return alone does not characterize bounded-policy geometry.

Figures & tables

Panel of Figure 1 — Figure 1: Study logic. The paper starts from an actuator-level phenomenon, separates mean placement from dispersion, changes only the space in which entropy is measured, measures the resulting gradients, and then tests the mechanism in an independent PPO implementation.

Condition	$R_{\mathrm{stoch}}$	Near bound (%)	log disp.
Motor, 14D	$12637\pm 941$	$9.49\pm 0.10$	$-1.594\pm 0.030$
Override, 80D	$13437\pm 730$	$87.38\pm 0.29$	$+0.832\pm 0.037$
Filter, 80D	$12389\pm 435$	$89.07\pm 0.16$	$+0.798\pm 0.029$
Symmetric $\tau$ , 80D	$12072\pm 250$	$88.73\pm 0.45$	$+0.852\pm 0.018$
Beta, 80D	$13698\pm 294$	$15.28\pm 0.30$	$-1.938\pm 0.008$

Table 1: Initial MyoLeg mechanism-isolation configurations. Values are mean

\pm

sample SD over three training seeds. Near-boundary occupancy uses an absolute margin of 0.05 from either bound of the configuration’s own action box, which is 5% of the muscle range

[0,1]

but 2.5% of the motor range

[-1,1]

; the motor row is therefore measured at a stricter relative threshold than the muscle rows, and a like-for-like 5%-of-range motor threshold is

|a|\geq 0.90

; the same canonical protocol gives

10.12\%\pm 0.06\%

at that threshold. All five configurations use the same entropy coefficient

c_{\mathrm{ent}}=10^{-3}

: the four Gaussian configurations on their latent Gaussian entropy

H(u)

, and the Beta control on the entropy of its Beta distribution. The motor reference uses its prescribed 78,000,000-step budget. For the Beta policy, log disp. is an effective log-dispersion: the state-averaged log of the per-dimension Beta standard deviation implied by the learned concentrations. Beta uses an effective log-dispersion and is not numerically identical to Gaussian

\log\sigma

Figure 2: The original MyoLeg phenomenon and its decomposition. A: Fixed-reset-seed traces for the first 300 control steps; no segment was selected after inspection. Motor actions are already in $[-1,1]$ . Muscle actions $a\in[0,1]$ are mapped to $y=2a-1$ only for display, so $|y|\geq 0.9$ is the same 5% boundary margin. The 10% and 89% annotations describe these single traces, not the multi-seed canonical aggregates. The Motor annotation is therefore at the like-for-like 5%-of-range threshold ( $|a|\geq 0.90$ ) noted in the Table 1 caption, not at the absolute 0.05 margin used in that table; the canonical aggregate at this matching threshold is $10.12\%\pm 0.06\%$ , so the closeness of the single-trace value to the table’s stricter-threshold 9.49% is coincidental. B: Same-state Filter decomposition at 145M steps, $n=3$ training seeds. Bars show near-boundary occupancy: $P_{11}=89.07\%$ , $P_{10}=83.83\%$ , and $P_{01}=77.51\%$ . Black diamonds show exact clipping-atom mass: 87.89%, 82.12%, and 75.61%. The large negative interaction reflects overlap under the 100% ceiling. C: 82.1% of state-conditioned Filter means lie outside $[0,1]$ and are exactly clipped; the mean-only cell $P_{10}$ is 83.8%.

Figure 3: Matched MyoLeg tanh experiment. A: Mean $\log\sigma$ during training. Thick lines and shading show the three-seed mean and min–max envelope over the full logged range; thin lines are individual seeds. Open circles are the three terminal 145M canonical values. A2: Separate late-training view of no entropy and $H(a)$ . B: Late-training same-state decomposition. No entropy has lower $P_{01}$ than $H(a)$ but higher $P_{10}$ and total occupancy. C: Frozen seed-0 gradient decomposition over 10 independent buffers. Positive loss gradient lowers $\sigma$ ; negative raises it. Under $H(u)$ and no entropy, $g_{\mu,\mathrm{ent}}\equiv 0$ , so its cosine with $\mu$ is undefined. Under $H(a)$ , the entropy mean-force cosine is approximately $+0.979$ , with essentially every evaluated state–action component receiving an inward contribution.

Condition	$\overline{\log\sigma}$	$R_{\mathrm{stoch}}$	$P_{11}$	$P_{10}$	$P_{01}$	$\mathbb{E}\|\mu\|$	$R_{\mathrm{det}}$	Det. near bound
$H(u)$	$+0.8835\pm 0.0499$	$13829\pm 635$	$71.42\pm 1.29$	$60.72\pm 2.23$	$51.80\pm 1.48$	$2.142\pm 0.148$	$899\pm 163$	$50.09\pm 1.72$
No entropy	$-0.6920\pm 0.0110$	$\mathbf{16011\pm 55}$	$29.76\pm 2.54$	$25.67\pm 2.62$	$\mathbf{1.53\pm 0.27}$	$1.035\pm 0.070$	$\mathbf{9154\pm 2480}$	$24.77\pm 1.71$
$H(a)$	$-0.5748\pm 0.0069$	$15539\pm 37$	$\mathbf{18.83\pm 0.28}$	$\mathbf{11.62\pm 0.56}$	$2.43\pm 0.10$	$\mathbf{0.716\pm 0.002}$	$5001\pm 781$	$\mathbf{10.35\pm 1.01}$

Table 2: Late-training MyoLeg tanh policies at 145,022,976 steps. Mean

\pm

sample SD over three training seeds.

P_{11}

and

P_{10}

are analytic probabilities (Section 3 ) on states visited by the stochastic policy;

P_{01}

depends only on the state-independent

\sigma

and involves no state set. The realized closed-loop occupancy agrees with

P_{11}

within 0.03 percentage points.

Figure 4: External replication on 38-dimensional DMC Dog-Stand with an independent, CleanRL-based PPO implementation. A: Canonical same-state geometry at 3M steps. The variance-only difference between no entropy and $H(a)$ is small; the large contrast is in mean placement. B: $P_{11}$ ordering is unchanged at 2.015M, 2.519M, and 3.000M for every training seed (nine of nine checkpoint–seed comparisons). This is late-training stability of the measured geometry, not an asymptotic convergence claim. C: Direct entropy force on the mean. Under $H(u)$ and no entropy the entropy mean-gradient vector is exactly zero and the cosine is not defined; crosses mark these zero-vector conditions. Under $H(a)$ , the cosine is approximately $+0.987$ across the three seeds.

Condition	$\overline{\log\sigma}$	$R_{\mathrm{stoch}}$	$R_{\mathrm{det}}$	$P_{11}$	$P_{10}$	$P_{01}$	$\mathbb{E}\|\mu\|$
$H(u)$	$-0.164\pm 0.113$	$470.5\pm 47.2$	$496.3\pm 59.3$	$45.57\pm 1.71$	$36.99\pm 1.46$	$17.74\pm 2.13$	$1.302\pm 0.043$
No entropy	$-1.313\pm 0.071$	$464.1\pm 73.5$	$470.2\pm 82.5$	$23.64\pm 1.47$	$22.25\pm 1.26$	$0.05\pm 0.05$	$0.964\pm 0.028$
Tanh 2 mean penalty	$-1.459\pm 0.048$	$483.9\pm 48.2$	$455.4\pm 52.1$	$0.20\pm 0.15$	$\mathbf{0.09\pm 0.10}$	$0.05\pm 0.04$	$\mathbf{0.208\pm 0.007}$
Pre-tanh L2	$-1.518\pm 0.040$	$465.1\pm 39.4$	$473.8\pm 32.3$	$2.49\pm 0.23$	$1.60\pm 0.25$	$0.01\pm 0.00$	$0.475\pm 0.015$
$H(a)$	$-0.918\pm 0.086$	$\mathbf{594.4\pm 34.0}$	$\mathbf{643.2\pm 58.4}$	$5.26\pm 1.46$	$1.91\pm 0.66$	$0.29\pm 0.12$	$0.489\pm 0.030$

Table 3: Dog-Stand canonical comparison at 3,000,320 steps. Mean

\pm

sample SD over three training seeds. The two direct mean regularizers were added after the primary three-condition experiment as mechanistic controls.

Figure 5: Dog-Stand mean–dispersion operating regimes. Small markers are individual training seeds; large markers are three-seed means and bars show sample SD. The horizontal axis is the canonical mean-only cell $P_{10}$ at the 5% margin; the vertical axis is mean $\log\sigma$ . Labels give canonical stochastic return. L2 and $H(a)$ have nearly matched mean geometry but substantially different variance and return regimes. The dashed connection highlights this comparison; it does not imply that retained dispersion causes the return difference.

Figure 6: Dog-Stand temporal execution structure, reported as secondary exploratory evidence. A: Opposite-boundary flip rate. A flip is a direct transition between $y\geq 0.9$ and $y\leq-0.9$ on consecutive control steps; the control period is 0.015 s (66.67 Hz). B: Mean dwell time inside the near-boundary set during stochastic rollout. C: Deterministic action variation, $\mathbb{E}|y_{t+1}-y_{t}|$ . Lower variation is consistent with smoother simulated action sequences; hardware consequences are not evaluated.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Margin	$H(u)$	No entropy	$H(a)$
Dog	1%	24.47 / 15.83	6.63 / 5.88	0.28 / 0.05
Dog	2.5%	35.34 / 26.31	14.18 / 13.02	1.63 / 0.44
Dog	5%	45.57 / 36.99	23.64 / 22.25	5.26 / 1.91
Dog	10%	57.77 / 50.46	37.58 / 36.10	14.69 / 7.44
MyoLeg	1%	56.40 / 40.46	10.30 / 7.69	2.77 / 0.82
MyoLeg	2.5%	64.70 / 51.62	19.51 / 15.81	9.28 / 4.75

Appendix

Table 4: Boundary-margin sensitivity. Entries are

P_{11}/P_{10}

in percent, averaged over three training seeds. The primary ordering

H(u)

>

no entropy

>

H(a)

holds for both quantities in every seed at every margin. Both 5% rows reproduce their frozen canonical values (Tables 2 and 3 ) exactly.

Donor	$H(u)$	No entropy	$H(a)$	Tanh 2 penalty	Pre-tanh L2
$H(u)$	37.05	28.31	4.56	0.59	5.30
No entropy	38.00	22.26	5.15	0.96	4.23
$H(a)$	42.06	28.62	1.88	0.57	4.52
Tanh 2 penalty	42.46	27.31	5.29	0.09	5.13
Pre-tanh L2	38.89	28.52	5.38	0.59	1.60
Equal-mixture pool	39.69	26.99	4.47	0.56	4.13

Appendix

Table 5: Dog common-state

P_{10}

(%), donor state set in rows and evaluated policy in columns, mean over three training seeds.

Condition	Entropy	Surrogate	Net
No entropy	$0$	$+1.015\!\times\!10^{-3}$	$+1.015\!\times\!10^{-3}$
		$[0.363,1.67]\!\times\!10^{-3}$	same
$H(u)$	$-1.000\!\times\!10^{-3}$	$+0.610\!\times\!10^{-3}$	$-0.390\!\times\!10^{-3}$
	exact	$[0.052,1.17]\!\times\!10^{-3}$	$[-0.949,0.168]\!\times\!10^{-3}$
$H(a)$	$-0.552\!\times\!10^{-3}$	$+0.604\!\times\!10^{-3}$	$+0.052\!\times\!10^{-3}$
		$[-0.175,1.38]\!\times\!10^{-3}$	$[-0.727,0.831]\!\times\!10^{-3}$

Appendix

Table 6: MyoLeg frozen-buffer

\log\sigma

gradient decomposition, 10 independent buffers per condition. Intervals are 95%

t

-intervals over buffers. Positive loss gradient lowers

\sigma

Setting	MyoLeg / Stable-Baselines3 PPO	Dog-Stand / CleanRL PPO
Observation / action dimensions	101 / 80 latent muscle actions	223 / 38 latent actions
Parallel environments	12, SubprocVecEnv	1, synchronous vector environment
Rollout length	2048 per environment	2048
Rollout batch	24,576 transitions	2,048 transitions
Minibatch size	128	64 (32 minibatches)
Update epochs	10	10

Appendix

Table 7: Training configuration for the primary tanh experiments. The Dog controls with direct mean penalties use the Dog column with

c_{\mathrm{ent}}=0

plus the stated penalty.

Component	MyoLeg	Dog-Stand
Actor mean	$101\rightarrow 512\rightarrow 512\rightarrow 80$ , ReLU hidden layers	$223\rightarrow 64\rightarrow 64\rightarrow 38$ , tanh hidden layers
Critic	$101\rightarrow 512\rightarrow 512\rightarrow 1$ , ReLU hidden layers	$223\rightarrow 64\rightarrow 64\rightarrow 1$ , tanh hidden layers
Actor dispersion	80 learned $\log\sigma_{i}$ parameters, initialized to zero	38 learned $\log\sigma_{i}$ parameters, initialized to zero
Initialization	Stable-Baselines3 MlpPolicy initialization	orthogonal; hidden gain $\sqrt{2}$ , actor-output gain 0.01, critic-output gain 1.0, zero biases
Trainable parameters	671,393	39,565

Appendix

Table 8: Policy and value network architecture. Both implementations use separate actor and critic MLPs and a state-independent learnable

\log\sigma

vector.

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 225 papers

Mar 5, 2026

BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning

Proximal constraints are fundamental to the stability of the Large Language Model reinforcement learning. While the canonical clipping mechanism in PPO serves as an efficient surrogate for trust regions, we identify a critical bottleneck: fixed bounds strictly constrain the upward update margin of low-probability actions, disproportionately suppressing high-advantage tail strategies and inducing rapid entropy collapse. To address this, we introduce Band-constrained Policy Optimization (BandPO). BandPO replaces canonical clipping with Band, a unified theoretical operator that projects trust regions defined by f-divergences into dynamic, probability-aware clipping intervals. Theoretical analysis confirms that Band effectively resolves this exploration bottleneck. We formulate this mapping as a convex optimization problem, guaranteeing a globally optimal numerical solution while deriving closed-form solutions for specific divergences. Extensive experiments across diverse models and datasets demonstrate that BandPO consistently outperforms canonical clipping and Clip-Higher, while robustly mitigating entropy collapse.

Yuan Li, Bo Wang, Yufei Gao +4

Fudan University · Shanghai Innovation Institute

RL for Language Models Proximal Policy Optimization Trust Region Policy Optimization

Ask PDF

Jun 22, 2026

KLip-PPO: A per-sample KL perspective on PPO-Clip

Proximal Policy Optimization (PPO) is the standard policy-gradient algorithm for on-policy reinforcement learning. The literature presents it in two forms, a clipped surrogate that bounds the importance ratio between successive policies and a Kullback-Leibler penalty between them. These forms are treated as separate algorithms with their own gradients, their own hyperparameters, and their own reference implementations, and a sizeable body of empirical work compares them. We show that the gradient of the clipped surrogate is reproduced exactly by a Kullback-Leibler surrogate whose coefficient varies per sample, with closed-form dependence on the importance ratio and the advantage. The identity holds at every minibatch step and across the entire inner loop, and on five MuJoCo continuous-control benchmarks the two losses produce indistinguishable training curves. The reformulation exposes a structural feature of the clipped surrogate that the min notation hides. PPO-Clip's implicit per-sample penalty is a step function at the boundary of the trust region, and the shape of this coefficient is the natural design axis for generalising the algorithm. We sketch the resulting follow-up directions in the discussion.

Riccardo Colletti, Robin Holzinger

University of California, Berkeley

KL-Regularized RL Policy Gradient Methods Proximal Policy Optimization Policy Learning

Ask PDF

Oct 1, 2026

FERPO: Forward Entropy-Regularized Policy Optimization

Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler (KL) divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy. By limiting the target distribution's deviation from the rollout policy, the KL regularization helps keep these importance weights well behaved. In contrast to reverse-KL objectives, which can favor a subset of the target distribution's modes, the forward-KL objective encourages coverage of multiple high-value modes and thereby promotes exploration. Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains. Computational benchmarks also demonstrate faster actor updates than Relative Entropy Pathwise Policy Optimization (REPPO).

Sebastian Sanokowski, Alireza Sarmadi, Majid Khadiv

Applied and Theoretical Aspects of Robot Intelligence (ATARI) Lab, Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich

KL-Regularized RL Policy Optimization Maximum Entropy RL Continuous Control Policy Learning Regularized RL

Ask PDF

Abstract

Policy Optimization

Explore similar work

BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning

KLip-PPO: A per-sample KL perspective on PPO-Clip

FERPO: Forward Entropy-Regularized Policy Optimization