The Price of Failure Risk: Clipped Barrier Policy Gradients with Convex Regularization

Pekka Malo; Lauri Viitasaari; Antti Suominen; Eeva Vilkkumaa; Olli Tahvonen

cs.LGNov 28, 2024

The Price of Failure Risk: Clipped Barrier Policy Gradients with Convex Regularization

Authors: Pekka Malo, Lauri Viitasaari, Antti Suominen, Eeva Vilkkumaa, Olli Tahvonen

Organizations: Aalto University, Department of Information and Service Management, Finland. · University of Helsinki, Department of Forest Sciences, Finland.

Abstract

In many control problems, actions consume a resource budget, such as a wear allowance, and the probability of exhausting it within a given horizon must stay below a tolerance. A simple, widely used approach is to augment the state with the remaining budget, subtract an interior barrier from the utility and penalize exhaustion, so that policy-gradient methods can be applied unchanged. We clip the combined one-step reward from below at a level m to keep the returns of exploratory policies finite. The clip level is a price of failure risk, since the clipped value decomposes exactly into the utility before failure, the cost of the interior margin and m times a geometric failure risk. Consequently, an approximately optimal policy maximizes utility net of margin cost among policies with no greater risk. Its risk exceeds that of any comparison policy by at most order 1/m and can be certified from its value or from validation data. Our main result proves exponential convergence of the population policy-gradient flow for every parameter regularizer that is convex along generalized geodesics and sufficiently strong relative to the price. This extends the mean-field analysis from relative entropy to weight decay and to finite networks. Relative entropy adds parameter noise and a pull toward a prior, whereas weight decay provides only the pull. Action regularization bounds changes in failure risk by the relative entropy to a reference policy. The bounds are evaluated on an exact maintenance benchmark, a fuel-tank controller, a solvable flow and a benchmark-suite task.

Figures & tables

Figure 1. The logic of the clipped barrier. (a) One-step reward against the next reserve

z^{\prime}

for an action of utility

u=0.6

, the barrier

B(z^{\prime})=0.15\log(1+1/z^{\prime})

and

m=1

. The unclipped reward

u-B(z^{\prime})

tends to

-\infty

at the boundary. Under the clipped barrier it is clipped at

-m

, a charge applied in every period from failure onward. Its interior loss (shaded) secures a margin before failure, which a failure-only price does not. (b) Schematic of the geometric risk against the utility net of interior loss for a policy class. The class has a positive risk floor, the smallest risk any of its policies attains, and its frontier is the largest utility net of interior loss attainable at each risk. A price

m

selects the point at which a line of slope

m/(1-\beta)

supports the frontier, a prescribed tolerance

\alpha

selects the frontier point at

\alpha

, and the risk between the two is unused. Below the risk floor no policy exists and no multiplier converges.

Figure 2. The return decomposition on a path of Example 2 , with constant rate

a=0.8

b=1

\beta=0.95

B(z^{\prime})=0.1\log(1+1/z^{\prime})

, and

m=0.3

. (a) Random wear exhausts the allowance at action

\tau=10

, before maintenance access on the illustrated realization. (b) Discounted revenue before breakdown, interior loss and the downtime price from breakdown onward. (c) Their exact pathwise totals, where

d=1-\beta

; the infinite downtime tail is summed analytically. Subscript

\omega

denotes this realization;

U

\mathcal{I}_{m}

and

V_{m}

in Eq. ( 12 ) are expectations. The path is recorded in the replication repository. The figure illustrates the accounting identity and does not show a trained policy.

Figure 3. The operational effect of the price in the wear-allowance model (

\beta=0.95

b=1

; barrier optima with

B(z^{\prime})=0.1\log(1+b/z^{\prime})

). (a) Exact policies reduce the operating rate as the allowance shrinks; with an interior barrier a margin is retained. (b) The utility–risk frontier, fixed-price and adaptive-multiplier PPO policies, and the tolerance

0.05

. (c) Geometric risk

q_{\beta}

against the price

m

and the

1/m

comparison bound. Exact solutions and all eight-seed learned configurations are detailed in Sect. 6.1 . With a finite price, useful throughput is obtained at the attained failure risk, without a prescribed level.

Figure 4. The curvature budget of Theorem 8 on a toy one-dimensional objective along a generalized geodesic, the position

\vartheta\in[0,1]

standing for the law, with

-V_{m}(\vartheta)=0.35(\vartheta-0.85)^{2}+0.025\cos(6\pi\vartheta)

, the prior at

\vartheta=0.15

and the regularizer

\kappa\mathcal{H}=\tfrac{1}{2}\kappa\lambda_{\mathcal{H}}(\vartheta-0.15)^{2}

; curves are shifted to a minimum of zero. (a) The negative clipped value

-V_{m}

is nonconvex, with curvature deficit

C_{m}

; the parameter regularizer

\kappa\mathcal{H}

adds the curvature

\kappa\lambda_{\mathcal{H}}

, and the sum is convex above the threshold

\kappa\lambda_{\mathcal{H}}>C_{m}

(at

1.5\,C_{m}

), not below it (at

C_{m}/2

). (b) The global minimizer against the strength

\kappa\lambda_{\mathcal{H}}/C_{m}

moves from the unregularized minimizer to the prior as the strength grows and is already near the prior at the threshold. This is the trade-off between convergence and bias.

Configuration	Setting	$q_{\beta}$ (%)	bound (%)	$U$	$\mathcal{I}_{m}$	gap (%)	pass	near (%)	at $0.9b$ (%)	$\mathcal{A}$	mean act. $q_{\beta}$ (%)
Fixed price	$m=0$	$59.90\pm 0.18$	–	$5.35\pm 0.02$	$1.01$	$-0.2$ ( $0.4$ )	$0/8$	$61.4$	$62.8$	–	$59.59$
	$m=0.1$ $\dagger$	$32.73\pm 23.09$	$100.0$	$5.14\pm 0.16$	$1.63$	$0.8$ ( $2.9$ )	$0/8$	$49.5$	$57.2$	–	$32.11$
	$m=0.3$	$4.42\pm 0.75$	$89.8$	$4.76\pm 0.05$	$3.82$	$0.9$ ( $1.5$ )	$3/8$	$29.4$	$45.1$	–	$3.91$
	$m=1$	$1.90\pm 0.26$	$27.2$	$4.50\pm 0.04$	$3.68$	$1.7$ ( $2.9$ )	$8/8$	$12.6$	$32.6$	–	$1.61$
	$m=3$	$0.97\pm 0.24$	$9.3$	$4.22\pm 0.08$	$3.27$	$3.8$ ( $4.9$ )	$8/8$	$3.9$	$17.7$	–	$0.68$
	$m=10$	$0.53\pm 0.07$	$3.0$	$4.01\pm 0.05$	$3.06$	$4.8$ ( $6.3$ )	$8/8$	$2.0$	$11.0$	–	$0.35$

Table 1. Learned policies in the wear-allowance example, eight seeds per configuration, with

q_{\beta}

and

U

(mean

\pm

between-seed standard deviation); bound, the right side of the price–risk law ( 15 ) with the idle policy as comparison and the measured optimization error, capped at one and averaged over the seeds, where the law applies; interior loss evaluated with

B(z)=0.1\log(1+b/z)

also for policies trained without that barrier; gap, the shortfall of

U

from the exact frontier at the attained

q_{\beta}

(mean and maximum); pass, the seeds whose binomial upper bound on

q_{\beta}

, simultaneous over all learned policies at confidence

95\%

, is at most

0.05

(each episode ends at an independent geometric time, so a breakdown before it is a Bernoulli event of probability

q_{\beta}

); near, the frequency of coming within

0.05\,b

of exhaustion; at

0.9\,b

, the breakdown probability when the true allowance is

0.9\,b

;

\mathcal{A}

, a pre-clipping Gaussian upper bound on action relative entropy to the incumbent rule (last row); and

q_{\beta}

of the mean-action policy, a different candidate.

Figure 5. Fuel-tank study, five seeds per configuration,

1000

fresh episodes per policy. (a) Exhaustion probability against the price for tanks of

22

21

and

20

units (small points seeds, large points means, thin lines joining the configurations of one seed at tank

22

, dotted lines the tolerances

0.05

and

0.02

). (b) Stopped utility against exhaustion probability at tank

22

for the fixed prices and the adaptive multiplier with the dual step

100

fixed before the runs and the step

5

added afterward; the dashed line is the utility without a tank. (c) Multiplier during continued training, mean and range over seeds. Exhaustion falls at the price

0.5

and crashes rise with the price. With the small dual step the multiplier approaches the lump-sum charge of the fixed price

m=0.5

, and with the step fixed before the runs it increases throughout training. Panels (a) and (b) support (O8) and panel (c) supports (O9); the values are given in Table F.3 of Online Resource 1.

Figure 6. The solvable flow for relative entropy alone and for

\mathcal{H}_{\omega,G}

with the power integrands

r\in\{3/2,2,3\}

(prior

N(0,1)

m=1

\omega=1

). (a) Objective gap relative to its initial value at the certified strength

\kappa=1.05\,C_{m}/\lambda_{W}

. The dashed line is the envelope

e^{-2\Lambda_{m}\ell}

of Eq. ( 27 ), the dash-dotted line the example-specific envelope

e^{-2(\kappa\lambda_{W}-C_{\sharp})\ell}

with the finite-difference curvature estimate

C_{\sharp}

of the example’s value curve (Appendix F.4 of Online Resource 1), and the dotted line

e^{-2\kappa\lambda_{W}\ell}

. (b) The same at

\kappa=1

, far below the threshold. (c) Improvement of the minimizer over the prior,

V_{m}(\mu^{*})-V_{m}(\Gamma)

, against

\kappa

; the horizontal line is the largest possible improvement, and with an integrand of faster growth the minimizer stays closer to the prior. Panels (a) and (b) support (O10), panel (c) supports (O11), and (O12) rests on Fig. F.3 of Online Resource 1.

Figure 7. Parameter anchoring of PPO toward the incumbent in the wear-allowance example at

m=0.03

, eight seeds per configuration. (a) Breakdown probability and (b) stopped utility against the anchor strength

\kappa_{p}

(

0

is the unanchored control) for the first set of runs (prior dispersion

0.37

, circles), the matched-prior set (dispersion

0.1

, diamonds) and the mean-only prior (triangles); dotted lines are the two prior policies, the lower one the incumbent rule, and the dashed line the exact optimum at this price. (c) Breakdown probability against the action relative entropy

\mathcal{A}

to the incumbent rule,

\kappa_{p}

increasing from right to left along each set and the action regularization strength

\eta\in\{0.01,0.1,1\}

(triangles) likewise; the first set inherits its prior policy, while the matched anchor and action regularization inherit the rule’s risk.

Figure 8. Roles of regularizers on the solvable flow (plateau initial law, prior

N(0,1)

, means over five seeds of

2000

units; configurations in Table F.9 of Online Resource 1). (a) Clipped value against learning time for relative entropy at

\kappa_{1}\in\{0.1,1\}

, the quadratic potential at

\kappa_{2}\in\{1,10\}

and their sum at

\kappa_{1}=0.1

\kappa_{2}\in\{1,10\}

; the potential alone and the sums with weak noise stay at the plateau value. (b) Spread of the units, which collapses under the potential alone (floor

10^{-3}

), stays close to the zero-objective reference under the strongly confined sum and grows under relative entropy. (c) Final clipped value (left axis) and final spread (right axis) against

\kappa_{1}

at the pull

\kappa_{2}=1

; the value leaves the plateau once the spread approaches the plateau’s width.

Policy Gradient Methods

Policy gradient methods optimize stochastic policies by estimating how parameter changes affect expected reward.

See all 165 papers

How Log-Barrier Helps Exploration in Policy Optimization
Mar 16, 2026Leonardo Cesani, Matteo Papini, Marcello RestelliRL Exploration Policy Gradient Methods
Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy
A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning
Sep 11, 2026Amine Andam, Jamal Bentahar, Mustapha HedabouKL-Regularized RL Adversarial Robustness
Mohammed VI Polytechnic University · Khalifa University · Concordia University
BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learning
Aug 31, 2026Dongsheng Hou, Yanqiao Chen, Yuhan RuiReinforcement Learning Constrained RL
Southern University of Science and Technology

Mar 16, 2026J/K move · Enter open · S save

How Log-Barrier Helps Exploration in Policy Optimization

Leonardo Cesani, Matteo Papini, Marcello Restelli

Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy

RL Exploration Policy Gradient Methods Multi-Armed Bandits

Open paper AskPDF arXiv

Recently, it has been shown that the Stochastic Gradient Bandit (SGB) algorithm converges to a globally optimal policy with a constant learning rate. However, these guarantees rely on unrealistic assumptions about the learning process, namely that the probability of the optimal action is always bounded away from zero. We attribute this to the lack of an explicit exploration mechanism in SGB. To address these limitations, we propose to regularize the SGB objective with a log-barrier on the parametric policy, structurally enforcing a minimal amount of exploration. We prove that Log-Barrier Stochastic Gradient Bandit (LB-SGB) matches the sample complexity of SGB, but also converges (at a slower rate) without any assumptions on the learning process. We also show a connection between the log-barrier regularization and Natural Policy Gradient, as both exploit the geometry of the policy space by controlling the Fisher information. We validate our theoretical findings through numerical simulations, showing the benefits of the log-barrier regularization.

Abstract

Policy Gradient Methods

Explore similar work

How Log-Barrier Helps Exploration in Policy Optimization

A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning

BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learning