The Price of Failure Risk: Clipped Barrier Policy Gradients with Convex Regularization
Authors: Pekka Malo, Lauri Viitasaari, Antti Suominen, Eeva Vilkkumaa, Olli Tahvonen
Organizations: Aalto University, Department of Information and Service Management, Finland. · University of Helsinki, Department of Forest Sciences, Finland.
In many control problems, actions consume a resource budget, such as a wear allowance, and the probability of exhausting it within a given horizon must stay below a tolerance. A simple, widely used approach is to augment the state with the remaining budget, subtract an interior barrier from the utility and penalize exhaustion, so that policy-gradient methods can be applied unchanged. We clip the combined one-step reward from below at a level m to keep the returns of exploratory policies finite. The clip level is a price of failure risk, since the clipped value decomposes exactly into the utility before failure, the cost of the interior margin and m times a geometric failure risk. Consequently, an approximately optimal policy maximizes utility net of margin cost among policies with no greater risk. Its risk exceeds that of any comparison policy by at most order 1/m and can be certified from its value or from validation data. Our main result proves exponential convergence of the population policy-gradient flow for every parameter regularizer that is convex along generalized geodesics and sufficiently strong relative to the price. This extends the mean-field analysis from relative entropy to weight decay and to finite networks. Relative entropy adds parameter noise and a pull toward a prior, whereas weight decay provides only the pull. Action regularization bounds changes in failure risk by the relative entropy to a reference policy. The bounds are evaluated on an exact maintenance benchmark, a fuel-tank controller, a solvable flow and a benchmark-suite task.
Figures & tables
Figure 1. The logic of the clipped barrier. (a) One-step reward against the next reserve z′ for an action of utility u=0.6 , the barrier B(z′)=0.15log(1+1/z′) and m=1 . The unclipped reward u−B(z′) tends to −∞ at the boundary. Under the clipped barrier it is clipped at −m , a charge applied in every period from failure onward. Its interior loss (shaded) secures a margin before failure, which a failure-only price does not. (b) Schematic of the geometric risk against the utility net of interior loss for a policy class. The class has a positive risk floor, the smallest risk any of its policies attains, and its frontier is the largest utility net of interior loss attainable at each risk. A price m selects the point at which a line of slope m/(1−β) supports the frontier, a prescribed tolerance α selects the frontier point at α , and the risk between the two is unused. Below the risk floor no policy exists and no multiplier converges.
Figure 2. The return decomposition on a path of Example 2 , with constant rate a=0.8 , b=1 , β=0.95 , B(z′)=0.1log(1+1/z′) , and m=0.3 . (a) Random wear exhausts the allowance at action τ=10 , before maintenance access on the illustrated realization. (b) Discounted revenue before breakdown, interior loss and the downtime price from breakdown onward. (c) Their exact pathwise totals, where d=1−β ; the infinite downtime tail is summed analytically. Subscript ω denotes this realization; U , Im and Vm in Eq. ( 12 ) are expectations. The path is recorded in the replication repository. The figure illustrates the accounting identity and does not show a trained policy.
Figure 3. The operational effect of the price in the wear-allowance model ( β=0.95 , b=1 ; barrier optima with B(z′)=0.1log(1+b/z′) ). (a) Exact policies reduce the operating rate as the allowance shrinks; with an interior barrier a margin is retained. (b) The utility–risk frontier, fixed-price and adaptive-multiplier PPO policies, and the tolerance 0.05 . (c) Geometric risk qβ against the price m and the 1/m comparison bound. Exact solutions and all eight-seed learned configurations are detailed in Sect. 6.1 . With a finite price, useful throughput is obtained at the attained failure risk, without a prescribed level.
Figure 4. The curvature budget of Theorem 8 on a toy one-dimensional objective along a generalized geodesic, the position ϑ∈[0,1] standing for the law, with −Vm(ϑ)=0.35(ϑ−0.85)2+0.025cos(6πϑ) , the prior at ϑ=0.15 and the regularizer κH=21κλH(ϑ−0.15)2 ; curves are shifted to a minimum of zero. (a) The negative clipped value −Vm is nonconvex, with curvature deficit Cm ; the parameter regularizer κH adds the curvature κλH , and the sum is convex above the threshold κλH>Cm (at 1.5Cm ), not below it (at Cm/2 ). (b) The global minimizer against the strength κλH/Cm moves from the unregularized minimizer to the prior as the strength grows and is already near the prior at the threshold. This is the trade-off between convergence and bias.
Configuration
Setting
qβ (%)
bound (%)
U
Im
gap (%)
pass
near (%)
at 0.9b (%)
A
mean act. qβ (%)
Fixed price
m=0
59.90±0.18
–
5.35±0.02
1.01
−0.2 ( 0.4 )
0/8
61.4
62.8
–
59.59
m=0.1†
32.73±23.09
100.0
5.14±0.16
1.63
0.8 ( 2.9 )
0/8
49.5
57.2
–
32.11
m=0.3
4.42±0.75
89.8
4.76±0.05
3.82
0.9 ( 1.5 )
3/8
29.4
45.1
–
3.91
m=1
1.90±0.26
27.2
4.50±0.04
3.68
1.7 ( 2.9 )
8/8
12.6
32.6
–
1.61
m=3
0.97±0.24
9.3
4.22±0.08
3.27
3.8 ( 4.9 )
8/8
3.9
17.7
–
0.68
m=10
0.53±0.07
3.0
4.01±0.05
3.06
4.8 ( 6.3 )
8/8
2.0
11.0
–
0.35
Table 1. Learned policies in the wear-allowance example, eight seeds per configuration, with qβ and U (mean ± between-seed standard deviation); bound, the right side of the price–risk law ( 15 ) with the idle policy as comparison and the measured optimization error, capped at one and averaged over the seeds, where the law applies; interior loss evaluated with B(z)=0.1log(1+b/z) also for policies trained without that barrier; gap, the shortfall of U from the exact frontier at the attained qβ (mean and maximum); pass, the seeds whose binomial upper bound on qβ , simultaneous over all learned policies at confidence 95% , is at most 0.05 (each episode ends at an independent geometric time, so a breakdown before it is a Bernoulli event of probability qβ ); near, the frequency of coming within 0.05b of exhaustion; at 0.9b , the breakdown probability when the true allowance is 0.9b ; A , a pre-clipping Gaussian upper bound on action relative entropy to the incumbent rule (last row); and qβ of the mean-action policy, a different candidate.
Figure 5. Fuel-tank study, five seeds per configuration, 1000 fresh episodes per policy. (a) Exhaustion probability against the price for tanks of 22 , 21 and 20 units (small points seeds, large points means, thin lines joining the configurations of one seed at tank 22 , dotted lines the tolerances 0.05 and 0.02 ). (b) Stopped utility against exhaustion probability at tank 22 for the fixed prices and the adaptive multiplier with the dual step 100 fixed before the runs and the step 5 added afterward; the dashed line is the utility without a tank. (c) Multiplier during continued training, mean and range over seeds. Exhaustion falls at the price 0.5 and crashes rise with the price. With the small dual step the multiplier approaches the lump-sum charge of the fixed price m=0.5 , and with the step fixed before the runs it increases throughout training. Panels (a) and (b) support (O8) and panel (c) supports (O9); the values are given in Table F.3 of Online Resource 1.
Figure 6. The solvable flow for relative entropy alone and for Hω,G with the power integrands r∈{3/2,2,3} (prior N(0,1) , m=1 , ω=1 ). (a) Objective gap relative to its initial value at the certified strength κ=1.05Cm/λW . The dashed line is the envelope e−2Λmℓ of Eq. ( 27 ), the dash-dotted line the example-specific envelope e−2(κλW−C♯)ℓ with the finite-difference curvature estimate C♯ of the example’s value curve (Appendix F.4 of Online Resource 1), and the dotted line e−2κλWℓ . (b) The same at κ=1 , far below the threshold. (c) Improvement of the minimizer over the prior, Vm(μ∗)−Vm(Γ) , against κ ; the horizontal line is the largest possible improvement, and with an integrand of faster growth the minimizer stays closer to the prior. Panels (a) and (b) support (O10), panel (c) supports (O11), and (O12) rests on Fig. F.3 of Online Resource 1.
Figure 7. Parameter anchoring of PPO toward the incumbent in the wear-allowance example at m=0.03 , eight seeds per configuration. (a) Breakdown probability and (b) stopped utility against the anchor strength κp ( 0 is the unanchored control) for the first set of runs (prior dispersion 0.37 , circles), the matched-prior set (dispersion 0.1 , diamonds) and the mean-only prior (triangles); dotted lines are the two prior policies, the lower one the incumbent rule, and the dashed line the exact optimum at this price. (c) Breakdown probability against the action relative entropy A to the incumbent rule, κp increasing from right to left along each set and the action regularization strength η∈{0.01,0.1,1} (triangles) likewise; the first set inherits its prior policy, while the matched anchor and action regularization inherit the rule’s risk.
Figure 8. Roles of regularizers on the solvable flow (plateau initial law, prior N(0,1) , means over five seeds of 2000 units; configurations in Table F.9 of Online Resource 1). (a) Clipped value against learning time for relative entropy at κ1∈{0.1,1} , the quadratic potential at κ2∈{1,10} and their sum at κ1=0.1 , κ2∈{1,10} ; the potential alone and the sums with weak noise stay at the plateau value. (b) Spread of the units, which collapses under the potential alone (floor 10−3 ), stays close to the zero-objective reference under the strongly confined sum and grows under relative entropy. (c) Final clipped value (left axis) and final spread (right axis) against κ1 at the pull κ2=1 ; the value leaves the plateau once the spread approaches the plateau’s width.
Recently, it has been shown that the Stochastic Gradient Bandit (SGB) algorithm converges to a globally optimal policy with a constant learning rate. However, these guarantees rely on unrealistic assumptions about the learning process, namely that the probability of the optimal action is always bounded away from zero. We attribute this to the lack of an explicit exploration mechanism in SGB. To address these limitations, we propose to regularize the SGB objective with a log-barrier on the parametric policy, structurally enforcing a minimal amount of exploration. We prove that Log-Barrier Stochastic Gradient Bandit (LB-SGB) matches the sample complexity of SGB, but also converges (at a slower rate) without any assumptions on the learning process. We also show a connection between the log-barrier regularization and Natural Policy Gradient, as both exploit the geometry of the policy space by controlling the Fisher information. We validate our theoretical findings through numerical simulations, showing the benefits of the log-barrier regularization.
Leonardo Cesani, Matteo Papini, Marcello Restelli
Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy
Regularization-based methods have become a standard approach for training Deep Reinforcement Learning policies against adversarial input perturbations. In this paper, we unify these methods by deriving new upper bounds on the performance gap between the nominal and worst-case policies. Each upper bound is expressed as an existing regularization objective plus a KL-divergence penalty between the nominal and worst-case policies, which further explains why adding a KL penalty improves robustness in practice. Building on these bounds, we formulate robust training as a constrained optimization problem, showing that existing methods correspond to the special case of a fixed Lagrange multiplier. We instead update the multiplier jointly with the policy to automatically tune the regularization weight. Finally, we conduct extensive adversarial evaluations across several continuous control tasks to validate our theoretical analysis.
Amine Andam, Jamal Bentahar, Mustapha Hedabou
Mohammed VI Polytechnic University · Khalifa University · Concordia University
Expected-cost constraints can still permit rare, high-cost events. Monte Carlo conditional value at risk (CVaR) gradients can be noisy at high confidence, whereas critics that model an outcome distribution add complexity. We propose BCPPO (Bachelier-Inspired Constrained Proximal Policy Optimization), a proximal policy optimization (PPO) method. Separately initialized cost-prediction networks (critics), trained with random sample masks, produce disagreement that marks predictions sensitive to which state-action regions occur in the training data and to critic training. A Bachelier formula for the expected amount above a reference level converts this disagreement into a smooth policy-update penalty. Gradients from this penalty do not alter the critics, so temporal-difference (TD) critic learning is unchanged. A saturation-aware controller adjusts the mean-cost penalty and stops accumulated error from growing while that penalty is clipped. Deployment retains only the policy network. The disagreement penalty is neither a tail-event probability nor a guaranteed error bound, and it provides no safety guarantee. Across 175 runs with shared tasks, costs, budgets, training steps, and evaluation seeds, no comparator attains both higher mean return and lower mean CVaR than BCPPO in any task. On Push1, BCPPO has no lower return and no higher CVaR than every comparator, with at least one strict gain. These results support a practical balance among reward, caution around cost predictions that vary across trained critics, and policy-only deployment.