SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

Dominik Wagner; Ankit Kanwar; Luke Ong

cs.LGDec 29, 2025

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

Authors: Dominik Wagner, Ankit Kanwar, Luke Ong

Organizations: NTU Singapore · Sony Corporation

Abstract

In safety-critical domains, reinforcement learning (RL) systems must satisfy strict, zero-cost safety constraints while achieving meaningful task performance. Existing model-free methods can struggle to achieve high safety without substantially compromising task performance. We introduce \emph{Safety-Biased Trust Region Policy Optimisation (SB-TRPO)}, a principled approach to RL with zero-cost constraints, which requires only a fixed fraction of the maximal cost reduction achievable within the trust region, thus retaining flexibility for reward optimisation. We show that the idealised update nevertheless converges to zero cost and maximal reward amongst zero-cost policies in finite MDPs. A practical gradient-based approximation provides local improvements in both safety and reward under suitable gradient alignment. Experiments on \emph{Safety Gymnasium} demonstrate high safety alongside strong task performance.

Figures & tables

Panel of Figure 1 — Figure 1: Deterministic MDP for which CPO yields reward-suboptimal policies.

Panel of Figure 2 — Figure 2: Visualisation of the adaptive convex combination $\Delta$ of $\Delta_{r}$ and $\Delta_{c}$ given by Eq. 4 for $\epsilon_{k}\coloneqq 1.4=-\beta\cdot\langle g_{c},\Delta_{c}\rangle$ , where $\beta\coloneqq 0.7$ , and the special case that $\Delta_{r}=g_{r}$ and $\Delta_{c}=-g_{c}$ .

Method	Car Circle		Point Push		Car Push		Point Goal		Car Goal		Hopper Velocity		Swimmer Velocity
Method	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑
Ours	$7.5\pm 1.5$	$\boldsymbol{0.99\pm 0.01}$	$\boldsymbol{0.33\pm 0.18}$	$\boldsymbol{0.79\pm 0.08}$	$\boldsymbol{0.38\pm 0.12}$	$0.71\pm 0.07$	$\boldsymbol{1.6\pm 0.5}$	$0.68\pm 0.07$	$\boldsymbol{1.4\pm 0.4}$	$0.66\pm 0.08$	$860\pm 378$	$\boldsymbol{1.0\pm 0.0}$	$48\pm 26$	$\boldsymbol{0.98\pm 0.02}$
PPO-Lag	$7.5\pm 2.4$	$0.86\pm 0.17$	$-0.29\pm 0.8$	$0.59\pm 0.22$	$-0.13\pm 0.34$	$0.65\pm 0.1$	$\mathit{1.3\pm 1.1}$	$\mathit{0.47\pm 0.13}$	$\mathit{1.2\pm 1.2}$	$\mathit{0.45\pm 0.16}$	$510\pm 500$	$0.58\pm 0.44$	$18\pm 38$	$0.57\pm 0.42$
TRPO-Lag	$\boldsymbol{11\pm 4}$	$0.81\pm 0.26$	$-0.49\pm 0.45$	$0.78\pm 0.16$	$-0.19\pm 0.25$	$0.89\pm 0.04$	$0.09\pm 0.29$	$\boldsymbol{0.79\pm 0.06}$	$0.23\pm 0.38$	$0.75\pm 0.07$	$72\pm 90$	$0.6\pm 0.5$	$\mathit{1.3\pm 5.1}$	$\mathit{0.33\pm 0.42}$
CPO	$0.96\pm 0.76$	$\boldsymbol{0.99\pm 0.03}$	$-1.2\pm 0.9$	$0.77\pm 0.19$	$-1.3\pm 0.3$	$0.93\pm 0.05$	$-1.3\pm 0.4$	$0.86\pm 0.05$	$-0.73\pm 0.23$	$0.82\pm 0.09$	$880\pm 182$	$\boldsymbol{1.0\pm 0.0}$	$-3.8\pm 9.3$	$0.99\pm 0.03$
CUP	$7.1\pm 3.5$	$0.75\pm 0.28$	$0.12\pm 0.19$	$0.67\pm 0.19$	$\mathit{-0.0062\pm 0.1729}$	$\mathit{0.47\pm 0.19}$	$\mathit{0.95\pm 0.74}$	$\mathit{0.15\pm 0.09}$	$\mathit{0.93\pm 0.74}$	$\mathit{0.15\pm 0.1}$	$\boldsymbol{1000\pm 289}$	$0.72\pm 0.19$	$34\pm 52$	$0.9\pm 0.1$

Table 1: Final metrics across level-2 Safety Gymnasium tasks. Red: negative (safe) reward, italics: safety probability

<50\%

, grey: Pareto-dominated pairs, bold: best value per metric (only among methods with positive (safe) reward and safety probability

\geq 50\%

for the other metric).

Panel of Figure 3 — Figure 3: Angles between policy updates and reward/cost gradients for Car Circle

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Safe navigation tasks of Safety Gymnasium ( Ji et al., 2023 ) (images taken from https://safety-gymnasium.readthedocs.io/en/latest/environments/safe_navigation.html )

Task	Objective	dim.
Circle	Circulate around the center of the circular area (green) while remaining within the boundaries (yellow), with rewards highest along the outermost circumference and increasing with speed.	28/40
Push	Push the box (yellow) to the goal location (green) while avoiding hazards and pillars (both blue).	76/88
Goal	Navigate to the goal (green) while avoiding hazards (blue) and vases (teal).	60/72

Appendix

Table 2: Overview of the Safety Gymnasium navigation tasks used in our experiments. The dimensionality refers to the state space for the Point and Car agents, respectively. (Recall that we always use level 2 of the tasks, which is the most difficult.)

Hyperparameter	SB-TRPO (Ours)		PPO-Lag	TRPO-Lag	CPO	CUP	CPPO-PID	FOCOPS	RCPO	PCPO	P3O	C-TRPO
	GAE	MC
Discount Factor $\gamma$	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99
Target KL	0.01	0.01	0.02	0.01	0.01	0.02	0.02	0.02	0.01	0.01	0.02	0.01
GAE $\lambda$	0.95	–	0.95	0.95	0.95	0.95	0.95	0.95	0.95	0.95	0.95	0.95
Timesteps per Epoch	20000	20000	20000	20000	20000	20000	20000	20000	20000	20000	20000	20000
Policy Hidden Layers	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}

Appendix

Table 3: Hyperparameters for SB-TRPO (Ours) and baselines.

Figure 6: Point Button: training longer, doing better.

Figure 7: Car Circle: staying safe and improving reward

	SB-TRPO (MC)	SB-TRPO (GAE)	TRPO-Lag	CPO	PPO-Lag	CUP	CPPO-PID	FOCOPS	RCPO	PCPO	P3O	C-TRPO
Time ↓	$\mathbf{0.26\pm 0.08}$	$8.12\pm 0.95$	$5.85\pm 0.79$	$6.59\pm 1.96$	$69.83\pm 8.29$	$84.60\pm 15.54$	$88.32\pm 18.82$	$53.54\pm 19.34$	$4.40\pm 0.35$	$5.74\pm 1.19$	$78.05\pm 20.14$	$3.32\pm 0.16$

Appendix

Table 4: Update times per epoch on Point Goal on a single, non-parallelized training run, based on the hyperparameters in Table 3 . Mean

\pm

STDEV is reported, rounded to 2 decimal places. The least update time consumed per epoch is in bold . Note that PPO-Lagrangian, in contrast to TRPO-Lagrangian, has a budget of up to 40 gradient steps per epoch (as per Table 3 ).

Panel of Figure 8 — Figure 8: Angles between policy updates and reward/cost gradients for Point Button

Method	Cost	Car Circle		Point Push		Point Goal		Swimmer Velocity
Method	limit	Safe Rew.	Safe Prob.	Safe Rew.	Safe Prob.	Safe Rew.	Safe Prob.	Safe Rew.	Safe Prob.
Ours	–	$7.5\pm 1.5$	$\boldsymbol{0.99\pm 0.01}$	$0.33\pm 0.18$	$\boldsymbol{0.79\pm 0.08}$	$\boldsymbol{1.6\pm 0.5}$	$0.68\pm 0.07$	$\boldsymbol{48\pm 26}$	$\boldsymbol{0.98\pm 0.02}$
TRPO-Lag	0	$\boldsymbol{11\pm 4}$	$0.81\pm 0.26$	$-0.49\pm 0.45$	$0.78\pm 0.16$	$0.09\pm 0.29$	$\boldsymbol{0.79\pm 0.06}$	$\mathit{1.3\pm 5.1}$	$\mathit{0.33\pm 0.42}$
TRPO-Lag	15	$\mathit{6.8\pm 1.5}$	$\mathit{0.44\pm 0.09}$	$0.29\pm 0.28$	$0.64\pm 0.12$	$-0.18\pm 0.53$	$0.82\pm 0.08$	$\mathit{0.0\pm 0.0}$	$\mathit{0.0\pm 0.0}$
CPO	0	$0.96\pm 0.76$	$\boldsymbol{0.99\pm 0.03}$	$-1.2\pm 0.9$	$0.77\pm 0.19$	$-1.3\pm 0.4$	$0.86\pm 0.05$	$-3.8\pm 9.3$	$0.99\pm 0.03$
CPO	15	$8.3\pm 1.6$	$0.54\pm 0.11$	$0.24\pm 0.14$	$0.61\pm 0.07$	$0.29\pm 0.67$	$0.69\pm 0.11$	$\mathit{0.0014\pm 0.0098}$	$\mathit{0.001\pm 0.0052}$

Appendix

Table 5: Selected baselines with a relaxed cost limit of 15 (using the same highlighting as in Table 1 ).

Panel of Figure 9 — Figure 9: Ablation study of safety bias $\beta$

Advantage	$\beta$	Car Goal		Point Goal		Car Circle		Swimmer Velocity
Advantage	$\beta$	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑
MC	$0.75$	${1.4\pm 0.4}$	$0.66\pm 0.08$	${1.6\pm 0.5}$	$0.68\pm 0.07$	$7.5\pm 1.5$	${0.99\pm 0.01}$	$48\pm 26$	${0.98\pm 0.02}$
GAE	$0.75$	$5.9\pm 0.9$	$0.40\pm 0.07$	$5.0\pm 1.3$	$0.51\pm 0.09$	$10.2\pm 2.2$	$0.96\pm 0.07$	$0.67\pm 8.1$	$1.0\pm 0.0$
GAE	$0.8$	$2.3\pm 0.35$	$0.62\pm 0.066$	$1.7\pm 0.92$	$0.69\pm 0.098$	$8\pm 3.1$	$0.99\pm 0.017$	$18\pm 35$	$0.98\pm 0.031$

Appendix

Table 6: Effect of advantage estimation on performance

Panel of Figure 10 — Figure 10: Histograms of relative approximation errors ( $\frac{\widehat{x}-x}{x}$ ).

Explore similar work

May 19, 2026cs.LG

Sampling-Based Safe Reinforcement Learning

Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.

Luca Vignola, Bruce D. Lee, Manish Prajapat +4

ETH Zurich

Safety Constraints Efficient Exploration

Ask PDF

Jun 12, 2026cs.AI

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep RL, they often suffer from delayed constraint correction, leading to oscillatory behavior and prolonged safety violations. In this paper, we propose Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method that incorporates local constraint sensitivity into policy updates. CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety, compensating for delayed Lagrange multiplier updates, reducing oscillations near the boundary, and preserving the KKT solutions of the original constrained problem. Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns compared to state-of-the-art primal-dual and penalty-based methods

Ayoub Belouadah, Sylvain Kubler, Yves Le Traon

SnT, University of Luxembourg

Safety Constraints

Ask PDF

Aug 10, 2026cs.LG

Boundary-Seeking Policy Gradient for Safe Reinforcement Learning

Safe reinforcement learning maximizes reward subject to safety constraints. For Constrained Markov Decision Processes, the linear-programming view over occupancy measures implies that whenever the constraint is active at optimality, the optimal policy lies exactly on the constraint boundary, yet standard gradient-based methods do not exploit this structure and often settle in the feasible interior. We introduce Boundary-Seeking Policy Gradient (BSPG), a first-order method whose update combines a tangential component that improves reward while preserving cost to first order with a signed, residual-driven normal component that regulates the policy toward the active boundary from either side; the combined direction admits an algebraic Lagrangian form with an induced coefficient and no learned dual variable. Under exact gradients and stated regularity conditions, the constraint residual converges to zero from either side with a finite-horizon

O(1/\sqrt{T})

bound, the tangential component is a reward-ascent direction on the boundary, and any convergent parameter sequence is stationary on the active constraint set, satisfying the KKT conditions when the limit is also a local maximizer over the feasible set. This complements existing analyses, which certify feasibility but do not characterize the constraint value at convergence. On a standard Safety-Gymnasium navigation task, BSPG attains higher reward while tracking the boundary more tightly than the compared baselines.

Chenhua Fan, Jiahui Zhu, Yuhang Zhang +1

School of EECS, Washington State University, Pullman, WA 99163, USA

Policy Gradient Safety Constraints Markov Decision Processes Structural Constraints

Ask PDF