SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

Dominik Wagner; Ankit Kanwar; Luke Ong

cs.LGDec 29, 2025

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

Authors: Dominik Wagner, Ankit Kanwar, Luke Ong

Organizations: NTU Singapore · Sony Corporation

Abstract

In safety-critical domains, reinforcement learning (RL) systems must satisfy strict, zero-cost safety constraints while achieving meaningful task performance. Existing model-free methods can struggle to achieve high safety without substantially compromising task performance. We introduce \emph{Safety-Biased Trust Region Policy Optimisation (SB-TRPO)}, a principled approach to RL with zero-cost constraints, which requires only a fixed fraction of the maximal cost reduction achievable within the trust region, thus retaining flexibility for reward optimisation. We show that the idealised update nevertheless converges to zero cost and maximal reward amongst zero-cost policies in finite MDPs. A practical gradient-based approximation provides local improvements in both safety and reward under suitable gradient alignment. Experiments on \emph{Safety Gymnasium} demonstrate high safety alongside strong task performance.

Figures & tables

Panel of Figure 1 — Figure 1: Deterministic MDP for which CPO yields reward-suboptimal policies.

Panel of Figure 2 — Figure 2: Visualisation of the adaptive convex combination $\Delta$ of $\Delta_{r}$ and $\Delta_{c}$ given by Eq. 4 for $\epsilon_{k}\coloneqq 1.4=-\beta\cdot\langle g_{c},\Delta_{c}\rangle$ , where $\beta\coloneqq 0.7$ , and the special case that $\Delta_{r}=g_{r}$ and $\Delta_{c}=-g_{c}$ .

Method	Car Circle		Point Push		Car Push		Point Goal		Car Goal		Hopper Velocity		Swimmer Velocity
Method	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑
Ours	$7.5\pm 1.5$	$\boldsymbol{0.99\pm 0.01}$	$\boldsymbol{0.33\pm 0.18}$	$\boldsymbol{0.79\pm 0.08}$	$\boldsymbol{0.38\pm 0.12}$	$0.71\pm 0.07$	$\boldsymbol{1.6\pm 0.5}$	$0.68\pm 0.07$	$\boldsymbol{1.4\pm 0.4}$	$0.66\pm 0.08$	$860\pm 378$	$\boldsymbol{1.0\pm 0.0}$	$48\pm 26$	$\boldsymbol{0.98\pm 0.02}$
PPO-Lag	$7.5\pm 2.4$	$0.86\pm 0.17$	$-0.29\pm 0.8$	$0.59\pm 0.22$	$-0.13\pm 0.34$	$0.65\pm 0.1$	$\mathit{1.3\pm 1.1}$	$\mathit{0.47\pm 0.13}$	$\mathit{1.2\pm 1.2}$	$\mathit{0.45\pm 0.16}$	$510\pm 500$	$0.58\pm 0.44$	$18\pm 38$	$0.57\pm 0.42$
TRPO-Lag	$\boldsymbol{11\pm 4}$	$0.81\pm 0.26$	$-0.49\pm 0.45$	$0.78\pm 0.16$	$-0.19\pm 0.25$	$0.89\pm 0.04$	$0.09\pm 0.29$	$\boldsymbol{0.79\pm 0.06}$	$0.23\pm 0.38$	$0.75\pm 0.07$	$72\pm 90$	$0.6\pm 0.5$	$\mathit{1.3\pm 5.1}$	$\mathit{0.33\pm 0.42}$
CPO	$0.96\pm 0.76$	$\boldsymbol{0.99\pm 0.03}$	$-1.2\pm 0.9$	$0.77\pm 0.19$	$-1.3\pm 0.3$	$0.93\pm 0.05$	$-1.3\pm 0.4$	$0.86\pm 0.05$	$-0.73\pm 0.23$	$0.82\pm 0.09$	$880\pm 182$	$\boldsymbol{1.0\pm 0.0}$	$-3.8\pm 9.3$	$0.99\pm 0.03$
CUP	$7.1\pm 3.5$	$0.75\pm 0.28$	$0.12\pm 0.19$	$0.67\pm 0.19$	$\mathit{-0.0062\pm 0.1729}$	$\mathit{0.47\pm 0.19}$	$\mathit{0.95\pm 0.74}$	$\mathit{0.15\pm 0.09}$	$\mathit{0.93\pm 0.74}$	$\mathit{0.15\pm 0.1}$	$\boldsymbol{1000\pm 289}$	$0.72\pm 0.19$	$34\pm 52$	$0.9\pm 0.1$

Table 1: Final metrics across level-2 Safety Gymnasium tasks. Red: negative (safe) reward, italics: safety probability

<50\%

, grey: Pareto-dominated pairs, bold: best value per metric (only among methods with positive (safe) reward and safety probability

\geq 50\%

for the other metric).

Panel of Figure 3 — Figure 3: Angles between policy updates and reward/cost gradients for Car Circle

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Safe navigation tasks of Safety Gymnasium ( Ji et al., 2023 ) (images taken from https://safety-gymnasium.readthedocs.io/en/latest/environments/safe_navigation.html )

Task	Objective	dim.
Circle	Circulate around the center of the circular area (green) while remaining within the boundaries (yellow), with rewards highest along the outermost circumference and increasing with speed.	28/40
Push	Push the box (yellow) to the goal location (green) while avoiding hazards and pillars (both blue).	76/88
Goal	Navigate to the goal (green) while avoiding hazards (blue) and vases (teal).	60/72

Appendix

Table 2: Overview of the Safety Gymnasium navigation tasks used in our experiments. The dimensionality refers to the state space for the Point and Car agents, respectively. (Recall that we always use level 2 of the tasks, which is the most difficult.)

Hyperparameter	SB-TRPO (Ours)		PPO-Lag	TRPO-Lag	CPO	CUP	CPPO-PID	FOCOPS	RCPO	PCPO	P3O	C-TRPO
	GAE	MC
Discount Factor $\gamma$	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99
Target KL	0.01	0.01	0.02	0.01	0.01	0.02	0.02	0.02	0.01	0.01	0.02	0.01
GAE $\lambda$	0.95	–	0.95	0.95	0.95	0.95	0.95	0.95	0.95	0.95	0.95	0.95
Timesteps per Epoch	20000	20000	20000	20000	20000	20000	20000	20000	20000	20000	20000	20000
Policy Hidden Layers	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}	{64, 64}

Appendix

Table 3: Hyperparameters for SB-TRPO (Ours) and baselines.

Figure 6: Point Button: training longer, doing better.

Figure 7: Car Circle: staying safe and improving reward

	SB-TRPO (MC)	SB-TRPO (GAE)	TRPO-Lag	CPO	PPO-Lag	CUP	CPPO-PID	FOCOPS	RCPO	PCPO	P3O	C-TRPO
Time ↓	$\mathbf{0.26\pm 0.08}$	$8.12\pm 0.95$	$5.85\pm 0.79$	$6.59\pm 1.96$	$69.83\pm 8.29$	$84.60\pm 15.54$	$88.32\pm 18.82$	$53.54\pm 19.34$	$4.40\pm 0.35$	$5.74\pm 1.19$	$78.05\pm 20.14$	$3.32\pm 0.16$

Appendix

Table 4: Update times per epoch on Point Goal on a single, non-parallelized training run, based on the hyperparameters in Table 3 . Mean

\pm

STDEV is reported, rounded to 2 decimal places. The least update time consumed per epoch is in bold . Note that PPO-Lagrangian, in contrast to TRPO-Lagrangian, has a budget of up to 40 gradient steps per epoch (as per Table 3 ).

Panel of Figure 8 — Figure 8: Angles between policy updates and reward/cost gradients for Point Button

Method	Cost	Car Circle		Point Push		Point Goal		Swimmer Velocity
Method	limit	Safe Rew.	Safe Prob.	Safe Rew.	Safe Prob.	Safe Rew.	Safe Prob.	Safe Rew.	Safe Prob.
Ours	–	$7.5\pm 1.5$	$\boldsymbol{0.99\pm 0.01}$	$0.33\pm 0.18$	$\boldsymbol{0.79\pm 0.08}$	$\boldsymbol{1.6\pm 0.5}$	$0.68\pm 0.07$	$\boldsymbol{48\pm 26}$	$\boldsymbol{0.98\pm 0.02}$
TRPO-Lag	0	$\boldsymbol{11\pm 4}$	$0.81\pm 0.26$	$-0.49\pm 0.45$	$0.78\pm 0.16$	$0.09\pm 0.29$	$\boldsymbol{0.79\pm 0.06}$	$\mathit{1.3\pm 5.1}$	$\mathit{0.33\pm 0.42}$
TRPO-Lag	15	$\mathit{6.8\pm 1.5}$	$\mathit{0.44\pm 0.09}$	$0.29\pm 0.28$	$0.64\pm 0.12$	$-0.18\pm 0.53$	$0.82\pm 0.08$	$\mathit{0.0\pm 0.0}$	$\mathit{0.0\pm 0.0}$
CPO	0	$0.96\pm 0.76$	$\boldsymbol{0.99\pm 0.03}$	$-1.2\pm 0.9$	$0.77\pm 0.19$	$-1.3\pm 0.4$	$0.86\pm 0.05$	$-3.8\pm 9.3$	$0.99\pm 0.03$
CPO	15	$8.3\pm 1.6$	$0.54\pm 0.11$	$0.24\pm 0.14$	$0.61\pm 0.07$	$0.29\pm 0.67$	$0.69\pm 0.11$	$\mathit{0.0014\pm 0.0098}$	$\mathit{0.001\pm 0.0052}$

Appendix

Table 5: Selected baselines with a relaxed cost limit of 15 (using the same highlighting as in Table 1 ).

Panel of Figure 9 — Figure 9: Ablation study of safety bias $\beta$

Advantage	$\beta$	Car Goal		Point Goal		Car Circle		Swimmer Velocity
Advantage	$\beta$	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑	Safe Rew. ↑	Safe Prob. ↑
MC	$0.75$	${1.4\pm 0.4}$	$0.66\pm 0.08$	${1.6\pm 0.5}$	$0.68\pm 0.07$	$7.5\pm 1.5$	${0.99\pm 0.01}$	$48\pm 26$	${0.98\pm 0.02}$
GAE	$0.75$	$5.9\pm 0.9$	$0.40\pm 0.07$	$5.0\pm 1.3$	$0.51\pm 0.09$	$10.2\pm 2.2$	$0.96\pm 0.07$	$0.67\pm 8.1$	$1.0\pm 0.0$
GAE	$0.8$	$2.3\pm 0.35$	$0.62\pm 0.066$	$1.7\pm 0.92$	$0.69\pm 0.098$	$8\pm 3.1$	$0.99\pm 0.017$	$18\pm 35$	$0.98\pm 0.031$

Appendix

Table 6: Effect of advantage estimation on performance

Panel of Figure 10 — Figure 10: Histograms of relative approximation errors ( $\frac{\widehat{x}-x}{x}$ ).

Explore similar work

Sampling-Based Safe Reinforcement Learning
May 19, 2026Luca Vignola, Bruce D. Lee, Manish Prajapat +4Safety Constraints Efficient Exploration
ETH Zurich
CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
Jun 12, 2026Ayoub Belouadah, Sylvain Kubler, Yves Le TraonSafety Constraints
SnT, University of Luxembourg
Boundary-Seeking Policy Gradient for Safe Reinforcement Learning
Aug 10, 2026Chenhua Fan, Jiahui Zhu, Yuhang Zhang +1Policy Gradient Safety Constraints
School of EECS, Washington State University, Pullman, WA 99163, USA

May 19, 2026, cs.LGJ/K move · Enter open · S save

Sampling-Based Safe Reinforcement Learning

Luca Vignola, Bruce D. Lee, Manish Prajapat, Manuel Wendl+3

ETH Zurich

Safety Constraints Efficient Exploration

Open paper AskPDF arXiv

Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.

Abstract

Explore similar work

Sampling-Based Safe Reinforcement Learning

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Boundary-Seeking Policy Gradient for Safe Reinforcement Learning