CORE-RL: Confidence-Oriented Reliability Evaluation of Black-Box Reinforcement Learning Policies

Santhosh GS; Ananya Ravi; Devika Jay; Abhishek Sarkar; Perepu Satheesh Kumar; Saurav Prakash; Kaushik Dey; Balaraman Ravindran

cs.AIOct 3, 2026

CORE-RL: Confidence-Oriented Reliability Evaluation of Black-Box Reinforcement Learning Policies

Authors: Santhosh GS, Ananya Ravi, Devika Jay, Abhishek Sarkar, Perepu Satheesh Kumar, Saurav Prakash, Kaushik Dey, Balaraman Ravindran

Organizations: Center for Responsible AI IIT Madras Chennai, India · Ericsson Research India

Abstract

The deployment of Reinforcement Learning (RL) agents in critical domains must be preceded with a pipeline to evaluate the alignment of the RL agent with complex multi-objective specifications and robustness under real-world environmental drift. However, to protect intellectual property, the RL agent may be delivered for evaluation as opaque executable or remote API, which makes traditional evaluation techniques based on the internals of the policies infeasible. To address this gap, CORE-RL: Confidence-Oriented Reliability Evaluation of black box RL policy is proposed in this paper. The CORE-RL pipeline introduces a Unified Reliability Metric that formally integrates early task termination and safety constraint violations, preventing unsafe policies from masking failures through premature episode halts. By subjecting the policy to a noise certification envelope of perceptual noise, actuation noise and change in environment dynamics, the pipeline computes the finite-sample Clopper-Pearson bounds on unified reliability metric and Hoeffdings' lower bound on reward and safety cost. The pipeline then defines safe operational design domain to report high-confidence certificates for safety and expected performance. Experiments on continuous control tasks demonstrate the CORE-RL pipeline's ability to automatically reject non-compliant policies and map the safe Operational Design Domain (ODD) of safety-aware policies. Thus CORE-RL provides an evaluation framework towards a quantitative, transparent and reproducible, statistical rationale necessary to safely evaluate, compare, and deploy black box RL solutions.

Figures & tables

Environment	Policy Class	Environment Deviation
SafetyAntVelocity-v1	PPO, PPO-Lag, TRPO, TRPO-Lag	Gaussian : $k_{a},k_{s}=\{0,0.05,0.1\}$
		Uniform - $k_{a},k_{s}=\{0,0.05,0.1\}$
		Black-box adversarial $L_{\infty}$ noise $k_{a},k_{s}=\{0,0.03,0.06\}$
SafetyPointGoal1-v0	PPO, PPO-Lag, TRPO, TRPO-Lag	Gaussian $k_{a},k_{s}=\{0,0.05,0.1\}$
		Uniform $k_{a},k_{s}=\{0,0.05,0.1\}$
		Black-box adversarial $L_{\infty}$ noise $k_{a},k_{s}=\{0,0.03,0.06\}$

Table 1: Evaluation coverage for 10,000 rollouts per policy–perturbation configuration.

Environment	Policy	Task-only failure		Safety-only failure		Unified failure		CORE-RL Evaluation
		Rate	B-CP UB	Rate	B-CP UB	Rate	B-CP UB
SafetyAnt	PPO	0.1555	0.1640	0.9751	0.9785	0.9913	0.9933	Fail
	PPO-Lag	0.0158	0.0189	0.7972	0.8063	0.8093	0.8182	Fail
	TRPO	0.0146	0.0176	0.9978	0.9987	0.9988	0.9994	Fail
	TRPO-Lag	0.0081	0.0104	0.0000	0.0004	0.0081	0.0104	Pass
SafetyCar	PPO	0.0000	0.0004	0.1952	0.2044	0.1952	0.2044	Fail

Table 2: Unified Reliability Metric under nominal evaluation at

d=100

Policy	$(\sigma_{a},\sigma_{s})$	Gaussian				Uniform
		Emp. fail	B-CP UB	Mean reward	Hoeffding LCB	Emp. fail	B-CP UB	Mean reward	Hoeffding LCB
PPO	$(0.05,0.00)$	0.1090	0.1163	26.45	-273.54	0.1086	0.1159	26.49	-273.50
PPO	$(0.10,0.00)$	0.1037	0.1109	26.34	-273.65	0.1081	0.1154	26.46	-273.54
PPO	$(0.10,0.10)$	0.4790	0.4905	2.01	-297.98	0.3221	0.3329	3.74	-296.25
PPO-Lag	$(0.05,0.00)$	0.0809	0.0873	11.10	-288.89	0.0821	0.0886	11.00	-289.00
PPO-Lag	$(0.10,0.00)$	0.0810	0.0875	11.05	-288.94	0.0800	0.0864	10.97	-289.02

Table 3: Matched SafetyPointGoal Gaussian and Uniform perturbation results at

d=100

Panel of Figure 1 — Figure 1: Operational Design Domain (ODD) for SafetyAnt under bounded adversarial $L_{\infty}$ perturbations at $d=100$ and $p^{\star}=0.05$ . Gaussian reliability certificates illustrating certified operational design domains (ODDs) for TRPO-Lag in the left pane attains certification at the stricter threshold d = 100, whereas PPO exhibits a high risk envelope.

Panel of Figure 2 — Figure 2: Operational Design Domain (ODD) for the SafetyCar TRPO-Lagrangian agent under environment perturbations at $d=100$ and $p^{\star}=0.05$ . In each panel, the left subplot shows the empirical mean reward over 10,000 episodes per perturbation configuration, while the right subplot shows the Bonferroni-corrected Hoeffding lower bound at 90% family-wise confidence. Together, these plots show how task performance changes across the action-noise and state-noise plane, while the lower-bound surface provides a conservative high-confidence estimate of reward under each perturbation configuration.

Appendix figures & tables65 assets

Supplementary material from the paper’s appendix.

Appendix

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0.00,0.00)$	0.0000	0.1952	0.1952	0.2044	FAIL
PPO	$(0.00,0.05)$	0.7014	0.1462	0.7450	0.7549	FAIL
PPO	$(0.00,0.10)$	0.8066	0.1379	0.8372	0.8456	FAIL
PPO	$(0.05,0.00)$	0.0000	0.1960	0.1960	0.2052	FAIL
PPO	$(0.05,0.05)$	0.7067	0.1496	0.7474	0.7573	FAIL
PPO	$(0.05,0.10)$	0.7984	0.1367	0.8272	0.8358	FAIL

Appendix

Table B.4: Reliability certificates for PPO on SafetyCar under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0.00,0.00)$	0.1010	0.0374	0.1369	0.1449	FAIL
PPO-Lag	$(0.00,0.05)$	0.8307	0.1846	0.8573	0.8652	FAIL
PPO-Lag	$(0.00,0.10)$	0.8776	0.2035	0.9002	0.9069	FAIL
PPO-Lag	$(0.05,0.00)$	0.1018	0.0400	0.1396	0.1477	FAIL
PPO-Lag	$(0.05,0.05)$	0.8259	0.1888	0.8545	0.8625	FAIL
PPO-Lag	$(0.05,0.10)$	0.8749	0.2009	0.8965	0.9034	FAIL

Appendix

Table B.5: Reliability certificates for PPO-Lagrangian on SafetyCar under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0.00,0.00)$	0.0000	0.1970	0.1970	0.2063	FAIL
TRPO	$(0.00,0.05)$	0.7137	0.2053	0.7672	0.7768	FAIL
TRPO	$(0.00,0.10)$	0.8285	0.2199	0.8610	0.8688	FAIL
TRPO	$(0.05,0.00)$	0.0000	0.2015	0.2015	0.2108	FAIL
TRPO	$(0.05,0.05)$	0.7153	0.2091	0.7697	0.7793	FAIL
TRPO	$(0.05,0.10)$	0.8293	0.2163	0.8605	0.8683	FAIL

Appendix

Table B.6: Reliability certificates for TRPO on SafetyCar under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\bar{R}$	$R_{\mathrm{H\text{-}LCB}}$	$\bar{C}$	$C_{\mathrm{H\text{-}UCB}}$
TRPO-Lag	$(0.00,0.00)$	23.35	-276.64	28.88	43.88
TRPO-Lag	$(0.00,0.05)$	-0.82	-300.81	37.04	52.04
TRPO-Lag	$(0.00,0.10)$	-1.54	-301.53	37.38	52.38
TRPO-Lag	$(0.05,0.00)$	23.61	-276.38	29.39	44.39
TRPO-Lag	$(0.05,0.05)$	-0.82	-300.81	35.39	50.39
TRPO-Lag	$(0.05,0.10)$	-1.56	-301.56	36.17	51.17

Appendix

Table B.7: Performance and cost certificates for TRPO-Lagrangian on SafetyCar under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0.00,0.00)$	0.0164	0.0300	0.0460	0.0510	FAIL
TRPO-Lag	$(0.00,0.05)$	0.6921	0.1417	0.7406	0.7506	FAIL
TRPO-Lag	$(0.00,0.10)$	0.8014	0.1424	0.8338	0.8422	FAIL
TRPO-Lag	$(0.05,0.00)$	0.0161	0.0288	0.0444	0.0493	PASS
TRPO-Lag	$(0.05,0.05)$	0.6971	0.1325	0.7434	0.7533	FAIL
TRPO-Lag	$(0.05,0.10)$	0.8124	0.1308	0.8418	0.8501	FAIL

Appendix

Table B.8: Reliability certificates for TRPO-Lagrangian on SafetyCar under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0.00,0.00)$	0.0000	0.1952	0.1952	0.2044	FAIL
PPO	$(0.00,0.05)$	0.5885	0.1534	0.6481	0.6590	FAIL
PPO	$(0.00,0.10)$	0.7327	0.1455	0.7738	0.7833	FAIL
PPO	$(0.05,0.00)$	0.0001	0.1997	0.1998	0.2091	FAIL
PPO	$(0.05,0.05)$	0.5858	0.1540	0.6433	0.6542	FAIL
PPO	$(0.05,0.10)$	0.7372	0.1447	0.7760	0.7855	FAIL

Appendix

Table B.9: Reliability certificates for PPO on SafetyCar under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0.00,0.00)$	0.1010	0.0374	0.1369	0.1449	FAIL
PPO-Lag	$(0.00,0.05)$	0.7376	0.1469	0.7723	0.7818	FAIL
PPO-Lag	$(0.00,0.10)$	0.8406	0.1853	0.8654	0.8731	FAIL
PPO-Lag	$(0.05,0.00)$	0.1046	0.0415	0.1445	0.1527	FAIL
PPO-Lag	$(0.05,0.05)$	0.7427	0.1486	0.7772	0.7867	FAIL
PPO-Lag	$(0.05,0.10)$	0.8411	0.1850	0.8679	0.8756	FAIL

Appendix

Table B.10: Reliability certificates for PPO-Lagrangian on SafetyCar under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0.00,0.00)$	0.0000	0.1970	0.1970	0.2063	FAIL
TRPO	$(0.00,0.05)$	0.4997	0.1943	0.5936	0.6048	FAIL
TRPO	$(0.00,0.10)$	0.7577	0.2233	0.8073	0.8162	FAIL
TRPO	$(0.05,0.00)$	0.0000	0.1976	0.1976	0.2069	FAIL
TRPO	$(0.05,0.05)$	0.5015	0.1956	0.5991	0.6103	FAIL
TRPO	$(0.05,0.10)$	0.7562	0.2098	0.8000	0.8091	FAIL

Appendix

Table B.11: Reliability certificates for TRPO on SafetyCar under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0.00,0.00)$	0.0164	0.0300	0.0460	0.0510	FAIL
TRPO-Lag	$(0.00,0.05)$	0.4989	0.1165	0.5620	0.5734	FAIL
TRPO-Lag	$(0.00,0.10)$	0.7289	0.1378	0.7705	0.7801	FAIL
TRPO-Lag	$(0.05,0.00)$	0.0182	0.0315	0.0490	0.0542	FAIL
TRPO-Lag	$(0.05,0.05)$	0.5022	0.1167	0.5670	0.5784	FAIL
TRPO-Lag	$(0.05,0.10)$	0.7308	0.1332	0.7713	0.7808	FAIL

Appendix

Table B.12: Reliability certificates for TRPO-Lagrangian on SafetyCar under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0.00,0.00)$	0.0000	0.1952	0.1952	0.2044	FAIL
PPO	$(0.00,0.03)$	0.0000	0.1948	0.1948	0.2040	FAIL
PPO	$(0.00,0.06)$	0.0000	0.2017	0.2017	0.2110	FAIL
PPO	$(0.03,0.00)$	0.0000	0.1969	0.1969	0.2062	FAIL
PPO	$(0.03,0.03)$	0.0001	0.1994	0.1995	0.2088	FAIL
PPO	$(0.03,0.06)$	0.0000	0.2024	0.2024	0.2118	FAIL

Appendix

Table B.13: Reliability certificates for PPO on SafetyCar under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0.00,0.00)$	0.1010	0.0374	0.1369	0.1449	FAIL
PPO-Lag	$(0.00,0.03)$	0.1048	0.0393	0.1425	0.1507	FAIL
PPO-Lag	$(0.00,0.06)$	0.0993	0.0403	0.1380	0.1461	FAIL
PPO-Lag	$(0.03,0.00)$	0.1036	0.0372	0.1391	0.1472	FAIL
PPO-Lag	$(0.03,0.03)$	0.1037	0.0393	0.1411	0.1492	FAIL
PPO-Lag	$(0.03,0.06)$	0.1009	0.0392	0.1377	0.1458	FAIL

Appendix

Table B.14: Reliability certificates for PPO-Lagrangian on SafetyCar under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0.00,0.00)$	0.1970	0.1970	0.2063	FAIL
TRPO	$(0.00,0.03)$	0.1948	0.1948	0.2040	FAIL
TRPO	$(0.00,0.06)$	0.1971	0.1971	0.2064	FAIL
TRPO	$(0.03,0.00)$	0.1994	0.1994	0.2087	FAIL
TRPO	$(0.03,0.03)$	0.1949	0.1949	0.2041	FAIL
TRPO	$(0.03,0.06)$	0.1997	0.1997	0.2090	FAIL

Appendix

Table B.15: Reliability certificates for TRPO on SafetyCar under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\bar{R}$	$R_{\mathrm{H\text{-}LCB}}$	$\bar{C}$	$C_{\mathrm{H\text{-}UCB}}$
TRPO-Lag	$(0.00,0.00)$	23.35	-276.64	28.88	43.88
TRPO-Lag	$(0.00,0.03)$	23.57	-276.42	29.42	44.42
TRPO-Lag	$(0.00,0.06)$	23.47	-276.52	29.70	44.70
TRPO-Lag	$(0.03,0.00)$	23.38	-276.61	29.93	44.93
TRPO-Lag	$(0.03,0.03)$	23.33	-276.66	29.46	44.46
TRPO-Lag	$(0.03,0.06)$	23.56	-276.44	29.37	44.37

Appendix

Table B.16: Performance and cost certificates for TRPO-Lagrangian on SafetyCar under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0.00,0.00)$	0.0164	0.0300	0.0460	0.0510	FAIL
TRPO-Lag	$(0.00,0.03)$	0.0177	0.0272	0.0443	0.0492	PASS
TRPO-Lag	$(0.00,0.06)$	0.0169	0.0311	0.0473	0.0524	FAIL
TRPO-Lag	$(0.03,0.00)$	0.0167	0.0320	0.0480	0.0531	FAIL
TRPO-Lag	$(0.03,0.03)$	0.0155	0.0330	0.0482	0.0533	FAIL
TRPO-Lag	$(0.03,0.06)$	0.0158	0.0306	0.0458	0.0508	FAIL

Appendix

Table B.17: Reliability certificates for TRPO-Lagrangian on SafetyCar under bounded

L_{\infty}

perturbations.

Policy	$(m,f,\zeta)$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0.75,0.75,1.00)$	0.0002	0.1932	0.1933	0.2025	FAIL
PPO	$(0.75,1.00,1.00)$	0.0000	0.1999	0.1999	0.2092	FAIL
PPO	$(0.75,1.25,1.00)$	0.0000	0.2122	0.2122	0.2217	FAIL
PPO	$(1.00,0.75,1.00)$	0.0000	0.1830	0.1830	0.1920	FAIL
PPO	$(1.00,1.00,1.00)$	0.0000	0.1992	0.1992	0.2085	FAIL
PPO	$(1.00,1.25,1.00)$	0.0001	0.2163	0.2164	0.2260	FAIL

Appendix

Table B.18: Reliability certificates for PPO on SafetyCar under parameter-shift perturbations.

Policy	$(m,f,\zeta)$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0.75,0.75,1.00)$	0.1427	0.0421	0.1814	0.1904	FAIL
PPO-Lag	$(0.75,1.00,1.00)$	0.0987	0.0435	0.1411	0.1492	FAIL
PPO-Lag	$(0.75,1.25,1.00)$	0.0671	0.0417	0.1082	0.1155	FAIL
PPO-Lag	$(1.00,0.75,1.00)$	0.1448	0.0389	0.1804	0.1894	FAIL
PPO-Lag	$(1.00,1.00,1.00)$	0.1024	0.0431	0.1446	0.1528	FAIL
PPO-Lag	$(1.00,1.25,1.00)$	0.0693	0.0425	0.1109	0.1183	FAIL

Appendix

Table B.19: Reliability certificates for PPO-Lagrangian on SafetyCar under parameter-shift perturbations.

Policy	$(m,f,\zeta)$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0.75,0.75,1.00)$	0.1966	0.1966	0.2059	FAIL
TRPO	$(0.75,1.00,1.00)$	0.2011	0.2011	0.2104	FAIL
TRPO	$(0.75,1.25,1.00)$	0.2191	0.2191	0.2287	FAIL
TRPO	$(1.00,0.75,1.00)$	0.1790	0.1790	0.1879	FAIL
TRPO	$(1.00,1.00,1.00)$	0.2007	0.2007	0.2100	FAIL
TRPO	$(1.00,1.25,1.00)$	0.2153	0.2153	0.2249	FAIL

Appendix

Table B.20: Reliability certificates for TRPO on SafetyCar under parameter-shift perturbations.

Policy	$(m,f,\zeta)$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0.75,0.75,1.00)$	0.0179	0.0351	0.0525	0.0578	FAIL
TRPO-Lag	$(0.75,1.00,1.00)$	0.0146	0.0339	0.0478	0.0529	FAIL
TRPO-Lag	$(0.75,1.25,1.00)$	0.0140	0.0323	0.0459	0.0509	FAIL
TRPO-Lag	$(1.00,0.75,1.00)$	0.0184	0.0289	0.0468	0.0519	FAIL
TRPO-Lag	$(1.00,1.00,1.00)$	0.0168	0.0318	0.0481	0.0532	FAIL
TRPO-Lag	$(1.00,1.25,1.00)$	0.0168	0.0314	0.0480	0.0531	FAIL

Appendix

Table B.21: Reliability certificates for TRPO-Lagrangian on SafetyCar under parameter-shift perturbations.

Policy	$(r_{\mathrm{dyn}},k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0.00,0.00,0.00)$	0.0000	0.2013	0.2013	0.2111	FAIL
PPO	$(0.00,0.00,0.05)$	0.5788	0.1536	0.6371	0.6486	FAIL
PPO	$(0.00,0.00,0.10)$	0.7352	0.1432	0.7716	0.7816	FAIL
PPO	$(0.10,0.00,0.00)$	0.0000	0.1973	0.1973	0.2070	FAIL
PPO	$(0.10,0.00,0.05)$	0.5797	0.1541	0.6386	0.6501	FAIL
PPO	$(0.10,0.00,0.10)$	0.7371	0.1465	0.7770	0.7869	FAIL

Appendix

Table B.22: Reliability certificates for PPO on SafetyCar under randomized-dynamics perturbations.

Policy	$(r_{\mathrm{dyn}},k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0.00,0.00,0.00)$	0.1059	0.0414	0.1454	0.1540	FAIL
PPO-Lag	$(0.00,0.00,0.05)$	0.7392	0.1447	0.7728	0.7828	FAIL
PPO-Lag	$(0.00,0.00,0.10)$	0.8454	0.1891	0.8702	0.8781	FAIL
PPO-Lag	$(0.10,0.00,0.00)$	0.1022	0.0400	0.1401	0.1486	FAIL
PPO-Lag	$(0.10,0.00,0.05)$	0.7306	0.1487	0.7659	0.7760	FAIL
PPO-Lag	$(0.10,0.00,0.10)$	0.8409	0.1843	0.8664	0.8744	FAIL

Appendix

Table B.23: Reliability certificates for PPO-Lagrangian on SafetyCar under randomized-dynamics perturbations.

Policy	$(r_{\mathrm{dyn}},k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0.00,0.00,0.00)$	0.0000	0.1952	0.1952	0.2049	FAIL
TRPO	$(0.00,0.00,0.05)$	0.4956	0.1867	0.5906	0.6024	FAIL
TRPO	$(0.00,0.00,0.10)$	0.7539	0.2162	0.7995	0.8090	FAIL
TRPO	$(0.10,0.00,0.00)$	0.0000	0.1980	0.1980	0.2077	FAIL
TRPO	$(0.10,0.00,0.05)$	0.4944	0.1812	0.5895	0.6013	FAIL
TRPO	$(0.10,0.00,0.10)$	0.7476	0.2124	0.7943	0.8039	FAIL

Appendix

Table B.24: Reliability certificates for TRPO on SafetyCar under randomized-dynamics perturbations.

Policy	$(r_{\mathrm{dyn}},k_{a},k_{s})$	$\bar{R}$	$R_{\mathrm{H\text{-}LCB}}$	$\bar{C}$	$C_{\mathrm{H\text{-}UCB}}$
TRPO-Lag	$(0.00,0.00,0.00)$	23.58	-285.85	28.92	44.39
TRPO-Lag	$(0.00,0.00,0.05)$	0.53	-308.90	32.90	48.37
TRPO-Lag	$(0.00,0.00,0.10)$	-1.07	-310.50	37.52	53.00
TRPO-Lag	$(0.10,0.00,0.00)$	23.31	-286.12	29.67	45.14
TRPO-Lag	$(0.10,0.00,0.05)$	0.55	-308.88	33.41	48.88
TRPO-Lag	$(0.10,0.00,0.10)$	-1.04	-310.48	36.65	52.12

Appendix

Table B.25: Performance and cost certificates for TRPO-Lagrangian on SafetyCar under randomized-dynamics perturbations.

Policy	$(r_{\mathrm{dyn}},k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0.00,0.00,0.00)$	0.0153	0.0310	0.0459	0.0512	FAIL
TRPO-Lag	$(0.00,0.00,0.05)$	0.5041	0.1126	0.5648	0.5767	FAIL
TRPO-Lag	$(0.00,0.00,0.10)$	0.7345	0.1399	0.7795	0.7894	FAIL
TRPO-Lag	$(0.10,0.00,0.00)$	0.0177	0.0307	0.0477	0.0530	FAIL
TRPO-Lag	$(0.10,0.00,0.05)$	0.5010	0.1120	0.5633	0.5752	FAIL
TRPO-Lag	$(0.10,0.00,0.10)$	0.7241	0.1348	0.7664	0.7765	FAIL

Appendix

Table B.26: Reliability certificates for TRPO-Lagrangian on SafetyCar under randomized-dynamics perturbations.

Policy	$(\Delta,k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0,0.00,0.00)$	0.0000	0.1952	0.1952	0.2049	FAIL
PPO	$(0,0.05,0.00)$	0.0001	0.1997	0.1998	0.2095	FAIL
PPO	$(0,0.10,0.00)$	0.0000	0.1966	0.1966	0.2063	FAIL
PPO	$(1,0.00,0.00)$	0.0000	0.1823	0.1823	0.1917	FAIL
PPO	$(1,0.05,0.00)$	0.0000	0.1752	0.1752	0.1845	FAIL
PPO	$(1,0.10,0.00)$	0.0000	0.1852	0.1852	0.1947	FAIL

Appendix

Table B.27: Reliability certificates for PPO on SafetyCar under delay/latency perturbations.

Policy	$(\Delta,k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0,0.00,0.00)$	0.1010	0.0374	0.1369	0.1453	FAIL
PPO-Lag	$(0,0.05,0.00)$	0.1046	0.0415	0.1445	0.1531	FAIL
PPO-Lag	$(0,0.10,0.00)$	0.1025	0.0397	0.1409	0.1494	FAIL
PPO-Lag	$(1,0.00,0.00)$	0.1428	0.0361	0.1761	0.1854	FAIL
PPO-Lag	$(1,0.05,0.00)$	0.1394	0.0405	0.1775	0.1868	FAIL
PPO-Lag	$(1,0.10,0.00)$	0.1416	0.0406	0.1794	0.1888	FAIL

Appendix

Table B.28: Reliability certificates for PPO-Lagrangian on SafetyCar under delay/latency perturbations.

Policy	$(\Delta,k_{a},k_{s})$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0,0.00,0.00)$	0.1970	0.1970	0.2067	FAIL
TRPO	$(0,0.05,0.00)$	0.1976	0.1976	0.2073	FAIL
TRPO	$(0,0.10,0.00)$	0.1968	0.1968	0.2065	FAIL
TRPO	$(1,0.00,0.00)$	0.1780	0.1780	0.1873	FAIL
TRPO	$(1,0.05,0.00)$	0.1773	0.1773	0.1866	FAIL
TRPO	$(1,0.10,0.00)$	0.1778	0.1778	0.1871	FAIL

Appendix

Table B.29: Reliability certificates for TRPO on SafetyCar under delay/latency perturbations.

Policy	$(\Delta,k_{a},k_{s})$	$\bar{R}$	$R_{\mathrm{H\text{-}LCB}}$	$\bar{C}$	$C_{\mathrm{H\text{-}UCB}}$
TRPO-Lag	$(0,0.00,0.00)$	23.35	-286.08	28.88	44.35
TRPO-Lag	$(0,0.05,0.00)$	23.30	-286.13	29.60	45.07
TRPO-Lag	$(0,0.10,0.00)$	23.53	-285.91	28.99	44.46
TRPO-Lag	$(1,0.00,0.00)$	22.98	-286.46	28.14	43.61
TRPO-Lag	$(1,0.05,0.00)$	23.28	-286.15	29.10	44.57
TRPO-Lag	$(1,0.10,0.00)$	23.43	-286.01	29.11	44.59

Appendix

Table B.30: Performance and cost certificates for TRPO-Lagrangian on SafetyCar under delay/latency perturbations.

Policy	$(\Delta,k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0,0.00,0.00)$	0.0164	0.0300	0.0460	0.0513	FAIL
TRPO-Lag	$(0,0.05,0.00)$	0.0182	0.0315	0.0490	0.0544	FAIL
TRPO-Lag	$(0,0.10,0.00)$	0.0148	0.0311	0.0451	0.0503	FAIL
TRPO-Lag	$(1,0.00,0.00)$	0.0171	0.0246	0.0412	0.0462	PASS
TRPO-Lag	$(1,0.05,0.00)$	0.0171	0.0294	0.0458	0.0510	FAIL
TRPO-Lag	$(1,0.10,0.00)$	0.0165	0.0297	0.0455	0.0507	FAIL

Appendix

Table B.31: Reliability certificates for TRPO-Lagrangian on SafetyCar under delay/latency perturbations.

Policy	$(\ell,k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(\text{nominal},0.00,0.00)$	0.0000	0.1952	0.1952	0.2052	FAIL
PPO	$(\text{nominal},0.05,0.00)$	0.0001	0.1997	0.1998	0.2099	FAIL
PPO	$(\text{nominal},0.10,0.00)$	0.0000	0.1966	0.1966	0.2066	FAIL
PPO	$(\text{nominal},0.00,0.05)$	0.5885	0.1534	0.6481	0.6599	FAIL
PPO	$(\text{nominal},0.00,0.10)$	0.7327	0.1455	0.7738	0.7841	FAIL
PPO	$(\text{mild},0.00,0.00)$	0.0000	0.9094	0.9094	0.9164	FAIL

Appendix

Table B.32: Reliability certificates for PPO on SafetyCar under layout-randomization perturbations.

Policy	$(\ell,k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(\text{nominal},0.00,0.00)$	0.1010	0.0374	0.1369	0.1456	FAIL
PPO-Lag	$(\text{nominal},0.05,0.00)$	0.1046	0.0415	0.1445	0.1534	FAIL
PPO-Lag	$(\text{nominal},0.10,0.00)$	0.1025	0.0397	0.1409	0.1497	FAIL
PPO-Lag	$(\text{nominal},0.00,0.05)$	0.7376	0.1469	0.7723	0.7826	FAIL
PPO-Lag	$(\text{nominal},0.00,0.10)$	0.8406	0.1853	0.8654	0.8737	FAIL
PPO-Lag	$(\text{mild},0.00,0.00)$	0.6155	0.0148	0.6287	0.6406	FAIL

Appendix

Table B.33: Reliability certificates for PPO-Lagrangian on SafetyCar under layout-randomization perturbations.

Policy	$(\ell,k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(\text{nominal},0.00,0.00)$	0.0000	0.1970	0.1970	0.2070	FAIL
TRPO	$(\text{nominal},0.05,0.00)$	0.0000	0.1976	0.1976	0.2076	FAIL
TRPO	$(\text{nominal},0.10,0.00)$	0.0000	0.1968	0.1968	0.2068	FAIL
TRPO	$(\text{nominal},0.00,0.05)$	0.4997	0.1943	0.5936	0.6058	FAIL
TRPO	$(\text{nominal},0.00,0.10)$	0.7577	0.2233	0.8073	0.8170	FAIL
TRPO	$(\text{mild},0.00,0.00)$	0.0000	0.9458	0.9458	0.9513	FAIL

Appendix

Table B.34: Reliability certificates for TRPO on SafetyCar under layout-randomization perturbations.

Policy	$(\ell,k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(\text{nominal},0.00,0.00)$	0.0164	0.0300	0.0460	0.0514	FAIL
TRPO-Lag	$(\text{nominal},0.05,0.00)$	0.0182	0.0315	0.0490	0.0546	FAIL
TRPO-Lag	$(\text{nominal},0.10,0.00)$	0.0148	0.0311	0.0451	0.0505	FAIL
TRPO-Lag	$(\text{nominal},0.00,0.05)$	0.4989	0.1165	0.5620	0.5743	FAIL
TRPO-Lag	$(\text{nominal},0.00,0.10)$	0.7289	0.1378	0.7705	0.7808	FAIL
TRPO-Lag	$(\text{mild},0.00,0.00)$	0.1410	0.0481	0.1891	0.1990	FAIL

Appendix

Table B.35: Reliability certificates for TRPO-Lagrangian on SafetyCar under layout-randomization perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0.00,0.00)$	0.1555	0.9751	0.9913	0.9933	FAIL
PPO	$(0.00,0.05)$	0.8570	0.0000	0.8570	0.8649	FAIL
PPO	$(0.00,0.10)$	0.8603	0.0000	0.8603	0.8681	FAIL
PPO	$(0.05,0.00)$	0.2569	0.9556	0.9875	0.9899	FAIL
PPO	$(0.05,0.05)$	0.8575	0.0000	0.8575	0.8654	FAIL
PPO	$(0.05,0.10)$	0.8716	0.0000	0.8716	0.8792	FAIL

Appendix

Table C.36: Reliability certificates for PPO on SafetyAnt under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0.00,0.00)$	0.0158	0.7972	0.8093	0.8182	FAIL
PPO-Lag	$(0.00,0.05)$	0.6255	0.0000	0.6255	0.6366	FAIL
PPO-Lag	$(0.00,0.10)$	0.6412	0.0000	0.6412	0.6522	FAIL
PPO-Lag	$(0.05,0.00)$	0.2548	0.4744	0.6802	0.6908	FAIL
PPO-Lag	$(0.05,0.05)$	0.6905	0.0000	0.6905	0.7010	FAIL
PPO-Lag	$(0.05,0.10)$	0.6991	0.0000	0.6991	0.7096	FAIL

Appendix

Table C.37: Reliability certificates for PPO-Lagrangian on SafetyAnt under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0.00,0.00)$	0.0146	0.9978	0.9988	0.9994	FAIL
TRPO	$(0.00,0.05)$	0.9356	0.0000	0.9356	0.9411	FAIL
TRPO	$(0.00,0.10)$	0.9408	0.0000	0.9408	0.9461	FAIL
TRPO	$(0.05,0.00)$	0.0600	0.9931	0.9983	0.9991	FAIL
TRPO	$(0.05,0.05)$	0.9351	0.0000	0.9351	0.9406	FAIL
TRPO	$(0.05,0.10)$	0.9417	0.0000	0.9417	0.9469	FAIL

Appendix

Table C.38: Reliability certificates for TRPO on SafetyAnt under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\bar{R}$	$R_{\mathrm{H\text{-}LCB}}$	$\bar{C}$	$C_{\mathrm{H\text{-}UCB}}$
TRPO-Lag	$(0.00,0.00)$	2863.30	2563.30	29.67	44.67
TRPO-Lag	$(0.00,0.05)$	-51.65	-351.65	2.55	17.55
TRPO-Lag	$(0.00,0.10)$	-56.38	-356.37	2.42	17.42
TRPO-Lag	$(0.05,0.00)$	2069.27	1769.27	31.17	46.17
TRPO-Lag	$(0.05,0.05)$	-56.09	-356.08	2.45	17.45
TRPO-Lag	$(0.05,0.10)$	-56.72	-356.72	2.18	17.18

Appendix

Table C.39: Performance and cost certificates for TRPO-Lagrangian on SafetyAnt under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0.00,0.00)$	0.0081	0.0000	0.0081	0.0104	PASS
TRPO-Lag	$(0.00,0.05)$	0.9198	0.0000	0.9198	0.9259	FAIL
TRPO-Lag	$(0.00,0.10)$	0.9200	0.0000	0.9200	0.9261	FAIL
TRPO-Lag	$(0.05,0.00)$	0.1304	0.0008	0.1312	0.1391	FAIL
TRPO-Lag	$(0.05,0.05)$	0.9188	0.0000	0.9188	0.9249	FAIL
TRPO-Lag	$(0.05,0.10)$	0.9274	0.0000	0.9274	0.9332	FAIL

Appendix

Table C.40: Reliability certificates for TRPO-Lagrangian on SafetyAnt under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0.00,0.00)$	0.1555	0.9751	0.9913	0.9933	FAIL
PPO	$(0.00,0.05)$	0.8496	0.0000	0.8496	0.8577	FAIL
PPO	$(0.00,0.10)$	0.8567	0.0000	0.8567	0.8646	FAIL
PPO	$(0.05,0.00)$	0.1831	0.9678	0.9889	0.9912	FAIL
PPO	$(0.05,0.05)$	0.8469	0.0000	0.8469	0.8550	FAIL
PPO	$(0.05,0.10)$	0.8560	0.0000	0.8560	0.8639	FAIL

Appendix

Table C.41: Reliability certificates for PPO on SafetyAnt under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0.00,0.00)$	0.0158	0.7972	0.8093	0.8182	FAIL
PPO-Lag	$(0.00,0.05)$	0.5858	0.0000	0.5858	0.5971	FAIL
PPO-Lag	$(0.00,0.10)$	0.6342	0.0000	0.6342	0.6452	FAIL
PPO-Lag	$(0.05,0.00)$	0.0354	0.7613	0.7847	0.7940	FAIL
PPO-Lag	$(0.05,0.05)$	0.6110	0.0000	0.6110	0.6222	FAIL
PPO-Lag	$(0.05,0.10)$	0.6546	0.0000	0.6546	0.6655	FAIL

Appendix

Table C.42: Reliability certificates for PPO-Lagrangian on SafetyAnt under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0.00,0.00)$	0.0146	0.9978	0.9988	0.9994	FAIL
TRPO	$(0.00,0.05)$	0.9272	0.0000	0.9272	0.9330	FAIL
TRPO	$(0.00,0.10)$	0.9371	0.0000	0.9371	0.9425	FAIL
TRPO	$(0.05,0.00)$	0.0267	0.9952	0.9981	0.9990	FAIL
TRPO	$(0.05,0.05)$	0.9254	0.0000	0.9254	0.9313	FAIL
TRPO	$(0.05,0.10)$	0.9322	0.0000	0.9322	0.9378	FAIL

Appendix

Table C.43: Reliability certificates for TRPO on SafetyAnt under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\bar{R}$	$R_{\mathrm{H\text{-}LCB}}$	$\bar{C}$	$C_{\mathrm{H\text{-}UCB}}$
TRPO-Lag	$(0.00,0.00)$	2863.30	2563.30	29.67	44.67
TRPO-Lag	$(0.00,0.05)$	-45.05	-345.04	3.08	18.08
TRPO-Lag	$(0.00,0.10)$	-52.60	-352.59	2.49	17.49
TRPO-Lag	$(0.05,0.00)$	2559.84	2259.84	43.21	58.21
TRPO-Lag	$(0.05,0.05)$	-46.45	-346.45	3.07	18.07
TRPO-Lag	$(0.05,0.10)$	-54.28	-354.27	2.49	17.49

Appendix

Table C.44: Performance and cost certificates for TRPO-Lagrangian on SafetyAnt under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0.00,0.00)$	0.0081	0.0000	0.0081	0.0104	PASS
TRPO-Lag	$(0.00,0.05)$	0.9124	0.0000	0.9124	0.9188	FAIL
TRPO-Lag	$(0.00,0.10)$	0.9231	0.0000	0.9231	0.9291	FAIL
TRPO-Lag	$(0.05,0.00)$	0.0334	0.0008	0.0341	0.0385	PASS
TRPO-Lag	$(0.05,0.05)$	0.9094	0.0000	0.9094	0.9159	FAIL
TRPO-Lag	$(0.05,0.10)$	0.9210	0.0000	0.9210	0.9271	FAIL

Appendix

Table C.45: Reliability certificates for TRPO-Lagrangian on SafetyAnt under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0.00,0.00)$	0.1466	0.9770	0.9922	0.9941	FAIL
PPO	$(0.00,0.03)$	0.1509	0.9760	0.9929	0.9947	FAIL
PPO	$(0.00,0.06)$	0.1703	0.9717	0.9916	0.9935	FAIL
PPO	$(0.03,0.00)$	0.1537	0.9723	0.9905	0.9926	FAIL
PPO	$(0.03,0.03)$	0.1568	0.9737	0.9910	0.9930	FAIL
PPO	$(0.03,0.06)$	0.1655	0.9709	0.9909	0.9929	FAIL

Appendix

Table C.46: Reliability certificates for PPO on SafetyAnt under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0.00,0.00)$	0.0169	0.7966	0.8088	0.8177	FAIL
PPO-Lag	$(0.00,0.03)$	0.0167	0.8003	0.8108	0.8197	FAIL
PPO-Lag	$(0.00,0.06)$	0.0221	0.7818	0.7962	0.8053	FAIL
PPO-Lag	$(0.03,0.00)$	0.0150	0.7909	0.8010	0.8101	FAIL
PPO-Lag	$(0.03,0.03)$	0.0207	0.7864	0.8008	0.8099	FAIL
PPO-Lag	$(0.03,0.06)$	0.0269	0.7689	0.7855	0.7948	FAIL

Appendix

Table C.47: Reliability certificates for PPO-Lagrangian on SafetyAnt under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0.00,0.00)$	0.0185	0.9979	0.9992	0.9997	FAIL
TRPO	$(0.00,0.03)$	0.0160	0.9970	0.9988	0.9994	FAIL
TRPO	$(0.00,0.06)$	0.0231	0.9969	0.9991	0.9996	FAIL
TRPO	$(0.03,0.00)$	0.0190	0.9977	0.9990	0.9996	FAIL
TRPO	$(0.03,0.03)$	0.0207	0.9973	0.9990	0.9996	FAIL
TRPO	$(0.03,0.06)$	0.0269	0.9959	0.9987	0.9994	FAIL

Appendix

Table C.48: Reliability certificates for TRPO on SafetyAnt under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\bar{R}$	$R_{\mathrm{H\text{-}LCB}}$	$\bar{C}$	$C_{\mathrm{H\text{-}UCB}}$
TRPO-Lag	$(0.00,0.00)$	2880.78	2580.78	29.78	44.78
TRPO-Lag	$(0.00,0.03)$	2789.97	2489.98	37.15	52.15
TRPO-Lag	$(0.00,0.06)$	2539.32	2239.33	48.75	63.75
TRPO-Lag	$(0.03,0.00)$	2811.96	2511.97	36.21	51.21
TRPO-Lag	$(0.03,0.03)$	2724.20	2424.20	42.56	57.56
TRPO-Lag	$(0.03,0.06)$	2446.67	2146.68	45.14	60.14

Appendix

Table C.49: Performance and cost certificates for TRPO-Lagrangian on SafetyAnt under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0.00,0.00)$	0.0076	0.0000	0.0076	0.0098	PASS
TRPO-Lag	$(0.00,0.03)$	0.0092	0.0001	0.0093	0.0117	PASS
TRPO-Lag	$(0.00,0.06)$	0.0265	0.0048	0.0313	0.0355	PASS
TRPO-Lag	$(0.03,0.00)$	0.0097	0.0001	0.0098	0.0123	PASS
TRPO-Lag	$(0.03,0.03)$	0.0163	0.0009	0.0172	0.0204	PASS
TRPO-Lag	$(0.03,0.06)$	0.0339	0.0025	0.0364	0.0409	PASS

Appendix

Table C.50: Reliability certificates for TRPO-Lagrangian on SafetyAnt under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0.00,0.00)$	0.0000	0.1060	0.1060	0.1132	FAIL
PPO	$(0.00,0.05)$	0.0476	0.2561	0.2898	0.3003	FAIL
PPO	$(0.00,0.10)$	0.3093	0.2324	0.4583	0.4698	FAIL
PPO	$(0.05,0.00)$	0.0000	0.1090	0.1090	0.1163	FAIL
PPO	$(0.05,0.05)$	0.0450	0.2612	0.2905	0.3010	FAIL
PPO	$(0.05,0.10)$	0.3154	0.2413	0.4697	0.4812	FAIL

Appendix

Table D.51: Reliability certificates for PPO on SafetyPointGoal under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0.00,0.00)$	0.0580	0.0202	0.0775	0.0838	FAIL
PPO-Lag	$(0.00,0.05)$	0.4898	0.1745	0.5747	0.5860	FAIL
PPO-Lag	$(0.00,0.10)$	0.7497	0.1662	0.7858	0.7951	FAIL
PPO-Lag	$(0.05,0.00)$	0.0574	0.0248	0.0809	0.0873	FAIL
PPO-Lag	$(0.05,0.05)$	0.5030	0.1655	0.5774	0.5887	FAIL
PPO-Lag	$(0.05,0.10)$	0.7542	0.1751	0.7927	0.8019	FAIL

Appendix

Table D.52: Reliability certificates for PPO-Lagrangian on SafetyPointGoal under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0.00,0.00)$	0.0000	0.1228	0.1228	0.1305	FAIL
TRPO	$(0.00,0.05)$	0.0207	0.2485	0.2628	0.2730	FAIL
TRPO	$(0.00,0.10)$	0.2058	0.2258	0.3761	0.3873	FAIL
TRPO	$(0.05,0.00)$	0.0000	0.1213	0.1213	0.1290	FAIL
TRPO	$(0.05,0.05)$	0.0238	0.2515	0.2686	0.2789	FAIL
TRPO	$(0.05,0.10)$	0.2043	0.2282	0.3768	0.3880	FAIL

Appendix

Table D.53: Reliability certificates for TRPO on SafetyPointGoal under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\bar{R}$	$R_{\mathrm{H\text{-}LCB}}$	$\bar{C}$	$C_{\mathrm{H\text{-}UCB}}$
TRPO-Lag	$(0.00,0.00)$	21.72	-278.27	29.28	44.28
TRPO-Lag	$(0.00,0.05)$	2.25	-297.74	31.82	46.82
TRPO-Lag	$(0.00,0.10)$	0.97	-299.02	39.67	54.67
TRPO-Lag	$(0.05,0.00)$	21.72	-278.27	29.95	44.95
TRPO-Lag	$(0.05,0.05)$	2.24	-297.75	32.65	47.65
TRPO-Lag	$(0.05,0.10)$	0.96	-299.03	40.84	55.84

Appendix

Table D.54: Performance and cost certificates for TRPO-Lagrangian on SafetyPointGoal under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0.00,0.00)$	0.0080	0.0209	0.0287	0.0328	PASS
TRPO-Lag	$(0.00,0.05)$	0.3095	0.1120	0.3889	0.4001	FAIL
TRPO-Lag	$(0.00,0.10)$	0.6201	0.1516	0.6739	0.6846	FAIL
TRPO-Lag	$(0.05,0.00)$	0.0090	0.0222	0.0310	0.0352	PASS
TRPO-Lag	$(0.05,0.05)$	0.3156	0.1197	0.3965	0.4078	FAIL
TRPO-Lag	$(0.05,0.10)$	0.6196	0.1558	0.6728	0.6835	FAIL

Appendix

Table D.55: Reliability certificates for TRPO-Lagrangian on SafetyPointGoal under Gaussian perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0.00,0.00)$	0.0000	0.1060	0.1060	0.1132	FAIL
PPO	$(0.00,0.05)$	0.0126	0.2468	0.2547	0.2648	FAIL
PPO	$(0.00,0.10)$	0.0846	0.2598	0.3177	0.3285	FAIL
PPO	$(0.05,0.00)$	0.0001	0.1086	0.1086	0.1159	FAIL
PPO	$(0.05,0.05)$	0.0140	0.2401	0.2496	0.2596	FAIL
PPO	$(0.05,0.10)$	0.0886	0.2560	0.3171	0.3279	FAIL

Appendix

Table D.56: Reliability certificates for PPO on SafetyPointGoal under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0.00,0.00)$	0.0580	0.0202	0.0775	0.0838	FAIL
PPO-Lag	$(0.00,0.05)$	0.2994	0.1553	0.4052	0.4165	FAIL
PPO-Lag	$(0.00,0.10)$	0.5791	0.1680	0.6401	0.6511	FAIL
PPO-Lag	$(0.05,0.00)$	0.0611	0.0226	0.0821	0.0886	FAIL
PPO-Lag	$(0.05,0.05)$	0.3019	0.1648	0.4129	0.4242	FAIL
PPO-Lag	$(0.05,0.10)$	0.5763	0.1770	0.6403	0.6513	FAIL

Appendix

Table D.57: Reliability certificates for PPO-Lagrangian on SafetyPointGoal under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0.00,0.00)$	0.0000	0.1228	0.1228	0.1305	FAIL
TRPO	$(0.00,0.05)$	0.0055	0.2372	0.2408	0.2507	FAIL
TRPO	$(0.00,0.10)$	0.0442	0.2435	0.2769	0.2873	FAIL
TRPO	$(0.05,0.00)$	0.0000	0.1202	0.1202	0.1278	FAIL
TRPO	$(0.05,0.05)$	0.0052	0.2297	0.2335	0.2433	FAIL
TRPO	$(0.05,0.10)$	0.0429	0.2451	0.2744	0.2847	FAIL

Appendix

Table D.58: Reliability certificates for TRPO on SafetyPointGoal under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\bar{R}$	$R_{\mathrm{H\text{-}LCB}}$	$\bar{C}$	$C_{\mathrm{H\text{-}UCB}}$
TRPO-Lag	$(0.00,0.00)$	21.72	-278.27	29.28	44.28
TRPO-Lag	$(0.00,0.05)$	3.57	-296.42	26.13	41.13
TRPO-Lag	$(0.00,0.10)$	1.75	-298.24	35.97	50.97
TRPO-Lag	$(0.05,0.00)$	21.75	-278.25	30.10	45.10
TRPO-Lag	$(0.05,0.05)$	3.58	-296.41	25.56	40.56
TRPO-Lag	$(0.05,0.10)$	1.74	-298.26	34.79	49.79

Appendix

Table D.59: Performance and cost certificates for TRPO-Lagrangian on SafetyPointGoal under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0.00,0.00)$	0.0080	0.0209	0.0287	0.0328	PASS
TRPO-Lag	$(0.00,0.05)$	0.1818	0.0824	0.2523	0.2624	FAIL
TRPO-Lag	$(0.00,0.10)$	0.4071	0.1307	0.4830	0.4945	FAIL
TRPO-Lag	$(0.05,0.00)$	0.0084	0.0219	0.0300	0.0341	PASS
TRPO-Lag	$(0.05,0.05)$	0.1791	0.0811	0.2486	0.2586	FAIL
TRPO-Lag	$(0.05,0.10)$	0.4120	0.1287	0.4871	0.4986	FAIL

Appendix

Table D.60: Reliability certificates for TRPO-Lagrangian on SafetyPointGoal under Uniform perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO	$(0.00,0.00)$	0.1060	0.1060	0.1132	FAIL
PPO	$(0.00,0.03)$	0.1108	0.1108	0.1182	FAIL
PPO	$(0.00,0.06)$	0.1058	0.1058	0.1130	FAIL
PPO	$(0.03,0.00)$	0.1056	0.1056	0.1128	FAIL
PPO	$(0.03,0.03)$	0.1085	0.1085	0.1158	FAIL
PPO	$(0.03,0.06)$	0.1058	0.1058	0.1130	FAIL

Appendix

Table D.61: Reliability certificates for PPO on SafetyPointGoal under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
PPO-Lag	$(0.00,0.00)$	0.0580	0.0202	0.0775	0.0838	FAIL
PPO-Lag	$(0.00,0.03)$	0.0580	0.0244	0.0815	0.0880	FAIL
PPO-Lag	$(0.00,0.06)$	0.0569	0.0216	0.0770	0.0833	FAIL
PPO-Lag	$(0.03,0.00)$	0.0600	0.0201	0.0792	0.0856	FAIL
PPO-Lag	$(0.03,0.03)$	0.0579	0.0223	0.0788	0.0852	FAIL
PPO-Lag	$(0.03,0.06)$	0.0575	0.0199	0.0765	0.0828	FAIL

Appendix

Table D.62: Reliability certificates for PPO-Lagrangian on SafetyPointGoal under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO	$(0.00,0.00)$	0.0000	0.1228	0.1228	0.1305	FAIL
TRPO	$(0.00,0.03)$	0.0000	0.1190	0.1190	0.1266	FAIL
TRPO	$(0.00,0.06)$	0.0000	0.1201	0.1201	0.1277	FAIL
TRPO	$(0.03,0.00)$	0.0001	0.1219	0.1219	0.1296	FAIL
TRPO	$(0.03,0.03)$	0.0000	0.1196	0.1196	0.1272	FAIL
TRPO	$(0.03,0.06)$	0.0000	0.1164	0.1164	0.1239	FAIL

Appendix

Table D.63: Reliability certificates for TRPO on SafetyPointGoal under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\bar{R}$	$R_{\mathrm{H\text{-}LCB}}$	$\bar{C}$	$C_{\mathrm{H\text{-}UCB}}$
TRPO-Lag	$(0.00,0.00)$	21.72	-278.27	29.28	44.28
TRPO-Lag	$(0.00,0.03)$	21.77	-278.23	29.42	44.42
TRPO-Lag	$(0.00,0.06)$	21.85	-278.14	29.74	44.74
TRPO-Lag	$(0.03,0.00)$	21.86	-278.13	29.35	44.35
TRPO-Lag	$(0.03,0.03)$	21.76	-278.23	30.06	45.06
TRPO-Lag	$(0.03,0.06)$	21.77	-278.22	29.23	44.23

Appendix

Table D.64: Performance and cost certificates for TRPO-Lagrangian on SafetyPointGoal under bounded

L_{\infty}

perturbations.

Policy	$(k_{a},k_{s})$	$\hat{p}_{\mathrm{task}}$	$\hat{p}_{\mathrm{safety}}$	$\hat{p}_{F}$	B-CP UB	Decision
TRPO-Lag	$(0.00,0.00)$	0.0080	0.0209	0.0287	0.0328	PASS
TRPO-Lag	$(0.00,0.03)$	0.0086	0.0187	0.0271	0.0310	PASS
TRPO-Lag	$(0.00,0.06)$	0.0078	0.0211	0.0286	0.0326	PASS
TRPO-Lag	$(0.03,0.00)$	0.0084	0.0217	0.0297	0.0338	PASS
TRPO-Lag	$(0.03,0.03)$	0.0085	0.0216	0.0297	0.0338	PASS
TRPO-Lag	$(0.03,0.06)$	0.0083	0.0229	0.0310	0.0352	PASS

Appendix

Table D.65: Reliability certificates for TRPO-Lagrangian on SafetyPointGoal under bounded

L_{\infty}

perturbations.

Environment	Perturbation	$d=25$	$d=50$	$d=100$	$d=250$	$d=500$	$d=750$	$d=1000$	$d=1500$
SafetyAnt	Gaussian	0.8361	0.7085	0.6766	0.6765	0.6765	0.6765	0.6765	0.6765
	Uniform	0.8547	0.7100	0.6404	0.6402	0.6402	0.6402	0.6402	0.6402
	$L_{\infty}$	0.7952	0.3762	0.0323	0.0309	0.0309	0.0309	0.0309	0.0309
SafetyCar	Gaussian	0.7337	0.6359	0.5503	0.5173	0.5150	0.5149	0.5149	0.5149
	Uniform	0.6740	0.5647	0.4703	0.4295	0.4252	0.4250	0.4250	0.4250
	$L_{\infty}$	0.4639	0.2196	0.0515	0.0229	0.0200	0.0194	0.0194	0.0194

Appendix

Table E.66: TRPO-Lagrangian reliability certificates under varying safety-budget thresholds

d

. Each entry reports the mean Bonferroni-corrected Clopper–Pearson upper bound

B_{\mathrm{CP}}^{\mathrm{UB}}

across the nine

(k_{a},k_{s})

perturbation configurations.

Environment	Perturbation	$B_{\mathrm{CP}}^{\mathrm{UB}}$	$p^{\star}=0.010$	$p^{\star}=0.025$	$p^{\star}=0.050$	$p^{\star}=0.100$
SafetyAnt	Gaussian	0.677	FAIL	FAIL	FAIL	FAIL
	Uniform	0.640	FAIL	FAIL	FAIL	FAIL
	$L_{\infty}$	0.032	FAIL	FAIL	PASS	PASS
SafetyCar	Gaussian	0.550	FAIL	FAIL	FAIL	FAIL
	Uniform	0.470	FAIL	FAIL	FAIL	FAIL
	$L_{\infty}$	0.052	FAIL	FAIL	FAIL	PASS

Appendix

Table E.67: TRPO-Lagrangian reliability certification under varying reliability thresholds

p^{\star}

. The Bonferroni-corrected Clopper–Pearson upper bound

B_{\mathrm{CP}}^{\mathrm{UB}}

is reported for each environment and perturbation type. A certificate passes when

B_{\mathrm{CP}}^{\mathrm{UB}}\leq p^{\star}

Observed Failures $k$	Minimum $N$	CP Upper Bound	Empirical $\hat{p}=k/N$
0	45	0.04988	0.00000
1	77	0.04958	0.01299
2	105	0.04989	0.01905
3	132	0.04991	0.02273
5	184	0.04982	0.02717
10	306	0.04990	0.03268

Appendix

Table 68: Minimum sample size

N

required to certify

p^{*}=0.05

90\%

confidence using Clopper–Pearson bounds (SafetyAntVelocity).

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,739 papers

Sep 14, 2026

Evaluation Metrics for Safe Reinforcement Learning

Safe reinforcement learning (RL) is commonly formalized as a Constrained Markov Decision Process (CMDP), in which an agent maximizes expected reward while keeping its expected cumulative cost below a specified safety bound. Existing safe RL benchmarks predominantly report whether an algorithm is safe on average, following this expectation-based guarantee. We argue that this convention is insufficient to reliably characterize an algorithm's true safety: it fails to capture how often and how severely the safety bound is violated, whether this holds consistently across tasks and safety bounds, and whether training-time behavior is representative of behavior of the final converged policy. Therefore, we introduce (i) evaluation metrics for safe RL that address each of these concerns and in addition allow for aggregation across tasks and safety bounds. We furthermore define (ii) a safety tier system to systematically categorize and compare algorithms in terms of safety and reliability at both training and for a final policy. Using this framework, we provide (iii) an empirical safety evaluation across multiple safety navigation tasks. Our results show that aggregate metrics, distributional reporting, and task- and safety bound-specific results each reveal information the other metrics cannot. We therefore recommend reporting all three jointly, rather than compressing this information into a single value, as is common practice. We provide SafeRLEval, an open-source evaluation suite to support the reliable characterization of safety in future safe RL research.

Lindsay Spoor, Aske Plaat, Thomas Moerland

Leiden Institute of Advanced Computer Science, Leiden University, Leiden, the Netherlands

RL Benchmarks Constrained RL Safe RL

Ask PDF

Apr 16, 2026

RL-STPA: Adapting System-Theoretic Hazard Analysis for Safety-Critical Reinforcement Learning

As reinforcement learning (RL) deployments expand into safety-critical domains, existing evaluation methods fail to systematically identify hazards arising from the black-box nature of neural network enabled policies and distributional shift between training and deployment. This paper introduces Reinforcement Learning System-Theoretic Process Analysis (RL-STPA), a framework that adapts conventional STPA's systematic hazard analysis to address RL's unique challenges through three key contributions: hierarchical subtask decomposition using both temporal phase analysis and domain expertise to capture emergent behaviors, coverage-guided perturbation testing that explores the sensitivity of state-action spaces, and iterative checkpoints that feed identified hazards back into training through reward shaping and curriculum design. We demonstrate RL-STPA in the safety-critical test case of autonomous drone navigation and landing, revealing potential loss scenarios that can be missed by standard RL evaluations. The proposed framework provides practitioners with a toolkit for systematic hazard analysis, quantitative metrics for safety coverage assessment, and actionable guidelines for establishing operational safety bounds. While RL-STPA cannot provide formal guarantees for arbitrary neural policies, it offers a practical methodology for systematically evaluating and improving RL safety and robustness in safety-critical applications where exhaustive verification methods remain intractable.

Steven A. Senczyszyn, Timothy C. Havens, Nathaniel Rice +3

Michigan Technological University · ARiA · DEVCOM

Safe RL Robust RL

Ask PDF

Oct 16, 2025

CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions

Reinforcement learning (RL), while powerful and expressive, can often prioritize performance at the expense of safety. Yet safety violations can lead to catastrophic outcomes in real-world deployments. Control Barrier Functions (CBFs) offer a principled method to enforce dynamic safety -- traditionally deployed online via safety filters. While the result is safe behavior, the fact that the RL policy does not have knowledge of the CBF can lead to conservative behaviors. This paper proposes CBF-RL, a framework for generating safe behaviors with RL by enforcing CBFs in training. CBF-RL has two key attributes: (1) minimally modifying a nominal RL policy to encode safety constraints via a CBF term, (2) and safety filtering of the policy rollouts in training. Theoretically, we prove that continuous-time safety filters can be deployed via closed-form expressions on discrete-time roll-outs. Practically, we demonstrate that CBF-RL internalizes the safety constraints in the learned policy -- both enforcing safer actions and biasing towards safer rewards -- enabling safe deployment without the need for an online safety filter. We validate our framework through ablation studies on navigation tasks and on the Unitree G1 humanoid robot, where CBF-RL enables safer exploration, faster convergence, and robust performance under uncertainty, enabling the humanoid robot to avoid obstacles and climb stairs safely in real-world settings without a runtime safety filter.

Lizhi Yang, Blake Werner, Massimiliano de Sa +1

Caltech MCE

RL for Robotics Control Barrier Functions Constrained RL Safety Filtering Safe RL

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Evaluation Metrics for Safe Reinforcement Learning

RL-STPA: Adapting System-Theoretic Hazard Analysis for Safety-Critical Reinforcement Learning

CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions