Integrating LTL Constraints into PPO for Safe Reinforcement Learning

Maifang Zhang; Hang Yu; Qian Zuo; Cheng Wang; Vaishak Belle; Fengxiang He

cs.LGMar 1, 2026

Integrating LTL Constraints into PPO for Safe Reinforcement Learning

Authors: Maifang Zhang, Hang Yu, Qian Zuo, Cheng Wang, Vaishak Belle, Fengxiang He

Organizations: School of Informatics, University of Edinburgh, Edinburgh EH8 9AB, Scotland. · School of Computer Science, Faculty of Engineering, University of Sydney, Darlington NSW 2008, Australia. · School of Engineering and Physical Sciences, Heriot-Watt University, Edinburgh EH14 4AS, Scotland.

Abstract

This paper proposes Proximal Policy Optimization with Linear Temporal Logic Constraints (PPO-LTL), a framework that integrates safety constraints written in LTL into PPO for safe reinforcement learning. LTL constraints offer rigorous representations of complex safety requirements, such as regulations that broadly exist in robotics, enabling systematic monitoring of safety requirements. Violations against LTL constraints are monitored by limit-deterministic Büchi automata, and then translated by a logic-to-cost mechanism into penalty signals. The signals are further employed for guiding the policy optimization via the Lagrangian scheme. Extensive experiments on the Zones and CARLA environments show that our PPO-LTL can consistently reduce safety violations, while maintaining competitive performance, against the state-of-the-art methods. The code is at https://github.com/EVIEHub/PPO-LTL.

Figures & tables

Fig. 1 : PPO-LTL: environment states are labeled with atomic propositions, monitored by LTL checkers to generate constraint costs, which are integrated with task rewards for policy optimization.

Method	Reward	Hit Wall Rate	$\lambda$
PPO	17.95 $\pm$ 0.84	3.7 $\pm$ 2.1%	-
PPO-Mask	10.14 $\pm$ 2.82	6.0 $\pm$ 2.0%	-
PPO-Shielding	15.92 $\pm$ 0.55	12.0 $\pm$ 2.0%	-
PPO-Lagrangian	23.23 $\pm$ 1.51	6.0 $\pm$ 2.0%	0.00
PPO-LTL-A	17.86 $\pm$ 0.73	4.3 $\pm$ 3.5%	0.0048
PPO-LTL-B	18.61 $\pm$ 0.67	4.7 $\pm$ 3.1%	0.0018

TABLE I : Results in ZonesEnv.

Methods	Collision Rate $\downarrow$	Routes $\uparrow$	Distance	Speed	Length	Col. Num	Cent. Dev	Cost	$\lambda$
PPO	0.262 $\pm$ 0.115	0.013 $\pm$ 0.008	6.58	0.45	4030	8.0	0.77	-	-
TIRL-PPO	0.173 $\pm$ 0.129	0.083 $\pm$ 0.111	6.37	0.07	9460	7.3	0.87	-	-
TIRL-SAC	0.336 $\pm$ 0.063	0.027 $\pm$ 0.011	6.92	0.92	926	57.3	0.12	-	-
PPO-Mask	0.408 $\pm$ 0.058	0.010 $\pm$ 0.012	2.38	1.06	1338	38.0	0.37	-	-
PPO-Shielding	0.267 $\pm$ 0.056	0.072 $\pm$ 0.036	18.87	10.63	93	164.3	0.52	-	-
PPO-Lagrangian	0.233 $\pm$ 0.089	0.077 $\pm$ 0.051	7.78	0.96	6616	103.0	0.60	0.00	0.01

TABLE II : Results in CARLA. Arrows (

\uparrow,\downarrow

) indicate the direction where values are strictly better.

Configuration	Collision Rate $\downarrow$	Routes $\uparrow$	Distance	Speed	Length	Col. Num	Cent. Dev	Cost	$\lambda$
Ablation of LTL Components
No collision	0.159 $\pm$ 0.027	0.009 $\pm$ 0.003	2.71	0.68	4123	11.0	0.64	0.07	0.00
No off-track	0.275 $\pm$ 0.129	0.032 $\pm$ 0.044	3.29	0.06	10501	5.7	0.62	0.07	0.00
No lane invasion	0.207 $\pm$ 0.046	0.024 $\pm$ 0.009	6.17	1.98	14323	21.7	0.57	0.06	0.01
Sensitivity of Cost Limit (cl) and Dual Learning Rate ( $\alpha_{\lambda}$ )
cl = 0.001	0.229 $\pm$ 0.153	0.033 $\pm$ 0.035	5.19	1.10	4749	21.0	0.78	0.06	0.01

TABLE III : Ablation studies and sensitivity analyses in CARLA.

Cost Limit	$\alpha_{\lambda}$	Reward $\uparrow$	Hit Wall Rate $\downarrow$	$\lambda$
0.03	0.008	18.04 $\pm$ 1.30	6.7 $\pm$ 1.2%	0.0072
0.05	0.010	17.86 $\pm$ 0.73	4.3 $\pm$ 3.5%	0.0048
0.07	0.015	18.61 $\pm$ 0.67	4.7 $\pm$ 3.1%	0.0018
0.10	0.020	18.35 $\pm$ 1.42	5.7 $\pm$ 2.1%	0.0002

TABLE IV : Sensitivity analysis of cost limit and Lagrangian learning rate (

\alpha_{\lambda}

) in ZonesEnv for PPO-LTL.

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 215 papers

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
Dec 29, 2025Dominik Wagner, Ankit Kanwar, Luke OngSafety-Critical Control Policy Optimization
NTU Singapore · Sony Corporation
CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
Jun 12, 2026Ayoub Belouadah, Sylvain Kubler, Yves Le TraonConstrained RL Safe RL
SnT, University of Luxembourg
PPO-EAL: Exact Augmented Lagrangian Proximal Policy Optimization for Safe Robotic Control
Jun 26, 2026Jiatao Ding, Songqun Gao, Andrea Del Prete +1Robotic Control Constrained RL
Department of Industrial Engineering, University of Trento, Via Sommarive 9, 38123, Trento, Italy

Dec 29, 2025J/K move · Enter open · S save

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

Dominik Wagner, Ankit Kanwar, Luke Ong

NTU Singapore · Sony Corporation

Safety-Critical Control Policy Optimization Constrained RL Trust Region Policy Optimization Safe RL Policy Learning

Open paper AskPDF arXiv

In safety-critical domains, reinforcement learning (RL) systems must satisfy strict, zero-cost safety constraints while achieving meaningful task performance. Existing model-free methods can struggle to achieve high safety without substantially compromising task performance. We introduce Safety-Biased Trust Region Policy Optimisation (SB-TRPO), a principled approach to RL with zero-cost constraints, which requires only a fixed fraction of the maximal cost reduction achievable within the trust region, thus retaining flexibility for reward optimisation. We show that the idealised update nevertheless converges to zero cost and maximal reward amongst zero-cost policies in finite MDPs. A practical gradient-based approximation provides local improvements in both safety and reward under suitable gradient alignment. Experiments on Safety Gymnasium demonstrate high safety alongside strong task performance.

Abstract

Safe RL

Explore similar work

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

PPO-EAL: Exact Augmented Lagrangian Proximal Policy Optimization for Safe Robotic Control