Safeguarding Multi-Agent Reinforcement Learning for Connected and Automated Vehicles with Recursively Feasible Control Barrier Functions

Jianye Xu; Yousef Sayed; Bassam Alrifaee

cs.RONov 3, 2026

Safeguarding Multi-Agent Reinforcement Learning for Connected and Automated Vehicles with Recursively Feasible Control Barrier Functions

Authors: Jianye Xu, Yousef Sayed, Bassam Alrifaee

Organizations: Department of Computer Science, RWTH Aachen University, Germany · Department of Aerospace Engineering, University of the Bundeswehr Munich, Germany

Abstract

Multi-Agent Reinforcement Learning (MARL) shows promise in coordinating connected and automated vehicles but offers no safety guarantee. Although Control Barrier Functions (CBFs) can add this guarantee, their optimization problem can become infeasible in practice due to control bounds. Moreover, they often overapproximate vehicle and road geometry, which makes them conservative. This paper proposes the Backup-Horizon CBF (BH-CBF), which evaluates safety along a horizon until all vehicles stop. BH-CBF is recursively feasible under control bounds, and it considers the exact vehicle and road geometry. Based on BH-CBF, we derive a fully decentralized safety filter that only needs a single broadcast of vehicle states and routes per step and stays real-time in dense traffic. We use BH-CBF to safeguard a MARL policy over its life cycle: as a reward during pre-training, as a run-time filter at deployment, and as a filter in the training loop that keeps fine-tuning on unseen maps collision-free. In simulations on real-world maps of a roundabout and an intersection, BH-CBF avoids all collisions of unsafe MARL policies and keeps its safety guarantee at every step. Existing filters either collide, lose the guarantee, or fall back to full braking far more often. In recorded roundabout traffic with 1264 human-driven vehicles over 115 km, our safeguarded policies shorten the median travel time by 14% compared with the human drivers, without any collision.

Figures & tables

Panel of Figure 1 — Fig. 1 : An unsafe MARL policy safeguarded by our BH-CBF. Light trails: predictions of braking to a standstill along the route. Red: standstill distance of a vehicle pair.

Panel of Figure 2 — Fig. 2 : Kinematic bicycle model. $\bm{p}=(p_{x},p_{y})$ : position; $\psi$ : heading; $v$ : speed; $\beta$ : slip angle; $\delta$ : steering angle; $\ell_{\mathrm{r}}$ : rear-axle distance; $\ell_{\mathrm{wb}}$ : wheelbase.

Panel of Figure 3 — Fig. 3 : Margin geometry. (a) Facet margin $h_{l}$ , whose minimum over all facets is the lane margin. $\bm{p}$ : position; $\bm{e}_{\mathrm{lon}},\bm{e}_{\mathrm{lat}}$ : body axes; $\bm{q}_{c}$ : corner; $\ell_{\mathrm{lon}},\ell_{\mathrm{lat}}$ : half-length, half-width; $\mathcal{F}_{l}=[\bm{b}_{l},\bm{b}_{l+1}]$ : facet of the left boundary ( $s=\mathrm{L}$ ); $\bm{n}_{l}$ : inward normal; red: clearance $\varepsilon_{\mathrm{lane}}$ . (b) Pair margin $h^{ij}$ . Red: clearance $\varepsilon_{\mathrm{veh}}$ ; $\bm{n}^{ij}$ : separating normal; dashed: separating line.

Panel of Figure 4 — Fig. 4 : Prediction of vehicle $i$ at step $k$ over $t$ (schematic). Blue: candidate prediction $\bm{\xi}^{i}_{k+t|k}(\bm{u}^{i})$ for a control $\bm{u}^{i}$ that accelerates and steers away from the route; gray: backup prediction $\bar{\bm{\xi}}^{i}_{k+t|k}$ , which tracks the route centerline (dashed). (a) Poses; $\bm{\xi}^{i}_{k+1|k}$ is the same for both predictions. (b) Speed. (c) Margin $h_{e}$ of the right lane constraint; ( 7 ) keeps the blue curve above $(1-\lambda)B_{e}(\bm{x}_{k})$ from $t=2$ on.

Panel of Figure 5 — Fig. 5 : Candidate predictions of vehicles $i$ (blue) and $j$ (green) and their backup predictions (gray) at step $k$ over $t$ (schematic). (a) Poses at a crossing; the backup predictions come closest at $t=3$ . (b) Pair margin $h^{ij}$ of the backup predictions (gray) and of the candidate predictions (black). The budget between the gray curve and $(1-\lambda)B^{ij}(\bm{x}_{k})$ is split into the shares of $i$ (blue fill) and $j$ (green fill).

Panel of Figure 6 — Fig. 6 : Decentralized corner inequalities ( 10 ) at step $k$ for one prediction step $t$ (schematic), drawn with the separating normal $\bm{n}^{ij}_{k+t|k}$ . Candidate predictions of vehicles $i$ (blue) and $j$ (green) and their backup predictions (gray). Blue and green fills: shares of $i$ and $j$ , as in Fig. 5 . Gray dot: closest corner of $\bar{\mathcal{R}}^{i}_{k+t|k}$ along $\bm{n}^{ij}_{k+t|k}$ ; blue dot: a corner of $\mathcal{R}^{i}_{k+t|k}$ that uses the full share $\rho^{ij}\eta^{ij}_{k+t|k}$ of the budget.

Panel of Figure 7 — Fig. 7 : BH-CBF pipeline. Solid: run-time safety filter. Dashed: training signals.

Panel of Figure 8 — Fig. 8 : The evaluation stages along a MARL policy’s life cycle.

Fig. 9 : Stage 1: training rollouts over pre-training by safety reward, smoothed over ten iterations, mean and standard error over five training seeds.

	Own map		Unseen map
	Collisions $\downarrow$	Throughput $\uparrow$	Collisions $\downarrow$	Throughput $\uparrow$
Reward method	[1/km]	[veh/s]	[1/km]	[veh/s]
TTC	0.86	1.08	1.27	0.92
One-Step CBF	0.71	1.03	1.61	0.80
BH-CBF (ours)	0.05	1.05	0.59	0.91

TABLE I : Stage 1: the 30 pre-trained policies (ten per safety reward, iteration 500) deployed without a filter.

Filter	Geometry	Condition	Split
BH-CBF (ours)	exact	backup horizon	corner budgets
A-Geometry	capsules	backup horizon	corner budgets
A-Condition	exact	one step	corner budgets
A-Split	exact	backup horizon	half split
A-Centralized	exact	backup horizon	centralized
B-SAT [ 19 ]	smoothed SAT	one step	half split

TABLE II : Compared safety filters; gray: as in BH-CBF.

	Collisions $\downarrow$	Runs $\downarrow$	Fallbacks $\downarrow$	Throughput $\uparrow$	Time $\downarrow$
Filter	[1/km]	$\notin\mathcal{C}$	[1/1000]	[veh/s]	[ms]
Unfiltered	1.39	60/60	–	0.87	–
B-Frenet	1.22	60/60	–	0.75	4.7
A-Condition	0.61	60/60	–	0.91	8.0
B-Disk	0.47	60/60	–	1.04	7.1
B-SAT	0.38	60/60	–	0.91	11.3

TABLE III : Stage 2: the ten unsafe source policies (TTC reward, iteration 100) deployed with fully decentralized filters, except A-Centralized; 60 runs (own and unseen map, three seeds each) per row. Runs

\notin\mathcal{C}

: runs that leave the certified set; time: computation time per step.

Panel of Figure 13 — Fig. 10 : Unsafe policies on the roundabout (a)–(d) and the intersection (e)–(h), without (top) and with (bottom) our BH-CBF filter. Blue: yielding vehicle 1; green: vehicle 2; red: collisions; gray: others (last footprint only); footprint interval: $1.0\text{\,}\mathrm{s}$ .

Panel of Figure 14 — Fig. 11 : Collisions of the TTC-reward policies over pre-training and fine-tuning, mean over runs (20 fine-tuning runs per setting, own and unseen map taken together) with standard deviation in ( 11(a) ) and standard error in ( 11(b) ).

	Travel time $\downarrow$	Stops $\downarrow$	Gap	RMS jerk $\downarrow$
Controlled by	median [s]	[%]	[m]	median [ $\mathrm{m}\text{\,}{\mathrm{s}}^{-3}$ ]
Humans	12.0	13.8	1.40	0.94
BH-CBF (ours)	10.3	2.0	1.22	3.66

TABLE IV : Stage 4: Human-comparison results over 115 km of 1264 recorded human-driven vehicles. Stops: stop before the ring entry; gap: 5th percentile of the minimum gap.

Panel of Figure 16 — Fig. 12 : Stage 4: two roundabout scenarios over $7\text{\,}\mathrm{s}$ . Filled: our BH-CBF-filtered policies; unfilled: recorded humans (time stamps with dashed lines). Other vehicles not shown.

Panel of Figure 17 — Fig. 13 : Stage 4: entering the ring. (b) Median and $2575\text{\,}\mathrm{\%}$ band. (c) Probability of entering the ring before the circulating vehicle (dots: share per time-to-arrival bin with $95\text{\,}\mathrm{\%}$ confidence intervals; thick curves: logistic fits with $95\text{\,}\mathrm{\%}$ confidence bands; thin blue curves: logistic fits of the ten single policies).

Panel of Figure 18 — Fig. 14 : Computation time of the BH-CBF filter per step on the roundabout, mean over 500 steps per bar; sampling period $\Delta t=$ 100\text{,}\mathrm{ms}$$ .

Control Barrier Functions

Control barrier functions encode safe states as constraints that keep a controlled system away from hazards.

See all 166 papers

May 16, 2026

Beyond Safety Filtering: Control Barrier Function-Informed Reinforcement Learning for Connected and Automated Vehicles

Reinforcement Learning (RL) uses rewards to guide learning, yet reward design is typically hand-crafted using heuristics that can be difficult to tune. We propose a Control Barrier Function (CBF)-informed reward design for Multi-Agent RL (MARL) that converts CBF constraint values under joint MARL actions into a reward signal that explicitly guides safe learning. We compare against two heuristic reward baselines in a four-way multi-lane intersection with connected and automated vehicles. Results show that our method achieves the highest task performance and is less sensitive to reward hyperparameters, yielding consistently strong performance across the tested hyperparameter range. Code for reproducing the experimental results and a video demonstration are available at https://github.com/bassamlab/SigmaRL.

Jianye Xu, Bassam Alrifaee

Department of Computer Science, RWTH Aachen University, Germany · Department of Aerospace Engineering, University of the Bundeswehr Munich, Germany

Reward Shaping Control Barrier Functions RL for Autonomous Driving Safe RL Multi-Agent Reinforcement Learning

Ask PDF

Oct 16, 2025

CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions

Reinforcement learning (RL), while powerful and expressive, can often prioritize performance at the expense of safety. Yet safety violations can lead to catastrophic outcomes in real-world deployments. Control Barrier Functions (CBFs) offer a principled method to enforce dynamic safety -- traditionally deployed online via safety filters. While the result is safe behavior, the fact that the RL policy does not have knowledge of the CBF can lead to conservative behaviors. This paper proposes CBF-RL, a framework for generating safe behaviors with RL by enforcing CBFs in training. CBF-RL has two key attributes: (1) minimally modifying a nominal RL policy to encode safety constraints via a CBF term, (2) and safety filtering of the policy rollouts in training. Theoretically, we prove that continuous-time safety filters can be deployed via closed-form expressions on discrete-time roll-outs. Practically, we demonstrate that CBF-RL internalizes the safety constraints in the learned policy -- both enforcing safer actions and biasing towards safer rewards -- enabling safe deployment without the need for an online safety filter. We validate our framework through ablation studies on navigation tasks and on the Unitree G1 humanoid robot, where CBF-RL enables safer exploration, faster convergence, and robust performance under uncertainty, enabling the humanoid robot to avoid obstacles and climb stairs safely in real-world settings without a runtime safety filter.

Lizhi Yang, Blake Werner, Massimiliano de Sa +1

Caltech MCE

RL for Robotics Control Barrier Functions Constrained RL Safety Filtering Safe RL

Ask PDF

Sep 8, 2026

Graph-Based Safe Reinforcement Learning for Multi-Agent Systems with Time-Varying Topology

This paper presents a graph-based safe multi-agent reinforcement learning (MARL) framework for cooperative navigation with time-varying topology. To address the critical challenge of ensuring safety in environments with sensing constraints, a safety-decoupled mechanism is introduced through a Control Barrier-Like Function (CBLF) action screening layer. This mechanism bridges the gap between discrete LiDAR perception and continuous safety constraints, ensuring that physical safety constraints are strictly satisfied regardless of the learning progress. Building upon this safety foundation, a unified structural architecture is proposed, integrating a attention-based actor and a Graph Attention Network (GAT) centralized critic. The actor utilizes a value vector reconstruction mechanism that explicitly encodes relative geometric relations through a collaborative tracking error matrix, enabling scale-insensitive policy learning under time-varying communication topologies. Meanwhile, the GAT-based critic models evolving interaction structures for accurate global value estimation. The proposed framework is validated on real differential-drive robot platforms, and experimental results demonstrate superior stability and safety in dynamic scenarios with limited fields-of-view.

Xiao Sizhe, Dong Lijing, Bai Rui +1

School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University, Beijing, China

Multi-Robot Coordination Safe RL Multi-Agent Path Finding Cooperative MARL

Ask PDF

Abstract

Control Barrier Functions

Explore similar work

Beyond Safety Filtering: Control Barrier Function-Informed Reinforcement Learning for Connected and Automated Vehicles

CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions

Graph-Based Safe Reinforcement Learning for Multi-Agent Systems with Time-Varying Topology