Safeguarding Multi-Agent Reinforcement Learning for Connected and Automated Vehicles with Recursively Feasible Control Barrier Functions

Jianye Xu; Yousef Sayed; Bassam Alrifaee

cs.RONov 3, 2026

Safeguarding Multi-Agent Reinforcement Learning for Connected and Automated Vehicles with Recursively Feasible Control Barrier Functions

Authors: Jianye Xu, Yousef Sayed, Bassam Alrifaee

Organizations: Department of Computer Science, RWTH Aachen University, Germany · Department of Aerospace Engineering, University of the Bundeswehr Munich, Germany

Abstract

Multi-Agent Reinforcement Learning (MARL) shows promise in coordinating connected and automated vehicles but offers no safety guarantee. Although Control Barrier Functions (CBFs) can add this guarantee, their optimization problem can become infeasible in practice due to control bounds. Moreover, they often overapproximate vehicle and road geometry, which makes them conservative. This paper proposes the Backup-Horizon CBF (BH-CBF), which evaluates safety along a horizon until all vehicles stop. BH-CBF is recursively feasible under control bounds, and it considers the exact vehicle and road geometry. Based on BH-CBF, we derive a fully decentralized safety filter that only needs a single broadcast of vehicle states and routes per step and stays real-time in dense traffic. We use BH-CBF to safeguard a MARL policy over its life cycle: as a reward during pre-training, as a run-time filter at deployment, and as a filter in the training loop that keeps fine-tuning on unseen maps collision-free. In simulations on real-world maps of a roundabout and an intersection, BH-CBF avoids all collisions of unsafe MARL policies and keeps its safety guarantee at every step. Existing filters either collide, lose the guarantee, or fall back to full braking far more often. In recorded roundabout traffic with 1264 human-driven vehicles over 115 km, our safeguarded policies shorten the median travel time by 14% compared with the human drivers, without any collision.

Figures & tables

Panel of Figure 1 — Fig. 1 : An unsafe MARL policy safeguarded by our BH-CBF. Light trails: predictions of braking to a standstill along the route. Red: standstill distance of a vehicle pair.

Panel of Figure 2 — Fig. 2 : Kinematic bicycle model. $\bm{p}=(p_{x},p_{y})$ : position; $\psi$ : heading; $v$ : speed; $\beta$ : slip angle; $\delta$ : steering angle; $\ell_{\mathrm{r}}$ : rear-axle distance; $\ell_{\mathrm{wb}}$ : wheelbase.

Panel of Figure 3 — Fig. 3 : Margin geometry. (a) Facet margin $h_{l}$ , whose minimum over all facets is the lane margin. $\bm{p}$ : position; $\bm{e}_{\mathrm{lon}},\bm{e}_{\mathrm{lat}}$ : body axes; $\bm{q}_{c}$ : corner; $\ell_{\mathrm{lon}},\ell_{\mathrm{lat}}$ : half-length, half-width; $\mathcal{F}_{l}=[\bm{b}_{l},\bm{b}_{l+1}]$ : facet of the left boundary ( $s=\mathrm{L}$ ); $\bm{n}_{l}$ : inward normal; red: clearance $\varepsilon_{\mathrm{lane}}$ . (b) Pair margin $h^{ij}$ . Red: clearance $\varepsilon_{\mathrm{veh}}$ ; $\bm{n}^{ij}$ : separating normal; dashed: separating line.

Panel of Figure 4 — Fig. 4 : Prediction of vehicle $i$ at step $k$ over $t$ (schematic). Blue: candidate prediction $\bm{\xi}^{i}_{k+t|k}(\bm{u}^{i})$ for a control $\bm{u}^{i}$ that accelerates and steers away from the route; gray: backup prediction $\bar{\bm{\xi}}^{i}_{k+t|k}$ , which tracks the route centerline (dashed). (a) Poses; $\bm{\xi}^{i}_{k+1|k}$ is the same for both predictions. (b) Speed. (c) Margin $h_{e}$ of the right lane constraint; ( 7 ) keeps the blue curve above $(1-\lambda)B_{e}(\bm{x}_{k})$ from $t=2$ on.

Panel of Figure 5 — Fig. 5 : Candidate predictions of vehicles $i$ (blue) and $j$ (green) and their backup predictions (gray) at step $k$ over $t$ (schematic). (a) Poses at a crossing; the backup predictions come closest at $t=3$ . (b) Pair margin $h^{ij}$ of the backup predictions (gray) and of the candidate predictions (black). The budget between the gray curve and $(1-\lambda)B^{ij}(\bm{x}_{k})$ is split into the shares of $i$ (blue fill) and $j$ (green fill).

Panel of Figure 6 — Fig. 6 : Decentralized corner inequalities ( 10 ) at step $k$ for one prediction step $t$ (schematic), drawn with the separating normal $\bm{n}^{ij}_{k+t|k}$ . Candidate predictions of vehicles $i$ (blue) and $j$ (green) and their backup predictions (gray). Blue and green fills: shares of $i$ and $j$ , as in Fig. 5 . Gray dot: closest corner of $\bar{\mathcal{R}}^{i}_{k+t|k}$ along $\bm{n}^{ij}_{k+t|k}$ ; blue dot: a corner of $\mathcal{R}^{i}_{k+t|k}$ that uses the full share $\rho^{ij}\eta^{ij}_{k+t|k}$ of the budget.

Panel of Figure 7 — Fig. 7 : BH-CBF pipeline. Solid: run-time safety filter. Dashed: training signals.

Panel of Figure 8 — Fig. 8 : The evaluation stages along a MARL policy’s life cycle.

Fig. 9 : Stage 1: training rollouts over pre-training by safety reward, smoothed over ten iterations, mean and standard error over five training seeds.

	Own map		Unseen map
	Collisions $\downarrow$	Throughput $\uparrow$	Collisions $\downarrow$	Throughput $\uparrow$
Reward method	[1/km]	[veh/s]	[1/km]	[veh/s]
TTC	0.86	1.08	1.27	0.92
One-Step CBF	0.71	1.03	1.61	0.80
BH-CBF (ours)	0.05	1.05	0.59	0.91

TABLE I : Stage 1: the 30 pre-trained policies (ten per safety reward, iteration 500) deployed without a filter.

Filter	Geometry	Condition	Split
BH-CBF (ours)	exact	backup horizon	corner budgets
A-Geometry	capsules	backup horizon	corner budgets
A-Condition	exact	one step	corner budgets
A-Split	exact	backup horizon	half split
A-Centralized	exact	backup horizon	centralized
B-SAT [ 19 ]	smoothed SAT	one step	half split

TABLE II : Compared safety filters; gray: as in BH-CBF.

	Collisions $\downarrow$	Runs $\downarrow$	Fallbacks $\downarrow$	Throughput $\uparrow$	Time $\downarrow$
Filter	[1/km]	$\notin\mathcal{C}$	[1/1000]	[veh/s]	[ms]
Unfiltered	1.39	60/60	–	0.87	–
B-Frenet	1.22	60/60	–	0.75	4.7
A-Condition	0.61	60/60	–	0.91	8.0
B-Disk	0.47	60/60	–	1.04	7.1
B-SAT	0.38	60/60	–	0.91	11.3

TABLE III : Stage 2: the ten unsafe source policies (TTC reward, iteration 100) deployed with fully decentralized filters, except A-Centralized; 60 runs (own and unseen map, three seeds each) per row. Runs

\notin\mathcal{C}

: runs that leave the certified set; time: computation time per step.

Panel of Figure 13 — Fig. 10 : Unsafe policies on the roundabout (a)–(d) and the intersection (e)–(h), without (top) and with (bottom) our BH-CBF filter. Blue: yielding vehicle 1; green: vehicle 2; red: collisions; gray: others (last footprint only); footprint interval: $1.0\text{\,}\mathrm{s}$ .

Panel of Figure 14 — Fig. 11 : Collisions of the TTC-reward policies over pre-training and fine-tuning, mean over runs (20 fine-tuning runs per setting, own and unseen map taken together) with standard deviation in ( 11(a) ) and standard error in ( 11(b) ).

	Travel time $\downarrow$	Stops $\downarrow$	Gap	RMS jerk $\downarrow$
Controlled by	median [s]	[%]	[m]	median [ $\mathrm{m}\text{\,}{\mathrm{s}}^{-3}$ ]
Humans	12.0	13.8	1.40	0.94
BH-CBF (ours)	10.3	2.0	1.22	3.66

TABLE IV : Stage 4: Human-comparison results over 115 km of 1264 recorded human-driven vehicles. Stops: stop before the ring entry; gap: 5th percentile of the minimum gap.

Panel of Figure 16 — Fig. 12 : Stage 4: two roundabout scenarios over $7\text{\,}\mathrm{s}$ . Filled: our BH-CBF-filtered policies; unfilled: recorded humans (time stamps with dashed lines). Other vehicles not shown.

Panel of Figure 17 — Fig. 13 : Stage 4: entering the ring. (b) Median and $2575\text{\,}\mathrm{\%}$ band. (c) Probability of entering the ring before the circulating vehicle (dots: share per time-to-arrival bin with $95\text{\,}\mathrm{\%}$ confidence intervals; thick curves: logistic fits with $95\text{\,}\mathrm{\%}$ confidence bands; thin blue curves: logistic fits of the ten single policies).

Panel of Figure 18 — Fig. 14 : Computation time of the BH-CBF filter per step on the roundabout, mean over 500 steps per bar; sampling period $\Delta t=$ 100\text{,}\mathrm{ms}$$ .

Control Barrier Functions

Control barrier functions encode safe states as constraints that keep a controlled system away from hazards.

See all 167 papers

Beyond Safety Filtering: Control Barrier Function-Informed Reinforcement Learning for Connected and Automated Vehicles
May 16, 2026Jianye Xu, Bassam AlrifaeeReward Shaping Control Barrier Functions
Department of Computer Science, RWTH Aachen University, Germany · Department of Aerospace Engineering, University of the Bundeswehr Munich, Germany
CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions
Oct 16, 2025Lizhi Yang, Blake Werner, Massimiliano de Sa +1RL for Robotics Control Barrier Functions
Caltech MCE
Graph-Based Safe Reinforcement Learning for Multi-Agent Systems with Time-Varying Topology
Sep 8, 2026Xiao Sizhe, Dong Lijing, Bai Rui +1Multi-Robot Coordination Safe RL
School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University, Beijing, China

May 16, 2026J/K move · Enter open · S save

Beyond Safety Filtering: Control Barrier Function-Informed Reinforcement Learning for Connected and Automated Vehicles

Jianye Xu, Bassam Alrifaee

Department of Computer Science, RWTH Aachen University, Germany · Department of Aerospace Engineering, University of the Bundeswehr Munich, Germany

Reward Shaping Control Barrier Functions RL for Autonomous Driving Safe RL Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Reinforcement Learning (RL) uses rewards to guide learning, yet reward design is typically hand-crafted using heuristics that can be difficult to tune. We propose a Control Barrier Function (CBF)-informed reward design for Multi-Agent RL (MARL) that converts CBF constraint values under joint MARL actions into a reward signal that explicitly guides safe learning. We compare against two heuristic reward baselines in a four-way multi-lane intersection with connected and automated vehicles. Results show that our method achieves the highest task performance and is less sensitive to reward hyperparameters, yielding consistently strong performance across the tested hyperparameter range. Code for reproducing the experimental results and a video demonstration are available at https://github.com/bassamlab/SigmaRL.

Abstract

Control Barrier Functions

Explore similar work

Beyond Safety Filtering: Control Barrier Function-Informed Reinforcement Learning for Connected and Automated Vehicles

CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions

Graph-Based Safe Reinforcement Learning for Multi-Agent Systems with Time-Varying Topology