Beyond Conservative Automated Driving in Multi-Agent Scenarios via Coupled Model Predictive Control and Deep Reinforcement Learning

Saeed Rahmani; Gözde Körpe; Zhenlin; Xu; Bruno Brito; Simeon Craig Calvert; Bart van Arem

cs.ROApr 15, 2026

Beyond Conservative Automated Driving in Multi-Agent Scenarios via Coupled Model Predictive Control and Deep Reinforcement Learning

Authors: Saeed Rahmani, Gözde Körpe, Zhenlin, Xu, Bruno Brito, Simeon Craig Calvert, Bart van Arem

Organizations: Gavin · TU Delft, Faculty of Civil Engineering and Geosciences, Department of Transport and Planning, Delft, The Netherlands · NVIDIA, San Jose, CA, USA

Abstract

Automated driving at unsignalized intersections is challenging due to complex multi-vehicle interactions and the need to balance safety and efficiency. Model Predictive Control (MPC) offers structured constraint handling through optimization but relies on hand-crafted rules that often produce overly conservative behavior. Deep Reinforcement Learning (RL) learns adaptive behaviors from experience but often struggles with safety assurance and generalization to unseen environments. In this study, we present an integrated MPC-RL framework to improve navigation performance in multi-agent scenarios. Experiments show that MPC-RL outperforms standalone MPC and end-to-end RL across three traffic-density levels. Collectively, MPC-RL reduces the collision rate by 21% and improves the success rate by 6.5% compared to pure MPC. We further evaluate zero-shot transfer to a highway merging scenario without retraining. Both MPC-based methods transfer substantially better than end-to-end PPO, which highlights the role of the MPC backbone in cross-scenario robustness. The framework also shows faster loss stabilization than end-to-end RL during training, which indicates a reduced learning burden. These results suggest that the integrated approach can improve the balance between safety performance and efficiency in multi-agent intersection scenarios, while the MPC component provides a strong foundation for generalization across driving environments. The implementation code is available open-source.

Figures & tables

Term	Formula	Role
$R_{\text{alive}}$	$0.2$	Survival bonus
$R_{\text{speed}}$	$1.5\cdot\min(v_{k}/v_{\max},\,1.5)$	Progress
$R_{\text{proximity}}$	$-2.0(1-2d_{\min})^{2}$ if $d_{\min}<0.5$	Safety
$R_{\text{smooth}}$	$-0.3\cdot\|\Delta a_{k}\|/a_{\max}$	Comfort
$R_{\text{center}}$	$0.3\left(1-\frac{\|l_{k}\|}{w_{\text{lane}}/2}\right)$	Lane keeping

TABLE I: Per-step reward components.

Parameter	Value
Training budget	4,000,000 steps
Batch size / Minibatch size	4,096 / 64
PPO epochs per update	10
Learning rate / Discount factor	$10^{-4}$ / 0.99
Policy network	FC [512, 256]
MPC prediction horizon	16 steps

TABLE II: Training and MPC hyperparameters

Fig. 2: Intersection scenario with ego vehicle and MPC reference trajectory.

Difficulty	Method	Success (%)	Collision (%)	Other (%)
Easy	MPC-RL	94.8	5.2	0.0
	Pure MPC	89.4	10.5	0.1
	Pure PPO	83.4	5.9	10.7
Moderate	MPC-RL	82.6	17.2	0.2
	Pure MPC	77.6	22.3	0.1
	Pure PPO	73.8	26.1	0.1

TABLE III: Intersection performance by difficulty level (1,000 episodes per method per level) and pooled across all levels (3,000 episodes per method), with pairwise statistical comparisons on the pooled data.

Fig. 3: Illustration of merging experiment with ego vehicle merging into the highway.

Difficulty	Method	Success (%)	Collision (%)	Off-road (%)	Sig.
Easy	MPC-RL	100.0	0.0	0.0	ns
	Pure MPC	100.0	0.0	0.0
	Pure PPO	0.0	0.0	100.0
Moderate	MPC-RL	100.0	0.0	0.0	ns
	Pure MPC	100.0	0.0	0.0
	Pure PPO	0.0	0.0	100.0

TABLE IV: Highway merging performance by difficulty level (1,000 episodes per method per level; zero-shot transfer for MPC-RL and Pure PPO).

Fig. 4: Training progress: Loss values for pure RL (PPO) and MPC-RL models.

Method	Mean (ms)	Std (ms)	Median (ms)
Pure PPO	0.71	0.09	0.70
Pure MPC	38.04	13.72	35.95
MPC-RL	38.36	10.06	36.98

TABLE V: Per-step inference time on the intersection scenario.

Autonomous Driving Planning

Autonomous driving planning selects safe, feasible vehicle actions and trajectories amid traffic, uncertainty, road rules, and changing goals.

See all 384 papers

Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections
Jul 23, 2026Gil Lifshits, Igal Bilik, Gilad KatzMulti-Agent Coordination Hierarchical RL
Ben-Gurion University of the Negev
Differentiable Model Predictive Safety for Heterogeneous Mobility at Urban Intersections
May 19, 2026Wenzhe Song, Hao ZhangMulti-Agent Coordination Intelligent Transportation Systems
School of Business Stevens Institute of Technology Hoboken, 07030, USA · Department of Mechanical Engineering Carnegie Mellon University Pittsburgh, 15213, USA
COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems
Mar 26, 2026Yifeng Zhang, Jieming Chen, Tingguang Zhou +4Centralized Training with Decentralized Execution Multi-Agent Coordination
Department of Mechanical Engineering, College of Design and Engineering, National University of Singapore, 21 Lower Kent Ridge Rd, 119077, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, 100872, China · School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China

Jul 23, 2026J/K move · Enter open · S save

Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections

Gil Lifshits, Igal Bilik, Gilad Katz

Ben-Gurion University of the Negev

Multi-Agent Coordination Hierarchical RL Autonomous Driving Decentralized MARL Cooperative MARL

Open paper AskPDF arXiv

Coordinating autonomous vehicles at unsignalized intersections remains a critical challenge for multi-agent reinforcement learning (MARL) systems, which typically struggle with combinatorial action spaces, reliance on privileged information, or rigid agent designs. We propose Master-Agent Proto-plan System (MAPS), a hierarchical deep reinforcement learning (DRL) architecture in which a centralized Master agent generates a compact, continuous embedding, denoted as proto-plan, that encodes a global coordination strategy. Decentralized Worker agents integrate this embedding with local observations to execute vehicle-specific control, decoupling strategic intent from tactical execution and enabling independent optimization of each module. As a proof-of-concept evaluation of this coordination mechanism, we test MAPS across 72 intersection configurations in HighwayEnv. MAPS achieves collision-free navigation while significantly reducing average travel time, outperforming state-of-the-art baselines. The learned proto-plans further exhibit robust generalization: a system trained with three agents achieves a 94% success rate when deployed zero-shot to five-agent scenarios, confirming that proto-plan-based hierarchical learning provides a promising framework for multi-vehicle coordination.

Abstract

Autonomous Driving Planning

Explore similar work

Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections

Differentiable Model Predictive Safety for Heterogeneous Mobility at Urban Intersections

COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems