Beyond Conservative Automated Driving in Multi-Agent Scenarios via Coupled Model Predictive Control and Deep Reinforcement Learning

Saeed Rahmani; Gözde Körpe; Zhenlin; Xu; Bruno Brito; Simeon Craig Calvert; Bart van Arem

cs.ROApr 15, 2026

Beyond Conservative Automated Driving in Multi-Agent Scenarios via Coupled Model Predictive Control and Deep Reinforcement Learning

Authors: Saeed Rahmani, Gözde Körpe, Zhenlin, Xu, Bruno Brito, Simeon Craig Calvert, Bart van Arem

Organizations: Gavin · TU Delft, Faculty of Civil Engineering and Geosciences, Department of Transport and Planning, Delft, The Netherlands · NVIDIA, San Jose, CA, USA

Abstract

Automated driving at unsignalized intersections is challenging due to complex multi-vehicle interactions and the need to balance safety and efficiency. Model Predictive Control (MPC) offers structured constraint handling through optimization but relies on hand-crafted rules that often produce overly conservative behavior. Deep Reinforcement Learning (RL) learns adaptive behaviors from experience but often struggles with safety assurance and generalization to unseen environments. In this study, we present an integrated MPC-RL framework to improve navigation performance in multi-agent scenarios. Experiments show that MPC-RL outperforms standalone MPC and end-to-end RL across three traffic-density levels. Collectively, MPC-RL reduces the collision rate by 21% and improves the success rate by 6.5% compared to pure MPC. We further evaluate zero-shot transfer to a highway merging scenario without retraining. Both MPC-based methods transfer substantially better than end-to-end PPO, which highlights the role of the MPC backbone in cross-scenario robustness. The framework also shows faster loss stabilization than end-to-end RL during training, which indicates a reduced learning burden. These results suggest that the integrated approach can improve the balance between safety performance and efficiency in multi-agent intersection scenarios, while the MPC component provides a strong foundation for generalization across driving environments. The implementation code is available open-source.

Figures & tables

Term	Formula	Role
$R_{\text{alive}}$	$0.2$	Survival bonus
$R_{\text{speed}}$	$1.5\cdot\min(v_{k}/v_{\max},\,1.5)$	Progress
$R_{\text{proximity}}$	$-2.0(1-2d_{\min})^{2}$ if $d_{\min}<0.5$	Safety
$R_{\text{smooth}}$	$-0.3\cdot\|\Delta a_{k}\|/a_{\max}$	Comfort
$R_{\text{center}}$	$0.3\left(1-\frac{\|l_{k}\|}{w_{\text{lane}}/2}\right)$	Lane keeping

TABLE I: Per-step reward components.

Parameter	Value
Training budget	4,000,000 steps
Batch size / Minibatch size	4,096 / 64
PPO epochs per update	10
Learning rate / Discount factor	$10^{-4}$ / 0.99
Policy network	FC [512, 256]
MPC prediction horizon	16 steps

TABLE II: Training and MPC hyperparameters

Fig. 2: Intersection scenario with ego vehicle and MPC reference trajectory.

Difficulty	Method	Success (%)	Collision (%)	Other (%)
Easy	MPC-RL	94.8	5.2	0.0
	Pure MPC	89.4	10.5	0.1
	Pure PPO	83.4	5.9	10.7
Moderate	MPC-RL	82.6	17.2	0.2
	Pure MPC	77.6	22.3	0.1
	Pure PPO	73.8	26.1	0.1

TABLE III: Intersection performance by difficulty level (1,000 episodes per method per level) and pooled across all levels (3,000 episodes per method), with pairwise statistical comparisons on the pooled data.

Fig. 3: Illustration of merging experiment with ego vehicle merging into the highway.

Difficulty	Method	Success (%)	Collision (%)	Off-road (%)	Sig.
Easy	MPC-RL	100.0	0.0	0.0	ns
	Pure MPC	100.0	0.0	0.0
	Pure PPO	0.0	0.0	100.0
Moderate	MPC-RL	100.0	0.0	0.0	ns
	Pure MPC	100.0	0.0	0.0
	Pure PPO	0.0	0.0	100.0

TABLE IV: Highway merging performance by difficulty level (1,000 episodes per method per level; zero-shot transfer for MPC-RL and Pure PPO).

Fig. 4: Training progress: Loss values for pure RL (PPO) and MPC-RL models.

Method	Mean (ms)	Std (ms)	Median (ms)
Pure PPO	0.71	0.09	0.70
Pure MPC	38.04	13.72	35.95
MPC-RL	38.36	10.06	36.98

TABLE V: Per-step inference time on the intersection scenario.

Autonomous Driving Planning

Autonomous driving planning selects safe, feasible vehicle actions and trajectories amid traffic, uncertainty, road rules, and changing goals.

See all 384 papers

Jul 23, 2026

Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections

Coordinating autonomous vehicles at unsignalized intersections remains a critical challenge for multi-agent reinforcement learning (MARL) systems, which typically struggle with combinatorial action spaces, reliance on privileged information, or rigid agent designs. We propose Master-Agent Proto-plan System (MAPS), a hierarchical deep reinforcement learning (DRL) architecture in which a centralized Master agent generates a compact, continuous embedding, denoted as proto-plan, that encodes a global coordination strategy. Decentralized Worker agents integrate this embedding with local observations to execute vehicle-specific control, decoupling strategic intent from tactical execution and enabling independent optimization of each module. As a proof-of-concept evaluation of this coordination mechanism, we test MAPS across 72 intersection configurations in HighwayEnv. MAPS achieves collision-free navigation while significantly reducing average travel time, outperforming state-of-the-art baselines. The learned proto-plans further exhibit robust generalization: a system trained with three agents achieves a 94% success rate when deployed zero-shot to five-agent scenarios, confirming that proto-plan-based hierarchical learning provides a promising framework for multi-vehicle coordination.

Gil Lifshits, Igal Bilik, Gilad Katz

Ben-Gurion University of the Negev

Multi-Agent Coordination Hierarchical RL Autonomous Driving Decentralized MARL Cooperative MARL

Ask PDF

May 19, 2026

Differentiable Model Predictive Safety for Heterogeneous Mobility at Urban Intersections

The imminent integration of autonomous vehicles and mobile robots in urban settings presents a critical safety challenge for future intelligent transportation systems. This paper addresses the complex problem of coordinating heterogeneous agents with disparate dynamics at unregulated intersections. We introduce a novel framework, differentiable model predictive safety (DMPS), which embeds the foresight of model-predictive control into a data-driven, end-to-end reinforcement learning architecture. DMPS agents learn a latent dynamics model to predict future trajectories contingent on their actions. A learned, differentiable safety critic then evaluates the risk of these trajectories. Crucially, by leveraging backpropagation through the entire unrolled predictive model, agents can efficiently compute the gradient of future safety with respect to their current action, enabling a minimal and precise online safety correction. Integrated into a multi-agent training scheme, DMPS virtually eliminates collisions to less than 5.6% in high-density, mixed vehicle-robot traffic simulations, demonstrating state-of-the-art safety without compromising energy and traffic efficiency.

Wenzhe Song, Hao Zhang

School of Business Stevens Institute of Technology Hoboken, 07030, USA · Department of Mechanical Engineering Carnegie Mellon University Pittsburgh, 15213, USA

Multi-Agent Coordination Intelligent Transportation Systems Model Predictive Control Safe RL Multi-Agent Reinforcement Learning

Ask PDF

Mar 26, 2026

COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems

Multi-Agent Self-Driving (MASD) systems provide an effective solution for coordinating autonomous vehicles to reduce congestion and enhance both safety and operational efficiency in future intelligent transportation systems. Multi-Agent Reinforcement Learning (MARL) has emerged as a promising approach for developing advanced end-to-end MASD systems. However, achieving efficient and safe collaboration in dynamic MASD systems remains a significant challenge in dense scenarios with complex agent interactions. To address this challenge, we propose a novel collaborative(CO-) interaction-aware(-IN) MARL framework, named COIN. Specifically, we develop a new counterfactual individual-global twin delayed deep deterministic policy gradient (CIG-TD3) algorithm, crafted in a "centralized training, decentralized execution" (CTDE) manner, which aims to jointly optimize the individual objectives (navigation) and the global objectives (collaboration) of agents. We further introduce a dual-level interaction-aware centralized critic architecture that captures both local pairwise interactions and global system-level dependencies, enabling more accurate global value estimation and improved credit assignment for collaborative policy learning. We conduct extensive simulation experiments in dense urban traffic environments, which demonstrate that COIN consistently outperforms other advanced baseline methods in both safety and efficiency across various system sizes. These results highlight its superiority in complex and dynamic MASD scenarios, as further validated through real-world robot demonstrations. Supplementary videos are available at https://marmotlab.github.io/COIN/

Yifeng Zhang, Jieming Chen, Tingguang Zhou +4

Department of Mechanical Engineering, College of Design and Engineering, National University of Singapore, 21 Lower Kent Ridge Rd, 119077, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, 100872, China · School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China

Centralized Training with Decentralized Execution Multi-Agent Coordination RL for Autonomous Driving Cooperative MARL Multi-Agent Reinforcement Learning

Ask PDF

Abstract

Autonomous Driving Planning

Explore similar work

Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections

Differentiable Model Predictive Safety for Heterogeneous Mobility at Urban Intersections

COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems