Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks

Giray Onur; Azita Dabiri; Bart De Schutter

eess.SYAug 21, 2026

Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks

Authors: Giray Onur, Azita Dabiri, Bart De Schutter

Organizations: Delft Center for Systems and Control, Delft University of Technology, Delft, 2628CD, The Netherlands

Abstract

Transportation networks, in particular multi-class transportation networks (i.e., networks with mixed vehicle types), are complex systems that are challenging to control. Recently, Deep Reinforcement Learning (DRL), which learns control policies from interactions with the environment, and Model Predictive Control (MPC), which uses a system model to optimize control inputs, have been increasingly utilized for transportation network control. However, nonlinear system dynamics and high-dimensional state spaces in large-scale networks limit DRL's learning capacity under time-constrained training and increase MPC's computation time, hindering real-time implementation with limited computational resources. Moreover, MPC depends on an accurate network model, which is often unavailable for complex systems such as multi-class transportation networks. This paper proposes a novel DRL-MPC framework for multi-class transportation networks that divides control authority between DRL and MPC, combining DRL's fast online computation and model independence with MPC's built-in optimization and constraint-handling capabilities. In the hierarchical framework, MPC operates at the higher level and determines low-frequency control inputs whose slower update rate accommodates its high computation time, while DRL operates at the lower level and determines high-frequency control inputs using its fast online deployment. The framework is evaluated on a multi-class freeway network against a hierarchical MPC controller and a hybrid state-feedback-MPC controller, including scenarios with model mismatch and noisy traffic demands. Results show that the proposed framework outperforms the hybrid state-feedback-MPC controller, substantially reduces online computation time compared with the hierarchical MPC controller, and provides more effective constraint enforcement under model mismatch.

Figures & tables

Figure 1: (a) Block diagram illustrating the proposed DRL-MPC framework that divides control inputs between the DRL agent and the MPC controller. (b) Time scales for the low-frequency control inputs of the high-level MPC controller and the high-frequency control inputs of the low-level DRL agent.

Notation	Definition
$k$	Network sampling step counter
$k_{\mathrm{l}}$	Low-level (DRL) control step counter
$k_{\mathrm{h}}$	High-level (MPC) control step counter
$T$	Network sampling time interval
$T_{\mathrm{l}}$	Low-level control sampling time interval
$T_{\mathrm{h}}$	High-level control sampling time interval

Table 1: Key notations used for the DRL-MPC framework

Figure 2: Relationship between the network sampling step counter

k

, low-level control step counter

k_{\mathrm{l}}

, and high-level control step counter

k_{\mathrm{h}}

for

m_{\mathrm{l}}=3

and

m_{\mathrm{h}}=6

Figure 3: Block diagram illustrating the training process of the DRL-MPC framework using the DDPG algorithm.

Figure 4: The benchmark freeway network with one variable message sign for route guidance and two metered on-ramps.

Parameter	Unit	Original Value	Perturbed Value
$T$	[s] a	10	10
$\tau_{c}$	[s] a	18	18
$\kappa_{c}$	[veh/km/lane]	40	40
$\eta_{c}$	[km 2 /h]	60	60
$L_{m}$	[km]	1	1
$L^{\mathrm{veh}}_{c}$	[m]	5	5

Table 2: Original and perturbed values of multi-class freeway network parameters in the case study.

Figure 5: The vehicle demand profiles used in the case study for the (a) mainstream origin and (b) on-ramp origins. The solid curves show the nominal demands, and the dashed curves show the estimated demands used for MPC predictions in the mismatch cases.

Figure 6: Cumulative reward curves of the five DDPG and five SAC agents under four scenarios. Bold curves show the mean cumulative reward across the five agents and shaded areas show the min–max range.

Figure 7: Total Time Spent (TTS) comparison for all control frameworks under the four evaluation scenarios.

Figure 8: Total queue length constraint violation comparison for all control frameworks under the four evaluation scenarios.

Figure 9: Maximum queue length constraint violation comparison for all control frameworks under the four evaluation scenarios.

Figure 10: Total input variation (TIV) comparison for all control frameworks under the four evaluation scenarios.

Figure 11: Soft objective cost (SOC) comparison, where SOC combines TTS, input-variation penalty, and the soft quadratic queue-length-violation penalty.

Controller	Scenario 1		Scenario 2		Scenario 3		Scenario 4
SF-MPC	$30.5$ $\,\pm\,$	$1.9$	$29.0$ $\,\pm\,$	$2.2$	$27.7$ $\,\pm\,$	$1.0$	$27.9$ $\,\pm\,$	$1.2$
Hier. MPC	$1778.9$ $\,\pm\,$	$3.7$	$1736.7$ $\,\pm\,$	$5.1$	$1638.2$ $\,\pm\,$	$33.0$	$1659.3$ $\,\pm\,$	$39.2$
DDPG-MPC	$55.8$ $\,\pm\,$	$3.5$	$52.9$ $\,\pm\,$	$2.3$	$54.6$ $\,\pm\,$	$2.8$	$54.3$ $\,\pm\,$	$2.3$
SAC-MPC	$76.9$ $\,\pm\,$	$2.6$	$78.8$ $\,\pm\,$	$3.1$	$62.4$ $\,\pm\,$	$1.8$	$64.9$ $\,\pm\,$	$2.1$

Table 3: Mean and standard deviation of the total control framework computation time (s) across all experimental runs for each control framework and scenario.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 12: Tuned PI-ALINEA parameters across the four scenarios. Each marker corresponds to one of the five independently tuned parameter sets per scenario, and the solid horizontal tick denotes the per-scenario mean.

Intelligent Transportation Systems

Intelligent Transportation Systems apply sensing, communication, AI, and optimization to make transport networks safer and more efficient.

See all 303 papers

Composite-Gradient Learning for Shared Control Authority Between Deep Reinforcement Learning and Model Predictive Control
Sep 15, 2026Giray Önür, Azita Dabiri, Bart De SchutterRL Control Reinforcement Learning
Delft Center for Systems and Control, Delft University of Technology, Delft, The Netherlands
Ramp Metering Control via Hybrid State Deep Reinforcement Learning in Partially Observable Connected Vehicle Environments
Oct 5, 2026Youcef Mehamlia, Nadir Farhi, Meriem BoualiRL Control Deep Q-Learning
Cosys-Grettia, Univ Gustave Eiffel, F-77454 Marne-la-Vallee, France. · Laboratoire LITAN, ESTIN, Amizour, Algeria.
Learned Terminal Costs for Short-Horizon Model Predictive Control of Autonomous Trucks in Interactive Highway Traffic
Oct 29, 2026Hooman Hamze, Benjamin Houshmand, Deepthi Pathare +3Autonomous Driving Model Predictive Control
Chalmers University of Technology · University of Gothenburg · Trucks Technology and Industrial Division, Volvo Group

Sep 15, 2026J/K move · Enter open · S save

Composite-Gradient Learning for Shared Control Authority Between Deep Reinforcement Learning and Model Predictive Control

Giray Önür, Azita Dabiri, Bart De Schutter

Delft Center for Systems and Control, Delft University of Technology, Delft, The Netherlands

RL Control Reinforcement Learning Intelligent Transportation Systems Model Predictive Control

Open paper AskPDF arXiv

Integrated deep reinforcement learning (DRL) and model predictive control (MPC) methods are increasingly used to control autonomous systems by combining their complementary capabilities. DRL learns control policies through interaction with the environment. MPC uses a system model to optimize control inputs while accounting for constraints. In DRL-MPC frameworks with shared control authority, both the DRL agent and the MPC controller each determine part of the control inputs. However, common learning formulations treat MPC as part of the environment and therefore do not explicitly account for MPC's contribution to control or its interaction with the DRL agent. This paper proposes a novel composite-gradient learning (CGL) method that integrates the MPC controller into the learning process by representing the DRL and MPC control inputs as a joint action and accounting for their interaction when updating the DRL agent during training. CGL is evaluated on two multi-class freeway traffic networks with different strengths of interaction between the DRL and MPC control inputs and it is compared with alternative methods that treat MPC as part of the environment or that only partially incorporate MPC into learning. The results show that CGL offers limited benefit under weak interaction, but learns higher-performing control policies than the alternative methods in a subset of training runs under strong interaction, although the average control-performance gains remain modest.

Abstract

Intelligent Transportation Systems

Explore similar work

Composite-Gradient Learning for Shared Control Authority Between Deep Reinforcement Learning and Model Predictive Control

Ramp Metering Control via Hybrid State Deep Reinforcement Learning in Partially Observable Connected Vehicle Environments

Learned Terminal Costs for Short-Horizon Model Predictive Control of Autonomous Trucks in Interactive Highway Traffic