Transportation networks, in particular multi-class transportation networks (i.e., networks with mixed vehicle types), are complex systems that are challenging to control. Recently, Deep Reinforcement Learning (DRL), which learns control policies from interactions with the environment, and Model Predictive Control (MPC), which uses a system model to optimize control inputs, have been increasingly utilized for transportation network control. However, nonlinear system dynamics and high-dimensional state spaces in large-scale networks limit DRL's learning capacity under time-constrained training and increase MPC's computation time, hindering real-time implementation with limited computational resources. Moreover, MPC depends on an accurate network model, which is often unavailable for complex systems such as multi-class transportation networks. This paper proposes a novel DRL-MPC framework for multi-class transportation networks that divides control authority between DRL and MPC, combining DRL's fast online computation and model independence with MPC's built-in optimization and constraint-handling capabilities. In the hierarchical framework, MPC operates at the higher level and determines low-frequency control inputs whose slower update rate accommodates its high computation time, while DRL operates at the lower level and determines high-frequency control inputs using its fast online deployment. The framework is evaluated on a multi-class freeway network against a hierarchical MPC controller and a hybrid state-feedback-MPC controller, including scenarios with model mismatch and noisy traffic demands. Results show that the proposed framework outperforms the hybrid state-feedback-MPC controller, substantially reduces online computation time compared with the hierarchical MPC controller, and provides more effective constraint enforcement under model mismatch.
Figures & tables
Figure 1: (a) Block diagram illustrating the proposed DRL-MPC framework that divides control inputs between the DRL agent and the MPC controller. (b) Time scales for the low-frequency control inputs of the high-level MPC controller and the high-frequency control inputs of the low-level DRL agent.
Notation
Definition
k
Network sampling step counter
kl
Low-level (DRL) control step counter
kh
High-level (MPC) control step counter
T
Network sampling time interval
Tl
Low-level control sampling time interval
Th
High-level control sampling time interval
Table 1: Key notations used for the DRL-MPC framework
Figure 2: Relationship between the network sampling step counter k , low-level control step counter kl , and high-level control step counter kh for ml=3 and mh=6 .
Figure 3: Block diagram illustrating the training process of the DRL-MPC framework using the DDPG algorithm.
Figure 4: The benchmark freeway network with one variable message sign for route guidance and two metered on-ramps.
Parameter
Unit
Original Value
Perturbed Value
T
[s] a
10
10
τc
[s] a
18
18
κc
[veh/km/lane]
40
40
ηc
[km 2 /h]
60
60
Lm
[km]
1
1
Lcveh
[m]
5
5
Table 2: Original and perturbed values of multi-class freeway network parameters in the case study.
Figure 5: The vehicle demand profiles used in the case study for the (a) mainstream origin and (b) on-ramp origins. The solid curves show the nominal demands, and the dashed curves show the estimated demands used for MPC predictions in the mismatch cases.
Figure 6: Cumulative reward curves of the five DDPG and five SAC agents under four scenarios. Bold curves show the mean cumulative reward across the five agents and shaded areas show the min–max range.
Figure 7: Total Time Spent (TTS) comparison for all control frameworks under the four evaluation scenarios.
Figure 8: Total queue length constraint violation comparison for all control frameworks under the four evaluation scenarios.
Figure 9: Maximum queue length constraint violation comparison for all control frameworks under the four evaluation scenarios.
Figure 10: Total input variation (TIV) comparison for all control frameworks under the four evaluation scenarios.
Figure 11: Soft objective cost (SOC) comparison, where SOC combines TTS, input-variation penalty, and the soft quadratic queue-length-violation penalty.
Controller
Scenario 1
Scenario 2
Scenario 3
Scenario 4
SF-MPC
30.5±
1.9
29.0±
2.2
27.7±
1.0
27.9±
1.2
Hier. MPC
1778.9±
3.7
1736.7±
5.1
1638.2±
33.0
1659.3±
39.2
DDPG-MPC
55.8±
3.5
52.9±
2.3
54.6±
2.8
54.3±
2.3
SAC-MPC
76.9±
2.6
78.8±
3.1
62.4±
1.8
64.9±
2.1
Table 3: Mean and standard deviation of the total control framework computation time (s) across all experimental runs for each control framework and scenario.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Tuned PI-ALINEA parameters across the four scenarios. Each marker corresponds to one of the five independently tuned parameter sets per scenario, and the solid horizontal tick denotes the per-scenario mean.
Integrated deep reinforcement learning (DRL) and model predictive control (MPC) methods are increasingly used to control autonomous systems by combining their complementary capabilities. DRL learns control policies through interaction with the environment. MPC uses a system model to optimize control inputs while accounting for constraints. In DRL-MPC frameworks with shared control authority, both the DRL agent and the MPC controller each determine part of the control inputs. However, common learning formulations treat MPC as part of the environment and therefore do not explicitly account for MPC's contribution to control or its interaction with the DRL agent. This paper proposes a novel composite-gradient learning (CGL) method that integrates the MPC controller into the learning process by representing the DRL and MPC control inputs as a joint action and accounting for their interaction when updating the DRL agent during training. CGL is evaluated on two multi-class freeway traffic networks with different strengths of interaction between the DRL and MPC control inputs and it is compared with alternative methods that treat MPC as part of the environment or that only partially incorporate MPC into learning. The results show that CGL offers limited benefit under weak interaction, but learns higher-performing control policies than the alternative methods in a subset of training runs under strong interaction, although the average control-performance gains remain modest.
Giray Önür, Azita Dabiri, Bart De Schutter
Delft Center for Systems and Control, Delft University of Technology, Delft, The Netherlands
Freeway on-ramp merges are major sources of congestion, causing significant economic and environmental costs. While Deep Reinforcement Learning (DRL) offers a promising solution for ramp metering, existing approaches rely primarily on aggregated macroscopic data. Connected vehicles (CVs) provide vehicle-level observations that can complement aggregate traffic measurements, but their limited penetration produces incomplete microscopic information. This paper proposes a hybrid observation representation combining macroscopic traffic measurements with a two-channel grid encoding observed CV presence and speed. A Dueling Double Deep Q-Network processes these inputs to select ramp-metering green durations. The controller is trained under varying traffic demands and CV penetration rates and evaluated against ALINEA and macroscopic-only DRL variants in SUMO. Across 50 matched evaluation scenarios, the hybrid controller under partial CV visibility reduces the reported total travel time by 11.4 % and mean spillback duration by 84.9 % relative to ALINEA. Evaluating the same trained policy with full CV visibility yields a further travel-time reduction of approximately 1.6 %. Analysis across penetration rates suggests that the performance gap decreases as microscopic observations become more complete. These results support the use of complementary macroscopic and sparse microscopic observations for learning-based ramp metering. The source code implementation of the model is available at: https://github.com/youcefMehamlia/Multimodal-DRL-RMC
Autonomous highway driving for heavy-duty trucks is a challenging control problem. The greater mass, dimensions and intricate dynamics increase the requirements for cooperation from surrounding traffic in critical maneuvers, requiring a careful balance between safety constraints and efficient long-horizon decision making. Model Predictive Control (MPC) provides rigorous constraint satisfaction but relies on terminal cost which can limit its effectiveness at short, computationally affordable horizons. Reinforcement Learning (RL) captures long-term driving behavior but offers no constraint guarantees and tends toward overly conservative policies when trained under sparse collision penalties. In this paper, we propose a hybrid MPC and RL framework for highway driving in autonomous trucks, where a finite-horizon MPC, solved via dynamic programming, uses a pre-trained RL model for terminal cost estimation. We evaluate the framework in a simulation environment with reactive surrounding traffic and compare it against pure MPC with zero terminal cost, rollout with terminal cost estimation using a base policy, and pure RL. Results show that the hybrid approach achieves zero collisions with the lowest cost of any configuration.
Hooman Hamze, Benjamin Houshmand, Deepthi Pathare +3
Chalmers University of Technology · University of Gothenburg · Trucks Technology and Industrial Division, Volvo Group