Ramp Metering Control via Hybrid State Deep Reinforcement Learning in Partially Observable Connected Vehicle Environments

Youcef Mehamlia; Nadir Farhi; Meriem Bouali

cs.LGOct 5, 2026

Ramp Metering Control via Hybrid State Deep Reinforcement Learning in Partially Observable Connected Vehicle Environments

Authors: Youcef Mehamlia, Nadir Farhi, Meriem Bouali

Organizations: Cosys-Grettia, Univ Gustave Eiffel, F-77454 Marne-la-Vallee, France. · Laboratoire LITAN, ESTIN, Amizour, Algeria.

Abstract

Freeway on-ramp merges are major sources of congestion, causing significant economic and environmental costs. While Deep Reinforcement Learning (DRL) offers a promising solution for ramp metering, existing approaches rely primarily on aggregated macroscopic data. Connected vehicles (CVs) provide vehicle-level observations that can complement aggregate traffic measurements, but their limited penetration produces incomplete microscopic information. This paper proposes a hybrid observation representation combining macroscopic traffic measurements with a two-channel grid encoding observed CV presence and speed. A Dueling Double Deep Q-Network processes these inputs to select ramp-metering green durations. The controller is trained under varying traffic demands and CV penetration rates and evaluated against ALINEA and macroscopic-only DRL variants in SUMO. Across 50 matched evaluation scenarios, the hybrid controller under partial CV visibility reduces the reported total travel time by 11.4 % and mean spillback duration by 84.9 % relative to ALINEA. Evaluating the same trained policy with full CV visibility yields a further travel-time reduction of approximately 1.6 %. Analysis across penetration rates suggests that the performance gap decreases as microscopic observations become more complete. These results support the use of complementary macroscopic and sparse microscopic observations for learning-based ramp metering. The source code implementation of the model is available at: https://github.com/youcefMehamlia/Multimodal-DRL-RMC

Figures & tables

Figure 1 : Freeway bottleneck and on-ramp configuration.

Figure 2 : Construction of the microscopic CV grid: (a) observation region, (b) spatial discretization, (c) removal of non-connected vehicles, and (d) two-channel encoding of CV presence and normalized speed.

Figure 3 : Hybrid two-stream architecture combining CNN-based microscopic features and fully connected macroscopic features before the Dueling heads.

Action index $a_{t}$	0	1	2	3	4	5	6	7
Green time $T_{g}$ (s)	5	10	15	20	25	30	35	40

Table 1 : Action Mapping for Green-Phase Durations

Strategy	Travel Time (s)	Time Loss (s)	Wait Time (s)	Spillback (s)	CO 2 ( $10^{9}$ mg)
No Control	$577,190\pm 72,494$	$338,777\pm 69,578$	$28,712\pm 6,539$	$\mathbf{1.48\pm 4.04}$	$2.11\pm 0.15$
ALINEA	$421,540\pm 41,909$	$170,540\pm 34,937$	$19,711\pm 5,514$	$455.7\pm 206.4$	$1.87\pm 0.12$
DQN Macro (No Lane)	$382,884\pm 35,719$	$130,092\pm 26,793$	$15,399\pm 6,771$	$94.40\pm 105.6$	$1.79\pm 0.12$
DQN Macro + Lane	$376,819\pm 31,966$	$124,718\pm 24,142$	$13,437\pm 4,737$	$44.00\pm 64.7$	$1.78\pm 0.10$
DQN Hybrid (Partial)	$373,633\pm 29,646$	$120,256\pm 21,292$	$12,404\pm 4,593$	$68.80\pm 95.6$	$1.78\pm 0.10$
DQN Hybrid (Full)	$\mathbf{367,825\pm 28,766}$	$\mathbf{112,530\pm 19,319}$	$\mathbf{12,345\pm 4,522}$	$37.60\pm 53.2$	$\mathbf{1.77\pm 0.10}$

Table 2 : Aggregated Performance Metrics Across Control Strategies (Mean

\pm

Std. Dev.). Bold and underlined values indicate the lowest and second-lowest means, respectively, across all listed controllers.

Figure 4 : Relative travel-time gap between partial and full CV visibility as a function of CV penetration for the same trained hybrid policy. The horizontal dashed line indicates equal performance.

Figure 5 : Training-return trajectory of the hybrid Dueling Double DQN controller.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Edge ID	Description	Length (m)	Lanes	Speed (m/s)
main_road	Upstream mainline	488.3	3	27.78
on_ramp	Ramp to signal	204.4	1	13.89
passage_area	Signal to merge	42.5	1	13.89
accel_area	Merge / acc. lane	193.8	4	22.22
end_main	Downstream section	193.1	3	27.78
off_ramp	Diverging off-ramp	161.3	2	13.89

Appendix

Table 3 : Key Edge Specifications in the SUMO Network

Component	Specification	Activation
Microscopic (CNN)	Conv2D (32, 64, 64 filters)	ELU
Macroscopic (FC)	Input: 14 features $\rightarrow$ FC(128)	ELU
Shared Dense	FC(512) $\rightarrow$ FC(256)	ELU
Value Head	FC(256) $\rightarrow$ FC(1)	Linear
Advantage Head	FC(256) $\rightarrow$ FC(8)	Linear

Appendix

Table 4 : Two-Stream Neural Network Architecture

Hyperparameter	Value	Reward Weight	Value
Total training steps	$2.1\times 10^{6}$	$w_{\text{speed\_merge}}$ (merge speed)	1.5
Learning rate ( $\alpha$ )	$1\times 10^{-4}$	$w_{\text{speed\_up}}$ (up speed)	1.0
Discount factor ( $\gamma$ )	$0.99$	$w_{\text{speed\_down}}$ (down speed)	0.5
Mini-batch size	32	$w_{\text{occ\_bottle}}$ (merge occ.)	2.0
Replay buffer size	$1\times 10^{6}$	$w_{\text{occ\_up}}$ (up occ.)	1.0
Target update ( $\tau$ )	$10^{-3}$	$w_{\text{queue}}$ (queue penalty)	1.0

Appendix

Table 5 : DRL Hyperparameters and Reward Weights

Explore similar work

Coordinated Lane-Level Variable Speed Limits and Ramp Metering for Successive Weaving Segments Considering Merging/Diverging Risks: A Hybrid Model Predictive Control and Multi-Agent Reinforcement Learning Approach
Sep 28, 2026Guodong Ma, Baofeng Sun, Wenyu Yang +1Standard Traffic Speed Benchmarks Traffic
School of Transportation, Jilin University, Changchun 130022, China · School of Transportation and Logistics, Southwest Jiaotong University, Chengdu 610031, China
Composite-Gradient Learning for Shared Control Authority Between Deep Reinforcement Learning and Model Predictive Control
Sep 15, 2026Giray Önür, Azita Dabiri, Bart De SchutterModel Predictive Control Multi-Agent Reinforcement Learning
Delft Center for Systems and Control, Delft University of Technology, Delft, The Netherlands
Momentum Based Reward Design for Low Emission Traffic Signal Control
May 28, 2026Chinmay Mundane, Amith Manoharan, Arun Kumar SinghTraffic Signal Control Potential-Based Reward Shaping
Institute of Technology, University of Tartu

Sep 28, 2026, math.OCJ/K move · Enter open · S save

Coordinated Lane-Level Variable Speed Limits and Ramp Metering for Successive Weaving Segments Considering Merging/Diverging Risks: A Hybrid Model Predictive Control and Multi-Agent Reinforcement Learning Approach

Guodong Ma, Baofeng Sun, Wenyu Yang, Zhihong Yao

School of Transportation, Jilin University, Changchun 130022, China · School of Transportation and Logistics, Southwest Jiaotong University, Chengdu 610031, China

Standard Traffic Speed Benchmarks Traffic Model Predictive Control Collision Prediction Multi-Agent Reinforcement Learning Sewing

Open paper AskPDF arXiv

Successive weaving segments (SWSs) on urban expressways are bottlenecks prone to recurrent congestion and collisions, requiring fine-grained active traffic management (ATM). Existing approaches struggle to balance the adaptive performance of data-driven optimization with the resilience and transferability of model-based control. We propose a hybrid framework to coordinate lane-level variable speed limits (VSLs) and ramp metering across SWSs. First, we reconstruct L-METANET, a lane-level macroscopic traffic flow model that captures free and forced lane changes. Second, we combine XGBoost-SHAP with a random-parameters binary logit (RPBL) model to derive analytical equations for merging and diverging collision risks and formulate system cost and reward functions. Third, we develop MPC-STMAPPO, a hierarchical controller integrating model predictive control (MPC) and multi-agent reinforcement learning (MARL). Its upper MPC layer uses L-METANET for long-horizon rolling optimization and generates baseline commands; its lower spatiotemporal MAPPO (ST-MAPPO) layer, enhanced with Mamba cells and graph attention, produces residual actions for short-horizon adjustment. Real-world experiments on the 18-km Eastern Expressway in Changchun, China, show that L-METANET accurately reproduces lane-changing-induced flow redistribution and capacity drops, with state evolution aligned with ground truth. XGBoost-SHAP-RPBL achieves AUCs above 0.80 in most tasks, outperforming conventional logit models. MPC-STMAPPO converges faster and performs better across multiple metrics than MPC- and MARL-based baselines. Under randomly fluctuating demand, it also significantly outperforms pure MARL in generalization, demonstrating strong potential for industrial deployment.

Abstract

Explore similar work

Coordinated Lane-Level Variable Speed Limits and Ramp Metering for Successive Weaving Segments Considering Merging/Diverging Risks: A Hybrid Model Predictive Control and Multi-Agent Reinforcement Learning Approach

Composite-Gradient Learning for Shared Control Authority Between Deep Reinforcement Learning and Model Predictive Control

Momentum Based Reward Design for Low Emission Traffic Signal Control