HALight: Heterogeneous Multi-Agent Reinforcement Learning for Traffic Signal Control under Lane-Level Missing Observations
Authors: Chenkai Hu, Yingqian Zhang, Ming Xie, Yi Fang, Chengxiang Jin, Chao Li
Organizations: Department of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China · Zhejiang Institute of Communications Co.,LTD., Hangzhou, Zhejiang, 310030, China · College of Control Science and Engineering, Zhejiang University, Hangzhou, Zhejiang, 310027, China · Institute of Cyberspace Security, College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China
Multi-agent reinforcement learning has shown great promise for traffic signal control. However, most existing methods rely on two key assumptions: (i) agents are assumed to be homogeneous with unified observation/action spaces; (ii) full and reliable lane-level traffic observations are available to all agents. In practice, urban road networks exhibit substantial structural heterogeneity (e.g., lane configurations and phase designs), while sensor failures and incomplete detector coverage often result in missing incoming lane measurements. Such issues can severely impair coordination and destabilize learning. To address these challenges, we propose HALight, a communication-enhanced heterogeneous multi-agent reinforcement learning framework for traffic signal control under lane-level missing observations. HALight supports intersection-specific observation and action modeling to accommodate heterogeneous intersections, and incorporates a lightweight one-hop neighborhood communication module to enrich spatial context when lane-level observations are partially unavailable. Moreover, HALight adopts a sequential policy update scheme to improve coordination among heterogeneous agents. Experimental results on two real-world road networks demonstrate that HALight outperforms representative homogeneous and heterogeneous baselines in terms of traffic efficiency and convergence behavior. The code and datasets used in this work are available at https://github.com/Huchenkaii/HALight.
Figures & tables
Fig. 1: Illustration of two prevalent forms of intersection heterogeneity in real-world road networks: (a) heterogeneity across different intersection types, such as crossing intersections and T intersections, (b) heterogeneity among intersections of the same type, characterized by variations in lane configurations, geometric layouts, and phase designs.
Fig. 2: Illustration of lane-level observation unavailability under non-uniform ETC sensing coverage.
Fig. 3: Figure of phase setting scheme for different intersection types: PA and PB correspond to two typical phase plans for crossing intersections, while PC represents a phase plan for a T intersection.
Mi1:mA^(s,a)⋅min(ωim,clip(ωim,1±ϵ)).
Algorithm 1 The HALight Algorithm
Fig. 4: An overview of HALight, including (a) the agent decision process, which consists of local observation encoding, one-hop neighborhood communication, attentive aggregation, information generation, and decision making, and (b) the sequential training pipeline with replay buffer sampling and sequential updates.
Fig. 5: Figures of the evaluated road networks, including (a) Xiasha road network and (b) Gudang road network.
Fig. 6: Learning curves of RL-based TSC methods on (a) the Xiasha road network and (b) the Gudang road network.
Road Networks
Arrival Rate (vehicles/s)
Road Length (m)
Moderate
High
Xiasha
0.4
0.6
400
Gudang
0.85
1.5
600
TABLE I: ROAD NETWORKS AND FLOW STATISTICS
Algorithms
Hyperparameters
Value
DQN-based
Learning rate α
0.0005
Replay buffer size
1,800,000
Hidden size
64
ϵ decay steps
1,800,000
Batch size
32
Training sequence length
16
TABLE II: HYPERPARAMETERS
Algorithms
Metric
Scenario
Moderate Demand
High Demand
0%
10%
20%
30%
40%
0%
10%
20%
30%
40%
Fixed Time
ATT
196.66 ± 2.11
196.66 ± 2.11
196.66 ± 2.11
196.66 ± 2.11
196.66 ± 2.11
212.25 ± 6.12
212.25 ± 6.12
212.25 ± 6.12
212.25 ± 6.12
212.25 ± 6.12
AQL
1.26 ± 0.04
1.26 ± 0.04
1.26 ± 0.04
1.26 ± 0.04
1.26 ± 0.04
2.74 ± 0.31
2.74 ± 0.31
2.74 ± 0.31
2.74 ± 0.31
2.74 ± 0.31
AWT
28.83 ± 1.05
28.83 ± 1.05
28.83 ± 1.05
28.83 ± 1.05
28.83 ± 1.05
43.50 ± 5.31
43.50 ± 5.31
43.50 ± 5.31
43.50 ± 5.31
43.50 ± 5.31
TP
1358.49 ± 4.11
1358.49 ± 4.11
1358.49 ± 4.11
1358.49 ± 4.11
1358.49 ± 4.11
2025.02 ± 8.71
2025.02 ± 8.71
2025.02 ± 8.71
2025.02 ± 8.71
2025.02 ± 8.71
TABLE III: Overall performance of HALight and baseline methods under different pmiss on the Xiasha road network. The best results are highlighted in bold, and the second-best results are underlined. Results for methods that fail to converge are omitted.
Algorithms
Metric
Scenario
Moderate Demand
High Demand
0%
10%
20%
30%
40%
0%
10%
20%
30%
40%
Fixed Time
ATT
308.19 ± 1.73
308.19 ± 1.73
308.19 ± 1.73
308.19 ± 1.73
308.19 ± 1.73
332.98 ± 5.29
332.98 ± 5.29
332.98 ± 5.29
332.98 ± 5.29
332.98 ± 5.29
AQL
1.80 ± 0.03
1.80 ± 0.03
1.80 ± 0.03
1.80 ± 0.03
1.80 ± 0.03
5.11 ± 0.39
5.11 ± 0.39
5.11 ± 0.39
5.11 ± 0.39
5.11 ± 0.39
AWT
33.24 ± 0.59
33.24 ± 0.59
33.24 ± 0.59
33.24 ± 0.59
33.24 ± 0.59
55.55 ± 4.29
55.55 ± 4.29
55.55 ± 4.29
55.55 ± 4.29
55.55 ± 4.29
TP
2784.86 ± 6.83
2784.86 ± 6.83
2784.86 ± 6.83
2784.86 ± 6.83
2784.86 ± 6.83
4856.63 ± 19.56
4856.63 ± 19.56
4856.63 ± 19.56
4856.63 ± 19.56
4856.63 ± 19.56
TABLE IV: Overall performance of HALight and baseline methods under different pmiss on the Gudang road network. The best results are highlighted in bold, and the second-best results are underlined. Results for methods that fail to converge are omitted.
Fig. 7: Ablation results of AWT on the Xiasha road network under different observation missing rates: (a) moderate demand and (b) high demand.
Variant
Policy Modeling
Comm
RNN
Sequential Update
PS-MAPPO-RNN
Homo
×
✓
×
PS-MAPPO-RNN+Comm
Homo
✓
✓
×
Het-MAPPO-RNN
Hetero
×
✓
×
Het-MAPPO-RNN+Comm
Hetero
✓
✓
×
HAPPO-RNN
Hetero
×
✓
✓
HALight-Linear
Hetero
✓
×
✓
TABLE V: Configuration of ablation variants. Homo and Hetero denote homogeneous and heterogeneous policy modeling.
Fig. 8: AQL performance under direction-level missing observations on the Xiasha road network: (a) moderate demand and (b) high demand.
Fig. 9: ATT of HALight under different communication conditions: (a) packet loss, (b) communication delay, and (c) communication interval.
Traffic signal control studies how to set intersection phases and green times to reduce congestion while balancing safety, emissions, and priority needs.
Department of Mechanical Engineering, National University of Singapore · Chandra Department of Electrical and Computer Engineering, The University of Texas at Austin · Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong · Shanghai AI Laboratory, Shanghai, China +1