HALight: Heterogeneous Multi-Agent Reinforcement Learning for Traffic Signal Control under Lane-Level Missing Observations
Authors: Chenkai Hu, Yingqian Zhang, Ming Xie, Yi Fang, Chengxiang Jin, Chao Li
Organizations: Department of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China · Zhejiang Institute of Communications Co.,LTD., Hangzhou, Zhejiang, 310030, China · College of Control Science and Engineering, Zhejiang University, Hangzhou, Zhejiang, 310027, China · Institute of Cyberspace Security, College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China
Multi-agent reinforcement learning has shown great promise for traffic signal control. However, most existing methods rely on two key assumptions: (i) agents are assumed to be homogeneous with unified observation/action spaces; (ii) full and reliable lane-level traffic observations are available to all agents. In practice, urban road networks exhibit substantial structural heterogeneity (e.g., lane configurations and phase designs), while sensor failures and incomplete detector coverage often result in missing incoming lane measurements. Such issues can severely impair coordination and destabilize learning. To address these challenges, we propose HALight, a communication-enhanced heterogeneous multi-agent reinforcement learning framework for traffic signal control under lane-level missing observations. HALight supports intersection-specific observation and action modeling to accommodate heterogeneous intersections, and incorporates a lightweight one-hop neighborhood communication module to enrich spatial context when lane-level observations are partially unavailable. Moreover, HALight adopts a sequential policy update scheme to improve coordination among heterogeneous agents. Experimental results on two real-world road networks demonstrate that HALight outperforms representative homogeneous and heterogeneous baselines in terms of traffic efficiency and convergence behavior. The code and datasets used in this work are available at https://github.com/Huchenkaii/HALight.
Figures & tables
Fig. 1: Illustration of two prevalent forms of intersection heterogeneity in real-world road networks: (a) heterogeneity across different intersection types, such as crossing intersections and T intersections, (b) heterogeneity among intersections of the same type, characterized by variations in lane configurations, geometric layouts, and phase designs.
Fig. 2: Illustration of lane-level observation unavailability under non-uniform ETC sensing coverage.
Fig. 3: Figure of phase setting scheme for different intersection types: PA and PB correspond to two typical phase plans for crossing intersections, while PC represents a phase plan for a T intersection.
Mi1:mA^(s,a)⋅min(ωim,clip(ωim,1±ϵ)).
Algorithm 1 The HALight Algorithm
Fig. 4: An overview of HALight, including (a) the agent decision process, which consists of local observation encoding, one-hop neighborhood communication, attentive aggregation, information generation, and decision making, and (b) the sequential training pipeline with replay buffer sampling and sequential updates.
Fig. 5: Figures of the evaluated road networks, including (a) Xiasha road network and (b) Gudang road network.
Fig. 6: Learning curves of RL-based TSC methods on (a) the Xiasha road network and (b) the Gudang road network.
Road Networks
Arrival Rate (vehicles/s)
Road Length (m)
Moderate
High
Xiasha
0.4
0.6
400
Gudang
0.85
1.5
600
TABLE I: ROAD NETWORKS AND FLOW STATISTICS
Algorithms
Hyperparameters
Value
DQN-based
Learning rate α
0.0005
Replay buffer size
1,800,000
Hidden size
64
ϵ decay steps
1,800,000
Batch size
32
Training sequence length
16
TABLE II: HYPERPARAMETERS
Algorithms
Metric
Scenario
Moderate Demand
High Demand
0%
10%
20%
30%
40%
0%
10%
20%
30%
40%
Fixed Time
ATT
196.66 ± 2.11
196.66 ± 2.11
196.66 ± 2.11
196.66 ± 2.11
196.66 ± 2.11
212.25 ± 6.12
212.25 ± 6.12
212.25 ± 6.12
212.25 ± 6.12
212.25 ± 6.12
AQL
1.26 ± 0.04
1.26 ± 0.04
1.26 ± 0.04
1.26 ± 0.04
1.26 ± 0.04
2.74 ± 0.31
2.74 ± 0.31
2.74 ± 0.31
2.74 ± 0.31
2.74 ± 0.31
AWT
28.83 ± 1.05
28.83 ± 1.05
28.83 ± 1.05
28.83 ± 1.05
28.83 ± 1.05
43.50 ± 5.31
43.50 ± 5.31
43.50 ± 5.31
43.50 ± 5.31
43.50 ± 5.31
TP
1358.49 ± 4.11
1358.49 ± 4.11
1358.49 ± 4.11
1358.49 ± 4.11
1358.49 ± 4.11
2025.02 ± 8.71
2025.02 ± 8.71
2025.02 ± 8.71
2025.02 ± 8.71
2025.02 ± 8.71
TABLE III: Overall performance of HALight and baseline methods under different pmiss on the Xiasha road network. The best results are highlighted in bold, and the second-best results are underlined. Results for methods that fail to converge are omitted.
Algorithms
Metric
Scenario
Moderate Demand
High Demand
0%
10%
20%
30%
40%
0%
10%
20%
30%
40%
Fixed Time
ATT
308.19 ± 1.73
308.19 ± 1.73
308.19 ± 1.73
308.19 ± 1.73
308.19 ± 1.73
332.98 ± 5.29
332.98 ± 5.29
332.98 ± 5.29
332.98 ± 5.29
332.98 ± 5.29
AQL
1.80 ± 0.03
1.80 ± 0.03
1.80 ± 0.03
1.80 ± 0.03
1.80 ± 0.03
5.11 ± 0.39
5.11 ± 0.39
5.11 ± 0.39
5.11 ± 0.39
5.11 ± 0.39
AWT
33.24 ± 0.59
33.24 ± 0.59
33.24 ± 0.59
33.24 ± 0.59
33.24 ± 0.59
55.55 ± 4.29
55.55 ± 4.29
55.55 ± 4.29
55.55 ± 4.29
55.55 ± 4.29
TP
2784.86 ± 6.83
2784.86 ± 6.83
2784.86 ± 6.83
2784.86 ± 6.83
2784.86 ± 6.83
4856.63 ± 19.56
4856.63 ± 19.56
4856.63 ± 19.56
4856.63 ± 19.56
4856.63 ± 19.56
TABLE IV: Overall performance of HALight and baseline methods under different pmiss on the Gudang road network. The best results are highlighted in bold, and the second-best results are underlined. Results for methods that fail to converge are omitted.
Fig. 7: Ablation results of AWT on the Xiasha road network under different observation missing rates: (a) moderate demand and (b) high demand.
Variant
Policy Modeling
Comm
RNN
Sequential Update
PS-MAPPO-RNN
Homo
×
✓
×
PS-MAPPO-RNN+Comm
Homo
✓
✓
×
Het-MAPPO-RNN
Hetero
×
✓
×
Het-MAPPO-RNN+Comm
Hetero
✓
✓
×
HAPPO-RNN
Hetero
×
✓
✓
HALight-Linear
Hetero
✓
×
✓
TABLE V: Configuration of ablation variants. Homo and Hetero denote homogeneous and heterogeneous policy modeling.
Fig. 8: AQL performance under direction-level missing observations on the Xiasha road network: (a) moderate demand and (b) high demand.
Fig. 9: ATT of HALight under different communication conditions: (a) packet loss, (b) communication delay, and (c) communication interval.
Traffic signal control studies how to set intersection phases and green times to reduce congestion while balancing safety, emissions, and priority needs.
Adaptive traffic signal control (ATSC) is crucial in alleviating congestion, maximizing throughput and promoting sustainable mobility in ever-expanding cities. Multi-Agent Reinforcement Learning (MARL) has recently shown significant potential in addressing complex traffic dynamics, but the intricacies of partial observability and coordination in decentralized environments still remain key challenges in formulating scalable and efficient control strategies. To address these challenges, we present CoordLight, a MARL-based framework designed to improve intra-neighborhood traffic by enhancing decision-making at individual junctions (agents), as well as coordination with neighboring agents, thereby scaling up to network-level traffic optimization. Specifically, we introduce the Queue Dynamic State Encoding (QDSE), a novel state representation based on vehicle queuing models, which strengthens the agents' capability to analyze, predict, and respond to local traffic dynamics. We further propose an advanced MARL algorithm, named Neighbor-aware Policy Optimization (NAPO). It integrates an attention mechanism that discerns the state and action dependencies among adjacent agents, aiming to facilitate more coordinated decision-making, and to improve policy learning updates through robust advantage calculation. This enables agents to identify and prioritize crucial interactions with influential neighbors, thus enhancing the targeted coordination and collaboration among agents. Through comprehensive evaluations against state-of-the-art traffic signal control methods over three real-world traffic datasets composed of up to 196 intersections, we empirically show that CoordLight consistently exhibits superior performance across diverse traffic networks with varying traffic flows. The code is available at https://github.com/marmotlab/CoordLight
Yifeng Zhang, Harsh Goel, Peizhuo Li +3
Department of Mechanical Engineering, National University of Singapore · Chandra Department of Electrical and Computer Engineering, The University of Texas at Austin · Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology
Adaptive Traffic Signal Control (ATSC) aims to optimize traffic flow and minimize delays by adjusting traffic lights in real time. Recent advances in Multi-agent Reinforcement Learning (MARL) have shown promise for ATSC, yet existing approaches still suffer from limited representational capacity, often leading to suboptimal performance and poor generalization in complex and dynamic traffic environments. On the other hand, Large Language Models (LLMs) excel at semantic representation, reasoning, and analysis, yet their propensity for hallucination and slow inference speeds often hinder their direct application to decision-making tasks. To address these challenges, we propose a novel learning paradigm named LATS that integrates LLMs and MARL, leveraging the former's strong prior knowledge and inductive abilities to enhance the latter's decision-making process. Specifically, we introduce a plug-and-play teacher-student learning module, where a trained embedding LLM serves as a teacher to generate rich semantic features that capture each intersection's topology structures and traffic dynamics. A much simpler (student) neural network then learns to emulate these features through knowledge distillation in the latent space, enabling the final model to operate independently from the LLM for downstream use in the RL decision-making process. This integration significantly enhances the overall model's representational capacity across diverse traffic scenarios, thus leading to more efficient and generalizable control strategies. Extensive experiments across diverse traffic datasets empirically demonstrate that our method enhances the representation learning capability of RL models, thereby leading to improved overall performance and generalization over both traditional RL and LLM-only approaches. [...]
Yifeng Zhang, Peizhuo Li, Tingguang Zhou +2
Department of Mechanical Engineering, National University of Singapore
Reinforcement learning (RL) has shown promise in traffic signal control (TSC). However, its reliance on predefined states limits responsiveness to observable open-world events that are absent from training data. IoT-enabled intersections provide heterogeneous observations from roadside sensors and cameras, creating opportunities to improve RL adaptability to such events. To this end, we propose ReasonLight, a multimodal foundation model-enhanced RL framework for zero-shot TSC. ReasonLight integrates three sources of information: structured traffic measurements, multi-view camera observations, and candidate phase decisions from a pre-trained RL controller. Given an RL-proposed phase, ReasonLight extracts visual semantics from multi-view images and aligns them with compact sensor-derived scene descriptions. This alignment enables a semantic-guided refinement module to either preserve or adjust the proposed action according to traffic rules and event semantics. To ensure operational reliability, refined actions are constrained by the set of available phases. Any invalid decision is rejected, and the system falls back to the original RL action. We evaluate ReasonLight on two types of rare events not seen during RL training: emergency vehicle priority and temporary traffic regulation. Experimental results show that ReasonLight achieves zero-shot adaptation without retraining. It reduces emergency vehicle waiting time by up to 88.7% compared with the RL-only backbone while preserving comparable routine traffic performance.
Aoyu Pang, Maonan Wang, Yuejiao Xie +3
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong · Shanghai AI Laboratory, Shanghai, China +1