TSC: Topology-Conditioned Stackelberg Coordination for Multi-Agent Reinforcement Learning in Interactive Driving
Authors: Xiaotong Zhang, Gang Xiong, Yuanjing Wang, Siyu Teng, Alois Knoll, Long Chen
Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Department of Natural Sciences, University of Durham, Durham DH1 3LE, United Kingdom · College of Civil and Transportation Engineering, Shenzhen University, Guangdong, 518060, China · Chair of Robotics, Artificial Intelligence and Realtime Systems, Technical University of Munich, 80333 Munich, Germany · State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China · Waytous Inc., Qingdao 266109, China
Safe and efficient autonomous driving in dense traffic is fundamentally a decentralized multi-agent coordination problem, where interactions at conflict points such as merging and weaving must be resolved reliably under partial observability. With only local and incomplete cues, interaction patterns can change rapidly, often causing unstable behaviors such as oscillatory yielding or unsafe commitments. Existing multi-agent reinforcement learning (MARL) approaches either adopt synchronous decision-making, which exacerbate non-stationarity, or depend on centralized sequencing mechanisms that scale poorly as traffic density increases. To address these limitations, we propose Topology-conditioned Stackelberg Coordination (TSC), a learning framework for decentralized interactive driving under communication-free execution, which extracts a time-varying directed priority graph from braid-inspired weaving relations between trajectories, thereby defining local leader-follower dependencies without constructing a global order of play. Conditioned on this graph, TSC endogenously factorizes dense interactions into graph-local Stackelberg subgames and, under centralized training and decentralized execution (CTDE), learns a sequential coordination policy that anticipates leaders via action prediction and trains followers through action-conditioned value learning to approximate local best responses, improving training stability and safety in dense traffic. Experiments across four dense traffic scenarios show that TSC achieves superior performance over representative MARL baselines across key metrics, most notably reducing collisions while maintaining competitive traffic efficiency and control smoothness.
Figures & tables
Fig. 1 : Topology-conditioned Stackelberg coordination from trajectory weaving in multi-vehicle traffic.
Fig. 2 : The TSC-Net architecture. Ego and neighbor encoders construct an interaction representation Hit , from which the TopoDecoder predicts pairwise priorities p^i←jt and node scores s^it to form a local priority graph. Guided by this graph, a TopK filter and TopoAttention aggregate the most influential neighbors into a compact decision state uit , which the PolicyHead maps to the continuous action ait . The same priority structure defines the local leader set Lit . The PredictHead estimates leader actions {a^j∣it}j∈Lit , and the ValueHead conditions on these predictions to approximate the local Stackelberg value, yielding TD advantages A^it for actor optimization under CTDE.
Fig. 3 : Evaluation scenarios for dense multi-vehicle interaction.
MFPO
SigmaRL
XP-MARL
TSC
Clover
CRAA↓
1.71
1.67
0.19
0.04
CRAM↓
0.00
0.00
0.03
0.03
CR↓
1.71
1.67
0.22
0.07
AS↑
86.6
87.1
84.8
88.2
SMLO↓
2.63
2.58
2.27
1.95
SMLA↓
11.2
10.4
8.95
8.89
TABLE I : Testing results on four scenarios from different MARL methods. Collision rate (CR [%] ) measures safety and is decomposed into agent-agent collisions CRAA and agent-map collisions CRAM . Average speed (AS [%] ) measures traffic efficiency. Smoothness (SM [%] ) measures control stability. Best is bold and second best is underlined within each scenario.
Fig. 4 : Comparison of safety, efficiency, and smoothness across four scenarios.
Fig. 5 : Collision rate of different methods under varying numbers of vehicles at test time.
Fig. 6 : Visualization of vehicle trajectories in one test episode. Black dots indicate collision events.
Fig. 7 : Qualitative trajectory visualization of TSC across four interaction scenarios, highlighting emergent behaviors including yielding, queuing, cutting in, bypassing, and reversing. For each vehicle, we plot a 25-step trajectory segment with distinct colors, where the luminosity fades from light to deep to indicate earlier to later positions.
Method
Clover
Weave
Merge
Bypass
TSC w/ random priority
0.30
1.62
1.11
1.79
TSC w/o Stackelberg
0.27
1.08
0.78
1.91
TSC w/o Top- K Filter
1.00
2.92
0.97
1.87
TSC
0.07
0.99
0.03
0.10
TABLE II : Ablation study on total collision rate (CR [%] ) across different scenarios.
Fig. 8 : Comparison of TSC with different TopoNbr-K settings.
Multi-Agent Self-Driving (MASD) systems provide an effective solution for coordinating autonomous vehicles to reduce congestion and enhance both safety and operational efficiency in future intelligent transportation systems. Multi-Agent Reinforcement Learning (MARL) has emerged as a promising approach for developing advanced end-to-end MASD systems. However, achieving efficient and safe collaboration in dynamic MASD systems remains a significant challenge in dense scenarios with complex agent interactions. To address this challenge, we propose a novel collaborative(CO-) interaction-aware(-IN) MARL framework, named COIN. Specifically, we develop a new counterfactual individual-global twin delayed deep deterministic policy gradient (CIG-TD3) algorithm, crafted in a "centralized training, decentralized execution" (CTDE) manner, which aims to jointly optimize the individual objectives (navigation) and the global objectives (collaboration) of agents. We further introduce a dual-level interaction-aware centralized critic architecture that captures both local pairwise interactions and global system-level dependencies, enabling more accurate global value estimation and improved credit assignment for collaborative policy learning. We conduct extensive simulation experiments in dense urban traffic environments, which demonstrate that COIN consistently outperforms other advanced baseline methods in both safety and efficiency across various system sizes. These results highlight its superiority in complex and dynamic MASD scenarios, as further validated through real-world robot demonstrations. Supplementary videos are available at https://marmotlab.github.io/COIN/
Yifeng Zhang, Jieming Chen, Tingguang Zhou +4
Department of Mechanical Engineering, College of Design and Engineering, National University of Singapore, 21 Lower Kent Ridge Rd, 119077, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, 100872, China · School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China
Coordinating autonomous vehicles at unsignalized intersections remains a critical challenge for multi-agent reinforcement learning (MARL) systems, which typically struggle with combinatorial action spaces, reliance on privileged information, or rigid agent designs. We propose Master-Agent Proto-plan System (MAPS), a hierarchical deep reinforcement learning (DRL) architecture in which a centralized Master agent generates a compact, continuous embedding, denoted as proto-plan, that encodes a global coordination strategy. Decentralized Worker agents integrate this embedding with local observations to execute vehicle-specific control, decoupling strategic intent from tactical execution and enabling independent optimization of each module. As a proof-of-concept evaluation of this coordination mechanism, we test MAPS across 72 intersection configurations in HighwayEnv. MAPS achieves collision-free navigation while significantly reducing average travel time, outperforming state-of-the-art baselines. The learned proto-plans further exhibit robust generalization: a system trained with three agents achieves a 94% success rate when deployed zero-shot to five-agent scenarios, confirming that proto-plan-based hierarchical learning provides a promising framework for multi-vehicle coordination.
Adaptive traffic signal control (ATSC) is crucial in alleviating congestion, maximizing throughput and promoting sustainable mobility in ever-expanding cities. Multi-Agent Reinforcement Learning (MARL) has recently shown significant potential in addressing complex traffic dynamics, but the intricacies of partial observability and coordination in decentralized environments still remain key challenges in formulating scalable and efficient control strategies. To address these challenges, we present CoordLight, a MARL-based framework designed to improve intra-neighborhood traffic by enhancing decision-making at individual junctions (agents), as well as coordination with neighboring agents, thereby scaling up to network-level traffic optimization. Specifically, we introduce the Queue Dynamic State Encoding (QDSE), a novel state representation based on vehicle queuing models, which strengthens the agents' capability to analyze, predict, and respond to local traffic dynamics. We further propose an advanced MARL algorithm, named Neighbor-aware Policy Optimization (NAPO). It integrates an attention mechanism that discerns the state and action dependencies among adjacent agents, aiming to facilitate more coordinated decision-making, and to improve policy learning updates through robust advantage calculation. This enables agents to identify and prioritize crucial interactions with influential neighbors, thus enhancing the targeted coordination and collaboration among agents. Through comprehensive evaluations against state-of-the-art traffic signal control methods over three real-world traffic datasets composed of up to 196 intersections, we empirically show that CoordLight consistently exhibits superior performance across diverse traffic networks with varying traffic flows. The code is available at https://github.com/marmotlab/CoordLight
Yifeng Zhang, Harsh Goel, Peizhuo Li +3
Department of Mechanical Engineering, National University of Singapore · Chandra Department of Electrical and Computer Engineering, The University of Texas at Austin · Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology