TSC: Topology-Conditioned Stackelberg Coordination for Multi-Agent Reinforcement Learning in Interactive Driving

Xiaotong Zhang; Gang Xiong; Yuanjing Wang; Siyu Teng; Alois Knoll; Long Chen

cs.ROFeb 27, 2026

TSC: Topology-Conditioned Stackelberg Coordination for Multi-Agent Reinforcement Learning in Interactive Driving

Authors: Xiaotong Zhang, Gang Xiong, Yuanjing Wang, Siyu Teng, Alois Knoll, Long Chen

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Department of Natural Sciences, University of Durham, Durham DH1 3LE, United Kingdom · College of Civil and Transportation Engineering, Shenzhen University, Guangdong, 518060, China ·

Abstract

Safe and efficient autonomous driving in dense traffic is fundamentally a decentralized multi-agent coordination problem, where interactions at conflict points such as merging and weaving must be resolved reliably under partial observability. With only local and incomplete cues, interaction patterns can change rapidly, often causing unstable behaviors such as oscillatory yielding or unsafe commitments. Existing multi-agent reinforcement learning (MARL) approaches either adopt synchronous decision-making, which exacerbate non-stationarity, or depend on centralized sequencing mechanisms that scale poorly as traffic density increases. To address these limitations, we propose Topology-conditioned Stackelberg Coordination (TSC), a learning framework for decentralized interactive driving under communication-free execution, which extracts a time-varying directed priority graph from braid-inspired weaving relations between trajectories, thereby defining local leader-follower dependencies without constructing a global order of play. Conditioned on this graph, TSC endogenously factorizes dense interactions into graph-local Stackelberg subgames and, under centralized training and decentralized execution (CTDE), learns a sequential coordination policy that anticipates leaders via action prediction and trains followers through action-conditioned value learning to approximate local best responses, improving training stability and safety in dense traffic. Experiments across four dense traffic scenarios show that TSC achieves superior performance over representative MARL baselines across key metrics, most notably reducing collisions while maintaining competitive traffic efficiency and control smoothness.

Figures & tables

Fig. 1 : Topology-conditioned Stackelberg coordination from trajectory weaving in multi-vehicle traffic.

Fig. 2 : The TSC-Net architecture. Ego and neighbor encoders construct an interaction representation $\mathbf{H}_{i}^{t}$ , from which the TopoDecoder predicts pairwise priorities $\mathbf{\hat{p}}_{i\leftarrow j}^{t}$ and node scores $\mathbf{\hat{s}}_{i}^{t}$ to form a local priority graph. Guided by this graph, a TopK filter and TopoAttention aggregate the most influential neighbors into a compact decision state $u_{i}^{t}$ , which the PolicyHead maps to the continuous action $a_{i}^{t}$ . The same priority structure defines the local leader set $\mathcal{L}_{i}^{t}$ . The PredictHead estimates leader actions $\{\hat{a}_{j\mid i}^{t}\}_{j\in\mathcal{L}_{i}^{t}}$ , and the ValueHead conditions on these predictions to approximate the local Stackelberg value, yielding TD advantages $\hat{A}_{i}^{t}$ for actor optimization under CTDE.

Fig. 3 : Evaluation scenarios for dense multi-vehicle interaction.

		MFPO	SigmaRL	XP-MARL	TSC
Clover	$\mathrm{CR}_{\mathrm{AA}}\downarrow$	1.71	1.67	0.19	0.04
	$\mathrm{CR}_{\mathrm{AM}}\downarrow$	0.00	0.00	0.03	0.03
	$\mathrm{CR}\downarrow$	1.71	1.67	0.22	0.07
	$\mathrm{AS}\uparrow$	86.6	87.1	84.8	88.2
	$\mathrm{SM}_{\mathrm{LO}}\downarrow$	2.63	2.58	2.27	1.95
	$\mathrm{SM}_{\mathrm{LA}}\downarrow$	11.2	10.4	8.95	8.89

TABLE I : Testing results on four scenarios from different MARL methods. Collision rate (CR

[\%]

) measures safety and is decomposed into agent-agent collisions

\mathrm{CR}_{\mathrm{AA}}

and agent-map collisions

\mathrm{CR}_{\mathrm{AM}}

. Average speed (AS

[\%]

) measures traffic efficiency. Smoothness (SM

[\%]

) measures control stability. Best is bold and second best is underlined within each scenario.

Fig. 4 : Comparison of safety, efficiency, and smoothness across four scenarios.

Fig. 5 : Collision rate of different methods under varying numbers of vehicles at test time.

Fig. 6 : Visualization of vehicle trajectories in one test episode. Black dots indicate collision events.

Fig. 7 : Qualitative trajectory visualization of TSC across four interaction scenarios, highlighting emergent behaviors including yielding, queuing, cutting in, bypassing, and reversing. For each vehicle, we plot a 25-step trajectory segment with distinct colors, where the luminosity fades from light to deep to indicate earlier to later positions.

Method	Clover	Weave	Merge	Bypass
TSC w/ random priority	0.30	1.62	1.11	1.79
TSC w/o Stackelberg	0.27	1.08	0.78	1.91
TSC w/o Top- $K$ Filter	1.00	2.92	0.97	1.87
TSC	0.07	0.99	0.03	0.10

TABLE II : Ablation study on total collision rate (CR

[\%]

) across different scenarios.

Fig. 8 : Comparison of TSC with different TopoNbr-K settings.

Multi-Agent Coordination

Multi-agent coordination studies how autonomous agents align actions, share information, and divide work to achieve individual or collective goals.

See all 501 papers

COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems
Mar 26, 2026Yifeng Zhang, Jieming Chen, Tingguang Zhou +4Centralized Training with Decentralized Execution Multi-Agent Coordination
Department of Mechanical Engineering, College of Design and Engineering, National University of Singapore, 21 Lower Kent Ridge Rd, 119077, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, 100872, China · School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China
Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections
Jul 23, 2026Gil Lifshits, Igal Bilik, Gilad KatzMulti-Agent Coordination Hierarchical RL
Ben-Gurion University of the Negev
CoordLight: Learning Decentralized Coordination for Network-Wide Traffic Signal Control
Mar 25, 2026Yifeng Zhang, Harsh Goel, Peizhuo Li +3Traffic Signal Control Multi-Agent Coordination
Department of Mechanical Engineering, National University of Singapore · Chandra Department of Electrical and Computer Engineering, The University of Texas at Austin · Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology

Mar 26, 2026J/K move · Enter open · S save

COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems

Yifeng Zhang, Jieming Chen, Tingguang Zhou, Tanishq Duhan+3

Department of Mechanical Engineering, College of Design and Engineering, National University of Singapore, 21 Lower Kent Ridge Rd, 119077, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, 100872, China · School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China

Centralized Training with Decentralized Execution Multi-Agent Coordination RL for Autonomous Driving Cooperative MARL Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Multi-Agent Self-Driving (MASD) systems provide an effective solution for coordinating autonomous vehicles to reduce congestion and enhance both safety and operational efficiency in future intelligent transportation systems. Multi-Agent Reinforcement Learning (MARL) has emerged as a promising approach for developing advanced end-to-end MASD systems. However, achieving efficient and safe collaboration in dynamic MASD systems remains a significant challenge in dense scenarios with complex agent interactions. To address this challenge, we propose a novel collaborative(CO-) interaction-aware(-IN) MARL framework, named COIN. Specifically, we develop a new counterfactual individual-global twin delayed deep deterministic policy gradient (CIG-TD3) algorithm, crafted in a "centralized training, decentralized execution" (CTDE) manner, which aims to jointly optimize the individual objectives (navigation) and the global objectives (collaboration) of agents. We further introduce a dual-level interaction-aware centralized critic architecture that captures both local pairwise interactions and global system-level dependencies, enabling more accurate global value estimation and improved credit assignment for collaborative policy learning. We conduct extensive simulation experiments in dense urban traffic environments, which demonstrate that COIN consistently outperforms other advanced baseline methods in both safety and efficiency across various system sizes. These results highlight its superiority in complex and dynamic MASD scenarios, as further validated through real-world robot demonstrations. Supplementary videos are available at https://marmotlab.github.io/COIN/

Abstract

Multi-Agent Coordination

Explore similar work

COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems

Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections

CoordLight: Learning Decentralized Coordination for Network-Wide Traffic Signal Control