Dual-Interaction-Aware Cooperative Control Strategy for Alleviating Mixed Traffic Congestion

Zhengxuan Liu; Yuxin Cai; Yijing Wang; Xiangkun He; Chen Lv; Zhiqiang Zuo

eess.SYMar 4, 2026

Dual-Interaction-Aware Cooperative Control Strategy for Alleviating Mixed Traffic Congestion

Authors: Zhengxuan Liu, Yuxin Cai, Yijing Wang, Xiangkun He, Chen Lv, Zhiqiang Zuo

Organizations: Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System, School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China · School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore · Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China

Abstract

As Intelligent Transportation System (ITS) develops, Connected and Automated Vehicles (CAVs) are expected to significantly reduce traffic congestion through cooperative strategies, such as in bottleneck areas. However, the uncertainty and diversity in the behaviors of Human-Driven Vehicles (HDVs) in mixed traffic environments present major challenges for CAV cooperation. This paper proposes a Dual-Interaction-Aware Cooperative Control (DIACC) strategy that enhances both local and global interaction perception within the Multi-Agent Reinforcement Learning (MARL) framework for Connected and Automated Vehicles (CAVs) in mixed traffic bottleneck scenarios. The DIACC strategy consists of three key innovations: 1) A Decentralized Interaction-Adaptive Decision-Making (D-IADM) module that enhances actor's local interaction perception by distinguishing CAV-CAV cooperative interactions from CAV-HDV observational interactions. 2) A Centralized Interaction-Enhanced Critic (C-IEC) that improves critic's global traffic understanding through interaction-aware value estimation, providing more accurate guidance for policy updates. 3) A reward design that employs softmin aggregation with temperature annealing to prioritize interaction-intensive scenarios in mixed traffic. Additionally, a lightweight Proactive Safety-based Action Refinement (PSAR) module applies rule-based corrections to accelerate training convergence. Experimental results demonstrate that DIACC significantly improves traffic efficiency and adaptability compared to rule-based and benchmark MARL models.

Figures & tables

Panel of Figure 1 — Fig. 1: Illustrations of non-cooperative merging in purely HDV traffic and cooperative merging with CAVs in a bottleneck scenario.

Full Name	Abbreviation
Centralized Training and Decentralized Execution	CTDE
Dual-Interaction-Aware Cooperative Control	DIACC
Decentralized Interaction-Adaptive Decision-Making	D-IADM
Trajectory-Aware Interaction Encoder	TAIE
Proactive Safety-Based Action Refinement	PSAR
Centralized Interaction-Enhanced Critic	C-IEC

TABLE I: Abbreviations.

Panel of Figure 7 — Fig. 6: Bottleneck settings with (a) 25% and (b) 50% lane reductions

Name	Aggressive	Normal	Cautious
D1	0.2	0.6	0.2
D2	0.2	0.4	0.4
D3	0.4	0.4	0.2

TABLE II: HDV Driving Style Distributions.

Panel of Figure 9 — Fig. 7: Integrated training-curve comparison. (a)–(c): baseline and ablation comparison. (d)–(f): softmin temperature $\tau$ selection. (g)–(i): ego reward weight $w_{e}$ selection.

Training Scenario
Scenario	Metric	SUMO (Pure HDV)	SUMO (Mixed Traffic)	MAPPO-IADM	DIACC
N=25, $\rho=0.4$ , D1	Speed (m/s)	$16.08\pm 7.94$	$17.05\pm 7.24$ (+6.0%)	$\bm{20.77\pm 2.88}$ (+29.2%)	$19.04\pm 6.19$ (+18.4%)
	p(WEs)	$11.20\pm 4.66\%$	$8.80\pm 7.76\%$ (-21.4%)	$\bm{0.00\pm 0.00\%}$ (-100.0%)	$4.00\pm 0.00\%$ (-64.3%)
	p(SCEs)	$7.20\pm 3.92\%$	$6.40\pm 9.33\%$ (-11.1%)	$\bm{0.00\pm 0.00\%}$ (-100.0%)	$\bm{0.00\pm 0.00\%}$ (-100.0%)
Zero-shot Testing Scenarios
N=25, $\rho=0.4$ , D2	Speed (m/s)	$14.44\pm 8.74$	$14.54\pm 9.07$ (+0.7%)	$\bm{20.13\pm 4.62}$ (+39.4%)	$19.05\pm 6.61$ (+31.9%)

TABLE III: Performance Comparison in 25% Capacity Reduction Bottleneck Scenario.

Training Scenario
Scenario	Metric	SUMO (Pure HDV)	SUMO (Mixed Traffic)	MAPPO-IADM	DIACC
N=25, $\rho=0.4$ , D1	Speed (m/s)	$10.00\pm 7.30$	$14.10\pm 6.52$ (+41.0%)	$17.47\pm 6.65$ (+74.8%)	$\bm{17.60\pm 7.77}$ (+76.0%)
	p(WEs)	$25.60\pm 9.83\%$	$13.23\pm 7.97\%$ (-48.3%)	$\bm{6.00\pm 2.00\%}$ (-76.6%)	$6.80\pm 5.38\%$ (-73.4%)
	p(SCEs)	$16.40\pm 7.68\%$	$7.60\pm 8.48\%$ (-53.7%)	$6.40\pm 7.84\%$ (-61.0%)	$\bm{0.00\pm 0.00\%}$ (-100.0%)
Zero-shot Testing Scenarios
N=25, $\rho=0.4$ , D2	Speed (m/s)	$10.55\pm 7.71$	$13.21\pm 6.67$ (+25.2%)	$16.47\pm 7.96$ (+56.1%)	$\bm{16.59\pm 8.48}$ (+57.2%)

TABLE IV: Performance Comparison in 50% Capacity Reduction Bottleneck Scenario.

Panel of Figure 12 — Fig. 8: Speed distribution comparison under mixed-traffic scales and capacity-reduction scenarios. (a)–(b) Box plot and KDE of overall speed distributions. (c)–(h) KDE of speed distributions across different capacity-reduction scenarios.

Indexes and Sets
Symbol	Definition
$i,j$	Vehicle indices (e.g., for CAVs/HDVs)
$t$	Time step index
$l$	Lane index
$n$	Number of CAVs ( $\|\mathcal{N}\|$ )
$\mathcal{N}$	Set of CAVs in the mixed traffic

TABLE V: Notations (organized by categories).

Zhengxuan Liu received the B.E. and M.S. degrees from Tianjin University, Tianjin, China in 2016 and 2019, respectively. From 2019 to 2021, she worked as an EEA Engineer at CATARC (Tianjin) Automotive Engineering Research Institute Co.,Ltd., specializing active safety domain. She is currently pursuing the Ph.D. degree in control science and engineering with the School of Electrical and Information Engineering. Her current research interests include cooperative control and security issue for mixed traffic system.

Table 14

Yuxin Cai received B.E. degree from Nanyang Technological University (NTU), Singapore in 2022. Now she is a PhD student at School of Mechanical and Aerospace Engineering, NTU, Singapore. Her main research interests are reinforcement learning, multi-robot cooperation and foundation models.

Table 15

Yijing Wang received her M.S. degree in control theory and control engineering from Yanshan University and the Ph.D. degree in control theory from Peking University, China, in 2000 and 2004, respectively. In 2004, she joined the School of Electrical and Information Engineering, Tianjin University, where she is a full professor. Her research interests are analysis and control of switched/hybrid systems, and robust control.

Table 16

Xiangkun He (Senior Member, IEEE) is currently a UESTC100 Young Professor at the University of Electronic Science and Technology of China. Previously, he was a Research Fellow at Nanyang Technological University, Singapore, and served as a Senior Research Scientist at Huawei Noah’s Ark Lab from 2019 to 2021. He earned his Ph.D. in 2019 from the School of Vehicle and Mobility at Tsinghua University. He has authored over 60 papers and holds 8 granted patents. His research interests include reinforcement learning, trustworthy AI, autonomous vehicles, and robotics. He has received many awards and honors, selectively including the Tsinghua University Outstanding Doctoral Dissertation Award in 2019, Best Paper Finalist at IEEE ICMA 2020, Huawei Major Technological Breakthrough Award in 2021, Best Paper Runner-Up Award at CVCI 2022, and Runner-Up in the Intelligent Algorithm Final of the 2022 Alibaba Global Future Vehicle Challenge. He serves as a reviewer for over 50 renowned journals and conferences.

Table 17

Chen Lv (Senior Member, IEEE) received the Ph.D. degree from the Department of Automotive Engineering, Tsinghua University, China, in 2016. From 2014 to 2015, he was a joint Ph.D. Researcher with the EECS Department, University of California, Berkeley. He is currently an Associate Professor with Nanyang Technology University, Singapore. His research focuses on advanced vehicles and human–machine systems, where he has contributed over 100 papers and received 12 granted patents in China. He received many awards and honors, including the Highly Commended Paper Award of IMechE U.K. in 2012, the Japan NSK Outstanding Mechanical Engineering Paper Award in 2014, the Tsinghua University Outstanding Doctoral Thesis Award in 2016, the IEEE IV Best Workshop/Special Session Paper Award in 2018, the Automotive Innovation Best Paper Award in 2020, the winner of Waymo Open Dataset Challenges at CVPR 2021, and the Machines Young Investigator Award in 2022. He serves as an Associate Editor for IEEE Transactions on Intelligent Transportation Systems, IEEE Transactions on Vehicular Technology, and IEEE Transactions on Intelligent Vehicles; and a Guest Editor for IEEE Intelligent Transportation Systems Magazine, IEEE/ASME Transactions on Mechatronics, and Applied Energy.

Table 18

Zhiqiang Zuo (Senior Member, IEEE) received the M.S. degree in control theory and control engineering in 2001 from Yanshan University and the Ph.D. degree in control theory in 2004 from Peking University, China. In 2004, he joined the School of Electrical and Information Engineering, Tianjin University, where he is a full professor. From 2008 to 2010, he was a Research Fellow in the Department of Mathematics, City University of Hong Kong. From 2013 to 2014, he was a visiting scholar at the University of California, Riverside. His research interests include nonlinear control, robust control, multi-agent systems and cyber-physical systems. Dr. Zuo is an associate editor of the Journal of the Franklin Institute (Elsevier).

Table 19

Intelligent Transportation Systems

Intelligent Transportation Systems apply sensing, communication, AI, and optimization to make transport networks safer and more efficient.

See all 290 papers

COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems
Mar 26, 2026Yifeng Zhang, Jieming Chen, Tingguang Zhou +4Centralized Training with Decentralized Execution Multi-Agent Coordination
Department of Mechanical Engineering, College of Design and Engineering, National University of Singapore, 21 Lower Kent Ridge Rd, 119077, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, 100872, China · School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China
Knowledge-Data-Dual-Driven Reinforcement Learning for Autonomous Vehicle Control in Mixed Traffic
Aug 14, 2026Jie Fang, Wei Zheng, Mengyun Xu +1Hybrid Action Space RL RL for Autonomous Driving
department of Civil Engineering, Fuzhou University, Fuzhou, 350108, China
HONEST-CAV: Hierarchical Optimization of Network Signals and Trajectories for Connected and Automated Vehicles with Multi-Agent Reinforcement Learning
Feb 21, 2026Ziyan Zhang, Changxin Wan, Peng Hao +5Intelligent Transportation Systems Traffic Signal Control
College of Engineering, Center for Environmental Research and Technology, University of California at Riverside, Riverside, CA, 92507. · InfoTech Labs, Toyota Motor North America, Mountain View, CA, 94043.

Mar 26, 2026J/K move · Enter open · S save

COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems

Yifeng Zhang, Jieming Chen, Tingguang Zhou, Tanishq Duhan+3

Department of Mechanical Engineering, College of Design and Engineering, National University of Singapore, 21 Lower Kent Ridge Rd, 119077, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, 100872, China · School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China

Centralized Training with Decentralized Execution Multi-Agent Coordination RL for Autonomous Driving Cooperative MARL Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Multi-Agent Self-Driving (MASD) systems provide an effective solution for coordinating autonomous vehicles to reduce congestion and enhance both safety and operational efficiency in future intelligent transportation systems. Multi-Agent Reinforcement Learning (MARL) has emerged as a promising approach for developing advanced end-to-end MASD systems. However, achieving efficient and safe collaboration in dynamic MASD systems remains a significant challenge in dense scenarios with complex agent interactions. To address this challenge, we propose a novel collaborative(CO-) interaction-aware(-IN) MARL framework, named COIN. Specifically, we develop a new counterfactual individual-global twin delayed deep deterministic policy gradient (CIG-TD3) algorithm, crafted in a "centralized training, decentralized execution" (CTDE) manner, which aims to jointly optimize the individual objectives (navigation) and the global objectives (collaboration) of agents. We further introduce a dual-level interaction-aware centralized critic architecture that captures both local pairwise interactions and global system-level dependencies, enabling more accurate global value estimation and improved credit assignment for collaborative policy learning. We conduct extensive simulation experiments in dense urban traffic environments, which demonstrate that COIN consistently outperforms other advanced baseline methods in both safety and efficiency across various system sizes. These results highlight its superiority in complex and dynamic MASD scenarios, as further validated through real-world robot demonstrations. Supplementary videos are available at https://marmotlab.github.io/COIN/

Abstract

Intelligent Transportation Systems

Explore similar work

COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems

Knowledge-Data-Dual-Driven Reinforcement Learning for Autonomous Vehicle Control in Mixed Traffic

HONEST-CAV: Hierarchical Optimization of Network Signals and Trajectories for Connected and Automated Vehicles with Multi-Agent Reinforcement Learning