SCORP: Scene-Consistent Multi-agent Diffusion Planning with Stable Online Reinforcement Post-Training for Cooperative Driving
Authors: Haojie Bai, Aimin Li, Ruoyu Yao, Xiongwei Zhao, Tingting Zhang, Xing Zhang, Lin Gao, and Jun Ma
Organizations: the School of Information Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518071, China · the Middle East Technology University (METU), Ankara, 06800, Turkiye · the Robotics and Autonomous Systems Thrust, Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · the School of Computer Science and Technology, Qinghai University, Xining, 810016, China
Cooperative driving is a safety- and efficiency-critical task that requires the coordination of diverse, interaction-realistic multi-agent trajectories. Although existing diffusion-based methods can capture multimodal behaviors from demonstrations, they often exhibit weak scene consistency and poor alignment with closed-loop cooperative objectives. This makes post-training necessary for further improvement, yet achieving stable online post-training in reactive multi-agent environments remains challenging. In this paper, we propose SCORP, a scene-consistent multi-agent diffusion planner with stable online reinforcement learning (RL) post-training for cooperative driving. For pre-training, we develop a scene-conditioned multi-agent denoising architecture that couples inter-agent self-attention with a dual-path conditioning mechanism: cross-attention provides direct scene-information injection, while AdaLN-Zero enables additional flexible and stable conditional modulation, thereby improving the scene consistency and road adherence of joint trajectories. For post-training, we formulate a two-layer Markov decision process (MDP) that explicitly integrates the reverse denoising chain with policy-environment interaction. We further co-design dense, well-shaped planning rewards and variance-gated group-relative policy optimization (VG-GRPO) to mitigate advantage collapse and gradient instability during closed-loop training. Extensive experiments show that SCORP outperforms strong open-source baselines on WOMD, with 10.47%-28.26% and 1.70%-7.22% improvements in core safety and efficiency metrics, respectively. Moreover, compared with alternative post-training methods, SCORP delivers significant and consistent gains in both driving safety and traffic efficiency, highlighting stable and sustained advances in closed-loop cooperative driving.
Figures & tables
Fig. 1: An illustration of a vehicle-cloud cooperative traffic scenario. A central cloud planner processes scene contexts and generates coordinated multi-agent trajectories using the proposed SCORP framework. Planned trajectories are transmitted to CAVs for execution.
Fig. 2: Architecture of the multi-agent diffusion planner. A symmetric scene encoder models scene elements and their relations in local coordinates through query-centric self-attention. A denoising decoder predicts joint future plans via inter-agent self-attention, scene-conditioned generation, and AdaLN-Zero modulation.
Fig. 3: Two-layer MDP formulation for reinforcement learning. We construct a two-layer MDP consisting of an inner denoising MDP and an outer environment-interaction MDP, and use Gaussian likelihoods at each denoising step so that the denoising process can be optimized via policy gradients.
Fig. 4: Online RL post-training framework. The pipeline consists of three components: (1) closed-loop policy rollout, where the policy samples multimodal multi-agent trajectories conditioned on the updated scene context; (2) rule-based reward evaluation, where each trajectory group is scored based on safety and efficiency metrics to compute variance-gated group-relative advantages; and (3) policy training, where mini-batches are randomly sampled from the rollout buffer to optimize the actor network with policy loss and KL regularization.
Method
Primary closed-loop objectives
Secondary diagnostics
CR (%) ↓
OR (%) ↓
AS (m/s) ↑
ADE (m) ↓
Kin (%) ↓
TrafficBotsV1.5 [ 45 ]
2.74 ± 0.21
1.79 ± 0.14
8.03 ± 0.48
1.68 ± 0.09
0.26 ± 0.02
SMART-large [ 27 ]
2.22 ± 0.09
1.58 ± 0.10
8.34 ± 0.30
1.30 ± 0.01
0.21 ± 0.01
VBD [ 8 ]
2.46 ± 0.14
1.92 ± 0.18
8.08 ± 0.52
1.41 ± 0.02
0.24 ± 0.01
SMART-tiny-CLSFT [ 46 ]
2.10 ± 0.10
1.53 ± 0.12
8.47 ± 0.44
1.23 ± 0.03
0.25 ± 0.02
SCORP
1.89 ± 0.12
1.36 ± 0.08
8.61 ± 0.46
1.36 ± 0.04
0.32 ± 0.03
TABLE I: Closed-loop benchmark results on the WOMD testing interactive split (primary: CR, OR, AS; secondary: ADE, Kin)
Fig. 5: Closed-loop planning visualizations in real traffic scenarios, showing interactive trajectories. We perform closed-loop simulation over an 8-second horizon using a viewpoint centered on Vehicle 1. The 1-second history trajectory is marked behind the vehicles.
Fig. 6: Visualization of diverse interaction behaviors under repeated inference in the same scenario, where Vehicles 0 and 1 exhibit different interaction outcomes: (a) Vehicle 1 goes first; (b) Vehicle 1 yields.
Fig. 7: Qualitative comparison between pre-training and RL post-training over an 8-second closed-loop rollout. (a) Pretrained planner. (b) Planner after 1M RL post-training steps. (c) Planner after 10M RL post-training steps. Safety and efficiency steadily improve as post-training progresses.
Setting
CR (%) ↓
OR (%) ↓
AS (m/s) ↑
ADE (m) ↓
Kin (%) ↓
w/o AdaLN-Zero
2.11
2.05
8.40
1.30
0.26
with AdaLN-Zero
2.04
1.68
8.36
1.28
0.25
TABLE III: Ablation on AdaLN-Zero module
Fig. 8: Qualitative comparison with and without the AdaLN-Zero module
Gating std1 / std2
Collapse step
CR [%]↓
OR [%]↓
AS [m/s]↑
ADE [m]↓
Kin [%]↓
w/o gating
≈ 0.5M
2.15
2.03
8.05
1.74
0.40
0.03/0.06
–
1.89
1.36
8.61
1.36
0.32
0.00/0.06
≈ 2.0M
2.01
1.57
8.42
1.42
0.30
0.03/0.09
–
1.96
1.50
8.49
1.30
0.29
TABLE IV: Ablation of the variance-gated mechanism (performance is evaluated at the last checkpoint before collapse)
Dataset type
CR (%) ↓
OR (%) ↓
AS (m/s) ↑
ADE (m) ↓
Kin (%) ↓
High-score
1.95
1.34
8.53
1.30
0.31
Low-score
2.18
1.99
8.49
1.40
0.37
Full
1.89
1.36
8.61
1.36
0.32
TABLE V: Ablation on post-training data distribution
High-level autonomous driving requires motion planners capable of modeling multimodal future uncertainties while remaining robust in closed-loop interactions. Although diffusion-based planners are effective at modeling complex trajectory distributions, they often suffer from stochastic instabilities and the lack of corrective negative feedback when trained purely with imitation learning. To address these issues, we propose RAD-2, a unified generator-discriminator framework for closed-loop planning. Specifically, a diffusion-based generator is used to produce diverse trajectory candidates, while an RL-optimized discriminator reranks these candidates according to their long-term driving quality. This decoupled design avoids directly applying sparse scalar rewards to the full high-dimensional trajectory space, thereby improving optimization stability. To further enhance reinforcement learning, we introduce Temporally Consistent Group Relative Policy Optimization, which exploits temporal coherence to alleviate the credit assignment problem. In addition, we propose On-policy Generator Optimization, which converts closed-loop feedback into structured longitudinal optimization signals and progressively shifts the generator toward high-reward trajectory manifolds. To support efficient large-scale training, we introduce BEV-Warp, a high-throughput simulation environment that performs closed-loop evaluation directly in Bird's-Eye View feature space via spatial warping. RAD-2 reduces the collision rate by 56% compared with strong diffusion-based planners. Real-world deployment further demonstrates improved perceived safety and driving smoothness in complex urban traffic.
Hao Gao, Shaoyu Chen, Yifan Zhu +4
Huazhong University of Science & Technology · Horizon Robotics
Collaborative driving systems leverage vehicle-to-everything (V2X) communication for multi-agent collaborative perception to enhance driving safety, yet they remain constrained by scarce annotated real-world V2X driving datasets and limited generalization across diverse driving conditions. While image generation technology offers a feasible solution for data augmentation, existing methods tailored for single-vehicle multi-view scenarios face two fundamental challenges in multi-agent driving settings: (1) the expansion of the learning objective degrades generation quality, and (2) the highly dynamic variations across agents hinder the modeling of consistency for physical attributes (e.g., color, category) in jointly observed objects. To bridge this gap, we propose V2XCrafter, the first framework for generating controllable and realistic collaborative driving scene across agents' camera views. For effective learning, we develop a progressive multi-agent diffusion model based on a single-agent backbone, using neighboring agents' latent states as reference signals to progressively guide the single-to-multi diffusion. To address cross-vehicle inconsistency, we propose a cross-agent attention module that leverages a collaboration view graph and learnable jointly observed object representation to model the dynamic cross-agent camera view relationships. Experiments have shown that V2XCrafter can generate high-fidelity and controllable street views with consistency across agents, thereby effectively enhancing the downstream collaborative 3D object detection tasks.
Yihang Tao, Yu Guo, Senkang Hu +4
Hong Kong JC STEM Lab of Smart City · City University of Hong Kong · Lingnan University
Multi-Agent Self-Driving (MASD) systems provide an effective solution for coordinating autonomous vehicles to reduce congestion and enhance both safety and operational efficiency in future intelligent transportation systems. Multi-Agent Reinforcement Learning (MARL) has emerged as a promising approach for developing advanced end-to-end MASD systems. However, achieving efficient and safe collaboration in dynamic MASD systems remains a significant challenge in dense scenarios with complex agent interactions. To address this challenge, we propose a novel collaborative(CO-) interaction-aware(-IN) MARL framework, named COIN. Specifically, we develop a new counterfactual individual-global twin delayed deep deterministic policy gradient (CIG-TD3) algorithm, crafted in a "centralized training, decentralized execution" (CTDE) manner, which aims to jointly optimize the individual objectives (navigation) and the global objectives (collaboration) of agents. We further introduce a dual-level interaction-aware centralized critic architecture that captures both local pairwise interactions and global system-level dependencies, enabling more accurate global value estimation and improved credit assignment for collaborative policy learning. We conduct extensive simulation experiments in dense urban traffic environments, which demonstrate that COIN consistently outperforms other advanced baseline methods in both safety and efficiency across various system sizes. These results highlight its superiority in complex and dynamic MASD scenarios, as further validated through real-world robot demonstrations. Supplementary videos are available at https://marmotlab.github.io/COIN/
Yifeng Zhang, Jieming Chen, Tingguang Zhou +4
Department of Mechanical Engineering, College of Design and Engineering, National University of Singapore, 21 Lower Kent Ridge Rd, 119077, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, 100872, China · School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China