SCORP: Scene-Consistent Multi-agent Diffusion Planning with Stable Online Reinforcement Post-Training for Cooperative Driving

Haojie Bai; Aimin Li; Ruoyu Yao; Xiongwei Zhao; Tingting Zhang; Xing Zhang; Lin Gao; and Jun Ma

cs.ROApr 13, 2026

SCORP: Scene-Consistent Multi-agent Diffusion Planning with Stable Online Reinforcement Post-Training for Cooperative Driving

Authors: Haojie Bai, Aimin Li, Ruoyu Yao, Xiongwei Zhao, Tingting Zhang, Xing Zhang, Lin Gao, and Jun Ma

Organizations: the School of Information Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518071, China · the Middle East Technology University (METU), Ankara, 06800, Turkiye · the Robotics and Autonomous Systems Thrust, Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · the School of Computer Science and Technology, Qinghai University, Xining, 810016, China

Abstract

Cooperative driving is a safety- and efficiency-critical task that requires the coordination of diverse, interaction-realistic multi-agent trajectories. Although existing diffusion-based methods can capture multimodal behaviors from demonstrations, they often exhibit weak scene consistency and poor alignment with closed-loop cooperative objectives. This makes post-training necessary for further improvement, yet achieving stable online post-training in reactive multi-agent environments remains challenging. In this paper, we propose SCORP, a scene-consistent multi-agent diffusion planner with stable online reinforcement learning (RL) post-training for cooperative driving. For pre-training, we develop a scene-conditioned multi-agent denoising architecture that couples inter-agent self-attention with a dual-path conditioning mechanism: cross-attention provides direct scene-information injection, while AdaLN-Zero enables additional flexible and stable conditional modulation, thereby improving the scene consistency and road adherence of joint trajectories. For post-training, we formulate a two-layer Markov decision process (MDP) that explicitly integrates the reverse denoising chain with policy-environment interaction. We further co-design dense, well-shaped planning rewards and variance-gated group-relative policy optimization (VG-GRPO) to mitigate advantage collapse and gradient instability during closed-loop training. Extensive experiments show that SCORP outperforms strong open-source baselines on WOMD, with 10.47%-28.26% and 1.70%-7.22% improvements in core safety and efficiency metrics, respectively. Moreover, compared with alternative post-training methods, SCORP delivers significant and consistent gains in both driving safety and traffic efficiency, highlighting stable and sustained advances in closed-loop cooperative driving.

Figures & tables

Fig. 1: An illustration of a vehicle-cloud cooperative traffic scenario. A central cloud planner processes scene contexts and generates coordinated multi-agent trajectories using the proposed SCORP framework. Planned trajectories are transmitted to CAVs for execution.

Fig. 2: Architecture of the multi-agent diffusion planner. A symmetric scene encoder models scene elements and their relations in local coordinates through query-centric self-attention. A denoising decoder predicts joint future plans via inter-agent self-attention, scene-conditioned generation, and AdaLN-Zero modulation.

Fig. 3: Two-layer MDP formulation for reinforcement learning. We construct a two-layer MDP consisting of an inner denoising MDP and an outer environment-interaction MDP, and use Gaussian likelihoods at each denoising step so that the denoising process can be optimized via policy gradients.

Fig. 4: Online RL post-training framework. The pipeline consists of three components: (1) closed-loop policy rollout, where the policy samples multimodal multi-agent trajectories conditioned on the updated scene context; (2) rule-based reward evaluation, where each trajectory group is scored based on safety and efficiency metrics to compute variance-gated group-relative advantages; and (3) policy training, where mini-batches are randomly sampled from the rollout buffer to optimize the actor network with policy loss and KL regularization.

Method	Primary closed-loop objectives			Secondary diagnostics
Method	CR (%) $\downarrow$	OR (%) $\downarrow$	AS (m/s) $\uparrow$	ADE (m) $\downarrow$	Kin (%) $\downarrow$
TrafficBotsV1.5 [ 45 ]	2.74 $\pm$ 0.21	1.79 $\pm$ 0.14	8.03 $\pm$ 0.48	1.68 $\pm$ 0.09	0.26 $\pm$ 0.02
SMART-large [ 27 ]	2.22 $\pm$ 0.09	1.58 $\pm$ 0.10	8.34 $\pm$ 0.30	1.30 $\pm$ 0.01	0.21 $\pm$ 0.01
VBD [ 8 ]	2.46 $\pm$ 0.14	1.92 $\pm$ 0.18	8.08 $\pm$ 0.52	1.41 $\pm$ 0.02	0.24 $\pm$ 0.01
SMART-tiny-CLSFT [ 46 ]	2.10 $\pm$ 0.10	1.53 $\pm$ 0.12	8.47 $\pm$ 0.44	1.23 $\pm$ 0.03	0.25 $\pm$ 0.02
SCORP	1.89 $\pm$ 0.12	1.36 $\pm$ 0.08	8.61 $\pm$ 0.46	1.36 $\pm$ 0.04	0.32 $\pm$ 0.03

TABLE I: Closed-loop benchmark results on the WOMD testing interactive split (primary: CR, OR, AS; secondary: ADE, Kin)

Method	Primary closed-loop objectives			Secondary diagnostics
Method	CR (%) $\downarrow$	OR (%) $\downarrow$	AS (m/s) $\uparrow$	ADE (m) $\downarrow$	Kin (%) $\downarrow$
Pre-trained only	2.04 $\pm$ 0.11	1.68 $\pm$ 0.10	8.36 $\pm$ 0.42	1.28 $\pm$ 0.02	0.25 $\pm$ 0.02
SFT	2.01 $\pm$ 0.07	1.64 $\pm$ 0.06	8.37 $\pm$ 0.36	1.15 $\pm$ 0.015	0.25 $\pm$ 0.01
DPO	1.97 $\pm$ 0.13	1.58 $\pm$ 0.09	8.15 $\pm$ 0.39	1.33 $\pm$ 0.04	0.27 $\pm$ 0.01
Offline RL	2.18 $\pm$ 0.09	1.82 $\pm$ 0.14	8.98 $\pm$ 0.68	1.37 $\pm$ 0.05	0.26 $\pm$ 0.02
SCORP (Online RL)	1.89 $\pm$ 0.12	1.36 $\pm$ 0.08	8.61 $\pm$ 0.46	1.36 $\pm$ 0.04	0.32 $\pm$ 0.03

TABLE II: Closed-loop comparison of post-training methods (primary: CR, OR, AS; secondary: ADE, Kin)

Fig. 5: Closed-loop planning visualizations in real traffic scenarios, showing interactive trajectories. We perform closed-loop simulation over an 8-second horizon using a viewpoint centered on Vehicle 1. The 1-second history trajectory is marked behind the vehicles.

Fig. 6: Visualization of diverse interaction behaviors under repeated inference in the same scenario, where Vehicles 0 and 1 exhibit different interaction outcomes: (a) Vehicle 1 goes first; (b) Vehicle 1 yields.

Fig. 7: Qualitative comparison between pre-training and RL post-training over an 8-second closed-loop rollout. (a) Pretrained planner. (b) Planner after 1M RL post-training steps. (c) Planner after 10M RL post-training steps. Safety and efficiency steadily improve as post-training progresses.

Setting	CR (%) $\downarrow$	OR (%) $\downarrow$	AS (m/s) $\uparrow$	ADE (m) $\downarrow$	Kin (%) $\downarrow$
w/o AdaLN-Zero	2.11	2.05	8.40	1.30	0.26
with AdaLN-Zero	2.04	1.68	8.36	1.28	0.25

TABLE III: Ablation on AdaLN-Zero module

Panel of Figure 11 — Fig. 8: Qualitative comparison with and without the AdaLN-Zero module

Gating $\text{std}_{1}$ / $\text{std}_{2}$	Collapse step	CR $[\%]$ $\downarrow$	OR $[\%]$ $\downarrow$	AS $[m/s]$ $\uparrow$	ADE $[m]$ $\downarrow$	Kin $[\%]$ $\downarrow$
w/o gating	$\approx$ 0.5M	2.15	2.03	8.05	1.74	0.40
0.03/0.06	–	1.89	1.36	8.61	1.36	0.32
0.00/0.06	$\approx$ 2.0M	2.01	1.57	8.42	1.42	0.30
0.03/0.09	–	1.96	1.50	8.49	1.30	0.29

TABLE IV: Ablation of the variance-gated mechanism (performance is evaluated at the last checkpoint before collapse)

Dataset type	CR (%) $\downarrow$	OR (%) $\downarrow$	AS (m/s) $\uparrow$	ADE (m) $\downarrow$	Kin (%) $\downarrow$
High-score	1.95	1.34	8.53	1.30	0.31
Low-score	2.18	1.99	8.49	1.40	0.37
Full	1.89	1.36	8.61	1.36	0.32

TABLE V: Ablation on post-training data distribution

Autonomous Driving Planning

Autonomous driving planning selects safe, feasible vehicle actions and trajectories amid traffic, uncertainty, road rules, and changing goals.

See all 384 papers

RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework
Apr 16, 2026Hao Gao, Shaoyu Chen, Yifan Zhu +4Diffusion-Based Planning RL for Diffusion Models
Huazhong University of Science & Technology · Horizon Robotics
V2XCrafter: Learning to Generate Driving Scene Across Agents
May 28, 2026Yihang Tao, Yu Guo, Senkang Hu +4Autonomous Driving Synthetic Data Generation for Robotics
Hong Kong JC STEM Lab of Smart City · City University of Hong Kong · Lingnan University
COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems
Mar 26, 2026Yifeng Zhang, Jieming Chen, Tingguang Zhou +4Centralized Training with Decentralized Execution Multi-Agent Coordination
Department of Mechanical Engineering, College of Design and Engineering, National University of Singapore, 21 Lower Kent Ridge Rd, 119077, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, 100872, China · School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China

Apr 16, 2026J/K move · Enter open · S save

RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework

Hao Gao, Shaoyu Chen, Yifan Zhu, Yuehao Song+3

Huazhong University of Science & Technology · Horizon Robotics

Diffusion-Based Planning RL for Diffusion Models Autonomous Driving Planning RL for Autonomous Driving Autonomous Driving Simulation

Open paper AskPDF arXiv

High-level autonomous driving requires motion planners capable of modeling multimodal future uncertainties while remaining robust in closed-loop interactions. Although diffusion-based planners are effective at modeling complex trajectory distributions, they often suffer from stochastic instabilities and the lack of corrective negative feedback when trained purely with imitation learning. To address these issues, we propose RAD-2, a unified generator-discriminator framework for closed-loop planning. Specifically, a diffusion-based generator is used to produce diverse trajectory candidates, while an RL-optimized discriminator reranks these candidates according to their long-term driving quality. This decoupled design avoids directly applying sparse scalar rewards to the full high-dimensional trajectory space, thereby improving optimization stability. To further enhance reinforcement learning, we introduce Temporally Consistent Group Relative Policy Optimization, which exploits temporal coherence to alleviate the credit assignment problem. In addition, we propose On-policy Generator Optimization, which converts closed-loop feedback into structured longitudinal optimization signals and progressively shifts the generator toward high-reward trajectory manifolds. To support efficient large-scale training, we introduce BEV-Warp, a high-throughput simulation environment that performs closed-loop evaluation directly in Bird's-Eye View feature space via spatial warping. RAD-2 reduces the collision rate by 56% compared with strong diffusion-based planners. Real-world deployment further demonstrates improved perceived safety and driving smoothness in complex urban traffic.

Abstract

Autonomous Driving Planning

Explore similar work

RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework

V2XCrafter: Learning to Generate Driving Scene Across Agents

COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems