Roadside-Cooperative Autonomous Driving: From Data Platform to Vision-Language End-to-End Reasoning

Yitao Xu; Tong Wu; Yiyan Wu; Guoji Xu; Yanbo Jiang; Jiahao Wang; Zehong Ke; Junkai Jiang

cs.ROAug 21, 2026

Roadside-Cooperative Autonomous Driving: From Data Platform to Vision-Language End-to-End Reasoning

Authors: Yitao Xu, Tong Wu, Yiyan Wu, Guoji Xu, Yanbo Jiang, Jiahao Wang, Zehong Ke, Junkai Jiang, +2 more

Organizations: School of Vehicle and Mobility, Tsinghua University · Suzhou Automotive Research Institute, Tsinghua University · State Key Laboratory of Intelligent Vehicle and Mobility, Tsinghua University

Abstract

Vehicle-to-Everything (V2X) cooperation enables beyond-line-of-sight perception, mitigating occlusions in single-vehicle sensing. However, existing V2X benchmarks provide limited support for closed-loop evaluation and language-grounded supervision, hindering the development of vision-language models (VLMs) for end-to-end cooperative driving. To address these limitations, we introduce V2XBench, a simulation platform featuring synchronized ego--roadside sensing and closed-loop evaluation, together with Chat-V2XBench, a progressively structured VQA dataset for cooperative reasoning. Building upon this benchmark infrastructure, we propose AURORA, an end-to-end cooperative driving framework. Equipped with a dual-view perception architecture, AURORA mitigates spatial and semantic discrepancies across ego and roadside viewpoints through a query-level Cross-View Query Alignment and Fusion (CQAF) module. Leveraging the resulting unified tokens, a LoRA-adapted VLM bridges semantic reasoning and generative trajectory planning. Extensive closed-loop evaluations on V2XBench demonstrate that AURORA achieves state-of-the-art performance in heavily occluded scenarios, with a Route Completion rate of 98.21% and a Driving Score of 76.02, while requiring low roadside communication bandwidth. Ultimately, this work pioneers an extensible V2X--VLM paradigm, paving the way for next-generation cooperative autonomous driving.

Figures & tables

Benchmark	Year	Real/ Sim.	View	Scenario Types	Closed- Loop	E2E- Sim	With Traffic Light	With VRUs	With VQA
nuPlan	2021	Real	Single-vehicle	73	✓	$\times$	✓	✓	$\times$
nuScenes	2019	Real	Single-vehicle	-	$\times$	$\times$	$\times$	✓	$\times$
CARLA LB V2	2023	Sim.	Single-vehicle	21	✓	✓	✓	✓	$\times$
Bench2Drive	2024	Sim.	Single-vehicle	44	✓	✓	✓	✓	$\times$
WIBAM	2021	Real	Infrastructure	1	$\times$	$\times$	✓	$\times$	$\times$
V2X-Sim 2.0	2022	Sim.	V2X	-	$\times$	$\times$	$\times$	$\times$	$\times$

Table 1: Comparison with representative public autonomous driving and V2X datasets. V2XBench is the only V2X benchmark to evaluate closed-loop E2E simulation with VQA capabilities.

Figure 1: Platform overview of V2XBench. V2XBench integrates route-adaptive roadside infrastructure, synchronized ego–RSU sensing, offline multi-agent data generation, Chat-V2XBench VQA construction, and online closed-loop cooperative driving evaluation within a unified CARLA-based simulation platform.

Figure 2: The pipeline of our AURORA, a holistic end-to-end framework for V2X cooperative driving. The architecture first utilizes dual-view perception to encode synchronized ego and roadside streams into spatial queries and semantic tokens. Roadside detection and map queries are transmitted and fused in the ego frame by the Cross-View Query Alignment and Fusion (CQAF) module, while ego semantic tokens are refined via query-conditioned cross-attention. The resulting fused tokens ground a VLM-guided generative planner that bridges semantic reasoning and action to roll out future trajectories conditioned on a dedicated waypoint token.

Method	Reference	Modality	Avg. mAP ↑	NDS ↑	Main-class mAP ↑				Trans. Cost (BPS) ↓
Method	Reference	Modality	Avg. mAP ↑	NDS ↑	Vehicle	VRUs	Traffic Light	Traffic Sign	Trans. Cost (BPS) ↓
No Fusion	–	Camera	0.478	0.497	0.513	0.532	0.204	0.301	–
V2X-ViT	ECCV 22	LiDAR	0.350	0.485	0.521	0.042	0.530	0.397	$2.10{\times}10^{8}$
CoDriving	IEEE TPAMI 25	LiDAR	0.393	0.343	0.528	0.627	0.019	0.161	$1.24{\times}10^{8}$
UniV2X	AAAI 25	Camera	0.271	0.382	0.305	0.383	0.131	0.247	$1.09{\times}\mathbf{10^{6}}$
UniMM-V2X	AAAI 26	Camera	0.338	0.466	0.396	0.499	0.056	0.178	$1.09{\times}\mathbf{10^{6}}$

Table 2: Cooperative 3D detection results on the V2XBench test split. All methods are trained and evaluated under the same ego–roadside cooperative setting. For clarity, we report class-wise mAP only for the main categories. BPS denotes bytes per second (B/s) for the roadside transmission cost.

Method	Reference	Modality	Closed-loop Metric				Open-loop Metric	Trans. Cost (BPS) ↓
Method	Reference	Modality	DS ↑	RC ↑	Efficiency ↑	Comfortness ↑	Avg. L2 (m) ↓	Trans. Cost (BPS) ↓
No Fusion	–	Camera	52.09	93.84	15.82	126.12	1.63	–
CoDriving	IEEE TPAMI 25	LiDAR	56.75	91.23	18.59	193.55	1.38	$1.24{\times}10^{8}$
UniV2X	AAAI 25	Camera	33.76	86.76	13.1	128.82	2.19	$4.05{\times}\mathbf{10^{6}}$
UniMM-V2X	AAAI 26	Camera	49.26	87.33	5.2	74.55	1.03	$4.66{\times}\mathbf{10^{6}}$
AURORA (Ours)	–	Camera	76.02	98.21	16.60	124.54	1.60	$4.84{\times}\mathbf{10^{6}}$

Table 3: Planning results on the V2XBench test split under both closed-loop and open-loop evaluation. DS and RC denote Driving Score and Route Completion, respectively. BPS denotes bytes per second (B/s) for the roadside transmission cost.

Setting	Perception		Closed-loop		Open-loop
Setting	mAP ↑	NDS ↑	DS ↑	RC ↑	Avg. L2 (m) ↓
w/o Fusion	0.478	0.497	52.09	93.84	1.63
w/ Fusion (Ours)	0.548	0.568	76.02	98.21	1.60

Table 4: Ablation on ego–roadside fusion. DS and RC denote Driving Score and Route Completion. “w/” and “w/o” denote with and without.

Planner	Closed-loop		Open-loop
Planner	DS ↑	RC ↑	Avg. L2 (m) ↓	Avg. Col. (%) ↓
MLP	56.98	88.38	1.94	5.18
Diffusion	66.21	92.61	1.73	3.91
VAE (Ours)	76.02	98.21	1.60	3.47

Table 5: Ablation on generative planners. DS and RC denote Driving Score and Route Completion.

Setting	Closed-loop		Open-loop
Setting	DS ↑	RC ↑	Avg. L2 (m) ↓	Avg. Col. (%) ↓
w/o VQA	67.98	92.11	1.54	3.57
w/ VQA (Ours)	76.02	98.21	1.60	3.47

Table 6: Ablation on VQA supervision. DS and RC denote Driving Score and Route Completion. “w/” and “w/o” denote with and without.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Sensor	Configuration	Mounting
Ego vehicle
$6\times$ RGB cam.	$1024{\times}512$ , FOV $110^{\circ}$	surround, $z{=}1.8$ m
$6\times$ Depth cam.	$1024{\times}512$ , FOV $110^{\circ}$	paired with RGB
$6\times$ Sem. cam.	$1024{\times}512$ , FOV $110^{\circ}$	paired with RGB
$1\times$ LiDAR	$10$ Hz, $0.6$ M pts/s	roof, $z{=}2.5$ m
BEV semantics	rasterized GT map	—

Appendix

Table 7: Sensor suite of the ego vehicle and each roadside unit (RSU) in V2XBench. All cameras additionally provide spatially aligned depth and (for the ego) semantic-segmentation streams.

Explore similar work

D2-V2X: Depth-Driven Cooperative V2X Reasoning for Autonomous Driving
May 22, 2026Kevin Richard, Alphin Varghese, Colin Pham +2VLMs for Autonomous Driving Cross-Modal Alignment
University of North Carolina at Charlotte
CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models
Aug 7, 2026Hsu-kuang Chiu, Stephen F. SmithAutonomous Driving Benchmarks End-to-End Autonomous Driving
Carnegie Mellon University, Robotics Institute
OmniV2X: A Generative Foundation Planner for Efficient End-to-End Cooperative Driving
Jun 19, 2026Juntong Peng, Juanwu Lu, Yupeng Zhou +3End-to-End Autonomous Driving LLM Planning
College of Engineering, Purdue University, West Lafayette, IN 47907, USA

May 22, 2026, cs.CVJ/K move · Enter open · S save

D2-V2X: Depth-Driven Cooperative V2X Reasoning for Autonomous Driving

Kevin Richard, Alphin Varghese, Colin Pham, David Oh+1

University of North Carolina at Charlotte

VLMs for Autonomous Driving Cross-Modal Alignment Autonomous Driving Perception VLM Reasoning Cooperative Perception 3D Visual Grounding

Open paper AskPDF arXiv

Single-vehicle Vision-Language Models (VLMs) are fundamentally constrained by sensor occlusions. While Vehicle-to-Everything (V2X) systems mitigate this, current benchmarks lack the cooperative reasoning required for resolving ambiguities in complex environments. We introduce D2-V2X, a spatially-aware Question-Rationale-Answer (QRA) benchmark featuring 8,500 triplets derived from multimodal vehicle and infrastructure sensors. We additionally establish a baseline that aligns 3D LiDAR features with the VLM's latent space. By enforcing natural language Chain-of-Thought rationales prior to structured JSON outputs, our model is forced to explicitly articulate spatial relations. Our experiments demonstrate that grounding VLMs in cooperative LiDAR achieves 24.4% recall in identifying occluded hazards compared to near-zero in zero-shot models and reduces spatial estimation error for visible objects by 77% compared to the zero-shot baseline. While the model achieves a functional decision-making F1-score of 53.5, we identify 3D-to-2D projection as a fundamental bottleneck in current VLM architectures, establishing a new baseline for future innovation. Data, code, and trained models available at https://github.com/KevinRichard1/D2-V2X

Stage	Batch Size	Epochs	Base LR	Freeze Backbone
Stage 1	12	4	$5{\times}10^{-5}$	False
Stage 2	12	4	$5{\times}10^{-5}$	False
Stage 3	8	4	$4{\times}10^{-5}$	False
Stage 4	8	6	$8{\times}10^{-5}$	True
Stage 5	6	4	$8{\times}10^{-5}$	True

Abstract

Explore similar work

D2-V2X: Depth-Driven Cooperative V2X Reasoning for Autonomous Driving

CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models

OmniV2X: A Generative Foundation Planner for Efficient End-to-End Cooperative Driving