cs.ROAug 21, 2026

Roadside-Cooperative Autonomous Driving: From Data Platform to Vision-Language End-to-End Reasoning

Authors: Yitao Xu, Tong Wu, Yiyan Wu, Guoji Xu, Yanbo Jiang, Jiahao Wang, Zehong Ke, Junkai Jiang, +2 more

Organizations: School of Vehicle and Mobility, Tsinghua University · Suzhou Automotive Research Institute, Tsinghua University · State Key Laboratory of Intelligent Vehicle and Mobility, Tsinghua University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vehicle-to-Everything (V2X) cooperation enables beyond-line-of-sight perception, mitigating occlusions in single-vehicle sensing. However, existing V2X benchmarks provide limited support for closed-loop evaluation and language-grounded supervision, hindering the development of vision-language models (VLMs) for end-to-end cooperative driving. To address these limitations, we introduce V2XBench, a simulation platform featuring synchronized ego--roadside sensing and closed-loop evaluation, together with Chat-V2XBench, a progressively structured VQA dataset for cooperative reasoning. Building upon this benchmark infrastructure, we propose AURORA, an end-to-end cooperative driving framework. Equipped with a dual-view perception architecture, AURORA mitigates spatial and semantic discrepancies across ego and roadside viewpoints through a query-level Cross-View Query Alignment and Fusion (CQAF) module. Leveraging the resulting unified tokens, a LoRA-adapted VLM bridges semantic reasoning and generative trajectory planning. Extensive closed-loop evaluations on V2XBench demonstrate that AURORA achieves state-of-the-art performance in heavily occluded scenarios, with a Route Completion rate of 98.21% and a Driving Score of 76.02, while requiring low roadside communication bandwidth. Ultimately, this work pioneers an extensible V2X--VLM paradigm, paving the way for next-generation cooperative autonomous driving.

Figures & tables

BenchmarkYearReal/ Sim.ViewScenario TypesClosed- LoopE2E- SimWith Traffic LightWith VRUsWith VQA
nuPlan2021RealSingle-vehicle73✓×\times✓✓×\times
nuScenes2019RealSingle-vehicle-×\times×\times×\times✓×\times
CARLA LB V22023Sim.Single-vehicle21✓✓✓✓×\times
Bench2Drive2024Sim.Single-vehicle44✓✓✓✓×\times
WIBAM2021RealInfrastructure1×\times×\times✓×\times×\times
V2X-Sim 2.02022Sim.V2X-×\times×\times×\times×\times×\times
Table 1: Comparison with representative public autonomous driving and V2X datasets. V2XBench is the only V2X benchmark to evaluate closed-loop E2E simulation with VQA capabilities.
Figure 1: Platform overview of V2XBench. V2XBench integrates route-adaptive roadside infrastructure, synchronized ego–RSU sensing, offline multi-agent data generation, Chat-V2XBench VQA construction, and online closed-loop cooperative driving evaluation within a unified CARLA-based simulation platform.
Figure 2: The pipeline of our AURORA, a holistic end-to-end framework for V2X cooperative driving. The architecture first utilizes dual-view perception to encode synchronized ego and roadside streams into spatial queries and semantic tokens. Roadside detection and map queries are transmitted and fused in the ego frame by the Cross-View Query Alignment and Fusion (CQAF) module, while ego semantic tokens are refined via query-conditioned cross-attention. The resulting fused tokens ground a VLM-guided generative planner that bridges semantic reasoning and action to roll out future trajectories conditioned on a dedicated waypoint token.
MethodReferenceModalityAvg. mAP ↑NDS ↑Main-class mAP ↑Trans. Cost (BPS) ↓
VehicleVRUsTraffic LightTraffic Sign
No Fusion–Camera0.4780.4970.5130.5320.2040.301–
V2X-ViTECCV 22LiDAR0.3500.4850.5210.0420.5300.3972.10×1082.10{\times}10^{8}
CoDrivingIEEE TPAMI 25LiDAR0.3930.3430.5280.6270.0190.1611.24×1081.24{\times}10^{8}
UniV2XAAAI 25Camera0.2710.3820.3050.3830.1310.2471.09×1061.09{\times}\mathbf{10^{6}}
UniMM-V2XAAAI 26Camera0.3380.4660.3960.4990.0560.1781.09×1061.09{\times}\mathbf{10^{6}}
Table 2: Cooperative 3D detection results on the V2XBench test split. All methods are trained and evaluated under the same ego–roadside cooperative setting. For clarity, we report class-wise mAP only for the main categories. BPS denotes bytes per second (B/s) for the roadside transmission cost.
MethodReferenceModalityClosed-loop MetricOpen-loop MetricTrans. Cost (BPS) ↓
DS ↑RC ↑Efficiency ↑Comfortness ↑Avg. L2 (m) ↓
No Fusion–Camera52.0993.8415.82126.121.63–
CoDrivingIEEE TPAMI 25LiDAR56.7591.2318.59193.551.381.24×1081.24{\times}10^{8}
UniV2XAAAI 25Camera33.7686.7613.1128.822.194.05×1064.05{\times}\mathbf{10^{6}}
UniMM-V2XAAAI 26Camera49.2687.335.274.551.034.66×1064.66{\times}\mathbf{10^{6}}
AURORA (Ours)–Camera76.0298.2116.60124.541.604.84×1064.84{\times}\mathbf{10^{6}}
Table 3: Planning results on the V2XBench test split under both closed-loop and open-loop evaluation. DS and RC denote Driving Score and Route Completion, respectively. BPS denotes bytes per second (B/s) for the roadside transmission cost.
SettingPerceptionClosed-loopOpen-loop
mAP ↑NDS ↑DS ↑RC ↑Avg. L2 (m) ↓
w/o Fusion0.4780.49752.0993.841.63
w/ Fusion (Ours)0.5480.56876.0298.211.60
Table 4: Ablation on ego–roadside fusion. DS and RC denote Driving Score and Route Completion. “w/” and “w/o” denote with and without.
PlannerClosed-loopOpen-loop
DS ↑RC ↑Avg. L2 (m) ↓Avg. Col. (%) ↓
MLP56.9888.381.945.18
Diffusion66.2192.611.733.91
VAE (Ours)76.0298.211.603.47
Table 5: Ablation on generative planners. DS and RC denote Driving Score and Route Completion.
SettingClosed-loopOpen-loop
DS ↑RC ↑Avg. L2 (m) ↓Avg. Col. (%) ↓
w/o VQA67.9892.111.543.57
w/ VQA (Ours)76.0298.211.603.47
Table 6: Ablation on VQA supervision. DS and RC denote Driving Score and Route Completion. “w/” and “w/o” denote with and without.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

SensorConfigurationMounting
Ego vehicle
6×6\times RGB cam.1024×5121024{\times}512 , FOV 110∘110^{\circ}surround, z=1.8z{=}1.8 m
6×6\times Depth cam.1024×5121024{\times}512 , FOV 110∘110^{\circ}paired with RGB
6×6\times Sem. cam.1024×5121024{\times}512 , FOV 110∘110^{\circ}paired with RGB
1×1\times LiDAR1010 Hz, 0.60.6 M pts/sroof, z=2.5z{=}2.5 m
BEV semanticsrasterized GT map—
Appendix
Table 7: Sensor suite of the ego vehicle and each roadside unit (RSU) in V2XBench. All cameras additionally provide spatially aligned depth and (for the ego) semantic-segmentation streams.
CategoryScenario Types
V2X scenariosCrossingBicycleFlow, DynamicObjectCrossing, NonSignalizedJunctionLeftTurn, NonSignalizedJunctionRightTurn, ParkingCrossingPedestrian, PedestrianCrossing, VehicleTurningRoute, VehicleTurningRoutePedestrian
Other scenariosAccident, AccidentTwoWays, BlockedIntersection, ConstructionObstacle, ConstructionObstacleTwoWays, ControlLoss, EnterActorFlow, EnterActorFlowV2, HardBreakRoute, HazardAtSideLane, HazardAtSideLaneTwoWays, HighwayCutIn, HighwayExit, InterurbanActorFlow, InterurbanAdvancedActorFlow, InvadingTurn, MergerIntoSlowTraffic, MergerIntoSlowTrafficV2, OppositeVehicleRunningRedLight, OppositeVehicleTakingPriority, ParkedObstacle, ParkedObstacleTwoWays, ParkingCutIn, ParkingExit, PriorityAtJunction, SignalizedJunctionLeftTurn, SignalizedJunctionRightTurn, StaticCutIn, VehicleOpensDoorTwoWays, YieldToEmergencyVehicle
Appendix
Table 8: The 3838 safety-critical scenario types in V2XBench, grouped into V2X scenarios (occlusion- and beyond-line-of-sight-dominated, where roadside infrastructure is most beneficial) and other challenging autonomous-driving scenarios.
Figure 3: Representative visualizations of synchronized V2XBench sensory data, including RGB images, spatially aligned depth maps, and LiDAR point clouds captured from ego and roadside views.
Figure 4: The Chat-V2XBench question bank. Each keyframe is annotated with questions drawn from the four levels of the progressive reasoning hierarchy.
Figure 5: The prompt used to auto-label the open-ended QA pairs with Qwen3-VL-Plus, conditioning the model on the multi-view imagery and the ground-truth object hints.
StageBatch SizeEpochsBase LRFreeze Backbone
Stage 11245×10−55{\times}10^{-5}False
Stage 21245×10−55{\times}10^{-5}False
Stage 3844×10−54{\times}10^{-5}False
Stage 4868×10−58{\times}10^{-5}True
Stage 5648×10−58{\times}10^{-5}True
Appendix
Table 9: Stage-wise training hyperparameters for the submitted AURORA checkpoint. Shared optimizer and schedule settings are described in the text.

Explore similar work

CardsList
May 22, 2026cs.CV

D2-V2X: Depth-Driven Cooperative V2X Reasoning for Autonomous Driving

Single-vehicle Vision-Language Models (VLMs) are fundamentally constrained by sensor occlusions. While Vehicle-to-Everything (V2X) systems mitigate this, current benchmarks lack the cooperative reasoning required for resolving ambiguities in complex environments. We introduce D2-V2X, a spatially-aware Question-Rationale-Answer (QRA) benchmark featuring 8,500 triplets derived from multimodal vehicle and infrastructure sensors. We additionally establish a baseline that aligns 3D LiDAR features with the VLM's latent space. By enforcing natural language Chain-of-Thought rationales prior to structured JSON outputs, our model is forced to explicitly articulate spatial relations. Our experiments demonstrate that grounding VLMs in cooperative LiDAR achieves 24.4% recall in identifying occluded hazards compared to near-zero in zero-shot models and reduces spatial estimation error for visible objects by 77% compared to the zero-shot baseline. While the model achieves a functional decision-making F1-score of 53.5, we identify 3D-to-2D projection as a fundamental bottleneck in current VLM architectures, establishing a new baseline for future innovation. Data, code, and trained models available at https://github.com/KevinRichard1/D2-V2X

Kevin Richard, Alphin Varghese, Colin Pham +2

University of North Carolina at Charlotte

VLMs for Autonomous DrivingCross-Modal AlignmentAutonomous Driving PerceptionVLM ReasoningCooperative Perception3D Visual Grounding
AskPDF
Aug 7, 2026cs.AI

CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models

Vision-Language-Action (VLA) models have recently achieved impressive performance for end-to-end autonomous driving, yet existing approaches are primarily designed for an individual single autonomous driving agent with limited support for cooperative perception, reasoning, and planning. We present Cooperative Multi-agent Unified Driving with Reasoning (CMU-Drive), a closed-loop end-to-end benchmark for evaluating cooperative autonomous driving with multiple connected autonomous vehicles (CAVs) operating in safety-critical driving scenarios with background traffic participants. We further propose Vehicle-to-Vehicle Vision-Language-Action (V2V-VLA), a cooperative VLA model that integrates cooperative driving into a single forward pass by jointly generating driving actions, future waypoints, language reasoning, and communication policies. Experiments on CMU-Drive establish the first benchmark and baseline for cooperative VLA driving and provide a foundation for future research on multi-agent, closed-loop, end-to-end cooperative autonomous driving. Our code, benchmark, and model checkpoint will be publicly released to facilitate open-source research.

Hsu-kuang Chiu, Stephen F. Smith

Carnegie Mellon University, Robotics Institute

Autonomous Driving BenchmarksEnd-to-End Autonomous DrivingAutonomous DrivingVision-Language-Action Models
AskPDF
Jun 19, 2026cs.RO

OmniV2X: A Generative Foundation Planner for Efficient End-to-End Cooperative Driving

We present OmniV2X, a generative foundation model for vehicle-to-everything (V2X) cooperative driving. The model directly interprets independent context sequences comprising multi-modal and multi-agent observations. The new design mitigates the computational cost of dense 3D perception, the vulnerability to data scarcity in cooperative scenarios, and the poor compliance with standardized messaging in existing methods that fuse multi-modal inputs into a shared representation. For training, we present an end-to-end supervised pipeline using a downstream trajectory generation loss, in which a high-capacity generative sequence planner implicitly learns to steer the model and leverage multi-modal inputs via cross-attention injection. As a foundation model, we demonstrate that OmniV2X pre-trained on large-scale single-agent planning datasets can efficiently adapt to cooperative environments by integrating the conditioning context with lightweight, standard-compliant V2X tokens. Evaluated on the DAIR-V2X-Seq dataset, OmniV2X outperforms existing end-to-end cooperative driving baselines, achieving state-of-the-art performance with less than 10% of the fine-tune V2X dataset and less than 1% of the communication bandwidth. We conduct comprehensive evaluations to demonstrate its computational efficiency and robustness under real-world constraints.

Juntong Peng, Juanwu Lu, Yupeng Zhou +3

College of Engineering, Purdue University, West Lafayette, IN 47907, USA

End-to-End Autonomous DrivingLLM PlanningAutonomous DrivingV2X CommunicationAutonomous Driving Planning
AskPDF