V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
Authors: Junwei You, Pei Li, Zhuoyu Jiang, Weizhe Tang, Zilin Huang, Rui Gan, Jiaxi Liu, Yan Zhao, +2 more
Organizations: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI, 53706, USA · Department of Civil and Architectural Engineering and Construction Management, University of Wyoming, Laramie, WY, 82071, USA · Kuaishou Technology, Beijing, 100085, China · School of Transportation, Southeast University, Nanjing, Jiangsu, 211189, China
Multimodal large language models (MLLMs) have shown strong potential for autonomous driving, yet existing benchmarks remain largely ego-centric and therefore cannot systematically assess model performance in infrastructure-centric and cooperative driving conditions. In this work, we introduce V2X-QA, a real-world dataset and benchmark for evaluating MLLMs across vehicle-side, infrastructure-side, and cooperative viewpoints. V2X-QA is built around a view-decoupled evaluation protocol that enables controlled comparison under vehicle-only, infrastructure-only, and cooperative driving conditions within a unified multiple-choice question answering (MCQA) framework. The benchmark is organized into a twelve-task taxonomy spanning perception, prediction, and reasoning and planning, and is constructed through expert-verified MCQA annotation to enable fine-grained diagnosis of viewpoint-dependent capabilities. Benchmark results across ten representative state-of-the-art proprietary and open-source models show that viewpoint accessibility substantially affects performance, and infrastructure-side reasoning supports meaningful macroscopic traffic understanding. Results also indicate that cooperative reasoning remains challenging since it requires cross-view alignment and evidence integration rather than simply additional visual input. To address these challenges, we introduce V2X-MoE, a benchmark-aligned baseline with explicit view routing and viewpoint-specific LoRA experts. The strong performance of V2X-MoE further suggests that explicit viewpoint specialization is a promising direction for multi-view reasoning in autonomous driving. Overall, V2X-QA provides a foundation for studying multi-perspective reasoning, reliability, and cooperative physical intelligence in connected autonomous driving. The dataset and V2X-MoE resources are publicly available at: https://github.com/junwei0001/V2X-QA.
Figures & tables
Figure 1: Overview of V2X-QA. Left: representative examples of the twelve viewpoint-aligned tasks under vehicle-side, infrastructure-side, and cooperative settings. Right: MCQA-based training and evaluation pipeline of V2X-MoE on the V2X-QA training and testing splits.
Category
Dataset
Year
Amount
Connectivity & Cooperation
Infrastructure
Multi-Perspective Decoupling Flexibility
Language QA
Safety-Critical Reasoning
Macroscopic Traffic Insight
I. Single-Vehicle QA (”Smart but Blind”)
BDD-X ( Kim et al. (2018) )
2018
26K
✗
✗
–
✓
✗
✗
DRAMA ( Malla et al. (2023) )
2022
14K
✗
✗
–
✓
✓
✗
NuScenes-SpatialQA ( Tian et al. (2025) )
2022
3.3M+
✗
✗
–
✓
✗
✗
VRU-Accident ( Kim et al. (2025) )
2023
6K
✗
✗
–
✓
✓
✗
NuScenes-QA ( Qian et al. (2024) )
2024
460K
✗
✗
–
✓
✓
✗
NuPlanQA ( Park et al. (2025) )
2024
1M
✗
✗
–
✓
✓
✗
Table 1: Comparison of driving QA benchmarks, V2X cooperative perception datasets, and cooperative QA resources. “Multi-perspective decoupling flexibility” indicates whether the benchmark supports controlled evaluation under different viewpoint accessibility conditions within a consistent protocol.
Figure 2: Construction pipeline of V2X-QA. The pipeline starts from the twelve viewpoint-aligned tasks under three settings, then proceeds to MCQA task bank construction, model-assisted answer selection with human verification, data split generation, and standardized benchmarking across proprietary and open-source MLLMs.
Figure 3: Statistics of V2X-QA. Left: overall task and functional distribution across the twelve viewpoint-aligned tasks. Right: distribution of perception, prediction, and reasoning & planning samples under vehicle-side, infrastructure-side, and cooperative views.
Figure 4: Answer-distribution diagnostics of V2X-QA. Left: distribution of correct option positions across the twelve tasks. Middle: majority-class ratio for each task. Right: answer entropy for each task.
Figure 5: Overall architecture and staged training pipeline of V2X-MoE. The model takes viewpoint-aligned visual evidence and MCQA prompts as input, uses a Qwen3-VL multimodal processor and a shared frozen backbone, activates one viewpoint-specific LoRA expert through an explicit router, and is trained by full MCQA training followed by CO-focused and IS-focused refinement.
Figure 6: View-specific LoRA injection with explicit hard routing in V2X-MoE. A viewpoint type signal selects one expert among the vehicle-side, infrastructure-side, and cooperative experts. The selected expert injects low-rank adaptations into the attention projections, while the base transformer parameters remain frozen.
Model
VS1
VS2
VS3
VS4
Avg.
IS1
IS2
IS3
IS4
Avg.
Closed-Source Models
GPT-5.2 ( OpenAI (2025b) )
65.3
43.8
26.0
43.0
44.5
69.8
42.6
38.5
47.5
49.6
GPT-5-Mini ( OpenAI (2025a) )
78.1
49.8
32.1
44.5
51.1
45.7
52.1
46.4
45.3
47.4
Gemini-3-Flash-Preview ( Google (2025) )
69.4
56.2
20.8
38.9
46.3
63.8
44.9
20.8
11.7
35.3
Gemini-2.5-Pro ( Comanici and others (2025) )
87.2
59.2
40.4
46.8
58.4
24.5
43.0
45.3
51.7
41.1
Qwen-3.5-Plus ( Qwen Team (2026) )
84.5
55.1
44.5
58.9
60.8
53.2
56.6
44.5
39.6
48.5
Table 2: Main benchmark results on the vehicle-side and infrastructure-side tasks of V2X-QA. Accuracy (%) is reported for each task and the corresponding average within each viewpoint group.
Model
CO1
CO2
CO3
CO4
Avg.
Closed-Source Models
GPT-5.2 ( OpenAI (2025b) )
34.0
36.2
49.4
27.2
36.7
GPT-5-Mini ( OpenAI (2025a) )
26.4
9.8
47.2
11.3
23.7
Gemini-3-Flash-Preview ( Google (2025) )
86.0
3.8
43.4
13.6
36.7
Gemini-2.5-Pro ( Comanici and others (2025) )
15.8
59.2
45.3
3.4
30.9
Qwen-3.5-Plus ( Qwen Team (2026) )
37.7
30.9
64.5
2.8
34.0
Table 3: Main benchmark results on the cooperative tasks of V2X-QA. Accuracy (%) is reported for each task and the average over the cooperative viewpoint group.
Figure 7: Task-level and viewpoint-level model performance comparison on V2X-QA.
View
ECE
Brier
No. of Samples
VS
0.0386
0.0854
1,060
IS
0.0387
0.0963
1,060
CO
0.0865
0.2292
1,060
All
0.0530
0.1370
3,180
Table 4: Calibration results of V2X-MoE across viewpoint groups.
Figure 8: Reliability diagram and confidence distribution of V2X-MoE across vehicle-side, infrastructure-side, and cooperative viewpoint groups.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Component
Content
System prompt
You are answering a multiple-choice autonomous driving question. Use only the provided image evidence and the question/options. Return exactly one uppercase letter only: A or B or C or D. Do not output any other words, punctuation, or explanation.
User prompt template
Task: {task_id} Viewpoint: {viewpoint} Image evidence: {image_note} Question: {question} Options: A. {opt_a} B. {opt_b} C. {opt_c} D. {opt_d} Return exactly one uppercase letter only: A, B, C, or D.
image_note (VS)
One image is provided: vehicle-side (ego) view.
image_note (IS)
One image is provided: infrastructure-side (RSU) view.
image_note (CO)
Two images are provided in order: (1) vehicle-side (ego) view, (2) infrastructure-side (RSU) view.
Appendix
Table 5: Unified benchmark prompt template used for standardized MCQA evaluation across all benchmarked models.
Fusing sensors with complementary modalities is crucial for maintaining a stable and comprehensive understanding of abnormal driving scenes. However, Multimodal Large Language Models (MLLMs) are underexplored for leveraging multi-sensor information to understand adverse driving scenarios in autonomous vehicles. To address this gap, we propose the DriveXQA, a multimodal dataset for autonomous driving VQA. In addition to four visual modalities, five sensor failure cases, and five weather conditions, it includes 102,505 QA pairs categorized into three types: global scene level, allocentric level, and ego-vehicle centric level. Since no existing MLLM framework adopts multiple complementary visual modalities as input, we design MVX-LLM, a token-efficient architecture with a Dual Cross-Attention (DCA) projector that fuses the modalities to alleviate information redundancy. Experiments demonstrate that our DCA achieves improved performance under challenging conditions such as foggy (GPTScore: 53.5 vs. 25.1 for the baseline).
Mingzhe Tao, Ruiping Liu, Junwei Zheng +6
Karlsruhe Institute of Technology · Mercedes-Benz Tech Innovation · Hunan University
Vision-language models (VLMs) have made substantial progress in autonomous driving, but their success has primarily been studied in ego-centric scenes. Infrastructure-side observations provide views beyond the ego vehicle's field of view, yet conventional cooperative-driving systems typically transform them into geometric representations for downstream perception and planning. Directly incorporating these views into VLMs offers an opportunity to improve cooperative scene understanding and trajectory planning. However, question answering and trajectory planning have not been jointly evaluated on the same real-world vehicle-infrastructure scenes. We present CoVLM-Bench, a benchmark for cooperative driving question answering (CDQA) and cooperative planning (CP) on vehicle-infrastructure paired scenes. CoVLM-Bench provides scene-grounded CDQA annotations, three-part rationales as auxiliary supervision, and future trajectory targets derived from recorded ego motion. It contains 2,196 paired frames with 35,136 CDQA annotations, while CP predicts six waypoints over a three-second horizon. The annotations combine model-assisted drafting, record-based computation, and human verification. Built upon CoVLM-Bench, we introduce CoVLM-Drive, a unified VLM baseline that directly uses paired views for both CDQA and CP. Experiments show that CDQA adaptation improves answer accuracy and that CoVLM-Drive reaches a lower FDE than the compared V2X planners; QA initialization and rationale supervision each reduce planning error. Together, CoVLM-Bench and CoVLM-Drive support the training and comparison of VLMs for cooperative scene understanding and planning.
Kang Yang, Shuai Liu, Hang Li +3
Renmin University of China · Hebei Key Laboratory of Real-virtual Integrated Autonomous Systems (RIAS)
Vehicle-to-Everything (V2X) cooperation enables beyond-line-of-sight perception, mitigating occlusions in single-vehicle sensing. However, existing V2X benchmarks provide limited support for closed-loop evaluation and language-grounded supervision, hindering the development of vision-language models (VLMs) for end-to-end cooperative driving. To address these limitations, we introduce V2XBench, a simulation platform featuring synchronized ego--roadside sensing and closed-loop evaluation, together with Chat-V2XBench, a progressively structured VQA dataset for cooperative reasoning. Building upon this benchmark infrastructure, we propose AURORA, an end-to-end cooperative driving framework. Equipped with a dual-view perception architecture, AURORA mitigates spatial and semantic discrepancies across ego and roadside viewpoints through a query-level Cross-View Query Alignment and Fusion (CQAF) module. Leveraging the resulting unified tokens, a LoRA-adapted VLM bridges semantic reasoning and generative trajectory planning. Extensive closed-loop evaluations on V2XBench demonstrate that AURORA achieves state-of-the-art performance in heavily occluded scenarios, with a Route Completion rate of 98.21% and a Driving Score of 76.02, while requiring low roadside communication bandwidth. Ultimately, this work pioneers an extensible V2X--VLM paradigm, paving the way for next-generation cooperative autonomous driving.
Yitao Xu, Tong Wu, Yiyan Wu +7
School of Vehicle and Mobility, Tsinghua University · Suzhou Automotive Research Institute, Tsinghua University · State Key Laboratory of Intelligent Vehicle and Mobility, Tsinghua University