V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
Authors: Junwei You, Pei Li, Zhuoyu Jiang, Weizhe Tang, Zilin Huang, Rui Gan, Jiaxi Liu, Yan Zhao, +2 more
Organizations: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI, 53706, USA · Department of Civil and Architectural Engineering and Construction Management, University of Wyoming, Laramie, WY, 82071, USA · Kuaishou Technology, Beijing, 100085, China · School of Transportation, Southeast University, Nanjing, Jiangsu, 211189, China
Multimodal large language models (MLLMs) have shown strong potential for autonomous driving, yet existing benchmarks remain largely ego-centric and therefore cannot systematically assess model performance in infrastructure-centric and cooperative driving conditions. In this work, we introduce V2X-QA, a real-world dataset and benchmark for evaluating MLLMs across vehicle-side, infrastructure-side, and cooperative viewpoints. V2X-QA is built around a view-decoupled evaluation protocol that enables controlled comparison under vehicle-only, infrastructure-only, and cooperative driving conditions within a unified multiple-choice question answering (MCQA) framework. The benchmark is organized into a twelve-task taxonomy spanning perception, prediction, and reasoning and planning, and is constructed through expert-verified MCQA annotation to enable fine-grained diagnosis of viewpoint-dependent capabilities. Benchmark results across ten representative state-of-the-art proprietary and open-source models show that viewpoint accessibility substantially affects performance, and infrastructure-side reasoning supports meaningful macroscopic traffic understanding. Results also indicate that cooperative reasoning remains challenging since it requires cross-view alignment and evidence integration rather than simply additional visual input. To address these challenges, we introduce V2X-MoE, a benchmark-aligned baseline with explicit view routing and viewpoint-specific LoRA experts. The strong performance of V2X-MoE further suggests that explicit viewpoint specialization is a promising direction for multi-view reasoning in autonomous driving. Overall, V2X-QA provides a foundation for studying multi-perspective reasoning, reliability, and cooperative physical intelligence in connected autonomous driving. The dataset and V2X-MoE resources are publicly available at: https://github.com/junwei0001/V2X-QA.
Figures & tables
Figure 1: Overview of V2X-QA. Left: representative examples of the twelve viewpoint-aligned tasks under vehicle-side, infrastructure-side, and cooperative settings. Right: MCQA-based training and evaluation pipeline of V2X-MoE on the V2X-QA training and testing splits.
Category
Dataset
Year
Amount
Connectivity & Cooperation
Infrastructure
Multi-Perspective Decoupling Flexibility
Language QA
Safety-Critical Reasoning
Macroscopic Traffic Insight
I. Single-Vehicle QA (”Smart but Blind”)
BDD-X ( Kim et al. (2018) )
2018
26K
✗
✗
–
✓
✗
✗
DRAMA ( Malla et al. (2023) )
2022
14K
✗
✗
–
✓
✓
✗
NuScenes-SpatialQA ( Tian et al. (2025) )
2022
3.3M+
✗
✗
–
✓
✗
✗
VRU-Accident ( Kim et al. (2025) )
2023
6K
✗
✗
–
✓
✓
✗
NuScenes-QA ( Qian et al. (2024) )
2024
460K
✗
✗
–
✓
✓
✗
NuPlanQA ( Park et al. (2025) )
2024
1M
✗
✗
–
✓
✓
✗
Table 1: Comparison of driving QA benchmarks, V2X cooperative perception datasets, and cooperative QA resources. “Multi-perspective decoupling flexibility” indicates whether the benchmark supports controlled evaluation under different viewpoint accessibility conditions within a consistent protocol.
Figure 2: Construction pipeline of V2X-QA. The pipeline starts from the twelve viewpoint-aligned tasks under three settings, then proceeds to MCQA task bank construction, model-assisted answer selection with human verification, data split generation, and standardized benchmarking across proprietary and open-source MLLMs.
Figure 3: Statistics of V2X-QA. Left: overall task and functional distribution across the twelve viewpoint-aligned tasks. Right: distribution of perception, prediction, and reasoning & planning samples under vehicle-side, infrastructure-side, and cooperative views.
Figure 4: Answer-distribution diagnostics of V2X-QA. Left: distribution of correct option positions across the twelve tasks. Middle: majority-class ratio for each task. Right: answer entropy for each task.
Figure 5: Overall architecture and staged training pipeline of V2X-MoE. The model takes viewpoint-aligned visual evidence and MCQA prompts as input, uses a Qwen3-VL multimodal processor and a shared frozen backbone, activates one viewpoint-specific LoRA expert through an explicit router, and is trained by full MCQA training followed by CO-focused and IS-focused refinement.
Figure 6: View-specific LoRA injection with explicit hard routing in V2X-MoE. A viewpoint type signal selects one expert among the vehicle-side, infrastructure-side, and cooperative experts. The selected expert injects low-rank adaptations into the attention projections, while the base transformer parameters remain frozen.
Model
VS1
VS2
VS3
VS4
Avg.
IS1
IS2
IS3
IS4
Avg.
Closed-Source Models
GPT-5.2 ( OpenAI (2025b) )
65.3
43.8
26.0
43.0
44.5
69.8
42.6
38.5
47.5
49.6
GPT-5-Mini ( OpenAI (2025a) )
78.1
49.8
32.1
44.5
51.1
45.7
52.1
46.4
45.3
47.4
Gemini-3-Flash-Preview ( Google (2025) )
69.4
56.2
20.8
38.9
46.3
63.8
44.9
20.8
11.7
35.3
Gemini-2.5-Pro ( Comanici and others (2025) )
87.2
59.2
40.4
46.8
58.4
24.5
43.0
45.3
51.7
41.1
Qwen-3.5-Plus ( Qwen Team (2026) )
84.5
55.1
44.5
58.9
60.8
53.2
56.6
44.5
39.6
48.5
Table 2: Main benchmark results on the vehicle-side and infrastructure-side tasks of V2X-QA. Accuracy (%) is reported for each task and the corresponding average within each viewpoint group.
Model
CO1
CO2
CO3
CO4
Avg.
Closed-Source Models
GPT-5.2 ( OpenAI (2025b) )
34.0
36.2
49.4
27.2
36.7
GPT-5-Mini ( OpenAI (2025a) )
26.4
9.8
47.2
11.3
23.7
Gemini-3-Flash-Preview ( Google (2025) )
86.0
3.8
43.4
13.6
36.7
Gemini-2.5-Pro ( Comanici and others (2025) )
15.8
59.2
45.3
3.4
30.9
Qwen-3.5-Plus ( Qwen Team (2026) )
37.7
30.9
64.5
2.8
34.0
Table 3: Main benchmark results on the cooperative tasks of V2X-QA. Accuracy (%) is reported for each task and the average over the cooperative viewpoint group.
Figure 7: Task-level and viewpoint-level model performance comparison on V2X-QA.
View
ECE
Brier
No. of Samples
VS
0.0386
0.0854
1,060
IS
0.0387
0.0963
1,060
CO
0.0865
0.2292
1,060
All
0.0530
0.1370
3,180
Table 4: Calibration results of V2X-MoE across viewpoint groups.
Figure 8: Reliability diagram and confidence distribution of V2X-MoE across vehicle-side, infrastructure-side, and cooperative viewpoint groups.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Component
Content
System prompt
You are answering a multiple-choice autonomous driving question. Use only the provided image evidence and the question/options. Return exactly one uppercase letter only: A or B or C or D. Do not output any other words, punctuation, or explanation.
User prompt template
Task: {task_id} Viewpoint: {viewpoint} Image evidence: {image_note} Question: {question} Options: A. {opt_a} B. {opt_b} C. {opt_c} D. {opt_d} Return exactly one uppercase letter only: A, B, C, or D.
image_note (VS)
One image is provided: vehicle-side (ego) view.
image_note (IS)
One image is provided: infrastructure-side (RSU) view.
image_note (CO)
Two images are provided in order: (1) vehicle-side (ego) view, (2) infrastructure-side (RSU) view.
Appendix
Table 5: Unified benchmark prompt template used for standardized MCQA evaluation across all benchmarked models.
School of Vehicle and Mobility, Tsinghua University · Suzhou Automotive Research Institute, Tsinghua University · State Key Laboratory of Intelligent Vehicle and Mobility, Tsinghua University