DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding

Mingzhe Tao; Ruiping Liu; Junwei Zheng; Yufan Chen; Kedi Ying; M. Saquib Sarfraz; Kailun Yang; Jiaming Zhang

cs.CVMar 11, 2026

DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding

Authors: Mingzhe Tao, Ruiping Liu, Junwei Zheng, Yufan Chen, Kedi Ying, M. Saquib Sarfraz, Kailun Yang, Jiaming Zhang, +1 more

Organizations: Karlsruhe Institute of Technology · Mercedes-Benz Tech Innovation · Hunan University

Abstract

Fusing sensors with complementary modalities is crucial for maintaining a stable and comprehensive understanding of abnormal driving scenes. However, Multimodal Large Language Models (MLLMs) are underexplored for leveraging multi-sensor information to understand adverse driving scenarios in autonomous vehicles. To address this gap, we propose the DriveXQA, a multimodal dataset for autonomous driving VQA. In addition to four visual modalities, five sensor failure cases, and five weather conditions, it includes $102,505$ QA pairs categorized into three types: global scene level, allocentric level, and ego-vehicle centric level. Since no existing MLLM framework adopts multiple complementary visual modalities as input, we design MVX-LLM, a token-efficient architecture with a Dual Cross-Attention (DCA) projector that fuses the modalities to alleviate information redundancy. Experiments demonstrate that our DCA achieves improved performance under challenging conditions such as foggy (GPTScore: $53.5$ vs. $25.1$ for the baseline).

Figures & tables

Figure 1: Left : Two corner cases of adverse driving scenes ( foggy condition causing poor visibility and camera over-exposure resulting in degraded image quality). Right : Performance (GPTScore) comparison on DriveXQA dataset.

Dataset	multi-view	modalities	SF	#VQA	FT	Hierarchical
DRAMA [ 29 ]	$\times$	1	$\times$	17K	$\checkmark$	$\times$
VLAAD [ 34 ]	$\times$	1	$\times$	64K	$\checkmark$	$\times$
LingoQA [ 31 ]	$\times$	1	$\times$	419K	$\checkmark$	$\times$
NuPlanQA [ 33 ]	$\checkmark$	1	$\times$	1M	$\checkmark$	$\triangle$
NuScenes-QA [ 36 ]	$\checkmark$	2	$\times$	460K	$\checkmark$	$\times$
NuScenes-MQA [ 21 ]	$\checkmark$	2	$\times$	–	$\checkmark$	$\times$

Table 1: Comparison of autonomous driving VQA datasets. MV : Multi-View RGB cameras. SF : Sensor Failure scenarios. #VQA: Number of VQA pairs. FT : Fine-tuned Evaluation. Hierarchical : Hierarchical question structure.

\triangle

means partially involved.

Figure 2: Statistics of DriveXQA dataset showing distribution of weather conditions (left) and distribution of sensor failure types (right): MB (Motion Blur), OE (Overexposure), UE (Underexposure), LJ (LiDAR Jitter), EL (Event Low-resolution).

Figure 3: Hierarchical XQA examples on DriveXQA dataset. The framework demonstrates three semantic levels: Global Scene Level , Allocentric Level , and Ego-Vehicle Centric Level .

Figure 4: Overview of MVX-LLM . The framework processes multi-modal sensor inputs (RGB, Depth, Event cameras from four viewpoints, and LiDAR point clouds) through specialized encoders. The DCA mechanism integrates RGB, Depth, and Event features before token replacement. The Question Answering component utilizes the fused representations to generate hierarchical responses across Global Scene, Allocentric, and Ego-Vehicle Centric levels under adverse driving conditions.

Method	CL	FG	NT	RN	SN	MB	OE	UE	LJ	EL	Average
prepend	61.2 /58.2	25.1/26.3	58.8 / 60.1	52.3/50.5	55.1/58.3	53.1/57.4	44.1/44.7	51.5/49.2	60.5 / 62.5	55.2 / 58.1	55.1/57.9
Token Fusion
Self-query	40.9/43.4	40.3/42.7	34.9/37.6	41.1/43.5	48.1/46.3	39.9/36.7	28.9/28.4	36.1/39.8	39.5/42.5	44.1/47.6	40.6/43.3
GAP	51.2/54.5	53.2/56.4	50.5/53.6	50.2/47.4	50.4/53.8	54.2/57.5	49.4/52.7	51.1/48.4	54.9/57.2	54.9/57.3	50.5/53.1
QAttn(Spectral)	56.5/58.7	53.7/55.4	46.8/47.9	53.0/56.3	53.2/54.5	54.3/57.7	46.0/44.9	51.6/54.4	55.2/57.3	55.3/57.2	52.8/55.9
QAttn(DepthGate)	38.3/40.2	40.1/43.6	33.0/34.2	41.3/44.6	50.0/53.4	41.1/43.4	27.5/24.8	48.3/51.9	43.9/45.4	40.3/44.6	40.0/44.1

Table 2: Results on DriveXQA dataset across weather conditions and sensor failures. Each cell shows GPT-4o-mini score/Human evaluation score. Blue indicates state-of-the-art results. Weather conditions: CL (Cloudy), FG (Foggy), NT (Night), RN (Rainy), SN (Sunny). Sensor failures: MB (Motion Blur), OE (Overexposure), UE (Underexposure), LJ (LiDAR Jitter), EL (Event Low-resolution).

sAttn	cAttn	QAttn	CIDEr	BLEU-4	ROUGE-L	METEOR	Sim	GPTScore
$\checkmark$			48.3	9.4	23.5	29.8	49.1	24.0
$\checkmark$		$\checkmark$	50.6	15.7	33.2	41.4	61.1	41.1
	$\checkmark$		31.4	6.5	19.7	26.6	46.8	20.4
	$\checkmark$	$\checkmark$	42.9	11.3	27.0	35.6	57.0	34.8
$\checkmark$	$\checkmark$		105.7	24.6	46.0	56.6	70.5	50.5
$\checkmark$	$\checkmark$	$\checkmark$	144.6	28.1	50.6	61.2	74.6	55.5

Table 3: Ablation study on attention mechanism components. Results demonstrate the individual and combined contributions of spatial cross-attention (sAttn), channel cross-attention (cAttn), and Query Attention (QAttn) for robust multi-modal fusion.

RGB	Depth	Event	LiDAR	CIDEr	BLEU-4	ROUGE-L	METEOR	Sim	GPTScore
$\checkmark$	$\checkmark$			163.4	19.5	31.6	38.1	53.4	23.4
$\checkmark$		$\checkmark$		152.6	26.1	43.4	48.2	63.1	42.4
	$\checkmark$	$\checkmark$		8.1	3.9	16.9	23.0	43.5	18.9
$\checkmark$	$\checkmark$	$\checkmark$		157.6	27.3	46.2	51.7	68.7	46.0
$\checkmark$	$\checkmark$		$\checkmark$	117.5	26.0	46.7	54.1	71.5	50.7
$\checkmark$		$\checkmark$	$\checkmark$	50.1	13.4	30.0	37.6	57.3	34.2

Table 4: Ablation study on sensor modality combinations. Results show the importance of each modality and its complementary effects for robust VQA performance under adverse conditions.

Figure 5: Qualitative analysis of multi-modal fusion performance under night conditions with camera overexposure.

Explore similar work

V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
Apr 3, 2026Junwei You, Pei Li, Zhuoyu Jiang +7Autonomous Driving Benchmarks VLMs for Autonomous Driving
Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI, 53706, USA · Department of Civil and Architectural Engineering and Construction Management, University of Wyoming, Laramie, WY, 82071, USA · Kuaishou Technology, Beijing, 100085, China +1
Technical Report on the CVPR 2026@AdvML Workshop Challenge
Jul 13, 2026Tianyuan Zhang, Zonglei Jing, Jiangfan Liu +47VLMs for Autonomous Driving Adversarial Attacks on VLA Models
WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
Nov 25, 2025Seungjun Yu, Seonho Lee, Namho Kim +5Autonomous Driving Datasets Visual Question Answering
KAIST Seoul, Republic of Korea · Hanyang University Seoul, Republic of Korea

Apr 3, 2026J/K move · Enter open · S save

V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views

Junwei You, Pei Li, Zhuoyu Jiang, Weizhe Tang+6

Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI, 53706, USA · Department of Civil and Architectural Engineering and Construction Management, University of Wyoming, Laramie, WY, 82071, USA · Kuaishou Technology, Beijing, 100085, China+1

Autonomous Driving Benchmarks VLMs for Autonomous Driving Autonomous Driving Datasets Autonomous Driving Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have shown strong potential for autonomous driving, yet existing benchmarks remain largely ego-centric and therefore cannot systematically assess model performance in infrastructure-centric and cooperative driving conditions. In this work, we introduce V2X-QA, a real-world dataset and benchmark for evaluating MLLMs across vehicle-side, infrastructure-side, and cooperative viewpoints. V2X-QA is built around a view-decoupled evaluation protocol that enables controlled comparison under vehicle-only, infrastructure-only, and cooperative driving conditions within a unified multiple-choice question answering (MCQA) framework. The benchmark is organized into a twelve-task taxonomy spanning perception, prediction, and reasoning and planning, and is constructed through expert-verified MCQA annotation to enable fine-grained diagnosis of viewpoint-dependent capabilities. Benchmark results across ten representative state-of-the-art proprietary and open-source models show that viewpoint accessibility substantially affects performance, and infrastructure-side reasoning supports meaningful macroscopic traffic understanding. Results also indicate that cooperative reasoning remains challenging since it requires cross-view alignment and evidence integration rather than simply additional visual input. To address these challenges, we introduce V2X-MoE, a benchmark-aligned baseline with explicit view routing and viewpoint-specific LoRA experts. The strong performance of V2X-MoE further suggests that explicit viewpoint specialization is a promising direction for multi-view reasoning in autonomous driving. Overall, V2X-QA provides a foundation for studying multi-perspective reasoning, reliability, and cooperative physical intelligence in connected autonomous driving. The dataset and V2X-MoE resources are publicly available at: https://github.com/junwei0001/V2X-QA.

Abstract

Explore similar work

V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views

Technical Report on the CVPR 2026@AdvML Workshop Challenge

WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving