Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering

Shaozu Ding; Linan Song; Dajiang Suo

cs.CVAug 28, 2026

Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering

Authors: Shaozu Ding, Linan Song, Dajiang Suo

Organizations: The Polytechnic School, Arizona State University

Abstract

Recent advances in visual question answering (VQA) and multimodal large language models (MLLMs) have enabled natural-language reasoning over traffic scenes. However, existing benchmarks are largely built from ego-vehicle views or 2D roadside videos, limiting their ability to evaluate 3D-grounded reasoning over real-world distances, trajectories, infrastructure topology, and safety-critical interactions. We introduce Inter-3D VQA, a large-scale roadside multimodal benchmark for 3D spatiotemporally grounded VQA at intersections. Built from synchronized point clouds and multi-view images, Inter-3D VQA contains 407K QA pairs covering lane-level positions, object relationships, motion patterns, and near-miss-oriented interaction reasoning. We further propose Inter-Geo, an MLLM baseline that integrates object- and scene-level aligned LiDAR representations, and Inter-Metrics, a unified evaluation framework for textual consistency, numerical accuracy, and semantic correctness. Experiments show that Inter-Geo outperforms image-based VLMs, especially on grounded spatial and temporal reasoning tasks. Our benchmark and codes are available at https://github.com/ASU-Suo-Lab/Inter-3D-VQA .

Figures & tables

Figure 1: Inter-3D VQA is the first roadside multimodal benchmark for 3D grounded VQA.

Figure 2: Inter-3D VQA is a large-scale roadside multimodal benchmark, aligning multi-view images with global point clouds to support diverse tasks from 3D perspective. Our MLLM Inter-Geo leverages object- and scene-level aligned LiDAR representations, achieving state-of-the-art performance on 3D grounded reasoning tasks.

Dataset	Domain	Main Tasks	QA Generation	QA Type	Modality	QAs
NuScenes-QA ( 2024 )	Driving	3D VQA	Temp.	Free-form	Image + PC	460k
LingoQA ( 2024 )	Driving	VQA	Man. + LLM	Free-form	Image	419k
DriveLM ( 2024 )	Driving	GVQA + T.P.	Man. + Temp.	Grounded	Image	443k
DriveLMM-o1 ( 2025 )	Driving	M.C.	Man. + LLM	Grounded	Image	18k
NuPlanQA ( 2025 )	Driving	M.C.	Man. + LLM	Free-form	Image	1M
SURDS ( 2025 )	Driving	VQA	Temp. + LLM	Free-form	Image	50.3k

Table 1: Comprehensive comparison of representative VQA benchmarks in autonomous driving, urban, and roadside scenarios. Temp.: Template, Man.: Manual, M.C.: Multiple Choice, T.P.: Trajectory Prediction, PC: Point Cloud.

Figure 3: Data construction pipeline of Inter-3D VQA. The pipeline consists of three stages: metadata curation from synchronized multi-view images, LiDAR point clouds, and HD maps, template-based QA generation with LLM-assisted expansion, and a multi-stage sanity check for quality control and data balancing.

Figure 4: Object referring mechanisms in Inter-3D VQA. Grounded QA provides spatiotemporal object references for cross-view and 3D grounding, while free-form QA uses topology-aware language descriptions to reduce ambiguity through infrastructure constraints.

Figure 5: Architecture of Inter-Geo model, which integrates multi-view images, text, and global point clouds. Object- and scene-level features are extracted by 3D encoder, then transformed into dual aligned LiDAR representations via DQNet, and finally injected into the LLM decoder through LDA for geometry-aware reasoning.

Types	Models	Free-form QA			Grounded QA
Types	Models	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
Open-source Generalist VLMs	LLaVA-NeXT-8B ( 2024 )	0.788	0.347	0.646	0.803	0.341	0.656
	Llama3.2-11B ( 2024 )	0.770	0.289	0.583	0.788	0.364	0.576
	Qwen3-VL-4B ( 2025 )	0.806	0.397	0.709	0.827	0.432	0.736
	Qwen3-VL-8B ( 2025 )	0.821	0.457	0.749	0.851	0.488	0.802
	InternVL3-4B ( 2025 )	0.794	0.363	0.656	0.811	0.405	0.666

Table 2: Performance comparison of Inter-Geo with various VLMs on both free-form and grounded Inter-3D VQA.

3D Enc.	DQNet		LDA	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
3D Enc.	Obj.	Scene	LDA	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
✓	✓	✓	✓	0.859	0.589	0.828
	✓	✓	✓	0.841	0.477	0.765
✓		✓	✓	0.857	0.566	0.822
✓	✓		✓	0.857	0.553	0.817
✓	✓	✓		0.827	0.432	0.736

Table 3: Contributions of LiDAR-related modules. Removing both DQNet branches is equivalent to removing LDA, as neither provides LiDAR information.

Backbone	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
Qwen3-VL-2B-Instruct	0.853	0.558	0.812
Qwen3-VL-4B-Instruct	0.859	0.589	0.828
Qwen3-VL-8B-Instruct	0.869	0.617	0.848

Table 4: Effect of backbone model scale on Inter-Geo.

Injection	Stage	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
Input Embed	Input	0.856	0.552	0.816
Vision Fusion	Input	0.850	0.549	0.810
LDA	Decoder	0.859	0.589	0.828

Table 5: Effect of LiDAR modality injection strategy.

Model	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
Qwen3-4B	0.769 (-10.5%)	0.365 (-38.0%)	0.510 (-38.4%)
Qwen3-8B	0.783 (-8.8%)	0.411 (-30.2%)	0.573 (-30.8%)

Table 6: Performance of text-only LLMs on the grounded Inter-3D VQA, with relative performance drops compared to Inter-Geo.

Model	Inference Speed
Qwen3-4B	3.1 FPS
Qwen3-VL-4B	2.6 FPS
Inter-Geo	1.3 FPS

Table 7: Inference speed comparison of Inter-Geo with text-only LLM and image-only VLM baselines.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Question-level statistics of Inter-3D VQA. The dataset exhibits a balanced distribution across task categories and provides rich topology-aware descriptions for 3D reasoning.

Panel of Figure 7 — Figure 7: Answer-level statistics of Inter-3D VQA. (a) Distribution of answer lengths, demonstrating concise and reasoning-oriented responses. (b) Word cloud of answers, revealing dominant semantic concepts in traffic participants, infrastructure, and motion.

Panel of Figure 8 — Figure 8: Behavior and object statistics of Inter-3D VQA. (a) Distribution of driving behaviors, covering both routine and safety-critical actions. (b) Distribution of object categories, including diverse traffic participants.

Figure 9: Distribution of numerical attributes in Inter-3D VQA, including object count, distance, and speed.

Figure 10: Spatial distribution of 3D coordinates in Inter-3D VQA.

Models	Free-form QA			Grounded QA
Models	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
GPT-5.1 ( 2025 )	0.715	0.251	0.407	0.523	0.081	0.098
Gemini-2.5-flash ( 2025 )	0.725	0.204	0.428	0.534	0.141	0.125
Claude-4.5-Haiku ( 2025 )	0.669	0.197	0.335	0.537	0.075	0.077

Appendix

Table 8: Zero-shot performance of closed-source API-only VLMs on free-form and grounded Inter-3D VQA.

Model	Backbone	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
CenterPoint Yin et al. (2021)	Sparse Convolution	0.862	0.540	0.828
TransFusion-L Bai et al. (2022)	Sparse Convolution	0.857	0.580	0.819
DSVT Wang et al. (2023)	Transformer	0.855	0.544	0.812
LION Liu et al. (2024)	Mamba	0.859	0.589	0.828

Appendix

Table 9: Comparison of different 3D encoders.

Insertion Position	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
First 4 layers	0.858	0.554	0.816
Middle 4 layers	0.858	0.552	0.822
Last 4 layers	0.859	0.589	0.828

Appendix

Table 10: Effect of LDA insertion position in the decoder on Inter-Geo.

# LDA Layers	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
2	0.858	0.544	0.821
4	0.859	0.589	0.828
8	0.857	0.553	0.816

Appendix

Table 11: Effect of the number of LDA layers in the decoder on Inter-Geo.

# Object Queries	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
8	0.861	0.569	0.827
16	0.859	0.589	0.828
24	0.858	0.545	0.818

Appendix

Table 12: Effect of the number of object queries in DQNet.

Models	Count $\downarrow$	Distance $\downarrow$	Speed $\downarrow$	Acceleration $\downarrow$	$\text{Coord}_{\text{3d}}$ $\downarrow$	$\text{Coord}_{\text{2d}}$ $\downarrow$	Waypoint $\downarrow$
GPT-5.1	0.95	0.91	1.00	1.00	0.78	0.93	0.86
Gemini-2.5-flash	0.97	0.91	0.51	0.99	0.85	0.94	0.84
Claude-4.5-Haiku	0.95	0.94	0.71	1.00	0.99	1.00	0.90
LLaVA-NeXT-8B	0.35	0.24	0.69	0.94	0.81	0.87	0.71
Llama3.2-11B	0.37	0.21	0.63	0.84	0.81	0.86	0.74
Qwen3-VL-4B	0.26	0.18	0.53	0.64	0.77	0.81	0.78

Appendix

Table 13: Detailed numerical error comparison on grounded Inter-3D VQA across different numerical categories.

# Scene Queries	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
32	0.859	0.567	0.824
64	0.859	0.589	0.828
128	0.858	0.545	0.820

Appendix

Table 14: Effect of the number of scene queries in DQNet.

Figure 11: System prompt for Inter-3D VQA evaluation. For image-based VLMs, LiDAR-related instructions are removed to avoid unavailable-modality interference.

Figure 12: Representative free-form question templates for Basic Perception.

Figure 13: Representative free-form question templates for Spatial Reasoning.

Figure 14: Representative free-form question templates for Temporal Reasoning.

Figure 15: Representative free-form question templates for Scene Understanding.

Figure 16: Representative grounded question templates for Basic Perception.

Figure 17: Representative grounded question templates for Spatial Reasoning.

Figure 18: Representative grounded question templates for Temporal Reasoning.

Figure 19: Representative grounded question templates for Scene Understanding.

Figure 20: Daytime examples of Spatial Reasoning and Temporal Reasoning.

Figure 21: Nighttime examples of Basic Perception and Scene Understanding.

3D VQA

Three-dimensional visual question answering uses spatial and geometric evidence to answer questions about scenes, objects, and volumetric images.

See all 46 papers

RISE: Roadside Infrastructure Sequence Understanding across 3D Tracking and Structured Vision-Language Reasoning
Aug 17, 2026Yanbo Jiang, Haotian Zheng, Jiahao Wang +10Temporal Reasoning in VLMs 3D MOT
School of Vehicle and Mobility, Tsinghua University · TsingCloud
OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning
Jun 14, 2026Maonan Wang, Zhengyan Huang, Kemou Jiang +13Autonomous Driving Datasets Spatiotemporal Reasoning
The Chinese University of Hong Kong, Shenzhen · Shanghai AI Lab · Beihang University +3
Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models
May 27, 2026Jiyao Zhang, Mingxu Zhang, Yitong Peng +8VLM Evaluation Spatial Reasoning Benchmarks
CFCS, School of CS, PKU Beijing, China · BUPT Beijing, China · PKU Beijing, China +2

Aug 17, 2026J/K move · Enter open · S save

RISE: Roadside Infrastructure Sequence Understanding across 3D Tracking and Structured Vision-Language Reasoning

Yanbo Jiang, Haotian Zheng, Jiahao Wang, Hanxiao Ren+9

School of Vehicle and Mobility, Tsinghua University · TsingCloud

Temporal Reasoning in VLMs 3D MOT Autonomous Driving Benchmarks VLM Reasoning Video QA VLMs for Autonomous Driving

Open paper AskPDF arXiv

We present RISE (Roadside Infrastructure Sequence Understanding and Evaluation), a framework spanning metric 3D tracking and structured vision-language reasoning in roadside sequences. For metric tracking, our image-only method combines SAM3 video identities with calibration-guided mask agreement for multi-view identity association, recovering persistent 3D tracks without LiDAR or task-specific 3D training. Its calibration-conditioned geometry allows the procedure to be instantiated at different calibrated multi-camera intersections without layout-specific retraining. On 20 human-reviewed clips from six intersections, the generated tracks achieve 66.9 MOTA within the defined multi-view evaluation scope. For structured vision-language reasoning, a human-reviewed MLLM pipeline mines high-value clips and uses a constrained full-context Oracle to construct bbox-grounded predictive QA without exposing future evidence to evaluated models. The resulting RISE-VQA dataset contains 33,910 QA pairs from 557 clips across 16 intersections and 61 roadside views. Its intersection-held-out RISE-Bench evaluates semantic choices, coordinates, future boxes, and interaction sets with deterministic task-specific metrics. Experiments show consistent benefits from domain adaptation and generally from temporal context, while revealing persistent challenges in spatial grounding, future localization, and interaction reasoning.

Abstract

3D VQA

Explore similar work

RISE: Roadside Infrastructure Sequence Understanding across 3D Tracking and Structured Vision-Language Reasoning

OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning

Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models