Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering

Shaozu Ding; Linan Song; Dajiang Suo

cs.CVAug 28, 2026

Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering

Authors: Shaozu Ding, Linan Song, Dajiang Suo

Organizations: The Polytechnic School, Arizona State University

Abstract

Recent advances in visual question answering (VQA) and multimodal large language models (MLLMs) have enabled natural-language reasoning over traffic scenes. However, existing benchmarks are largely built from ego-vehicle views or 2D roadside videos, limiting their ability to evaluate 3D-grounded reasoning over real-world distances, trajectories, infrastructure topology, and safety-critical interactions. We introduce Inter-3D VQA, a large-scale roadside multimodal benchmark for 3D spatiotemporally grounded VQA at intersections. Built from synchronized point clouds and multi-view images, Inter-3D VQA contains 407K QA pairs covering lane-level positions, object relationships, motion patterns, and near-miss-oriented interaction reasoning. We further propose Inter-Geo, an MLLM baseline that integrates object- and scene-level aligned LiDAR representations, and Inter-Metrics, a unified evaluation framework for textual consistency, numerical accuracy, and semantic correctness. Experiments show that Inter-Geo outperforms image-based VLMs, especially on grounded spatial and temporal reasoning tasks. Our benchmark and codes are available at https://github.com/ASU-Suo-Lab/Inter-3D-VQA .

Figures & tables

Figure 1: Inter-3D VQA is the first roadside multimodal benchmark for 3D grounded VQA.

Figure 2: Inter-3D VQA is a large-scale roadside multimodal benchmark, aligning multi-view images with global point clouds to support diverse tasks from 3D perspective. Our MLLM Inter-Geo leverages object- and scene-level aligned LiDAR representations, achieving state-of-the-art performance on 3D grounded reasoning tasks.

Dataset	Domain	Main Tasks	QA Generation	QA Type	Modality	QAs
NuScenes-QA ( 2024 )	Driving	3D VQA	Temp.	Free-form	Image + PC	460k
LingoQA ( 2024 )	Driving	VQA	Man. + LLM	Free-form	Image	419k
DriveLM ( 2024 )	Driving	GVQA + T.P.	Man. + Temp.	Grounded	Image	443k
DriveLMM-o1 ( 2025 )	Driving	M.C.	Man. + LLM	Grounded	Image	18k
NuPlanQA ( 2025 )	Driving	M.C.	Man. + LLM	Free-form	Image	1M
SURDS ( 2025 )	Driving	VQA	Temp. + LLM	Free-form	Image	50.3k

Table 1: Comprehensive comparison of representative VQA benchmarks in autonomous driving, urban, and roadside scenarios. Temp.: Template, Man.: Manual, M.C.: Multiple Choice, T.P.: Trajectory Prediction, PC: Point Cloud.

Figure 3: Data construction pipeline of Inter-3D VQA. The pipeline consists of three stages: metadata curation from synchronized multi-view images, LiDAR point clouds, and HD maps, template-based QA generation with LLM-assisted expansion, and a multi-stage sanity check for quality control and data balancing.

Figure 4: Object referring mechanisms in Inter-3D VQA. Grounded QA provides spatiotemporal object references for cross-view and 3D grounding, while free-form QA uses topology-aware language descriptions to reduce ambiguity through infrastructure constraints.

Figure 5: Architecture of Inter-Geo model, which integrates multi-view images, text, and global point clouds. Object- and scene-level features are extracted by 3D encoder, then transformed into dual aligned LiDAR representations via DQNet, and finally injected into the LLM decoder through LDA for geometry-aware reasoning.

Types	Models	Free-form QA			Grounded QA
Types	Models	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
Open-source Generalist VLMs	LLaVA-NeXT-8B ( 2024 )	0.788	0.347	0.646	0.803	0.341	0.656
	Llama3.2-11B ( 2024 )	0.770	0.289	0.583	0.788	0.364	0.576
	Qwen3-VL-4B ( 2025 )	0.806	0.397	0.709	0.827	0.432	0.736
	Qwen3-VL-8B ( 2025 )	0.821	0.457	0.749	0.851	0.488	0.802
	InternVL3-4B ( 2025 )	0.794	0.363	0.656	0.811	0.405	0.666

Table 2: Performance comparison of Inter-Geo with various VLMs on both free-form and grounded Inter-3D VQA.

3D Enc.	DQNet		LDA	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
3D Enc.	Obj.	Scene	LDA	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
✓	✓	✓	✓	0.859	0.589	0.828
	✓	✓	✓	0.841	0.477	0.765
✓		✓	✓	0.857	0.566	0.822
✓	✓		✓	0.857	0.553	0.817
✓	✓	✓		0.827	0.432	0.736

Table 3: Contributions of LiDAR-related modules. Removing both DQNet branches is equivalent to removing LDA, as neither provides LiDAR information.

Backbone	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
Qwen3-VL-2B-Instruct	0.853	0.558	0.812
Qwen3-VL-4B-Instruct	0.859	0.589	0.828
Qwen3-VL-8B-Instruct	0.869	0.617	0.848

Table 4: Effect of backbone model scale on Inter-Geo.

Injection	Stage	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
Input Embed	Input	0.856	0.552	0.816
Vision Fusion	Input	0.850	0.549	0.810
LDA	Decoder	0.859	0.589	0.828

Table 5: Effect of LiDAR modality injection strategy.

Model	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
Qwen3-4B	0.769 (-10.5%)	0.365 (-38.0%)	0.510 (-38.4%)
Qwen3-8B	0.783 (-8.8%)	0.411 (-30.2%)	0.573 (-30.8%)

Table 6: Performance of text-only LLMs on the grounded Inter-3D VQA, with relative performance drops compared to Inter-Geo.

Model	Inference Speed
Qwen3-4B	3.1 FPS
Qwen3-VL-4B	2.6 FPS
Inter-Geo	1.3 FPS

Table 7: Inference speed comparison of Inter-Geo with text-only LLM and image-only VLM baselines.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Question-level statistics of Inter-3D VQA. The dataset exhibits a balanced distribution across task categories and provides rich topology-aware descriptions for 3D reasoning.

Panel of Figure 7 — Figure 7: Answer-level statistics of Inter-3D VQA. (a) Distribution of answer lengths, demonstrating concise and reasoning-oriented responses. (b) Word cloud of answers, revealing dominant semantic concepts in traffic participants, infrastructure, and motion.

Panel of Figure 8 — Figure 8: Behavior and object statistics of Inter-3D VQA. (a) Distribution of driving behaviors, covering both routine and safety-critical actions. (b) Distribution of object categories, including diverse traffic participants.

Figure 9: Distribution of numerical attributes in Inter-3D VQA, including object count, distance, and speed.

Figure 10: Spatial distribution of 3D coordinates in Inter-3D VQA.

Models	Free-form QA			Grounded QA
Models	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
GPT-5.1 ( 2025 )	0.715	0.251	0.407	0.523	0.081	0.098
Gemini-2.5-flash ( 2025 )	0.725	0.204	0.428	0.534	0.141	0.125
Claude-4.5-Haiku ( 2025 )	0.669	0.197	0.335	0.537	0.075	0.077

Appendix

Table 8: Zero-shot performance of closed-source API-only VLMs on free-form and grounded Inter-3D VQA.

Model	Backbone	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
CenterPoint Yin et al. (2021)	Sparse Convolution	0.862	0.540	0.828
TransFusion-L Bai et al. (2022)	Sparse Convolution	0.857	0.580	0.819
DSVT Wang et al. (2023)	Transformer	0.855	0.544	0.812
LION Liu et al. (2024)	Mamba	0.859	0.589	0.828

Appendix

Table 9: Comparison of different 3D encoders.

Insertion Position	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
First 4 layers	0.858	0.554	0.816
Middle 4 layers	0.858	0.552	0.822
Last 4 layers	0.859	0.589	0.828

Appendix

Table 10: Effect of LDA insertion position in the decoder on Inter-Geo.

# LDA Layers	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
2	0.858	0.544	0.821
4	0.859	0.589	0.828
8	0.857	0.553	0.816

Appendix

Table 11: Effect of the number of LDA layers in the decoder on Inter-Geo.

# Object Queries	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
8	0.861	0.569	0.827
16	0.859	0.589	0.828
24	0.858	0.545	0.818

Appendix

Table 12: Effect of the number of object queries in DQNet.

Models	Count $\downarrow$	Distance $\downarrow$	Speed $\downarrow$	Acceleration $\downarrow$	$\text{Coord}_{\text{3d}}$ $\downarrow$	$\text{Coord}_{\text{2d}}$ $\downarrow$	Waypoint $\downarrow$
GPT-5.1	0.95	0.91	1.00	1.00	0.78	0.93	0.86
Gemini-2.5-flash	0.97	0.91	0.51	0.99	0.85	0.94	0.84
Claude-4.5-Haiku	0.95	0.94	0.71	1.00	0.99	1.00	0.90
LLaVA-NeXT-8B	0.35	0.24	0.69	0.94	0.81	0.87	0.71
Llama3.2-11B	0.37	0.21	0.63	0.84	0.81	0.86	0.74
Qwen3-VL-4B	0.26	0.18	0.53	0.64	0.77	0.81	0.78

Appendix

Table 13: Detailed numerical error comparison on grounded Inter-3D VQA across different numerical categories.

# Scene Queries	$S_{\text{text}}\uparrow$	$S_{\text{num}}\uparrow$	$S_{\text{sem}}\uparrow$
32	0.859	0.567	0.824
64	0.859	0.589	0.828
128	0.858	0.545	0.820

Appendix

Table 14: Effect of the number of scene queries in DQNet.

Figure 11: System prompt for Inter-3D VQA evaluation. For image-based VLMs, LiDAR-related instructions are removed to avoid unavailable-modality interference.

Figure 12: Representative free-form question templates for Basic Perception.

Figure 13: Representative free-form question templates for Spatial Reasoning.

Figure 14: Representative free-form question templates for Temporal Reasoning.

Figure 15: Representative free-form question templates for Scene Understanding.

Figure 16: Representative grounded question templates for Basic Perception.

Figure 17: Representative grounded question templates for Spatial Reasoning.

Figure 18: Representative grounded question templates for Temporal Reasoning.

Figure 19: Representative grounded question templates for Scene Understanding.

Figure 20: Daytime examples of Spatial Reasoning and Temporal Reasoning.

Figure 21: Nighttime examples of Basic Perception and Scene Understanding.

3D VQA

Three-dimensional visual question answering uses spatial and geometric evidence to answer questions about scenes, objects, and volumetric images.

See all 46 papers

Aug 17, 2026

RISE: Roadside Infrastructure Sequence Understanding across 3D Tracking and Structured Vision-Language Reasoning

We present RISE (Roadside Infrastructure Sequence Understanding and Evaluation), a framework spanning metric 3D tracking and structured vision-language reasoning in roadside sequences. For metric tracking, our image-only method combines SAM3 video identities with calibration-guided mask agreement for multi-view identity association, recovering persistent 3D tracks without LiDAR or task-specific 3D training. Its calibration-conditioned geometry allows the procedure to be instantiated at different calibrated multi-camera intersections without layout-specific retraining. On 20 human-reviewed clips from six intersections, the generated tracks achieve 66.9 MOTA within the defined multi-view evaluation scope. For structured vision-language reasoning, a human-reviewed MLLM pipeline mines high-value clips and uses a constrained full-context Oracle to construct bbox-grounded predictive QA without exposing future evidence to evaluated models. The resulting RISE-VQA dataset contains 33,910 QA pairs from 557 clips across 16 intersections and 61 roadside views. Its intersection-held-out RISE-Bench evaluates semantic choices, coordinates, future boxes, and interaction sets with deterministic task-specific metrics. Experiments show consistent benefits from domain adaptation and generally from temporal context, while revealing persistent challenges in spatial grounding, future localization, and interaction reasoning.

Yanbo Jiang, Haotian Zheng, Jiahao Wang +10

School of Vehicle and Mobility, Tsinghua University · TsingCloud

Temporal Reasoning in VLMs 3D MOT Autonomous Driving Benchmarks VLM Reasoning Video QA VLMs for Autonomous Driving

Ask PDF

Jun 14, 2026

OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning

Traffic scene understanding requires models to reason beyond object recognition, including lane topology, multi-view geometry, temporal evolution, and signal-phase semantics. However, existing traffic-oriented multimodal benchmarks largely emphasize passive visual recognition or isolated video understanding, offering limited support for evaluating structure-aware traffic reasoning under controlled conditions. We introduce OmniTraffic, a controllable generation pipeline and benchmark for spatio-temporal traffic reasoning. Built around 12 real-world intersections reconstructed into editable 3D traffic environments and complemented by surveillance footage from two countries, OmniTraffic supports both controlled and natural-condition evaluation. It defines a three-level task hierarchy spanning scene perception, multi-view and temporal reasoning, and decision support. Using structured traffic metadata, OmniTraffic generates synchronized multi-view VQA samples covering vehicle states, lane functions, view--BEV correspondence, temporal dynamics, and signal-phase analysis, resulting in 8M VQA samples and a 3K human-verified test set. Evaluation of eleven frontier MLLMs reveals a large human--model gap, with the most pronounced failures in topology-grounded and spatio-temporal reasoning tasks. Fine-tuning a lightweight MLLM on simulated OmniTraffic data further improves performance on real-world traffic scenes, demonstrating the value of simulation-generated supervision for traffic-specific multimodal reasoning. Beyond a fixed dataset, OmniTraffic provides an extensible pipeline with configurable intersections, camera views, traffic demands, signal phases, visual conditions, and rare events.

Maonan Wang, Zhengyan Huang, Kemou Jiang +13

The Chinese University of Hong Kong, Shenzhen · Shanghai AI Lab · Beihang University +3

Autonomous Driving Datasets Spatiotemporal Reasoning Autonomous Driving Perception Multimodal Reasoning Multimodal Model Evaluation

Ask PDF

May 27, 2026

Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models

Are current Vision Language Models (VLMs) ready to comprehend and reason about complex embodied interactions in 3D environments? We introduce Embodied3DBench, a robot-centric benchmark targeting low-level spatial intelligence in embodied 3D environments. To systematically evaluate these foundational perceptual capabilities, the benchmark includes 6 task categories divided into two core groups: Spatial Structural Understanding (Grounding, Spatial Relation Prediction, and Multi-view Correspondence) and Interaction-Oriented Perception (Affordance Prediction, Grasp Point Prediction, and Trajectory Prediction). The benchmark spans 12 subcategories and contains over 21k high-quality question-answer pairs. We evaluate 13 state-of-the-art models, and the results show that while current models exhibit relatively strong high-level spatial reasoning, such as understanding object-to-object positional relations, they remain fragile in interaction-oriented perception, highlighting a significant lack of robust 3D-aware interaction priors. To actively bridge this capability gap revealed by our benchmark, we further synthesize a large-scale training dataset comprising 1.3M QA pairs. Notably, fine-tuning on this dataset yields significant improvements in low-level spatial intelligence. Ultimately, Embodied3DBench fills a critical gap by providing both a systematic evaluation framework and a scalable data solution, setting a clear target for the development of interaction-aware multimodal systems.

Jiyao Zhang, Mingxu Zhang, Yitong Peng +8

CFCS, School of CS, PKU Beijing, China · BUPT Beijing, China · PKU Beijing, China +2

VLM Evaluation Spatial Reasoning Benchmarks Robotic Grasping 3D Spatial Reasoning Robot Manipulation Benchmarks 3D Visual Grounding

Ask PDF

Abstract

3D VQA

Explore similar work

RISE: Roadside Infrastructure Sequence Understanding across 3D Tracking and Structured Vision-Language Reasoning

OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning

Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models