UDVideoQA: A Traffic Video Question Answering Dataset for Multi-Object Spatio-Temporal Reasoning in Urban Dynamics

Joseph Raj Vishal; Nagasiri Poluri; Katha Naik; Rutuja Patil; Kashyap Hegde Kota; Krishna Vinod; Prithvi Jai Ramesh; Mohammad Farhadi

cs.CVFeb 24, 2026

UDVideoQA: A Traffic Video Question Answering Dataset for Multi-Object Spatio-Temporal Reasoning in Urban Dynamics

Authors: Joseph Raj Vishal, Nagasiri Poluri, Katha Naik, Rutuja Patil, Kashyap Hegde Kota, Krishna Vinod, Prithvi Jai Ramesh, Mohammad Farhadi, +2 more

Organizations: Arizona State University

Abstract

Understanding the complex, multi-agent dynamics of urban traffic remains a fundamental challenge for video language models. This paper introduces Urban Dynamics VideoQA, a benchmark dataset that captures the unscripted real-world behavior of dynamic urban scenes. UDVideoQA is curated from 16 hours of traffic footage recorded at multiple city intersections under diverse traffic, weather, and lighting conditions. It employs an event-driven dynamic blur technique to ensure privacy preservation without compromising scene fidelity. Using a unified annotation pipeline, the dataset contains 28K question-answer pairs generated across 8 hours of densely annotated video, averaging one question per second. Its taxonomy follows a hierarchical reasoning level, spanning basic understanding and attribution to event reasoning, reverse reasoning, and counterfactual inference, enabling systematic evaluation of both visual grounding and causal reasoning. Comprehensive experiments benchmark 10 SOTA VideoLMs on UDVideoQA and 8 models on a complementary video question generation benchmark. Results reveal a persistent perception-reasoning gap, showing models that excel in abstract inference often fail with fundamental visual grounding. While models like Gemini Pro achieve the highest zero-shot accuracy, fine-tuning the smaller Qwen2.5-VL 7B model on UDVideoQA bridges this gap, achieving performance comparable to proprietary systems. In VideoQGen, Gemini 2.5 Pro, and Qwen3 Max generate the most relevant and complex questions, though all models exhibit limited linguistic diversity, underscoring the need for human-centric evaluation. The UDVideoQA suite, including the dataset, annotation tools, and benchmarks for both VideoQA and VideoQGen, provides a foundation for advancing robust, privacy-aware, and real-world multimodal reasoning. UDVideoQA is available at https://ud-videoqa.github.io/UD-VideoQA/UD-VideoQA/.

Figures & tables

Panel of Table 1 — Table 1: Comparison of UDVideoQA with existing traffic VideoQA datasets. It highlights the key features of various datasets, showcasing UDVideoQA ’s advancements in data scale, annotation density, and its unique focus on hallucination robustness and multi-event scenarios from a roadside perspective.

Figure 2: Illustrates the pipeline for creating the UDVideoQA dataset. The process begins with traffic video recording, which is segmented and temporally clipped into $10$ s clips. These clips undergo dynamic anonymity blurring. The QA taxonomy and generation module then uses model based on VideoQGen benchmark and prompt engineering to create a question pool, which undergoes rigorous human in the loop quality control and refinement.

Model Name	Relevance					Answerability					Diversity					Pedestrian Centric					Vehicle Centric					Specific (Grounded)					Generic
	BU	Atr	ER	RR	CI	BU	Atr	ER	RR	CI	BU	Atr	ER	RR	CI	BU	Atr	ER	RR	CI	BU	Atr	ER	RR	CI	BU	Atr	ER	RR	CI	BU	Atr	ER	RR	CI
Gemini 2.5 Pro	92	84	82	86	80	90	80	80	84	82	64	62	66	64	60	35	30	45	40	50	55	60	50	55	45	70	80	65	60	55	30	20	35	40	45
Qwen3 Max	82	84	84	85	85	92	88	88	88	88	50	50	48	50	50	33	41	49	46	43	67	59	51	54	57	38	45	56	57	49	62	55	44	43	51
Gemini Flash	78	84	82	84	80	77	81	80	83	82	64	43	50	47	41	36	43	50	47	41	64	57	50	53	59	39	45	58	56	50	61	55	42	44	51
Qwen3-VL-235B	70	76	79	81	74	76	80	80	82	77	56	59	62	63	58	31	44	49	46	42	69	48	55	58	50	42	48	55	58	50	58	52	45	42	50
Gemini 2.5 Flash	74	70	72	68	72	86	85	84	83	85	34	34	34	33	33	38	44	49	46	42	62	56	51	54	58	41	47	55	57	46	59	53	45	43	54

Table 2: The VideoQGen Benchmark. Evaluating multimodal models across reasoning types, including basic understanding ( BU ), attribution ( Atr ), event reasoning ( ER ), reverse reasoning ( RR ), and counterfactual inference ( CI ). Gemini

2.5

Pro leads overall across relevance, answerability, and diversity dimensions, showing strong consistency across all reasoning categories.

Figure 3: UDVideoQA dataset statistics. (a) Word frequency distribution by question type, (b) Distribution of question categories across semantic domains, including pedestrians, vehicles, and environmental signage. (c) Plot illustrating the spread of question sets across six contextual dimensions: pedestrian, temporal, vehicle, event, and signage, and (d) Frequency of questions containing key interrogative terms ( “what”, “where”, and “is” ) across reasoning categories.

		Morning						Afternoon						Evening
Model Type	Model Name	BU	Atr	ER	RR	CI	Overall	BU	Atr	ER	RR	CI	Overall	BU	Atr	ER	RR	CI	Overall
Proprietary	Gemini 2.5 Pro	90.00	22.22	88.89	86.10	91.70	75.78	78.27	65.44	75.42	73.81	72.56	73.10	87.39	50.00	62.09	76.37	79.80	71.13
	Gemini 2.5 Flash	80.00	22.22	77.78	77.78	89.88	69.53	82.87	75.61	74.91	75.65	65.85	74.98	50.00	50.00	68.68	72.53	61.60	60.56
	GPT-5	80.00	25.00	50.00	63.89	82.22	60.22	74.83	69.96	70.68	70.13	68.90	70.90	58.79	50.00	47.80	54.94	58.50	54.01
	GPT-4o	75.00	25.00	69.44	47.22	85.88	60.51	58.74	19.68	35.32	22.56	45.61	36.38	50.00	8.24	56.59	58.79	76.38	50.00
Open Source	Qwen 2.5 32B	75.66	36.11	66.67	25.00	77.78	56.24	64.37	48.59	53.66	44.17	51.83	52.52	72.07	40.78	45.96	78.03	53.16	58.00

Table 3: The VideoQA Benchmark. It presents comparative performance across proprietary and open-source video reasoning models evaluated under varying illumination conditions: high intensity (afternoon), medium intensity (morning), and low intensity (evening). Overall results show that Gemini 2.5 Pro achieved the highest accuracy in the Morning (75.78%) and Evening (71.13%) conditions, while Gemini 2.5 Flash scored the highest in the Afternoon(74.98%). Llava-NeXT-Video exhibited the weakest overall performance.

Figure 4: Cross-model performance comparison on the UDVideoQA benchmark. Each bar shows the range from worst to best performance of VideoLMs used in this study.

Dataset	Overall Score across QA Pairs
Dataset	Baseline	Fine Tuned
RoadSocial [ 29 ]	52.75	60
SUTDTraffic [ 44 ]	59.05	62.7

Table 4: Cross-dataset generalization results of Qwen2.5-VL-7B evaluated on RoadSocial and SUTDTrafficQA. The table reports the overall QA accuracy (in %) for the baseline model and after fine-tuning on UDVideoQA .

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Deployment sites of traffic cameras used for data collection. The selected intersections capture diverse lighting and weather conditions, providing representative coverage of real-world urban traffic environments.

Figure 6: Strategic placement of traffic-monitoring cameras showing varied intersection types, traffic densities, and environmental conditions.

Figure 7: Qualitative comparison of semantic and motion-based anonymization methods. (a) Original frame, (b) YOLOv11 + SAM2 semantic baseline, (c) IEBCS, (d) ESIM, and (e) V2E. The semantic approach (b) fails to blur the pedestrian highlighted in purple, whereas the event-based methods (c–e) accurately anonymize moving agents while preserving static background details.

	High Intensity Light				Low Intensity Light
Event Simulator	Fraction Blurred	Mean Edge Detection	Mean SSIM Reduction	CPU Processing time(s)	Fraction Blurred	Mean Edge Detection	Mean SSIM Reduction	CPU Processing Time(s)
Yolo + SAM2	0.28	0.36	0.22	1500	0.51	0.49	0.17	1500
IEBCS	0.24	0.30	0.15	105.3	0.37	0.39	0.13	104.6
ESIM	0.025	2.2	0.70	79.8	0.09	0.24	0.68	80
V2E	0.94	0.87	0.64	132	0.96	0.91	0.56	132

Appendix

Table 5: Comparative evaluation of event-based simulators against the YOLOv11 + SAM2 semantic baseline for dynamic, privacy-preserving blurring. The selected method, IEBCS, achieves the best trade-off between targeted anonymization and preservation of scene detail, particularly under low-light conditions.

Panel of Figure 8 — Figure 8: Representative examples from the UDVideoQA question–answer taxonomy.

Panel of Figure 9 — Figure 9: Structured system prompt used for generating grounded and diverse question–answer pairs in the UDVideoQA benchmark.

Panel of Figure 10 — Figure 10: Two-stage prompting pipeline used to generate structured and visually grounded QA pairs for the UDVideoQA dataset.

Figure 11: Overall performance of VideoQGen models across core metrics - relevance , answerability , and diversity . Gemini 2.5 Pro and Qwen3 Max achieve the highest scores, while GPT-4o performs consistently lower and generates less diverse questions.

Model	Questions
	Low Intensity Light	Medium Intensity Light	High Intensity Light
Gemini 2.5 Pro	Are there any discernible architectural features on the buildings flanking the street?	What was the pedestrian doing as the dark car was approaching the crosswalk to make its turn?	Before the pedestrian in the dark shirt begins crossing the street at 0:07, what is the state of the intersection?
Qwen3 Max	Are there any temporary construction barriers or signs present on the sidewalk near the right edge of the frame?	If the traffic light for the horizontal street had remained red, would the black sedan have entered the intersection at $0.6$ seconds?	If the white van had begun to turn right at $0:02$ , which pedestrian group would it have encountered?
Gemini Flash	Counting all visible vehicles, both moving and stationary, how many are white?	Is it true that a cyclist uses the green-painted bike lane at any point in the video?	A person in a white shirt and dark pants enters from the bottom left. What path do they take?
Qwen3-VL-23B	Was there any person standing on the sidewalk near the orange traffic signs on the bottom right?	If the silver sedan had not entered the intersection at $0.3$ s, would the white autonomous vehicle have been blocked from turning left?	If the traffic light for the crosswalk had turned red at $5$ seconds, would the pedestrians have been in the middle of the street?
Gemini 2.5 Flash	Is it true that any pedestrian attempted to cross the street without using a designated crosswalk?	If the pedestrian at the bottom right had decided to cross at 0:03 , would they have entered the silver car’s path?	How many silver/grey passenger cars are moving away from the camera in the upper lanes?

Appendix

Table 6: Representative examples of generated questions across lighting conditions (low, medium, and high intensity). Gemini 2.5 Pro achieves the highest reasoning quality, while GPT-4o produces repetitive, low-diversity questions.

Figure 12: Distinct performance profiles of VideoQGen models across evaluation metrics. Gemini 2.5 Pro excels in specific (grounded) and relevance , while Qwen3 Max leads in answerability . The focus mean serves as a baseline for comparison.

Figure 13: User interface of the UDVideoQA Annotation Tool showing (top) the primary manual annotation workspace, (middle) real-time statistics for QA pairs, and (bottom) fine-grained blur tagging using YOLOv11 and SAM2.

Panel of Figure 14 — Figure 14: Prompt template used for VideoQA model evaluation, instructing models to generate factual, visually grounded answers based solely on the provided video content.

Panel of Figure 15 — Figure 15: Grading prompt used for VideoQA benchmark evaluation. The LLM-based grader enforces strict, non-inferential correctness through canonicalized categories, subtype detection, and category-weighted scoring.

Figure 16: Average VideoQA performance of proprietary and open-source models across five reasoning categories under different lighting conditions: (a) day, (b) afternoon, and (c) nighttime scenarios.

Figure 17: Comparison of four leading multimodal models ( Gemini 2.5 Pro , Gemini 2.5 Flash , GPT-4o , and Qwen 2.5 7B ) across five VideoQA reasoning dimensions, highlighting variations in perceptual and causal understanding.

Model Type	Model Name	Pedestrian-centric						Vehicular-centric
Model Type	Model Name	BU	Atr	ER	RR	CI	Overall	BU	Atr	ER	RR	CI	Overall
Proprietary	Gemini 2.5 Pro	93.43	83.03	73.32	76.39	92.81	87.90	66.67	33.84	69.96	77.10	84.72	68.59
	Gemini 2.5 Flash	68.93	87.27	72.12	78.12	94.77	80.21	71.11	38.62	72.46	73.68	78.60	67.93
	GPT-5	68.93	80.61	72.12	76.39	92.16	78.01	71.11	39.49	55.13	59.75	82.10	60.21
	GPT-4o	59.42	25.45	56.49	28.91	83.01	55.29	42.22	7.32	47.25	44.58	71.53	44.56
Open Source	Qwen 2.5 32B	87.90	81.82	59.38	41.38	86.93	80.07	53.33	63.48	47.95	54.49	65.07	56.74

Appendix

Table 7: Comparison of model performance across pedestrian-centric and vehicular-centric scenarios. Color intensity indicates relative accuracy within each reasoning subcategory.

Model	Gemini 2.5 Pro	Gemini 2.5 Flash	GPT-4o	GPT-5	Qwen 2.5 VL - 7B (Fine Tuned)	Qwen 2.5 VL - 7B	InternVL3-38B	Qwen 2.5 VL 32B	VideoLlama3	NVILA 8B	Llava-NeXT-Video
Question	Is there any overtaking observed?
Ground Truth	No
Best Answer	No overtaking is observed as there are no moving vehicles	No	No	No	No overtaking is observed in the image.	No, there is no overtaking observed […]. The road appears to be empty with no vehicles in motion […]	No, there is no overtaking observed.	No, there is no overtaking observed […]. The scene shows an empty road with no vehicles or pedestrians present. […]	The video does not show any overtaking.	No overtaking is observed in the image.	Based on the image provided, it appears that there is no over- taking place. […]
Question	How many cars exited the foreground?
Ground Truth	Four
Worst Answer	Zero cars exited the foreground as the car moved away from the camera.	One car exited the fore- ground	Two	0	One car exited the fore-ground in the image.	[…] there is one car that appears to have just exited the foreground on the right side […]	One car exited the foreground	[…] there is one car that exits the foreground. It can be seen moving from the right side […] eventually exiting the frame.	[…] it appears that one car has exited the foreground. The road is mostly empty […]	There is one car that exited the foreground.	There are three cars in the foreground of the image.

Appendix

Table 8: Representative examples of best and worst model-generated answers under the daytime scenario.

Model	Gemini 2.5 Pro	Gemini 2.5 Flash	GPT-4o	GPT-5	Qwen 2.5 VL - 7B (Fine Tuned)	Qwen 2.5 VL - 7B	InternVL3-38B	Qwen 2.5 VL 32B	VideoLlama3	NVILA 8B	Llava-NeXT-Video
Question	Is the road surface wet from rain?
Ground Truth	No the road looks dry
Best Answer	No, the road surface is not wet from rain; it appears completely dry.	No, the road surface appears dry, not wet from rain.	No	No	No, the road surfaceis not wet from rain. The road appears dry and clear of any water.	No, there is no wetness from rain observed […]. The road appears to be dry with no visible signs of water […]	No, the road surface appears dry.	No, there is no wetness from rain observed […]. The road surface appears to be dry with no visible signs of water or reflections […]	The road surface appears dry, with no visible signs of rain or wetness.	No, the road surface is not wet from rain.	No, there is no wetness from rain observed […]. The road surface appears to be dry and clear with it not raining at the time […]
Question	What large marking is painted in the central lane near the camera?
Ground Truth	A white ‘X’ road marking
Worst Answer	A white bicycle symbol.	A large diamond shape (?) is painted in the central lane.	Straight white arrow	A large white marking is painted in the central lane.	The large marking painted in the central lane near the camera is a white arrow pointing to the right	The large marking appears to be a white arrow pointing to the right with it indicating a turn lane […]	The large marking painted in the central lane near the camera is a ’no parking’ sign	The large marking appears to be a ”Bus Only” or ”HOV” lane marking with this being indicated by the red color […]	The large marking appears to be a white arrow pointing to the right with it indicating a lane change or turn direction […]	The large marking painted in the central lane near the camera is a white arrow.	The large marking painted in the central lane near the camera is a ”No Parking” sign.

Appendix

Table 9: Representative examples of best and worst model-generated answers under the afternoon scenario.

Model	Gemini 2.5 Pro	Gemini 2.5 Flash	GPT-4o	GPT-5	Qwen 2.5 VL - 7B (Fine Tuned)	Qwen 2.5 VL - 7B	InternVL3-38B	Qwen 2.5 VL 32B	VideoLlama3	NVILA 8B	Llava-NeXT-Video
Question	Is the road surface wet from rain?
Ground Truth	No the road looks dry
Best Answer	No, the road surface is not wet from rain; it appears completely dry.	No, the road surface appears dry, not wet from rain.	No	No	No, the road surfaceis not wet from rain. The road appears dry and clear of any water.	No, there is no wetness from rain observed […]. The road appears to be dry with no visible signs of water […]	No, the road surface appears dry.	No, there is no wetness from rain observed […]. The road surface appears to be dry with no visible signs of water or reflections […]	The road surface appears dry, with no visible signs of rain or wetness.	No, the road surface is not wet from rain.	No, there is no wetness from rain observed […]. The road surface appears to be dry and clear with it not raining at the time […]
Question	What large marking is painted in the central lane near the camera?
Ground Truth	A white ‘X’ road marking
Worst Answer	A white bicycle symbol.	A large diamond shape (?) is painted in the central lane.	Straight white arrow	A large white marking is painted in the central lane.	The large marking painted in the central lane near the camera is a white arrow pointing to the right	The large marking appears to be a white arrow pointing to the right with it indicating a turn lane […]	The large marking painted in the central lane near the camera is a ’no parking’ sign	The large marking appears to be a ”Bus Only” or ”HOV” lane marking with this being indicated by the red color […]	The large marking appears to be a white arrow pointing to the right with it indicating a lane change or turn direction […]	The large marking painted in the central lane near the camera is a white arrow.	The large marking painted in the central lane near the camera is a ”No Parking” sign.

Appendix

Table 10: Representative examples of best and worst model-generated answers under the evening scenario

Figure 18: Analysis of multi-agent dynamics: (a) most frequent multi-agent scenario observed in the dataset, (b) temporal stability of pedestrian activity across video segments, and (c) proximity density plots highlighting frequent close interactions between vehicle–pedestrian and pedestrian–pedestrian pairs.

Figure 19: Example frame illustrating the model’s use of visual evidence, specifically the “AP” license plate prefix-to support its prediction in the RoadSocial dataset.

	QA-Group
Training Configuration	Factual	Complex	Imaginative	Adversarial	Overall Score
Fine Tuned Model	68.2	41.8	74.4	55	60
Baseline	66.2	3	83.7	58.1	52.75

Appendix

Table 11: Evaluation of the UDVideoQA fine-tuned model’s generalization performance on the RoadSocial dataset across four QA categories - factual , complex , imaginative , and adversarial .

	QA-Group
Training Configuration	BU	Att	CI	ER	Overall Score
Fine Tuned Model	54.5	73.3	75	48	62.7
Baseline	48.9	69.6	73.4	44	59.05

Appendix

Table 12: Evaluation of the UDVideoQA fine-tuned model’s generalization performance on the SUTDTraffic dataset across four QA categories factual , complex , imaginative , and adversarial .

Panel of Figure 20 — Figure 20: Example VideoQGen outputs across lighting conditions.

Figure 21: Example of a reverse reasoning question evaluating fine-grained temporal understanding. Models must track two distinct objects and determine their correct sequential order. Gemini 2.5 Pro provides the most accurate response by focusing on the temporal relationships within the sequence.

Figure 22: . Example of an event reasoning challenge illustrating model failures in temporal grounding. Although the ground truth indicates that no vehicle approaches the camera during the final three seconds, several models (e.g., GPT and LLaVA-NeXT-Video ) incorrectly respond “Yes”, revealing hallucinated motion and poor temporal localization.

Figure 23: Example of an attribution question assessing a specific visual attribute, the number of stopped vehicles at the start of the clip. Most models correctly identify zero vehicles, consistent with the ground truth.

Figure 24: Example of a basic understanding question evaluating simple environmental perception, specifically, road surface wetness. All models correctly identify the road as dry, demonstrating high accuracy on this foundational reasoning task.

Figure 25: Example of a counterfactual inference question evaluating a model’s ability to reject a false premise. While most models correctly classify the statement as false, LLaVA-NeXT incorrectly confirms the non-existent event, revealing a failure in precise factual verification.

Explore similar work

Aug 13, 2026

UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations

Traffic video understanding has become an important problem in intelligent transportation, as road videos provide direct evidence for accidents, violations, and interactions between vehicles and vulnerable road users. A useful system should explain how a traffic event develops, why it happens, and when the relevant interaction occurs, yet this remains difficult for multimodal large language models (MLLMs) because traffic videos contain sparse events and varied viewpoints. We introduce UniTraffic-Agent, the MR-CAS solution for Track~3 of the 10th AI City Challenge, which includes Traffic Anomaly Reasoning (TAR) and two out-of-domain evaluations: FETV for fisheye traffic events and PSI-VQA for pedestrian intention reasoning. UniTraffic-Agent follows an observe--reason--act--verify workflow that samples timestamped visual evidence, reasons over all questions from the same clip in one request, and converts responses through task-specific action adapters. On the official Public leaderboards, MR-CAS ranks 16th on TAR with a score of 0.5780, 2nd on FETV with 0.4884, and 4th on PSI-VQA with 64.4161. The code is available at https://github.com/Roclp/UniTraffic-Agent.

Peng Li, Qianqian Xu, Shilong Bao +2

School of Computer Science and Technology, University of Chinese Academy of Sciences (UCAS), Beijing, China · State Key Laboratory of AI Safety, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China +1

Pedestrian Intention Prediction Intelligent Transportation Systems Video Reasoning VLM Reasoning Multimodal Agents Anomaly Detection

Ask PDF

Apr 4, 2026

SGTA: Scene-Graph Based Multi-Modal Traffic Agent for Video Understanding

We present Scene-Graph Based Multi-Modal Traffic Agent (SGTA), a modular framework for traffic video understanding that combines structured scene graphs with multi-modal reasoning. It constructs a traffic scene graph from roadside videos using detection, tracking, and lane extraction, followed by tool-based reasoning over both symbolic graph queries and visual inputs. SGTA adopts ReAct to process interleaved reasoning traces from large language models with tool invocations, enabling interpretable decision-making for complex video questions. Experiments on selected TUMTraffic VideoQA dataset sample demonstrate that SGTA achieves competitive accuracy across multiple question types while providing transparent reasoning steps. These results highlight the potential of integrating structured scene representations with multi-modal agents for traffic video understanding.

Xingcheng Zhou, Mingyu Liu, Walter Zimmer +2

Technical University of Munich, Chair of Robotics, Artificial Intelligence (AI) and Embedded Systems, Boltzmannstraße 3, 85748 Garching bei München, Germany

Video QA Visual Question Answering Video Reasoning Intelligent Transportation Systems Multimodal Agents Graph Reasoning

Ask PDF

Jun 14, 2026

OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning

Traffic scene understanding requires models to reason beyond object recognition, including lane topology, multi-view geometry, temporal evolution, and signal-phase semantics. However, existing traffic-oriented multimodal benchmarks largely emphasize passive visual recognition or isolated video understanding, offering limited support for evaluating structure-aware traffic reasoning under controlled conditions. We introduce OmniTraffic, a controllable generation pipeline and benchmark for spatio-temporal traffic reasoning. Built around 12 real-world intersections reconstructed into editable 3D traffic environments and complemented by surveillance footage from two countries, OmniTraffic supports both controlled and natural-condition evaluation. It defines a three-level task hierarchy spanning scene perception, multi-view and temporal reasoning, and decision support. Using structured traffic metadata, OmniTraffic generates synchronized multi-view VQA samples covering vehicle states, lane functions, view--BEV correspondence, temporal dynamics, and signal-phase analysis, resulting in 8M VQA samples and a 3K human-verified test set. Evaluation of eleven frontier MLLMs reveals a large human--model gap, with the most pronounced failures in topology-grounded and spatio-temporal reasoning tasks. Fine-tuning a lightweight MLLM on simulated OmniTraffic data further improves performance on real-world traffic scenes, demonstrating the value of simulation-generated supervision for traffic-specific multimodal reasoning. Beyond a fixed dataset, OmniTraffic provides an extensible pipeline with configurable intersections, camera views, traffic demands, signal phases, visual conditions, and rare events.

Maonan Wang, Zhengyan Huang, Kemou Jiang +13

The Chinese University of Hong Kong, Shenzhen · Shanghai AI Lab · Beihang University +3

Autonomous Driving Datasets Spatiotemporal Reasoning Autonomous Driving Perception Multimodal Reasoning Multimodal Model Evaluation

Ask PDF