cs.CVFeb 24, 2026
SaveUDVideoQA: A Traffic Video Question Answering Dataset for Multi-Object Spatio-Temporal Reasoning in Urban Dynamics
Organizations: Arizona State University
Abstract
Understanding the complex, multi-agent dynamics of urban traffic remains a fundamental challenge for video language models. This paper introduces Urban Dynamics VideoQA, a benchmark dataset that captures the unscripted real-world behavior of dynamic urban scenes. UDVideoQA is curated from 16 hours of traffic footage recorded at multiple city intersections under diverse traffic, weather, and lighting conditions. It employs an event-driven dynamic blur technique to ensure privacy preservation without compromising scene fidelity. Using a unified annotation pipeline, the dataset contains 28K question-answer pairs generated across 8 hours of densely annotated video, averaging one question per second. Its taxonomy follows a hierarchical reasoning level, spanning basic understanding and attribution to event reasoning, reverse reasoning, and counterfactual inference, enabling systematic evaluation of both visual grounding and causal reasoning. Comprehensive experiments benchmark 10 SOTA VideoLMs on UDVideoQA and 8 models on a complementary video question generation benchmark. Results reveal a persistent perception-reasoning gap, showing models that excel in abstract inference often fail with fundamental visual grounding. While models like Gemini Pro achieve the highest zero-shot accuracy, fine-tuning the smaller Qwen2.5-VL 7B model on UDVideoQA bridges this gap, achieving performance comparable to proprietary systems. In VideoQGen, Gemini 2.5 Pro, and Qwen3 Max generate the most relevant and complex questions, though all models exhibit limited linguistic diversity, underscoring the need for human-centric evaluation. The UDVideoQA suite, including the dataset, annotation tools, and benchmarks for both VideoQA and VideoQGen, provides a foundation for advancing robust, privacy-aware, and real-world multimodal reasoning. UDVideoQA is available at https://ud-videoqa.github.io/UD-VideoQA/UD-VideoQA/.
Figures & tables
Table 1: Comparison of UDVideoQA with existing traffic VideoQA datasets. It highlights the key features of various datasets, showcasing UDVideoQA ’s advancements in data scale, annotation density, and its unique focus on hallucination robustness and multi-event scenarios from a roadside perspective.
Figure 2: Illustrates the pipeline for creating the UDVideoQA dataset. The process begins with traffic video recording, which is segmented and temporally clipped into s clips. These clips undergo dynamic anonymity blurring. The QA taxonomy and generation module then uses model based on VideoQGen benchmark and prompt engineering to create a question pool, which undergoes rigorous human in the loop quality control and refinement.
| Model Name | Relevance | Answerability | Diversity | Pedestrian Centric | Vehicle Centric | Specific (Grounded) | Generic | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BU | Atr | ER | RR | CI | BU | Atr | ER | RR | CI | BU | Atr | ER | RR | CI | BU | Atr | ER | RR | CI | BU | Atr | ER | RR | CI | BU | Atr | ER | RR | CI | BU | Atr | ER | RR | CI | |
| Gemini 2.5 Pro | 92 | 84 | 82 | 86 | 80 | 90 | 80 | 80 | 84 | 82 | 64 | 62 | 66 | 64 | 60 | 35 | 30 | 45 | 40 | 50 | 55 | 60 | 50 | 55 | 45 | 70 | 80 | 65 | 60 | 55 | 30 | 20 | 35 | 40 | 45 |
| Qwen3 Max | 82 | 84 | 84 | 85 | 85 | 92 | 88 | 88 | 88 | 88 | 50 | 50 | 48 | 50 | 50 | 33 | 41 | 49 | 46 | 43 | 67 | 59 | 51 | 54 | 57 | 38 | 45 | 56 | 57 | 49 | 62 | 55 | 44 | 43 | 51 |
| Gemini Flash | 78 | 84 | 82 | 84 | 80 | 77 | 81 | 80 | 83 | 82 | 64 | 43 | 50 | 47 | 41 | 36 | 43 | 50 | 47 | 41 | 64 | 57 | 50 | 53 | 59 | 39 | 45 | 58 | 56 | 50 | 61 | 55 | 42 | 44 | 51 |
| Qwen3-VL-235B | 70 | 76 | 79 | 81 | 74 | 76 | 80 | 80 | 82 | 77 | 56 | 59 | 62 | 63 | 58 | 31 | 44 | 49 | 46 | 42 | 69 | 48 | 55 | 58 | 50 | 42 | 48 | 55 | 58 | 50 | 58 | 52 | 45 | 42 | 50 |
| Gemini 2.5 Flash | 74 | 70 | 72 | 68 | 72 | 86 | 85 | 84 | 83 | 85 | 34 | 34 | 34 | 33 | 33 | 38 | 44 | 49 | 46 | 42 | 62 | 56 | 51 | 54 | 58 | 41 | 47 | 55 | 57 | 46 | 59 | 53 | 45 | 43 | 54 |
Table 2: The VideoQGen Benchmark. Evaluating multimodal models across reasoning types, including basic understanding ( BU ), attribution ( Atr ), event reasoning ( ER ), reverse reasoning ( RR ), and counterfactual inference ( CI ). Gemini Pro leads overall across relevance, answerability, and diversity dimensions, showing strong consistency across all reasoning categories.
Figure 3: UDVideoQA dataset statistics. (a) Word frequency distribution by question type, (b) Distribution of question categories across semantic domains, including pedestrians, vehicles, and environmental signage. (c) Plot illustrating the spread of question sets across six contextual dimensions: pedestrian, temporal, vehicle, event, and signage, and (d) Frequency of questions containing key interrogative terms ( “what”, “where”, and “is” ) across reasoning categories.
| Morning | Afternoon | Evening | |||||||||||||||||
| Model Type | Model Name | BU | Atr | ER | RR | CI | Overall | BU | Atr | ER | RR | CI | Overall | BU | Atr | ER | RR | CI | Overall |
| Proprietary | Gemini 2.5 Pro | 90.00 | 22.22 | 88.89 | 86.10 | 91.70 | 75.78 | 78.27 | 65.44 | 75.42 | 73.81 | 72.56 | 73.10 | 87.39 | 50.00 | 62.09 | 76.37 | 79.80 | 71.13 |
| Gemini 2.5 Flash | 80.00 | 22.22 | 77.78 | 77.78 | 89.88 | 69.53 | 82.87 | 75.61 | 74.91 | 75.65 | 65.85 | 74.98 | 50.00 | 50.00 | 68.68 | 72.53 | 61.60 | 60.56 | |
| GPT-5 | 80.00 | 25.00 | 50.00 | 63.89 | 82.22 | 60.22 | 74.83 | 69.96 | 70.68 | 70.13 | 68.90 | 70.90 | 58.79 | 50.00 | 47.80 | 54.94 | 58.50 | 54.01 | |
| GPT-4o | 75.00 | 25.00 | 69.44 | 47.22 | 85.88 | 60.51 | 58.74 | 19.68 | 35.32 | 22.56 | 45.61 | 36.38 | 50.00 | 8.24 | 56.59 | 58.79 | 76.38 | 50.00 | |
| Open Source | Qwen 2.5 32B | 75.66 | 36.11 | 66.67 | 25.00 | 77.78 | 56.24 | 64.37 | 48.59 | 53.66 | 44.17 | 51.83 | 52.52 | 72.07 | 40.78 | 45.96 | 78.03 | 53.16 | 58.00 |
Table 3: The VideoQA Benchmark. It presents comparative performance across proprietary and open-source video reasoning models evaluated under varying illumination conditions: high intensity (afternoon), medium intensity (morning), and low intensity (evening). Overall results show that Gemini 2.5 Pro achieved the highest accuracy in the Morning (75.78%) and Evening (71.13%) conditions, while Gemini 2.5 Flash scored the highest in the Afternoon(74.98%). Llava-NeXT-Video exhibited the weakest overall performance.
Figure 4: Cross-model performance comparison on the UDVideoQA benchmark. Each bar shows the range from worst to best performance of VideoLMs used in this study.
| Dataset | Overall Score across QA Pairs | |
|---|---|---|
| Baseline | Fine Tuned | |
| RoadSocial [ 29 ] | 52.75 | 60 |
| SUTDTraffic [ 44 ] | 59.05 | 62.7 |
Table 4: Cross-dataset generalization results of Qwen2.5-VL-7B evaluated on RoadSocial and SUTDTrafficQA. The table reports the overall QA accuracy (in %) for the baseline model and after fine-tuning on UDVideoQA .
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Deployment sites of traffic cameras used for data collection. The selected intersections capture diverse lighting and weather conditions, providing representative coverage of real-world urban traffic environments.
Figure 6: Strategic placement of traffic-monitoring cameras showing varied intersection types, traffic densities, and environmental conditions.
Figure 7: Qualitative comparison of semantic and motion-based anonymization methods. (a) Original frame, (b) YOLOv11 + SAM2 semantic baseline, (c) IEBCS, (d) ESIM, and (e) V2E. The semantic approach (b) fails to blur the pedestrian highlighted in purple, whereas the event-based methods (c–e) accurately anonymize moving agents while preserving static background details.
| High Intensity Light | Low Intensity Light | |||||||
|---|---|---|---|---|---|---|---|---|
| Event Simulator | Fraction Blurred | Mean Edge Detection | Mean SSIM Reduction | CPU Processing time(s) | Fraction Blurred | Mean Edge Detection | Mean SSIM Reduction | CPU Processing Time(s) |
| Yolo + SAM2 | 0.28 | 0.36 | 0.22 | 1500 | 0.51 | 0.49 | 0.17 | 1500 |
| IEBCS | 0.24 | 0.30 | 0.15 | 105.3 | 0.37 | 0.39 | 0.13 | 104.6 |
| ESIM | 0.025 | 2.2 | 0.70 | 79.8 | 0.09 | 0.24 | 0.68 | 80 |
| V2E | 0.94 | 0.87 | 0.64 | 132 | 0.96 | 0.91 | 0.56 | 132 |
Table 5: Comparative evaluation of event-based simulators against the YOLOv11 + SAM2 semantic baseline for dynamic, privacy-preserving blurring. The selected method, IEBCS, achieves the best trade-off between targeted anonymization and preservation of scene detail, particularly under low-light conditions.
Figure 8: Representative examples from the UDVideoQA question–answer taxonomy.
Figure 9: Structured system prompt used for generating grounded and diverse question–answer pairs in the UDVideoQA benchmark.
Figure 10: Two-stage prompting pipeline used to generate structured and visually grounded QA pairs for the UDVideoQA dataset.
Figure 11: Overall performance of VideoQGen models across core metrics - relevance , answerability , and diversity . Gemini 2.5 Pro and Qwen3 Max achieve the highest scores, while GPT-4o performs consistently lower and generates less diverse questions.
| Model | Questions | ||
|---|---|---|---|
| Low Intensity Light | Medium Intensity Light | High Intensity Light | |
| Gemini 2.5 Pro | Are there any discernible architectural features on the buildings flanking the street? | What was the pedestrian doing as the dark car was approaching the crosswalk to make its turn? | Before the pedestrian in the dark shirt begins crossing the street at 0:07, what is the state of the intersection? |
| Qwen3 Max | Are there any temporary construction barriers or signs present on the sidewalk near the right edge of the frame? | If the traffic light for the horizontal street had remained red, would the black sedan have entered the intersection at seconds? | If the white van had begun to turn right at , which pedestrian group would it have encountered? |
| Gemini Flash | Counting all visible vehicles, both moving and stationary, how many are white? | Is it true that a cyclist uses the green-painted bike lane at any point in the video? | A person in a white shirt and dark pants enters from the bottom left. What path do they take? |
| Qwen3-VL-23B | Was there any person standing on the sidewalk near the orange traffic signs on the bottom right? | If the silver sedan had not entered the intersection at s, would the white autonomous vehicle have been blocked from turning left? | If the traffic light for the crosswalk had turned red at seconds, would the pedestrians have been in the middle of the street? |
| Gemini 2.5 Flash | Is it true that any pedestrian attempted to cross the street without using a designated crosswalk? | If the pedestrian at the bottom right had decided to cross at 0:03 , would they have entered the silver car’s path? | How many silver/grey passenger cars are moving away from the camera in the upper lanes? |
Table 6: Representative examples of generated questions across lighting conditions (low, medium, and high intensity). Gemini 2.5 Pro achieves the highest reasoning quality, while GPT-4o produces repetitive, low-diversity questions.
Figure 12: Distinct performance profiles of VideoQGen models across evaluation metrics. Gemini 2.5 Pro excels in specific (grounded) and relevance , while Qwen3 Max leads in answerability . The focus mean serves as a baseline for comparison.
Figure 13: User interface of the UDVideoQA Annotation Tool showing (top) the primary manual annotation workspace, (middle) real-time statistics for QA pairs, and (bottom) fine-grained blur tagging using YOLOv11 and SAM2.
Figure 14: Prompt template used for VideoQA model evaluation, instructing models to generate factual, visually grounded answers based solely on the provided video content.
Figure 15: Grading prompt used for VideoQA benchmark evaluation. The LLM-based grader enforces strict, non-inferential correctness through canonicalized categories, subtype detection, and category-weighted scoring.
Figure 16: Average VideoQA performance of proprietary and open-source models across five reasoning categories under different lighting conditions: (a) day, (b) afternoon, and (c) nighttime scenarios.
Figure 17: Comparison of four leading multimodal models ( Gemini 2.5 Pro , Gemini 2.5 Flash , GPT-4o , and Qwen 2.5 7B ) across five VideoQA reasoning dimensions, highlighting variations in perceptual and causal understanding.
| Model Type | Model Name | Pedestrian-centric | Vehicular-centric | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BU | Atr | ER | RR | CI | Overall | BU | Atr | ER | RR | CI | Overall | ||
| Proprietary | Gemini 2.5 Pro | 93.43 | 83.03 | 73.32 | 76.39 | 92.81 | 87.90 | 66.67 | 33.84 | 69.96 | 77.10 | 84.72 | 68.59 |
| Gemini 2.5 Flash | 68.93 | 87.27 | 72.12 | 78.12 | 94.77 | 80.21 | 71.11 | 38.62 | 72.46 | 73.68 | 78.60 | 67.93 | |
| GPT-5 | 68.93 | 80.61 | 72.12 | 76.39 | 92.16 | 78.01 | 71.11 | 39.49 | 55.13 | 59.75 | 82.10 | 60.21 | |
| GPT-4o | 59.42 | 25.45 | 56.49 | 28.91 | 83.01 | 55.29 | 42.22 | 7.32 | 47.25 | 44.58 | 71.53 | 44.56 | |
| Open Source | Qwen 2.5 32B | 87.90 | 81.82 | 59.38 | 41.38 | 86.93 | 80.07 | 53.33 | 63.48 | 47.95 | 54.49 | 65.07 | 56.74 |
Table 7: Comparison of model performance across pedestrian-centric and vehicular-centric scenarios. Color intensity indicates relative accuracy within each reasoning subcategory.
| Model | Gemini 2.5 Pro | Gemini 2.5 Flash | GPT-4o | GPT-5 | Qwen 2.5 VL - 7B (Fine Tuned) | Qwen 2.5 VL - 7B | InternVL3-38B | Qwen 2.5 VL 32B | VideoLlama3 | NVILA 8B | Llava-NeXT-Video |
| Question | Is there any overtaking observed? | ||||||||||
| Ground Truth | No | ||||||||||
| Best Answer | No overtaking is observed as there are no moving vehicles | No | No | No | No overtaking is observed in the image. | No, there is no overtaking observed […]. The road appears to be empty with no vehicles in motion […] | No, there is no overtaking observed. | No, there is no overtaking observed […]. The scene shows an empty road with no vehicles or pedestrians present. […] | The video does not show any overtaking. | No overtaking is observed in the image. | Based on the image provided, it appears that there is no over- taking place. […] |
| Question | How many cars exited the foreground? | ||||||||||
| Ground Truth | Four | ||||||||||
| Worst Answer | Zero cars exited the foreground as the car moved away from the camera. | One car exited the fore- ground | Two | 0 | One car exited the fore-ground in the image. | […] there is one car that appears to have just exited the foreground on the right side […] | One car exited the foreground | […] there is one car that exits the foreground. It can be seen moving from the right side […] eventually exiting the frame. | […] it appears that one car has exited the foreground. The road is mostly empty […] | There is one car that exited the foreground. | There are three cars in the foreground of the image. |
Table 8: Representative examples of best and worst model-generated answers under the daytime scenario.
| Model | Gemini 2.5 Pro | Gemini 2.5 Flash | GPT-4o | GPT-5 | Qwen 2.5 VL - 7B (Fine Tuned) | Qwen 2.5 VL - 7B | InternVL3-38B | Qwen 2.5 VL 32B | VideoLlama3 | NVILA 8B | Llava-NeXT-Video |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Question | Is the road surface wet from rain? | ||||||||||
| Ground Truth | No the road looks dry | ||||||||||
| Best Answer | No, the road surface is not wet from rain; it appears completely dry. | No, the road surface appears dry, not wet from rain. | No | No | No, the road surfaceis not wet from rain. The road appears dry and clear of any water. | No, there is no wetness from rain observed […]. The road appears to be dry with no visible signs of water […] | No, the road surface appears dry. | No, there is no wetness from rain observed […]. The road surface appears to be dry with no visible signs of water or reflections […] | The road surface appears dry, with no visible signs of rain or wetness. | No, the road surface is not wet from rain. | No, there is no wetness from rain observed […]. The road surface appears to be dry and clear with it not raining at the time […] |
| Question | What large marking is painted in the central lane near the camera? | ||||||||||
| Ground Truth | A white ‘X’ road marking | ||||||||||
| Worst Answer | A white bicycle symbol. | A large diamond shape (?) is painted in the central lane. | Straight white arrow | A large white marking is painted in the central lane. | The large marking painted in the central lane near the camera is a white arrow pointing to the right | The large marking appears to be a white arrow pointing to the right with it indicating a turn lane […] | The large marking painted in the central lane near the camera is a ’no parking’ sign | The large marking appears to be a ”Bus Only” or ”HOV” lane marking with this being indicated by the red color […] | The large marking appears to be a white arrow pointing to the right with it indicating a lane change or turn direction […] | The large marking painted in the central lane near the camera is a white arrow. | The large marking painted in the central lane near the camera is a ”No Parking” sign. |
Table 9: Representative examples of best and worst model-generated answers under the afternoon scenario.
| Model | Gemini 2.5 Pro | Gemini 2.5 Flash | GPT-4o | GPT-5 | Qwen 2.5 VL - 7B (Fine Tuned) | Qwen 2.5 VL - 7B | InternVL3-38B | Qwen 2.5 VL 32B | VideoLlama3 | NVILA 8B | Llava-NeXT-Video |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Question | Is the road surface wet from rain? | ||||||||||
| Ground Truth | No the road looks dry | ||||||||||
| Best Answer | No, the road surface is not wet from rain; it appears completely dry. | No, the road surface appears dry, not wet from rain. | No | No | No, the road surfaceis not wet from rain. The road appears dry and clear of any water. | No, there is no wetness from rain observed […]. The road appears to be dry with no visible signs of water […] | No, the road surface appears dry. | No, there is no wetness from rain observed […]. The road surface appears to be dry with no visible signs of water or reflections […] | The road surface appears dry, with no visible signs of rain or wetness. | No, the road surface is not wet from rain. | No, there is no wetness from rain observed […]. The road surface appears to be dry and clear with it not raining at the time […] |
| Question | What large marking is painted in the central lane near the camera? | ||||||||||
| Ground Truth | A white ‘X’ road marking | ||||||||||
| Worst Answer | A white bicycle symbol. | A large diamond shape (?) is painted in the central lane. | Straight white arrow | A large white marking is painted in the central lane. | The large marking painted in the central lane near the camera is a white arrow pointing to the right | The large marking appears to be a white arrow pointing to the right with it indicating a turn lane […] | The large marking painted in the central lane near the camera is a ’no parking’ sign | The large marking appears to be a ”Bus Only” or ”HOV” lane marking with this being indicated by the red color […] | The large marking appears to be a white arrow pointing to the right with it indicating a lane change or turn direction […] | The large marking painted in the central lane near the camera is a white arrow. | The large marking painted in the central lane near the camera is a ”No Parking” sign. |
Table 10: Representative examples of best and worst model-generated answers under the evening scenario
Figure 18: Analysis of multi-agent dynamics: (a) most frequent multi-agent scenario observed in the dataset, (b) temporal stability of pedestrian activity across video segments, and (c) proximity density plots highlighting frequent close interactions between vehicle–pedestrian and pedestrian–pedestrian pairs.
Figure 19: Example frame illustrating the model’s use of visual evidence, specifically the “AP” license plate prefix-to support its prediction in the RoadSocial dataset.
| QA-Group | |||||
| Training Configuration | Factual | Complex | Imaginative | Adversarial | Overall Score |
| Fine Tuned Model | 68.2 | 41.8 | 74.4 | 55 | 60 |
| Baseline | 66.2 | 3 | 83.7 | 58.1 | 52.75 |
Table 11: Evaluation of the UDVideoQA fine-tuned model’s generalization performance on the RoadSocial dataset across four QA categories - factual , complex , imaginative , and adversarial .
| QA-Group | |||||
|---|---|---|---|---|---|
| Training Configuration | BU | Att | CI | ER | Overall Score |
| Fine Tuned Model | 54.5 | 73.3 | 75 | 48 | 62.7 |
| Baseline | 48.9 | 69.6 | 73.4 | 44 | 59.05 |
Table 12: Evaluation of the UDVideoQA fine-tuned model’s generalization performance on the SUTDTraffic dataset across four QA categories factual , complex , imaginative , and adversarial .
Figure 20: Example VideoQGen outputs across lighting conditions.
Figure 21: Example of a reverse reasoning question evaluating fine-grained temporal understanding. Models must track two distinct objects and determine their correct sequential order. Gemini 2.5 Pro provides the most accurate response by focusing on the temporal relationships within the sequence.
Figure 22: . Example of an event reasoning challenge illustrating model failures in temporal grounding. Although the ground truth indicates that no vehicle approaches the camera during the final three seconds, several models (e.g., GPT and LLaVA-NeXT-Video ) incorrectly respond “Yes”, revealing hallucinated motion and poor temporal localization.
Figure 23: Example of an attribution question assessing a specific visual attribute, the number of stopped vehicles at the start of the clip. Most models correctly identify zero vehicles, consistent with the ground truth.
Figure 24: Example of a basic understanding question evaluating simple environmental perception, specifically, road surface wetness. All models correctly identify the road as dry, demonstrating high accuracy on this foundational reasoning task.
Figure 25: Example of a counterfactual inference question evaluating a model’s ability to reject a false premise. While most models correctly classify the statement as false, LLaVA-NeXT incorrectly confirms the non-existent event, revealing a failure in precise factual verification.