TAU-R1: Visual Language Model for Traffic Anomaly Understanding
Authors: Yuqiang Lin, Kehua Chen, Sam Lockyer, Arjun Yadav, Mingxuan Sui, Shucheng Zhang, Yan Shi, Bingzhang Wang, +7 more
Organizations: University of Bath, BA2 7AY, United Kingdom · University of Washington, WA 98105, USA · City of Carmel, Indiana, USA · Starwit GmbH, Germany
Traffic Anomaly Understanding (TAU) is important for traffic safety in Intelligent Transportation Systems. Recent vision-language models (VLMs) have shown strong capabilities in video understanding. However, progress on TAU remains limited due to the lack of benchmarks and task-specific methodologies. To address this limitation, we introduce Roundabout-TAU, a dataset constructed from real-world roundabout videos collected in collaboration with the City of Carmel, Indiana. The dataset contains 342 clips and is annotated with more than 2,000 question-answer pairs covering multiple aspects of traffic anomaly understanding. Building on this benchmark, we propose TAU-R1, a two-layer vision-language framework for TAU. The first layer is a lightweight anomaly classifier that performs coarse anomaly categorisation, while the second layer is a larger anomaly reasoner that generates detailed event summaries. To improve task-specific reasoning, we introduce a two-stage training strategy consisting of decomposed-QA-enhanced supervised fine-tuning followed by TAU-GRPO, a GRPO-based post-training method with TAU-specific reward functions. Experimental results show that TAU-R1 achieves strong performance on both anomaly classification and reasoning tasks while maintaining deployment efficiency. The dataset and code are available at: https://github.com/siri-rouser/TAU-R1
Figures & tables
Figure 1: Roundabout-TAU dataset statistics. (a) Overview of Roundabout-TAU, including the map of 28 camera sites and one example video. (1b) Distribution of video lengths in Roundabout-TAU. (2b) Proportion of different QA categories in Roundabout-TAU. (c) Distribution of anomaly classes.
Figure 2: TAU-R1 framework and training pipeline. Top: the two-layer framework, where a lightweight classifier filters the video stream and a larger reasoner summarises anomalous clips. Bottom: the two-stage training strategy for both modules, including decomposed-QA enhanced SFT followed by TAU-GRPO post-training.
Figure 3: Decomposed-QA enhanced SFT. To help the model learn the intermediate scene knowledge required for traffic anomaly understanding, we decompose the task into five question-answer types: environment QA, object grounding QA, anomaly time window QA, anomaly reasoning QA, and anomaly description QA. The figure shows one example for each question type.
Model
Params.
Classification
Summarization
4-cls AP
4-cls F1
2-cls AP
2-cls F1
BLEU
ROUGE-L
METEOR
G-Score
Commercial MLLMs
GPT-5 [ 35 ]
N/A
0.2143
0.1115
0.2857
0.2367
0.0353
0.2539
0.2842
3.5485
Qwen3.5-Plus [ 37 ]
N/A
0.2143
0.1074
0.2143
0.1085
0.0313
0.2295
0.2801
2.2947
Gemini-3.1-pro [ 17 ]
N/A
0.4048
0.4086
0.5952
0.6372
0.0417
0.2423
0.2660
2.5598
Open-Source MLLMs
Table 1: Comparison of different approaches on the task of traffic anomaly understanding over Roundabout-TAU dataset. Bold values indicate the best performance, while underlined values indicate the second-best performance.
Metric
Classifier
Summarizer
End-to-End
Total runtime (sec)
809.13
3441.92
4251.05
Avg latency per clip (sec/clip)
2.37
12.47
12.43
Real-Time Factor (runtime/video_time)
0.09
0.44
0.47
Table 2: Jetson AGX Orin deployment efficiency on Roundabout-TAU.
Method
Classification
Summarization
4-cls AP
4-cls F1
2-cls AP
2-cls F1
FP
FN
BLEU
ROUGE-L
METEOR
G-Score
Qwen3-VL
0.2381
0.1967
0.6190
0.6452
6
10
0.0325
0.2534
0.2756
2.7583
+SFT
0.5476
0.5121
0.8810
0.8580
5
0
0.1749
0.4050
0.4104
3.7858
+Decomposed-QA SFT
0.6429
0.6400
0.9048
0.9048
2
2
0.2110
0.4171
0.4298
4.4480
+TAU-GRPO
0.7381
0.7310
0.9286
0.9218
3
0
0.2091
0.4234
0.4252
4.6441
Table 3: Ablation study of the proposed training pipeline on the Roundabout-TAU dataset. The classification ablation is performed on the 2B model, whereas the summarization ablation is performed on the 8B model. FP and FN denote the false positives and false negatives of binary classification, respectively.
Figure 4: Qualitative comparison on Roundabout-TAU dataset.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Sites
Scenes
Anom. Clips
Total Clips
Hours
Text Ann.
ShanghaiTech [ 61 ]
13
Campus
130
437
-
×
UCSD-Ped [ 23 ]
2
Campus
48
98
-
×
CUHK Avenue [ 31 ]
1
Campus
21
37
0.3h
×
MSAD [ 63 ]
500
Multiple
240
720
-
×
Roundabout-TAU
28
Traffic
276
342
2.5h
✓
Appendix
Table 4: Comparison of video anomaly detection/understanding datasets using real-world collected video sources.
Figure 5: Prompt templates for classification and summarization task.
Figure 6: Prompt template for other decomposed-QA tasks.
Figure 7: Prompt template for the object grounding task.
Figure 8: Prompt template for GPT-based evaluation.
Figure 9: Prompt template for GRPO reward function.
Classification
Summarization
Training Strategy
4-cls AP
4-cls F1
2-cls AP
2-cls F1
BLEU
ROUGE-L
METEOR
G_Score
LoRA
0.4523
0.4251
0.6904
0.7150
0.1316
0.3704
0.3639
3.1869
Full Parameter
0.7381
0.7310
0.9286
0.9218
0.2091
0.4234
0.4252
4.6441
Appendix
Table 5: Comparison between LoRA-based training and full-parameter fine-tuning on classification and summarization tasks.
Method
BLEU
ROUGE-L
METEOR
G_Score
TAU-R1 summarizer (without prior label)
0.1914
0.3976
0.3967
4.1922
TAU-R1 summarizer (with prior label)
0.2091
0.4234
0.4252
4.6441
Appendix
Table 6: Effect of using prior anomaly labels on summarization performance.
Figure 10: Additional qualitative example where TAU-R1 successfully identifies the anomaly in a long video, while competing VAU models fail to capture the key event.
Figure 11: Additional qualitative example in a challenging far-field and partially occluded scene, where all models fail to fully capture the anomaly event.
Figure 12: Out-of-domain traffic scene qualitative examples for TAU-R1.
Traffic Anomaly Understanding (TAU) requires models and systems to detect, reason about, and explain anomalous events in transportation videos. To address this challenge, we propose TAU-Agent, an agentic retrieval-augmented framework for traffic anomaly understanding. Given a task query, a central retrieval agent orchestrates two visual perception tools, namely a Video Captioning Tool and an Open-Vocabulary Tracking Tool, to retrieve and select query-relevant evidence, including captions, temporal intervals, and object trajectories. The selected evidence, together with sampled video frames and the input query, is provided to a supervised fine-tuned vision-language model for final reasoning and answer generation. We evaluate TAU-Agent on both the in-domain and the out-of-domain benchmarks from the AI City Challenge 2026. TAU-Agent achieves scores of 0.6779 on Track 3, 0.3998 on Track 7, and 67.9275 on Track 8, ranking second, twelfth, and fifth, respectively. Code is available at: https://github.com/siri-rouser/TAU-Agent.
Yuqiang Lin, Yan Shi, Sam Lockyer +5
University of Bath, Bath BA2 7AY, UK · University of Washington, Seattle, WA 98195, USA
Detecting a traffic anomaly does not establish whether a video-language model can explain what happened, localize it in time, or identify its causes. We introduce TAR (Traffic Anomaly Reasoning) and TAR-Bench, paired resources for training and evaluating these complementary capabilities across 10 tasks spanning question answering, temporal reasoning, and scene understanding. TAR contains 44,040 automatically generated annotations with chain-of-thought traces for 3,670 CCTV videos from eight public datasets. TAR-Bench provides 960 human-curated annotations for 80 held-out clips from 17 public YouTube videos. Evaluation of eleven vision-language models reveals a gap between question-answering performance and temporal or scene reasoning. Progressively adding task groups during supervised fine-tuning improves aggregate performance on both Cosmos-Reason2-8B and Qwen3-VL-8B-Instruct. Training on all 10 tasks raises their mean benchmark scores from 34.3 to 55.7 and from 30.9 to 53.9, respectively. These results support joint training across complementary tasks as a promising approach to traffic anomaly understanding, while highlighting persistent limitations in temporal precision and causal attribution. TAR and TAR-Bench serve as the official training and in-domain evaluation resources for AI City Challenge 2026 Track 3.The dataset is available at https://huggingface.co/datasets/nvidia/PhysicalAI-Traffic-Anomaly-Reasoning
Traffic video understanding has become an important problem in intelligent transportation, as road videos provide direct evidence for accidents, violations, and interactions between vehicles and vulnerable road users. A useful system should explain how a traffic event develops, why it happens, and when the relevant interaction occurs, yet this remains difficult for multimodal large language models (MLLMs) because traffic videos contain sparse events and varied viewpoints. We introduce UniTraffic-Agent, the MR-CAS solution for Track~3 of the 10th AI City Challenge, which includes Traffic Anomaly Reasoning (TAR) and two out-of-domain evaluations: FETV for fisheye traffic events and PSI-VQA for pedestrian intention reasoning. UniTraffic-Agent follows an observe--reason--act--verify workflow that samples timestamped visual evidence, reasons over all questions from the same clip in one request, and converts responses through task-specific action adapters. On the official Public leaderboards, MR-CAS ranks 16th on TAR with a score of 0.5780, 2nd on FETV with 0.4884, and 4th on PSI-VQA with 64.4161. The code is available at https://github.com/Roclp/UniTraffic-Agent.
Peng Li, Qianqian Xu, Shilong Bao +2
School of Computer Science and Technology, University of Chinese Academy of Sciences (UCAS), Beijing, China · State Key Laboratory of AI Safety, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China +1