TAU-R1: Visual Language Model for Traffic Anomaly Understanding

Yuqiang Lin; Kehua Chen; Sam Lockyer; Arjun Yadav; Mingxuan Sui; Shucheng Zhang; Yan Shi; Bingzhang Wang

cs.CVMar 19, 2026

TAU-R1: Visual Language Model for Traffic Anomaly Understanding

Authors: Yuqiang Lin, Kehua Chen, Sam Lockyer, Arjun Yadav, Mingxuan Sui, Shucheng Zhang, Yan Shi, Bingzhang Wang, +7 more

Organizations: University of Bath, BA2 7AY, United Kingdom · University of Washington, WA 98105, USA · City of Carmel, Indiana, USA · Starwit GmbH, Germany

Abstract

Traffic Anomaly Understanding (TAU) is important for traffic safety in Intelligent Transportation Systems. Recent vision-language models (VLMs) have shown strong capabilities in video understanding. However, progress on TAU remains limited due to the lack of benchmarks and task-specific methodologies. To address this limitation, we introduce Roundabout-TAU, a dataset constructed from real-world roundabout videos collected in collaboration with the City of Carmel, Indiana. The dataset contains 342 clips and is annotated with more than 2,000 question-answer pairs covering multiple aspects of traffic anomaly understanding. Building on this benchmark, we propose TAU-R1, a two-layer vision-language framework for TAU. The first layer is a lightweight anomaly classifier that performs coarse anomaly categorisation, while the second layer is a larger anomaly reasoner that generates detailed event summaries. To improve task-specific reasoning, we introduce a two-stage training strategy consisting of decomposed-QA-enhanced supervised fine-tuning followed by TAU-GRPO, a GRPO-based post-training method with TAU-specific reward functions. Experimental results show that TAU-R1 achieves strong performance on both anomaly classification and reasoning tasks while maintaining deployment efficiency. The dataset and code are available at: https://github.com/siri-rouser/TAU-R1

Figures & tables

Figure 1: Roundabout-TAU dataset statistics. (a) Overview of Roundabout-TAU, including the map of 28 camera sites and one example video. (1b) Distribution of video lengths in Roundabout-TAU. (2b) Proportion of different QA categories in Roundabout-TAU. (c) Distribution of anomaly classes.

Figure 2: TAU-R1 framework and training pipeline. Top: the two-layer framework, where a lightweight classifier filters the video stream and a larger reasoner summarises anomalous clips. Bottom: the two-stage training strategy for both modules, including decomposed-QA enhanced SFT followed by TAU-GRPO post-training.

Figure 3: Decomposed-QA enhanced SFT. To help the model learn the intermediate scene knowledge required for traffic anomaly understanding, we decompose the task into five question-answer types: environment QA, object grounding QA, anomaly time window QA, anomaly reasoning QA, and anomaly description QA. The figure shows one example for each question type.

Commercial MLLMs
Model	Params.	Classification				Summarization
Model	Params.	4-cls AP	4-cls F1	2-cls AP	2-cls F1	BLEU	ROUGE-L	METEOR	G-Score
GPT-5 [ 35 ]	N/A	0.2143	0.1115	0.2857	0.2367	0.0353	0.2539	0.2842	3.5485
Qwen3.5-Plus [ 37 ]	N/A	0.2143	0.1074	0.2143	0.1085	0.0313	0.2295	0.2801	2.2947
Gemini-3.1-pro [ 17 ]	N/A	0.4048	0.4086	0.5952	0.6372	0.0417	0.2423	0.2660	2.5598
Open-Source MLLMs

Table 1: Comparison of different approaches on the task of traffic anomaly understanding over Roundabout-TAU dataset. Bold values indicate the best performance, while underlined values indicate the second-best performance.

Metric	Classifier	Summarizer	End-to-End
Total runtime (sec)	809.13	3441.92	4251.05
Avg latency per clip (sec/clip)	2.37	12.47	12.43
Real-Time Factor (runtime/video_time)	0.09	0.44	0.47

Table 2: Jetson AGX Orin deployment efficiency on Roundabout-TAU.

Method	Classification						Summarization
Method	4-cls AP	4-cls F1	2-cls AP	2-cls F1	FP	FN	BLEU	ROUGE-L	METEOR	G-Score
Qwen3-VL	0.2381	0.1967	0.6190	0.6452	6	10	0.0325	0.2534	0.2756	2.7583
+SFT	0.5476	0.5121	0.8810	0.8580	5	0	0.1749	0.4050	0.4104	3.7858
+Decomposed-QA SFT	0.6429	0.6400	0.9048	0.9048	2	2	0.2110	0.4171	0.4298	4.4480
+TAU-GRPO	0.7381	0.7310	0.9286	0.9218	3	0	0.2091	0.4234	0.4252	4.6441

Table 3: Ablation study of the proposed training pipeline on the Roundabout-TAU dataset. The classification ablation is performed on the 2B model, whereas the summarization ablation is performed on the 8B model. FP and FN denote the false positives and false negatives of binary classification, respectively.

Figure 4: Qualitative comparison on Roundabout-TAU dataset.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Sites	Scenes	Anom. Clips	Total Clips	Hours	Text Ann.
ShanghaiTech [ 61 ]	13	Campus	130	437	-	$\times$
UCSD-Ped [ 23 ]	2	Campus	48	98	-	$\times$
CUHK Avenue [ 31 ]	1	Campus	21	37	0.3h	$\times$
MSAD [ 63 ]	500	Multiple	240	720	-	$\times$
Roundabout-TAU	28	Traffic	276	342	2.5h	✓

Appendix

Table 4: Comparison of video anomaly detection/understanding datasets using real-world collected video sources.

Figure 5: Prompt templates for classification and summarization task.

Figure 6: Prompt template for other decomposed-QA tasks.

Figure 7: Prompt template for the object grounding task.

Figure 8: Prompt template for GPT-based evaluation.

Figure 9: Prompt template for GRPO reward function.

	Classification				Summarization
Training Strategy	4-cls AP	4-cls F1	2-cls AP	2-cls F1	BLEU	ROUGE-L	METEOR	G_Score
LoRA	0.4523	0.4251	0.6904	0.7150	0.1316	0.3704	0.3639	3.1869
Full Parameter	0.7381	0.7310	0.9286	0.9218	0.2091	0.4234	0.4252	4.6441

Appendix

Table 5: Comparison between LoRA-based training and full-parameter fine-tuning on classification and summarization tasks.

Method	BLEU	ROUGE-L	METEOR	G_Score
TAU-R1 summarizer (without prior label)	0.1914	0.3976	0.3967	4.1922
TAU-R1 summarizer (with prior label)	0.2091	0.4234	0.4252	4.6441

Appendix

Table 6: Effect of using prior anomaly labels on summarization performance.

Figure 10: Additional qualitative example where TAU-R1 successfully identifies the anomaly in a long video, while competing VAU models fail to capture the key event.

Figure 11: Additional qualitative example in a challenging far-field and partially occluded scene, where all models fail to fully capture the anomaly event.

Figure 12: Out-of-domain traffic scene qualitative examples for TAU-R1.

Explore similar work

TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding
Aug 26, 2026Yuqiang Lin, Yan Shi, Sam Lockyer +5Anomaly Detection Video Reasoning
University of Bath, Bath BA2 7AY, UK · University of Washington, Seattle, WA 98195, USA
From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoning
Aug 10, 2026Han Zhang, Yilin Zhao, Zaid Pervaiz Bhat +5Video-Language Model Evaluation Video Anomaly Detection
NVIDIA · Santa Clara University
UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations
Aug 13, 2026Peng Li, Qianqian Xu, Shilong Bao +2Pedestrian Intention Prediction Intelligent Transportation Systems
School of Computer Science and Technology, University of Chinese Academy of Sciences (UCAS), Beijing, China · State Key Laboratory of AI Safety, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China +1

Aug 26, 2026J/K move · Enter open · S save

TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding

Yuqiang Lin, Yan Shi, Sam Lockyer, Harish Tayyar Madabushi+4

University of Bath, Bath BA2 7AY, UK · University of Washington, Seattle, WA 98195, USA

Anomaly Detection Video Reasoning Agentic RAG

Open paper AskPDF arXiv

Traffic Anomaly Understanding (TAU) requires models and systems to detect, reason about, and explain anomalous events in transportation videos. To address this challenge, we propose TAU-Agent, an agentic retrieval-augmented framework for traffic anomaly understanding. Given a task query, a central retrieval agent orchestrates two visual perception tools, namely a Video Captioning Tool and an Open-Vocabulary Tracking Tool, to retrieve and select query-relevant evidence, including captions, temporal intervals, and object trajectories. The selected evidence, together with sampled video frames and the input query, is provided to a supervised fine-tuned vision-language model for final reasoning and answer generation. We evaluate TAU-Agent on both the in-domain and the out-of-domain benchmarks from the AI City Challenge 2026. TAU-Agent achieves scores of 0.6779 on Track 3, 0.3998 on Track 7, and 67.9275 on Track 8, ranking second, twelfth, and fifth, respectively. Code is available at: https://github.com/siri-rouser/TAU-Agent.

Abstract

Explore similar work

TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding

From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoning

UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations