Traffic Anomaly Understanding (TAU) requires models and systems to detect, reason about, and explain anomalous events in transportation videos. To address this challenge, we propose TAU-Agent, an agentic retrieval-augmented framework for traffic anomaly understanding. Given a task query, a central retrieval agent orchestrates two visual perception tools, namely a Video Captioning Tool and an Open-Vocabulary Tracking Tool, to retrieve and select query-relevant evidence, including captions, temporal intervals, and object trajectories. The selected evidence, together with sampled video frames and the input query, is provided to a supervised fine-tuned vision-language model for final reasoning and answer generation. We evaluate TAU-Agent on both the in-domain and the out-of-domain benchmarks from the AI City Challenge 2026. TAU-Agent achieves scores of 0.6779 on Track 3, 0.3998 on Track 7, and 67.9275 on Track 8, ranking second, twelfth, and fifth, respectively. Code is available at: https://github.com/siri-rouser/TAU-Agent.
Figures & tables
Figure 1: TAU-Agent pipeline example with comparison of other end-to-end unified sampled video anomaly models. TAU-Agent follows multi-stage pipeline to decompose the complex TAU task.
Figure 2: Overview of the TAU-Agent framework. The main agent invokes the Video Captioning Tool and the Open-Vocabulary Tracking Tool to retrieve and select query-relevant evidence and determine the relevant frame range. The input question, sampled video frames, and retrieved evidence are then passed to a fine-tuned question-answering VLM to generate the final prediction.
Figure 3: Overview of the hybrid detection and tracking pipeline. Queries shown in blue correspond to traffic-related COCO categories or fine-grained subclasses and are routed through the YOLO-based branch. Queries shown in orange correspond to non-COCO open-vocabulary categories and are processed by GroundingDINO. Detections from both branches are associated across frames using ByteTrack to generate object tracks.
Step and Operation
Description
1. Query Interpretation
Analyze the input question to identify the referenced events, objects, interactions, anomaly, and temporal context.
2. Video Caption Retrieval
Invoke the Video Captioning Tool to obtain a high-level understanding of the video and retrieve potentially relevant caption segments.
3. Temporal Evidence Selection
Use the question and caption evidence to determine candidate frame ranges and select relevant caption segments.
4. Object-Track Retrieval
If object-level evidence is required, invoke the Open-Vocabulary Tracking Tool to retrieve relevant tracks.
5. Evidence Refinement
Jointly reason over the question, captions, and object tracks to refine the selected frame range, select supporting evidence, and assign a relevance score to each selected item.
6. Iterative Retrieval
Perform additional tool calls if the retrieved evidence remains insufficient or ambiguous.
Table 1: Workflow of the main RAG agent.
Rank
Team
Mean
BCQ
MCQ
BCQ OE
MCQ OE
Open QA
Causal
Scene
Temporal
Summary
1
25
0.6788
1.0000
0.9500
0.6686
0.9693
0.4986
0.5310
0.4373
0.5137
0.5409
2
Ours
0.6779
1.0000
0.9500
0.6685
0.9176
0.5150
0.5503
0.4314
0.5164
0.5516
3
309
0.6748
0.9750
0.9500
0.6660
0.9520
0.5215
0.5253
0.4445
0.4975
0.5416
4
270
0.6741
0.9750
0.9500
0.6663
0.9530
0.5177
0.5181
0.4439
0.4980
0.5453
5
60
0.6703
0.9750
0.9500
0.6663
0.9530
0.5177
0.5181
0.4095
0.4980
0.5453
Table 2: Results on the in-domain TAR leaderboard. Our submission is shown in italics and best item are bold.
Rank
Team
Final Score
Description
Categorical Mean
1
30
0.4891
0.4171
0.5612
2
219
0.4889
0.4166
0.5612
3
139
0.4884
0.4411
0.5358
12
Ours
0.3998
0.3513
0.4484
13
61
0.3997
0.3476
0.4518
Table 3: Results on the FETV leaderboard. Our submission is shown in italics .
Rank
Team
Final
BCQ F1
BCQ Acc.
Open QA F1
MCQ Acc.
Temp. mIoU
1
220
70.8947
0.7084
0.7273
0.5833
0.7912
0.7529
2
30
70.6397
0.7084
0.7273
0.5833
0.7912
0.7427
3
257
69.0698
0.6136
0.6182
0.6674
0.7912
0.6906
4
84
68.2765
0.7136
0.7273
0.6206
0.6813
0.7155
5
Ours
67.9275
0.6167
0.7091
0.7791
0.7253
0.5960
Table 4: Results on the out-of-domain PSI-VQA benchmark. Our submission is shown in italics and best item are bold.
School of Computer Science and Technology, University of Chinese Academy of Sciences (UCAS), Beijing, China · State Key Laboratory of AI Safety, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China +1