Reason or Recite: Complementary Specialists with an LLM Judge for Traffic Anomaly Reasoning

Mayur Kulkarni

cs.CVOct 26, 2026

Reason or Recite: Complementary Specialists with an LLM Judge for Traffic Anomaly Reasoning

Authors: Mayur Kulkarni

Organizations: Stellarview AI, Pune, India

Abstract

Road anomalies such as collisions, violations, and sudden hazards demand more from video understanding than detection: a practical system must verify what happened, describe the scene, and explain the cause. We find these competences pull in opposite directions for vision-language models: closed-form questions reward precise, evidence-grounded reasoning, while open-ended answers are scored by similarity to reference text, where matching the annotation style matters as much as reasoning correctly. We present Reason-or-Recite (RoR), a two-specialist framework for the Traffic Anomaly Reasoning track of the 2026 AI City Challenge. RoR (1) rewrites the official chain-of-thought annotations into a structured, temporally grounded schema and LoRA-tunes a 9B vision-language model on the enhanced traces; (2) fine-tunes a 4-bit-quantized 27B model on answers alone, absorbing the annotation style; and (3) selects between the two candidates per question with a training-free LLM judge. RoR ranks first on the in-domain leaderboard with mean score 0.6788.

Figures & tables

Figure 1: Four defect classes of the official TAR annotations, with verbatim training examples; each is detailed in Sec. 3.2 . Panel (c) shows UCF-Crime [ 35 ] clips.

Task	#Items	TS %	Official	TAR-RE	Official	TAR-RE
			Words		R-L vs. answer
BCQ-OE	7,340	59.8	109	429	0.128	0.058
MCQ-OE	3,670	53.0	123	463	0.141	0.065
Open QA	3,670	57.2	126	471	0.246	0.151
Scene description	3,670	3.0	118	483	0.326	0.212
Video summarization	3,670	89.0	117	482	0.289	0.233

Table 1: Auditing the official TAR training annotations, and the effect of reasoning regeneration (TAR-RE, Sec. 4.2 ). TS = share of annotations containing at least one timestamp; Words = mean reasoning length; R-L = mean token-level ROUGE-L F1 between a reasoning trace and its gold answer (higher = more restatement; omitted where the answer is a single word, option letter, or timestamp pair). Plain BCQ and MCQ match their -OE rows to within 0.3 and are omitted.

Figure 2: The TAR-RE rewrite ( Sec. 4.2 ) on one training item: the official annotation (left) is regenerated under our contract (center) into the five-part trace (right), preserving the gold outcome. Bottom: timestamp canonicalization.

Figure 3: Reason or recite, on two open-ended validation items: the auto-labeled reference beside both specialists’ answers; green marks phrasing shared with the reference, amber accurate but unrewarded content; BERTScore F1 at top right. (a) The reference narrates one of three vehicles; matching its selection outscores reporting all three. (b) Both perceive correctly; near clause-for-clause parallelism beats different words.

#	Team	Mean	BCQ	MCQ	BCQ-OE	MCQ-OE	OQA	Causal	Scene	T-Desc	Summ.
1	Stellarview AI (ours)	0.6788	1.000	0.950	0.669	0.969	0.499	0.531	0.437	0.514	0.541
2	FPT AI Vision	0.6696	0.963	0.950	0.631	0.952	0.522	0.525	0.445	0.498	0.542
3	Smart Vision	0.6669	1.000	0.925	0.634	0.967	0.501	0.528	0.437	0.485	0.525
4	CHT-IVS	0.6563	0.988	0.938	0.643	0.957	0.465	0.516	0.406	0.491	0.503
5	CHT_IVS	0.6562	0.988	0.938	0.642	0.957	0.465	0.516	0.405	0.491	0.503
6	first_gogogo	0.6369	0.994	0.975	0.619	0.973	0.464	0.470	0.374	0.445	0.420

Table 2: Top 6 teams on the AI City Challenge 2026 Track 3 in-domain (TAR) leaderboard. BCQ and MCQ are exact-match accuracy; the remaining seven tasks (-OE: with explanation; OQA: open-ended QA; T-Desc: temporal event description; Summ.: video summarization) are BERTScore F1; Mean is the unweighted average over the nine scored task types. Best value per column in bold .

Task	#Items	Answer-only	TAR-RE reasoning	$\Delta$
BCQ	140	0.9571	0.9714	$+0.0143$
MCQ	70	0.9429	1.0000	$+0.0571$
BCQ-OE	140	0.6972	0.6460	$-0.0512$
Open QA	70	0.4426	0.4618	$+0.0192$
Scene description	70	0.4453	0.4620	$+0.0167$

Table 3: Reasoning-supervision ablation on the 9B backbone: identical recipe ( Sec. 5.1 ), training target either the gold answer alone or the full TAR-RE think-then-answer trace ( Sec. 4.3 ); greedy decoding on identical items from a subset of the internal test split. BCQ and MCQ are exact-match accuracy, the rest BERTScore F1;

\Delta

is reasoning minus answer-only.

Selection strategy	Mean score
Reasoning specialist (9B) only	0.4460
Answer specialist (27B) only	0.4958
Judge, neutral rubric	0.4887
Judge-guided selection (ours)	0.5043
Oracle per-question selection	0.5347

Table 4: Ablation of the selection strategy on the internal test split:

70

videos,

490

items over the six free-text task types (one item per video and task; two for BCQ-OE). Mean score is the average over items.

Task	Lever	$\Delta$	Failure mode
Rewriting the answer after generation
OQA	Text-only LLM rewrites the answers	$-0.071$	polishes away gold-matching detail
BCQ-OE	LLM cleanup of drafts	$-0.012$	deletes words that match the gold
BCQ-OE	LLM fusion of four candidates	$-0.016$	invents detail the drafts agree is absent
Scene	Trim last sentence / keep first three	$-0.028$ / $-0.074$	the metric is recall-driven
Injecting content from outside the model

Table 5: Levers that did not work. Deltas are BERTScore F1 against the base configuration on the validation data (leaderboard where marked LB); paired bootstrap puts the noise floor at roughly

0.012

–

0.016

, so smaller deltas are ties.

Video Reasoning

Video reasoning studies how AI systems use video evidence to infer events, actions, temporal relations, causes, and future states.

See all 208 papers

UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations
Aug 13, 2026Peng Li, Qianqian Xu, Shilong Bao +2Pedestrian Intention Prediction Intelligent Transportation Systems
School of Computer Science and Technology, University of Chinese Academy of Sciences (UCAS), Beijing, China · State Key Laboratory of AI Safety, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China +1
TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding
Aug 26, 2026Yuqiang Lin, Yan Shi, Sam Lockyer +5Traffic Anomaly Detection Anomaly Detection
University of Bath, Bath BA2 7AY, UK · University of Washington, Seattle, WA 98195, USA
From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoning
Aug 10, 2026Han Zhang, Yilin Zhao, Zaid Pervaiz Bhat +5Video-Language Model Evaluation Video Anomaly Detection
NVIDIA · Santa Clara University

Aug 13, 2026J/K move · Enter open · S save

UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations

Peng Li, Qianqian Xu, Shilong Bao, Yangbangyan Jiang+1

School of Computer Science and Technology, University of Chinese Academy of Sciences (UCAS), Beijing, China · State Key Laboratory of AI Safety, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China+1

Pedestrian Intention Prediction Intelligent Transportation Systems Video Reasoning VLM Reasoning Multimodal Agents Traffic Anomaly Detection+1

Open paper AskPDF arXiv

Traffic video understanding has become an important problem in intelligent transportation, as road videos provide direct evidence for accidents, violations, and interactions between vehicles and vulnerable road users. A useful system should explain how a traffic event develops, why it happens, and when the relevant interaction occurs, yet this remains difficult for multimodal large language models (MLLMs) because traffic videos contain sparse events and varied viewpoints. We introduce UniTraffic-Agent, the MR-CAS solution for Track~3 of the 10th AI City Challenge, which includes Traffic Anomaly Reasoning (TAR) and two out-of-domain evaluations: FETV for fisheye traffic events and PSI-VQA for pedestrian intention reasoning. UniTraffic-Agent follows an observe--reason--act--verify workflow that samples timestamped visual evidence, reasons over all questions from the same clip in one request, and converts responses through task-specific action adapters. On the official Public leaderboards, MR-CAS ranks 16th on TAR with a score of 0.5780, 2nd on FETV with 0.4884, and 4th on PSI-VQA with 64.4161. The code is available at https://github.com/Roclp/UniTraffic-Agent.

Abstract

Video Reasoning

Explore similar work

UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations

TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding

From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoning