Learning Temporal Evidence Selection for Video Anomaly Understanding

Hangil Park; Seungheon Kim; Tae-Kyun Kim

cs.CVOct 12, 2026

Learning Temporal Evidence Selection for Video Anomaly Understanding

Authors: Hangil Park, Seungheon Kim, Tae-Kyun Kim

Organizations: Korea Advanced Institute of Science and Technology (KAIST)

Abstract

Video anomaly understanding (VAU) aims to generate language-level interpretations of anomalous events in long videos, requiring visual evidence that supports explanation beyond anomaly detection alone. Existing VAU pipelines commonly rely on anomaly-centric or predefined temporal sampling strategies, which efficiently emphasize suspicious moments but overlook contextual cues, object interactions, and temporally distributed evidence needed for reasoning. Motivated by recent progress in learnable temporal selection for long-video question answering, we revisit VAU temporal sampling as a downstream-aligned evidence selection problem. We introduce the semantic-conditioned Temporal Sampling Q-Former that learns to select temporal visual evidence through the VAU generation objective. Since generic VAU prompts provide weak video-specific guidance, we condition the sampler on CLIP-embedded object tags automatically extracted from the input video. These tags are used only for temporal evidence selection and are not appended to the LLM generation prompt, separating sampler conditioning from prompt augmentation. The proposed sampler operates in visual-token space, computes soft temporal selection weights from frame-level representations, and merges duplicated selections before response generation. Experiments on HIVAU-70K show that our model achieves the best performance among the compared fine-tuned models, while ablations demonstrate that learnable temporal sampling improves over fixed anomaly-focused sampling and that CLIP-embedded object-tag conditioning further improves over generic-prompt conditioning. Our results suggest that temporal sampling in VAU should be aligned with explanation-oriented evidence utility rather than anomaly saliency alone.

Figures & tables

Figure 1: Conceptual illustration of the evidence mismatch in video anomaly understanding. Anomaly-centric selection captures salient anomaly peaks, but its non-peak selections are not necessarily useful for explaining why the event is anomalous. Explanation-supporting selection retrieves anomaly evidence together with explanation-helpful non-peak cues, such as scene context, object roles, and surrounding reactions, leading to a more complete natural-language explanation.

Figure 2: Overview of the proposed temporal evidence selection framework. The video is encoded by a frozen visual encoder, and the temporal Q-Former selects visual evidence from uniformly sampled candidate frames. Object-centric tags are encoded by a frozen CLIP text encoder and mapped to the temporal Q-Former hidden space by a trainable semantic projection layer. The resulting object-tag embedding sequence $\mathbf{Z}_{c}$ is used only to condition the temporal sampler. The selected evidence tokens are projected and then consumed by the LLM with the original prompt $q$ .

Zero-shot MLLMs
Method	Params	BLEU ( Papineni et al., 2002 ) ( $\uparrow$ )			CIDEr ( Vedantam et al., 2015 ) ( $\uparrow$ )			METEOR ( Banerjee and Lavie, 2005 ) ( $\uparrow$ )			ROUGE ( Lin, 2004 ) ( $\uparrow$ )
Method	Params	C	E	V	C	E	V	C	E	V	C	E	V
Video-ChatGPT ( Maaz et al., 2024 )	7B	0.152	0.068	0.066	0.033	0.011	0.013	0.102	0.069	0.044	0.153	0.048	0.079
Video-LLAMA ( Zhang et al., 2023a )	7B	0.151	0.079	0.104	0.024	0.014	0.017	0.112	0.076	0.057	0.156	0.067	0.090
Video-LLAVA ( Lin et al., 2024 )	7B	0.164	0.046	0.055	0.032	0.009	0.013	0.097	0.022	0.014	0.132	0.023	0.045
LLaVA-Next-Video ( Zhang et al., 2024 )	7B	0.435	0.091	0.120	0.102	0.015	0.031	0.117	0.085	0.096	0.198	0.080	0.106

Table 1: Evaluation on the HIVAU-70K benchmark. We report the performance at three temporal granularities: clip-level (C), event-level (E), and video-level (V). Ours denotes the Temporal Q-Former with Object Tags (CLIP Embedding), which is used as our final model. Rows marked with † are evaluated or reproduced using our implementations; unmarked baseline results are from ( Zhang et al., 2025a ; Cheng et al., 2026 ) . The best results among the compared methods are shown in bold , and the second best results are underlined .

Evidence Selector	Conditioning Signal	BLEU ( $\uparrow$ )	CIDEr ( $\uparrow$ )	METEOR ( $\uparrow$ )	ROUGE ( $\uparrow$ )
Uniform Sampling	–	0.9457	0.8396	0.2027	0.3744
ATS ( Zhang et al., 2025a )	–	0.9144	0.8537	0.2008	0.3738
CAES ( Cheng et al., 2026 )	–	0.9851	0.8582	0.1989	0.3690
Temporal Q-Former	Generic Prompt	1.1227	0.9015	0.1996	0.3729
Temporal Q-Former	Object Tags (Text)	1.1274	0.9188	0.1996	0.3759
Temporal Q-Former	Object Tags (CLIP Embedding)	1.1482	0.9334	0.2028	0.3776

Table 2: Ablation study on temporal evidence selection and sampler conditioning. Scores are computed directly on the full HIVAU-70K evaluation set, including clip-, event-, and video-level samples. The upper block compares fixed temporal sampling strategies, while the lower block compares Temporal Q-Former variants with different conditioning signals. For the fixed-sampler baselines, we use a shared InternVL2-2B+LoRA checkpoint and change only the inference-time sampling rule. For the Temporal Q-Former variants, we train separate checkpoints for each conditioning signal while keeping the backbone and LoRA training configuration.

Figure 3: Qualitative comparison of temporal evidence selection on HIVAU-70K. We compare ATS, CAES, and the object-tag-conditioned Temporal Q-Former. The proposed sampler retrieves explanation-supporting evidence missed by fixed anomaly-focused samplers, including brief causal moments and temporally distributed contextual cues needed for anomaly reasoning.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Sampling Strategy	BLEU ( $\uparrow$ )			CIDEr ( $\uparrow$ )			METEOR ( $\uparrow$ )			ROUGE ( $\uparrow$ )
Sampling Strategy	C	E	V	C	E	V	C	E	V	C	E	V
Uniform Sampling	1.1297	0.6712	0.4999	0.5061	1.4220	1.3516	0.2151	0.1996	0.1420	0.3618	0.4059	0.3858
ATS ( Zhang et al., 2025a )	1.0937	0.6028	0.5348	0.5037	1.4533	1.4251	0.2128	0.1952	0.1440	0.3599	0.4067	0.3904
CAES ( Cheng et al., 2026 )	1.0840	0.7322	0.7589	0.5265	1.4070	1.4223	0.2074	0.1997	0.1506	0.3533	0.4023	0.3947
Temporal Q-Former (Generic Prompt)	1.1702	0.9363	0.7610	0.5266	1.5137	1.5763	0.2050	0.2086	0.1577	0.3532	0.4145	0.4050
Temporal Q-Former (Object Tags)	1.2050	0.9564	0.7817	0.5657	1.5745	1.5247	0.2081	0.2137	0.1600	0.3596	0.4201	0.3987

Appendix

Table 3: C/E/V breakdown of sampling strategy ablations. This table reports clip-, event-, and video-level results for the different sampling strategies.

Sampler Conditioning	BLEU ( $\uparrow$ )			CIDEr ( $\uparrow$ )			METEOR ( $\uparrow$ )			ROUGE ( $\uparrow$ )
Sampler Conditioning	C	E	V	C	E	V	C	E	V	C	E	V
Generic Prompt	1.1702	0.9363	0.7610	0.5266	1.5137	1.5763	0.2050	0.2086	0.1577	0.3532	0.4145	0.4050
Object Tags (Text)	1.1845	0.9417	0.7558	0.5302	1.5584	1.6293	0.2044	0.2117	0.1581	0.3556	0.4204	0.4063
Object Tags (CLIP Embedding)	1.2050	0.9564	0.7817	0.5657	1.5745	1.5247	0.2081	0.2137	0.1600	0.3596	0.4201	0.3987

Appendix

Table 4: C/E/V breakdown of sampler-conditioning ablations. This table reports clip-, event-, and video-level results for the Temporal Q-Former variants with different conditioning signals. Text denotes using the object-tag string as sampler text context, while CLIP Embedding denotes encoding individual object tags with a CLIP text encoder, mapping them through a trainable semantic projection layer, and using the projected embeddings as the sampler conditioning sequence.

Panel of Figure 4 — Figure 4: Task-wise sampling analysis on HIVAU-70K. Each cell reports the corpus-level metric difference over ATS for a given granularity/task pair; positive values indicate improvement over ATS. Uniform sampling and ATS share the same checkpoint and differ only in the inference-time sampling rule. The Temporal Q-Former comparison uses a separately trained end-to-end variant, so it should be interpreted as the effect of replacing the fixed sampler with a learned evidence selector.

Comparison $(A-B)$	BLEU $\Delta$ [95% CI]	CIDEr $\Delta$ [95% CI]	METEOR $\Delta$ [95% CI]	ROUGE $\Delta$ [95% CI]
Object Tags $-$ Prompt	$+0.0254~[+0.0053,+0.0449]^{\ast}$	$+0.0320~[-0.0009,+0.0632]$	$+0.0033~[+0.0012,+0.0053]^{\ast}$	$+0.0047~[+0.0009,+0.0083]^{\ast}$
Object Tags $-$ CAES	$+0.1630~[+0.1130,+0.2135]^{\ast}$	$+0.0753~[+0.0287,+0.1255]^{\ast}$	$+0.0039~[+0.0014,+0.0068]^{\ast}$	$+0.0085~[+0.0037,+0.0131]^{\ast}$
Object Tags $-$ ATS	$+0.2337~[+0.1755,+0.2863]^{\ast}$	$+0.0797~[+0.0356,+0.1255]^{\ast}$	$+0.0020~[-0.0009,+0.0049]$	$+0.0038~[-0.0003,+0.0082]$
Object Tags $-$ Uniform	$+0.2025~[+0.1395,+0.2549]^{\ast}$	$+0.0938~[+0.0501,+0.1395]^{\ast}$	$+0.0001~[-0.0029,+0.0028]$	$+0.0032~[-0.0009,+0.0075]$

Appendix

Table 5: Paired bootstrap confidence intervals for ablation comparisons. Each cell reports

\Delta=\mathrm{Score}(A)-\mathrm{Score}(B)

with a 95% confidence interval. Positive values indicate that method

A

outperforms method

B

. ∗ denotes that the 95% confidence interval does not include zero. “Prompt” denotes Temporal Q-Former with generic-prompt conditioning, and “Object Tags” denotes Temporal Q-Former with CLIP-embedded object-tag conditioning.

Panel of Figure 5 — Figure 5: Full sampled-frame sequences visualization for Fig. 3 .

Sampler Conditioning	BLEU ( $\uparrow$ )				CIDEr ( $\uparrow$ )				METEOR ( $\uparrow$ )				ROUGE ( $\uparrow$ )
Sampler Conditioning	C	E	V	Overall	C	E	V	Overall	C	E	V	Overall	C	E	V	Overall
Object Tags (CLIP Embedding)	1.2050	0.9564	0.7817	1.1482	0.5657	1.5745	1.5247	0.9334	0.2081	0.2137	0.1600	0.2028	0.3596	0.4201	0.3987	0.3776
Task+Object Tags (CLIP Embedding)	1.1977	0.9816	0.7990	1.1413	0.5657	1.6147	1.5797	0.9488	0.2064	0.2139	0.1635	0.2020	0.3596	0.4246	0.4090	0.3798

Appendix

Table 6: Comparison between Ours and Task-aware conditioning extension. This table reports clip-, event-, video- and full evaluation results for Ours and the task-aware conditioning extension.

Explore similar work

LATERN: Test-Time Context-Aware Explainable Video Anomaly Detection
May 14, 2026Mitchell Piehl, Muchao YeInterpretable Anomaly Detection VLM Reasoning
The University of Iowa, Iowa City IA 52245, USA
Glance, Scrutinize, and Think: Advancing Video Anomaly Detection from Training-Free to Agentic Reasoning
Aug 7, 2026Shibo Gao, Peipei Yang, Xu-Yao Zhang +1Anomaly Localization Video Anomaly Detection
School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China. · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China.
AgenticVAU: Multi-Agent Explore-Verify Reasoning for Video Anomaly Understanding
Aug 4, 2026Yuxiang Duan, Huining Li, Ao Li +6Multi-Agent Systems Video Anomaly Detection
C-FAIR & School of software, Shandong University · Renmin University of China · Nanjing Agricultural University +1

May 14, 2026J/K move · Enter open · S save

LATERN: Test-Time Context-Aware Explainable Video Anomaly Detection

Mitchell Piehl, Muchao Ye

The University of Iowa, Iowa City IA 52245, USA

Interpretable Anomaly Detection VLM Reasoning Multimodal Anomaly Detection Video Anomaly Detection Temporal Video Understanding VLM Inference

Open paper AskPDF arXiv

Vision-language models (VLMs) have recently emerged as a promising paradigm for video anomaly detection (VAD) due to their strong visual reasoning ability and natural language-based explainability. In this paper, we aim to address a key limitation of such pipelines, which perform segment-level inference independently owing to token constraints and reason without structured temporal context, allowing VLMs to interpret anomalies as deviations from evolving video dynamics rather than producing fragmented predictions and explanations. To specify, we propose a context-aware framework named LATERN, which reformulates VAD as a temporal evidence aggregation process. LATERN consists of two complementary modules: Context-Aware Anomaly Scoring (CEA) and Recursive Evidence Aggregation (REA). CEA introduces a novel image-grounded memory mechanism, which selectively chooses historical content via frame diversity and visual-textual alignment as expanded context to help generate reliable anomaly scores. Building upon these scores, REA performs recursive temporal aggregation to identify coherent anomaly intervals and produce event-level decisions and explanations grounded in visual-textual evidence. Extensive experiments on challenging benchmarks, including UCF-Crime and XD-Violence, show that LATERN enhances detection accuracy and explanation consistency for frozen VLMs during test time, while generating temporally coherent and semantically grounded event-level explanations.

Abstract

Explore similar work

LATERN: Test-Time Context-Aware Explainable Video Anomaly Detection

Glance, Scrutinize, and Think: Advancing Video Anomaly Detection from Training-Free to Agentic Reasoning

AgenticVAU: Multi-Agent Explore-Verify Reasoning for Video Anomaly Understanding