ESOM: Efficiently Understanding Streaming Video Anomalies with Open-world Dynamic Definitions

Zihao Liu; Xiaoyu Wu; Wenna Li; Jianqin Wu; Linlin Yang

cs.CVApr 9, 2026

ESOM: Efficiently Understanding Streaming Video Anomalies with Open-world Dynamic Definitions

Authors: Zihao Liu, Xiaoyu Wu, Wenna Li, Jianqin Wu, Linlin Yang

Organizations: State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, China

Abstract

Open-world video anomaly detection (OWVAD) aims to detect and explain abnormal events under different anomaly definitions, which is important for applications such as intelligent surveillance and live-streaming content moderation. Recent MLLM-based methods have shown promising open-world generalization, but still suffer from three major limitations: inefficiency for practical deployment, lack of streaming processing adaptation, and limited support for dynamic anomaly definitions in both modeling and evaluation. To address these issues, this paper proposes ESOM, an efficient streaming OWVAD model that operates in a training-free manner. ESOM includes a Definition Normalization module to structure user prompts for reducing hallucination, an Inter-frame-matched Intra-frame Token Merging module to compress redundant visual tokens, a Hybrid Streaming Memory module for efficient causal inference, and a Probabilistic Scoring module that converts interval-level textual outputs into frame-level anomaly scores. In addition, this paper introduces OpenDef-Bench, a new benchmark with clean surveillance videos and diverse natural anomaly definitions for evaluating performance under varying conditions. Extensive experiments show that ESOM achieves real-time efficiency on a single GPU and state-of-the-art performance in anomaly temporal localization, classification, and description generation. The code and benchmark will be released at https://github.com/Kamino666/ESOM_OpenDef-Bench.

Figures & tables

Fig. 1: Motivation of ESOM and OpenDef-Bench. (a) ESOM addresses three key challenges in OWVAD: efficiency, streaming processing, and ability of generating frame-level anomaly scores and textual explanations under dynamic anomaly definitions. (b) OpenDef-Bench provides diverse, scene-dependent anomaly definitions. In contrast, Traditional settings [ 2 ] rely on static definitions, while recent Drift@5 [ 7 ] suffers from limited diversity and weak alignment with real-world scenarios.

Fig. 2: ESOM is a training-free streaming framework for open-world video anomaly detection under dynamic anomaly definitions. Given a raw user prompt, The DN module first converts it into a structured anomaly definition table. And the input video is processed in a sliding-window manner, where each window is encoded into visual tokens, and the IIM module compresses redundant tokens before they are fed into the LLM. Meanwhile, the HSM module manages the model’s cross-window memory states to maintain temporal continuity during streaming inference. Based on the LLM outputs, including anomaly reasoning, classification, and temporal segments, the PS module further converts interval-level textual predictions into frame-level soft anomaly scores.

Fig. 3: The construction pipeline and example samples of the proposed OpenDef-Bench. Each video is annotated with multiple definitions containing multiple candidate categories. A sample is a triplet of a video, a definition, and the corresponding answer.

Benchmark	Dynamic	Clean	# Videos	# Samples	Duration (h)	Resolution
	Definitions
UCF-Crime [ 41 ]	$\times$	$\times$	290	290	10.3	240P
XD-Violence [ 42 ]	$\times$	$\times$	800	800	27.0	360P
MSAD [ 43 ]	$\times$	$\times$	240	240	1.4	720P, 1080P
ShanghaiTech [ 37 ]	$\times$	$\checkmark$	107	107	0.5	480P
NWPU [ 38 ]	$\times$	$\checkmark$	242	242	4.3	1080P

TABLE I: Comparison of OpenDef-Bench with existing VAD benchmarks. Clean: w/o watermarks, editing, or visual effects.

Panel of Figure 5 — Fig. 4: Statistics of OpenDef-Bench. With high video resolution, long video duration, and diverse answer options, OpenDef-Bench presents a challenging benchmark for evaluation.

Fig. 5: T-SNE visualization of text embeddings of anomaly definitions from OpenDef-Bench and existing VAD benchmarks. Definitions from existing benchmarks cluster in localized regions, while those from OpenDef-Bench are more uniformly distributed across the semantic space.

Model	Type	Online	UCF-Crime			XD-Violence			ShanghaiTech
Model	Type	Online	AUC	LaAP	F1	AP	LaAP	F1	AUC	LaAP
VadCLIP [ 8 ]	T	$\times$	80.16	21.70	10.52	58.29	60.16	26.16	60.73	9.20
LaGoVAD [ 7 ]	T	$\times$	81.11	22.48	16.64	74.25	75.82	63.80	45.74	4.94
Qwen3-VL-8B	TF	$\times$	63.64	6.33	21.79	44.36	33.92	69.02	58.68	4.38
LAVAD [ 5 ]	TF	$\times$	79.21	24.01	–	–	–	–	57.58	6.52
VERA [ 11 ]	GF	$\times$	85.57	29.82	–	56.24	56.61	–	59.10	16.36

TABLE II: Comparison with existing methods on traditional zero-shot video anomaly detection benchmarks. T denotes Training, GF denotes Gradient-free, and TF denotes Training-free.

Model	Type	Online	OpenDef-Bench				XD-Violence		HIVAU-70k
			Temporal Localization			Category Selection	Drift@5		Video Description
			AUC	AP	LaAP	ACC	AUC	AP	Win Rate
HolmesVAU [ 6 ]	T	$\times$	46.35	3.74	–	0.71	–	–	–
LaGoVAD [ 7 ]	T	$\times$	48.19	6.40	6.43	9.34	85.7	37.1	–
LAVAD [ 5 ]	TF	$\times$	50.13	4.91	4.84	–	81.7	34.8	–
Qwen3-VL-8B	TF	$\times$	54.86	10.22	7.02	12.74	78.8	34.0	61.7%

TABLE III: Comparison of dynamic definition performance and explainability with state-of-the-art methods on OpenDef-Bench, XD-Violence and HIVAU-70k. T denotes Training and TF denotes Training-free.

Model	Single Model	Param.	FPS	RTF $\downarrow$	UCF-Crime	OpenDef-Bench
Model	Single Model	Param.	FPS	RTF $\downarrow$	AUC $\uparrow$	AUC $\uparrow$	AP $\uparrow$
LAVAD [ 5 ]	$\times$	$\approx$ 43B	2	11.05	79.21	50.13	4.91
VERA [ 11 ]	$\times$	$\approx$ 9B	2	5.74	75.27	–	–
ESOM	$\checkmark$	8B	2	0.27	82.13	57.84	15.43
ESOM	$\checkmark$	8B	4	0.52	86.18	59.22	15.88

TABLE IV: Comparison of inference efficiency. FPS denotes the video sampling rate.

Method	UCF-Crime		OpenDef-Bench		Token Ratio (%)	Prefill Time (s)	Decode Time (s)
Method	AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	Token Ratio (%)	Prefill Time (s)	Decode Time (s)
ESOM	86.18	41.26	59.22	32.74	0.22	0.381	8.31
$-$ w/o IIM	85.75	40.71	58.70	31.65	0.44	0.865	10.94
$\hskip 9.24994pt-$ w/o HSM	80.53	40.70	54.87	31.86	1.00	2.692	18.07
$\hskip 18.49988pt-$ w/o PS	76.11	40.70	52.41	31.86	1.00	2.692	18.07
$\hskip 9.24994pt\hskip 18.49988pt-$ w/o DN	71.99	35.63	52.73	27.35	1.00	2.692	18.07

TABLE V: Ablation study of each module.

Fig. 6: Comparison of different token compression methods [ 47 , 48 , 49 ] under varying token ratios on UCF-Crime.

Table 12

Fig. 7: Qualitative results of ESOM. Due to space limitations, some text is truncated and marked with [MORE…] .

GoF Size	P-frame Interval	UCF-Crime
GoF Size	P-frame Interval	AUC $\uparrow$	F1 $\uparrow$
15	7	84.31	40.93
8	7	86.18	41.26
4	3	83.81	38.46
4	1	86.00	41.93

TABLE VIII: Ablation study of the IIM module.

Explore similar work

Rethinking Open-World Video Anomaly Detection: Diagnosing Definition Blindness
Jul 22, 2026Inpyo Song, Jangwon LeeUnsupervised Anomaly Detection Video Anomaly Detection
ReactVAU: A Slow-Fast Decoupled Framework for Streaming Video Anomaly Understanding
Sep 7, 2026Chia-Hui Chen, Shih-Ying Yeh, Fu-En Yang +2Streaming Video Understanding Video Anomaly Detection
National Tsing Hua University, Hsinchu, Taiwan · NVIDIA, Taipei, Taiwan
No Need For Real Anomaly: MLLM Empowered Zero-Shot Video Anomaly Detection
Feb 22, 2026Zunkai Dai, Ke Li, Jiajia Liu +2Video Anomaly Detection Zero-Shot Anomaly Detection
Beijing University of Posts and Telecommunications · Northwestern Polytechnical University

Jul 22, 2026J/K move · Enter open · S save

Rethinking Open-World Video Anomaly Detection: Diagnosing Definition Blindness

Inpyo Song, Jangwon Lee

Unsupervised Anomaly Detection Video Anomaly Detection

Open paper AskPDF arXiv

Open-world video anomaly detection (OWVAD) is expected to detect events that match a user-specified definition of abnormality. This requirement is stronger than generic anomaly localization: in the same video, changing the definition should change which temporal regions are scored as anomalous. We show that current OWVAD evaluation largely fails to isolate this conditional behavior. Standard VAD metrics and the dynamic-definition protocol can be dominated by target-versus-normal separation, allowing models to obtain strong scores while remaining nearly insensitive to the queried definition. We call this failure mode definition blindness. To explain why it is missed, we decompose dynamic-definition evaluation into target-versus-normal detection and target-versus-other-anomaly discrimination, and find that the former receives 7.2-26.8

\times

more weight across common VAD benchmarks. Motivated by this diagnosis, we introduce three definition-conditioned evaluation metrics, DC-Disc, DC-Det

Δ

, and DC-Sel

Δ

, which progressively remove normal-frame, generic-anomaly, and multi-event selection shortcuts. Experiments on UCF-Crime, XD-Violence, and MSAD reveal that several strong VAD, OWVAD, and general vision language model baselines localize anomalous moments but exhibit weak definition following, often with near-zero definition-response margins. To validate that the failure is actionable, we further introduce DeCoS, a definition-contrastive scoring rule that subtracts anomaly evidence shared across definitions. DeCoS improves the strongest baseline by 7.3-16.0 AUROC points on DC-Disc and 15.5-28.3 points on DC-Det

Δ

. Overall, our results argue that OWVAD should be evaluated as definition-conditioned anomaly scoring, not as anomaly detection under different prompt labels.

Abstract

Explore similar work

Rethinking Open-World Video Anomaly Detection: Diagnosing Definition Blindness

ReactVAU: A Slow-Fast Decoupled Framework for Streaming Video Anomaly Understanding

No Need For Real Anomaly: MLLM Empowered Zero-Shot Video Anomaly Detection