Look Inside Each Video: Rethinking How Video Anomaly Detection Is Evaluated

Inpyo Song; Jangwon Lee

cs.CVAug 22, 2026

Look Inside Each Video: Rethinking How Video Anomaly Detection Is Evaluated

Authors: Inpyo Song, Jangwon Lee

Organizations: SungKyunKwan University South Korea

Abstract

Video anomaly detection (VAD) aims to localize anomalous events by identifying when they occur. A common evaluation pools frame-level anomaly scores across all test videos and measures whether anomalous frames receive higher scores than normal frames (Micro-AUROC). This pooling creates comparisons both within the same video and across different videos. However, the pooled evaluation protocol commonly used in current VAD benchmarks makes Micro-AUROC heavily dependent on cross-video comparisons, which do not directly test whether anomalous frames rank above normal frames within the same video. To quantify this imbalance, we decompose Micro-AUROC into within-video and cross-video comparisons. Across four widely used weakly supervised VAD benchmarks, within-video pairs account for only 0.071-0.388% of all anomalous-normal frame pairs. This imbalance can make Micro-AUROC a poor indicator of within-video ranking. Video-level classifiers that assign a single score to each video achieve 81.40-97.18% Micro-AUROC while obtaining 50% Within-AUROC. For existing weakly supervised VAD detectors, replacing all frame scores in each video with their mean sets Within-AUROC to 50%. Across 120 runs, this replacement retains a median 98.8% of the original Micro-AUROC margin above chance. Because the decomposition depends on the evaluation protocol rather than the training supervision, the same distinction applies whenever VAD frame scores are pooled across videos. High Micro-AUROC alone therefore does not establish temporal localization. Our analysis reveals a limitation of pooled VAD evaluation and motivates reporting within-video ranking alongside Micro-AUROC.

Figures & tables

Figure 1: Micro-AUROC closely matches Cross-AUROC. (a) Anomalous–normal frame pairs are divided into within-video and cross-video comparisons. (b) Micro-AUROC is compared with Within-AUROC and Cross-AUROC across four benchmarks and five WSVAD methods. Detectors with similar Micro-AUROC can have substantially different Within-AUROC.

Benchmark	$V$	$w$	Max. gain (pp)
ShanghaiTech	199	$0.140\%$	$0.070$
UCF-Crime	290	$0.388\%$	$0.194$
XD-Violence	800	$0.071\%$	$0.036$
MSAD	240	$0.206\%$	$0.103$

Table 1: Within-video pair fraction and maximum direct contribution to Micro-AUROC.

Figure 2: Within-video pair fractions across VAD benchmarks. Colored points denote the four WSVAD benchmarks used in our main experiments.

Classifier	Feature	ShanghaiTech	UCF-Crime	XD-Violence	MSAD
(a) Mean-pool	CLIP	96.87 $\pm 0.00$	78.50 $\pm 0.04$	88.76 $\pm 0.02$	89.76 $\pm 0.01$
(b) Sequence-linear	CLIP	96.60 $\pm 0.01$	80.03 $\pm 0.21$	90.78 $\pm 0.04$	89.34 $\pm 0.02$
(c) Bi-GRU	CLIP	97.16 $\pm 0.01$	77.84 $\pm 0.92$	89.92 $\pm 0.20$	88.99 $\pm 0.50$
(d) Transformer	CLIP	97.18 $\pm 0.15$	80.40 $\pm 0.87$	88.13 $\pm 1.77$	88.82 $\pm 0.27$
(a) Mean-pool	I3D	92.06 $\pm 0.04$	79.21 $\pm 0.10$	89.22 $\pm 0.02$	86.20 $\pm 0.05$
(b) Sequence-linear	I3D	91.18 $\pm 0.07$	81.40 $\pm 0.06$	89.33 $\pm 0.20$	86.47 $\pm 0.08$

Table 2: Micro-AUROC (%) from video-level anomaly classifiers. Each classifier predicts a single anomaly score per video and assigns that score to all frames in the video. Results are averaged over three runs. The oracle uses frame-level test labels to order videos by their ground-truth fraction of anomalous frames. Within-AUROC is

50\%

for every row.

Figure 3: Architecture of the four video classifiers.

Figure 4: Micro-AUROC remains high after video-mean replacement. All frame scores in each video are replaced with that video’s mean score. The horizontal axis shows the original Micro-AUROC, and the vertical axis shows Micro-AUROC after replacement. (a) Each point represents one of 40 combinations of method, feature, and benchmark, averaged over three training runs. (b) Each point represents one of 13 released results. The diagonal denotes no change.

Reproduced runs
Benchmark	$n$	Median $R_{\mathrm{mean}}$	Minimum $R_{\mathrm{mean}}$	$R_{\mathrm{mean}}\geq 85\%$	$R_{\mathrm{mean}}\geq 90\%$
ShanghaiTech	30	$99.3\%$	$95.0\%$	$30/30$	$30/30$
UCF-Crime	30	$92.6\%$	$78.1\%$	$29/30$	$26/30$
XD-Violence	30	$100.0\%$	$94.3\%$	$30/30$	$30/30$
MSAD	30	$100.1\%$	$95.5\%$	$30/30$	$30/30$
All	120	$98.8\%$	$78.1\%$	$119/120$	$116/120$

Table 3: Retained Micro-AUROC margin above chance after video-mean replacement. Each reproduced run or released result contributes one

R_{\mathrm{mean}}

value as defined in Equation 8 . The final two columns count results retaining at least

85\%

and

90\%

of the original margin, respectively. Reproduced runs and released results are summarized separately.

Semi-supervised
Benchmark	Method	$w$ (%)	Original scores			After replacement
Benchmark	Method	$w$ (%)	Micro-AUROC	Cross-AUROC	Within-AUROC	Micro-AUROC
ShanghaiTech	TrajVAD-P	0.814	88.59	88.57	91.58	61.92
ShanghaiTech	UniVAD	0.814	89.32	89.30	92.22	61.30
Avenue	Jigsaw-VAD	5.079	90.32	90.21	92.40	65.27
UCSD Ped2	Jigsaw-VAD	6.802	86.41	85.96	92.57	61.72

Table 4: Cross-video comparisons remain dominant beyond weak supervision. We report the same detectors across multiple benchmarks. AUROC values and the within-video pair fraction

w

are percentages. The final column gives Micro-AUROC after video-mean replacement, which sets Within-AUROC to

50\%

for every row.

	Within-AUROC		Macro-AUROC		AUC A
Benchmark	Reversals	$\rho$	Reversals	$\rho$	Reversals	$\rho$
ShanghaiTech	23	$-0.10$	21	$0.10$	19	$0.20$
UCF-Crime	9	$0.78$	10	$0.68$	13	$0.58$
XD-Violence	15	$0.44$	18	$0.26$	25	$-0.08$
MSAD	31	$-0.53$	28	$-0.30$	27	$-0.35$
All (180 pairs)	78 ( $43.3\%$ )	–	77 ( $42.8\%$ )	–	84 ( $46.7\%$ )	–

Table 5: Changing the ranking metric changes detector comparisons. For each benchmark, 5 methods evaluated with 2 feature representations give 10 configurations, which are ranked by their 3-run mean performance. We compare all 45 unique pairs of configurations under Micro-AUROC and each alternative metric. A reversal occurs when the two metrics give opposite orderings for a pair. Ties are excluded from the reversal count.

\rho

is Spearman’s rank correlation.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	$V$	$V_{+}$	Frames	$w$	Max. gain (pp)	Maximum Micro-AUROC
Weakly-supervised test splits
ShanghaiTech	199	44	142,912	$0.140\%$	$0.070$	$98.35$
UCF-Crime	290	140	1,114,144	$0.388\%$	$0.194$	$92.95$
XD-Violence	800	500	2,330,384	$0.071\%$	$0.036$	$95.46$
MSAD (Protocol II)	240	120	146,410	$0.206\%$	$0.103$	$96.90$
LAD2000	560	280	628,569	$0.176\%$	$0.088$	$95.42$

Appendix

Table 6: Test-set composition across VAD settings.

V_{+}

counts videos containing an anomaly.

w

is the within-video pair fraction. Max. gain is the Micro-AUROC increase in percentage points from improving Within-AUROC from 50% to 100% while holding Cross-AUROC fixed. The final column gives the maximum Micro-AUROC (%) with one anomaly score per video.

Benchmark	Method	Features	Micro	Mean Micro	Cross	Within	Macro
Three-run averages
ShanghaiTech	Sultani †	CLIP	$97.24$	$96.87$	$97.29$	$66.24$	$66.33$
ShanghaiTech	Sultani †	I3D	$85.00$	$86.16$	$85.01$	$78.43$	$77.49$
ShanghaiTech	RTFM †	CLIP	$96.05$	$96.32$	$96.13$	$44.99$	$44.34$
ShanghaiTech	RTFM †	I3D	$93.14$	$94.01$	$93.17$	$75.45$	$75.07$
ShanghaiTech	CLIP-TSA †	CLIP	$97.74$	$95.93$	$97.76$	$82.59$	$81.85$

Appendix

Table 9: Detector AUROC results (%). Mean Micro is Micro-AUROC after video-mean replacement.

\dagger

denotes the mean over three training runs. Training-free AUROC results are in Table 14 .

Detector training runs
Benchmark	$n$	Median $R_{\mathrm{mean}}$	Min. $R_{\mathrm{mean}}$	$R_{\mathrm{mean}}\geq 85\%$	$R_{\mathrm{mean}}\geq 90\%$
ShanghaiTech	30	$99.3\%$	$95.0\%$	$30/30$	$30/30$
UCF-Crime	30	$92.6\%$	$78.1\%$	$29/30$	$26/30$
XD-Violence	30	$100.0\%$	$94.3\%$	$30/30$	$30/30$
MSAD	30	$100.1\%$	$95.5\%$	$30/30$	$30/30$
All	120	$98.8\%$	$78.1\%$	$119/120$	$116/120$

Appendix

Table 10: Micro-AUROC retention after video-mean replacement. Each training run contributes one

R_{\mathrm{mean}}

value as defined in Equation 8 . Results from released checkpoints and scores are summarized separately. The final two columns count results retaining at least

85\%

and

90\%

, respectively.

Benchmark	Method	Features	Original AP	Video mean AP
Three-run detector averages
ShanghaiTech	Sultani †	CLIP	$55.83$	$50.69$
ShanghaiTech	Sultani †	I3D	$43.07$	$40.23$
ShanghaiTech	RTFM †	CLIP	$45.48$	$48.24$
ShanghaiTech	RTFM †	I3D	$47.34$	$39.38$
ShanghaiTech	CLIP-TSA †	CLIP	$63.24$	$42.55$

Appendix

Table 12: Frame-level AP (%) before and after video-mean replacement.

\dagger

denotes the mean over three training runs. These are the same WSVAD predictions as in .

ShanghaiTech (107-video test split)
Method	$w$	Micro	Mean Micro	Cross	Within	Macro
TrajVAD-P	$0.814$	$88.59$	$61.92$	$88.57$	$91.58$	$91.45$
TrajVAD-T	$0.814$	$84.89$	$63.98$	$84.89$	$84.98$	$87.00$
MoCoDAD	$0.814$	$76.68$	$58.77$	$76.66$	$79.32$	$77.52$
SPARTA	$0.814$	$84.83$	$56.57$	$84.80$	$87.72$	$89.69$
STG-NF	$0.814$	$85.94$	$56.64$	$85.92$	$87.63$	$89.20$

Appendix

Table 13: Semi-supervised detector AUROC (%). Mean Micro is Micro-AUROC after video-mean replacement, and

w

is the within-video pair fraction (%).

\dagger

denotes three-run means. For these rows, Micro and Mean Micro include sample standard deviations.

UCF-Crime
Method	$w$	Micro	Mean Micro	Cross	Within	Macro
LAVAD	$0.388$	$80.28$	$79.66$	$80.36$	$61.75$	$57.85$
VADTree	$0.388$	$84.74$	$82.10$	$84.78$	$74.17$	$63.70$
VERA	$0.388$	$85.58$	$80.82$	$85.62$	$73.46$	$70.27$
XD-Violence
LAVAD	$0.071$	$85.40$	$85.32$	$85.42$	$64.78$	$59.94$

Appendix

Table 14: Additional VLM-based detector AUROC (%) on the evaluated test splits. Mean Micro is Micro-AUROC after video-mean replacement, and

w

is the within-video pair fraction (%). Each row evaluates one set of anomaly scores.

Weakly-supervised
Method	Features	Micro	Mean Micro	Cross	Within	Macro
BN-WVAD †	CLIP	$54.09$	$49.55$	$53.95$	$61.03$	$61.42$
BN-WVAD †	I3D	$56.84$	$52.31$	$56.58$	$70.30$	$71.35$
CLIP-TSA †	CLIP	$68.58$	$45.22$	$68.32$	$82.13$	$81.37$
CLIP-TSA †	I3D	$61.07$	$44.68$	$60.76$	$76.96$	$74.24$
RTFM †	CLIP	$46.85$	$50.05$	$46.88$	$45.11$	$44.16$

Appendix

Table 15: AUROC (%) on the same 44 ShanghaiTech videos and 17,472 frames. All rows have

w=1.891\%

. Mean Micro is Micro-AUROC after video-mean replacement.

\dagger

denotes the mean over three runs.

MSAD, RTFM, I3D features
Prediction	Anomaly-event AP
Prediction	Average	tIoU $0.5$
Original scores	46.64	21.82
Video-mean replacement	0.00	0.00
Whole video, detector mean	61.12	35.69
Whole video, mean-pool	63.04	36.33

Appendix

Table 18: Interval extraction and whole-video predictions give different results. Binary anomaly-event AP (%) on all test videos. The first two rows in each block use interval extraction. Whole-video rows submit one interval per video. Detector results use released checkpoints, and mean-pool results average three runs with the indicated features. Average AP uses tIoU thresholds

0.1,0.2,0.3,0.4,

and

0.5

			Average AP		[email protected]
Benchmark	Method	Features	Original	Whole video	Original	Whole video
Three-run detector means
ShanghaiTech	Sultani †	CLIP	31.85	28.04	8.96	4.85
ShanghaiTech	Sultani †	I3D	21.41	23.31	8.07	5.51
ShanghaiTech	RTFM †	CLIP	23.70	27.89	7.51	6.71
ShanghaiTech	RTFM †	I3D	27.79	23.21	8.45	4.26

Appendix

Table 19: Binary anomaly-event AP (%) on all test videos. Original denotes interval extraction from the original scalar scores. Whole video denotes one interval per video scored by the detector mean. Video-mean replacement followed by extraction gives zero AP throughout.

\dagger

marks three-run means.

All test videos, Average AP
Classifier	Features	ShanghaiTech	UCF-Crime	XD-Violence	MSAD
Mean-pool	CLIP	28.11	9.99	12.49	66.44
Sequence-linear	CLIP	27.13	11.11	14.17	67.04
Bi-GRU	CLIP	29.17	10.92	13.31	66.40
Transformer	CLIP	29.51	11.08	11.28	66.37
Mean-pool	I3D	23.28	9.43	13.89	63.04

Appendix

Table 20: Binary anomaly-event AP (%) for whole-video intervals scored by video-level classifiers. Each entry averages three runs. The interval construction is identical for every classifier.

		All test videos		Anomaly-containing videos
Benchmark	Method	Average	[email protected]	Average	[email protected]
UCF-Crime	VadCLIP	6.68	2.93	6.92	3.04
UCF-Crime	DSANet	13.01	8.00	13.17	8.06
XD-Violence	VadCLIP	24.70	14.34	18.16	9.66
XD-Violence	DSANet	28.87	17.89	18.88	9.45

Appendix

Table 21: Class-wise mAP (%) from released VadCLIP and DSANet checkpoints. Each population retains the classes recorded by the released evaluation, 14 on UCF-Crime and 7 on XD-Violence. Every class-wise video-mean-replacement run produces zero proposals and zero mAP at all reported thresholds.

Metric	WSVAD	Semi-supervised	Training-free	Overall
	$(n=59)$	$(n=45)$	$(n=9)$	$(n=113)$
Frame-level AUROC or AP	$98.3\%$ $(58)$	$100\%$ $(45)$	$100\%$ $(9)$	$\mathbf{99.1\%}$ $(112)$
AUC A	$18.6\%$ $(11)$	$0\%$ $(0)$	$0\%$ $(0)$	$9.7\%$ $(11)$
Macro-AUROC	$0\%$ $(0)$	$22.2\%$ $(10)$	$0\%$ $(0)$	$8.8\%$ $(10)$
Event-level mAP at temporal IoU	$13.6\%$ $(8)$	$0\%$ $(0)$	$0\%$ $(0)$	$7.1\%$ $(8)$
Reported FAR	$13.6\%$ $(8)$	$0\%$ $(0)$	$0\%$ $(0)$	$7.1\%$ $(8)$

Appendix

Table 22: Frame-level AUROC or AP dominates VAD evaluation. Values show the percentage of surveyed papers reporting each metric, with paper counts in parentheses. Only papers reporting their own numerical results on a benchmark in Table 6 are counted. AUC A pools frames from all videos containing anomalies. FAR follows the definition reported in each paper.

Explore similar work

Auditing Frame-Level AUC in Weakly Supervised Video Anomaly Detection: Granularity, Resolution, and Scene Bias
Aug 12, 2026Sara Abdulaziz, Egor BondarevAnomaly Localization Algorithmic Bias
Eindhoven University of Technology, 5612 AE Eindhoven, Netherlands
From Frames to Events: Rethinking Evaluation in Human-Centric Video Anomaly Detection
Apr 10, 2026Narges Rashvand, Shanle Yao, Armin Danesh Pazho +2Video Anomaly Detection Temporal Action Detection
University of North Carolina at Charlotte, Charlotte, NC, USA
MuST-VAD: Mutual Structured Learning for Video Anomaly Detection
Aug 7, 2026Satoshi Hashimoto, Hitoshi Nishimura, Mori KurokawaVLM Adaptation Weakly Supervised Video Anomaly Detection
KDDI Research, Inc. Fujimino, Saitama, Japan

Aug 12, 2026J/K move · Enter open · S save

Auditing Frame-Level AUC in Weakly Supervised Video Anomaly Detection: Granularity, Resolution, and Scene Bias

Sara Abdulaziz, Egor Bondarev

Eindhoven University of Technology, 5612 AE Eindhoven, Netherlands

Anomaly Localization Algorithmic Bias Video Anomaly Detection Weakly Supervised Video Anomaly Detection Video Surveillance

Open paper AskPDF arXiv

Frame-level area under the ROC curve (AUC) is the dominant evaluation metric for weakly supervised video anomaly detection (WSVAD). Its standard form measures whether an anomalous frame outranks a normal frame drawn from anywhere in the test set. We refer to this comparison as pooled AUC, since it aggregates frame pairs across test videos regardless of source. Pooled AUC therefore credits both event localization and differences between video sources. We audit this protocol on UCF-Crime across recent state-of-the-art models spanning different backbone families. Holding each model's frame scores fixed, we read them under three pairing granularities: global, per anomaly category, and within each video, then repeat the same three-granularity readout on zero-shot scores computed from the models' internal representations. We assess ranking reliability with a paired video bootstrap. Three findings follow. First, pooled AUC does not reliably predict within-video anomaly localization: models with similar pooled scores exhibit large localization differences and rank reversals under stricter granularities. Second, at the benchmark's test-split size, pooled AUC lacks the resolution to support state-of-the-art margins reported in the field. Within each backbone family, it resolves no comparison at those margins, while within-video AUC resolves several over identical predictions. Learned representations further reveal that within-video anomaly structure and detector localization are decoupled. Third, on normal footage alone, every model we examine separates videos by recording properties, such as resolution and color encoding, indicating that scene sensitivity is shared across the setting rather than specific to any architecture. We publicly release a granularity-aware protocol computable from existing predictions and scene-factor annotations for UCF-Crime.

Abstract

Explore similar work

Auditing Frame-Level AUC in Weakly Supervised Video Anomaly Detection: Granularity, Resolution, and Scene Bias

From Frames to Events: Rethinking Evaluation in Human-Centric Video Anomaly Detection

MuST-VAD: Mutual Structured Learning for Video Anomaly Detection