cs.CVOct 8, 2026

A Unified Score Matching Paradigm for Video Anomaly Detection and Anticipation

Authors: Congqi Cao, Zhenhe Liang, Hanwen Zhang, Yifan Zhao, Qinyi Lv, Lingtong Min, Yanning Zhang

Organizations: National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science, Northwestern Polytechnical University, Xi’an 710129, China · School of Electronics and Information, Northwestern Polytechnical University, Xi’an 710129, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Video anomaly detection (VAD) is a fundamental and safety-critical task in computer vision. Recent generative approaches detect anomalies from a distributional perspective, but remain limited by local anomaly modes. Meanwhile, video anomaly anticipation (VAA), as a proactive extension beyond post-hoc detection, introduces additional challenges. In particular, the contrastive inference paradigm in VAD, which relies on ground-truth frames, is not applicable to VAA, hindering its development. To address these challenges, we propose a unified score-driven framework, termed Uni-DSM, based on denoising score matching (DSM), which models anomaly patterns through likelihood estimation and score functions over the learned data distribution. Within this unified framework, we adopt a shared noise-conditioned score transformer backbone with scene-dependent embeddings and motion-aware weighting for distribution-level modeling. Instead of introducing separate architectures, Uni-DSM unifies VAD and VAA through different inference and supervision paradigms built upon the same score-based formulation. For VAD, we instantiate an autoregressive denoising score matching (ADSM) mechanism, which progressively accumulates anomalous evidence via autoregressive denoising, enabling enhanced perception of local modes beyond visual cues. For VAA, we extend the same architecture by incorporating a lightweight auxiliary decoder and a novel self-distilled denoising score matching (SDSM) mechanism. By constructing supervision from output discrepancies instead of relying on unavailable future ground truth, our method achieves efficient training suitable or early anomaly anticipation. Extensive experiments on multiple benchmark datasets demonstrate state-of-the-art performance in both VAD and VAA while maintaining high efficiency, establishing a unified and scalable pipeline from anomaly detection to anticipation.

Figures & tables

Fig. 1: An illustration of the local modes of anomalies. We explore two regions in which the anomalies are residing. The likelihood-based method is highly sensitive to regions of anomalies with inherently low probability likelihood while blind to anomalies located in local modes near the learned distribution.
Fig. 2: An illustration of three video anomaly anticipation frameworks, where GT denotes ground truth. Compared with semantic memory bank and bidirectional frame prediction frameworks, our self-distilled DSM (SDSM) within Uni-DSM adopts a lightweight unidirectional main-auxiliary architecture, enabling efficient video anomaly anticipation with lower GPU memory consumption and faster inference speed.
Fig. 3: An overview of the Uni-DSM framework, a unified score-driven architecture based on denoising score matching (DSM). The red flow represents autoregressive denoising score matching (ADSM) for video anomaly detection, while the purple flow denotes self-distilled score matching (SDSM) for video anomaly anticipation.
Fig. 4: Schematic illustration of the video anomaly detection (VAD) and video anomaly anticipation (VAA) tasks.
Fig. 5: Overview of the proposed noise-conditional score transformer (NCST) architecture used in Fig. 3 .
Fig. 6: Detailed architecture of the NCST block utilized within the noise-conditional score transformer. MLP, FFN, and MHA stand for the multi-layer perception layer, the feed-forward neural network, and the multi-head attention layer, respectively.
Fig. 7: An overview of the proposed autoregressive denoising score matching process.
MethodAvenueShanghaiTechNWPU
MicroMacroMicroMacroMicroMacro
◇ FFP [ 14 ]84.9-72.8---
◇ AMMC-Net [ 15 ]86.6-73.7-64.5-
◇ HF 2 -VAD [ 16 ]91.193.576.283.163.7-
◇ VABD [ 17 ]86.6-78.2---
+ MTP [ 18 ]82.9-76.0---
TABLE I: Comparison of different methods on the Avenue, ShanghaiTech, and NWPU datasets in AUCs ( %\% ) metric. The best result on each dataset is shown in bold. We also list the inputs of the methods, in which method without marking denotes single modal input.
α\alpha0.5s1s1.5s2s2.5s3s
Chance50.050.050.050.050.050.0
Human–––––90.4
FBAE [ 12 ]65.865.364.964.664.264.0
SSAE [ 13 ]69.569.368.768.267.166.9
Uni-DSM (ours)71.371.170.570.269.569.1
TABLE II: Anticipation Results of different methods on the NWPU Dataset
MethodShanghaiTechNWPU
MicroMacroMicroMacro
L-ADSM77.687.568.173.2
Uni-DSM84.593.276.978.1
TABLE III: AUCs ( %\% ) of our model in latent space and original data space on ShanghaiTech and NWPU datasets.
MethodShanghaiTechNWPU
MicroMacroMicroMacro
MSMA [ 33 ]81.390.472.375.1
MULDE [ 34 ]82.690.473.174.5
Uni-DSM84.593.276.978.1
TABLE IV: AUCs ( %\% ) of different likelihood-based methods in the original data space on ShanghaiTech and NWPU datasets.
 DSMADSMSceneMotionAppearanceShanghaiTech
     MicroMacro
1✓    73.978.5
2✓✓   76.484.1
3✓✓✓  76.986.3
4✓✓ ✓ 78.687.7
5✓✓  ✓82.189.9
TABLE V: AUCs( %\% ) of our model variants on the ShanghaiTech dataset for the VAD task.
 DSMSDSMSceneMotionNWPU datasets
1✓   67.5
2✓✓  68.1
3✓✓✓ 68.6
4✓✓ ✓69.3
5✓✓✓✓70.5
TABLE VI: AUCs(%) of our model variants on the NWPU Dataset for the VAA task.
ModelGPU Memory (G)Running Speed (s)Frame Rate (FPS)Performance (%)
⋄ HF 2 -VAD [ 16 ]–0.066715.076.2
⋄ VABD [ 17 ]–0.014768.078.2
LLSH [ 45 ]–0.039225.577.6
FPDM [ 31 ]–0.12827.878.6
SSAE [ 13 ]240.109610.180.5
Uni-DSM (ours)110.019451.584.5
TABLE VII: Comparison of computational resource consumption during VAD inference on the ShanghaiTech dataset.
ModelGPU Memory (G)Running Speed (s)Frame Rate (FPS)Performance (%)
EPAP-Net [ 36 ]–0.021945.7–
FBAE [ 12 ]210.027436.564.9
SSAE [ 13 ]240.109610.168.7
SDSM (ours)160.016361.370.5
TABLE VIII: Comparison of Computational Resource Consumption During VAA Inference on the NWPU Dataset
Fig. 8: Visualization results on the NWPU dataset for the VAD task. The red boxes indicate that an anomaly is occurring, while the green boxes denote that the anomaly has disappeared.
Fig. 9: Visualization results on the NWPU dataset for the VAA task. The orange boxes indicate that an anomaly is about to occur, the red boxes denote that an anomaly is currently happening, and the yellow boxes signify that the anomaly is disappearing.
Figure 18
Figure 19
Figure 20
Figure 21
Figure 22
Figure 23
Figure 24

Explore similar work

CardsList

  1. Glance, Scrutinize, and Think: Advancing Video Anomaly Detection from Training-Free to Agentic Reasoning

    Aug 7, 2026Shibo Gao, Peipei Yang, Xu-Yao Zhang +1Anomaly LocalizationVideo Anomaly Detection

    School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China. · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China.

  2. Probe-VAD: Ordinal Likelihood Probing for Training-Free Video Anomaly Detection

    Sep 15, 2026Jiawei Gu, Qilin Zhao, Tengkuo Guo +6Video Anomaly DetectionZero-Shot Anomaly Detection

    School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China · Faculty of Life Science and Medicine, School of Medicine and Health, Harbin Institute of Technology, Harbin 150001, China · School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin 150001, China +2

  3. Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding

    Oct 1, 2026Mohd Ubaid Wani, Sara Atito, Josef Kittler +1CoT ReasoningInterpretable Anomaly Detection

    Centre for Vision, Speech and Signal Processing (CVSSP) University of Surrey, UK · Surrey Institute for People-Centred AI (PAI) University of Surrey, UK