Frequency-Guided Diffusion Model with Perturbation Training for Skeleton-Based Video Anomaly Detection
Authors: Xiaofeng Tan, Xin Geng, Liang Wang, Hongsong Wang
Organizations: School of Computer Science and Engineering, Southeast University, Nanjing 211189, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA) · School of Artificial Intelligence, University of Chinese Academy of Sciences
Video anomaly detection (VAD) is a vital yet complex open-set task in computer vision, commonly tackled through reconstruction-based methods. However, these methods struggle with two key limitations: (1) insufficient robustness in open-set scenarios, where unseen normal motions are frequently misclassified as anomalies, and (2) an overemphasis on, but restricted capacity for, local motion reconstruction, which are inherently difficult to capture accurately due to their diversity. To overcome these challenges, we introduce a novel frequency-guided diffusion model with perturbation training. First, we enhance robustness by training a generator to produce perturbed samples, which are similar to normal samples and target the weakness of the reconstruction model. This training paradigm expands the reconstruction domain of the model, improving its generalization to unseen normal motions. Second, to address the overemphasis on motion details, we employ the 2D Discrete Cosine Transform (DCT) to separate high-frequency (local) and low-frequency (global) motion components. By guiding the diffusion model with observed high-frequency information, we prioritize the reconstruction of low-frequency components, enabling more accurate and robust anomaly detection. Extensive experiments on five widely used VAD datasets demonstrate that our approach surpasses state-of-the-art methods, underscoring its effectiveness in open-set scenarios and diverse motion contexts. Our project website is https://xiaofeng-tan.github.io/projects/FG-Diff.
Figures & tables
Fig. 1: The data illustration. (a) The training and testing data, where the training data is composed of seen normal motions and the testing data contains unseen normal and abnormal motions. Although seen and unseen motions represent the same action (e.g., walking), their local details, such as stride length, arm swing amplitude, and joint angles, exhibit significant differences. (b) The frequency analyses of motions. This analysis reveals that a motion retaining only 70% of its low-frequency information remains largely similar to the original motion in terms of global structure, with minor differences observed in the low-frequency regions. Note that low-frequency and high-frequency regions do not correspond directly to specific joints. Instead, low-frequency regions are defined as areas where joints predominantly contain low-frequency information while also exhibiting a relatively higher proportion of high-frequency details.
Fig. 2: Comparison between our proposed method and existing methods. During the training phase, we employ adversarial training for the perturbation generator and denoiser to enhance model robustness. Specifically, the perturbation generator attacks the observed motion, producing motions that are challenging to reconstruct yet resemble normal motions. These perturbed motions are then used to train the denoiser, thereby improving its robustness. During the inference phase, we apply DCT to separate observed motion into global and local components, represented as low-frequency and high-frequency information. By leveraging high-frequency information as guidance, our method can accurately reconstruct observed motion compared to existing methods.
Fig. 3: The illustration of reconstruction-based method. The reconstruction-based method struggles to capture the fine details of unseen actions (e.g., the same action from a different camera view in Fig.(b)) but excels at reconstructing the global action information (as shown by the blue line).
Fig. 4: The framework of the proposed method. The model is trained utilizing generated perturbation examples. The training phase includes two processes: minimizing the mean square error to train the noise predictor εθ and maximizing this error to train the perturbation generator Gϕ . During the testing phase, the high-frequency information of observed motions and the low-frequency information of generated motions are fused for effective anomaly detection.
Fig. 5: The illustration of perturbation training. In Fig. (a), the green and yellow points denote the original training xk and perturbed motion x^k , respectively. The red region represents the distribution of unseen normal samples. Accordingly, Fig. (b) demonstrates that the reconstruction domain is extended by our proposed perturbation training.
Fig. 6: The visualization of human motions processed by 2D-DCT. (a) original motions; (b) motions with low-frequency information only; (c) the comparison between (a) and (b); (d) the skeletal example. Note that the red lines in (d) denote the discarded high-frequency information, and the red circles represent the high-frequency joints w.r.t. temporal and spatial dimension.
Algorithm 2 Frequency-Guided Motion Denoising Process
Type
Method
Venue
Modality
Avenue
HR-Avenue
HR-STC
UBnormal
HR-UBnormal
Pred.
MPED-RNN-Pred. [ 7 ]
CVPR’ 2019
Skeleton
-
-
74.5
-
-
Multi-Time. Pred. [ 18 ]
WACV’ 2020
Skeleton
-
88.3
77.0
-
-
PoseCVAE [ 75 ]
ICPR’ 2021
Skeleton
-
87.8
75.7
-
-
AMMC [ 1 ]
AAAI’ 2021
RGB
86.6
-
-
-
-
F 2 PN [ 76 ]
T-PAMI’ 2022
RGB
85.7
-
-
-
-
TrajREC-Ftr. [ 17 ]
WACV’ 2024
Skeleton
-
89.4
77.9
68.0
68.2
TABLE I: Comparison of the proposed method against other SoTA methods. The best results across all methods are in bold, the second-best ones are underlined, “Pred.” and “Rec.” denote predict-based and reconstruction-based methods, respectively.
Method
Training
Params
UBnormal
Memory
FPS
Sultani et al. [ 79 ]
S.
-
50.3
<11 GB
30.7
AED-SSMTL [ 80 ]
S.
> 80M
61.3
<11 GB
23.9
TimeSformer [ 81 ]
S.
121M
68.5
<11 GB
10.2
AED-SSMTL [ 80 ]
W.S.
> 80M
59.3
<11 GB
23.9
FG-Diff (Ours)
U.S.
556K
68.9
<6 GB
50.6
TABLE II: Comparison with supervised and weakly supervised methods. “W.S.”, “U.S.”, and “S.” denote weakly supervised, unsupervised and supervised methods, respectively. The FPS rates are measured on a single 1080Ti GPU with 11GB of VRAM.
Method
HR-Avenue
HR-STC
HR-UBnormal
Baseline
87.5 ( ↓ 3.2)
75.2 ( ↓ 3.4)
64.4 ( ↓ 4.6)
Ours w/o IP
90.4 ( ↓ 0.3)
77.4 ( ↓ 1.2)
68.7 ( ↓ 0.3)
Ours w/ double IP
90.7 (-)
78.5 ( ↓ 0.1)
68.6 ( ↓ 0.4)
Ours w/o DCT-Mask
89.9 ( ↓ 0.8)
78.0 ( ↓ 0.6)
68.1 ( ↓ 0.9)
Ours
90.7
78.6
69.0
TABLE III: Ablation studies of each component in the proposed method.
λPI
w/ PT
w/o PT
HR-A.
HR-S.
HR-U.
HR-A.
HR-S.
HR-U.
0.00
90.7
78.7
68.5
88.7
76.2
67.4
0.02
90.6
78.4
68.8
86.9
75.0
66.8
0.05
90.7
78.2
68.7
84.7
73.8
64.2
0.10
90.7
78.6
69.0
80.3
71.2
63.5
0.15
90.5
77.9
68.1
76.4
69.5
61.3
TABLE IV: Robust analysis of perturbations training. “w/ PT” refers to the method utilizing our perturbation training, while “w/o PT” indicates the approach without perturbation training, but with Gaussian noise augmentation instead. The symbol “ λPI ” denotes the perturbation intensity during inference.
Fig. 7: Sensitivity analyses of (a) DCT-Mask threshold λdct , and (b) perturbation training weight λp . As the model is probability-based, we performed 10 independent evaluations and reported the resulting average value.
Fig. 8: Performance comparison on unseen motions from UBnormal dataset. (a) qualitative visualization; (b) quantitative results.
PointNet++ [ 82 ]
DGCNN [ 83 ]
SPIL [ 84 ]
ST-GCN [ 85 ]
PDSF [ 86 ]
Ours
78.2
80.6
89.3
83.3
82.5
91.6
TABLE V: The performance comparison of the violence action detection methods on the RWF-2000 dataset.
Fig. 9: Performance analysis of specific behaviors.
Fig. 10: Performance of occluded actions with (a) spatial masks and (b) temporal masks.
Fig. 11: Visualization of denoising process in 2D t-SNE space. The inherent stochasticity enables the reconstruction model to handle diverse actions, but the diversity of the generative poses challenges for detail reconstruction.
Fig. 12: The low-frequency information w.r.t (a) spatiality and (b) temporality.
Fig. 13: Anomaly score curves on the Avenue and HR-UBnormal datasets. (a) Avenue dataset; (b) HR-UBnormal dataset. The horizontal axis represents the frame index, the red circles in the clip of each figure denote the abnormal events, and the green circles represent the normal ones.
Video Anomaly Detection (VAD) is inherently challenging due to the scarcity of anomalies and the large visual variability in surveillance footage, including changes in lighting, viewpoint, and human appearance. To mitigate visual noise and address privacy concerns, recent work has shifted to pose-based VAD, which focuses on motion dynamics rather than raw video data. However, existing pose-based approaches model human behavior in continuous latent spaces, limiting their ability to learn compact motion patterns necessary for robust behavior analysis. We address this by proposing Vector-Quantized Video Anomaly Detection (VQ-VAD), a novel human-centric anomaly detection framework that learns discrete motion representations. VQ-VAD adapts Vector-Quantized GAN (VQ-GAN), originally developed for image generation, to operate on keypoint sequences and construct a motion codebook of normal behavior. Trained exclusively on normal motion sequences, VQ-VAD detects anomalies by identifying high reconstruction errors when an observed motion sequence cannot be mapped to the learned codebook. We conduct extensive experiments across three complementary evaluation settings, including in-domain, cross-domain, and cross-dataset generalization, on four anomaly detection benchmarks. VQ-VAD achieves strong in-domain accuracy (81.83% on HR-SHT [15]), effective cross-domain transfer from CMU Panoptic [14] (76.69% on HR-SHT [15] without retraining), and competitive cross-dataset robustness. The code base for this work is available at https://github.com/TeCSAR-UNCC/VQ-VAD.
Narges Rashvand, Ghazal Alinezhad Noghre, Shanle Yao +2
Electrical and Computer Engineering Department University of North Carolina at Charlotte Charlotte, NC, USA
Video Anomaly Detection (VAD) aims to temporally localize abnormal events in videos. Most existing approaches rely on dataset-specific training and curated annotations, limiting generalization in open-set scenarios. Recent zero-shot methods based on Large Vision- Language Models (LVLMs) alleviate this dependency but often lack temporal continuity and structured reasoning. We propose Cog-VADU, a fully training-free framework that reformulates VAD as a sequential cognitive reasoning task. Cog-VADU introduces Chain-of- Anomaly Detection Thought Prompting (CoADTP), which unrolls an LVLM into a recurrent reasoning chain across video segments. By propagating structured rationales over time, the model maintains implicit temporal memory, enabling robust discrimination between com- plex anomalies and high-motion normal activities. To improve reliability, we further design a cross-modal re-ranking stage that aligns textual rationales with visual embeddings, enforcing semantic consistency and temporal coherence for refined and stable predictions. Extensive experiments on multiple public VAD benchmarks demonstrate that Cog-VADU achieves competitive zero-shot performance. Moreover, cross-model evaluations show that CoADTP consistently enhances reasoning-based anomaly detection in a model-agnostic manner, pro- viding interpretable and generalizable anomaly understanding for real-world applications.
Mohd Ubaid Wani, Sara Atito, Josef Kittler +1
Centre for Vision, Speech and Signal Processing (CVSSP) University of Surrey, UK · Surrey Institute for People-Centred AI (PAI) University of Surrey, UK
The collection and detection of video anomaly data has long been a challenging problem due to its rare occurrence and spatio-temporal scarcity. Existing video anomaly detection (VAD) methods under perform in open-world scenarios. Key contributing factors include limited dataset diversity, and inadequate understanding of context-dependent anomalous semantics. To address these issues, i) we propose LAVIDA, an end-to-end zero-shot video anomaly detection framework. ii) LAVIDA employs an Anomaly Exposure Sampler that transforms segmented objects into pseudo-anomalies to enhance model adaptability to unseen anomaly categories. It further integrates a Multimodal Large Language Model (MLLM) to bolster semantic comprehension capabilities. Additionally, iii) we design a token compression approach based on reverse attention to handle the spatio-temporal scarcity of anomalous patterns and decrease computational cost. The training process is conducted solely on pseudo anomalies without any VAD data. Evaluations across four benchmark VAD datasets demonstrate that LAVIDA achieves SOTA performance in both frame-level and pixel-level anomaly detection under the zero-shot setting. Our code is available in https://github.com/VitaminCreed/LAVIDA.
Zunkai Dai, Ke Li, Jiajia Liu +2
Beijing University of Posts and Telecommunications · Northwestern Polytechnical University