Frequency-Guided Diffusion Model with Perturbation Training for Skeleton-Based Video Anomaly Detection

Xiaofeng Tan; Xin Geng; Liang Wang; Hongsong Wang

cs.CVDec 4, 2024

Frequency-Guided Diffusion Model with Perturbation Training for Skeleton-Based Video Anomaly Detection

Authors: Xiaofeng Tan, Xin Geng, Liang Wang, Hongsong Wang

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing 211189, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA) · School of Artificial Intelligence, University of Chinese Academy of Sciences

Abstract

Video anomaly detection (VAD) is a vital yet complex open-set task in computer vision, commonly tackled through reconstruction-based methods. However, these methods struggle with two key limitations: (1) insufficient robustness in open-set scenarios, where unseen normal motions are frequently misclassified as anomalies, and (2) an overemphasis on, but restricted capacity for, local motion reconstruction, which are inherently difficult to capture accurately due to their diversity. To overcome these challenges, we introduce a novel frequency-guided diffusion model with perturbation training. First, we enhance robustness by training a generator to produce perturbed samples, which are similar to normal samples and target the weakness of the reconstruction model. This training paradigm expands the reconstruction domain of the model, improving its generalization to unseen normal motions. Second, to address the overemphasis on motion details, we employ the 2D Discrete Cosine Transform (DCT) to separate high-frequency (local) and low-frequency (global) motion components. By guiding the diffusion model with observed high-frequency information, we prioritize the reconstruction of low-frequency components, enabling more accurate and robust anomaly detection. Extensive experiments on five widely used VAD datasets demonstrate that our approach surpasses state-of-the-art methods, underscoring its effectiveness in open-set scenarios and diverse motion contexts. Our project website is https://xiaofeng-tan.github.io/projects/FG-Diff.

Figures & tables

Fig. 1: The data illustration. (a) The training and testing data, where the training data is composed of seen normal motions and the testing data contains unseen normal and abnormal motions. Although seen and unseen motions represent the same action (e.g., walking), their local details, such as stride length, arm swing amplitude, and joint angles, exhibit significant differences. (b) The frequency analyses of motions. This analysis reveals that a motion retaining only 70% of its low-frequency information remains largely similar to the original motion in terms of global structure, with minor differences observed in the low-frequency regions. Note that low-frequency and high-frequency regions do not correspond directly to specific joints. Instead, low-frequency regions are defined as areas where joints predominantly contain low-frequency information while also exhibiting a relatively higher proportion of high-frequency details.

Fig. 2: Comparison between our proposed method and existing methods. During the training phase, we employ adversarial training for the perturbation generator and denoiser to enhance model robustness. Specifically, the perturbation generator attacks the observed motion, producing motions that are challenging to reconstruct yet resemble normal motions. These perturbed motions are then used to train the denoiser, thereby improving its robustness. During the inference phase, we apply DCT to separate observed motion into global and local components, represented as low-frequency and high-frequency information. By leveraging high-frequency information as guidance, our method can accurately reconstruct observed motion compared to existing methods.

Fig. 3: The illustration of reconstruction-based method. The reconstruction-based method struggles to capture the fine details of unseen actions (e.g., the same action from a different camera view in Fig.(b)) but excels at reconstructing the global action information (as shown by the blue line).

Fig. 4: The framework of the proposed method. The model is trained utilizing generated perturbation examples. The training phase includes two processes: minimizing the mean square error to train the noise predictor

\varepsilon_{\theta}

and maximizing this error to train the perturbation generator

\mathcal{G}_{\phi}

. During the testing phase, the high-frequency information of observed motions and the low-frequency information of generated motions are fused for effective anomaly detection.

Fig. 5: The illustration of perturbation training. In Fig. (a), the green and yellow points denote the original training

x_{k}

and perturbed motion

\hat{x}_{k}

, respectively. The red region represents the distribution of unseen normal samples. Accordingly, Fig. (b) demonstrates that the reconstruction domain is extended by our proposed perturbation training.

Fig. 6: The visualization of human motions processed by 2D-DCT. (a) original motions; (b) motions with low-frequency information only; (c) the comparison between (a) and (b); (d) the skeletal example. Note that the red lines in (d) denote the discarded high-frequency information, and the red circles represent the high-frequency joints w.r.t. temporal and spatial dimension.

\small\mathbf{x}_{t-1}^{g}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}^{c}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(\mathbf{x}_{t}^{c},t,c)\right)+(1-\alpha_{t})\varepsilon

Algorithm 2 Frequency-Guided Motion Denoising Process

Type	Method	Venue	Modality	Avenue	HR-Avenue	HR-STC	UBnormal	HR-UBnormal
Pred.	MPED-RNN-Pred. [ 7 ]	CVPR’ 2019	Skeleton	-	-	74.5	-	-
	Multi-Time. Pred. [ 18 ]	WACV’ 2020	Skeleton	-	88.3	77.0	-	-
	PoseCVAE [ 75 ]	ICPR’ 2021	Skeleton	-	87.8	75.7	-	-
	AMMC [ 1 ]	AAAI’ 2021	RGB	86.6	-	-	-	-
	F 2 PN [ 76 ]	T-PAMI’ 2022	RGB	85.7	-	-	-	-
	TrajREC-Ftr. [ 17 ]	WACV’ 2024	Skeleton	-	89.4	77.9	68.0	68.2

TABLE I: Comparison of the proposed method against other SoTA methods. The best results across all methods are in bold, the second-best ones are underlined, “Pred.” and “Rec.” denote predict-based and reconstruction-based methods, respectively.

Method	Training	Params	UBnormal	Memory	FPS
Sultani et al. [ 79 ]	S.	-	50.3	$<11$ GB	30.7
AED-SSMTL [ 80 ]	S.	$>$ 80M	61.3	$<11$ GB	23.9
TimeSformer [ 81 ]	S.	121M	68.5	$<11$ GB	10.2
AED-SSMTL [ 80 ]	W.S.	$>$ 80M	59.3	$<11$ GB	23.9
FG-Diff (Ours)	U.S.	556K	68.9	$<6$ GB	50.6

TABLE II: Comparison with supervised and weakly supervised methods. “W.S.”, “U.S.”, and “S.” denote weakly supervised, unsupervised and supervised methods, respectively. The FPS rates are measured on a single 1080Ti GPU with 11GB of VRAM.

Method	HR-Avenue	HR-STC	HR-UBnormal
Baseline	87.5 ( $\downarrow$ 3.2)	75.2 ( $\downarrow$ 3.4)	64.4 ( $\downarrow$ 4.6)
Ours w/o IP	90.4 ( $\downarrow$ 0.3)	77.4 ( $\downarrow$ 1.2)	68.7 ( $\downarrow$ 0.3)
Ours w/ double IP	90.7 (-)	78.5 ( $\downarrow$ 0.1)	68.6 ( $\downarrow$ 0.4)
Ours w/o DCT-Mask	89.9 ( $\downarrow$ 0.8)	78.0 ( $\downarrow$ 0.6)	68.1 ( $\downarrow$ 0.9)
Ours	90.7	78.6	69.0

TABLE III: Ablation studies of each component in the proposed method.

$\lambda_{PI}$	w/ PT			w/o PT
$\lambda_{PI}$	HR-A.	HR-S.	HR-U.	HR-A.	HR-S.	HR-U.
0.00	90.7	78.7	68.5	88.7	76.2	67.4
0.02	90.6	78.4	68.8	86.9	75.0	66.8
0.05	90.7	78.2	68.7	84.7	73.8	64.2
0.10	90.7	78.6	69.0	80.3	71.2	63.5
0.15	90.5	77.9	68.1	76.4	69.5	61.3

TABLE IV: Robust analysis of perturbations training. “w/ PT” refers to the method utilizing our perturbation training, while “w/o PT” indicates the approach without perturbation training, but with Gaussian noise augmentation instead. The symbol “

\lambda_{PI}

” denotes the perturbation intensity during inference.

Panel of Figure 12 — Fig. 7: Sensitivity analyses of (a) DCT-Mask threshold $\lambda_{\text{dct}}$ , and (b) perturbation training weight $\lambda_{\text{p}}$ . As the model is probability-based, we performed 10 independent evaluations and reported the resulting average value.

Panel of Figure 13 — Fig. 8: Performance comparison on unseen motions from UBnormal dataset. (a) qualitative visualization; (b) quantitative results.

PointNet++ [ 82 ]	DGCNN [ 83 ]	SPIL [ 84 ]	ST-GCN [ 85 ]	PDSF [ 86 ]	Ours
78.2	80.6	89.3	83.3	82.5	91.6

TABLE V: The performance comparison of the violence action detection methods on the RWF-2000 dataset.

Fig. 9: Performance analysis of specific behaviors.

Panel of Figure 16 — Fig. 10: Performance of occluded actions with (a) spatial masks and (b) temporal masks.

Fig. 11: Visualization of denoising process in 2D t-SNE space. The inherent stochasticity enables the reconstruction model to handle diverse actions, but the diversity of the generative poses challenges for detail reconstruction.

Panel of Figure 18 — Fig. 12: The low-frequency information w.r.t (a) spatiality and (b) temporality.

Panel of Figure 19 — Fig. 13: Anomaly score curves on the Avenue and HR-UBnormal datasets. (a) Avenue dataset; (b) HR-UBnormal dataset. The horizontal axis represents the frame index, the red circles in the clip of each figure denote the abnormal events, and the green circles represent the normal ones.

Explore similar work

VQ-VAD: Vector-quantized Motion Representation Learning for Human-centric Video Anomaly Detection
Aug 5, 2026Narges Rashvand, Ghazal Alinezhad Noghre, Shanle Yao +2Vector Quantization Reconstruction-Based Anomaly Detection
Electrical and Computer Engineering Department University of North Carolina at Charlotte Charlotte, NC, USA
Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding
Oct 1, 2026Mohd Ubaid Wani, Sara Atito, Josef Kittler +1CoT Reasoning Interpretable Anomaly Detection
Centre for Vision, Speech and Signal Processing (CVSSP) University of Surrey, UK · Surrey Institute for People-Centred AI (PAI) University of Surrey, UK
No Need For Real Anomaly: MLLM Empowered Zero-Shot Video Anomaly Detection
Feb 22, 2026Zunkai Dai, Ke Li, Jiajia Liu +2Video Anomaly Detection Zero-Shot Anomaly Detection
Beijing University of Posts and Telecommunications · Northwestern Polytechnical University

Aug 5, 2026J/K move · Enter open · S save

VQ-VAD: Vector-quantized Motion Representation Learning for Human-centric Video Anomaly Detection

Narges Rashvand, Ghazal Alinezhad Noghre, Shanle Yao, Gabriel Maldonado+1

Electrical and Computer Engineering Department University of North Carolina at Charlotte Charlotte, NC, USA

Vector Quantization Reconstruction-Based Anomaly Detection Video Anomaly Detection

Open paper AskPDF arXiv

Video Anomaly Detection (VAD) is inherently challenging due to the scarcity of anomalies and the large visual variability in surveillance footage, including changes in lighting, viewpoint, and human appearance. To mitigate visual noise and address privacy concerns, recent work has shifted to pose-based VAD, which focuses on motion dynamics rather than raw video data. However, existing pose-based approaches model human behavior in continuous latent spaces, limiting their ability to learn compact motion patterns necessary for robust behavior analysis. We address this by proposing Vector-Quantized Video Anomaly Detection (VQ-VAD), a novel human-centric anomaly detection framework that learns discrete motion representations. VQ-VAD adapts Vector-Quantized GAN (VQ-GAN), originally developed for image generation, to operate on keypoint sequences and construct a motion codebook of normal behavior. Trained exclusively on normal motion sequences, VQ-VAD detects anomalies by identifying high reconstruction errors when an observed motion sequence cannot be mapped to the learned codebook. We conduct extensive experiments across three complementary evaluation settings, including in-domain, cross-domain, and cross-dataset generalization, on four anomaly detection benchmarks. VQ-VAD achieves strong in-domain accuracy (81.83% on HR-SHT [15]), effective cross-domain transfer from CMU Panoptic [14] (76.69% on HR-SHT [15] without retraining), and competitive cross-dataset robustness. The code base for this work is available at https://github.com/TeCSAR-UNCC/VQ-VAD.

Abstract

Explore similar work

VQ-VAD: Vector-quantized Motion Representation Learning for Human-centric Video Anomaly Detection

Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding

No Need For Real Anomaly: MLLM Empowered Zero-Shot Video Anomaly Detection