STEP: Score-Based Temporal Energy for Human Pose Video Anomaly Detection
Authors: Jakub Micorek, Mateusz Koziński, Horst Possegger
Organizations: Institute of Visual Computing, Graz University of Technology, Austria · Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Austria
Skeleton-based Video Anomaly Detection (VAD) offers a robust, privacy-preserving solution for identifying abnormal behaviors. To model the distribution of normal static and moving poses, recent methods train Energy-Based Models (EBMs) via Denoising Score Matching (DSM). However, directly injecting noise, required for training, into raw joint coordinates creates physically impossible poses, and this structural collapse severely worsens as the temporal window expands. To address this, we introduce STEP, a simple framework that utilizes Principal Component Analysis (PCA) to project pose sequences into a compact, whitened PC-space. Learning the data density within this well-behaved PC-space ensures that the injected noise translates into physically plausible variations, which allows the model to process longer video sequences without the performance collapse of raw coordinate baselines. Additionally, to mitigate inherent pose estimation inaccuracies arising from occlusions or motion blur, we integrate a sequence-level weighting mechanism based on the estimator's confidence scores. Operating at real-time computational efficiency, our simple and lightweight framework outperforms the previous skeleton-based state-of-the-art by 12.2% (90.1% AUROC) on the challenging UBnormal dataset and achieves highly competitive results by improving on the ShanghaiTech benchmark.
Figures & tables
Figure 1: The STEP Pipeline. Left: Human poses are extracted from the raw video. We highlight a static person (blue), a walking person (green), and an anomalous cyclist (red). Naively adding independent Gaussian noise to pixel coordinates on the static person destroys the kinematic structure, creating physically impossible poses (gray). Middle: In contrast, by projecting the sequences into a compact, whitened PC-space, the injected isotropic noise translates into semantically meaningful action variations ( e.g . , smoothly inducing motion in the static blue pose sequence) while yielding structurally plausible bone lengths and human proportions. Right: Consequently, the Energy-Based Model learns smoothed, well-behaved energy landscapes at different σ scales (top). During inference, we evaluate the sequence across distinct noise scales ( σlow to σhigh ). Normal actions (static and walking) fall into lower-energy basins across these scales, while the anomalous cyclist yields a high-energy score.
Figure 2: The principal components correspond to semantically meaningful actions : The image shows the effect of shifting a static pose along the first two principal components. PC1 corresponds to walking across the field of view, while PC2 represents walking towards or away from the camera.
Figure 3: STEP Architecture. Left: Overview of our energy model. The noise scale σ is integrated via early fusion with the whitened pose sequence π(x) and additionally input to every stacked block. The network fθ(π(x),σ) outputs a scalar energy estimate. The dashed green path illustrates the energy gradient calculation, which is required only during training. Right: Internal structure of the modulated block, combining conditioning on σ with a residual skip connection.
Method
ShanghaiTech
UBnormal
Full
HR
Full
HR
BiPOCO [ 16 ]
-
74.9
50.7
52.3
MPED-RNN [ 24 ]
73.4
75.4
60.6
61.2
MTP [ 27 ]
76.0
77.0
-
-
GEPC [ 22 ]
76.1
74.8
53.4
55.2
PoseCVAE [ 15 ]
-
75.5
-
-
Table 1: State-of-the-art comparison on ShanghaiTech and UBnormal. STEP sets a new state-of-the-art on UBnormal and matches performance on ShanghaiTech. Results are reported for both the Full test set and the Human-Related (HR) splits. To ensure rigorous evaluation, the STEP results are reported as the mean and standard deviation across 20 independent training runs. Best results per column are in bold , second best are underlined . Results show AUROC (%). † reproduced by us.
Figure 4: Qualitative results on ShanghaiTech (clip 01_0025 ). Each person’s track is assigned a unique color (its text box matches the time-series plot below each frame); skeleton joints are colored from green (low anomaly score) to red (high anomaly score). The number shown above each person is the mean sequence confidence; below it is the confidence-weighted anomaly score. The temporal plot shows all active tracks’ scores over time; the red-shaded region marks the ground-truth anomaly interval. Left, frames 150–200: STEP correctly detects a passing cyclist (cyan track) as anomalous, producing a sharp anomaly score spike. Right, frames 300–370: An insightful false positive: a pedestrian (orange track) turns and walks backwards, receiving an elevated score despite no anomaly label, revealing the model has learned forward-facing walking as normal.
Method
PCA ( K )
Conf.
Temporal Window ( T )
1
2
4
8
12
16
24
32
MULDE †
×
×
80.6
80.7
78.7
75.7
72.2
72.6
72.6
70.6
STEP
×
✓
86.0
87.2
86.3
82.4
81.3
80.5
78.3
73.1
STEP
48
×
-
82.1
83.2
84.1
84.0
83.6
82.3
81.3
STEP
16
✓
86.7
87.9
88.0
87.3
86.4
85.0
82.9
81.6
STEP
32
✓
86.7
89.5
88.4
88.6
88.9
89.0
87.5
85.7
Table 2: Ablation of temporal window ( T ), PCA dimension ( K ), and confidence weighting on UBnormal. Naively applying DSM to raw coordinates (MULDE) causes structural collapse as T expands. Projecting sequences onto our PC-space stabilizes this degradation, while confidence weighting mitigates tracking noise. The combination reveals a clear capacity trade-off, with the highest-performing configurations shifting diagonally as T increases. Results show AUROC (%). Best results per row are in bold , second best are underlined . Missing values (-) denote invalid configurations where the target PCA dimension K exceeds the input dimension. All results are reported for a fixed random seed. † reproduced by us.
Configuration
ShanghaiTech
UBnormal
Full STEP (Ours)
86.3
Δ
90.4
Δ
Architecture & PC-space
w/o σ− modulation Residual MLP (Vanilla MLP)
85.6
-0.7
89.3
-1.1
w/o Whitening
85.5
-0.8
88.2
-2.2
Training Confidence Handling
Drop Pose (Confidence <0.1 )
86.4
+0.1
90.5
+0.1
Table 3: Component Ablation of STEP across both datasets at T=12,K=48 . Removing our core architectural choices or aggressively dropping uncertain training poses results in consistent performance drops across both datasets. All AUROC (%) results are reported for a fixed seed.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Successful anomaly detection on UBnormal. This four-frame sequence captures an extreme anomaly where a van strikes a pedestrian (identified by the red box). Our STEP framework accurately detects the severe kinematic violation: the anomaly score sharply spikes upon the initial impact and remains heavily elevated while the person is lying unnaturally on the ground. Both the dynamic fall and the static abnormal pose are correctly flagged as highly anomalous (indicated by the red skeletal overlay). Video clip: abnormal_scene_24_scenario_2 .
Figure S2 : Qualitative example on ShanghaiTech. This four-frame sequence captures both a successful detection and an insightful false positive. First, our model successfully detects a passing cyclist ( i.e . cyan ID) as a clear anomaly. In the subsequent three frames, three pedestrians walk from left to right. One pedestrian ( i.e . orange ID) turns and proceeds to walk backwards. While backward walking is not labeled as an anomaly in the dataset ground truth, our system flags it with an elevated anomaly score. This highlights that our model effectively learned the standard, forward-facing pedestrian kinematics that define normality in the training set, causing it to intuitively recognize this rare backward motion as an anomalous event. Video clip: 01_0025 .
Figure S3 : Tracking failure and subsequent detection on UBnormal. This sequence highlights a fundamental limitation of skeleton-based approaches: the reliance on upstream pose estimators. Initially, a pedestrian ( i.e . blue ID) experiences a seizure and collapses. However, due to the extreme pose and environmental factors ( e.g . , fog), the tracker completely loses the subject mid-fall. Because no skeleton is detected, our framework cannot evaluate the motion. Later in the sequence, a doctor ( i.e . orange ID) rushes to assist the fallen individual. The tracker successfully captures this new subject, and our system correctly flags the fast-paced running motion as anomalous. This demonstrates that while STEP is robust, its performance is ultimately bottlenecked by the reliability of the foundational pose extraction. Video clip: abnormal_scene_8_scenario_1_fog .
Figure S4 : Confidence Weighting on UBnormal: We show the impact of using the confidence weighting. Left column without, right column with confidence weighting. The occluded person sitting on the bench has lower confidence. Without confidence reweighting, the normal sitting person would be wrongly detected as an anomaly. Video clip: abnormal_scene_10_scenario_3 .
Figure S5 : Exhaustive temporal and dimensional sweep. We evaluate the impact of the temporal window length ( T ) and manifold dimensionality ( K ) on UBnormal (left) and ShanghaiTech (right). Across both datasets, our PCA-projected STEP framework maintains stable, high performance at extended temporal window lengths, whereas the performance of baselines operating directly on raw coordinates degrades substantially as T increases.
Figure S6 : Architectural comparison. We evaluate the performance differences across varying temporal lengths ( T ) and PCA dimensions ( K ) on UBnormal (left) and ShanghaiTech (right). The σ -modulated Residual MLP (diamonds) consistently outperforms the Vanilla MLP (pentagons) across almost all configurations, proving the benefit of explicitly conditioning the network on the noise scale.
Modality
Method
ShanghaiTech
UBnormal
D
M
S
Full
HR
Full
HR
Multi-Modal / Pixel-Based Methods
✓
sRNN [ 20 ]
68.0
-
-
-
✓
Conv-AE [ 10 ]
70.4
69.8
-
-
✓
LSA [ 1 ]
72.5
-
-
-
✓
✓
FFP [ 18 ]
72.8
72.7
-
-
Appendix
Table S1 : Extended state-of-the-art comparison using the AUROC (%) metric. Input modality acronyms denote deep features (D), motion (M), and skeletons (S). STEP strictly outperforms all previous methods, including heavy multi-modal architectures, on both benchmarks. † reproduced by us.
Dataset
Method
AUROC
AP
Full
HR
Full
HR
UBnormal
STG-NF [ 11 ]
71.8
71.5
62.7
67.2
SeeKer [ 4 ]
77.9
78.9
80.3
79.8
STEP
90.3
90.9
91.6
91.7
ShanghaiTech
STG-NF [ 11 ]
85.9
87.4
77.6
81.4
SeeKer [ 4 ]
85.5
86.9
80.0
81.5
Appendix
Table S2 : Comparison on AUROC (%) and Average Precision (AP %) on UBnormal and ShanghaiTech. Best results are in bold , and second-best are underlined .
Figure S7 : UBnormal T/K Heatmaps: The best performing PCA/kNN configurations do not strictly align with our EBM’s optimal configuration. However, even the simple PCA/kNN baseline comfortably outperforms the current SOTA.
Figure S8 : ShanghaiTech T/K Heatmaps: Similar to UBnormal, the optimal temporal and dimensional capacities for distance-based scoring (kNN) differ from density-based scoring (STEP).
T
σhigh=1
UBnormal ( K )
ShanghaiTech ( K )
σlow
32
48
64
32
48
64
12
1e-4
89.13
89.56
89.38
85.52
86.05
85.22
0.001
89.33
89.24
89.48
85.43
85.97
85.42
0.01
88.93
89.70
89.48
84.98
85.85
85.68
0.1
88.91
90.37
90.54
85.64
86.28
85.45
0.5
86.37
86.91
86.47
83.72
83.88
82.65
Appendix
Table S3 : Impact of minimum noise scale ( σlow ) across different temporal windows ( T ) and PCA dimensions ( K ) with a fixed σhigh=1 . Results are in AUROC (%). For a given T/K setting we mark the best performing in bold , underline the second best and the third best is in italic .
T
σlow=0.1
UBnormal ( K )
ShanghaiTech ( K )
σhigh
32
48
64
32
48
64
12
0.2
89.49
89.87
90.14
86.03
86.38
85.77
0.5
89.48
90.57
90.24
85.74
86.49
86.02
1
88.91
90.37
90.54
85.64
86.28
85.54
2
89.12
89.89
90.43
85.77
86.10
85.29
16
0.2
88.50
89.00
89.71
85.08
86.05
85.96
Appendix
Table S4 : Impact of maximum noise scale ( σhigh ) across different temporal windows ( T ) and PCA dimensions ( K ) with a fixed σlow=0.1 . Results are in AUROC (%). For a given T/K setting we mark the best performing in bold , underline the second best and the third best is in italic .
T
Drop Conf.
UBnormal ( K )
ShanghaiTech ( K )
16
32
48
64
16
32
48
64
8
0
87.25
88.61
89.22
89.96
84.61
86.04
85.92
85.11
0.1
87.25
88.89
89.14
90.27
84.84
86.16
86.03
85.25
0.2
87.86
89.17
89.49
89.23
84.98
85.74
85.81
85.07
0.4
86.37
86.43
86.51
85.43
84.12
84.95
84.43
83.36
12
0
86.43
88.91
90.37
90.54
84.67
85.64
86.28
85.45
Appendix
Table S5 : Impact of dropping noisy poses during training across different temporal windows ( T ) and PCA dimensions ( K ). Poses below the confidence threshold are discarded. Results are in AUROC (%). For a given T/K setting we mark the best performing in bold , underline the second best and the third best is in italic .
Aggregation Strategy
PCA Manifold Dimension ( K )
8
16
32
48
64
128
256
Agg-Sum
79.7
83.3
87.2
87.2
87.1
84.8
81.5
Best Individual σ
81.9
86.9
89.6
90.3
90.1
89.2
85.3
Agg-Max (Ours)
82.3
86.4
88.9
90.4
90.1
88.8
85.7
Appendix
Table S6 : Impact of the multiscale aggregation strategy on the UBnormal dataset ( T=12 ). Agg-Max successfully captures anomalies across all structural frequencies without diluting the energy signal, drastically outperforming summation ( Agg-Sum ) and competing directly with the retrospectively chosen best individual scale.
Aggregation
UBnormal
ShanghaiTech
BT Raw
BT Smooth
On Raw
On Smooth
BT Raw
BT Smooth
On Raw
On Smooth
AUC
AP
AUC
AP
AUC
AP
AUC
AP
AUC
AP
AUC
AP
AUC
AP
AUC
AP
Full Test-set
Agg-Max
87.73
91.31
90.99
92.28
85.03
89.84
90.59
91.90
83.93
81.08
85.39
83.20
83.55
81.41
86.32
84.52
Agg-Sum
85.76
89.58
88.17
91.41
83.53
88.45
86.92
90.64
81.95
79.08
83.71
81.36
81.86
79.48
84.88
82.75
Agg-Mean-Top2
87.67
91.27
90.89
92.21
84.98
89.80
90.50
91.84
83.78
80.95
85.26
83.07
83.41
81.27
86.20
84.38
Appendix
Table S7 : Comprehensive evaluation of aggregation strategies across different inference protocols on the Full and Human-Related (HR) splits. We evaluate both offline/non-causal Backtrack (BT) and real-time/causal Online (On) settings, with and without temporal 1D-Gaussian smoothing. Results are in AUROC (%) and AP (%). We mark the best performing strategy in bold , underline the second best, and the third best is in italic .
Test \ Train
ShanghaiTech
UBnormal
Both
ShanghaiTech
86.3
77.2
86.0
UBnormal
82.4
90.6
88.9
Appendix
Table S8 : Cross-Dataset Evaluation: We evaluate the generalization of STEP by testing models on datasets they were not trained on. Finally, we jointly train both datasets in a single model and evaluate on both datasets. Results reported are AUROC (%).
Method
Variant
AUROC (%)
BT
On
Raw
Smooth
Raw
Smooth
STG-NF [ 11 ]
MSAD-HR
55.7 †
SeeKer [ 4 ]
MSAD-HR
61.1 †
STEP (Ours)
MSAD (includes all: HR and non-HR)
57.6
59.9
56.4
59.6
MSAD-HR / ST-Style (drop non-HR clips)
71.7
72.8
69.9
71.6
Appendix
Table S9 : AUROC (%) on MSAD [ 36 ] under evaluation protocol (i): 360 held-out clips (120 normal + 240 abnormal). ST-Style (as in ShanghaiTech): excludes non-HR clips. UB-Style (as in UBnormal): marks non-HR anomalous frames as normal. BT : Backtrack (non-causal). On : Online (causal, real-time). Raw / Smooth : without/with temporal 1D-Gaussian smoothing. † HR split style unspecified.
K
T
Persons/Frame
Latency (ms)
FPS
Peak Mem (MB)
48
12
1
0.86
1167
21.5
48
12
10
0.93
1077
22.8
48
12
20
0.90
1109
25.2
48
12
30
0.92
1087
28.6
48
12
40
0.95
1051
30.0
48
12
50
0.97
1026
32.6
Appendix
Table S10 : Inference performance of STEP once poses are extracted. Latency covers the entire pipeline from raw coordinates to final frame-level scores ( L=10 noise scales σ ). K denotes the PCA dimension and T represents the temporal segment length.
Figure S9 : Latent Traversal in the whitened PC-space. We systematically add structured variance ( σ ) along the first two principal components. Top (Static Pose): Adding variance to a stationary person induces a coherent walking motion. Bottom (Moving Pose): Altering the latent coordinates of an already walking person smoothly alters the style and spatial orientation. The ghosting gray skeleton represents the zero-noise center pose for reference.
Figure S10 : PCA as a Low-Pass Filter. Top Row: Adding independent Gaussian noise directly to raw pixel coordinates destroys the structure of a moving person. Bottom Rows: Projecting these corrupted sequences through various PCA bottlenecks filters the noise. An overly tight bottleneck ( K=8 ) fails to recover complex motion, while a loose bottleneck ( K=256 ) retains too many high-frequency components and remains sensitive to the noise. An appropriate K=48 configuration successfully removes unstructured jitter while preserving underlying motion.
Figure S11 : Reconstruction Error under Input Noise ( T=12 , UBnormal). As we increase the noise on the input pose coordinates, the poses can still be well reconstructed within the appropriate PCA capacity range of K∈[32,64] . For larger PCA dimensions, the PC-space becomes too permissive and attempts to reconstruct the structural noise, causing the error to rise.
Method
Pose Jitter ( σpixel )
0
0.01
0.05
0.10
0.20
0.30
0.50
1.00
UBnormal
Agg-Max
90.62
90.63
90.44
89.12
82.46
74.80
68.95
64.24
Agg-Sum
87.57
87.58
87.70
87.67
86.41
82.14
72.21
62.89
Best Ind.
90.49
90.49
90.21
89.53
86.98
85.50
79.13
65.82
ShanghaiTech
Appendix
Table S11 : Robustness to raw coordinate jitter during inference on UBnormal and ShanghaiTech ( T=12,K=48 ). We inject independent Gaussian noise ( σpixel ) into the raw input coordinates prior to the PCA projection. Up to σpixel=0.1 , the PCA effectively acts as a low-pass filter, maintaining state-of-the-art performance. Tested on a single model with a fixed random seed.
Projection Method
PCA
AE
VAE
Pretrain using Anomalies
×
✓
×
✓
×
✓
AUROC (%)
90.5
90.5
89.5
90.1
84.9
86.2
Appendix
Table S12 : Comparison of linear and non-linear extractors on the UBnormal benchmark ( T=12,K=48 ). Pretrain using Anomalies indicates the projection manifold was constructed using both normal and abnormal training data, rather than strictly normal data. The linear PCA projection outperforms deep Autoencoders, even when the latter are provided with abnormal training sequences.
As embodied perception systems increasingly bridge digital and physical realms in interactive multimedia applications, the need for privacy-preserving approaches to understand human activities in physical environments has become paramount. Video anomaly detection is a critical task in such embodied multimedia systems for intelligent surveillance and forensic analysis. Skeleton-based approaches have emerged as a privacy-preserving alternative that processes physical world information through abstract human pose representations while discarding sensitive visual attributes such as identity and facial features. However, existing skeleton-based methods predominantly model continuous motion trajectories in a monolithic manner, failing to capture the hierarchical nature of human activities composed of discrete semantic primitives and fine-grained kinematic details, which leads to reduced discriminability when anomalies manifest at different abstraction levels. In this regard, we propose Motion Semantics Guided Normalizing Flow (MSG-Flow) that decomposes skeleton-based VAD into hierarchical motion semantics modeling. It employs vector quantized variational auto-encoder to discretize continuous motion into interpretable primitives, an autoregressive Transformer to model semantic-level temporal dependencies, and a conditional normalizing flow to capture detail-level pose variations. Extensive experiments on benchmarks (HR-ShanghaiTech & HR-UBnormal) demonstrate that MSG-Flow achieves state-of-the-art performance with 88.1% and 75.8% AUC respectively.
Yang Liu, Boan Chen, Yuanyuan Meng +5
Tongji Univ. · SJTU · Shanghai Creative Studies Institute +4
Video Anomaly Detection (VAD) is inherently challenging due to the scarcity of anomalies and the large visual variability in surveillance footage, including changes in lighting, viewpoint, and human appearance. To mitigate visual noise and address privacy concerns, recent work has shifted to pose-based VAD, which focuses on motion dynamics rather than raw video data. However, existing pose-based approaches model human behavior in continuous latent spaces, limiting their ability to learn compact motion patterns necessary for robust behavior analysis. We address this by proposing Vector-Quantized Video Anomaly Detection (VQ-VAD), a novel human-centric anomaly detection framework that learns discrete motion representations. VQ-VAD adapts Vector-Quantized GAN (VQ-GAN), originally developed for image generation, to operate on keypoint sequences and construct a motion codebook of normal behavior. Trained exclusively on normal motion sequences, VQ-VAD detects anomalies by identifying high reconstruction errors when an observed motion sequence cannot be mapped to the learned codebook. We conduct extensive experiments across three complementary evaluation settings, including in-domain, cross-domain, and cross-dataset generalization, on four anomaly detection benchmarks. VQ-VAD achieves strong in-domain accuracy (81.83% on HR-SHT [15]), effective cross-domain transfer from CMU Panoptic [14] (76.69% on HR-SHT [15] without retraining), and competitive cross-dataset robustness. The code base for this work is available at https://github.com/TeCSAR-UNCC/VQ-VAD.
Narges Rashvand, Ghazal Alinezhad Noghre, Shanle Yao +2
Electrical and Computer Engineering Department University of North Carolina at Charlotte Charlotte, NC, USA
Video anomaly detection is critical for public safety and security, yet remains highly challenging despite extensive research due to large variations in appearance, viewpoint, and scene dynamics. Among existing approaches, human pose-based methods have emerged as a major line of research, showing strong performance since many anomalies in public datasets involve humans and pose representations are robust to appearance changes while providing compact motion descriptions. However, these methods often overlook bounding-box trajectories, although such information is inherently available in pose-based pipelines. In this paper, we explicitly leverage these trajectories as a primary anomaly cue. We present TrajVAD, a framework that models multi-class bounding-box trajectories using normalizing flows to learn normal kinematic patterns. Its trajectory-only variant, TrajVAD-T, eliminates pose estimation, reaches 87.7 AP on ShanghaiTech, and achieves the best results on MSAD among compared methods. TrajVAD-P adds a reliability-gated pose branch and improves performance to 88.6 AUROC and 90.9 AP on ShanghaiTech, establishing bounding-box trajectories as an effective modality for video anomaly detection.