STEP: Score-Based Temporal Energy for Human Pose Video Anomaly Detection

Jakub Micorek; Mateusz Koziński; Horst Possegger

cs.CVAug 20, 2026

STEP: Score-Based Temporal Energy for Human Pose Video Anomaly Detection

Authors: Jakub Micorek, Mateusz Koziński, Horst Possegger

Organizations: Institute of Visual Computing, Graz University of Technology, Austria · Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Austria

Abstract

Skeleton-based Video Anomaly Detection (VAD) offers a robust, privacy-preserving solution for identifying abnormal behaviors. To model the distribution of normal static and moving poses, recent methods train Energy-Based Models (EBMs) via Denoising Score Matching (DSM). However, directly injecting noise, required for training, into raw joint coordinates creates physically impossible poses, and this structural collapse severely worsens as the temporal window expands. To address this, we introduce STEP, a simple framework that utilizes Principal Component Analysis (PCA) to project pose sequences into a compact, whitened PC-space. Learning the data density within this well-behaved PC-space ensures that the injected noise translates into physically plausible variations, which allows the model to process longer video sequences without the performance collapse of raw coordinate baselines. Additionally, to mitigate inherent pose estimation inaccuracies arising from occlusions or motion blur, we integrate a sequence-level weighting mechanism based on the estimator's confidence scores. Operating at real-time computational efficiency, our simple and lightweight framework outperforms the previous skeleton-based state-of-the-art by 12.2% (90.1% AUROC) on the challenging UBnormal dataset and achieves highly competitive results by improving on the ShanghaiTech benchmark.

Figures & tables

Figure 1: The STEP Pipeline. Left: Human poses are extracted from the raw video. We highlight a static person (blue), a walking person (green), and an anomalous cyclist (red). Naively adding independent Gaussian noise to pixel coordinates on the static person destroys the kinematic structure, creating physically impossible poses (gray). Middle: In contrast, by projecting the sequences into a compact, whitened PC-space, the injected isotropic noise translates into semantically meaningful action variations ( e.g . , smoothly inducing motion in the static blue pose sequence) while yielding structurally plausible bone lengths and human proportions. Right: Consequently, the Energy-Based Model learns smoothed, well-behaved energy landscapes at different $\sigma$ scales (top). During inference, we evaluate the sequence across distinct noise scales ( $\sigma_{low}$ to $\sigma_{high}$ ). Normal actions (static and walking) fall into lower-energy basins across these scales, while the anomalous cyclist yields a high-energy score.

Figure 2: The principal components correspond to semantically meaningful actions : The image shows the effect of shifting a static pose along the first two principal components. PC1 corresponds to walking across the field of view, while PC2 represents walking towards or away from the camera.

Panel of Figure 3 — Figure 3: STEP Architecture. Left: Overview of our energy model. The noise scale $\sigma$ is integrated via early fusion with the whitened pose sequence $\pi(\mathbf{x})$ and additionally input to every stacked block. The network $f_{\theta}(\pi(\mathbf{x}),\sigma)$ outputs a scalar energy estimate. The dashed green path illustrates the energy gradient calculation, which is required only during training. Right: Internal structure of the modulated block, combining conditioning on $\sigma$ with a residual skip connection.

Method	ShanghaiTech		UBnormal
Method	Full	HR	Full	HR
BiPOCO [ 16 ]	-	74.9	50.7	52.3
MPED-RNN [ 24 ]	73.4	75.4	60.6	61.2
MTP [ 27 ]	76.0	77.0	-	-
GEPC [ 22 ]	76.1	74.8	53.4	55.2
PoseCVAE [ 15 ]	-	75.5	-	-

Table 1: State-of-the-art comparison on ShanghaiTech and UBnormal. STEP sets a new state-of-the-art on UBnormal and matches performance on ShanghaiTech. Results are reported for both the Full test set and the Human-Related (HR) splits. To ensure rigorous evaluation, the STEP results are reported as the mean and standard deviation across 20 independent training runs. Best results per column are in bold , second best are underlined . Results show AUROC (%). † reproduced by us.

Panel of Figure 4 — Figure 4: Qualitative results on ShanghaiTech (clip 01_0025 ). Each person’s track is assigned a unique color (its text box matches the time-series plot below each frame); skeleton joints are colored from green (low anomaly score) to red (high anomaly score). The number shown above each person is the mean sequence confidence; below it is the confidence-weighted anomaly score. The temporal plot shows all active tracks’ scores over time; the red-shaded region marks the ground-truth anomaly interval. Left, frames 150–200: STEP correctly detects a passing cyclist (cyan track) as anomalous, producing a sharp anomaly score spike. Right, frames 300–370: An insightful false positive: a pedestrian (orange track) turns and walks backwards, receiving an elevated score despite no anomaly label, revealing the model has learned forward-facing walking as normal.

Method	PCA ( $K$ )	Conf.	Temporal Window ( $T$ )
Method	PCA ( $K$ )	Conf.	1	2	4	8	12	16	24	32
MULDE †	$\times$	$\times$	80.6	80.7	78.7	75.7	72.2	72.6	72.6	70.6
STEP	$\times$	✓	86.0	87.2	86.3	82.4	81.3	80.5	78.3	73.1
STEP	48	$\times$	-	82.1	83.2	84.1	84.0	83.6	82.3	81.3
STEP	16	✓	86.7	87.9	88.0	87.3	86.4	85.0	82.9	81.6
STEP	32	✓	86.7	89.5	88.4	88.6	88.9	89.0	87.5	85.7

Table 2: Ablation of temporal window (

T

), PCA dimension (

K

), and confidence weighting on UBnormal. Naively applying DSM to raw coordinates (MULDE) causes structural collapse as

T

expands. Projecting sequences onto our PC-space stabilizes this degradation, while confidence weighting mitigates tracking noise. The combination reveals a clear capacity trade-off, with the highest-performing configurations shifting diagonally as

T

increases. Results show AUROC (%). Best results per row are in bold , second best are underlined . Missing values (-) denote invalid configurations where the target PCA dimension

K

exceeds the input dimension. All results are reported for a fixed random seed. † reproduced by us.

Configuration	ShanghaiTech		UBnormal
Full STEP (Ours)	86.3	$\Delta$	90.4	$\Delta$
Architecture & PC-space
w/o $\sigma-$ modulation Residual MLP (Vanilla MLP)	85.6	-0.7	89.3	-1.1
w/o Whitening	85.5	-0.8	88.2	-2.2
Training Confidence Handling
Drop Pose (Confidence $<0.1$ )	86.4	+0.1	90.5	+0.1

Table 3: Component Ablation of STEP across both datasets at

T=12,K=48

. Removing our core architectural choices or aggressively dropping uncertain training poses results in consistent performance drops across both datasets. All AUROC (%) results are reported for a fixed seed.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure S1 : Successful anomaly detection on UBnormal. This four-frame sequence captures an extreme anomaly where a van strikes a pedestrian (identified by the red box). Our STEP framework accurately detects the severe kinematic violation: the anomaly score sharply spikes upon the initial impact and remains heavily elevated while the person is lying unnaturally on the ground. Both the dynamic fall and the static abnormal pose are correctly flagged as highly anomalous (indicated by the red skeletal overlay). Video clip: abnormal_scene_24_scenario_2 .

Panel of Figure 9 — Figure S2 : Qualitative example on ShanghaiTech. This four-frame sequence captures both a successful detection and an insightful false positive. First, our model successfully detects a passing cyclist ( i.e . cyan ID) as a clear anomaly. In the subsequent three frames, three pedestrians walk from left to right. One pedestrian ( i.e . orange ID) turns and proceeds to walk backwards. While backward walking is not labeled as an anomaly in the dataset ground truth, our system flags it with an elevated anomaly score. This highlights that our model effectively learned the standard, forward-facing pedestrian kinematics that define normality in the training set, causing it to intuitively recognize this rare backward motion as an anomalous event. Video clip: 01_0025 .

Panel of Figure 10 — Figure S3 : Tracking failure and subsequent detection on UBnormal. This sequence highlights a fundamental limitation of skeleton-based approaches: the reliance on upstream pose estimators. Initially, a pedestrian ( i.e . blue ID) experiences a seizure and collapses. However, due to the extreme pose and environmental factors ( e.g . , fog), the tracker completely loses the subject mid-fall. Because no skeleton is detected, our framework cannot evaluate the motion. Later in the sequence, a doctor ( i.e . orange ID) rushes to assist the fallen individual. The tracker successfully captures this new subject, and our system correctly flags the fast-paced running motion as anomalous. This demonstrates that while STEP is robust, its performance is ultimately bottlenecked by the reliability of the foundational pose extraction. Video clip: abnormal_scene_8_scenario_1_fog .

Panel of Figure 11 — Figure S4 : Confidence Weighting on UBnormal: We show the impact of using the confidence weighting. Left column without, right column with confidence weighting. The occluded person sitting on the bench has lower confidence. Without confidence reweighting, the normal sitting person would be wrongly detected as an anomaly. Video clip: abnormal_scene_10_scenario_3 .

Panel of Figure 12 — Figure S5 : Exhaustive temporal and dimensional sweep. We evaluate the impact of the temporal window length ( $T$ ) and manifold dimensionality ( $K$ ) on UBnormal (left) and ShanghaiTech (right). Across both datasets, our PCA-projected STEP framework maintains stable, high performance at extended temporal window lengths, whereas the performance of baselines operating directly on raw coordinates degrades substantially as $T$ increases.

Panel of Figure 13 — Figure S6 : Architectural comparison. We evaluate the performance differences across varying temporal lengths ( $T$ ) and PCA dimensions ( $K$ ) on UBnormal (left) and ShanghaiTech (right). The $\sigma$ -modulated Residual MLP (diamonds) consistently outperforms the Vanilla MLP (pentagons) across almost all configurations, proving the benefit of explicitly conditioning the network on the noise scale.

Multi-Modal / Pixel-Based Methods
Modality			Method	ShanghaiTech		UBnormal
D	M	S	Method	Full	HR	Full	HR
✓			sRNN [ 20 ]	68.0	-	-	-
✓			Conv-AE [ 10 ]	70.4	69.8	-	-
✓			LSA [ 1 ]	72.5	-	-	-
✓	✓		FFP [ 18 ]	72.8	72.7	-	-

Appendix

Table S1 : Extended state-of-the-art comparison using the AUROC (%) metric. Input modality acronyms denote deep features (D), motion (M), and skeletons (S). STEP strictly outperforms all previous methods, including heavy multi-modal architectures, on both benchmarks. † reproduced by us.

Dataset	Method	AUROC		AP
Dataset	Method	Full	HR	Full	HR
UBnormal	STG-NF [ 11 ]	71.8	71.5	62.7	67.2
	SeeKer [ 4 ]	77.9	78.9	80.3	79.8
	STEP	90.3	90.9	91.6	91.7
ShanghaiTech	STG-NF [ 11 ]	85.9	87.4	77.6	81.4
	SeeKer [ 4 ]	85.5	86.9	80.0	81.5

Appendix

Table S2 : Comparison on AUROC (%) and Average Precision (AP %) on UBnormal and ShanghaiTech. Best results are in bold , and second-best are underlined .

Figure S7 : UBnormal $T/K$ Heatmaps: The best performing PCA/kNN configurations do not strictly align with our EBM’s optimal configuration. However, even the simple PCA/kNN baseline comfortably outperforms the current SOTA.

Figure S8 : ShanghaiTech $T/K$ Heatmaps: Similar to UBnormal, the optimal temporal and dimensional capacities for distance-based scoring (kNN) differ from density-based scoring (STEP).

$T$	$\sigma_{high}=1$	UBnormal ( $K$ )			ShanghaiTech ( $K$ )
$T$	$\sigma_{low}$	32	48	64	32	48	64
12	1e-4	89.13	89.56	89.38	85.52	86.05	85.22
	0.001	89.33	89.24	89.48	85.43	85.97	85.42
	0.01	88.93	89.70	89.48	84.98	85.85	85.68
	0.1	88.91	90.37	90.54	85.64	86.28	85.45
	0.5	86.37	86.91	86.47	83.72	83.88	82.65

Appendix

Table S3 : Impact of minimum noise scale (

\sigma_{low}

) across different temporal windows (

T

) and PCA dimensions (

K

) with a fixed

\sigma_{high}=1

. Results are in AUROC (%). For a given

T/K

setting we mark the best performing in bold , underline the second best and the third best is in italic .

$T$	$\sigma_{low}=0.1$	UBnormal ( $K$ )			ShanghaiTech ( $K$ )
$T$	$\sigma_{high}$	32	48	64	32	48	64
12	0.2	89.49	89.87	90.14	86.03	86.38	85.77
	0.5	89.48	90.57	90.24	85.74	86.49	86.02
	1	88.91	90.37	90.54	85.64	86.28	85.54
	2	89.12	89.89	90.43	85.77	86.10	85.29
16	0.2	88.50	89.00	89.71	85.08	86.05	85.96

Appendix

Table S4 : Impact of maximum noise scale (

\sigma_{high}

) across different temporal windows (

T

) and PCA dimensions (

K

) with a fixed

\sigma_{low}=0.1

. Results are in AUROC (%). For a given

T/K

setting we mark the best performing in bold , underline the second best and the third best is in italic .

$T$	Drop Conf.	UBnormal ( $K$ )				ShanghaiTech ( $K$ )
$T$	Drop Conf.	16	32	48	64	16	32	48	64
8	0	87.25	88.61	89.22	89.96	84.61	86.04	85.92	85.11
	0.1	87.25	88.89	89.14	90.27	84.84	86.16	86.03	85.25
	0.2	87.86	89.17	89.49	89.23	84.98	85.74	85.81	85.07
	0.4	86.37	86.43	86.51	85.43	84.12	84.95	84.43	83.36
12	0	86.43	88.91	90.37	90.54	84.67	85.64	86.28	85.45

Appendix

Table S5 : Impact of dropping noisy poses during training across different temporal windows (

T

) and PCA dimensions (

K

). Poses below the confidence threshold are discarded. Results are in AUROC (%). For a given

T/K

setting we mark the best performing in bold , underline the second best and the third best is in italic .

Aggregation Strategy	PCA Manifold Dimension ( $K$ )
Aggregation Strategy	8	16	32	48	64	128	256
Agg-Sum	79.7	83.3	87.2	87.2	87.1	84.8	81.5
Best Individual $\sigma$	81.9	86.9	89.6	90.3	90.1	89.2	85.3
Agg-Max (Ours)	82.3	86.4	88.9	90.4	90.1	88.8	85.7

Appendix

Table S6 : Impact of the multiscale aggregation strategy on the UBnormal dataset (

T=12

). Agg-Max successfully captures anomalies across all structural frequencies without diluting the energy signal, drastically outperforming summation ( Agg-Sum ) and competing directly with the retrospectively chosen best individual scale.

Aggregation	UBnormal								ShanghaiTech
	BT Raw		BT Smooth		On Raw		On Smooth		BT Raw		BT Smooth		On Raw		On Smooth
	AUC	AP	AUC	AP	AUC	AP	AUC	AP	AUC	AP	AUC	AP	AUC	AP	AUC	AP
Full Test-set
Agg-Max	87.73	91.31	90.99	92.28	85.03	89.84	90.59	91.90	83.93	81.08	85.39	83.20	83.55	81.41	86.32	84.52
Agg-Sum	85.76	89.58	88.17	91.41	83.53	88.45	86.92	90.64	81.95	79.08	83.71	81.36	81.86	79.48	84.88	82.75
Agg-Mean-Top2	87.67	91.27	90.89	92.21	84.98	89.80	90.50	91.84	83.78	80.95	85.26	83.07	83.41	81.27	86.20	84.38

Appendix

Table S7 : Comprehensive evaluation of aggregation strategies across different inference protocols on the Full and Human-Related (HR) splits. We evaluate both offline/non-causal Backtrack (BT) and real-time/causal Online (On) settings, with and without temporal 1D-Gaussian smoothing. Results are in AUROC (%) and AP (%). We mark the best performing strategy in bold , underline the second best, and the third best is in italic .

Test \ Train	ShanghaiTech	UBnormal	Both
ShanghaiTech	86.3	77.2	86.0
UBnormal	82.4	90.6	88.9

Appendix

Table S8 : Cross-Dataset Evaluation: We evaluate the generalization of STEP by testing models on datasets they were not trained on. Finally, we jointly train both datasets in a single model and evaluate on both datasets. Results reported are AUROC (%).

Method	Variant	AUROC (%)
		BT		On
		Raw	Smooth	Raw	Smooth
STG-NF [ 11 ]	MSAD-HR	55.7 †
SeeKer [ 4 ]	MSAD-HR	61.1 †
STEP (Ours)	MSAD (includes all: HR and non-HR)	57.6	59.9	56.4	59.6
	MSAD-HR / ST-Style (drop non-HR clips)	71.7	72.8	69.9	71.6

Appendix

Table S9 : AUROC (%) on MSAD [ 36 ] under evaluation protocol (i): 360 held-out clips (120 normal

+

240 abnormal). ST-Style (as in ShanghaiTech): excludes non-HR clips. UB-Style (as in UBnormal): marks non-HR anomalous frames as normal. BT : Backtrack (non-causal). On : Online (causal, real-time). Raw / Smooth : without/with temporal 1D-Gaussian smoothing. † HR split style unspecified.

$K$	$T$	Persons/Frame	Latency (ms)	FPS	Peak Mem (MB)
48	12	1	0.86	1167	21.5
48	12	10	0.93	1077	22.8
48	12	20	0.90	1109	25.2
48	12	30	0.92	1087	28.6
48	12	40	0.95	1051	30.0
48	12	50	0.97	1026	32.6

Appendix

Table S10 : Inference performance of STEP once poses are extracted. Latency covers the entire pipeline from raw coordinates to final frame-level scores (

L=10

noise scales

\sigma

K

denotes the PCA dimension and

T

represents the temporal segment length.

Panel of Figure 26 — Figure S9 : Latent Traversal in the whitened PC-space. We systematically add structured variance ( $\sigma$ ) along the first two principal components. Top (Static Pose): Adding variance to a stationary person induces a coherent walking motion. Bottom (Moving Pose): Altering the latent coordinates of an already walking person smoothly alters the style and spatial orientation. The ghosting gray skeleton represents the zero-noise center pose for reference.

Figure S10 : PCA as a Low-Pass Filter. Top Row: Adding independent Gaussian noise directly to raw pixel coordinates destroys the structure of a moving person. Bottom Rows: Projecting these corrupted sequences through various PCA bottlenecks filters the noise. An overly tight bottleneck ( $K=8$ ) fails to recover complex motion, while a loose bottleneck ( $K=256$ ) retains too many high-frequency components and remains sensitive to the noise. An appropriate $K=48$ configuration successfully removes unstructured jitter while preserving underlying motion.

Figure S11 : Reconstruction Error under Input Noise ( $T=12$ , UBnormal). As we increase the noise on the input pose coordinates, the poses can still be well reconstructed within the appropriate PCA capacity range of $K\in[32,64]$ . For larger PCA dimensions, the PC-space becomes too permissive and attempts to reconstruct the structural noise, causing the error to rise.

Method	Pose Jitter ( $\sigma_{pixel}$ )
Method	0	0.01	0.05	0.10	0.20	0.30	0.50	1.00
UBnormal
Agg-Max	90.62	90.63	90.44	89.12	82.46	74.80	68.95	64.24
Agg-Sum	87.57	87.58	87.70	87.67	86.41	82.14	72.21	62.89
Best Ind.	90.49	90.49	90.21	89.53	86.98	85.50	79.13	65.82
ShanghaiTech

Appendix

Table S11 : Robustness to raw coordinate jitter during inference on UBnormal and ShanghaiTech (

T=12,K=48

). We inject independent Gaussian noise (

\sigma_{pixel}

) into the raw input coordinates prior to the PCA projection. Up to

\sigma_{pixel}=0.1

, the PCA effectively acts as a low-pass filter, maintaining state-of-the-art performance. Tested on a single model with a fixed random seed.

Projection Method	PCA		AE		VAE
Pretrain using Anomalies	$\times$	✓	$\times$	✓	$\times$	✓
AUROC (%)	90.5	90.5	89.5	90.1	84.9	86.2

Appendix

Table S12 : Comparison of linear and non-linear extractors on the UBnormal benchmark (

T=12,K=48

). Pretrain using Anomalies indicates the projection manifold was constructed using both normal and abnormal training data, rather than strictly normal data. The linear PCA projection outperforms deep Autoencoders, even when the latter are provided with abnormal training sequences.

Explore similar work

Motion Semantics Guided Normalizing Flow for Privacy-Preserving Video Anomaly Detection
Mar 23, 2026Yang Liu, Boan Chen, Yuanyuan Meng +5Video Anomaly Detection Interpretable Anomaly Detection
Tongji Univ. · SJTU · Shanghai Creative Studies Institute +4
VQ-VAD: Vector-quantized Motion Representation Learning for Human-centric Video Anomaly Detection
Aug 5, 2026Narges Rashvand, Ghazal Alinezhad Noghre, Shanle Yao +2Vector Quantization Reconstruction-Based Anomaly Detection
Electrical and Computer Engineering Department University of North Carolina at Charlotte Charlotte, NC, USA
Bounding-Box Trajectories Matter for Video Anomaly Detection
May 21, 2026Inpyo Song, Jangwon LeeVideo Anomaly Detection
Sungkyunkwan University, Seoul, Republic of Korea

Mar 23, 2026J/K move · Enter open · S save

Motion Semantics Guided Normalizing Flow for Privacy-Preserving Video Anomaly Detection

Yang Liu, Boan Chen, Yuanyuan Meng, Jing Liu+4

Tongji Univ. · SJTU · Shanghai Creative Studies Institute+4

Video Anomaly Detection Interpretable Anomaly Detection Normalizing Flows

Open paper AskPDF arXiv

As embodied perception systems increasingly bridge digital and physical realms in interactive multimedia applications, the need for privacy-preserving approaches to understand human activities in physical environments has become paramount. Video anomaly detection is a critical task in such embodied multimedia systems for intelligent surveillance and forensic analysis. Skeleton-based approaches have emerged as a privacy-preserving alternative that processes physical world information through abstract human pose representations while discarding sensitive visual attributes such as identity and facial features. However, existing skeleton-based methods predominantly model continuous motion trajectories in a monolithic manner, failing to capture the hierarchical nature of human activities composed of discrete semantic primitives and fine-grained kinematic details, which leads to reduced discriminability when anomalies manifest at different abstraction levels. In this regard, we propose Motion Semantics Guided Normalizing Flow (MSG-Flow) that decomposes skeleton-based VAD into hierarchical motion semantics modeling. It employs vector quantized variational auto-encoder to discretize continuous motion into interpretable primitives, an autoregressive Transformer to model semantic-level temporal dependencies, and a conditional normalizing flow to capture detail-level pose variations. Extensive experiments on benchmarks (HR-ShanghaiTech & HR-UBnormal) demonstrate that MSG-Flow achieves state-of-the-art performance with 88.1% and 75.8% AUC respectively.

Abstract

Explore similar work

Motion Semantics Guided Normalizing Flow for Privacy-Preserving Video Anomaly Detection

VQ-VAD: Vector-quantized Motion Representation Learning for Human-centric Video Anomaly Detection

Bounding-Box Trajectories Matter for Video Anomaly Detection