Real-Time Video Anomaly Detection Using YOLO Pose Estimation and CLIP-Based Semantic Scoring

Vanodhya G. Warnasooriya; Amir Hajian; Watchara Ruangsang; Supavadee Aramvith

cs.CVAug 31, 2026

Real-Time Video Anomaly Detection Using YOLO Pose Estimation and CLIP-Based Semantic Scoring

Authors: Vanodhya G. Warnasooriya, Amir Hajian, Watchara Ruangsang, Supavadee Aramvith

Organizations: Dept. of Electrical Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok 10330, Thailand · Media Technology Program, King Mongkut’s Univ. of Technology Thonburi, Bangkok 10150, Thailand

Abstract

We propose a lightweight two-stage framework for real-time video anomaly detection. The first stage employs YOLO v11n-pose to detect persons and extract seventeen skeletal keypoints in a single forward pass. The second stage encodes each cropped person region through CLIP ViT-B/32 and computes cosine similarity against predefined textual descriptions of anomalous behaviors. This architecture eliminates the need for optical flow, standalone pose estimators, and density-based scoring modules. Experiments on CUHK Avenue, ShanghaiTech Campus, and a custom indoor dataset collected at Chulalongkorn University demonstrate an end-to-end throughput of approximately 51 FPS on an NVIDIA Titan XP GPU, a 3.36x speedup over the multi-feature baseline, while maintaining frame-level AUROC values of 89.26%, 70.26%, and 84.13%, respectively.

Explore similar work

VigilFormer: Deformable Attention for Video Anomaly Detection with Causal Risk Inference
May 31, 2026Xinze ZhangVideo Anomaly Understanding Anomaly Detection
Bounding-Box Trajectories Matter for Video Anomaly Detection
May 21, 2026Inpyo Song, Jangwon LeeVideo Anomaly Understanding Trajectory Prediction
Context-structured Video Anomaly Detection with Large Vision-Language Models
Jul 21, 2026Dongjun Kim, Changjae Oh, Andrea Cavallaro +1Video Anomaly Understanding Anomalous Frame Detection

May 31, 2026 · cs.CVJ/K move · Enter open · S save

VigilFormer: Deformable Attention for Video Anomaly Detection with Causal Risk Inference

Xinze Zhang

University of Southern California, Los Angeles, CA 90007, USA

Video anomaly detection in surveillance settings must balance detection accuracy against real-time throughput, a tension that existing methods address either through stronger feature extractors or more efficient architectures, but rarely both. We present VigilFormer, a unified framework that combines deformable spatio-temporal attention with causal temporal modeling to detect anomalies in untrimmed surveillance video. The proposed Deformable Spatio-Temporal Encoder (DSTE) attends to a sparse set of informative locations across frames, avoiding the quadratic cost of dense attention while retaining the ability to capture irregular motion patterns. A Causal Anomaly Classifier (CAC) applies dilated causal convolutions over snippet-level features and optimizes a contrastive multiple-instance learning objective that separates anomalous and normal representations without frame-level labels. To meet deployment constraints, an Adaptive Confidence Scheduler (ACS) dynamically skips low-information frames at inference time, reducing redundant computation in static scenes. Evaluated on UCF-Crime, ShanghaiTech, and CUHK Avenue, VigilFormer achieves AUC scores of 87.83%, 97.21%, and 89.74% respectively, at 41.5 FPS on a single GPU, outperforming recent weakly-supervised methods in both accuracy and speed.

Video Anomaly Understanding Anomaly Detection Spatial Attention Convolutional Encoders

Open paper Ask about it PDF

Abstract

Explore similar work

VigilFormer: Deformable Attention for Video Anomaly Detection with Causal Risk Inference

Bounding-Box Trajectories Matter for Video Anomaly Detection

Context-structured Video Anomaly Detection with Large Vision-Language Models