Interpretable Human Activity Recognition for Subtle Robbery Detection in Surveillance Videos
Authors: Bryan Jhoan Cazáres Leyva, Ulises Gachuz Davila, José Juan González Fonseca, Juan Irving Vasquez, Vanessa A. Camacho-Vázquez, Sergio Isahí Garrido-Castañeda
Organizations: Escuela Superior de Cómputo, Instituto Politécnico Nacional, México City, México. · Centro de Innovación y Desarollo Tecnológico en Cómputo, Instituto Politécnico Nacional, México City, México.
Non-violent street robberies (snatch-and-run) are difficult to detect automatically because they are brief, subtle, and often indistinguishable from benign human interactions in unconstrained surveillance footage. This paper presents a hybrid, pose-driven approach for detecting snatch-and-run events that combines real-time perception with an interpretable classification stage suitable for edge deployment. The system uses a YOLO-based pose estimator to extract body keypoints for each tracked person and computes kinematic and interaction features describing hand speed, arm extension, proximity, and relative motion between an aggressor-victim pair. A Random Forest classifier is trained on these descriptors, and a temporal hysteresis filter is applied to stabilize frame-level predictions and reduce spurious alarms. We evaluate the method on a staged dataset and on a disjoint test set collected from internet videos, demonstrating promising generalization across different scenes and camera viewpoints. Finally, we implement the complete pipeline on an NVIDIA Jetson Nano and report real-time performance, supporting the feasibility of proactive, on-device robbery detection.
Figures & tables
Figure 1: General flow of the proposed method for the recognition of robbery in surveillance videos.
Rank
Feature
Imp. ( 10−2 )
Rank
Feature
Imp. ( 10−2 )
1
dist_p95
5.236
6
distancet_max
4.188
2
handToHip_max
4.802
7
handToTorso_median
3.340
3
handToTorso_mean
4.781
8
handToHip_p95
2.763
4
handToTorso_p95
4.297
9
distance_mean
2.459
5
handToTorso_max
4.239
10
closeHandPct
2.352
Table 1: Top 10 features ranked by Random Forest importance (Two columns).
Figure 2: Class visualization using PCA. As we can observe, the projected samples are in a general way distinguishable. This graph is only a projection of the feature vectors of 10 elements.
Figure 3: Representative frames from the training/validation video dataset recorded for this work. Faces were deliberately blurred for this paper.
Figure 4: Representative frames from the test video dataset. These videos are publicly available on YouTube. Faces were deliberately blurred for this paper.
Figure 5: Example sequence of the functioning of the implemented method. The sequence shows the detection of a robbery.
Class
Acc.
Prec.
Recall
F1-Score
Support
Non-Robbery
0.83
0.91
0.83
0.87
12
Robbery
0.83
0.71
0.83
0.77
6
Table 2: Classification results for validation stage.
Class
Acc.
Precision
Recall
F1-Score
Support
Non-Robbery
73.3
0.78
0.83
0.81
30
Robbery
58.8
0.67
0.59
0.62
17
Table 3: Classification results for the test experiment.
Physical violence in public spaces is a significant public health concern, with minor incidents such as pushing often serving as precursors to more severe escalations. This research develops an automated system for the real-time detection of moderate physical violence, specifically pushing, in surveillance camera footage. The proposed solution integrates state-of-the-art computer vision models, utilizing YOLO11 and YOLO11-Pose for human detection and skeletal keypoint extraction. By calculating body inclination and joint angles between shoulders and hips, a Random Forest classifier was trained to distinguish between normal behavior and aggressive physical contact. The system's performance was evaluated through three progressive case studies representing increasing levels of difficulty. In controlled environments with frontal views, the model achieved a precision of 0.98. In the most challenging scenario, featuring high-altitude, steep-angle recordings from real-world surveillance infrastructure, the system maintained a precision of 0.72 despite significant perspective distortion and visual noise. These results demonstrate the feasibility of using skeletal analysis for early violence intervention in urban security contexts.
Luis Angel Aparicio Borjas, Victor Elias Nieto, Juan Irving Vasquez +2
ESCOM-IPN, Mexico City, México · Centro de Innovación y Desarrollo Tecnológico en Cómputo, Instituto Politécnico Nacional, Mexico City, México
We propose a lightweight two-stage framework for real-time video anomaly detection. The first stage employs YOLO v11n-pose to detect persons and extract seventeen skeletal keypoints in a single forward pass. The second stage encodes each cropped person region through CLIP ViT-B/32 and computes cosine similarity against predefined textual descriptions of anomalous behaviors. This architecture eliminates the need for optical flow, standalone pose estimators, and density-based scoring modules. Experiments on CUHK Avenue, ShanghaiTech Campus, and a custom indoor dataset collected at Chulalongkorn University demonstrate an end-to-end throughput of approximately 51 FPS on an NVIDIA Titan XP GPU, a 3.36x speedup over the multi-feature baseline, while maintaining frame-level AUROC values of 89.26%, 70.26%, and 84.13%, respectively.
Vanodhya G. Warnasooriya, Amir Hajian, Watchara Ruangsang +1
Dept. of Electrical Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok 10330, Thailand · Media Technology Program, King Mongkut’s Univ. of Technology Thonburi, Bangkok 10150, Thailand
Articulated human pose provides detailed body-configuration information beyond coarse spatial relationships, but whether this detail yields greater discriminative information when the downstream pipeline is held fixed remains unclear. We examine this through early violence detection. Holding the tracker, temporal head, supervision, folds, and evaluation fixed, we compare five interaction representations spanning coarse bounding-box geometry, a matched handcrafted pose analogue, enriched pose descriptors, and a matched-capacity encoder learned from raw joints, under video-level evaluation with cluster-bootstrap intervals. No pose-based representation outperforms coarse geometry, though with fifteen anomalous videos this subset cannot rule out small effects. Extending the pipeline to frozen visual encoders, and repeating the comparison on XD-Violence (137 anomalous videos, nine times our UCF-Crime sample), person-crop appearance and whole-frame context both exceed geometry by a wide margin, yet context matches appearance on UCF-Crime and exceeds it on the larger split: cropping to the interacting people yields no advantage over encoding the whole frame. This prompts a direct test of what the benchmark measures. Scoring anomalous videos using only frames preceding the annotated onset, under a control removing sequence length as a cue, retains 39-91% of above-chance separation on both benchmarks, including for seven hand-designed geometric channels. Inspection of the tightest pre-onset windows identifies concrete provenance artifacts: editorial title cards and platform watermarks absent from the surveillance footage supplying the normal class. Video-level AUC here is thus a composite of event evidence and pre-event source cues, a shared source of discrimination that can obscure differences between representations. The diagnostic requires only annotations these benchmarks already ship.