Beyond Isolated Entities: Relation-Aware Multi-Entity Modeling for Unsupervised Video Anomaly Detection
Authors: Zhongpeng Pan, Xina Cheng, Kailun Yang
Organizations: School of Artificial Intelligence, Xidian University, China · School of Artificial Intelligence and Robotics, Hunan University, China · National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China
Video anomaly detection for patrol robots and surveillance systems must recognize abnormal interactions among familiar entities. Existing pixel-reconstruction and isolated-entity methods may fail when individual entities appear normal but their spatial or motion relations are abnormal. This work presents Interaction-Centric Network for Temporal Entity-Relation Analysis and Consistency Testing (INTERACT), a framework for unsupervised video anomaly detection. Person-Object Appearance-Motion Interaction (POAMI) moves beyond isolated-entity encoding by jointly modeling the appearance, motion, and spatial configurations of persons and objects in each frame. The resulting representations capture both individual entity states and their cross-entity context. Based on these representations, Target Geometry-Guided Relational Interaction Prediction (TGRIP) predicts target entity interaction states from historical relational memory and target-frame geometry without explicit identity tracking. Motion-Interaction Reconstruction and Alignment (MIRA) then evaluates these predictions through conditional flow reconstruction and semantic consistency checking, providing complementary anomaly evidence beyond prediction error alone. INTERACT achieves state-of-the-art performance, obtaining a frame-level AUC of 84.5% on the ShanghaiTech benchmark. Ablation studies show that removing cross-entity attention causes the largest performance drop, demonstrating the necessity of relational modeling. INTERACT is particularly effective for anomalies caused by changes in relations among people and objects, while maintaining competitive performance in general scenarios. The source code will be made publicly available at https://github.com/ppworkhard/INTERACT.
Figures & tables
Fig. 1: Conceptual illustration of relational anomaly detection. A person first walks with a suitcase, stops beside it, and then leaves it behind, forming an abnormal person–object relation although both entities remain individually familiar. Object-centric modeling (bottom left) processes entities and their appearance and motion cues independently, missing this relational change. INTERACT with POAMI (bottom right) jointly models the appearance, motion, and position of multiple entities; the curves show their evolving distances in the POAMI feature space, enabling detection of the abnormal interaction.
Fig. 2: Overview of INTERACT. POAMI encodes multi-entity appearance, motion, and spatial relations. The Relation Prediction Head forecasts the global relation token, while TGRIP predicts entity tokens from relational history and target geometry. MIRA verifies the entity predictions through motion reconstruction and semantic alignment.
Fig. 3: Cross-entity attention and occlusion sensitivity in scenes with more people (left) and fewer people (right). Red/yellow boxes denote persons/objects, and arrows show stronger attention connections retained for readability. Heatmap colors indicate increases (red), decreases (blue), or minimal changes (near-white) in the Stage 1 anomaly score after local occlusion.
Method
Venue
Avenue
ShanghaiTech
NWPU Campus
FFP [ 20 ]
CVPR18
84.9
72.8
–
MemAE [ 18 ]
ICCV19
84.9
72.8
61.9
CAE [ 26 ]
CVPR19
87.4
78.7
–
MNAD [ 19 ]
CVPR20
88.5
70.5
62.5
AMMC-Net [ 33 ]
AAAI21
86.6
73.7
64.5
HF 2 -VAD [ 21 ]
ICCV21
91.1
76.2
63.7
TABLE I: Frame-level ROC-AUC (%) comparison with existing methods on video anomaly detection benchmarks. INTERACT achieves state-of-the-art performance on ShanghaiTech [ 14 ] ( 84.5% ) while maintaining competitive results on Avenue [ 31 ] and NWPU Campus_sub [ 32 ] .
ID
Variant
Cross-entity Attn.
Prediction Score
Verification
Token Consistency
AUC (%)
Δ
A0
INTERACT (Full)
✓
✓
✓
✓
84.50
0.00
A1
Entity-Independent Variant
×
✓
✓
✓
79.18
-5.32
A2
Prediction Only
✓
✓
×
–
80.93
-3.57
A3
Verification Only
✓
×
✓
✓
82.93
-1.57
A4
Without Token Consistency
✓
✓
✓
×
83.43
-1.07
TABLE II: Ablation study on ShanghaiTech. Delta is computed relative to the full INTERACT model (A0).
Fig. 4: Preliminary evaluation on a real-world Unitree Go2 quadruped robot platform. Left: the robot during campus patrol. Right: two first-person anomaly sequences—(a) Fall and (b) Jump—showing sample frames (green borders: normal, red borders: anomalous) and anomaly score curves from Prediction Only, Verification Only, and the proposed full INTERACT solution. Pink shaded regions indicate ground-truth anomalous intervals.
Video anomaly detection in surveillance settings must balance detection accuracy against real-time throughput, a tension that existing methods address either through stronger feature extractors or more efficient architectures, but rarely both. We present VigilFormer, a unified framework that combines deformable spatio-temporal attention with causal temporal modeling to detect anomalies in untrimmed surveillance video. The proposed Deformable Spatio-Temporal Encoder (DSTE) attends to a sparse set of informative locations across frames, avoiding the quadratic cost of dense attention while retaining the ability to capture irregular motion patterns. A Causal Anomaly Classifier (CAC) applies dilated causal convolutions over snippet-level features and optimizes a contrastive multiple-instance learning objective that separates anomalous and normal representations without frame-level labels. To meet deployment constraints, an Adaptive Confidence Scheduler (ACS) dynamically skips low-information frames at inference time, reducing redundant computation in static scenes. Evaluated on UCF-Crime, ShanghaiTech, and CUHK Avenue, VigilFormer achieves AUC scores of 87.83%, 97.21%, and 89.74% respectively, at 41.5 FPS on a single GPU, outperforming recent weakly-supervised methods in both accuracy and speed.
Xinze Zhang
University of Southern California, Los Angeles, CA 90007, USA
Video anomaly detection is critical for public safety and security, yet remains highly challenging despite extensive research due to large variations in appearance, viewpoint, and scene dynamics. Among existing approaches, human pose-based methods have emerged as a major line of research, showing strong performance since many anomalies in public datasets involve humans and pose representations are robust to appearance changes while providing compact motion descriptions. However, these methods often overlook bounding-box trajectories, although such information is inherently available in pose-based pipelines. In this paper, we explicitly leverage these trajectories as a primary anomaly cue. We present TrajVAD, a framework that models multi-class bounding-box trajectories using normalizing flows to learn normal kinematic patterns. Its trajectory-only variant, TrajVAD-T, eliminates pose estimation, reaches 87.7 AP on ShanghaiTech, and achieves the best results on MSAD among compared methods. TrajVAD-P adds a reliability-gated pose branch and improves performance to 88.6 AUROC and 90.9 AP on ShanghaiTech, establishing bounding-box trajectories as an effective modality for video anomaly detection.
Video Anomaly Detection (VAD) aims to temporally localize abnormal events in videos. Most existing approaches rely on dataset-specific training and curated annotations, limiting generalization in open-set scenarios. Recent zero-shot methods based on Large Vision- Language Models (LVLMs) alleviate this dependency but often lack temporal continuity and structured reasoning. We propose Cog-VADU, a fully training-free framework that reformulates VAD as a sequential cognitive reasoning task. Cog-VADU introduces Chain-of- Anomaly Detection Thought Prompting (CoADTP), which unrolls an LVLM into a recurrent reasoning chain across video segments. By propagating structured rationales over time, the model maintains implicit temporal memory, enabling robust discrimination between com- plex anomalies and high-motion normal activities. To improve reliability, we further design a cross-modal re-ranking stage that aligns textual rationales with visual embeddings, enforcing semantic consistency and temporal coherence for refined and stable predictions. Extensive experiments on multiple public VAD benchmarks demonstrate that Cog-VADU achieves competitive zero-shot performance. Moreover, cross-model evaluations show that CoADTP consistently enhances reasoning-based anomaly detection in a model-agnostic manner, pro- viding interpretable and generalizable anomaly understanding for real-world applications.
Mohd Ubaid Wani, Sara Atito, Josef Kittler +1
Centre for Vision, Speech and Signal Processing (CVSSP) University of Surrey, UK · Surrey Institute for People-Centred AI (PAI) University of Surrey, UK