Beyond Isolated Entities: Relation-Aware Multi-Entity Modeling for Unsupervised Video Anomaly Detection

Zhongpeng Pan; Xina Cheng; Kailun Yang

eess.IVSep 23, 2026

Beyond Isolated Entities: Relation-Aware Multi-Entity Modeling for Unsupervised Video Anomaly Detection

Authors: Zhongpeng Pan, Xina Cheng, Kailun Yang

Organizations: School of Artificial Intelligence, Xidian University, China · School of Artificial Intelligence and Robotics, Hunan University, China · National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China

Abstract

Video anomaly detection for patrol robots and surveillance systems must recognize abnormal interactions among familiar entities. Existing pixel-reconstruction and isolated-entity methods may fail when individual entities appear normal but their spatial or motion relations are abnormal. This work presents Interaction-Centric Network for Temporal Entity-Relation Analysis and Consistency Testing (INTERACT), a framework for unsupervised video anomaly detection. Person-Object Appearance-Motion Interaction (POAMI) moves beyond isolated-entity encoding by jointly modeling the appearance, motion, and spatial configurations of persons and objects in each frame. The resulting representations capture both individual entity states and their cross-entity context. Based on these representations, Target Geometry-Guided Relational Interaction Prediction (TGRIP) predicts target entity interaction states from historical relational memory and target-frame geometry without explicit identity tracking. Motion-Interaction Reconstruction and Alignment (MIRA) then evaluates these predictions through conditional flow reconstruction and semantic consistency checking, providing complementary anomaly evidence beyond prediction error alone. INTERACT achieves state-of-the-art performance, obtaining a frame-level AUC of 84.5% on the ShanghaiTech benchmark. Ablation studies show that removing cross-entity attention causes the largest performance drop, demonstrating the necessity of relational modeling. INTERACT is particularly effective for anomalies caused by changes in relations among people and objects, while maintaining competitive performance in general scenarios. The source code will be made publicly available at https://github.com/ppworkhard/INTERACT.

Figures & tables

Fig. 1: Conceptual illustration of relational anomaly detection. A person first walks with a suitcase, stops beside it, and then leaves it behind, forming an abnormal person–object relation although both entities remain individually familiar. Object-centric modeling (bottom left) processes entities and their appearance and motion cues independently, missing this relational change. INTERACT with POAMI (bottom right) jointly models the appearance, motion, and position of multiple entities; the curves show their evolving distances in the POAMI feature space, enabling detection of the abnormal interaction.

Fig. 2: Overview of INTERACT. POAMI encodes multi-entity appearance, motion, and spatial relations. The Relation Prediction Head forecasts the global relation token, while TGRIP predicts entity tokens from relational history and target geometry. MIRA verifies the entity predictions through motion reconstruction and semantic alignment.

Fig. 3: Cross-entity attention and occlusion sensitivity in scenes with more people (left) and fewer people (right). Red/yellow boxes denote persons/objects, and arrows show stronger attention connections retained for readability. Heatmap colors indicate increases (red), decreases (blue), or minimal changes (near-white) in the Stage 1 anomaly score after local occlusion.

Method	Venue	Avenue	ShanghaiTech	NWPU Campus
FFP [ 20 ]	CVPR18	84.9	72.8	–
MemAE [ 18 ]	ICCV19	84.9	72.8	61.9
CAE [ 26 ]	CVPR19	87.4	78.7	–
MNAD [ 19 ]	CVPR20	88.5	70.5	62.5
AMMC-Net [ 33 ]	AAAI21	86.6	73.7	64.5
HF 2 -VAD [ 21 ]	ICCV21	91.1	76.2	63.7

TABLE I: Frame-level ROC-AUC (%) comparison with existing methods on video anomaly detection benchmarks. INTERACT achieves state-of-the-art performance on ShanghaiTech [ 14 ] (

84.5\%

) while maintaining competitive results on Avenue [ 31 ] and NWPU Campus_sub [ 32 ] .

ID	Variant	Cross-entity Attn.	Prediction Score	Verification	Token Consistency	AUC (%)	$\Delta$
A0	INTERACT (Full)	✓	✓	✓	✓	84.50	0.00
A1	Entity-Independent Variant	$\times$	✓	✓	✓	79.18	-5.32
A2	Prediction Only	✓	✓	$\times$	–	80.93	-3.57
A3	Verification Only	✓	$\times$	✓	✓	82.93	-1.57
A4	Without Token Consistency	✓	✓	✓	$\times$	83.43	-1.07

TABLE II: Ablation study on ShanghaiTech. Delta is computed relative to the full INTERACT model (A0).

Fig. 4: Preliminary evaluation on a real-world Unitree Go2 quadruped robot platform. Left: the robot during campus patrol. Right: two first-person anomaly sequences—(a) Fall and (b) Jump—showing sample frames (green borders: normal, red borders: anomalous) and anomaly score curves from Prediction Only, Verification Only, and the proposed full INTERACT solution. Pink shaded regions indicate ground-truth anomalous intervals.

Entities	Params (M)	GFLOPs/clip	FPS
1	57.18	41.84	8.61
3	57.18	61.84	5.75
5	57.18	81.73	5.30

TABLE III: Computational efficiency of INTERACT.

Explore similar work

VigilFormer: Deformable Attention for Video Anomaly Detection with Causal Risk Inference
May 31, 2026Xinze ZhangTransformer Attention Deformable Attention
University of Southern California, Los Angeles, CA 90007, USA
Bounding-Box Trajectories Matter for Video Anomaly Detection
May 21, 2026Inpyo Song, Jangwon LeeVideo Anomaly Detection
Sungkyunkwan University, Seoul, Republic of Korea
Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding
Oct 1, 2026Mohd Ubaid Wani, Sara Atito, Josef Kittler +1CoT Reasoning Interpretable Anomaly Detection
Centre for Vision, Speech and Signal Processing (CVSSP) University of Surrey, UK · Surrey Institute for People-Centred AI (PAI) University of Surrey, UK

May 31, 2026J/K move · Enter open · S save

VigilFormer: Deformable Attention for Video Anomaly Detection with Causal Risk Inference

Xinze Zhang

University of Southern California, Los Angeles, CA 90007, USA

Transformer Attention Deformable Attention Video Anomaly Detection Weakly Supervised Video Anomaly Detection Video Surveillance

Open paper AskPDF arXiv

Video anomaly detection in surveillance settings must balance detection accuracy against real-time throughput, a tension that existing methods address either through stronger feature extractors or more efficient architectures, but rarely both. We present VigilFormer, a unified framework that combines deformable spatio-temporal attention with causal temporal modeling to detect anomalies in untrimmed surveillance video. The proposed Deformable Spatio-Temporal Encoder (DSTE) attends to a sparse set of informative locations across frames, avoiding the quadratic cost of dense attention while retaining the ability to capture irregular motion patterns. A Causal Anomaly Classifier (CAC) applies dilated causal convolutions over snippet-level features and optimizes a contrastive multiple-instance learning objective that separates anomalous and normal representations without frame-level labels. To meet deployment constraints, an Adaptive Confidence Scheduler (ACS) dynamically skips low-information frames at inference time, reducing redundant computation in static scenes. Evaluated on UCF-Crime, ShanghaiTech, and CUHK Avenue, VigilFormer achieves AUC scores of 87.83%, 97.21%, and 89.74% respectively, at 41.5 FPS on a single GPU, outperforming recent weakly-supervised methods in both accuracy and speed.

Abstract

Explore similar work

VigilFormer: Deformable Attention for Video Anomaly Detection with Causal Risk Inference

Bounding-Box Trajectories Matter for Video Anomaly Detection

Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding