Strictly Causal Streaming Video Anomaly Detection with a Theoretically-Grounded State-Space Core

Yogesh Kumar

cs.AIAug 25, 2026

Strictly Causal Streaming Video Anomaly Detection with a Theoretically-Grounded State-Space Core

Authors: Yogesh Kumar

Organizations: Indian Institute of Technology Jodhpur, Jodhpur, India

Abstract

Recent work has applied Mamba style state space models (SSMs) to video anomaly detection, yet existing approaches still rely on buffering clips or windows internally, lack a theoretical account of how temporal memory relates to detection latency, and benchmark efficiency only through GPU throughput rather than the edge hardware these methods are intended to target. We introduce a strictly causal streaming anomaly detector whose fixed size state is updated in O(1) time and memory per incoming frame, with no lookahead and no clip buffering. Its temporal core is a diagonal linear state space recurrence with an input and state dependent decay gate, trained self supervised through causal next embedding prediction on a frozen visual backbone. We derive a closed form relationship between the recurrence decay spectrum and both detection delay and the shortest anomaly it can reliably capture, then validate empirically on UCSD Ped2 and CUHK Avenue. The settling delay bound predicted from the learned base decay (57 to 59 frames) sits far above the measured detection delay (1.6 and 18.4 frames), showing that the event boundary gate, not the base decay, governs responsiveness. We further report end to end latency and throughput measured directly on Apple M3 Pro hardware, 0.74 ms and 0.77 ms per frame (over 1300 FPS), rather than simulated GPU numbers. With an untuned initial configuration the method reaches 67.9 percent and 70.2 percent frame level AUC on Ped2 and Avenue, trailing prior non causal SSM baselines in accuracy. Ablations over decay rate, state size, and gating reveal that the gate contribution is dataset size dependent, hurting accuracy on the smaller Ped2 training set but helping on the larger Avenue one. Closing this accuracy gap and extending evaluation to a third, larger benchmark are immediate next steps.

Figures & tables

Dataset	Method	Frame-AUC $\uparrow$	EER $\downarrow$	Latency (ms) $\downarrow$	FPS $\uparrow$	Hardware
UCSD Ped2	Ours (streaming SSM)	67.9	36.8	0.74	1343.62	mps
CUHK Avenue	Ours (streaming SSM)	70.2	34.9	0.77	1290.74	mps

Table 1: Main results: standard accuracy metrics alongside on-device streaming latency/throughput.

Dataset	Layer	Mean decay $a$	Theory delay (frames)	Empirical delay (frames)
UCSD Ped2	0	0.9493	57.5	1.6 ( $n{=}$ 12)
	1	0.9501	58.6
CUHK Avenue	0	0.9493	57.5	18.4 ( $n{=}$ 21)
	1	0.9501	58.6

Table 2: Predicted settling delay (Eq. 9 ), from each dataset’s trained mean decay, vs. empirically measured detection delay.

Variant	Frame-AUC $\uparrow$	EER $\downarrow$	Latency (ms) $\downarrow$	FPS $\uparrow$
Baseline (gate on, N=128, decay in [.9,.999])	67.9	36.8	0.74	1344
Gate off	76.6	24.6	0.49	2047
Decay fast, decay in [.5,.9]	67.0	38.2	0.75	1330
Decay slow, decay in [.98,.9999]	67.8	37.0	0.76	1311
State size N=32	74.7	33.1	0.77	1302
State size N=256	66.5	38.7	0.77	1298

Table 3: Ablations on UCSD Ped2: architectural variants, all else held fixed. Latency/FPS measured the same way as Table 1 .

Variant	Frame-AUC $\uparrow$	EER $\downarrow$	Latency (ms) $\downarrow$	FPS $\uparrow$
Baseline (gate on, N=128, decay in [.9,.999])	70.2	34.9	0.77	1291
Gate off	60.0	43.8	0.51	1953
Decay fast, decay in [.5,.9]	70.6	34.4	0.75	1327
Decay slow, decay in [.98,.9999]	70.3	34.8	0.77	1303
State size N=32	70.3	35.0	0.76	1319
State size N=256	71.1	34.5	0.77	1302

Table 4: Ablations on CUHK Avenue: architectural variants, all else held fixed. Latency/FPS measured the same way as Table 1 .

Explore similar work

Sep 7, 2026

ReactVAU: A Slow-Fast Decoupled Framework for Streaming Video Anomaly Understanding

In this paper, we propose ReactVAU, a Slow-Fast Decoupled Framework for real-time streaming Video Anomaly Understanding (VAU). Existing VAU methods rely on offline inference with global temporal sampling, which violates causality and prevents deployment in live surveillance streams. Conversely, general streaming video models satisfy causal access but dilute rare transient anomalies during memory compression and often invoke heavyweight MLLMs uniformly over long normal intervals. React VAU addresses this gap with three synergistic components: a lightweight Fast Detection Module based on Spatial Grid Folding (SGF) for continuous anomaly filtering; an Anomaly-Aware Persistent Memory (AAPM) that protects critical visual cues from temporal decay; and a heavyweight Slow Reasoning Module that remains dormant during normal streams and is awakened only by suspicious events for semantic verification and causal description. Extensive experiments on multiple benchmarks demonstrate that ReactVAU operates under strict streaming constraints while simultaneously achieving competitive performance in both anomaly detection and causal reasoning, alongside significantly enhanced computational efficiency by minimizing heavyweight MLLM invocations. Project page is available at https://huiyuiui.github.io/ReactVAU/

Chia-Hui Chen, Shih-Ying Yeh, Fu-En Yang +2

National Tsing Hua University, Hsinchu, Taiwan · NVIDIA, Taipei, Taiwan

Streaming Video Understanding Video Anomaly Detection Causal Reasoning in Videos

Ask PDF

May 31, 2026

VigilFormer: Deformable Attention for Video Anomaly Detection with Causal Risk Inference

Video anomaly detection in surveillance settings must balance detection accuracy against real-time throughput, a tension that existing methods address either through stronger feature extractors or more efficient architectures, but rarely both. We present VigilFormer, a unified framework that combines deformable spatio-temporal attention with causal temporal modeling to detect anomalies in untrimmed surveillance video. The proposed Deformable Spatio-Temporal Encoder (DSTE) attends to a sparse set of informative locations across frames, avoiding the quadratic cost of dense attention while retaining the ability to capture irregular motion patterns. A Causal Anomaly Classifier (CAC) applies dilated causal convolutions over snippet-level features and optimizes a contrastive multiple-instance learning objective that separates anomalous and normal representations without frame-level labels. To meet deployment constraints, an Adaptive Confidence Scheduler (ACS) dynamically skips low-information frames at inference time, reducing redundant computation in static scenes. Evaluated on UCF-Crime, ShanghaiTech, and CUHK Avenue, VigilFormer achieves AUC scores of 87.83%, 97.21%, and 89.74% respectively, at 41.5 FPS on a single GPU, outperforming recent weakly-supervised methods in both accuracy and speed.

Xinze Zhang

University of Southern California, Los Angeles, CA 90007, USA

Transformer Attention Deformable Attention Video Anomaly Detection Weakly Supervised Video Anomaly Detection Video Surveillance

Ask PDF

Apr 9, 2026

ESOM: Efficiently Understanding Streaming Video Anomalies with Open-world Dynamic Definitions

Open-world video anomaly detection (OWVAD) aims to detect and explain abnormal events under different anomaly definitions, which is important for applications such as intelligent surveillance and live-streaming content moderation. Recent MLLM-based methods have shown promising open-world generalization, but still suffer from three major limitations: inefficiency for practical deployment, lack of streaming processing adaptation, and limited support for dynamic anomaly definitions in both modeling and evaluation. To address these issues, this paper proposes ESOM, an efficient streaming OWVAD model that operates in a training-free manner. ESOM includes a Definition Normalization module to structure user prompts for reducing hallucination, an Inter-frame-matched Intra-frame Token Merging module to compress redundant visual tokens, a Hybrid Streaming Memory module for efficient causal inference, and a Probabilistic Scoring module that converts interval-level textual outputs into frame-level anomaly scores. In addition, this paper introduces OpenDef-Bench, a new benchmark with clean surveillance videos and diverse natural anomaly definitions for evaluating performance under varying conditions. Extensive experiments show that ESOM achieves real-time efficiency on a single GPU and state-of-the-art performance in anomaly temporal localization, classification, and description generation. The code and benchmark will be released at https://github.com/Kamino666/ESOM_OpenDef-Bench.

Zihao Liu, Xiaoyu Wu, Wenna Li +2

State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, China

Video Anomaly Detection Anomaly Localization Streaming Video Understanding

Ask PDF