MoG-VLM: Efficient Video Anomaly Detection through Motion-Guided Vision-Language Models

Khaled Waleed Dawoud; Muhammad Zaigham Zaheer; Khan Muhammad; Muhammad Haris Khan

cs.CVOct 22, 2026

MoG-VLM: Efficient Video Anomaly Detection through Motion-Guided Vision-Language Models

Authors: Khaled Waleed Dawoud, Muhammad Zaigham Zaheer, Khan Muhammad, Muhammad Haris Khan

Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · Sungkyunkwan University, Seoul 03063, South Korea

Abstract

Vision-Language Models (VLMs) have shown strong potential for video anomaly detection, but dense frame-level inference makes them impractical for real-time deployment. Existing acceleration commonly relies on uniform sampling, which ignores content dynamics and can either waste computation on static segments or miss localized anomalies. We propose MoG-VLM a motion-adaptive anomaly detection framework that leverages motion purely as a lightweight control signal to decide when VLM inference is necessary. To this end, we introduce Temporal Gradient Magnitude (TGM) which performs content-aware sampling to concentrate computation on informative regions while avoiding redundant processing. We further introduce a hierarchical dual-stream design in which dense multi-frame VLM reasoning is activated only around frames with elevated anomaly likelihood, via anchor-triggered context-window expansion with early termination. A fusion module then reconciles these instantaneous semantic cues with temporal action dynamics. Across four benchmarks, our method achieves 91.91%, 84.30%, 87.88%, and 78.96% AUC-ROC on XD-Violence, UCF-Crime, MSAD, and ShanghaiTech, respectively, establishing new state of the art on three benchmarks under the zero-shot, training-free setting. On XD-Violence, we improve over the previous best result by +0.57% while delivering a 1.31x inference speedup. These results demonstrate that content-aware allocation of VLM capacity, processing fewer frames more intelligently rather than all frames uniformly, is a more effective axis of progress than uniformly densifying VLM inference across frames. Code is available on GitHub: https://github.com/kdawoud91/MoG-VLM.

Figures & tables

Figure 1: Comparison of frame sampling strategies for video anomaly detection. (a) Dense sampling, incurring prohibitively high computational cost [ 20 ] . (b) Uniform sampling, which can waste computation on static segments [ 36 , 12 ] . (c) Our motion-adaptive sampling method based on Temporal Gradient Magnitude (TGM) dynamically selects frames according to motion patterns, reducing processed frames. (d) Our introduced anchor-triggered processing selectively expands time windows around high-confidence detections.

Figure 2: Overview of the MoG-VLM framework. Motion-adaptive sampling via TGM categorizes frames into three tiers: baseline uniform (orange), TGM-triggered densification (green), and anchor-triggered (magenta). Baseline stride frames and TGM-triggered frames with high anomaly scores become anchors, initiating hierarchical lookback windows ( $W_{1}$ , $W_{2}$ , $W_{3}$ ). Stream B conducts temporal-level Video-VLM multi-frame reasoning and LLM scoring. Tier-aware adaptive fusion combines both streams using motion-tier-inherited weights ( $\lambda_{\text{base}}$ , $\lambda_{\text{motion}}$ , $\lambda_{\text{anchor}}$ ) without training.

Method	Zero-shot	Training-free	XD-Violence	UCF-Crime	MSAD	ShanghaiTech
Method	Zero-shot	Training-free	AUC(%)	AUC(%)	AUC(%)	AUC(%)
FusedVision [ 6 ]	✗	✗	–	–	–	83.58
RTFM [ 23 ]	✗	✗	61.56	–	85.67	–
Jigsaw [ 24 ]	✗	✗	–	–	–	84.30
MGFN [ 5 ]	✗	✗	–	86.98	85.0	–
STPrompt [ 30 ]	✗	✗	–	88.08	–	–

Table 1: VAD Performance comparison across XD-Violence, UCF-Crime, MSAD and ShanghaiTech. ✓ / ✗ indicate whether a method is zero-shot and training-free. Our MoG-VLM results are mean

\pm

std over 3 runs.

Table 4

Table 5

Table 6: Ablation of multiscale TGM window sizes

\mathcal{W}

. AUC-ROC (%) is reported at the frame-level (motion-adaptive sampling only). The default configuration is shaded.

\mathcal{W}

XD UCF MSAD Shang

\{1,2,4\}

84.69 68.31 81.51 72.38

\{1,3,5\}

84.53 66.76 79.07 71.38

\{2,4,8\}

85.36 68.61 79.38 70.67

Table 7: Effect of

\theta_{\text{trigger}}

on AUC-ROC (%) across datasets. Default conf. is shaded.

\theta_{\text{trigger}}

XD UCF MSAD Shang 0.4 87.56 70.21 82.92 73.44 0.5 87.29 70.07 82.31 73.10 0.6 85.84 68.46 82.22 73.09 0.7 85.29 67.95 82.14 73.16 0.8 84.80 67.61 82.07 73.16

Table 7

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Table 8

$\lambda_{\text{base}}$	MSAD	UCF	XD	Shang.
0.2	87.57	84.47	91.79	77.99
0.3 $\star$	87.93	84.48	91.97	79.02
0.4	88.20	84.39	92.10	79.79
0.5	88.40	84.20	92.17	80.35
0.6	88.56	83.90	92.18	80.56

Appendix

Table 10: Sensitivity to

\lambda_{\text{base}}

(

\lambda_{\text{motion}}{=}0.4

\lambda_{\text{anchor}}{=}0.7

). AUC (%).

\star

: paper configuration.

Table 10

Figure 5: Sensitivity of frame-level AUC-ROC to the Gaussian smoothing width

\sigma

across datasets.

Prompt variant	Final
Default (paper — full prompt)	91.91
V1 (minimal — indicator lists removed)	91.98
V2 (no structure — lists + rules cut)	91.34
V3 (rephrased — same content, reworded)	92.05

Appendix

Table 13: Prompt sensitivity on XD (AUC-ROC, %).

Sampling (AUC%/h)	XD	UCF	MSAD	Shang
Uniform 1 FPS	84.00/7.96	65.02/3.51	77.57/0.78	71.54/0.13
Uniform 3 FPS	84.44/23.9	65.18/10.5	77.99/2.3	73.67 /0.4
Ours (adaptive 1 $\to$ 3 FPS)	84.69 /8.42	68.31 /4.12	81.51 /0.85	72.38/0.14

Appendix

Table 14: Sampling Schedule Ablation.

Video Anomaly Detection

Video anomaly detection identifies unusual events or behaviors in video, often when examples of the anomalies are scarce or unknown.

See all 113 papers

LATERN: Test-Time Context-Aware Explainable Video Anomaly Detection
May 14, 2026Mitchell Piehl, Muchao YeInterpretable Anomaly Detection VLM Reasoning
The University of Iowa, Iowa City IA 52245, USA
GridVAD: Open-Set Video Anomaly Detection via Spatial Reasoning over Stratified Frame Grids
Mar 26, 2026Mohamed Eltahir, Ahmed O. Ibrahim, Obada Siralkhatim +2Video Anomaly Detection Training-Free Video Anomaly Detection
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · African Institute for Mathematical Sciences (AIMS), Cape Town, South Africa · Independent Researcher +1
COPRA: Conditional Parameter Adaptation with Reinforcement Learning for Video Anomaly Detection
May 14, 2026Darryl Cherian Jacob, Xinyu Liu, Kai Wang +1VLM Adaptation Video Anomaly Detection
Auburn University · Tencent Hunyuan

May 14, 2026J/K move · Enter open · S save

LATERN: Test-Time Context-Aware Explainable Video Anomaly Detection

Mitchell Piehl, Muchao Ye

The University of Iowa, Iowa City IA 52245, USA

Interpretable Anomaly Detection VLM Reasoning Multimodal Anomaly Detection Video Anomaly Detection Temporal Video Understanding VLM Inference

Open paper AskPDF arXiv

Vision-language models (VLMs) have recently emerged as a promising paradigm for video anomaly detection (VAD) due to their strong visual reasoning ability and natural language-based explainability. In this paper, we aim to address a key limitation of such pipelines, which perform segment-level inference independently owing to token constraints and reason without structured temporal context, allowing VLMs to interpret anomalies as deviations from evolving video dynamics rather than producing fragmented predictions and explanations. To specify, we propose a context-aware framework named LATERN, which reformulates VAD as a temporal evidence aggregation process. LATERN consists of two complementary modules: Context-Aware Anomaly Scoring (CEA) and Recursive Evidence Aggregation (REA). CEA introduces a novel image-grounded memory mechanism, which selectively chooses historical content via frame diversity and visual-textual alignment as expanded context to help generate reliable anomaly scores. Building upon these scores, REA performs recursive temporal aggregation to identify coherent anomaly intervals and produce event-level decisions and explanations grounded in visual-textual evidence. Extensive experiments on challenging benchmarks, including UCF-Crime and XD-Violence, show that LATERN enhances detection accuracy and explanation consistency for frozen VLMs during test time, while generating temporally coherent and semantically grounded event-level explanations.

Abstract

Video Anomaly Detection

Explore similar work

LATERN: Test-Time Context-Aware Explainable Video Anomaly Detection

GridVAD: Open-Set Video Anomaly Detection via Spatial Reasoning over Stratified Frame Grids

COPRA: Conditional Parameter Adaptation with Reinforcement Learning for Video Anomaly Detection