cs.CVApr 7, 2026

Reasoning-Guided Grounding: Elevating Video Anomaly Detection through Multimodal Large Language Models

Authors: Sakshi Agarwal, Aishik Konwer, Ankit Parag Shah

Organizations: Center of Advanced AI, Accenture.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Video Anomaly Detection (VAD) has traditionally been framed as binary classification or outlier detection, providing neither interpretable reasoning nor precise spatial localization of anomalous events. While Vision-Language Models (VLMs) offer rich scene understanding, they struggle with reliable spatial grounding - often producing hallucinated or geometrically invalid bounding boxes when asked to localize objects. We propose VANGUARD (Video Anomaly Understanding through Reasoning and Grounding), a framework that unifies anomaly classification, spatial grounding, and chain-of-thought reasoning within a single VLM. VANGUARD introduces a three-stage curriculum that progressively layers training objectives: (1) classifier warmup on frozen backbone features, (2) LoRA-adapted spatial grounding, and (3) chain-of-thought generation. To overcome the sparse annotation typical of VAD benchmarks, we employ a teacher-student annotation pipeline in which a VLM (Qwen3-VL-4B) generates structured per-subclip reasoning trajectories based on manual annotations available from the UCA Dataset. Further, GroundingDINO provides bounding box supervision. On UCF-Crime, VANGUARD achieves 94% ROC-AUC with 84% F1 while simultaneously producing interpretable chain-of-thought explanations and spatial grounding of anomalous objects - capabilities absent from prior VAD methods. Ablations confirm that staged training outperforms monolithic optimization, and that structured reasoning acts as an implicit regularizer yielding more balanced predictions than classification-only fine-tuning. Zero-shot transfer to XD-Violence and ShanghaiTech demonstrates cross-domain generalization without target-domain adaptation.

Figures & tables

Figure 1: VANGUARD-Bench dataset creation pipeline and sample annotations. Left: Starting from UCF-Crime surveillance videos, we extract CLIP-based keyframes to segment each video into temporally distinct subclips. A vision-language model (Qwen3-VL) generates per-object annotations with event classification ( Normal / Abnormal ) and natural language reasoning. GroundingDINO then localizes each object with bounding boxes via Hungarian matching for one-to-one assignment. Right: Example annotations across three anomaly types. Each column shows the grounding frame of a subclip with spatially grounded objects ( red = abnormal, green = normal) and the VLM-generated reasoning below. The pipeline produces structured {Event, Reason, label, box} annotations that enable joint classification and spatial grounding supervision.
Figure 2: Overview of Vanguard ’s three-stage curriculum training procedure. Stage 1 trains only the classification head on video-level labels using Lbce\mathcal{L}_{\text{bce}} . Stage 2 unfreezes LoRA adapters and introduces mixed data (80% image-level detection, 20% video-level CoT) with spatial grounding losses Llm\mathcal{L}_{\text{lm}} and Lgiou\mathcal{L}_{\text{giou}} . Stage 3 refines reasoning on video-level CoT data only, dropping the GIoU loss. Each stage loads the checkpoint from the previous one, progressively building from classification to spatial grounding to chain-of-thought reasoning.
MethodUCF-CrimeXD-ViolenceShanghaiTech Campus
AUC ↑\uparrowF1 ↑\uparrowmeanIoU ↑\uparrowR@25 ↑\uparrowAUC ↑\uparrowF1 ↑\uparrowAUC ↑\uparrowF1 ↑\uparrow
ASK-Hint0.8983–––0.9031–––
VERA0.8655–––0.8826–––
LaVAD0.8028–––0.8536–––
Flashback (2025)0.8729–––0.9054–––
HolmesVAD0.72990.3721––0.95880.80000.5000 ⋆0.0000 ⋆
Table 1: Comparison with recent video anomaly detection methods on UCF-Crime, XD-Violence, and ShanghaiTech Campus datasets. We report detection metrics (AUC, F1) and spatial localization metrics (meanIoU, R@25). ↑\uparrow indicates higher is better. ⋆\star denotes frame-level evaluation.
Figure 3: Qualitative spatial grounding results on UCF-Crime test samples. Our trained model accurately localizes objects in surveillance frames by predicting bounding boxes (blue) that closely align with ground-truth annotations (green). Overlapping labels indicate correctly identified object categories. The model demonstrates robust grounding across diverse scenes including residential areas, streets, fire incidents, and nighttime surveillance.
Figure 4: Qualitative CoT-grounded reasoning for video anomaly detection on UCF-Crime test samples. Our model localizes objects in surveillance frames by predicting bounding boxes and provides per-object reasoning while recording observations. VANGUARD identifies anomalous regions in its analysis and predicts the final answer as ”Abnormal” or ”Normal.” The results demonstrate robust explainability and interpretability for video anomaly detection.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Qualitative comparison of spatial grounding across methods on UCF-Crime test samples. Each row shows a different test video frame. Columns from left to right: ground-truth annotations (green boxes with object labels), SpatialVLM, Kosmos-2, and Vanguard (red boxes). Vanguard produces tighter, more accurate bounding boxes with correct object-level anomaly attribution—localizing anomalous entities and their context (e.g., fire and van in Arson009 (a), motorcycle on the ground in Stealing079 (d))—while baseline methods either output single coarse bounding boxes covering the entire scene (Kosmos-2), or detect generic foreground objects irrespective of anomaly relevance (SpatialVLM). Notably in Shoplifting044 (e), SpatialVLM localizes bystanders rather than the shoplifting act. Vanguard also demonstrates stronger robustness in low-visibility night scenes such as Arson009 (a) and Shooting026 (b), where both baselines either fail to produce any boxes or yield imprecise localizations.
Figure 6: Qualitative comparison of CoT-grounded reasoning for video anomaly detection on UCF-Crime test samples. Each row shows sampled frames from a different test video. Columns display the generated reasoning from Vanguard (Ours), HolmesVAD, and Vad-R1. Vanguard produces structured, spatially-grounded reasoning—identifying anomalous objects with bounding box coordinates and per-object explanations (e.g., intruder breaking into a house at night in Burglary018 (b), person with a gun in Arrest007 (c))—while HolmesVAD generates brief responses that frequently misclassify anomalous scenes as normal, and Vad-R1 provides scene-level descriptions without explicit spatial grounding. The results demonstrate that Vanguard ’s curriculum-trained CoT reasoning acts as both an interpretability mechanism and an implicit regularizer, producing more balanced and explainable predictions than classification-only or prompt-only baselines.
Figure 7: Pipeline output for a subclip from Arrest002. The VLM detects two men engaged in physical aggression (Abnormal) alongside passive scene objects—a ladder, floor, and wall (Normal). GroundingDINO localizes each object with bounding boxes.
Figure 8: Spatial grounding output for an abnormal subclip from Arrest002. GroundingDINO localizes VLM-detected objects on the keyframe: two men involved in a physical altercation (red, Abnormal) and a stationary ladder in the background (green, Normal). Scene-level objects such as “floor” and “wall” were excluded as their bounding boxes exceeded 50% of the frame area. A greedy deduplication step ensures each detected person is assigned a distinct bounding box.
MetricValue
Videos (Normal / Abnormal)1,288 (644 / 644)
Total subclips2,092
Normal / Abnormal1,012 / 1,080
Subclips / video (mean / median)1.9 / 2
Subclip duration, median21.7 s
p25 / p755.5 s / 69.5 s
Appendix
Table 2: Phase 1: Subclipping strategy statistics. Videos are segmented via CLIP-based keyframe extraction and subsampled to at most two subclips per video.
MetricValue
Total samples (Normal / Abnormal)7,807 (3,941 / 3,866)
Subclips / video (mean / median)6.1 / 7
Total object instances29,203
Unique object labels2,606
Singleton labels1,412 (54.2%)
Rare labels (2–5 occurrences)729 (28.0%)
Appendix
Table 3: Phase 2: Object grounding statistics. Up to ten image-grounded subclips are sampled per video, each annotated with VLM-based object detection and GroundingDINO spatial localization on a single anchor frame.
MetricValue
Total CoT annotations2,092
Answer: Normal / Abnormal1,012 / 1,080
Section completeness (Obs / Ana / Ans)98.2 / 98.2 / 100%
Overall length
Words (mean / median)149 / 132
Characters (mean / median)915 / 818
Appendix
Table 4: Phase 3: Chain-of-thought annotation statistics. CoT text is generated by converting VLM object detections and bounding boxes into a structured Observations–Analysis–Answer format.
HyperparameterStage 1Stage 2Stage 3
Epochs233
Peak learning rate1×10−31\times 10^{-3}5×10−45\times 10^{-4}5×10−45\times 10^{-4}
LR schedulerCosineCosineCosine
Warmup ratio0.10.050.05
Batch size (per device)444
Gradient accumulation steps222
Appendix
Table 5: Training hyperparameters per curriculum stage.
Loss weightStage 1Stage 2Stage 3
λbce\lambda_{\text{bce}} (Binary CE)1.01.01.0
λlm\lambda_{\text{lm}} (Text generation)0.00.50.5
λgiou\lambda_{\text{giou}} (Text-coord GIoU)0.01.00.0
Data: Image-level detection–80%–
Data: UCF-Train (Videos-label)100%––
Data: Vanguard-Bench–20%100%
Appendix
Table 6: Loss weight ( λ\lambda ) configuration per curriculum stage.
λlm\lambda_{\text{lm}}λgiou\lambda_{\text{giou}}AUC ∗F1 ∗meanIoURecall@25
0.10.50.95530.90380.57050.4732
0.50.50.95530.90380.57050.4732
0.51.00.96350.90160.61910.5061
0.52.00.95510.89270.58050.4792
0.53.00.95230.88750.57640.4703
Appendix
Table 7: Sensitivity analysis of GIoU loss weight on UCF-Crime validation set ( λbce=1.0\lambda_{\text{bce}}=1.0 , λlm=0.5\lambda_{\text{lm}}=0.5 fixed). Bold indicates the configuration used in our final model. Note that ∗ AUC and F1 are computed at the subclip level in Vanguard-Bench.
Training StrategyAUCF1Prec.Rec.meanIoUR@25
Stage 1 only (2 ep.)0.94360.77500.82350.7317––
Stage 1 →\to Joint (all losses, 2 + 6 ep.)0.96700.87820.90840.85000.20640.089
Stage 1 →\to 2 (2+3 ep.)0.92980.81040.78920.83330.580.46
Stage 1 →\to 2 →\to 3 (2+3+3 ep.)0.93780.83600.81080.86270.620.51
Appendix
Table 8: Curriculum training vs. joint optimization on UCF-Crime. The curriculum achieves substantially stronger spatial grounding than monolithic training, with competitive classification performance.
Subclips/videoTrain samplesAUCF1
22,0922{,}0920.93780.8360
4∼\sim 4,2000.94290.8509
Appendix
Table 9: Effect of subclips per video on UCF-Crime performance (Stage 3 trained with different subclips/video). Increasing subclips improves performance up to a saturation point.
CategoryTotalTPFNAcc.Recall
Abuse2201.0001.000
Arrest5501.0001.000
Arson9901.0001.000
Assault3301.0001.000
Burglary131210.9230.923
Explosion212010.9520.952
Appendix
Table 10: Per-category anomaly detection accuracy on the UCF-Crime test set. Categories involving abrupt visual events achieve near-perfect recall, while Shoplifting—a visually subtle, slow-onset category—drops to 66.7%. Normal video accuracy (72.7%) reflects a 27.3% false-positive rate.

Explore similar work

CardsList
Aug 7, 2026

Glance, Scrutinize, and Think: Advancing Video Anomaly Detection from Training-Free to Agentic Reasoning

Video Anomaly Detection (VAD) aims to identify anomalous events and localize their temporal intervals. Existing approaches exhibit a "when-what" dissociation: traditional DNN-based methods localize when anomalies occur but lack semantic understanding, whereas LLM-based methods explain what happens but neglect precise temporal grounding. We attribute this to the absence of a unified reasoning paradigm. Inspired by how humans inspect surveillance videos - glancing globally to form temporal hypotheses, scrutinizing suspicious segments, and thinking iteratively to correct errors - we study this global-to-local paradigm from two perspectives. We first propose Glance then Scrutinize (GtS), a training-free framework using static and dynamic textual guidance for coarse-to-fine anomaly grounding and understanding, balancing accuracy and speed. To break the ceiling imposed by frozen external modules, we further propose a tool-augmented agentic VAD method, where a multimodal large language model learns to invoke a video cropping tool, inspect densely resampled frames, and self-correct mislocalized hypotheses, via cold-start supervised fine-tuning followed by reinforcement learning with a joint answer-grounding reward. For training and evaluation, we extend our prior VAGU benchmark into VAGU-T (Video Anomaly Grounding, Understanding, and Thinking), comprising 7,567 real-world videos over 21 anomaly categories with human-validated grounding, explanations, QA pairs, and chain-of-thought tool-calling traces. We further introduce JeAUG, a metric jointly evaluating semantic interpretability and temporal precision. Experiments show that GtS substantially surpasses training-free baselines, while the agentic model delivers both higher accuracy and faster inference.

Shibo Gao, Peipei Yang, Xu-Yao Zhang +1

School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China. · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China.

Anomaly LocalizationVideo Anomaly DetectionMultimodal Agents
AskPDF
Oct 1, 2026

Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding

Video Anomaly Detection (VAD) aims to temporally localize abnormal events in videos. Most existing approaches rely on dataset-specific training and curated annotations, limiting generalization in open-set scenarios. Recent zero-shot methods based on Large Vision- Language Models (LVLMs) alleviate this dependency but often lack temporal continuity and structured reasoning. We propose Cog-VADU, a fully training-free framework that reformulates VAD as a sequential cognitive reasoning task. Cog-VADU introduces Chain-of- Anomaly Detection Thought Prompting (CoADTP), which unrolls an LVLM into a recurrent reasoning chain across video segments. By propagating structured rationales over time, the model maintains implicit temporal memory, enabling robust discrimination between com- plex anomalies and high-motion normal activities. To improve reliability, we further design a cross-modal re-ranking stage that aligns textual rationales with visual embeddings, enforcing semantic consistency and temporal coherence for refined and stable predictions. Extensive experiments on multiple public VAD benchmarks demonstrate that Cog-VADU achieves competitive zero-shot performance. Moreover, cross-model evaluations show that CoADTP consistently enhances reasoning-based anomaly detection in a model-agnostic manner, pro- viding interpretable and generalizable anomaly understanding for real-world applications.

Mohd Ubaid Wani, Sara Atito, Josef Kittler +1

Centre for Vision, Speech and Signal Processing (CVSSP) University of Surrey, UK · Surrey Institute for People-Centred AI (PAI) University of Surrey, UK

CoT ReasoningInterpretable Anomaly DetectionVideo Anomaly DetectionZero-Shot Anomaly DetectionTemporal Video Understanding
AskPDF
May 22, 2026

CoReVAD: A Contextual Reasoning Framework for Training-Free Video Anomaly Detection

Existing Video Anomaly Detection (VAD) methods typically rely on task-specific training, leading to strong domain dependency and high training costs. Moreover, most existing methods output only scalar anomaly scores, providing limited insight into why specific events are considered abnormal. Recent advances in Vision-Language Models (VLMs) have enabled both anomaly detection and human-interpretable reasoning. However, many VLM-based approaches still require additional training steps (e.g., instruction tuning or verbalized learning) or external Large Language Models (LLMs), incurring further training costs and inference overhead. To address these challenges, we propose CoReVAD, a contextual reasoning framework for training-free video anomaly detection that operates with a single frozen VLM. CoReVAD directly generates anomaly scores and temporal descriptions from the VLM. To mitigate noise in generative outputs, we introduce a Local Response Cleaning (LRC) module based on local vision-text alignment. Furthermore, global temporal context and progression are incorporated through softmax-based refinement, Gaussian smoothing, and position weighting. Experiments on UCF-Crime and XD-Violence demonstrate that CoReVAD achieves competitive performance among training-free methods while providing reliable and interpretable explanations. Our official code is available at: https://github.com/Muk-00/CoReVAD

Hyeongmuk Lim, Youngbum Hur

Department of Industrial Engineering, Inha University, Incheon, Republic of Korea

Interpretable Anomaly DetectionVLM ReasoningVideo Anomaly DetectionZero-Shot Anomaly Detection
AskPDF