VESSI - VLM-Enhanced Support for Surveillance and Investigations

Saverio Cavasin; Pietro Tedeschi; Mattia Tamiazzo; Alessandro Brighente; Simone Milani; Mauro Conti

cs.CVOct 8, 2026

VESSI - VLM-Enhanced Support for Surveillance and Investigations

Authors: Saverio Cavasin, Pietro Tedeschi, Mattia Tamiazzo, Alessandro Brighente, Simone Milani, Mauro Conti

Organizations: Department of Mathematics, University of Padua, Padua, Italy · Innovation & Research Center, CY4GATE S.p.A., part of ELT Group, Rome, Italy · Department of Information Engineering, University of Padua, Padua, Italy · Örebro University, Örebro, Sweden

Abstract

Automated video surveillance analysis has become a critical component of intelligence infrastructures and Law Enforcement agencies. Traditional systems lack the semantic module for comprehensive situational awareness and forensic tasks, limiting their ability to interpret events meaningfully or support post-incident investigations. This slows operational insight and increases the burden on human analysts. Recent advances in Vision-Language Models (VLMs) offer promising pathways to bridge this gap. To address this, we propose VLM-Enhanced Support for Surveillance and Investigations (VESSI), a VLM-based framework designed to enhance automated video surveillance analysis through prompt-driven interrogation of video sequences where salient visual features are converted into textual descriptions. We test our framework with four state-of-the-art models. Since most datasets for this task are unlabeled, we also propose the Composite Model Utility Score (CMUS) to assess VLM performance. Experimental results show that our solution substantially improves the analysis capabilities of human operators and enhances the flexibility of automated surveillance systems. In our evaluation, the most reliable model flagged potentially relevant activity in more than 66% of the videos while reducing review time by more than 85%, offering a practical balance between selectivity and efficiency. The model ordering produced by the reference-free CMUS evaluation was reproduced by the normal-video CMUS evaluation and matched the false-positive-rate ordering obtained from 5,909 manually referenced frames. This agreement supports the operational use of the score within the evaluated setting.

Figures & tables

Feature / Capability	UCF-Crime [ 16 ]	UCA [ 17 ]	VIRAT [ 20 ]	LAVAD [ 25 ]	VERA [ 26 ]	VANE-Bench [ 29 ]	ASK-HINT [ 27 ]	CPVAD [ 28 ]	VESSI
Multimodal Data Analysis (video + language)	$\mdlgwhtcircle$	$\mdlgblkcircle$	$\mdlgwhtcircle$	$\mdlgblkcircle$	$\mdlgblkcircle$	$\mdlgblkcircle$	$\mdlgblkcircle$	$\mdlgblkcircle$	$\mdlgblkcircle$
Frame-level Semantic Analysis	$\mdlgwhtcircle$	$\mdlgblkcircle$	$\mdlgwhtcircle$	$\mdlgblkcircle$	$\mdlgblkcircle$	$\circlelefthalfblack$	$\circlelefthalfblack$	$\circlelefthalfblack$	$\mdlgblkcircle$
Interactive prompt-based querying	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\circlelefthalfblack$	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\mdlgblkcircle$
Real-time forensic applicability	$\mdlgblkcircle$	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\circlelefthalfblack$	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\circlelefthalfblack$	$\mdlgblkcircle$
Lightweight SOTA model deployed ( $\leq$ 7B)	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\mdlgwhtcircle$	$\mdlgblkcircle$	$\mdlgwhtcircle$	$\mdlgblkcircle$
No manual annotation requirement	$\mdlgblkcircle$	$\mdlgwhtcircle$	$\mdlgblkcircle$	$\mdlgblkcircle$	$\circlelefthalfblack$	$\mdlgwhtcircle$	$\mdlgblkcircle$	$\mdlgblkcircle$	$\mdlgblkcircle$

Table I: Comparison of VESSI with surveillance datasets, benchmarks, and recent VLM -based anomaly-detection systems. Legend:

\mdlgblkcircle

Feature fully fulfilled,

\circlelefthalfblack

Feature partially fulfilled,

\mdlgwhtcircle

Feature not fulfilled.

Class	Number of Videos	Total Duration (min)
Abuse	$50$	$107.50$
Arrest	$50$	$165.22$
Arson	$50$	$151.06$
Assault	$50$	$72.19$
Burglary	$100$	$261.78$
Explosion	$50$	$140.22$

Table II: Dataset classes, number of videos, and total duration (in minutes) per class.

Panel of Figure 2 — Figure 2: Examples of successful detections from three models across different actions and prompt difficulties. Confidence values are internal generation scores used to filter and rank detections; they are not calibrated probabilities of correctness.

	DeepSeek-VL		InstructBLIP		LLaVA		Qwen2.5-VL
Category	Cov. (%)	Det. Rate	Cov. (%)	Det. Rate	Cov. (%)	Det. Rate	Cov. (%)	Det. Rate
Abuse	$98.67$	$0.8353$	$98.00$	$0.8582$	$50.67$	$0.1652$	$32.67$	$0.0292$
Arrest	$100.00$	$0.7011$	$100.00$	$0.8283$	$86.00$	$0.3258$	$80.00$	$0.2454$
Arson	$96.67$	$0.4096$	$100.00$	$0.6046$	$62.00$	$0.0969$	$71.33$	$0.0800$
Assault	$96.67$	$0.6202$	$98.67$	$0.8525$	$70.67$	$0.2162$	$59.33$	$0.1327$
Burglary	$95.33$	$0.6059$	$100.00$	$0.9717$	$89.00$	$0.4351$	$79.00$	$0.2271$

Table III: Average coverage (%) and detection rate per model and category. Values are averaged across the three prompt difficulty levels (easy, medium, hard). Normal videos are excluded.

Model	$<10\%$	$10-40\%$	$40-70\%$	$70-95\%$	$\geq 95\%$	$H_{m}$
Qwen2.5-VL	$1591$	$753$	$344$	$150$	$12$	$\bf{-0.1598}$
InstructBLIP	$106$	$182$	$193$	$337$	$2032$	$1.2030$
DeepSeek-VL	$390$	$454$	$476$	$650$	$880$	$0.7063$
LLaVA	$1223$	$658$	$491$	$355$	$123$	$0.0609$

Table IV: Number of video–prompt pairs falling into detection-rate bands.

Model	Processing time (min)	Time saved (%)
DeepSeek-VL	$40.29$	$74.13$
InstructBLIP	$14.09$	$90.96$
LLaVA	$55.17$	$64.59$
Qwen2.5-VL	$22.82$	$85.32$

Table V: Mean processing time and time saved across categories.

Model	$C_{m}$	$T_{m}$	$D_{m}$	$H_{m}$	CMUS score
Qwen2.5-VL	$0.6690$	$0.8532$	$0.1579$	$-0.1598$	$\mathbf{1.6031}$
InstructBLIP	$0.9908$	$0.9096$	$0.8304$	$1.2030$	$0.2822$
DeepSeek-VL	$0.9562$	$0.7413$	$0.6172$	$0.7063$	$0.6826$
LLaVA	$0.7406$	$0.6459$	$0.2695$	$0.0609$	$1.1909$

Table VI: Normalized CMUS inputs and resulting scores.

Model	$\mathrm{FPR}_{\mathrm{human}}$	$\mathrm{TPR}_{\mathrm{human}}$
Qwen2.5-VL	$0.0644$	$0.2535$
LLaVA	$0.1781$	$0.4073$
DeepSeek-VL	$0.5497$	$0.7832$
InstructBLIP	$0.8195$	$0.9471$

Table VII: Human-reference false-positive and true-positive rates aggregated across the three prompt formulations.

Model	$C_{m}$	$T_{m}$	$D_{m}$	$H_{m}$	CMUS score
Qwen2.5-VL	$0.0154$	$0.8421$	$0.0007$	$-0.5000$	$\bf{1.3571}$
InstructBLIP	$0.6205$	$0.9069$	$0.4191$	$0.29$	$1.0279$
DeepSeek-VL	$0.3877$	$0.7266$	$0.1711$	$-0.1767$	$1.2054$
LLaVA	$0.0959$	$0.6362$	$0.0143$	$-0.4867$	$1.2116$

Table VIII: Control-group evaluation on 50 normal videos queried for all 13 crime categories at three prompt levels. All reported quantities are computed specifically on the control group.

Figure 3: Mean and standard deviation of each metric across all models, prompts, and categories. Each bar aggregates over $150{,}000$ frames.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Description
Frame ID	Unique frame index
Timestamp (s)	Relative time since start
Prompt	Input natural language query
Model	Model name (e.g., InstructBLIP)
Tokens	Prompt token count
Readability	Flesch-Kincaid grade level

Appendix

Table S1: Logged metrics per frame.

Frame	Time (s)	Model	VRAM (MB)	Power (W)	Conf.
$30$	$1.00$	InstructBLIP	$16,444.94$	$174.87$	$0.78$
$60$	$2.00$	InstructBLIP	$16,450.94$	$167.93$	$0.78$
$90$	$3.00$	InstructBLIP	$16,436.94$	$167.32$	$0.78$

Appendix

Table S2: Example GPU monitoring entries for the Arson class using InstructBLIP with a hard prompt.

Panel of Figure 14 — Figure S1: Computational profiles of the four models. Panels (a)–(e) report frame-level distributions across the full experimental dataset; panel (f) summarizes normalized model-level averages.

	Time in minutes					Perc. of time saved
Class	Original	DeepSeek-VL	InstructBLIP	LLaVA	Qwen2.5-VL	DeepSeek-VL	InstructBLIP	LLaVA	Qwen2.5-VL
Abuse	$107.50$	$28.58$	$9.71$	$38.25$	$17.25$	$73.41$ %	$90.97$ %	$64.42$ %	$83.95$ %
Arrest	$165.22$	$42.99$	$14.97$	$58.61$	$24.13$	$73.98$ %	$90.94$ %	$64.53$ %	$85.40$ %
Arson	$151.06$	$39.22$	$13.63$	$53.50$	$22.94$	$74.04$ %	$90.98$ %	$64.58$ %	$84.81$ %
Assault	$72.19$	$18.62$	$6.49$	$25.47$	$10.69$	$74.21$ %	$91.01$ %	$64.72$ %	$85.19$ %
Burglary	$261.78$	$67.56$	$23.66$	$92.71$	$38.31$	$74.19$ %	$90.96$ %	$64.58$ %	$85.37$ %

Appendix

Table S3: Processing time and percentage saved per category, averaged across prompt levels. The final row reports the arithmetic mean across categories.

Dataset and Models
$\mathcal{V}$	Set of videos = $\{\mathcal{V}_{1},\dots,\mathcal{V}_{13}\}$ , where each subset $\mathcal{V}_{c}$ contains all videos $v$ belonging to the $c$ -th crime class.
$\mathcal{M}$	Model space = { Qwen2.5-VL , InstructBLIP , DeepSeek-VL , and LLaVA-1.5 }
$\mathcal{F}$	Frame set, with frame $f\in\mathcal{F}$ .
Inference and Efficiency
$T_{or}$	Total original video duration (in minutes).
$T_{in}$	Total inference time per model.

Appendix

Table S4: Summary of notation and symbols.

Explore similar work

Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows
Sep 23, 2026Xiyuan Shen, Jiuyang Lyu, Seokhyun Hwang +4Video Understanding Video-Language Model Evaluation
University of Washington, USA · Georgia Institute of Technology, USA · Google DeepMind, USA
Parser-Free VLM Verification for Federated Weakly Supervised Video Anomaly Detection
Sep 7, 2026Sébastien Thuau, Amira Gran, Siba Haidar +1Multimodal Anomaly Detection Weakly Supervised Video Anomaly Detection
esieaLab, ESIEA, Paris, France · ETIS Laboratory, CNRS, UMR8051, CY Cergy Paris University, Paris, France
BLUE: Semantics-Preserving Video Compression for Efficient Vision-Language Surveillance Analytics
Jul 21, 2026Shubham Baid, Akash James, Sahil Chachra +2Efficient VLM Inference Video Compression
KGraph AI Solutions Pvt. Ltd., Bangalore, India – 560016

Sep 23, 2026, cs.CVJ/K move · Enter open · S save

Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows

Xiyuan Shen, Jiuyang Lyu, Seokhyun Hwang, Huanfen Yao+3

University of Washington, USA · Georgia Institute of Technology, USA · Google DeepMind, USA

Video Understanding Video-Language Model Evaluation Human-in-the-Loop Annotation

Open paper AskPDF arXiv

Video provides a rich record of human behavior, interaction, and situated contexts, offering important evidence for understanding people and conducting human-centered research. As vision-language models (VLMs) become increasingly capable of analyzing video, they offer opportunities to automate this traditionally human-intensive process. Yet a central question remains: when can VLMs analyze human-centered video independently, and when does reliable analysis still require human involvement? To address this question, we first characterize video analysis practices in human-centered research. We systematically analyze all 1,702 CHI 2026 full papers and identify 125 that annotate videos. Through iterative coding, we derive a five-dimensional taxonomy spanning analytic purpose, viewpoint, phenomenon, reasoning requirement, and annotation authority. Grounded in recurring annotation tasks captured by this taxonomy, we construct a benchmark of 15 representative tasks from open datasets to map the capabilities and limitations of a general-purpose VLM. We examine the division of labor between humans and VLMs by comparing three annotation workflows: VLM alone, human alone, and human verification of VLM outputs. Across tasks, VLM-alone annotation approaches human accuracy on average (HNS = 97.0, where 100 denotes human-alone performance), demonstrating substantial potential to automate human-centered video analysis. Human verification achieves the highest accuracy (HNS = 121.5) while reducing human annotation time by 48.9% and monetary cost by 31.3%-44.5% relative to human-alone annotation. Our findings connect real-world human-centered video analysis tasks and current VLM capabilities, and clarify how human-AI collaboration can make VLM-assisted analysis reliable and efficient.

Abstract

Explore similar work

Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows

Parser-Free VLM Verification for Federated Weakly Supervised Video Anomaly Detection

BLUE: Semantics-Preserving Video Compression for Efficient Vision-Language Surveillance Analytics