SOVABench: A Vehicle Surveillance Action Retrieval Benchmark for Multimodal Large Language Models

Oriol Rabasseda; Zenjie Li; Kamal Nasrollahi; Sergio Escalera

cs.CVJan 8, 2026

SOVABench: A Vehicle Surveillance Action Retrieval Benchmark for Multimodal Large Language Models

Authors: Oriol Rabasseda, Zenjie Li, Kamal Nasrollahi, Sergio Escalera

Organizations: Universitat de Barcelona and Computer Vision Center Gran Via de les Corts Catalanes 585, Barcelona Campus UAB, Edifici O, Cerdanyola del Vall`es · Milestone Systems A/S Banemarksvej 50, Brøndby · Aalborg Universitet Fredrik Bajers Vej 7K, Aalborg Øst

Abstract

Automatic identification of events and recurrent behavior analysis are critical for video surveillance. However, most existing content-based video retrieval benchmarks focus on scene-level similarity and do not evaluate the action discrimination required in surveillance. To address this gap, we introduce SOVABench (Surveillance Opposite Vehicle Actions Benchmark), a real-world retrieval benchmark built from surveillance footage and centered on vehicle-related actions. SOVABench defines two evaluation protocols (inter-pair and intra-pair) to assess cross-action discrimination and temporal direction understanding. Although action distinctions are generally intuitive for human observers, our experiments show that they remain challenging for state-of-the-art vision and multimodal models. Leveraging the visual reasoning and instruction-following capabilities of Multimodal Large Language Models (MLLMs), we present a training-free framework for producing interpretable embeddings from MLLM-generated descriptions for both images and videos. The framework achieves strong performance on SOVABench as well as on several spatial and counting benchmarks where contrastive Vision-Language Models often fail. The code, annotations, and instructions to construct the benchmark are publicly available.

Figures & tables

Panel of Figure 2 — Figure 2: Samples and performance in SOVABench. (a) Illustrative samples of the constructed benchmark of four different actions (close trunk, open vehicle door, start, and turn left), and (b) comparison of methods for the two evaluation protocols in SOVABench. Methods include MLLMs using the MLLM-to-Embedding framework to obtaining embeddings (bold) and contrastive VLMs. For reference, random values are 3.4 mAP and 50.3 Pair-mAP in Inter-pair and Intra-pair protocols respectively.

Action	Opposite	Inter-	Intra-
	Action	pair	pair
Drive forward	Reverse	✗	✔
Enter vehicle	Exit vehicle	✔	✔
Load vehicle	Unload vehicle	✔	✔
Open trunk	Close trunk	✔	✔
Open vehicle door	Close vehicle door	✔	✔

Table 1: Opposite vehicle-related actions included in SOVABench. Check marks indicate the use of the pair for each evaluation protocol.

Panel of Figure 3 — Figure 3: Statistical overview of the SOVABench (Inter-pair) benchmark. The class count of the SOVABench (Intra-pair) benchmark can be accessed in the supplementary material. (a) Sample count per class in queries. (b) Distribution of clip durations. (c) Resolution distribution resulting from spatial cropping.

Dataset	Data	Metric	#Samples	#Queries	Retrieval task
FIVR-200K [ 14 ]	Video	mAP	226k	100	Fine-grained Incident Retrieval
VCDB [ 11 ]	Video	Prec. & Recall	100.5k	528	Near-Duplicate Retrieval
EVVE [ 27 ]	Video	mAP	102.4k	620	Event Retrieval
SOVABench (Inter-pair)	Video	mAP	9.9k	1,423	Action Retrieval
SOVABench (Intra-pair)	Video	Pair-mAP	2.3k	2,300	Opposite Action Retrieval

Table 2: Comparison of representative CBVR benchmarks to SOVABench. SOVABench provides a large number of queries, which are the samples used for evaluation. Retrieval performance is measured using mAP in most of the benchmarks.

Figure 4: Overview of the MLLM-to-Embedding framework. Given an image/video and the textual instruction, an MLLM first generates a descriptive textual response. The output is then split into individual sentences, each encoded using a sentence-similarity text encoder. The similarity between two samples is computed using the maximum pairwise cosine similarity between their sentence embeddings (Equation ( 3 )). The filmstrip is extracted from SOVABench.

Model	SpatialBench		VSR	What’s Up		CountBench	Visual7W-	Spatial Avg.	Count Avg.	Avg.
	InD.	OutD.		A	B		Count
Random	29.8	29.8	50.0	25.0	25.0	10.0	10.0	31.9	17.5	24.7
CLIP-ViT-B/32	28.9	36.4	53.7	31.3	32.4	29.2	33.8	36.5	31.5	34.0
CLIP-ViT-H-14 † [ 19 ]	–	–	54.5	–	–	–	–	–	–	–
XVLM 16M † [ 13 ]	–	–	–	50.7	33.1	–	–	–	–	–
BLIP 14M † [ 13 ]	–	–	–	38.8	38.2	–	–	–	–	–

Table 3: Comparison of CLIP and state-of-the-art approaches to the MLLM-to-Embedding framework across spatial understanding and object counting classification benchmarks. For SpatialBench, we report the Indoor (InD.) and Outdoor (OutD.) datasets and for What’s Up we report the subsets A and B. The final three columns average the results for spatial, counting, and spatial&counting. In all cases, the metric is accuracy. † Results obtained from the respective paper.

Model	Inter-	Intra-
	pair	pair
Random	3.4	50.3
Contrastive Image-VLMs
CLIP-ViT-L-14 [ 26 ]	29.1	51.3
SigLIP2-Giant [ 33 ]	30.6	51.1
MERU [ 8 ]	28.6	51.3

Table 4: Performance comparison of models in SOVABench. Models considered include contrastive image-VLMs, contrastive video-VLMs, general MLLMs, video-MLLMs, and API MLLMs. Metrics are mAP for SOVABench (Inter-pair) and Pair-mAP for SOVABench (Intra-pair).

Model	SpatialBench		VSR	What’s Up		CountBench	Visual7W-	Spatial Avg.	Count Avg.	Avg.
	InD.	OutD.		A	B		Count
InternVL3.5 2B ${}_{\text{GENERAL}}$
(w/ Sent. split + max)	37.1	39.0	51.9	62.4	28.7	39.1	50.4	43.8	44.8	44.3
InternVL3.5 2B ${}_{\text{GENERAL}}$
(w/o Sent. split + max)	35.0	28.9	53.0	32.5	23.0	48.7	50.7	34.5	49.7	42.1
InternVL3.5 2B ${}_{\text{TASK-AWARE}}$

Table 5: Evaluation of the inclusion of the sentence splitting with maximum aggregator. The model used is InternVL3.5 2B and both general and task-aware prompting strategies are reported. All metrics are accuracy.

Model	Params.	Emb. size	mAP
CLIP-ViT-L-14 (Text tower)	123M	768	33.5
all-MiniLM-L6-v2 5 5 5 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2	23M	384	37.8
EmbeddingGemma [ 34 ]	308M	768	36.7
GTE-Large-8152	409M	1024	38.3
mE5 Large Instruct [ 35 ]	560M	1024	37.7
Qwen3 Embedding 0.6B [ 46 ]	596M	1024	36.9

Table 6: Sensitivity of the MLLM-to-Embedding framework of the choice of sentence embedder. Executions are done with MiniCPM-V 4.5

{}_{\text{TASK-AWARE}}

on SOVABench (Inter-pair) with 1 FPS. The number of parameters (Params.) and the dimensionality of the embedding space (Emb. size) are reported.

Frame rate	Inter-Pair	Intra-Pair
	(mAP)	(Pair-mAP)
1 FPS	38.3	53.6
3 FPS	37.3	54.0
5 FPS	36.3	53.8
7 FPS	36.2	53.4

Table 7: Sensitivity of SOVABench evaluations on the sampling rate. Executions are done with MiniCPM-V 4.5

{}_{\text{TASK-AWARE}}

Panel of Figure 11 — Figure A.1 : Number of video samples per action class within each opposite pair in the SOVABench (Intra-Pair) benchmark.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Efficiency	Inter-pair
		(Constr.)
Random	–	23.7
Contrastive Image-VLMs
CLIP-ViT-L-14	22.88	37.4
SigLIP2-Giant	3.94	38.7
MERU	21.25	36.9

Appendix

Table C.1 : Performance comparison of models in SOVABench (Inter-pair) restricted to the set of query samples, and efficiency comparison. Efficiency is calculated as instances processed per second in the evaluation of the default SOVABench (Inter-pair) benchmark including distracting samples. SOVABench (Inter-pair) uses mAP.

Panel of Figure 14 — Figure E.2 : Examples of generation errors obtained from MiniCPM-V 4.5 ${}_{\text{TASK-AWARE}}$ in the $<$ Open trunk, Close trunk $>$ pair. Errors are prioritized from more to less sever, including Generation Errors / Hallucination (a)-(c), Under-Description (d)-(e), Temporal Misunderstanding (f)-(g), and Action Assumption (h)-(i). In addition, (j) shows a successful case. Each example is composed by a filmstrip of the video, the model answer and an observation indicating why is wrong.

Video Retrieval

Video retrieval matches natural-language, visual, or multimodal queries to relevant videos, clips, and moments.

See all 61 papers

CrossView: Can Vision-Language Models Reason Across Cameras?
Aug 16, 2026Sahil Shah, S P Sharan, Harsh Goel +4VLM Evaluation Video QA
The University of Texas at Austin, Austin TX, USA · Independent Researcher, USA
Benchmarking Visual State Tracking in Multimodal Video Understanding
Jun 2, 2026Sihyun Yu, Nanye Ma, Pinzhi Huang +8Multimodal Large Language Models Video Reasoning
New York University · KAIST
VESSI - VLM-Enhanced Support for Surveillance and Investigations
Oct 8, 2026Saverio Cavasin, Pietro Tedeschi, Mattia Tamiazzo +3Video Surveillance Vision-Language Models
Department of Mathematics, University of Padua, Padua, Italy · Innovation & Research Center, CY4GATE S.p.A., part of ELT Group, Rome, Italy · Department of Information Engineering, University of Padua, Padua, Italy +1

Aug 16, 2026J/K move · Enter open · S save

CrossView: Can Vision-Language Models Reason Across Cameras?

Sahil Shah, S P Sharan, Harsh Goel, Manvik Pasula+3

The University of Texas at Austin, Austin TX, USA · Independent Researcher, USA

VLM Evaluation Video QA VLMs for Autonomous Driving

Open paper AskPDF arXiv

Video understanding benchmarks have long centered on single-camera settings, where modern multi-modal language models achieve strong performance across image and video tasks. Yet, the real world runs on multi-camera networks: autonomous vehicles, security systems, and robots all gather data across many simultaneous views. We argue that this is not simply "more" of the single-camera problem; it is fundamentally different. Multi-camera reasoning requires handling context that scales with the number of views, resolving occlusions visible from only a subset of cameras, judging which views matter, and integrating evidence across perspectives that may overlap or diverge. Current models struggle with exactly these challenges, yet no benchmark systematically targets them. We introduce CrossView, a multi-camera video question-answering benchmark spanning autonomous driving, security surveillance, egocentric/exocentric video, and robotics. Evaluation of proprietary models, such as GPT-5.2, and open-source models, like Qwen3-VL, reveals consistently low accuracy, with open-source models trailing by a wide margin. Performance scales strongly with a model's ability to jointly process multiple viewpoints, positioning CrossView as a rigorous benchmark for multi-camera video. We open-source our code and dataset at https://utaustin-swarmlab.github.io/CrossView.

Abstract

Video Retrieval

Explore similar work

CrossView: Can Vision-Language Models Reason Across Cameras?

Benchmarking Visual State Tracking in Multimodal Video Understanding

VESSI - VLM-Enhanced Support for Surveillance and Investigations