In the literature, prior research on Security-oriented Video Understanding (SVU) has predominantly focused on detecting and localize the threats (e.g., shootings, robberies) in videos, while largely lacking the effective capability to generate and evaluate the threat causes. Motivated by these gaps, this paper introduces a new chat paradigm SVU task, i.e., In-depth Security-oriented Video Understanding (DeepSVU), which aims to not only identify and locate the threats but also attribute and evaluate the causes threatening segments. Furthermore, this paper reveals two key challenges in the proposed task: 1) how to effectively model the coarse-to-fine physical-world information (e.g., human behavior, object interactions and background context) to boost the DeepSVU task; and 2) how to adaptively trade off these factors. To tackle these challenges, this paper proposes a new Unified Physical-world Regularized MoE (UPRM) approach. Specifically, UPRM incorporates two key components: the Unified Physical-world Enhanced MoE (UPE) Block and the Physical-world Trade-off Regularizer (PTR), to address the above two challenges, respectively. Extensive experiments conduct on our DeepSVU instructions datasets (i.e., UCF-C instructions and CUVA instructions) demonstrate that UPRM outperforms several advanced Video-LLMs as well as non-VLM approaches. Such information.These justify the importance of the coarse-to-fine physical-world information in the DeepSVU task and demonstrate the effectiveness of our UPRM in capturing such information.
Figures & tables
Figure 1. An example illustrating the DeepSVU task and the output of the proposed UPRM approach. In a surveillance video, UPRM can not only precisely identify and locate threatening segments but also attribute their causes. Overview figure for the DeepSVU task and UPRM outputs.
Figure 2. (a) The overall architecture of the Unified Physical-world Regularized MoE (UPRM) approach. (b) Unified Physical-world Enhanced MoE Block. It consists of two main components: Coarse-to-Fine Experts and Physical-world Trade-off Regularizer. (c) Coarse-grained and Fine-grained Experts, which is used to model the physical-world information.
Figure 3. Physical-world information statistics of our dataset.
Figure 4. Data composition for training and inference.
Approach
Identifying
Locating
Attributing
FNRs ↓
F2
mAP@tIoU
SB
Rouge
BLEU
GPT
Human
0.1
0.3
0.5
Average
Video Chat
32.65
69.93
90.55
78.61
58.71
75.95
0.39
0.36
27.94
6.54
3.23
Valley
58.91
45
92.04
67.66
14.43
58.04
0.3
0.31
22.1
5.93
2.97
PandaGPT
23.36
78.32
93.53
81.59
70.15
81.76
0.44
0.40
35.51
7.06
3.61
mPLUG-Owl
42.35
60.19
89.55
67.66
48.76
68.66
0.42
0.33
29.46
6.04
3.06
Table 1. Comparison of several Video-LLMs and UPRM on the Identifying and Locating capability of the CUVA instructions datasets. The ↓ next to FNRs indicates that a lower metric corresponds to better performance.
Identifying
Locating
mAP@tIoU
Models
FNRs ↓
F2
0.1
0.2
0.3
Average
VadClip †
67.54
31.62
22.69
13.03
7.98
14.57
BiConvLSTM †
81.99
18.95
21.32
8.46
4.78
11.52
X3D †
77.2
23.16
18.68
10.89
5.84
11.8
CLIP-TSA †
65.05
31.5
22.22
12.26
4.98
13.15
Table 2. Comparison of Identifying and Locating capabilities among several well-performing Video-LLMs and traditional non-LLM approaches conducted on the UCF-C instructions datasets, where † denotes non-LLM approaches.
Identifying
Locating
mAP@tIoU
Models
FNRs ↓
F2
0.1
0.3
0.5
Average
VadClip
63.29
37.44
87.06
39.4
13.12
46.53
BiConvLSTM
62.1
42.02
81.59
43.78
19.4
48.26
X3D
43.96
58.74
89.55
64.18
45.77
66.5
CLIP-TSA
61.45
46.22
84.12
49.56
15.72
49.8
Table 3. Comparison of Identifying and Locating capabilities among several well-performing non-LLM threat detection models on the CUVA instructions datasets.
Figure 5. Comparisons of Weights Assigned by Different Experts. w/o PTR is exactly the basic MoE expert weighting.
Figure 6. Convergence analysis of UPRM and other Video-LLMs on (a) CUVA and (b) UCF-C instruction datasets.
Figure 7. Two samples to compare UPRM with other Video-LLMs.
Figure 8. The statistics of physical-world information for (a) the UCF-C instructions dataset and (b) the CUVA instructions. These pie charts illustrate the imbalanced distribution of these information, with coarse-grained information and human-pose comprising the largest and second-largest segments. The surrounding images provide examples corresponding to each fine-grained physical-world information. The proportions are computed based on the frequency of each physical-world descriptions in the dataset, with segment size reflecting its relative occurrence.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9. Convergence Analysis. Investigation of the effect of component removal within the UPRM model on (a) CUVA and (b) UCF-C instructions datasets, illustrating the differences in training losses when specific components are excluded. “Faster" refers to how many times faster our method’s convergence point is compared to the slowest baseline.
Figure 10. Comparison of the average inference time for a one-minute video between UPRM and other Video-LLMs.
Approaches
Abuse
Arrest
Arson
Assault
Accidenct
Burglary
Explosion
Fighting
Robbery
Shooting
Stealing
Shoplifting
Vandalism
VadCLIP †
100
89.93
68.79
92.73
95.05
48.01
72.44
78.71
53.38
68.42
48.31
49.28
88.31
BiConvLSTM †
100
85.07
68.57
95.49
92.08
77.36
70.21
91.56
86.47
77.99
88.41
92.09
68.83
X3D †
100
69.03
65.36
95.49
81.19
67.03
89.72
85.71
92.48
61.84
79.71
96.4
51.95
CLIP-TSA †
100
46.27
72.14
61.81
83.17
61.78
86.52
71.43
77.44
56.82
48.31
73.38
53.25
Video Chat
87.5
42.16
81.07
78.47
47.52
56.34
74.11
49.91
68.35
54.6
64.25
71.58
66.23
Video ChatGPT
93.75
70.7
76.62
94.45
83.67
78.45
78.9
71.76
65.42
70.34
79.95
87.59
83.12
Appendix
Table 4. Comparison of our UPRM with several advanced Video-LLMs on different physical-world scenes (a total of 13 scenes) from the UCF-C instructions dataset, using FNRs.
Approaches
Fire
Violations
Pedestrain
Animals
Water Incident
Vandalism
Traffic Accidents
Robbery
Theft
Fighting
Burning
VadCLIP †
66.99
73.81
55.36
63.34
59.53
58.64
66.03
61.33
68.57
57
65.39
BiConvLSTM †
64.42
71.34
50.92
64.07
56.91
57.94
63.72
68.74
72.42
51.35
64.93
X3D †
41.58
21.56
43.42
64.29
42.94
35.7
40.23
17.22
38.55
36.85
48.81
CLIP-TSA †
59.76
58.29
58.34
59.24
58.11
60.02
59.87
60.45
56.93
55.62
57.88
Video Chat
47.73
21.9
47.29
30.78
20.61
27.26
20.19
44.81
63.06
50.34
26.84
Video ChatGPT
20.94
46.15
36.09
51.68
40.27
38.3
35.19
45.67
38.23
32.36
36.38
Appendix
Table 5. Comparison of our UPRM with several advanced Video-LLMs on different physical-world scenes (a total of 11 scenes) from the CUVA instructions dataset, using FNRs.
Though Multimodal Large Language Models (MLLMs) have shown impressive potential in video understanding, long video understanding (LVU) remains challenging since distracting noise in complex and lengthy contexts can obscure localized details, misleading MLLMs to produce incorrect answers. Recent works mitigate these issues by incentivizing deep reasoning to include relevant evidence. However, these methods have two main problems: First, the reinforcement fine-tuning framework (RFT) they leveraged incurs substantial training overheads, including high annotation costs and complicated reward designs. Second, the self-reflective and iterative-perception mechanism in some methods causes lengthy outputs and high inference latency. To alleviate these problems, we propose a novel Segment-to-Video Supervision} method (S2V) to efficiently enhance fine-grained reasoning in LVU. Specifically, we generate question answer pairs (VQA) based on localized segments, and then transfer these segment-based VQA back to the whole video for training. Due to focusing on short segments, segment-based VQA can naturally notice details which tend to be overlooked from a whole-video perspective. Training on such data can enforce MLLMs to correctly associate fine-grained details with QA while avoiding distracting noise in the whole video. The S2V training involves just reinforcement learning (RL) with a simple accuracy reward based on only 10K VQA samples and the resulting S2V model predicts answer using a single forward pass with limited output tokens. Experimental results demonstrate that S2V can consistently improve LVU performance across multiple LVU benchmarks, outperforming both general MLLMs and reasoning-based methods not only in LVU accuracy but also in training and inference efficiency.
Beibei Zhang, Chao Xu, Jun Lan +4
State Key Laboratory for Novel Software Technology, Nanjing University · Ant Group · School of Computer Science, Shanghai Jiao Tong University
Open-world video understanding often requires a model to locate sparse visual evidence and acquire external knowledge that is absent from the video and its parametric memory. While Thinking-with-Videos enables active temporal perception and Deep Research supports multi-step information seeking, the two capabilities are typically developed in isolation. We introduce VideoRover, a unified Video Deep Research framework that iteratively coordinates video cropping, multimodal search, and webpage browsing. Given a video-question pair, VideoRover uses each tool result to select the next action, so localized video clips guide external retrieval and retrieved evidence triggers further video inspection and verification. To develop this capability, we construct an automated data curation pipeline, producing 26K verified SFT trajectories and 3K challenging RL instances. We also introduce VideoRover-Bench, a benchmark stratified by video duration and research difficulty. Experiments on VideoDR and VideoRover-Bench show that our VideoRover-8B-RL achieves performance comparable to proprietary models in the direct-answer setting without tool use while outperforming larger open-source models equipped with the same tool suite. Ablation studies and training dynamics further validate the complementary roles of active video grounding, external retrieval, and long-horizon reinforcement learning.
Wenqi Liu, Shijie Ma, Yunxiao Wang +21
Shandong University · Institute of Automation, Chinese Academy of Sciences · Beihang University +4
As large language models (LLMs) become capable of processing increasingly diverse modalities and longer temporal contexts, an emerging line of work is moving beyond fixed video-language inference toward agentic systems that actively decide what information to inspect, retain, verify, and act upon. This survey reviews video understanding agents: systems that use video as the primary information source and solve understanding tasks through adaptive state construction and action selection. We first formalize an agent loop for video understanding, then address a central question: why do agents matter for video understanding? To answer this, we organize the literature through a challenge-to-design taxonomy, linking context bottlenecks to hierarchical evidence memory, evidence sparsity to active evidence acquisition, temporal causality to state and process tracking, and multimodal ambiguity to role-specialized coordination. We further review state space paradigms, learning paradigms, supervision signals, benchmarks, and evaluation protocols. Finally, we identify open directions toward agentic-native temporal modeling and video-native agents. Project page: https://github.com/DXY0711/Awesome-Agentic-Video-Understanding