In the literature, prior research on Security-oriented Video Understanding (SVU) has predominantly focused on detecting and localize the threats (e.g., shootings, robberies) in videos, while largely lacking the effective capability to generate and evaluate the threat causes. Motivated by these gaps, this paper introduces a new chat paradigm SVU task, i.e., In-depth Security-oriented Video Understanding (DeepSVU), which aims to not only identify and locate the threats but also attribute and evaluate the causes threatening segments. Furthermore, this paper reveals two key challenges in the proposed task: 1) how to effectively model the coarse-to-fine physical-world information (e.g., human behavior, object interactions and background context) to boost the DeepSVU task; and 2) how to adaptively trade off these factors. To tackle these challenges, this paper proposes a new Unified Physical-world Regularized MoE (UPRM) approach. Specifically, UPRM incorporates two key components: the Unified Physical-world Enhanced MoE (UPE) Block and the Physical-world Trade-off Regularizer (PTR), to address the above two challenges, respectively. Extensive experiments conduct on our DeepSVU instructions datasets (i.e., UCF-C instructions and CUVA instructions) demonstrate that UPRM outperforms several advanced Video-LLMs as well as non-VLM approaches. Such information.These justify the importance of the coarse-to-fine physical-world information in the DeepSVU task and demonstrate the effectiveness of our UPRM in capturing such information.
Figures & tables
Figure 1. An example illustrating the DeepSVU task and the output of the proposed UPRM approach. In a surveillance video, UPRM can not only precisely identify and locate threatening segments but also attribute their causes. Overview figure for the DeepSVU task and UPRM outputs.
Figure 2. (a) The overall architecture of the Unified Physical-world Regularized MoE (UPRM) approach. (b) Unified Physical-world Enhanced MoE Block. It consists of two main components: Coarse-to-Fine Experts and Physical-world Trade-off Regularizer. (c) Coarse-grained and Fine-grained Experts, which is used to model the physical-world information.
Figure 3. Physical-world information statistics of our dataset.
Figure 4. Data composition for training and inference.
Approach
Identifying
Locating
Attributing
FNRs ↓
F2
mAP@tIoU
SB
Rouge
BLEU
GPT
Human
0.1
0.3
0.5
Average
Video Chat
32.65
69.93
90.55
78.61
58.71
75.95
0.39
0.36
27.94
6.54
3.23
Valley
58.91
45
92.04
67.66
14.43
58.04
0.3
0.31
22.1
5.93
2.97
PandaGPT
23.36
78.32
93.53
81.59
70.15
81.76
0.44
0.40
35.51
7.06
3.61
mPLUG-Owl
42.35
60.19
89.55
67.66
48.76
68.66
0.42
0.33
29.46
6.04
3.06
Table 1. Comparison of several Video-LLMs and UPRM on the Identifying and Locating capability of the CUVA instructions datasets. The ↓ next to FNRs indicates that a lower metric corresponds to better performance.
Identifying
Locating
mAP@tIoU
Models
FNRs ↓
F2
0.1
0.2
0.3
Average
VadClip †
67.54
31.62
22.69
13.03
7.98
14.57
BiConvLSTM †
81.99
18.95
21.32
8.46
4.78
11.52
X3D †
77.2
23.16
18.68
10.89
5.84
11.8
CLIP-TSA †
65.05
31.5
22.22
12.26
4.98
13.15
Table 2. Comparison of Identifying and Locating capabilities among several well-performing Video-LLMs and traditional non-LLM approaches conducted on the UCF-C instructions datasets, where † denotes non-LLM approaches.
Identifying
Locating
mAP@tIoU
Models
FNRs ↓
F2
0.1
0.3
0.5
Average
VadClip
63.29
37.44
87.06
39.4
13.12
46.53
BiConvLSTM
62.1
42.02
81.59
43.78
19.4
48.26
X3D
43.96
58.74
89.55
64.18
45.77
66.5
CLIP-TSA
61.45
46.22
84.12
49.56
15.72
49.8
Table 3. Comparison of Identifying and Locating capabilities among several well-performing non-LLM threat detection models on the CUVA instructions datasets.
Figure 5. Comparisons of Weights Assigned by Different Experts. w/o PTR is exactly the basic MoE expert weighting.
Figure 6. Convergence analysis of UPRM and other Video-LLMs on (a) CUVA and (b) UCF-C instruction datasets.
Figure 7. Two samples to compare UPRM with other Video-LLMs.
Figure 8. The statistics of physical-world information for (a) the UCF-C instructions dataset and (b) the CUVA instructions. These pie charts illustrate the imbalanced distribution of these information, with coarse-grained information and human-pose comprising the largest and second-largest segments. The surrounding images provide examples corresponding to each fine-grained physical-world information. The proportions are computed based on the frequency of each physical-world descriptions in the dataset, with segment size reflecting its relative occurrence.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9. Convergence Analysis. Investigation of the effect of component removal within the UPRM model on (a) CUVA and (b) UCF-C instructions datasets, illustrating the differences in training losses when specific components are excluded. “Faster" refers to how many times faster our method’s convergence point is compared to the slowest baseline.
Figure 10. Comparison of the average inference time for a one-minute video between UPRM and other Video-LLMs.
Approaches
Abuse
Arrest
Arson
Assault
Accidenct
Burglary
Explosion
Fighting
Robbery
Shooting
Stealing
Shoplifting
Vandalism
VadCLIP †
100
89.93
68.79
92.73
95.05
48.01
72.44
78.71
53.38
68.42
48.31
49.28
88.31
BiConvLSTM †
100
85.07
68.57
95.49
92.08
77.36
70.21
91.56
86.47
77.99
88.41
92.09
68.83
X3D †
100
69.03
65.36
95.49
81.19
67.03
89.72
85.71
92.48
61.84
79.71
96.4
51.95
CLIP-TSA †
100
46.27
72.14
61.81
83.17
61.78
86.52
71.43
77.44
56.82
48.31
73.38
53.25
Video Chat
87.5
42.16
81.07
78.47
47.52
56.34
74.11
49.91
68.35
54.6
64.25
71.58
66.23
Video ChatGPT
93.75
70.7
76.62
94.45
83.67
78.45
78.9
71.76
65.42
70.34
79.95
87.59
83.12
Appendix
Table 4. Comparison of our UPRM with several advanced Video-LLMs on different physical-world scenes (a total of 13 scenes) from the UCF-C instructions dataset, using FNRs.
Approaches
Fire
Violations
Pedestrain
Animals
Water Incident
Vandalism
Traffic Accidents
Robbery
Theft
Fighting
Burning
VadCLIP †
66.99
73.81
55.36
63.34
59.53
58.64
66.03
61.33
68.57
57
65.39
BiConvLSTM †
64.42
71.34
50.92
64.07
56.91
57.94
63.72
68.74
72.42
51.35
64.93
X3D †
41.58
21.56
43.42
64.29
42.94
35.7
40.23
17.22
38.55
36.85
48.81
CLIP-TSA †
59.76
58.29
58.34
59.24
58.11
60.02
59.87
60.45
56.93
55.62
57.88
Video Chat
47.73
21.9
47.29
30.78
20.61
27.26
20.19
44.81
63.06
50.34
26.84
Video ChatGPT
20.94
46.15
36.09
51.68
40.27
38.3
35.19
45.67
38.23
32.36
36.38
Appendix
Table 5. Comparison of our UPRM with several advanced Video-LLMs on different physical-world scenes (a total of 11 scenes) from the CUVA instructions dataset, using FNRs.