AIGC Video Detection based on the fusion of spatial-frequency-optical flow multimodal features
Authors: S. Hong, X. Q. Wang, C. Zhang, J. C. Wang, P. X. Duan, Y. W. Wang
Organizations: School of Cyber Science and Technology, Beihang University, Beijing 100191, China · School of Information and Engineering, Nanchang University, Nanchang 330031, China · Continuing Education College, Beihang University, Beijing 100191, China · DBAPPSecurity Co., Ltd. (DAS-Security), Beijing 100195, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
The rapid evolution of generative AI (e.g., Sora, Hunyuan) makes it essential to develop effective detection strategies that can generalize across ever-evolving synthesis techniques. This study is motivated by the observation of a fundamental challenge in generative models: the inherent difficulty of maintaining cross-modal consistency between appearance and motion. To this end, we propose a multi-modal framework for AIGC video forgery detection tasks, named Cross-Attention based Video Forgery Detector (CrossAtt-VFD), based on joint multi-view analysis of content.Methodologically, we introduce a dual-branch architecture that simultaneously extracts spatial-frequency and optical-flow features.This approach enables the modeling of videos from complementary perceptual perspectives.The core of this process is a dedicated cross-attention mechanism, which governs the alignment of the two modalities and translates cross-modal inconsistencies into a potent diagnostic signal. This multi-modal strategy facilitates the detection of motion that is statistically inconsistent with the visual appearance of a scene. Comprehensive experimental results demonstrated that our model achieves an accuracy of 94.22%, a precision of 91.67 %,and a recall of 96.25 %, effectively verifying the advantages of the multi-modal fusion strategy.
Figures & tables
Figure 1: Example of inter-frame mutation in AIGC video. (a) Normal scene of the hand not yet appearing. (b) Sudden hand deformity appearing on the shoulder area.
Figure 2: Multi-modal framework overview.
Figure 3: Optical flow processing results of different video scenarios
Dataset
Real video
Fake video
Total
Training set
1 312
1 312
2 624
Validation set
448
448
896
Test set
640
703
1 343
Table 1: Composition of dataset videos
Index
Selecting keyframes
Entering full video directly
Loss function
0.2115±0.0024
0.2244±0.0030
Accuracy rate
93.30%±0.17%
92.50%±0.26%
Precision rate
91.28%±0.71%
90.73%±0.66%
Recall rate
94.25%±0.58%
93.00%±0.71%
F1-score
92.74%±0.45%
91.85%±0.32%
VRAM usage
9539MiB±17MiB
9693MiB±26MiB
Table 2: Comparison of training metrics for two different data loading methods
Figure 4: Training loss curves for different loading methods
Model
Accuracy (%)
Precision (%)
Recall (%)
F1-score (%)
CNN-LSTM-Attention
77.56
73.54
79.70
76.50
EfficientFormer
81.28
73.16
93.40
82.05
ViTranSP
89.77
87.14
91.12
89.09
CrossAtt-VFD
94.32
91.67
96.25
93.90
Table 3: Quantitative comparison with SOTA methods
Figure 5: Performance metric comparison curves of different detection models
Indicator
Only spatial mode
Spatial + frequency domain
Spatial + optical flow
Full triple-modality
Loss function
0.2115±0.0024
0.2027±0.0033
0.1910±0.0022
0.2252±0.0026
Accuracy rate
93.30%±0.17%
93.86%±0.24%
93.98%±0.14%
94.32%±0.10%
Precision rate
91.28%±0.71%
90.05%±0.99%
93.27%±0.56%
91.67%±0.45%
Recall rate
94.25%±0.58%
97.25%±1.39%
93.50%±0.40%
96.25%±0.57%
F1-score
92.74%±0.45%
93.51%±0.28%
93.38%±0.17%
93.90%±0.10%
VRAM usage
9539MiB±17MiB
9693MiB±12MiB
13830MiB±10MiB
13978MiB±8MiB
Table 4: Comparison of training indicators for different modal combinations
Figure 6: Evaluation indicators for different mode combinations
Figure 7: Display of test set detection results
Figure 8: Detection results on the open-source dataset DVF