EDVD-LLaMA: Explainable Deepfake Video Detection via Multimodal Large Language Model Reasoning
Authors: Haoran Sun, Chen Cai, Huiping Zhuang, Kong Aik Lee, Lap-Pui Chau, Yi Wang
Organizations: Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, Guangzhou 511442, China
The rapid development of deepfake video technology has not only facilitated artistic creation but also made it easier to spread misinformation, which is increasingly difficult to identify. Traditional deepfake video detection (DVD) methods face issues such as a lack of transparency in their principles and insufficient generalization capabilities to cope with evolving forgery techniques. This highlights an urgent need for detectors that can identify forged content and provide verifiable reasoning explanations. This paper proposes the explainable deepfake video detection (EDVD) task and designs the EDVD-LLaMA multimodal, a large language model (MLLM) reasoning framework, which provides traceable reasoning processes alongside accurate detection results and trustworthy explanations. Our approach first incorporates a Spatio-Temporal Subtle Information Tokenization (ST-SIT) to extract and fuse global and local cross-frame deepfake features, providing rich spatio-temporal semantic information input for MLLM reasoning. Second, we construct a Fine-grained Multimodal Chain-of-Thought (Fg-MCoT) mechanism, which introduces facial feature data as hard constraints during the reasoning process to achieve pixel-level spatio-temporal video localization, suppress hallucinated outputs, and enhance the reliability of the chain of thought. In addition, we build an Explainable Reasoning FF++ benchmark dataset (ER-FF++set), leveraging structured data to annotate videos and ensure quality control, thereby supporting dual supervision for reasoning and detection. Extensive experiments demonstrate that EDVD-LLaMA achieves outstanding performance and robustness in terms of detection accuracy, explainability, and its ability to handle cross-forgery methods and cross-dataset scenarios. Compared to previous DVD methods, it provides a more explainable and superior solution.
Figures & tables
Figure 1: Left: Performance comparison between EDVD-LLaMA and MLLMs on the DVD task. Right: Performance comparison between EDVD-LLaMA and traditional methods on cross-forgery and cross-dataset detection tasks. EDVD-LLaMA demonstrates superior performance in the above tasks.
Figure 2: (a) Conventional deepfake video detection: only outputs a binary real/fake label. (b) Our method (MLLM + Fg-MCoT) fuses consecutive frame images and fine-grained facial landmarks to generate a multimodal reasoning chain, enabling a multimodal large language model to achieve multimodal interaction and provide explicit continuous reasoning evidence. “(omitted)” indicates content omitted for brevity.
Figure 3: EDVD-LLaMA pipeline. The candidate video is processed by Spatio-Temporal Subtle Information Tokenization (ST-SIT) to obtain video tokens, while the Face Landmark Detection Module provides facial metrics in parallel. In the two-stage Fine-grained Multimodal Chain-of-Thought (Fg-MCoT), Stage 1 combines the video tokens, facial metrics, and thought prompt to generate a rationale. Stage 2 combines the video tokens, rationale, and question prompt to produce the final answer. “omitted” indicates content omitted for brevity.
Figure 4: Construction pipeline of the proposed ER-FF++set. We sample videos for multiple manipulation types (Deepfakes, Face2Face, FaceShifter, FaceSwap) from FaceForensics++, and then apply manipulation-type–specific prompts to steer the Assistant LLM in producing structured rationales for each decision.
Methods
LLM
ER-FF++set
Acc. ↑
AUC ↑
F1 ↑
Video-LLaVA
Vicuna-7B
51.63
54.12
51.94
Video-ChatGPT
LLaVA-7B
52.27
56.53
52.38
VideoChat
Vicuna-7B
56.42
59.20
57.19
VideoLLaMA3
Qwen2.5-7B
72.48
75.93
73.50
EDVD-LLaMA (Ours)
Qwen2.5-7B
84.75
87.64
85.13
Table 1: Performance comparison of EDVD-LLaMA with MLLMs on the ER-FF++set for deepfake video detection.
Methods
Rationale Generation
CIDEr ↑
ROUGE_L ↑
BLEU-4 ↑
METEOR ↑
CSS ↑
Video-LLaVA
2.1738
0.6231
0.4165
0.3579
0.6342
Video-ChatGPT
2.2416
0.6542
0.4457
0.3748
0.6573
VideoChat
2.1952
0.6318
0.4221
0.3615
0.6457
VideoLLaMA3
2.3154
0.6679
0.4523
0.3842
0.6628
Ours
2.4579
0.7013
0.5048
0.4236
0.7135
Table 2: Performance comparison of MLLMs on the ER-FF++set for explanatory text generation.
Methods
Cross Dataset
Cross Avg.
WildDF
CelebDF
Conventional
Xception
58.03
66.35
62.19
DSP-FWA
53.39
57.11
55.25
B4
63.47
73.75
68.61
B4ATT
62.65
69.29
65.97
Table 3: Cross-dataset evaluation results on WildDF and CelebDF. Both MLLM-based and Conventional DVD methods are compared. Cross Avg. denotes the average score across the two public datasets.
Train
Methods
Test(Intra Dataset——Cross Forgery)
Cross Avg.
DF
F2F
FS
FSh
NT
DeepFake
Xception
-
51.25
49.87
52.03
52.92
51.52
DSP-FWA
-
51.73
52.03
54.94
53.17
52.97
B4
-
52.83
49.77
54.53
55.69
53.21
B4ATT
-
52.13
49.66
52.82
54.24
52.21
MCX-API
-
53.34
48.82
53.61
54.56
52.58
Table 4: Cross-forgery evaluation results on the ER-FF++set. Models are trained on one type of forgery and tested on others. Cross Avg. denotes the average score across forgery methods.
Figure 5: Qualitative results on ER-FF++set. We compare EDVD-LLaMA’s test results on deepfake and real videos from ER-FF++ with VideoLLaVA, VideoChatGPT, VideoChat, and VideoLLaMA3. Green boxes indicate correct judgments, and red boxes indicate incorrect decisions. “omitted” indicates content omitted for brevity.
#
Sdse
Scvc
Across
Acc. ↑
AUC ↑
F1 ↑
1
✓
✓
✓
84.75
87.64
85.13
2
✓
✓
×
80.83
84.06
81.27
3
✓
×
✓
82.98
86.17
83.44
4
✓
×
×
76.41
79.82
76.95
5
×
✓
×
74.52
77.10
74.96
6
×
×
×
72.48
75.93
73.50
Table 5: Ablation study results on the ER-FF++set. The ST-SIT module makes a significant contribution to the overall performance of EDVD-LLaMA.
Methods
Acc. ↑
AUC ↑
F1 ↑
Ours (w/ Fg-MCoT)
84.75
87.64
85.13
w/ Mc & w/o MΔ
82.13
86.57
82.74
w/o Mc & w/o MΔ
79.04
82.91
79.42
w/o Fg-MCoT
77.15
80.06
77.92
Table 6: Ablation study results on Fg-MCoT. Both the Mc and MΔ contribute significantly to the overall performance of EDVD-LLaMA.
Nc×Pf
Size
Acc.
AUC
F1
8×9
224×224
84.75
87.64
85.13
8×4
224×224
83.42
86.23
83.81
4×9
224×224
82.11
84.86
82.48
4×4
224×224
78.67
81.49
79.01
Table 7: Ablation study results on Sampling Strategy.