EDVD-LLaMA: Explainable Deepfake Video Detection via Multimodal Large Language Model Reasoning
Authors: Haoran Sun, Chen Cai, Huiping Zhuang, Kong Aik Lee, Lap-Pui Chau, Yi Wang
Organizations: Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, Guangzhou 511442, China
The rapid development of deepfake video technology has not only facilitated artistic creation but also made it easier to spread misinformation, which is increasingly difficult to identify. Traditional deepfake video detection (DVD) methods face issues such as a lack of transparency in their principles and insufficient generalization capabilities to cope with evolving forgery techniques. This highlights an urgent need for detectors that can identify forged content and provide verifiable reasoning explanations. This paper proposes the explainable deepfake video detection (EDVD) task and designs the EDVD-LLaMA multimodal, a large language model (MLLM) reasoning framework, which provides traceable reasoning processes alongside accurate detection results and trustworthy explanations. Our approach first incorporates a Spatio-Temporal Subtle Information Tokenization (ST-SIT) to extract and fuse global and local cross-frame deepfake features, providing rich spatio-temporal semantic information input for MLLM reasoning. Second, we construct a Fine-grained Multimodal Chain-of-Thought (Fg-MCoT) mechanism, which introduces facial feature data as hard constraints during the reasoning process to achieve pixel-level spatio-temporal video localization, suppress hallucinated outputs, and enhance the reliability of the chain of thought. In addition, we build an Explainable Reasoning FF++ benchmark dataset (ER-FF++set), leveraging structured data to annotate videos and ensure quality control, thereby supporting dual supervision for reasoning and detection. Extensive experiments demonstrate that EDVD-LLaMA achieves outstanding performance and robustness in terms of detection accuracy, explainability, and its ability to handle cross-forgery methods and cross-dataset scenarios. Compared to previous DVD methods, it provides a more explainable and superior solution.
Figures & tables
Figure 1: Left: Performance comparison between EDVD-LLaMA and MLLMs on the DVD task. Right: Performance comparison between EDVD-LLaMA and traditional methods on cross-forgery and cross-dataset detection tasks. EDVD-LLaMA demonstrates superior performance in the above tasks.
Figure 2: (a) Conventional deepfake video detection: only outputs a binary real/fake label. (b) Our method (MLLM + Fg-MCoT) fuses consecutive frame images and fine-grained facial landmarks to generate a multimodal reasoning chain, enabling a multimodal large language model to achieve multimodal interaction and provide explicit continuous reasoning evidence. “(omitted)” indicates content omitted for brevity.
Figure 3: EDVD-LLaMA pipeline. The candidate video is processed by Spatio-Temporal Subtle Information Tokenization (ST-SIT) to obtain video tokens, while the Face Landmark Detection Module provides facial metrics in parallel. In the two-stage Fine-grained Multimodal Chain-of-Thought (Fg-MCoT), Stage 1 combines the video tokens, facial metrics, and thought prompt to generate a rationale. Stage 2 combines the video tokens, rationale, and question prompt to produce the final answer. “omitted” indicates content omitted for brevity.
Figure 4: Construction pipeline of the proposed ER-FF++set. We sample videos for multiple manipulation types (Deepfakes, Face2Face, FaceShifter, FaceSwap) from FaceForensics++, and then apply manipulation-type–specific prompts to steer the Assistant LLM in producing structured rationales for each decision.
Methods
LLM
ER-FF++set
Acc. ↑
AUC ↑
F1 ↑
Video-LLaVA
Vicuna-7B
51.63
54.12
51.94
Video-ChatGPT
LLaVA-7B
52.27
56.53
52.38
VideoChat
Vicuna-7B
56.42
59.20
57.19
VideoLLaMA3
Qwen2.5-7B
72.48
75.93
73.50
EDVD-LLaMA (Ours)
Qwen2.5-7B
84.75
87.64
85.13
Table 1: Performance comparison of EDVD-LLaMA with MLLMs on the ER-FF++set for deepfake video detection.
Methods
Rationale Generation
CIDEr ↑
ROUGE_L ↑
BLEU-4 ↑
METEOR ↑
CSS ↑
Video-LLaVA
2.1738
0.6231
0.4165
0.3579
0.6342
Video-ChatGPT
2.2416
0.6542
0.4457
0.3748
0.6573
VideoChat
2.1952
0.6318
0.4221
0.3615
0.6457
VideoLLaMA3
2.3154
0.6679
0.4523
0.3842
0.6628
Ours
2.4579
0.7013
0.5048
0.4236
0.7135
Table 2: Performance comparison of MLLMs on the ER-FF++set for explanatory text generation.
Methods
Cross Dataset
Cross Avg.
WildDF
CelebDF
Conventional
Xception
58.03
66.35
62.19
DSP-FWA
53.39
57.11
55.25
B4
63.47
73.75
68.61
B4ATT
62.65
69.29
65.97
Table 3: Cross-dataset evaluation results on WildDF and CelebDF. Both MLLM-based and Conventional DVD methods are compared. Cross Avg. denotes the average score across the two public datasets.
Train
Methods
Test(Intra Dataset——Cross Forgery)
Cross Avg.
DF
F2F
FS
FSh
NT
DeepFake
Xception
-
51.25
49.87
52.03
52.92
51.52
DSP-FWA
-
51.73
52.03
54.94
53.17
52.97
B4
-
52.83
49.77
54.53
55.69
53.21
B4ATT
-
52.13
49.66
52.82
54.24
52.21
MCX-API
-
53.34
48.82
53.61
54.56
52.58
Table 4: Cross-forgery evaluation results on the ER-FF++set. Models are trained on one type of forgery and tested on others. Cross Avg. denotes the average score across forgery methods.
Figure 5: Qualitative results on ER-FF++set. We compare EDVD-LLaMA’s test results on deepfake and real videos from ER-FF++ with VideoLLaVA, VideoChatGPT, VideoChat, and VideoLLaMA3. Green boxes indicate correct judgments, and red boxes indicate incorrect decisions. “omitted” indicates content omitted for brevity.
#
Sdse
Scvc
Across
Acc. ↑
AUC ↑
F1 ↑
1
✓
✓
✓
84.75
87.64
85.13
2
✓
✓
×
80.83
84.06
81.27
3
✓
×
✓
82.98
86.17
83.44
4
✓
×
×
76.41
79.82
76.95
5
×
✓
×
74.52
77.10
74.96
6
×
×
×
72.48
75.93
73.50
Table 5: Ablation study results on the ER-FF++set. The ST-SIT module makes a significant contribution to the overall performance of EDVD-LLaMA.
Methods
Acc. ↑
AUC ↑
F1 ↑
Ours (w/ Fg-MCoT)
84.75
87.64
85.13
w/ Mc & w/o MΔ
82.13
86.57
82.74
w/o Mc & w/o MΔ
79.04
82.91
79.42
w/o Fg-MCoT
77.15
80.06
77.92
Table 6: Ablation study results on Fg-MCoT. Both the Mc and MΔ contribute significantly to the overall performance of EDVD-LLaMA.
Nc×Pf
Size
Acc.
AUC
F1
8×9
224×224
84.75
87.64
85.13
8×4
224×224
83.42
86.23
83.81
4×9
224×224
82.11
84.86
82.48
4×4
224×224
78.67
81.49
79.01
Table 7: Ablation study results on Sampling Strategy.
Current Vision-Language Models (VLMs) for deepfake detection excel at identifying spatial artifacts but overlook a critical dimension: temporal inconsistencies in video forgeries. Adapting VLMs to reason about these dynamic cues remains a distinct challenge. To bridge this gap, we propose Forensic Answer-Questioning (FAQ), a large-scale benchmark that formulates temporal deepfake analysis as a multiple-choice task. FAQ introduces a three-level hierarchy to progressively evaluate and equip VLMs with forensic capabilities: (1) Facial Perception, testing the ability to identify static visual artifacts; (2) Temporal Deepfake Grounding, requiring the localization of dynamic forgery artifacts across frames; and (3) Forensic Reasoning, challenging models to synthesize evidence for final authenticity verdicts. We evaluate a range of VLMs on FAQ and generate a corresponding instruction-tuning set, FAQ-IT. Extensive experiments show that models fine-tuned on FAQ-IT achieve advanced performance on both in-domain and cross-dataset detection benchmarks. Ablation studies further validate the impact of our key design choices, confirming that FAQ is the driving force behind the temporal reasoning capabilities of these VLMs.
Zheyuan Gu, Qingsong Zhao, Yusong Wang +6
Institute of Artificial Intelligence, China Telecom (TeleAI) · Peking University · Fudan University
Recent Deepfake Video Detection (DFD) studies have demonstrated that pre-trained Vision-Language Models (VLMs) such as CLIP exhibit strong generalization capabilities in detecting artifacts across different identities. However, existing approaches focus on leveraging visual features only, overlooking their most distinctive strength -- the rich vision-language semantics embedded in the latent space. We propose VLAForge, a novel DFD framework that unleashes the potential of such cross-modal semantics to enhance model's discriminability in deepfake detection. This work i) enhances the visual perception of VLM through a ForgePerceiver, which acts as an independent learner to capture diverse, subtle forgery cues both granularly and holistically, while preserving the pretrained Vision-Language Alignment (VLA) knowledge, and ii) provides a complementary discriminative cue -- Identity-Aware VLA score, derived by coupling cross-modal semantics with the forgery cues learned by ForgePerceiver. Notably, the VLA score is augmented by an identity prior-informed text prompting to capture authenticity cues tailored to each identity, thereby enabling more discriminative cross-modal semantics. Comprehensive experiments on video DFD benchmarks, including classical face-swapping forgeries and recent full-face generation forgeries, demonstrate that our VLAForge substantially outperforms state-of-the-art methods at both frame and video levels. Code is available at https://github.com/mala-lab/VLAForge.
Jiawen Zhu, Yunqi Miao, Xueyi Zhang +2
Singapore Management University, Singapore · The University of Warwick, UK · Nanyang Technological University, Singapore +1
Multimodal large language models (MLLMs) offer a promising path toward interpretable deepfake detection by generating textual explanations. However, the reasoning process of current MLLM-based methods combines evidence generation and manipulation localization into a unified step. This combination blurs the boundary between faithful observations and hallucinated explanations, leading to unreliable conclusions. Building on this, we present VIGIL, a part-centric structured forensic framework inspired by expert forensic practice through a plan-then-examine pipeline: the model first plans which facial parts warrant inspection based on global visual cues, then examines each part with independently sourced forensic evidence. A stage-gated injection mechanism delivers part-level forensic evidence only during examination, ensuring that part selection remains driven by the model's own perception rather than biased by external signals. We further propose a progressive three-stage training paradigm whose reinforcement learning stage employs part-aware rewards to enforce anatomical validity and evidence--conclusion coherence. To enable rigorous generalizability evaluation, we construct OmniFake, a hierarchical 5-Level benchmark where the model, trained on only three foundational generators, is progressively tested up to in-the-wild social-media data. Extensive experiments on OmniFake and cross-dataset evaluations demonstrate that VIGIL consistently outperforms both expert detectors and concurrent MLLM-based methods across all generalizability levels.
Xinghan Li, Junhao Xu, Jingjing Chen
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China.