VRAG-DFD: Verifiable Retrieval-Augmentation for MLLM-based Deepfake Detection

Hui Han; Shunli Wang; Yandan Zhao; Taiping Yao; Shouhong Ding

cs.CVApr 15, 2026

VRAG-DFD: Verifiable Retrieval-Augmentation for MLLM-based Deepfake Detection

Authors: Hui Han, Shunli Wang, Yandan Zhao, Taiping Yao, Shouhong Ding

Organizations: Shanghai Jiao Tong University · Tencent Youtu Lab

Abstract

In Deepfake Detection (DFD) tasks, researchers proposed two types of MLLM-based methods: complementary combination with small DFD detectors, or static forgery knowledge injection. The lack of professional forgery knowledge hinders the performance of these DFD-MLLMs. To solve this, we deeply considered two insightful issues: How to provide high-quality associated forgery knowledge for MLLMs? AND How to endow MLLMs with critical reasoning abilities given noisy reference information? Notably, we attempted to address above two questions with preliminary answers by leveraging the combination of Retrieval-Augmented Generation (RAG) and Reinforcement Learning (RL). Through RAG and RL techniques, we propose the VRAG-DFD framework with accurate dynamic forgery knowledge retrieval and powerful critical reasoning capabilities. Specifically, in terms of data, we constructed two datasets with RAG: Forensic Knowledge Database (FKD) for DFD knowledge annotation, and Forensic Chain-of-Thought Dataset (F-CoT), for critical CoT construction. In terms of model training, we adopt a three-stage training method (Alignment->SFT->GRPO) to gradually cultivate the critical reasoning ability of the MLLM. In terms of performance, VRAG-DFD achieved SOTA and competitive performance on DFD generalization testing.

Figures & tables

Figure 1: Comparison between the previous method and the proposed VRAG-DFD framework. Previous methods predicted based on a single image and prompt with severe hallucinations. The proposed VRAG-DFD effectively reduce hallucinations by constructing highly relevant references through RAG and making decisions through critical reasoning. (Faces are selected from publicly available FF++ [ 40 ] dataset.)

Figure 2: The anotation phases of the Forensic Knowledge Database (FKD) and Forensic Chain-of-Thought (F-CoT) Dataset. (Faces are selected from FF++ [ 40 ] dataset.)

Figure 3: Framework of the proposed VRAG-DFD. Firstly, the dynamic forensic retriever captures top- $k$ samples from the multimodal FKD database. Afterwards, MLLM conducts a critical reasoning process ( e.g. , ①-②-③) based on the text information retrieved by RAG , and ultimately outputs the final conclusion ( e.g. , ④). (Faces are selected from FF++ [ 40 ] dataset.)

Figure 4: Training stages of VRAG-DFD. Stage-1 aims to equip the MLLM with basic forgery artifacts recognition capabilities. Stage-2 aims to guide the model to predict in the form of F-CoT. Stage-3 adopts GRPO with detailed reward function designs to encourage the MLLM to have the capabilities to critical reasoning given top- $k$ RAG evidence. (Faces are selected from FF++ [ 40 ] dataset.)

Method	Venue	Explainable	Research Topic	CDF-v1	CDF-v2	DFDC	FFIW	WDF
F 3 -Net [ 36 ]	ECCV 2020	✗	Frequency + CNN	81.11	77.92	67.35	70.11	72.80
LTW [ 43 ]	AAAI 2021	✗	Meta Training	-	77.14	69.00	76.63	-
SPSL [ 28 ]	CVPR 2021	✗	Phase Spectrum	-	79.90	77.00	79.40	70.20
SRM [ 30 ]	CVPR 2021	✗	Attention + FPN	-	84.00	69.50	80.60	72.20
PCL+I2G [ 54 ]	ICCV 2021	✗	Blending	-	90.03	67.52	-	-
SBI [ 41 ]	CVPR 2022	✗	Blending	93.44	93.18	72.42	84.83	-

Table 1: Cross-dataset performance comparison with SOTA face forgery detection methods on CDF-v1 [ 25 ] , CDF-v2 [ 25 ] , DFDC [ 8 ] , FFIW [ 58 ] , and WDF [ 60 ] . All methods are trained on FF++ [ 40 ] . The best results are shown in bold , and the second results are underlined . The Video-level AUC metric is adopted for comparison.

Stage	Objective	Epoch	lr	Batch	Param.
Stage-1	Alignment	3	$5\times 10^{-5}$	512	-
Stage-2	SFT	2	$3\times 10^{-5}$	64	-
Stage-3	GRPO	1	$1\times 10^{-6}$	32	$\beta=0.001$

Table 2: Training configurations of three training stages.

Metrics	Test Set AUC
Metrics	CDF1	CDF2	DFDC	FFIW	WDF
w/o RAG Module	91.13	87.85	72.42	78.62	71.62
VRAG-DFD	99.60	95.97	81.82	93.49	88.96

Table 3: Ablation study of RAG (AUC %). w/o RAG removes the forensic retriever and all associated critical reasoning components.

Metrics	Test Set AUC
Metrics	CDF1	CDF2	DFDC	FFIW	WDF
Stage-1	96.77	94.03	78.54	90.89	87.63
Stage-1&2	91.53	95.8	81.58	92.37	89.13
VRAG-DFD	99.60	95.97	81.82	93.49	88.96

Table 4: Ablation study of the three-stage training pipeline. We report AUC (%) on five generalization benchmarks.

Retrievers	Test Set AUC					Avg.
Retrievers	CDF1	CDF2	DFDC	FFIW	WDF	Avg.
Effort [ 49 ]	99.60	95.97	81.82	93.49	88.96	91.62
CLIP-LoRA [ 17 ]	100.00	95.77	82.47	94.00	88.42	92.13

Table 5: Ablation study of retriever (AUC %). ‘CLIP-LoRA’ replaces Effort [ 49 ] with a standard CLIP model fine-tuned by LoRA.

Model	GPT-4o	Gemini2.5 Pro	Avg.
GPT-4o	4.60	3.25	3.93
Gemini2.5 Pro	7.31	7.02	7.16
VRAG-DFD	7.55	7.78	7.66

Table 6: Explanation quality evaluation. GPT-4o and Gemini 2.5 Pro serve as evaluators. VRAG-DFD achieves the highest average score.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Metrics	Test Set AUC					Avg.
Metrics	CDF1	CDF2	DFDC	FFIW	WDF	Avg.
DD-VQA [ 53 ]	96.17	94.83	79.48	92.50	85.88	89.77
FKD (Ours)	99.60	95.97	81.82	93.49	88.96	91.62

Appendix

Table 7: Data Quality Comparison between FKD and DDVQA. Replacing FKD with DDVQA leads to a significant performance drop, indicating that forensic-grade knowledge is essential for effective reasoning.

Stage	Config	Test Set AUC					$\Delta$
Stage	Config	CDF1	CDF2	DFDC	FFIW	WDF	$\Delta$
Stage-1	w/o RAG	89.72	86.30	73.31	78.59	78.73	+8.16
Stage-1	w/ RAG	96.77	94.03	78.54	90.89	87.63	+8.16
Stage-2	w/o RAG	91.33	86.96	71.57	76.43	70.05	+10.66
Stage-2	w/ RAG	91.53	95.80	81.58	92.37	89.13	+10.66
Stage-3	w/o RAG	91.53	86.98	72.05	76.51	70.04	+12.2

Appendix

Table 8: Ablation study on the impact of the RAG module across three training stages. We report the AUC (%) on five generalization benchmarks. The column ‘

\Delta

Avg.’ highlights the performance gain attributable to the RAG module within each stage. Crucially, we observe that RAG provides substantial improvements consistently across all stages , demonstrating that dynamic retrieval of expert knowledge is fundamental to bridging the generalization gap, even in the early visual alignment phase.

Metrics	Test Set AUC
Metrics	CDF1	CDF2	DFDC	FFIW	WDF
Qwen2.5VL-7B	66.00	73.35	56.82	71.91	66.82
Qwen2.5VL-7B + ours	99.60	95.97	81.82	93.49	88.96
Intern3VL-8B	66.33	72.06	57.19	70.86	65.75
Intern3VL-8B + ours	99.19	95.09	80.16	92.45	90.00

Appendix

Table 9: Effectiveness across different base MLLMs (AUC %). We show the performance of each model before (zero-shot) and after (’+ VRAG-DFD (Ours)’) applying our three-stage VRAG-DFD pipeline.

	Retrieval		Inference		Total
Metric	GFLOPs	Ratio	GFLOPs	Ratio	GFLOPs	Ratio
Value	$\sim$ 81	0.35%	$\sim$ 22,760	99.65%	$\sim$ 22,841	100.00%

Appendix

Table 10: Computational complexity analysis of VRAG-DFD.

Samp. Strategy	Total	Test Set AUC					Avg.
Samp. Strategy	Total	CDF1	CDF2	DFDC	FFIW	WDF	Avg.
Sparse (8 fms)	$\sim$ 28k	92.14	76.85	70.69	70.40	77.95	77.61
Dense (All fms)	$\sim$ 80k	89.72	86.30	73.71	78.59	78.73	81.41
Gain ( $\Delta$ )	-	-2.42	+9.45	+3.02	+8.19	+0.78	+3.80

Appendix

Table 11: Ablation study on frame sampling density in Stage-1. We compare sparse sampling (8 frames/video) against dense sampling (all frames). The results demonstrate a substantial performance leap in the average AUC (+3.80%), confirming that dense visual supervision is indispensable for learning robust forgery features across diverse datasets.

Metric	Test Set AUC					Weigh. Avg.
Metric	CDF-v1	CDF-v2	DFDC	FFIW	WDF	Weigh. Avg.
# Adv. Samp.	34	429	534	1,374	5,226	7,597
# VRAG_DFD Cor.	32	417	520	1,345	5,048	7,362
Robust. Rate (%)	94.12	97.20	97.38	97.89	96.59	96.91

Appendix

Table 12: Robustness against misleading retrieval evidence. We evaluate a subset of adversarial samples where the initial visual perception is correct (

s_{1}

=Correct), but the RAG evidence is misleading (

RAG

=False). The results demonstrate that VRAG-DFD maintains an exceptionally high Robustness Rate (

>96\%

on avg.), effectively rejecting misleading external noise.

Panel of Figure 5 — Figure 5: Qualitative analysis of VRAG-DFD (real case 1).

Panel of Figure 7 — Figure 7: Qualitative analysis of VRAG-DFD (fake case 1).

Multimodal Deepfake Detection

See all 27 papers

VIGIL: Part-Grounded Structured Reasoning for Generalizable Deepfake Detection
Mar 23, 2026Xinghan Li, Junhao Xu, Jingjing ChenExplainable Deepfake Detection Deepfake Detection
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China.
PATE-Forensics: Perception-as-Tool for Explainable Deepfake Forensics with General-Purpose MLLMs
Aug 19, 2026Yaqi Li, Jielun Peng, Yabin Wang +2Explainable Deepfake Detection Digital Image Forensics
Harbin Institute of Technology
Deepfake Forensics Adapter: A Dual-Stream Network for Generalizable Deepfake Detection
Mar 2, 2026Jianfeng Liao, Yichen Wei, Raymond Chan Ching Bon +3Deepfake Detection Digital Image Forensics
Shenzhen Technology University, Guangdong, China · Singapore Institute of Technology, Singapore, Singapore · The University of Hong Kong, Hong Kong, China +1

Mar 23, 2026J/K move · Enter open · S save

VIGIL: Part-Grounded Structured Reasoning for Generalizable Deepfake Detection

Xinghan Li, Junhao Xu, Jingjing Chen

College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China.

Explainable Deepfake Detection Deepfake Detection VLM Reasoning Multimodal Deepfake Detection

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) offer a promising path toward interpretable deepfake detection by generating textual explanations. However, the reasoning process of current MLLM-based methods combines evidence generation and manipulation localization into a unified step. This combination blurs the boundary between faithful observations and hallucinated explanations, leading to unreliable conclusions. Building on this, we present VIGIL, a part-centric structured forensic framework inspired by expert forensic practice through a plan-then-examine pipeline: the model first plans which facial parts warrant inspection based on global visual cues, then examines each part with independently sourced forensic evidence. A stage-gated injection mechanism delivers part-level forensic evidence only during examination, ensuring that part selection remains driven by the model's own perception rather than biased by external signals. We further propose a progressive three-stage training paradigm whose reinforcement learning stage employs part-aware rewards to enforce anatomical validity and evidence--conclusion coherence. To enable rigorous generalizability evaluation, we construct OmniFake, a hierarchical 5-Level benchmark where the model, trained on only three foundational generators, is progressively tested up to in-the-wild social-media data. Extensive experiments on OmniFake and cross-dataset evaluations demonstrate that VIGIL consistently outperforms both expert detectors and concurrent MLLM-based methods across all generalizability levels.

Abstract

Multimodal Deepfake Detection

Explore similar work

VIGIL: Part-Grounded Structured Reasoning for Generalizable Deepfake Detection

PATE-Forensics: Perception-as-Tool for Explainable Deepfake Forensics with General-Purpose MLLMs

Deepfake Forensics Adapter: A Dual-Stream Network for Generalizable Deepfake Detection