Unleashing Vision-Language Semantics for Deepfake Video Detection

Jiawen Zhu; Yunqi Miao; Xueyi Zhang; Jiankang Deng; Guansong Pang

cs.CVMar 25, 2026

Unleashing Vision-Language Semantics for Deepfake Video Detection

Authors: Jiawen Zhu, Yunqi Miao, Xueyi Zhang, Jiankang Deng, Guansong Pang

Organizations: Singapore Management University, Singapore · The University of Warwick, UK · Nanyang Technological University, Singapore · Imperial College London, UK

Abstract

Recent Deepfake Video Detection (DFD) studies have demonstrated that pre-trained Vision-Language Models (VLMs) such as CLIP exhibit strong generalization capabilities in detecting artifacts across different identities. However, existing approaches focus on leveraging visual features only, overlooking their most distinctive strength -- the rich vision-language semantics embedded in the latent space. We propose VLAForge, a novel DFD framework that unleashes the potential of such cross-modal semantics to enhance model's discriminability in deepfake detection. This work i) enhances the visual perception of VLM through a ForgePerceiver, which acts as an independent learner to capture diverse, subtle forgery cues both granularly and holistically, while preserving the pretrained Vision-Language Alignment (VLA) knowledge, and ii) provides a complementary discriminative cue -- Identity-Aware VLA score, derived by coupling cross-modal semantics with the forgery cues learned by ForgePerceiver. Notably, the VLA score is augmented by an identity prior-informed text prompting to capture authenticity cues tailored to each identity, thereby enabling more discriminative cross-modal semantics. Comprehensive experiments on video DFD benchmarks, including classical face-swapping forgeries and recent full-face generation forgeries, demonstrate that our VLAForge substantially outperforms state-of-the-art methods at both frame and video levels. Code is available at https://github.com/mala-lab/VLAForge.

Figures & tables

Figure 1: Visualization of (a) visual attention map of CLIP, (b) forgery localization map of ForgePerceiver, and (c) VLA attention map. Without proper adaptation, CLIP focuses on task-irrelevant visual cues. ForgePerceiver improves this case by highlighting potential forgery areas, but it provides coarse spatial guidance only. Augmented by discriminative identity priors, the VLA attention map offers more fine-grained, stronger forgery indication.

Figure 2: Overview of VLAForge . It exploit the potential of VLMs in deepfake detection by i) ForgePerceiver, which acts as an independent learner to capture diverse, subtle forgery cues granularly and holistically; and ii) Identity-Aware VLA Scoring, which is driven by identity prior-informed text prompting and its enriched cross-modal semantics coupled with the visual forgery cues from ForgePerceiver.

Frame-level AUROC						Video-level AUROC
Method	Venue	CDF-v1	CDF-v2	DFDC	DFD	Method	Venue	CDF-v2	DFDC	DFD
Xception [ 34 ]	ICCV’19	77.9	73.7	70.8	81.6	TALL [ 42 ]	ICCV’23	83.1	69.3	83.3
EfficientB4 [ 27 ]	ICML‘19	79.1	74.9	69.6	81.5	SeeABLE [ 20 ]	ICCV’23	87.3	75.9	-
X-ray [ 21 ]	CVPR’20	70.9	67.9	63.3	76.6	IID [ 18 ]	CVPR’23	83.8	70.0	93.9
FFD [ 10 ]	CVPR’20	78.4	74.4	70.3	80.2	SFDG [ 40 ]	CVPR’23	75.8	73.6	88.0
SPSL [ 26 ]	CVPR’21	81.5	76.5	70.4	81.2	CADDM [ 12 ]	CVPR’23	93.9	73.9	-

Table 1: AUROC results of frame-level and video-level deepfake detection. † indicates results reproduced by us. The best and second-best results are respectively highlighed in blue and yellow .

Setting	Method	VQGAN	StyleGAN	SiT	DiT	PixArt
F-level	RepDFD	80.5	82.2	58.2	58.7	88.5
	ForAda	93.9	92.5	69.0	62.0	96.5
	Ours	98.4	98.0	77.4	70.7	97.2
V-level	RepDFD	85.0	86.3	69.3	58.9	94.8
	ForAda	98.1	97.9	76.0	67.1	98.5
	Ours	99.7	99.7	85.9	80.3	99.5

Table 2: AUROC results on frame (F)- and video (V)-level detection of GAN- and diffusion-generated full-face forgeries.

Setting	Model	CDF-v2	DFDC	DFD	VQGAN	SiT
F-level	Base	58.3	64.0	77.5	74.8	52.9
	+ T1	76.3	76.0	74.6	89.7	69.3
	+ T2	82.3	80.9	87.4	95.1	74.6
	+ T3	90.8	86.5	92.8	97.6	76.8
	+ T4	91.2	87.0	93.6	98.4	77.4
V-level	Base	60.3	64.6	83.9	79.4	59.2

Table 3: Frame (F)- and video (V)-level module ablation results.

Setting	$\mathcal{L}_{VLA}$	$\mathcal{L}_{orth}$	CDF-v2	DFDC	DFD	VQGAN	SiT
F-level	$\times$	$\times$	88.4	84.0	91.0	96.2	74.8
	$\times$	✓	89.1	84.7	92.1	97.2	75.0
	✓	$\times$	91.0	86.9	92.8	97.3	76.4
	✓	✓	91.2	87.0	93.6	98.4	77.4
V-level	$\times$	$\times$	93.6	86.6	95.1	97.8	85.0
	$\times$	✓	93.9	87.3	96.1	99.2	85.1

Table 4: Frame (F)- and video (V)-level loss ablation results.

Figure 3: Attention visualization of forgery faces produced by different models: (a) Attention from original CLIP; (b) Attention of forgery-aware masks from ForgePrecever.

Figure 4: Visualization of VLA attention maps with (w.) and without (w/o.) injecting identity prior into text prompts.

Figure 5: Frame-level and Video-level AUROC based on different value of $q$ ( Left ) and $\alpha$ ( Right ).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Sythesis Methods	Real	Fake	Total
FaceForensics++	4	1000	4000	5000
CelebDF v1	1	408	795	1203
CelebDF v2	1	590	5634	6229
DFDC	8	23654	104500	128154
DFD	5	363	3000	3363

Appendix

Table 5: Data statistics of face-swapping forgery datasets.

Dataset	Synthesis Type	Real	Fake	Total
VQGAN	GAN based	590	5634	6229
StyleGAN-XL	GAN based	590	5634	6229
SiT-XL/2	Latent Diffusion	590	5634	6229
DiT	Latent Diffusion	590	5634	6229
PixArt	Latent Diffusion	590	5634	6229

Appendix

Table 6: Data statistics of full-face synthesized data generated based on GAN and Diffusion-based methods.

Methods	Param.	Training (ms)	Inference (ms)	CDF-v2	DFDC	AVG
DCL	19.35M	-	-	82.3	76.7	79.5
FTCN	26.6M	-	-	86.9	74.0	80.5
CFM	25.37M	-	-	89.7	80.2	85.0
RepDFD	0.078M	1832.4±7.5	42.3±0.6	89.9	81.0	85.5
ForAda	5.7M	933.3±1.4	21.2±0.2	95.7	87.2	91.5
Ours	3.28 M	1410.0±2.2	41.7±0.9	96.8	89.6	93.2

Appendix

Table 7: Model complexity analysis in Video-level AUROC.

	Method	CDF-v2	DFDC	DFD	VQGAN	SiT
F-level	LLaVA	89.8	85.8	92.1	98.1	76.6
	LLM-Prompt	89.5	86.3	92.7	97.5	75.1
	Arc2Face-ID	90.6	86.4	92.5	98.0	76.9
	Learn-Prompt	83.8	82.3	90.1	96.6	76.4
	$w/o$ Real/Fake	89.5	86.3	92.7	97.5	75.1
	VLAForge	91.2	87.0	93.6	98.4	77.4

Appendix

Table 8: Frame (F)- and video (V)-level AUROC using different prompt variants.

Figure 6: T-sNE visualization comparison of the DFD features between visual-only model (‘ T2 ’ in Table 3) and complete VLAForge (‘ T4 ’ in Table 3).

Figure 7: More visualization of VLA attention maps with (w.) and without (w/o.) injecting identity prior into text prompts.).

	SEED	CDF-v2	DFDC	DFD	VQGAN	SiT-XL/2
F-level	1024	91.2	87.0	93.6	98.3	77.4
	0000	91.4	87.9	92.8	98.6	76.9
	1111	90.7	86.7	93.2	97.9	77.6
	AVG	91.1	87.2	93.2	98.3	77.3
V-level	1024	96.8	89.6	97.2	99.7	85.9
	0000	97.0	89.9	96.4	99.6	85.4

Appendix

Table 9: Frame (F)- and video (V)-level AUROC using different random seeds.

Explore similar work

Beyond Static Artifacts: A Forensic Benchmark for Video Deepfake Reasoning in Vision Language Models
Feb 25, 2026Zheyuan Gu, Qingsong Zhao, Yusong Wang +6Deepfake Detection Video-Language Model Evaluation
Institute of Artificial Intelligence, China Telecom (TeleAI) · Peking University · Fudan University
Deepfake Forensics Adapter: A Dual-Stream Network for Generalizable Deepfake Detection
Mar 2, 2026Jianfeng Liao, Yichen Wei, Raymond Chan Ching Bon +3Deepfake Detection Digital Image Forensics
Shenzhen Technology University, Guangdong, China · Singapore Institute of Technology, Singapore, Singapore · The University of Hong Kong, Hong Kong, China +1
CAM-VFD: Cross-Attention Multimodal Video Forgery Detection
May 16, 2026Hoda Osama Elkhodary, Sherin Mostafa Youssef, Marwa Elshenawy +1Multimodal Robustness Deepfake Detection
Computer Engineering Department, College of Engineering and Technology, Arab Academy for Science, Technology and Maritime Transport, Alexandria, Egypt

Feb 25, 2026J/K move · Enter open · S save

Beyond Static Artifacts: A Forensic Benchmark for Video Deepfake Reasoning in Vision Language Models

Zheyuan Gu, Qingsong Zhao, Yusong Wang, Zhaohong Huang+5

Institute of Artificial Intelligence, China Telecom (TeleAI) · Peking University · Fudan University

Deepfake Detection Video-Language Model Evaluation VLM Reasoning Temporal Video Grounding

Open paper AskPDF arXiv

Current Vision-Language Models (VLMs) for deepfake detection excel at identifying spatial artifacts but overlook a critical dimension: temporal inconsistencies in video forgeries. Adapting VLMs to reason about these dynamic cues remains a distinct challenge. To bridge this gap, we propose Forensic Answer-Questioning (FAQ), a large-scale benchmark that formulates temporal deepfake analysis as a multiple-choice task. FAQ introduces a three-level hierarchy to progressively evaluate and equip VLMs with forensic capabilities: (1) Facial Perception, testing the ability to identify static visual artifacts; (2) Temporal Deepfake Grounding, requiring the localization of dynamic forgery artifacts across frames; and (3) Forensic Reasoning, challenging models to synthesize evidence for final authenticity verdicts. We evaluate a range of VLMs on FAQ and generate a corresponding instruction-tuning set, FAQ-IT. Extensive experiments show that models fine-tuned on FAQ-IT achieve advanced performance on both in-domain and cross-dataset detection benchmarks. Ablation studies further validate the impact of our key design choices, confirming that FAQ is the driving force behind the temporal reasoning capabilities of these VLMs.

Abstract

Explore similar work

Beyond Static Artifacts: A Forensic Benchmark for Video Deepfake Reasoning in Vision Language Models

Deepfake Forensics Adapter: A Dual-Stream Network for Generalizable Deepfake Detection

CAM-VFD: Cross-Attention Multimodal Video Forgery Detection