Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities

Jingtong Dou; Chuancheng Shi; Jian Wang; Fei Shen; Zhiyong Wang; Tat-Seng Chua

cs.CVApr 9, 2026

Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities

Authors: Jingtong Dou, Chuancheng Shi, Jian Wang, Fei Shen, Zhiyong Wang, Tat-Seng Chua

Organizations: The University of Sydney · Nanjing University of Posts and Telecommunications · National University of Singapore

Abstract

As generative artificial intelligence evolves, deepfake attacks have escalated from single-modality manipulations to complex, multimodal threats. Existing forensic techniques face a severe generalization bottleneck: by relying excessively on superficial, modality-specific artifacts, they neglect the shared latent forgery knowledge hidden beneath variable physical appearances. Consequently, these models suffer catastrophic performance degradation when confronted with unseen "dark modalities." To break this limitation, this paper introduces a paradigm shift that redefines multimodal forensics from conventional "feature fusion" to "modality generalization." We propose the first modality-agnostic forgery (MAF) detection framework. By explicitly decoupling modality-specific styles, MAF precisely extracts the essential, cross-modal latent forgery knowledge. Furthermore, we define two progressive dimensions to quantify model generalization: transferability toward semantically correlated modalities (Weak MAF), and robustness against completely isolated signals of "dark modality" (Strong MAF). To rigorously assess these generalization limits, we introduce the DeepModal-Bench benchmark, which integrates diverse multimodal forgery detection algorithms and adapts state-of-the-art generalized learning methods. This study not only empirically proves the existence of universal forgery traces but also achieves significant performance breakthroughs on unknown modalities via the MAF framework, offering a pioneering technical pathway for universal multimodal defense.

Figures & tables

Fig. 1: From modality-binding to modality-agnostic forgery (MAF). Unlike existing tasks that overfit to specific physical representations seen during training, our proposed MAF task focuses on extracting shared latent forgery knowledge. This modality-agnostic feature learning enables robust source-free domain generalization against unknown modalities (e.g., unseen audio), systematically tackling both Weak and Strong MAF challenges.

Fig. 2: Architecture of the MAF framework. Following training on known modalities, the universal detector $\mathcal{F}$ evaluates an unseen modality $\mathcal{M}^{K+1}$ under two settings: Weak MAF utilizes a semantically aligned perceptor, while Strong MAF employs an independent perceptor to assess forensic generalization under strict physical and semantic isolation.

Fig. 3: Comparison of feature distributions between semantic and forensics spaces. Top (Semantic Space): Forgery features exhibit strong modality dependence and lack shared fingerprints, resulting in isolated distributions across modalities. Bottom (Forensics Space): Through feature decoupling, the distributions of both known and completely unseen modalities converge, successfully achieving cross-modal universality of latent forgery knowledge.

Dataset	Video	Audio	Image	Instances	Category
LAV-DF [ 7 ]	$\checkmark$	$\checkmark$	$\checkmark$	101,675	Aligned
FakeAVCeleb [ 21 ]	$\checkmark$	$\checkmark$	$\checkmark$	20,000	Aligned
ASVspoof5 [ 55 ]	✗	$\checkmark$	✗	1,002,889	Unaligned
Celeb-DF++ [ 32 ]	$\checkmark$	✗	$\checkmark$	53,786	Unaligned

TABLE I: Overview of the experimental datasets in DeepModal-Bench. We use “

\checkmark

” and “ ✗ ” to indicate the presence or absence of observable modalities, respectively. The “Instances” reports the combined total of genuine and forged samples used for benchmarking across different semantic categories.

Fig. 4: Performance evaluation under the Weak MAF setting across three pre-trained perceptors (ImageBind, LanguageBind, and UniBind). Each bar reports the average AUC across the video, audio, and image modalities, utilizing the Oracle validation strategy for model selection. Dashed horizontal lines denote the mean performance of multi-modal learning (MML) and domain generalization (DG) methods for each corresponding perceiver.

Fig. 7: Performance evaluation under Strong MAF across three pre-trained perceptors (ImageBind, LanguageBind, and UniBind). The bar chart illustrates the average accuracy of various detection algorithms across different modalities. The dashed lines represent the overall performance benchmarks for multi-modal learning (MML) and domain generalization (DG) methods, respectively, under each corresponding perceiver.

Method	LAV-DF	FakeAVCeleb	Celeb+ASV.
Single-modality only	51.7 (-8.0)	52.4 (-8.5)	49.9 (-7.9)
Random initialization	53.4 (-6.3)	55.9 (-5.0)	52.1 (-5.7)
Strong MAF (Ours)	59.7	60.9	57.8

TABLE II: Ablation study on the source of cross-modal forensic knowledge under the Strong MAF setting. We compare our complete framework against two degraded baselines to isolate the contribution of joint multimodal learning.

Fig. 10: Cross-modal distribution consistency analysis. The heatmaps illustrate the Kullback-Leibler (KL) divergence between different modalities in the semantic space (left) and the proposed forensic space (right).

Feature Space	Label	Audio	Image	Video
Semantic Space (IB)	Real	105	192	199
Semantic Space (IB)	Fake	102	199	199
	Real	3 ( $\downarrow$ 97%)	50 ( $\downarrow$ 74%)	40 ( $\downarrow$ 80%)
Forensic Space (MAF)	Fake	2 ( $\downarrow$ 98%)	44 ( $\downarrow$ 78%)	32 ( $\downarrow$ 84%)

TABLE III: Intrinsic dimensionality analysis. We report the PCA effective dimensionality (

k_{95}

) of real and fake features in both the semantic and forensic spaces across three modalities. The sharp drop in

k_{95}

empirically validates that forgery features converge into a highly compact, low-dimensional subspace.

Fig. 11: Cross-modal neuron co-activation. Visualizing the top-64 activated MLP neurons per modality reveals peripheral modality-specific clusters and a robust central core of 16 neurons co-activated across all media, mechanistically confirming the extraction of universal forgery fingerprints.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Algorithm	Hyperparameter	Default Value	Distribution
		batchsize	32	$2^{\mathcal{U}(3,5.5)}$
MML	All	lr	0.001	$10^{\mathcal{U}(-4,-2)}$
		momentum	0.9	$\mathcal{U}(0.85,0.95)$
		weight decay	0.0001	$10^{\mathcal{U}(-6,-2)}$
		patience	70	$\mathcal{U}(60,80)$
	OGM [ 39 ]	alpha	0.1	$\mathcal{U}(0.1,0.3)$

Appendix

TABLE IV: Hyperparameter Search Space and Default Values

Perceptor	Method		LAV-DF [ 7 ]				Fakeavcele [ 21 ]				Cele+Asv [ 55 , 32 ]
Perceptor	Method		Vid	Aud	Img	Avg	Vid	Aud	Img	Avg	Vid	Aud	Img	Avg
ImageBind [ 15 ]	MML	Concat	$63.12_{\pm 0.41}$	$62.32_{\pm 0.22}$	$63.42_{\pm 0.88}$	62.95	$64.59_{\pm 0.15}$	$67.05_{\pm 1.34}$	$69.42_{\pm 0.55}$	67.02	$58.14_{\pm 0.72}$	$59.10_{\pm 0.09}$	$59.27_{\pm 0.43}$	58.84
		OGM [ 39 ]	$60.66_{\pm 0.12}$	$61.48_{\pm 0.64}$	$59.92_{\pm 0.37}$	60.69	$62.08_{\pm 0.91}$	$64.47_{\pm 0.56}$	$63.42_{\pm 0.82}$	63.32	$56.34_{\pm 0.28}$	$57.17_{\pm 0.11}$	$57.94_{\pm 0.74}$	57.15
		DLMG [ 61 ]	$61.64_{\pm 0.53}$	$61.25_{\pm 0.49}$	$59.83_{\pm 1.12}$	60.91	$64.75_{\pm 0.04}$	$64.87_{\pm 0.32}$	$63.24_{\pm 0.71}$	64.29	$58.56_{\pm 0.44}$	$57.34_{\pm 0.85}$	$58.67_{\pm 0.21}$	58.19
	DG	ERM	$70.02_{\pm 0.66}$	$66.86_{\pm 0.19}$	$67.61_{\pm 0.33}$	68.16	$62.00_{\pm 0.52}$	$64.60_{\pm 0.78}$	$77.12_{\pm 0.25}$	67.91	$60.12_{\pm 0.91}$	$62.02_{\pm 0.44}$	$64.22_{\pm 0.12}$	62.12
		IRM [ 4 ]	$68.15_{\pm 0.42}$	$66.45_{\pm 0.08}$	$70.05_{\pm 1.67}$	68.22	$63.40_{\pm 0.62}$	$68.35_{\pm 0.55}$	$74.15_{\pm 0.33}$	68.63	$60.15_{\pm 0.27}$	$61.05_{\pm 0.81}$	$62.75_{\pm 0.59}$	61.32

Appendix

TABLE V: Classification performance (mean

\pm

std) under the Weak MAF setting, using test-modality validation set (oracle) for model selection.

Perceptor	Method		LAV-DF [ 7 ]				Fakeavcele [ 21 ]				Cele+Asv [ 55 , 32 ]
Perceptor	Method		Vid	Aud	Img	Avg	Vid	Aud	Img	Avg	Vid	Aud	Img	Avg
ImageBind [ 15 ]	MML	Concat	$61.20_{\pm 1.93}$	$60.65_{\pm 1.22}$	$60.93_{\pm 1.46}$	60.93	$63.38_{\pm 0.36}$	$64.78_{\pm 2.19}$	$64.46_{\pm 1.33}$	64.21	$58.44_{\pm 1.89}$	$57.82_{\pm 1.42}$	$57.40_{\pm 1.94}$	57.89
		OGM [ 39 ]	$60.00_{\pm 0.46}$	$60.82_{\pm 0.34}$	$61.24_{\pm 0.35}$	60.69	$61.69_{\pm 0.43}$	$65.59_{\pm 0.39}$	$63.37_{\pm 0.26}$	63.55	$56.91_{\pm 1.02}$	$56.12_{\pm 1.41}$	$56.87_{\pm 0.89}$	56.63
		DLMG [ 61 ]	$61.66_{\pm 1.24}$	$59.72_{\pm 0.99}$	$60.49_{\pm 1.00}$	60.62	$64.44_{\pm 1.23}$	$63.45_{\pm 0.16}$	$63.06_{\pm 0.82}$	63.65	$57.30_{\pm 0.88}$	$58.01_{\pm 1.65}$	$57.39_{\pm 1.41}$	57.57
	DG	ERM	$71.03_{\pm 0.28}$	$66.13_{\pm 0.95}$	$68.01_{\pm 0.89}$	68.39	$62.07_{\pm 0.14}$	$63.60_{\pm 0.27}$	$72.97_{\pm 1.02}$	66.21	$58.40_{\pm 1.11}$	$61.81_{\pm 1.01}$	$62.36_{\pm 1.27}$	60.86
		IRM [ 4 ]	$67.50_{\pm 0.88}$	$66.55_{\pm 1.25}$	$68.79_{\pm 1.25}$	67.61	$62.30_{\pm 0.03}$	$64.95_{\pm 0.03}$	$71.31_{\pm 0.83}$	66.19	$60.03_{\pm 1.18}$	$61.90_{\pm 1.74}$	$61.76_{\pm 1.92}$	61.23

Appendix

TABLE VI: Classification performance (mean

\pm

std) under the Weak MAF setting, using leave-one-modality-out cross-validation for model selection.

Perceptor	Method		LAV-DF [ 7 ]				Fakeavcele [ 21 ]				Cele+Asv [ 55 , 32 ]
Perceptor	Method		Vid	Aud	Img	Avg	Vid	Aud	Img	Avg	Vid	Aud	Img	Avg
ImageBind [ 15 ]	MML	Concat	$64.21_{\pm 0.78}$	$62.66_{\pm 0.72}$	$63.51_{\pm 0.73}$	63.46	$64.50_{\pm 0.99}$	$66.57_{\pm 0.35}$	$65.60_{\pm 0.47}$	65.56	$58.71_{\pm 0.03}$	$58.96_{\pm 1.39}$	$58.86_{\pm 0.79}$	58.84
		OGM [ 39 ]	$59.88_{\pm 1.16}$	$61.23_{\pm 0.72}$	$62.02_{\pm 0.24}$	61.04	$64.00_{\pm 0.14}$	$61.59_{\pm 1.47}$	$63.70_{\pm 0.04}$	63.10	$58.13_{\pm 0.95}$	$56.91_{\pm 0.81}$	$56.32_{\pm 0.41}$	57.12
		DLMG [ 61 ]	$60.03_{\pm 0.73}$	$61.29_{\pm 0.20}$	$61.23_{\pm 0.39}$	60.85	$63.85_{\pm 0.40}$	$62.88_{\pm 0.38}$	$63.44_{\pm 0.37}$	63.39	$58.07_{\pm 0.66}$	$58.09_{\pm 0.68}$	$57.65_{\pm 0.54}$	57.94
	DG	ERM [ 50 ]	$71.35_{\pm 0.66}$	$65.85_{\pm 1.44}$	$67.81_{\pm 0.30}$	68.34	$62.83_{\pm 0.02}$	$61.08_{\pm 0.84}$	$76.01_{\pm 0.50}$	66.64	$59.91_{\pm 0.65}$	$61.96_{\pm 0.73}$	$63.71_{\pm 1.39}$	61.86
		IRM [ 4 ]	$68.64_{\pm 0.15}$	$66.89_{\pm 0.90}$	$70.84_{\pm 0.45}$	68.79	$61.72_{\pm 0.35}$	$64.63_{\pm 0.58}$	$72.44_{\pm 0.65}$	66.26	$60.49_{\pm 0.50}$	$61.60_{\pm 0.06}$	$63.04_{\pm 0.92}$	61.71

Appendix

TABLE VII: Classification performance (mean

\pm

std) under the Weak MAF setting, using training-modality validation set for model selection.

Perceptor	Method		LAV-DF [ 7 ]				Fakeavcele [ 21 ]				Cele+Asv [ 55 , 32 ]
Perceptor	Method		Vid	Aud	Img	Avg	Vid	Aud	Img	Avg	Vid	Aud	Img	Avg
ImageBind [ 15 ]	MML	Concat	$61.76_{\pm 0.37}$	$61.24_{\pm 1.48}$	$62.32_{\pm 0.29}$	61.77	$64.27_{\pm 0.63}$	$65.24_{\pm 1.14}$	$64.47_{\pm 0.85}$	64.66	$59.27_{\pm 1.73}$	$59.49_{\pm 0.42}$	$59.47_{\pm 1.26}$	59.41
		OGM [ 39 ]	$60.42_{\pm 0.51}$	$60.72_{\pm 0.14}$	$60.37_{\pm 0.76}$	60.50	$62.97_{\pm 0.92}$	$62.81_{\pm 1.68}$	$63.47_{\pm 0.39}$	63.08	$58.57_{\pm 1.05}$	$58.22_{\pm 0.83}$	$58.18_{\pm 1.44}$	58.32
		DLMG [ 61 ]	$60.35_{\pm 1.12}$	$60.71_{\pm 0.85}$	$58.77_{\pm 1.34}$	59.94	$63.30_{\pm 0.48}$	$64.56_{\pm 0.27}$	$62.65_{\pm 1.51}$	63.50	$59.32_{\pm 0.73}$	$58.84_{\pm 1.29}$	$59.17_{\pm 1.96}$	59.11
	DG	ERM	$59.52_{\pm 0.19}$	$59.14_{\pm 1.62}$	$59.11_{\pm 0.33}$	59.26	$61.79_{\pm 1.47}$	$64.03_{\pm 0.81}$	$59.07_{\pm 1.24}$	61.63	$57.83_{\pm 0.68}$	$57.10_{\pm 1.85}$	$58.31_{\pm 0.96}$	57.75
		IRM [ 4 ]	$61.46_{\pm 0.84}$	$60.30_{\pm 0.45}$	$61.16_{\pm 1.71}$	60.97	$61.23_{\pm 0.26}$	$61.66_{\pm 1.37}$	$65.15_{\pm 0.52}$	62.68	$58.84_{\pm 1.18}$	$59.00_{\pm 0.91}$	$58.74_{\pm 0.13}$	58.86

Appendix

TABLE VIII: Classification performance (mean

\pm

std) under the Strong MAF setting, using test-modality validation set (oracle) set for model selection.

Perceptor	Method		LAV-DF [ 7 ]				Fakeavcele [ 21 ]				Cele+Asv [ 55 , 32 ]
Perceptor	Method		Vid	Aud	Img	Avg	Vid	Aud	Img	Avg	Vid	Aud	Img	Avg
ImageBind [ 15 ]	MML	Concat	$59.62_{\pm 1.99}$	$59.31_{\pm 0.88}$	$59.67_{\pm 1.77}$	59.53	$63.33_{\pm 1.28}$	$64.07_{\pm 0.35}$	$61.78_{\pm 1.71}$	63.06	$59.15_{\pm 0.62}$	$59.41_{\pm 1.36}$	$60.09_{\pm 0.98}$	59.55
		OGM [ 39 ]	$59.35_{\pm 1.03}$	$59.10_{\pm 0.65}$	$60.92_{\pm 1.19}$	59.79	$62.93_{\pm 0.25}$	$61.69_{\pm 0.85}$	$63.13_{\pm 0.30}$	62.58	$57.95_{\pm 1.69}$	$59.33_{\pm 0.61}$	$58.10_{\pm 1.03}$	58.46
		DLMG [ 61 ]	$60.72_{\pm 1.67}$	$62.01_{\pm 1.78}$	$60.53_{\pm 1.93}$	61.09	$62.88_{\pm 0.14}$	$62.09_{\pm 1.94}$	$63.48_{\pm 1.28}$	62.82	$57.28_{\pm 1.22}$	$58.35_{\pm 0.51}$	$57.18_{\pm 1.72}$	57.60
	DG	ERM	$61.36_{\pm 1.88}$	$60.71_{\pm 0.45}$	$58.69_{\pm 0.95}$	60.25	$60.37_{\pm 1.39}$	$61.11_{\pm 0.42}$	$62.17_{\pm 1.84}$	61.22	$59.42_{\pm 0.93}$	$58.76_{\pm 1.28}$	$58.43_{\pm 1.20}$	58.87
		IRM [ 4 ]	$63.83_{\pm 1.13}$	$61.86_{\pm 1.42}$	$62.22_{\pm 0.12}$	62.64	$61.00_{\pm 0.43}$	$61.10_{\pm 1.10}$	$63.96_{\pm 0.30}$	62.02	$59.00_{\pm 0.29}$	$57.41_{\pm 1.47}$	$59.28_{\pm 1.30}$	58.56

Appendix

TABLE IX: Classification performance (mean

\pm

std) under the Strong MAF setting, using leave-one-modality-out cross-validation set for model selection.

Perceptor	Method		LAV-DF [ 7 ]				Fakeavcele [ 21 ]				Cele+Asv [ 55 , 32 ]
Perceptor	Method		Vid	Aud	Img	Avg	Vid	Aud	Img	Avg	Vid	Aud	Img	Avg
ImageBind [ 15 ]	MML	Concat	$61.05_{\pm 0.72}$	$59.47_{\pm 0.54}$	$60.08_{\pm 1.02}$	60.20	$63.88_{\pm 0.71}$	$62.63_{\pm 0.66}$	$64.50_{\pm 1.25}$	63.67	$59.75_{\pm 1.91}$	$58.49_{\pm 1.65}$	$58.70_{\pm 1.84}$	58.98
		OGM [ 39 ]	$61.34_{\pm 0.26}$	$61.53_{\pm 1.02}$	$58.80_{\pm 0.36}$	60.56	$63.55_{\pm 1.55}$	$64.56_{\pm 1.82}$	$63.88_{\pm 0.16}$	64.00	$56.72_{\pm 0.84}$	$58.38_{\pm 0.56}$	$58.76_{\pm 1.01}$	57.95
		DLMG [ 61 ]	$61.53_{\pm 1.09}$	$62.05_{\pm 1.35}$	$60.60_{\pm 0.29}$	61.39	$64.58_{\pm 1.31}$	$64.56_{\pm 0.19}$	$63.52_{\pm 0.22}$	64.22	$57.52_{\pm 0.69}$	$58.22_{\pm 1.25}$	$58.13_{\pm 0.84}$	57.96
	DG	ERM	$60.51_{\pm 1.12}$	$62.05_{\pm 1.61}$	$61.86_{\pm 1.00}$	61.47	$62.58_{\pm 0.97}$	$62.29_{\pm 0.34}$	$61.97_{\pm 1.69}$	62.28	$59.18_{\pm 1.15}$	$56.62_{\pm 1.42}$	$56.42_{\pm 1.52}$	57.41
		IRM [ 4 ]	$61.45_{\pm 1.94}$	$62.03_{\pm 1.90}$	$60.56_{\pm 0.99}$	61.35	$63.79_{\pm 1.72}$	$62.96_{\pm 1.09}$	$64.44_{\pm 0.22}$	63.73	$60.20_{\pm 1.34}$	$60.21_{\pm 0.53}$	$57.59_{\pm 1.99}$	59.33

Appendix

TABLE X: Classification performance (mean

\pm

std) under the Strong MAF setting, using training-modality validation set for model selection.

Multimodal Deepfake Detection

See all 29 papers

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints
Apr 29, 2026Wasim Ahmad, Wei Zhang, Xuerui MaoAudio Deepfake Detection Cross-Modal Representation Learning
School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China
Revisiting Cross-Reconstruction for Generalizable Deepfake Detection
Oct 1, 2026Bingjian Yang, Shilei Zhao, Zheng WangImage Forgery Detection Digital Image Forensics
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, China
CAM-VFD: Cross-Attention Multimodal Video Forgery Detection
May 16, 2026Hoda Osama Elkhodary, Sherin Mostafa Youssef, Marwa Elshenawy +1Multimodal Deepfake Detection Multimodal Robustness
Computer Engineering Department, College of Engineering and Technology, Arab Academy for Science, Technology and Maritime Transport, Alexandria, Egypt

Apr 29, 2026J/K move · Enter open · S save

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints

Wasim Ahmad, Wei Zhang, Xuerui Mao

School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China

Audio Deepfake Detection Cross-Modal Representation Learning Digital Forensics Deepfake Detection Multimodal Deepfake Detection Multimodal Deception Detection

Open paper AskPDF arXiv

Audio-visual deepfakes have reached a level of realism that makes perceptual detection unreliable, threatening media integrity and biometric security. While multimodal detection has shown promise, most approaches are binary classification tasks that often latch onto dataset-specific artifacts rather than genuine generative traces. We argue that a detector incapable of identifying how a video was forged is likely learning the wrong signal. Unlike binary detection, attribution-guided learning imposes a stronger geometric constraint on the shared embedding space, forcing the model to encode generator-specific forensic content rather than shortcuts. We propose the Attribution-Guided Multimodal Deepfake Detection (AMDD) framework, which jointly learns to detect and attribute manipulation. AMDD treats generator attribution as a structured regularization that constrains representation geometry toward forensically meaningful features. We introduce a Cross-Modal Forensic Fingerprint Consistency (CMFFC) loss to enforce alignment between generator-induced artifacts in visual and audio streams. This exploits the fact that coherent manipulation leaves correlated traces across modalities, grounded in the physical coupling between speech and facial articulation that synthetic pipelines routinely disrupt. Architecturally, we pair a ResNet50 with temporal attention for visual encoding against a pretrained ResNet18 for mel spectrograms, closing the encoder capacity gap found in prior models. On FakeAVCeleb, AMDD achieves 99.7% balanced accuracy and 99.8% AUC with 95.9% attribution accuracy. Cross-dataset evaluation on DeepfakeTIMIT, DFDM, and LAV-DF confirms that real video detection generalizes robustly, while fake detection on unseen generators remains an open challenge that we analyze in depth.

Abstract

Multimodal Deepfake Detection

Explore similar work

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints

Revisiting Cross-Reconstruction for Generalizable Deepfake Detection

CAM-VFD: Cross-Attention Multimodal Video Forgery Detection