X-AVDT: Audio-Visual Cross-Attention for Robust Deepfake Detection

Youngseo Kim; Kwan Yun; Seokhyeon Hong; Sihun Cha; Colette Suhjung Koo; Junyong Noh

cs.CVMar 9, 2026

X-AVDT: Audio-Visual Cross-Attention for Robust Deepfake Detection

Authors: Youngseo Kim, Kwan Yun, Seokhyeon Hong, Sihun Cha, Colette Suhjung Koo, Junyong Noh

Organizations: Visual Media Lab, KAIST

Abstract

The surge of highly realistic synthetic videos produced by contemporary generative systems has significantly increased the risk of malicious use, challenging both humans and existing detectors. Against this backdrop, we take a generator-side view and observe that internal cross-attention mechanisms in these models encode fine-grained speech-motion alignment, offering useful correspondence cues for forgery detection. Building on this insight, we propose X-AVDT, a robust and generalizable deepfake detector that probes generator-internal audio-visual signals accessed via DDIM inversion to expose these cues. X-AVDT extracts two complementary signals: (i) a video composite capturing inversion-induced discrepancies, and (ii) an audio-visual cross-attention feature reflecting modality alignment enforced during generation. To enable faithful cross-generator evaluation, we further introduce MMDF, a new multimodal deepfake dataset spanning diverse manipulation types and rapidly evolving synthesis paradigms, including GANs, diffusion, and flow-matching. Extensive experiments demonstrate that X-AVDT achieves leading performance on MMDF and generalizes strongly to external benchmarks and unseen generators, outperforming existing methods with accuracy improved by 13.1%. Our findings highlight the importance of leveraging internal audio-visual consistency cues for robustness to future generators in deepfake detection.

Figures & tables

Figure 1: Temporally averaged cross-attention maps. For each video, we extract audio–visual cross-attention during DDIM inversion and average the maps over all frames to obtain a single heatmap. Real vs. fake samples exhibit consistent disparities.

Dataset	Modality	Method			Model
Dataset	Modality	FS	RE	TH	GAN	DF	FM
Celeb-DF [ 48 ]	V	✓	✗	✗	✓	✗	✗
DF-Platter [ 60 ]	V	✓	✓	✗	✓	✗	✗
KoDF [ 40 ]	AV	✓	✓	✓	✓	✗	✗
FaceForensics++ [ 71 ]	AV	✓	✓	✗	✓	✗	✗
DFDC [ 21 ]	AV	✓	✗	✗	✓	✗	✗

Table 1: Comparison of deepfake video datasets. FS:Face swapping, RE: Self-reenactment, TH: Talking-head generation, DF: Diffusion model, FM: Flow-matching.

Figure 2: Input representations with complementary features. (a) video composite $\boldsymbol{\phi}$ is obtained from video $x$ and audio $c$ by running DDIM inversion and reconstruction, decoding both the noisy and clean latents, and computing the residual. We then concatenate four components channel-wise: $x$ , $D(\hat{z}_{T})$ , $D(\hat{z}_{0})$ , and $\lvert x-D(\hat{z}_{0})\rvert$ . (b) AV cross-attention feature $\boldsymbol{\psi}$ is extracted during DDIM inversion from the diffusion U-Net and summarized as a frame-aligned tensor. These complementary cues (a) and (b) capture appearance information and modality alignment, respectively. For clarity, all visual elements shown ( $D(\hat{z}_{T})$ , $D(\hat{z}_{0})$ , and $\lvert x-D(\hat{z}_{0})\rvert$ ) are decoded images.

Figure 3: The overall framework of X-AVDT. From each audio-visual pair, we form two inputs $\boldsymbol{\phi}$ and $\boldsymbol{\psi}$ . Two 3D encoders map them to features that are concatenated and passed through the Feature Fusion Decoder to produce a fused feature. A classification head outputs the real/fake score, while an embedding head is trained with a triplet objective to improve robustness.

Split	Generator	Model	Method	#Clips (real/fake)
Train	Hallo2 [ 17 ]	Diffusion	Talking-Head	4k/4k
	LivePortrait [ 29 ]	GAN	Self-Reenactment	4k/4k
	FaceAdapter [ 32 ]	Diffusion	Face Swapping	4k/4k
Test	HunyuanAvatar [ 13 ]	Flow Matching	Talking-Head	0.8k/0.8k
	MegActor- $\Sigma$ [ 91 ]	Diffusion Transformer	Self-Reenactment	0.8k/0.8k
	AniPortrait [ 87 ]	Diffusion	Talking-Head	0.8k/0.8k

Table 2: Composition of the dataset. This coverage better reflects current synthesis trends than outdated datasets that focus primarily on GANs, and permits assessing cross-generator generalization.

Dataset	Sync-C $\uparrow$	Sync-D $\downarrow$	LPIPS $\downarrow$	FVD $\downarrow$	HFAR $\uparrow$
FaceForensics++ [ 71 ]	3.32	11.06	0.27	370.23	0.22
FakeAVCeleb [ 38 ]	5.87	8.38	0.19	170.61	0.34
MMDF ( Ours )	7.36	7.35	0.07	121.39	0.41

Table 3: Audio-visual quality of manipulated videos. Sync-C, Sync-D, LPIPS, and FVD are computational metrics, whereas HFAR denotes the human false acceptance rate measured in a user study.

	AniPortrait [ 87 ]				MegActor- $\Sigma$ [ 91 ]				HunyuanAvatar [ 13 ]				Average
Model	AUROC	AP	Acc@EER	Acc	AUROC	AP	Acc@EER	Acc	AUROC	AP	Acc@EER	Acc	AUROC	AP	Acc@EER	Acc
Official-pretrained
LipForensics [ 31 ]	79.49	81.45	73.25	74.50	72.17	73.56	65.85	67.11	63.66	56.50	59.91	60.66	74.24	74.54	71.91	72.38
RealForensics [ 30 ]	86.31	85.23	78.65	77.56	62.93	61.60	58.79	58.71	64.90	60.30	61.40	60.85	77.65	75.71	72.15	65.70
AVAD [ 23 ]	73.25	74.98	66.05	67.49	50.87	56.96	51.26	51.18	75.37	74.54	74.14	74.39	67.47	67.29	64.75	65.29
FACTOR [ 69 ]	94.21	93.54	86.29	86.55	75.44	72.83	68.82	69.33	36.69	46.18	38.27	56.15	68.78	70.85	64.46	70.68

Table 4: Quantitative comparison on the MMDF dataset. Detectors are evaluated using the official pretrained checkpoint (first panel) and after retrained on the MMDF training set (Hallo2 [ 17 ] , LivePortrait [ 29 ] , and FaceAdapter [ 32 ] ) (second panel). Best in bold ; second-best underlined . * Note: FACTOR is a zero-shot method, while AVAD and AVH-Align are unsupervised methods.

	Cross-Attention			Temporal-Attention			Spatial-Attention
Timestep $t$	AUROC	AP	Acc@EER	AUROC	AP	Acc@EER	AUROC	AP	Acc@EER
$t=24$	91.56	86.90	81.51	83.92	87.08	75.01	64.57	68.00	61.82
$t=249$	81.30	69.52	77.13	68.25	71.94	64.10	57.42	52.00	56.04
$t=499$	68.11	57.97	67.44	66.29	66.29	59.30	52.38	53.14	50.02

Table 6: Ablation results for different attention features and diffusion timesteps. As the diffusion timestep

t

grows, the latent becomes noisier and conditioning weakens. Results at

t\in\{24,249,499\}

show that cross-attention performs best, and performance degrades as

t

increases.

Figure 4: Comparison of attention features across diffusion timesteps. Red box denote the configuration used in our method.

Figure 5: Robustness against unseen corruptions. AUROC (%) across five severity levels. Per corruption scales are shown on the $x$ –axes. Average denotes the mean AUROC across all corruptions at each severity level.

Figure 6: Grad-CAM visualizations. Red activation indicates regions where our model focuses most (i.e., pixels that make a strong positive contribution to the predicted class), while cooler colors denote weak or no contribution.

Method	AUROC	AP	Acc@EER
(a) Ablation on Input Representations
w/o AV Cross-Attn ( $\boldsymbol{\psi}$ )	88.22	87.25	83.70
w/o Video Composite ( $\boldsymbol{\phi}$ )	90.21	90.57	84.32
w/o Residual ( $\lvert x-D(\hat{z}_{0})\rvert$ )	93.82	92.25	89.00
(b) Ablation on Loss Design
w/o $\mathcal{L}_{\text{tri}}$	92.64	92.26	86.32

Table 7: Ablation on input representations and loss design. (a) removing any of the input representation degrades performance,

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Epochs	1	2 (Ours)	5	10	20
AUROC	93.24	95.29	95.01	95.19	95.13

Appendix

Table A.1.3 : Effect of training epochs on X-AVDT.

FakeAVCeleb [ 38 ]			FaceForensics++ [ 71 ]
FSGAN	FaceSwap	Wav2Lip	Deepfakes	FaceSwap	Face2Face
99.73	99.79	99.92	99.62	99.24	99.63

Appendix

Table B.1 : In-domain AUROC on the benchmark dataset. Cross-manipulation generalization is evaluated by training on two manipulation methods and testing on the remaining one (e.g., train on the first two columns and test on the third columns).

	AniPortrait [ 87 ]			MegActor- $\Sigma$ [ 91 ]			HunyuanAvatar [ 13 ]			Average
Model	AUROC	AP	Acc@EER	AUROC	AP	Acc@EER	AUROC	AP	Acc@EER	AUROC	AP	Acc@EER
SpeechForensics [ 49 ]	99.99	99.99	99.88	98.69	98.82	94.46	92.12	91.98	82.90	96.93	96.93	92.41
X-AVDT ( Ours )	99.10	98.89	96.54	90.17	88.05	83.11	97.79	97.44	97.69	95.29	94.03	91.15

Appendix

Table B.2.1 : Additional quantitative comparison results on the MMDF dataset.

Dataset	AUROC	AP	Acc@EER	Acc
DeepSpeak v1.0 [ 5 ]	94.29	95.06	95.39	94.94
KoDF [ 40 ]	93.07	92.88	91.13	91.87
Deepfake-Eval2024 [ 10 ]	75.02	72.73	71.68	71.36

Appendix

Table B.2.2 : Additional quantitative results for X-AVDT.

	AniPortrait [ 87 ]			MegActor- $\Sigma$ [ 91 ]			HunyuanAvatar [ 13 ]			Average
Method	AUROC	AP	Acc@EER	AUROC	AP	Acc@EER	AUROC	AP	Acc@EER	AUROC	AP	Acc@EER
w/o Inversion	73.55	67.56	69.95	63.18	63.69	59.81	41.81	46.36	43.55	62.22	61.07	56.69
Text-driven	90.22	90.85	81.56	76.15	75.25	69.91	49.01	49.00	50.51	74.48	76.94	65.75
Audio-driven ( Ours )	96.55	98.83	90.20	89.87	89.25	83.86	97.71	97.04	90.94	94.71	95.04	88.33

Appendix

Table C.1 : Ablation on video composite

\boldsymbol{\phi}

. When training, we fix the backbone and vary only the conditioning signal used during inversion. Audio-driven setting ranked first across datasets, while removing inversion cues yielded the weakest composite.

Figure C.2.1 : Robustness to unseen corruptions. AP (%) is shown in the first row and Acc@EER (%) is shown in the second row across five severity levels. Per corruption scales are shown on the $x$ –axes. Average denotes the mean AP in the top row and Acc@EER in the bottom row, respectively, across all corruptions at each severity level.

	Audio Desynchronizaiton			Audio Codec Artifacts
Metric	-0.5 sec	0	+0.5 sec	0	8k	32k
AUROC	90.90	93.70	91.31	93.70	91.80	90.17
AP	91.10	94.30	91.74	94.30	88.80	86.51
Acc@EER	83.40	86.40	83.97	86.40	85.90	81.56

Appendix

Table C.2.2 : Robustness to unseen audio perturbations. Performance under audio desynchronization (temporal offsets) and audio codec artifacts (low-bitrate re-encoding) at varying severity levels.

Figure C.3 : Visualization of attention features across diffusion timesteps.

Figure D.1 : PCA embeddings with a shared LDA decision boundary. Embeddings are extracted from the baseline and from the independently trained X-AVDT, without any fine-tuning.

Figure D.2 : Top- $q$ Attention Mass Coverage within the Face ROI (Left) and $\Delta$ Cross-Attention Maps (Right). In the $\Delta$ maps, red indicates regions with higher fake cross-attention than real ( $\Delta>0$ ), and blue indicates the opposite ( $\Delta<0$ ).

Figure D.3 : Temporally averaged cross-attention heatmaps.

Figure E.1 : Statistics of the MMDF dataset.

Figure F : Speed-accuracy trade-off with DDIM inversion steps.

Figure D.2.1 : Qualitative visualization of the input representations $\boldsymbol{\phi}$ and $\boldsymbol{\psi}$ .

Figure D.2.2 : Qualitative visualization of the input representations $\boldsymbol{\phi}$ and $\boldsymbol{\psi}$ .

Figure D.3.1 : Qualitative comparison of real and fake videos generated by Hallo2 [ 17 ] in the MMDF.

Figure D.3.2 : Qualitative comparison of real and fake videos generated by LivePortrait [ 29 ] in the MMDF.

Figure D.3.3 : Qualitative comparison of real and fake videos generated by FaceAdater [ 32 ] in the MMDF.

Figure D.3.4 : Qualitative comparison of real and fake videos generated by HunyuanAvatar [ 13 ] in the MMDF.

Figure D.3.5 : Qualitative comparison of real and fake videos generated by MegActor- $\Sigma$ [ 91 ] in the MMDF.

Figure D.3.6 : Qualitative comparison of real and fake videos generated by AniPortrait [ 87 ] in the MMDF.

Explore similar work

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints
Apr 29, 2026Wasim Ahmad, Wei Zhang, Xuerui MaoAudio Deepfake Detection Cross-Modal Representation Learning
School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China
Leave No Stone Unturned: Uncovering Holistic Audio-Visual Intrinsic Coherence for Deepfake Detection
Mar 25, 2026Jielun Peng, Yabin Wang, Yaqi Li +2Deepfake Detection Audio Deepfake Detection
Harbin Institute of Technology
Leveraging large multimodal models for audio-video deepfake detection: a pilot study
Feb 25, 2026Songjun Cao, Yuqi Li, Yunpeng Luo +2Deepfake Detection Audio Deepfake Detection
Tencent YouTu Lab, China · Tencent Youtu Lab, China · Fudan University, China

Apr 29, 2026J/K move · Enter open · S save

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints

Wasim Ahmad, Wei Zhang, Xuerui Mao

School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China

Audio Deepfake Detection Cross-Modal Representation Learning Digital Forensics Deepfake Detection

Open paper AskPDF arXiv

Audio-visual deepfakes have reached a level of realism that makes perceptual detection unreliable, threatening media integrity and biometric security. While multimodal detection has shown promise, most approaches are binary classification tasks that often latch onto dataset-specific artifacts rather than genuine generative traces. We argue that a detector incapable of identifying how a video was forged is likely learning the wrong signal. Unlike binary detection, attribution-guided learning imposes a stronger geometric constraint on the shared embedding space, forcing the model to encode generator-specific forensic content rather than shortcuts. We propose the Attribution-Guided Multimodal Deepfake Detection (AMDD) framework, which jointly learns to detect and attribute manipulation. AMDD treats generator attribution as a structured regularization that constrains representation geometry toward forensically meaningful features. We introduce a Cross-Modal Forensic Fingerprint Consistency (CMFFC) loss to enforce alignment between generator-induced artifacts in visual and audio streams. This exploits the fact that coherent manipulation leaves correlated traces across modalities, grounded in the physical coupling between speech and facial articulation that synthetic pipelines routinely disrupt. Architecturally, we pair a ResNet50 with temporal attention for visual encoding against a pretrained ResNet18 for mel spectrograms, closing the encoder capacity gap found in prior models. On FakeAVCeleb, AMDD achieves 99.7% balanced accuracy and 99.8% AUC with 95.9% attribution accuracy. Cross-dataset evaluation on DeepfakeTIMIT, DFDM, and LAV-DF confirms that real video detection generalizes robustly, while fake detection on unseen generators remains an open challenge that we analyze in depth.

Abstract

Explore similar work

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints

Leave No Stone Unturned: Uncovering Holistic Audio-Visual Intrinsic Coherence for Deepfake Detection

Leveraging large multimodal models for audio-video deepfake detection: a pilot study