The surge of highly realistic synthetic videos produced by contemporary generative systems has significantly increased the risk of malicious use, challenging both humans and existing detectors. Against this backdrop, we take a generator-side view and observe that internal cross-attention mechanisms in these models encode fine-grained speech-motion alignment, offering useful correspondence cues for forgery detection. Building on this insight, we propose X-AVDT, a robust and generalizable deepfake detector that probes generator-internal audio-visual signals accessed via DDIM inversion to expose these cues. X-AVDT extracts two complementary signals: (i) a video composite capturing inversion-induced discrepancies, and (ii) an audio-visual cross-attention feature reflecting modality alignment enforced during generation. To enable faithful cross-generator evaluation, we further introduce MMDF, a new multimodal deepfake dataset spanning diverse manipulation types and rapidly evolving synthesis paradigms, including GANs, diffusion, and flow-matching. Extensive experiments demonstrate that X-AVDT achieves leading performance on MMDF and generalizes strongly to external benchmarks and unseen generators, outperforming existing methods with accuracy improved by 13.1%. Our findings highlight the importance of leveraging internal audio-visual consistency cues for robustness to future generators in deepfake detection.
Figures & tables
Figure 1: Temporally averaged cross-attention maps. For each video, we extract audio–visual cross-attention during DDIM inversion and average the maps over all frames to obtain a single heatmap. Real vs. fake samples exhibit consistent disparities.
Dataset
Modality
Method
Model
FS
RE
TH
GAN
DF
FM
Celeb-DF [ 48 ]
V
✓
✗
✗
✓
✗
✗
DF-Platter [ 60 ]
V
✓
✓
✗
✓
✗
✗
KoDF [ 40 ]
AV
✓
✓
✓
✓
✗
✗
FaceForensics++ [ 71 ]
AV
✓
✓
✗
✓
✗
✗
DFDC [ 21 ]
AV
✓
✗
✗
✓
✗
✗
Table 1: Comparison of deepfake video datasets. FS:Face swapping, RE: Self-reenactment, TH: Talking-head generation, DF: Diffusion model, FM: Flow-matching.
Figure 2: Input representations with complementary features. (a) video composite ϕ is obtained from video x and audio c by running DDIM inversion and reconstruction, decoding both the noisy and clean latents, and computing the residual. We then concatenate four components channel-wise: x , D(z^T) , D(z^0) , and ∣x−D(z^0)∣ . (b) AV cross-attention feature ψ is extracted during DDIM inversion from the diffusion U-Net and summarized as a frame-aligned tensor. These complementary cues (a) and (b) capture appearance information and modality alignment, respectively. For clarity, all visual elements shown ( D(z^T) , D(z^0) , and ∣x−D(z^0)∣ ) are decoded images.
Figure 3: The overall framework of X-AVDT. From each audio-visual pair, we form two inputs ϕ and ψ . Two 3D encoders map them to features that are concatenated and passed through the Feature Fusion Decoder to produce a fused feature. A classification head outputs the real/fake score, while an embedding head is trained with a triplet objective to improve robustness.
Split
Generator
Model
Method
#Clips (real/fake)
Train
Hallo2 [ 17 ]
Diffusion
Talking-Head
4k/4k
LivePortrait [ 29 ]
GAN
Self-Reenactment
4k/4k
FaceAdapter [ 32 ]
Diffusion
Face Swapping
4k/4k
Test
HunyuanAvatar [ 13 ]
Flow Matching
Talking-Head
0.8k/0.8k
MegActor- Σ [ 91 ]
Diffusion Transformer
Self-Reenactment
0.8k/0.8k
AniPortrait [ 87 ]
Diffusion
Talking-Head
0.8k/0.8k
Table 2: Composition of the dataset. This coverage better reflects current synthesis trends than outdated datasets that focus primarily on GANs, and permits assessing cross-generator generalization.
Dataset
Sync-C ↑
Sync-D ↓
LPIPS ↓
FVD ↓
HFAR ↑
FaceForensics++ [ 71 ]
3.32
11.06
0.27
370.23
0.22
FakeAVCeleb [ 38 ]
5.87
8.38
0.19
170.61
0.34
MMDF ( Ours )
7.36
7.35
0.07
121.39
0.41
Table 3: Audio-visual quality of manipulated videos. Sync-C, Sync-D, LPIPS, and FVD are computational metrics, whereas HFAR denotes the human false acceptance rate measured in a user study.
AniPortrait [ 87 ]
MegActor- Σ [ 91 ]
HunyuanAvatar [ 13 ]
Average
Model
AUROC
AP
Acc@EER
Acc
AUROC
AP
Acc@EER
Acc
AUROC
AP
Acc@EER
Acc
AUROC
AP
Acc@EER
Acc
Official-pretrained
LipForensics [ 31 ]
79.49
81.45
73.25
74.50
72.17
73.56
65.85
67.11
63.66
56.50
59.91
60.66
74.24
74.54
71.91
72.38
RealForensics [ 30 ]
86.31
85.23
78.65
77.56
62.93
61.60
58.79
58.71
64.90
60.30
61.40
60.85
77.65
75.71
72.15
65.70
AVAD [ 23 ]
73.25
74.98
66.05
67.49
50.87
56.96
51.26
51.18
75.37
74.54
74.14
74.39
67.47
67.29
64.75
65.29
FACTOR [ 69 ]
94.21
93.54
86.29
86.55
75.44
72.83
68.82
69.33
36.69
46.18
38.27
56.15
68.78
70.85
64.46
70.68
Table 4: Quantitative comparison on the MMDF dataset. Detectors are evaluated using the official pretrained checkpoint (first panel) and after retrained on the MMDF training set (Hallo2 [ 17 ] , LivePortrait [ 29 ] , and FaceAdapter [ 32 ] ) (second panel). Best in bold ; second-best underlined . * Note: FACTOR is a zero-shot method, while AVAD and AVH-Align are unsupervised methods.
Cross-Attention
Temporal-Attention
Spatial-Attention
Timestep t
AUROC
AP
Acc@EER
AUROC
AP
Acc@EER
AUROC
AP
Acc@EER
t=24
91.56
86.90
81.51
83.92
87.08
75.01
64.57
68.00
61.82
t=249
81.30
69.52
77.13
68.25
71.94
64.10
57.42
52.00
56.04
t=499
68.11
57.97
67.44
66.29
66.29
59.30
52.38
53.14
50.02
Table 6: Ablation results for different attention features and diffusion timesteps. As the diffusion timestep t grows, the latent becomes noisier and conditioning weakens. Results at t∈{24,249,499} show that cross-attention performs best, and performance degrades as t increases.
Figure 4: Comparison of attention features across diffusion timesteps. Red box denote the configuration used in our method.
Figure 5: Robustness against unseen corruptions. AUROC (%) across five severity levels. Per corruption scales are shown on the x –axes. Average denotes the mean AUROC across all corruptions at each severity level.
Figure 6: Grad-CAM visualizations. Red activation indicates regions where our model focuses most (i.e., pixels that make a strong positive contribution to the predicted class), while cooler colors denote weak or no contribution.
Method
AUROC
AP
Acc@EER
(a) Ablation on Input Representations
w/o AV Cross-Attn ( ψ )
88.22
87.25
83.70
w/o Video Composite ( ϕ )
90.21
90.57
84.32
w/o Residual ( ∣x−D(z^0)∣ )
93.82
92.25
89.00
(b) Ablation on Loss Design
w/o Ltri
92.64
92.26
86.32
Table 7: Ablation on input representations and loss design. (a) removing any of the input representation degrades performance,
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Epochs
1
2 (Ours)
5
10
20
AUROC
93.24
95.29
95.01
95.19
95.13
Appendix
Table A.1.3 : Effect of training epochs on X-AVDT.
FakeAVCeleb [ 38 ]
FaceForensics++ [ 71 ]
FSGAN
FaceSwap
Wav2Lip
Deepfakes
FaceSwap
Face2Face
99.73
99.79
99.92
99.62
99.24
99.63
Appendix
Table B.1 : In-domain AUROC on the benchmark dataset. Cross-manipulation generalization is evaluated by training on two manipulation methods and testing on the remaining one (e.g., train on the first two columns and test on the third columns).
AniPortrait [ 87 ]
MegActor- Σ [ 91 ]
HunyuanAvatar [ 13 ]
Average
Model
AUROC
AP
Acc@EER
AUROC
AP
Acc@EER
AUROC
AP
Acc@EER
AUROC
AP
Acc@EER
SpeechForensics [ 49 ]
99.99
99.99
99.88
98.69
98.82
94.46
92.12
91.98
82.90
96.93
96.93
92.41
X-AVDT ( Ours )
99.10
98.89
96.54
90.17
88.05
83.11
97.79
97.44
97.69
95.29
94.03
91.15
Appendix
Table B.2.1 : Additional quantitative comparison results on the MMDF dataset.
Dataset
AUROC
AP
Acc@EER
Acc
DeepSpeak v1.0 [ 5 ]
94.29
95.06
95.39
94.94
KoDF [ 40 ]
93.07
92.88
91.13
91.87
Deepfake-Eval2024 [ 10 ]
75.02
72.73
71.68
71.36
Appendix
Table B.2.2 : Additional quantitative results for X-AVDT.
AniPortrait [ 87 ]
MegActor- Σ [ 91 ]
HunyuanAvatar [ 13 ]
Average
Method
AUROC
AP
Acc@EER
AUROC
AP
Acc@EER
AUROC
AP
Acc@EER
AUROC
AP
Acc@EER
w/o Inversion
73.55
67.56
69.95
63.18
63.69
59.81
41.81
46.36
43.55
62.22
61.07
56.69
Text-driven
90.22
90.85
81.56
76.15
75.25
69.91
49.01
49.00
50.51
74.48
76.94
65.75
Audio-driven ( Ours )
96.55
98.83
90.20
89.87
89.25
83.86
97.71
97.04
90.94
94.71
95.04
88.33
Appendix
Table C.1 : Ablation on video composite ϕ . When training, we fix the backbone and vary only the conditioning signal used during inversion. Audio-driven setting ranked first across datasets, while removing inversion cues yielded the weakest composite.
Figure C.2.1 : Robustness to unseen corruptions. AP (%) is shown in the first row and Acc@EER (%) is shown in the second row across five severity levels. Per corruption scales are shown on the x –axes. Average denotes the mean AP in the top row and Acc@EER in the bottom row, respectively, across all corruptions at each severity level.
Audio Desynchronizaiton
Audio Codec Artifacts
Metric
-0.5 sec
0
+0.5 sec
0
8k
32k
AUROC
90.90
93.70
91.31
93.70
91.80
90.17
AP
91.10
94.30
91.74
94.30
88.80
86.51
Acc@EER
83.40
86.40
83.97
86.40
85.90
81.56
Appendix
Table C.2.2 : Robustness to unseen audio perturbations. Performance under audio desynchronization (temporal offsets) and audio codec artifacts (low-bitrate re-encoding) at varying severity levels.
Figure 20
Figure C.3 : Visualization of attention features across diffusion timesteps.
Figure D.1 : PCA embeddings with a shared LDA decision boundary. Embeddings are extracted from the baseline and from the independently trained X-AVDT, without any fine-tuning.
Figure D.2 : Top- q Attention Mass Coverage within the Face ROI (Left) and Δ Cross-Attention Maps (Right). In the Δ maps, red indicates regions with higher fake cross-attention than real ( Δ>0 ), and blue indicates the opposite ( Δ<0 ).
Audio-visual deepfakes have reached a level of realism that makes perceptual detection unreliable, threatening media integrity and biometric security. While multimodal detection has shown promise, most approaches are binary classification tasks that often latch onto dataset-specific artifacts rather than genuine generative traces. We argue that a detector incapable of identifying how a video was forged is likely learning the wrong signal. Unlike binary detection, attribution-guided learning imposes a stronger geometric constraint on the shared embedding space, forcing the model to encode generator-specific forensic content rather than shortcuts. We propose the Attribution-Guided Multimodal Deepfake Detection (AMDD) framework, which jointly learns to detect and attribute manipulation. AMDD treats generator attribution as a structured regularization that constrains representation geometry toward forensically meaningful features. We introduce a Cross-Modal Forensic Fingerprint Consistency (CMFFC) loss to enforce alignment between generator-induced artifacts in visual and audio streams. This exploits the fact that coherent manipulation leaves correlated traces across modalities, grounded in the physical coupling between speech and facial articulation that synthetic pipelines routinely disrupt. Architecturally, we pair a ResNet50 with temporal attention for visual encoding against a pretrained ResNet18 for mel spectrograms, closing the encoder capacity gap found in prior models. On FakeAVCeleb, AMDD achieves 99.7% balanced accuracy and 99.8% AUC with 95.9% attribution accuracy. Cross-dataset evaluation on DeepfakeTIMIT, DFDM, and LAV-DF confirms that real video detection generalizes robustly, while fake detection on unseen generators remains an open challenge that we analyze in depth.
Wasim Ahmad, Wei Zhang, Xuerui Mao
School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China
The rapid progress of generative AI has enabled hyper-realistic audio-visual deepfakes, intensifying threats to personal security and social trust. Most existing deepfake detectors rely either on uni-modal artifacts or audio-visual discrepancies, failing to jointly leverage both sources of information. Moreover, detectors that rely on generator-specific artifacts tend to exhibit degraded generalization when confronted with unseen forgeries. We argue that robust and generalizable detection should be grounded in intrinsic audio-visual coherence within and across modalities. Accordingly, we propose HAVIC, a Holistic Audio-Visual Intrinsic Coherence-based deepfake detector. HAVIC first learns priors of modality-specific structural coherence, inter-modal micro- and macro-coherence by pre-training on authentic videos. Based on the learned priors, HAVIC further performs holistic adaptive aggregation to dynamically fuse audio-visual features for deepfake detection. Additionally, we introduce HiFi-AVDF, a high-fidelity audio-visual deepfake dataset featuring both text-to-video and image-to-video forgeries from state-of-the-art commercial generators. Extensive experiments across several benchmarks demonstrate that HAVIC significantly outperforms existing state-of-the-art methods, achieving improvements of 9.39% AP and 9.37% AUC on the most challenging cross-dataset scenario. Our code and dataset are available at https://github.com/tuffy-studio/HAVIC.
Audio-visual deepfake detection (AVD) is increasingly important as modern generators can fabricate convincing speech and video. Most current multimodal detectors are small, task-specific models: they work well on curated tests but scale poorly and generalize weakly across domains. We introduce AV-LMMDetect, a supervised fine-tuned (SFT) large multimodal model that casts AVD as a prompted yes/no classification - "Is this video real or fake?". Built on Qwen 2.5 Omni, it jointly analyzes audio and visual streams for deepfake detection and is trained in two stages: lightweight LoRA alignment followed by audio-visual encoder full fine-tuning. On FakeAVCeleb and Mavos-DD, AV-LMMDetect matches or surpasses prior methods and sets a new state of the art on Mavos-DD datasets.
Songjun Cao, Yuqi Li, Yunpeng Luo +2
Tencent YouTu Lab, China · Tencent Youtu Lab, China · Fudan University, China