The rapid progress of generative AI has enabled hyper-realistic audio-visual deepfakes, intensifying threats to personal security and social trust. Most existing deepfake detectors rely either on uni-modal artifacts or audio-visual discrepancies, failing to jointly leverage both sources of information. Moreover, detectors that rely on generator-specific artifacts tend to exhibit degraded generalization when confronted with unseen forgeries. We argue that robust and generalizable detection should be grounded in intrinsic audio-visual coherence within and across modalities. Accordingly, we propose HAVIC, a Holistic Audio-Visual Intrinsic Coherence-based deepfake detector. HAVIC first learns priors of modality-specific structural coherence, inter-modal micro- and macro-coherence by pre-training on authentic videos. Based on the learned priors, HAVIC further performs holistic adaptive aggregation to dynamically fuse audio-visual features for deepfake detection. Additionally, we introduce HiFi-AVDF, a high-fidelity audio-visual deepfake dataset featuring both text-to-video and image-to-video forgeries from state-of-the-art commercial generators. Extensive experiments across several benchmarks demonstrate that HAVIC significantly outperforms existing state-of-the-art methods, achieving improvements of 9.39% AP and 9.37% AUC on the most challenging cross-dataset scenario. Our code and dataset are available at https://github.com/tuffy-studio/HAVIC.
Figures & tables
Dataset
Modality
T2V
I2V
Person #
Year
FaceForensics++ [ 70 ]
V
✗
✗
–
2019
WildDeepfake [ 100 ]
V
✗
✗
–
2020
KoDF [ 45 ]
V
✗
✗
403
2021
DF-Platter [ 58 ]
V
✗
✗
454
2023
FakeAVCeleb [ 39 ]
AV
✗
✗
500
2021
DefakeAVMiT [ 93 ]
AV
✗
✗
86
2023
Table 1: Comparison of representative deepfake datasets. Columns indicate manipulated modality (V: visual-only, AV: audio-visual), support for Text-to-Video (T2V) and Image-to-Video (I2V) generation, number of person, and publication year. Our HiFi-AVDF dataset contains forged content in both audio and visual modalities, and supports modern T2V and I2V generation.
Figure 1: Overview of our proposed HAVIC. (a) Holistic Coherence Priors Pre-training phase . The visible tokens xv,vis and xa,vis are encoded by Ev and Ea to produce hierarchical features {vl}l=1H and {al}l=1H . The high-level representations vH and aH are further processed by the Audio-Visual Interaction Module I to yield interaction-aware features vinter and ainter . These features are decoded by modality-specific decoders Dv and Da , where each layer integrates hierarchical features for input reconstruction. In parallel, vinter and ainter are fed into cross-modal decoders Dv→a and Da→v to reconstruct the counterpart semantics a^Hg and v^Hg , supervised by gradient-stopped targets aHg and vHg . (b) Holistic Adaptive Aggregation Classification phase . The pre-trained Ev , Ea , and I with learned holistic coherence priors are used to extract features from the input sample. These features are aggregated by the Adaptive Feature Aggregation module A and then fed into the classifiers to predict both modality-specific and overall authenticity.
Figure 2: Illustrations of three self-supervised objectives in the Holistic Coherence Priors Pre-training phase . (a) Each decoder layer reconstructs inputs using hierarchical encoder features, enforcing modality-specific structural coherence (illustrated with the visual modality). (b) Audio and visual features are temporally partitioned and aligned segment by segment, with a soft negative pairs strategy to capture inter-modal micro-coherence (only one temporal segment and the visual-to-audio direction are illustrated for simplicity). (c) One modality reconstructs the global semantic representation of the other, ensuring inter-modal macro-coherence.
Figure 3: Overview of the HiFi-AVDF dataset . The left panel shows the deepfake generation process, where audio tracks, reference frames, and video captions of real videos are fed into generation models. The right panel shows dataset statistics, detailing the proportion of samples generated by each generative model and the distribution of different manipulation strategies. The bottom panels present representative examples generated by different models.
Figure 1: Examples of real–fake video pairs generated by the six models in HiFi-AVDF. For each model, we display one representative pair, where the top row shows a real video clip and the bottom row shows the corresponding forged clip produced by that model.
Table 1: Summary of representative deepfake datasets. This table compares commonly used visual and audio-visual deepfake datasets in terms of manipulated modality, support for T2V/I2V generation, generation methods, number of person, and counts of samples. HiFi-AVDF provides high-quality audio-visual forgeries generated by diverse state-of-the-art models, supporting both T2V and I2V modalities.
Category
Attribute
Percentage (%)
Gender
Male
62.99
Female
37.01
Race/Ethnicity
White
56.43
Black
3.73
Latino Hispanic
4.41
East Asian
11.86
Table 2: Demographic distribution analysis results on the HiFi-AVDF dataset.
Figure 2: Modality-Specific Hierarchical Reconstruction visualizations . For each clip, the first row shows the original audio spectrograms and visual frames, while the second and third rows depict the masked inputs and the corresponding reconstructions from the final decoder layer, respectively. Details of the reconstructions can be seen by zooming in.
Loss weight λcl
FakeAVCeleb
KoDF
HiFi-AVDF
ACC
AUC
AP
AUC
AP
AUC
0.001
99.8
99.8
97.3
96.7
72.3
67.1
0.005
99.8
99.9
98.5
99.0
74.7
74.4
0.01
99.8
99.9
99.2
98.8
75.4
75.7
0.05
99.8
99.8
98.3
98.4
73.9
75.5
0.1
99.6
99.7
97.8
97.9
72.4
73.8
Table 8: Ablation study on loss weight for Fine-grained Audio-visual Contrastive Loss.
Figure 3: t-SNE visualization of the learned audio-visual embeddings . From left to right: (1) embeddings without Adaptive Aggregation, (2) embeddings without Auxiliary Classifiers, and (3) embeddings of the complete model.
Method
Parameters (M)
Throughput (samples/s)
AP
AUC
VideoMAE [ 79 ]
235.1
74.7
69.1
71.3
HiCMAE [ 75 ]
241.0
72.5
71.8
73.5
AVFF [ 61 ]
196.8
34.2
66.0
65.2
HAVIC (Ours)
243.1
66.8
75.4
75.7
Table 9: Comparison of model complexity and performance. The number of parameters and inference throughput are reported for the pre-training models, while AP and AUC are evaluated on the HiFi-AVDF dataset to assess detection performance.
Figure 4: Examples of challenging failure cases. From left to right: (1) masked faces, (2) profile or side faces, and (3) highly realistic forgeries.