The rapid progress of generative AI has enabled hyper-realistic audio-visual deepfakes, intensifying threats to personal security and social trust. Most existing deepfake detectors rely either on uni-modal artifacts or audio-visual discrepancies, failing to jointly leverage both sources of information. Moreover, detectors that rely on generator-specific artifacts tend to exhibit degraded generalization when confronted with unseen forgeries. We argue that robust and generalizable detection should be grounded in intrinsic audio-visual coherence within and across modalities. Accordingly, we propose HAVIC, a Holistic Audio-Visual Intrinsic Coherence-based deepfake detector. HAVIC first learns priors of modality-specific structural coherence, inter-modal micro- and macro-coherence by pre-training on authentic videos. Based on the learned priors, HAVIC further performs holistic adaptive aggregation to dynamically fuse audio-visual features for deepfake detection. Additionally, we introduce HiFi-AVDF, a high-fidelity audio-visual deepfake dataset featuring both text-to-video and image-to-video forgeries from state-of-the-art commercial generators. Extensive experiments across several benchmarks demonstrate that HAVIC significantly outperforms existing state-of-the-art methods, achieving improvements of 9.39% AP and 9.37% AUC on the most challenging cross-dataset scenario. Our code and dataset are available at https://github.com/tuffy-studio/HAVIC.
Figures & tables
Dataset
Modality
T2V
I2V
Person #
Year
FaceForensics++ [ 70 ]
V
✗
✗
–
2019
WildDeepfake [ 100 ]
V
✗
✗
–
2020
KoDF [ 45 ]
V
✗
✗
403
2021
DF-Platter [ 58 ]
V
✗
✗
454
2023
FakeAVCeleb [ 39 ]
AV
✗
✗
500
2021
DefakeAVMiT [ 93 ]
AV
✗
✗
86
2023
Table 1: Comparison of representative deepfake datasets. Columns indicate manipulated modality (V: visual-only, AV: audio-visual), support for Text-to-Video (T2V) and Image-to-Video (I2V) generation, number of person, and publication year. Our HiFi-AVDF dataset contains forged content in both audio and visual modalities, and supports modern T2V and I2V generation.
Figure 1: Overview of our proposed HAVIC. (a) Holistic Coherence Priors Pre-training phase . The visible tokens xv,vis and xa,vis are encoded by Ev and Ea to produce hierarchical features {vl}l=1H and {al}l=1H . The high-level representations vH and aH are further processed by the Audio-Visual Interaction Module I to yield interaction-aware features vinter and ainter . These features are decoded by modality-specific decoders Dv and Da , where each layer integrates hierarchical features for input reconstruction. In parallel, vinter and ainter are fed into cross-modal decoders Dv→a and Da→v to reconstruct the counterpart semantics a^Hg and v^Hg , supervised by gradient-stopped targets aHg and vHg . (b) Holistic Adaptive Aggregation Classification phase . The pre-trained Ev , Ea , and I with learned holistic coherence priors are used to extract features from the input sample. These features are aggregated by the Adaptive Feature Aggregation module A and then fed into the classifiers to predict both modality-specific and overall authenticity.
Figure 2: Illustrations of three self-supervised objectives in the Holistic Coherence Priors Pre-training phase . (a) Each decoder layer reconstructs inputs using hierarchical encoder features, enforcing modality-specific structural coherence (illustrated with the visual modality). (b) Audio and visual features are temporally partitioned and aligned segment by segment, with a soft negative pairs strategy to capture inter-modal micro-coherence (only one temporal segment and the visual-to-audio direction are illustrated for simplicity). (c) One modality reconstructs the global semantic representation of the other, ensuring inter-modal macro-coherence.
Figure 3: Overview of the HiFi-AVDF dataset . The left panel shows the deepfake generation process, where audio tracks, reference frames, and video captions of real videos are fed into generation models. The right panel shows dataset statistics, detailing the proportion of samples generated by each generative model and the distribution of different manipulation strategies. The bottom panels present representative examples generated by different models.
Figure 1: Examples of real–fake video pairs generated by the six models in HiFi-AVDF. For each model, we display one representative pair, where the top row shows a real video clip and the bottom row shows the corresponding forged clip produced by that model.
Table 1: Summary of representative deepfake datasets. This table compares commonly used visual and audio-visual deepfake datasets in terms of manipulated modality, support for T2V/I2V generation, generation methods, number of person, and counts of samples. HiFi-AVDF provides high-quality audio-visual forgeries generated by diverse state-of-the-art models, supporting both T2V and I2V modalities.
Category
Attribute
Percentage (%)
Gender
Male
62.99
Female
37.01
Race/Ethnicity
White
56.43
Black
3.73
Latino Hispanic
4.41
East Asian
11.86
Table 2: Demographic distribution analysis results on the HiFi-AVDF dataset.
Figure 2: Modality-Specific Hierarchical Reconstruction visualizations . For each clip, the first row shows the original audio spectrograms and visual frames, while the second and third rows depict the masked inputs and the corresponding reconstructions from the final decoder layer, respectively. Details of the reconstructions can be seen by zooming in.
Loss weight λcl
FakeAVCeleb
KoDF
HiFi-AVDF
ACC
AUC
AP
AUC
AP
AUC
0.001
99.8
99.8
97.3
96.7
72.3
67.1
0.005
99.8
99.9
98.5
99.0
74.7
74.4
0.01
99.8
99.9
99.2
98.8
75.4
75.7
0.05
99.8
99.8
98.3
98.4
73.9
75.5
0.1
99.6
99.7
97.8
97.9
72.4
73.8
Table 8: Ablation study on loss weight for Fine-grained Audio-visual Contrastive Loss.
Figure 3: t-SNE visualization of the learned audio-visual embeddings . From left to right: (1) embeddings without Adaptive Aggregation, (2) embeddings without Auxiliary Classifiers, and (3) embeddings of the complete model.
Method
Parameters (M)
Throughput (samples/s)
AP
AUC
VideoMAE [ 79 ]
235.1
74.7
69.1
71.3
HiCMAE [ 75 ]
241.0
72.5
71.8
73.5
AVFF [ 61 ]
196.8
34.2
66.0
65.2
HAVIC (Ours)
243.1
66.8
75.4
75.7
Table 9: Comparison of model complexity and performance. The number of parameters and inference throughput are reported for the pre-training models, while AP and AUC are evaluated on the HiFi-AVDF dataset to assess detection performance.
Figure 4: Examples of challenging failure cases. From left to right: (1) masked faces, (2) profile or side faces, and (3) highly realistic forgeries.
The surge of highly realistic synthetic videos produced by contemporary generative systems has significantly increased the risk of malicious use, challenging both humans and existing detectors. Against this backdrop, we take a generator-side view and observe that internal cross-attention mechanisms in these models encode fine-grained speech-motion alignment, offering useful correspondence cues for forgery detection. Building on this insight, we propose X-AVDT, a robust and generalizable deepfake detector that probes generator-internal audio-visual signals accessed via DDIM inversion to expose these cues. X-AVDT extracts two complementary signals: (i) a video composite capturing inversion-induced discrepancies, and (ii) an audio-visual cross-attention feature reflecting modality alignment enforced during generation. To enable faithful cross-generator evaluation, we further introduce MMDF, a new multimodal deepfake dataset spanning diverse manipulation types and rapidly evolving synthesis paradigms, including GANs, diffusion, and flow-matching. Extensive experiments demonstrate that X-AVDT achieves leading performance on MMDF and generalizes strongly to external benchmarks and unseen generators, outperforming existing methods with accuracy improved by 13.1%. Our findings highlight the importance of leveraging internal audio-visual consistency cues for robustness to future generators in deepfake detection.
Audio-visual deepfake detection (AVD) is increasingly important as modern generators can fabricate convincing speech and video. Most current multimodal detectors are small, task-specific models: they work well on curated tests but scale poorly and generalize weakly across domains. We introduce AV-LMMDetect, a supervised fine-tuned (SFT) large multimodal model that casts AVD as a prompted yes/no classification - "Is this video real or fake?". Built on Qwen 2.5 Omni, it jointly analyzes audio and visual streams for deepfake detection and is trained in two stages: lightweight LoRA alignment followed by audio-visual encoder full fine-tuning. On FakeAVCeleb and Mavos-DD, AV-LMMDetect matches or surpasses prior methods and sets a new state of the art on Mavos-DD datasets.
Songjun Cao, Yuqi Li, Yunpeng Luo +2
Tencent YouTu Lab, China · Tencent Youtu Lab, China · Fudan University, China
Generative AI has rapidly expanded audio-visual forgery beyond human-centric deepfakes into general scenes. Existing AIGC detection methods assume audio-visual content correspondence, identifying forgeries by spotting cross-modal inconsistencies. However, we empirically find that this assumption does not consistently hold in general scenarios. We argue that, for general audio-visual AIGC detection, decision-level fusion is a more robust alternative to feature-level fusion. Therefore, we propose DAV-Det, a decoupled audio-visual AIGC detection system that independently models forensic evidence from each modality. The visual detector leverages multi-granularity representations at global, patch, and segment levels to capture spatial forgery cues, while the audio detector exploits both temporal and spectral irregularities via a gated temporal-spectral dual-branch architecture to model acoustic artifacts. Our method ranks 1st in the General AIGC Audio-Video Detection Challenge of the IJCAI-ECAI 2026 DDL 2.0 Workshop, with a final score of 0.8460. Code is available at https://github.com/tuffy-studio/DAV-Det.
Jielun Peng, Yabin Wang, Yaqi Li +3
Harbin Institute of Technology · University of Agder