SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment

Sahibzada Adil Shahzad; Ammarah Hashmi; Junichi Yamagishi; Yusuke Yasuda; Yu Tsao; Chia-Wen Lin; Yan-Tsung Peng; Hsin-Min Wang

cs.CVMar 26, 2026

SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment

Authors: Sahibzada Adil Shahzad, Ammarah Hashmi, Junichi Yamagishi, Yusuke Yasuda, Yu Tsao, Chia-Wen Lin, Yan-Tsung Peng, Hsin-Min Wang

Organizations: Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Institute of Information Science, Academia Sinica, Taipei 11529, Taiwan · Department of Computer Science, National Chengchi University, Taipei 11529, Taiwan · Institute of Information Systems and Applications, National Tsing Hua University, Hsinchu 300044, Taiwan · National Institute of Informatics, Tokyo 101-8430, Japan · Research Center for Information Technology Innovation, Academia Sinica, Taipei 11529, Taiwan · Department of Electrical Engineering and the Institute of Communications Engineering, National Tsing Hua University, Hsinchu 300044, Taiwan · Institute of Information Science, Academia Sinica, Taipei 11529, Taiwan

Abstract

Multimodal deepfakes can exhibit subtle visual artifacts and cross-modal inconsistencies, which remain challenging to detect, especially when detectors are trained primarily on curated synthetic forgeries. Such synthetic dependence can introduce dataset and generator bias, limiting scalability and robustness to unseen manipulations. We propose SAVe, a self-supervised audio-visual deepfake detection framework that learns entirely on authentic videos. SAVe generates on-the-fly, identity-preserving, region-aware self-blended pseudo-manipulations to emulate tampering artifacts, enabling the model to learn complementary visual cues across multiple facial granularities. To capture cross-modal evidence, SAVe also models lip-speech synchronization via an audio-visual alignment component that detects temporal misalignment patterns characteristic of audio-visual forgeries. Experiments on FakeAVCeleb and AV-LipSync-TIMIT demonstrate competitive in-domain performance and strong cross-dataset generalization, highlighting self-supervised learning as a scalable paradigm for multimodal deepfake detection.

Figures & tables

Fig. 1: Prior visual/audio-visual deepfake detectors typically rely on manipulated training data and task-specific preprocessing, using either visual-only models or audio-visual synchronization cues with ad-hoc fusion, which can lead to poor generalization under unseen generators. In contrast, the proposed model, SAVe, is trained exclusively on authentic videos by generating on-the-fly, identity-preserving self-blended pseudo-manipulations. SAVe combines FaceBlend , LipBlend , LowerFaceBlend , and AVSync features via a Fusion head for robust real/fake prediction.

Fig. 2: Overview of the proposed SAVe framework, a self-supervised learning (SSL) audio-visual deepfake detection method trained solely using authentic data. Given an input video, visual frames and audio are extracted, in the AVSync branch, both modalities are encoded by AV-HuBERT to obtain audio ( $F_{a}$ ) and visual ( $F_{v}$ ) representations, which are then processed by an alignment network to produce an audio-visual misalignment score that reflects lip-speech synchronization consistency. In parallel, the SS-VPFG module synthesizes on-the-fly pseudo-forged visual samples via identity-preserving self-blending over multiple facial regions, enabling region-aware supervision: (face (real $\text{I}_{\text{FR}}$ vs. fake $\text{I}_{\text{FB}}$ ), lip (real $\text{I}_{\text{LR}}$ vs. fake $\text{I}_{\text{LB}}$ ), and lower-face (real $\text{I}_{\text{LFR}}$ vs. fake $\text{I}_{\text{LFB}}$ )) to provide region-aware supervision. Finally, region-specific visual features and AVSync misalignment cues are aggregated by the Fusion module to output the final prediction (Real/Fake).

Fig. 3: Overview of the proposed Self-Supervised Visual Pseudo-Forgery Generation (SS-VPFG) module. Given a Base Image, the module first performs source-target augmentation and facial landmark detection to produce aligned source and target images. Three region-specific blending pipelines are then applied: FaceBlend , LipBlend , and LowerFaceBlend . Each pipeline extracts the corresponding region of interest (ROI), generates region masks, applies targeted augmentations, and outputs forged variants, namely $\text{I}_{\text{FB}}$ , $\text{I}_{\text{LB}}$ , and $\text{I}_{\text{LFB}}$ , alongside their intermediate augmented results ( $\text{I}_{\text{FR}}$ , $\text{I}_{\text{LR}}$ , and $\text{I}_{\text{LFR}}$ ). These pseudo forgeries provide diverse self-supervised signals for training visual forgery detectors.

Manipulation	FaceBlend		LowerFaceBlend		LipBlend
Manipulation	AUC	AP	AUC	AP	AUC	AP
Wav2Lip	$0.9999$	$0.9999$	$0.9895$	$0.9994$	$0.9826$	$0.9981$
FaceSwap	$0.9781$	$0.9859$	$0.9642$	$0.9736$	$0.9718$	$0.9740$
FaceSwap+Wav2Lip	$0.9999$	$0.9999$	$0.9955$	$0.9991$	$0.9924$	$0.9972$
FSGAN	$0.9999$	$0.9999$	$0.9803$	$0.9971$	$0.9736$	$0.9946$
FSGAN+Wav2Lip	$0.9999$	$0.9999$	$0.9957$	$0.9994$	$0.9910$	$0.9972$

TABLE I: In-domain performance on FakeAVCeleb for visual self-supervised branches trained only on real videos with on-the-fly region-wise self-blending. We report AUC and AP for FaceBlend, LowerFaceBlend, and LipBlend, evaluated per manipulation type.

Trained on LipSyncTIMIT (All compressions: original, c23, c40)
Method	FaceBlend		LowerFaceBlend		LipBlend
Method	AUC	AP	AUC	AP	AUC	AP
Wav2Lip	$0.8266$	$0.9865$	$0.7680$	$0.9825$	$0.8342$	$0.9875$
FaceSwap	$0.6676$	$0.7334$	$0.7978$	$0.8508$	$0.9345$	$0.9536$
FaceSwap+Wav2Lip	$0.8165$	$0.9516$	$0.8560$	$0.9637$	$0.9565$	$0.9898$
FSGAN	$0.8326$	$0.9720$	$0.7854$	$0.9626$	$0.8429$	$0.9755$

TABLE II: Cross-dataset generalization from LipSyncTIMIT to FakeAVCeleb for visual self-supervised branches. Models are trained on LipSyncTIMIT under different compression settings and evaluated on FakeAVCeleb across unseen manipulation types. We report AUC and AP.

Manipulation	AVSync
Manipulation	AUC	AP
Wav2Lip	$0.9623$	$0.9945$
FaceSwap	$0.8630$	$0.8737$
FaceSwap+Wav2Lip	$0.9558$	$0.9800$
FSGAN	$0.8863$	$0.9757$
FSGAN+Wav2Lip	$0.9477$	$0.9834$

TABLE III: In-domain performance of the audio-visual synchronization module ( AVSync ) on FakeAVCeleb. We report AUC and AP per manipulation type and over the entire dataset.

Trained on LipSyncTIMIT (original)
Manipulation	AVSync
Manipulation	AUC	AP
Wav2Lip	$0.8920$	$0.9893$
FaceSwap	$0.6874$	$0.7644$
FaceSwap+Wav2Lip	$0.8791$	$0.9585$
FSGAN	$0.7165$	$0.9475$

TABLE IV: Cross-dataset generalization of AVSync from LipSyncTIMIT to FakeAVCeleb. AVSync is trained on LipSyncTIMIT under different compression settings and evaluated on FakeAVCeleb. We report AUC and AP.

Model	Train Type	Modality	FakeAVCeleb-LS	Original	c23	c40
				LipSyncTIMIT
Xception [ 16 ]	sup.	V	$0.90$	$0.90$	$0.85$	$0.56$
LSDA [ 59 ]	sup.	V	-	$0.91$	$0.84$	$0.60$
FTCN [ 21 ]	sup.	V	$0.97$	$0.84$	$0.84$	$0.59$
LIPINC-V2 [ 56 ]	sup.	V	$0.99$	$0.97$	$0.96$	$0.76$
FaceBlend	unsup.	V	$0.9999$	$0.8508$	$0.8047$	$0.4856$

TABLE V: Comparison with baselines (AUC). Supervised methods (sup.) are trained on labeled real and fake data. Self-supervised methods (unsup.) are trained solely on real videos. AVSync is a component branch of SAVe , and Proposed (ours) denotes average-logit fusion of FB/LB/LFB/AVSync .

Explore similar work

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints
Apr 29, 2026Wasim Ahmad, Wei Zhang, Xuerui MaoAudio Deepfake Detection Cross-Modal Representation Learning
School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China
Leveraging large multimodal models for audio-video deepfake detection: a pilot study
Feb 25, 2026Songjun Cao, Yuqi Li, Yunpeng Luo +2Deepfake Detection Audio Deepfake Detection
Tencent YouTu Lab, China · Tencent Youtu Lab, China · Fudan University, China
MSCT: Differential Cross-Modal Attention for Deepfake Detection
Apr 9, 2026Fangda Wei, Miao Liu, Yingxue Wang +3Deepfake Detection Cross-Modal Attention
Beijing Institute of Technology, China · China Academy of Electronics and Information Technology, China

Apr 29, 2026J/K move · Enter open · S save

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints

Wasim Ahmad, Wei Zhang, Xuerui Mao

School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China

Audio Deepfake Detection Cross-Modal Representation Learning Digital Forensics Deepfake Detection

Open paper AskPDF arXiv

Audio-visual deepfakes have reached a level of realism that makes perceptual detection unreliable, threatening media integrity and biometric security. While multimodal detection has shown promise, most approaches are binary classification tasks that often latch onto dataset-specific artifacts rather than genuine generative traces. We argue that a detector incapable of identifying how a video was forged is likely learning the wrong signal. Unlike binary detection, attribution-guided learning imposes a stronger geometric constraint on the shared embedding space, forcing the model to encode generator-specific forensic content rather than shortcuts. We propose the Attribution-Guided Multimodal Deepfake Detection (AMDD) framework, which jointly learns to detect and attribute manipulation. AMDD treats generator attribution as a structured regularization that constrains representation geometry toward forensically meaningful features. We introduce a Cross-Modal Forensic Fingerprint Consistency (CMFFC) loss to enforce alignment between generator-induced artifacts in visual and audio streams. This exploits the fact that coherent manipulation leaves correlated traces across modalities, grounded in the physical coupling between speech and facial articulation that synthetic pipelines routinely disrupt. Architecturally, we pair a ResNet50 with temporal attention for visual encoding against a pretrained ResNet18 for mel spectrograms, closing the encoder capacity gap found in prior models. On FakeAVCeleb, AMDD achieves 99.7% balanced accuracy and 99.8% AUC with 95.9% attribution accuracy. Cross-dataset evaluation on DeepfakeTIMIT, DFDM, and LAV-DF confirms that real video detection generalizes robustly, while fake detection on unseen generators remains an open challenge that we analyze in depth.

Abstract

Explore similar work

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints

Leveraging large multimodal models for audio-video deepfake detection: a pilot study

MSCT: Differential Cross-Modal Attention for Deepfake Detection