Leveraging large multimodal models for audio-video deepfake detection: a pilot study

Songjun Cao; Yuqi Li; Yunpeng Luo; Jianjun Yin; Long Ma

cs.SDFeb 25, 2026

Leveraging large multimodal models for audio-video deepfake detection: a pilot study

Authors: Songjun Cao, Yuqi Li, Yunpeng Luo, Jianjun Yin, Long Ma

Organizations: Tencent YouTu Lab, China · Tencent Youtu Lab, China · Fudan University, China

Abstract

Audio-visual deepfake detection (AVD) is increasingly important as modern generators can fabricate convincing speech and video. Most current multimodal detectors are small, task-specific models: they work well on curated tests but scale poorly and generalize weakly across domains. We introduce AV-LMMDetect, a supervised fine-tuned (SFT) large multimodal model that casts AVD as a prompted yes/no classification - "Is this video real or fake?". Built on Qwen 2.5 Omni, it jointly analyzes audio and visual streams for deepfake detection and is trained in two stages: lightweight LoRA alignment followed by audio-visual encoder full fine-tuning. On FakeAVCeleb and Mavos-DD, AV-LMMDetect matches or surpasses prior methods and sets a new state of the art on Mavos-DD datasets.

Figures & tables

Figure 1: Performance comparison with Qwen 2.5 Omni [ 13 ] , highlighting the improved results of our AV-LMMDETECT. The video data is from the open-source MAVOS-DD dataset [ 14 ] .

Figure 2: Overview of our AV-LMMDetect . We reformulate audio-visual deepfake detection as a multimodal question answering task. During two-stage fine-tuning, Stage 1 employs LoRA for efficient alignment, while Stage 2 opens vision and audio encoders for audio-visual encoder full learning. The video data is from the open-source MAVOS-DD dataset [ 14 ] .

Method	Modality	AUC	acc
MesoNet [ 1 ]	V	60.9	57.3
Capsule [ 2 ]	V	70.9	68.8
Xception [ 3 ]	V	70.5	67.9
LipForensics [ 4 ]	V	82.4	80.2
Multiple-Attention [ 5 ]	V	79.3	77.6
SLADD [ 6 ]	V	72.1	70.5

Table 1: Intra-manipulation evaluation on FakeAVCeleb [ 16 ] . Following standard protocol, we use 70% as training set and 30% as test set. We report Area Under the Curve (AUC) and Accuracy (Acc) scores (%).

Method	Fine-tuned	In-domain			Open-set model			Open-set language			Open-set full
Method	Fine-tuned	mAP	AUC	acc	mAP	AUC	acc	mAP	AUC	acc	mAP	AUC	acc
AVFF [ 12 ]	✗	0.51	0.51	52.45	0.50	0.50	22.58	0.51	0.51	59.46	0.50	0.50	35.34
MRDF [ 17 ]	✗	0.50	0.46	44.04	0.52	0.52	58.04	0.46	0.41	39.35	0.51	0.49	50.78
TALL [ 18 ]	✗	0.49	0.48	50.74	0.50	0.51	39.22	0.48	0.47	50.78	0.50	0.49	44.63
Qwen 2.5 Omni [ 13 ]	✗	0.47	0.44	49.25	0.77	0.45	20.84	0.38	0.43	55.50	0.61	0.41	32.26
AVFF [ 12 ]	✓	0.95	0.95	86.93	0.85	0.89	75.34	0.90	0.90	84.26	0.87	0.89	77.68

Table 2: Comparison on MAVOS-DD [ 14 ] . Best results are in bold ; second-best are underlined .

Training Strategy	mAP	AUC	acc
Zero-shot	0.61	0.41	32.26
Stage 1 only	0.82	0.66	73.40
Stage 2 only	0.86	0.83	80.61
Stage 1 + Stage 2 (Ours)	0.96	0.92	85.09

Table 3: Ablation study on MAVOS-DD [ 14 ] Open-set full scenario comparing four training configurations.

Figure 3: Confusion matrices for AVFF, MRDF, TALL, and our AV-LMMDetect on MAVOS-DD Open-set full scenario.

Explore similar work

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints
Apr 29, 2026Wasim Ahmad, Wei Zhang, Xuerui MaoAudio Deepfake Detection Cross-Modal Representation Learning
School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China
SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment
Mar 26, 2026Sahibzada Adil Shahzad, Ammarah Hashmi, Junichi Yamagishi +5Deepfake Detection Audio Deepfake Detection
Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Institute of Information Science, Academia Sinica, Taipei 11529, Taiwan · Department of Computer Science, National Chengchi University, Taipei 11529, Taiwan · Institute of Information Systems and Applications, National Tsing Hua University, Hsinchu 300044, Taiwan +4
X-AVDT: Audio-Visual Cross-Attention for Robust Deepfake Detection
Mar 9, 2026Youngseo Kim, Kwan Yun, Seokhyeon Hong +3Deepfake Detection Audio Deepfake Detection
Visual Media Lab, KAIST

Apr 29, 2026J/K move · Enter open · S save

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints

Wasim Ahmad, Wei Zhang, Xuerui Mao

School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China

Audio Deepfake Detection Cross-Modal Representation Learning Digital Forensics Deepfake Detection

Open paper AskPDF arXiv

Audio-visual deepfakes have reached a level of realism that makes perceptual detection unreliable, threatening media integrity and biometric security. While multimodal detection has shown promise, most approaches are binary classification tasks that often latch onto dataset-specific artifacts rather than genuine generative traces. We argue that a detector incapable of identifying how a video was forged is likely learning the wrong signal. Unlike binary detection, attribution-guided learning imposes a stronger geometric constraint on the shared embedding space, forcing the model to encode generator-specific forensic content rather than shortcuts. We propose the Attribution-Guided Multimodal Deepfake Detection (AMDD) framework, which jointly learns to detect and attribute manipulation. AMDD treats generator attribution as a structured regularization that constrains representation geometry toward forensically meaningful features. We introduce a Cross-Modal Forensic Fingerprint Consistency (CMFFC) loss to enforce alignment between generator-induced artifacts in visual and audio streams. This exploits the fact that coherent manipulation leaves correlated traces across modalities, grounded in the physical coupling between speech and facial articulation that synthetic pipelines routinely disrupt. Architecturally, we pair a ResNet50 with temporal attention for visual encoding against a pretrained ResNet18 for mel spectrograms, closing the encoder capacity gap found in prior models. On FakeAVCeleb, AMDD achieves 99.7% balanced accuracy and 99.8% AUC with 95.9% attribution accuracy. Cross-dataset evaluation on DeepfakeTIMIT, DFDM, and LAV-DF confirms that real video detection generalizes robustly, while fake detection on unseen generators remains an open challenge that we analyze in depth.

Abstract

Explore similar work

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints

SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment

X-AVDT: Audio-Visual Cross-Attention for Robust Deepfake Detection