MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark

Florinel-Alin Croitoru; Vlad Hondru; Marius Popescu; Radu Tudor Ionescu; Fahad Shahbaz Khan; Mubarak Shah

cs.CVMay 16, 2025

MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark

Authors: Florinel-Alin Croitoru, Vlad Hondru, Marius Popescu, Radu Tudor Ionescu, Fahad Shahbaz Khan, Mubarak Shah

Organizations: University of Bucharest · MBZ University of Artificial Intelligence · Linköping University · University of Central Florida

Abstract

We present the first large-scale open-set benchmark for multilingual audio-video deepfake detection. Our dataset comprises nearly 350 hours of real and fake videos across eight languages, with 59% of data being generated. For each language, the fake videos are generated with several distinct audio and video deepfake generation models, selected based on their quality, via visual inspection. We organize the training, validation and test splits such that only a subset of the chosen generative models and languages are available during training, thus creating several challenging open-set evaluation setups. We perform experiments with various pre-trained and fine-tuned deepfake detectors proposed in recent literature. Our results show that state-of-the-art detectors are not currently able to maintain their performance levels when tested in our open-set scenarios. We publicly release our data and code for non-commercial use at https://github.com/CroitoruAlin/MAVOS-DD.

Figures & tables

Fig. 1 : In MAVOS-DD, the training set and in-domain test set contain real and fake videos sampled from the same distribution, comprising six languages and six generative approaches. The open-set model test set extends the in-domain test set with fake samples generated by unseen models (Sonic, HifiFace, Roop, VC2 and Sonic/HifiFace/Roop+VC2). The open-set language test set extends the in-domain test set with samples in unseen languages (German and Hindi). The open-set full test set adds samples generated by unseen models in unseen languages. One fake sample from each data distribution is shown on the right-hand side. Groups of voice conversion methods (VC1 and VC2) are disjoint across all languages. Best viewed in color.

Dataset	File count		Length (h)			#Generative methods	#Languages	Open-set	Multimodal
Dataset	#Real	#Fake	Real	Fake	Total	#Generative methods	#Languages	Open-set	Multimodal
FaceForensics++ [ 37 ]	1,000	4,000	4.7	17.0	21.7	4	0	✗	✗
DFDC [ 52 ]	23,654	104,500	64.4	288.9	353.3	5	0	✗	✗
DeeperForensics [ 53 ]	50,000	10,000	46.3	116.7	163.0	1	0	✗	✗
ForgeryNet [ 69 ]	99,630	121,617	13.3	13.5	26.8	15	0	✗	✗
Celeb-DF [ 36 ]	590	5,639	2.1	20.4	22.5	1	0	✗	✗

TABLE I : Comparison between MAVOS-DD and other video and audio-video (multimodal) datasets. MAVOS-DD is the largest dataset for multilingual audio-video open-set deepfake detection.

Panel of Figure 3 — Fig. 2 : Distribution of videos per language and per generative method. MAVOS-DD comprises videos in eight languages, generated with various methods. The languages are coded as follows: Arabic (AR), English (EN), German (DE), Hindi (HI), Mandarin (MD), Romanian (RO), Russian (RU) and Spanish (ES).

Split		Video type	File count				Total count	Total length (h)
Split		Video type	Core set	Extra langs	Extra models	Extra models & langs	Total count	Total length (h)
Train		Real	10,297	0	0	0	10,297	38.5
Train		Fake	11,073	0	0	0	11,073	49.1
Validation		Real	1,715	0	0	0	1,715	6.4
Validation		Fake	2,180	0	0	0	2,180	9.5
Test	In-domain	Real	5,185	0	0	0	5,185	19.3

TABLE II : Number of real and fake videos included in the training, validation and test splits of MAVOS-DD. The test data is divided into four subsets, which generate an in-domain evaluation scenario and three open-set evaluation scenarios. The core set includes six languages (Arabic, English, Mandarin, Romanian, Russian, Spanish) and four methods (EchoMimic, Memo, LivePortrait, Inswapper). The extra languages are German and Hindi. The extra models are Sonic, HifiFace, Roop and VC2. The length (in hours) of the real and fake content in each split is reported in the last column.

	RVRA	FVRA	RVFA	FVFA
Sample count	35,557	35,169	8,939	7,373
Length (h)	128	160	32	29

TABLE III : MAVOS-DD splits across real and fake modalities. Legend: RV – real video; FV – fake video; RA – real audio; FA – fake audio.

Fig. 3 : Fake video frames generated by several deepfake generation methods. Best viewed in color.

Method	Fine-tuned	In-domain			Open-set model			Open-set language			Open-set full
Method	Fine-tuned	mAP	AUC	acc	mAP	AUC	acc	mAP	AUC	acc	mAP	AUC	acc
AVFF	✗	$0.53$	$0.54$	$49.23$	$0.54$	$0.56$	$56.78$	$0.53$	$0.52$	$56.73$	$0.53$	$0.55$	$53.85$
MRDF	✗	$0.58$	$0.58$	$54.81$	$0.59$	$0.60$	$49.57$	$0.57$	$0.58$	$55.71$	$0.59$	$0.59$	$53.33$
TALL	✗	$0.50$	$0.50$	$49.94$	$0.47$	$0.45$	$43.60$	$0.48$	$0.48$	$49.70$	$0.47$	$0.46$	$46.38$
AVH-Align	✗	0.56	0.57	55.18	0.51	0.51	42.28	0.58	0.60	57.82	0.54	0.54	46.28
AVH-Align/sup	✗	0.52	0.51	48.30	0.48	0.46	63.80	0.56	0.56	42.10	0.52	0.52	57.70

TABLE IV : Results obtained by pre-trained and fine-tuned versions of AVFF [ 25 ] , MRDF [ 48 ] , TALL [ 49 ] and AVH-Align [ 47 ] on the MAVOS-DD official test sets: in-domain, open-set model, open-set language and open-set full. The best and second-best results on each column are highlighted in bold blue and orange , respectively.

Model	Pre-trained			Fine-tuned on MAVOS-DD
Model	mAP	AUC	acc	mAP	AUC	acc
AVFF	0.49	0.48	54.28	0.95	0.95	70.78
TALL	0.55	0.49	44.69	0.83	0.82	74.66
MRDF	0.52	0.52	48.56	0.53	0.53	52.12

TABLE V : Results on BioDeepAV [ 17 ] for pre-trained vs. fine-tuned models.

Fig. 4 : A real video and its corresponding fake sample generated using LivePortrait. The MRDF detector incorrectly classifies the fake sample as real. Best viewed in color.

Train forgery	Test forgery	AVFF			MRDF
Train forgery	Test forgery	mAP	AUC	acc	mAP	AUC	acc
FVRA, RVFA, FVFA	FVFA	0.99	0.99	93.31	0.92	0.90	80.40
FVRA, RVFA	FVFA	0.98	0.98	92.73	0.88	0.88	78.04

TABLE VI : Cross-forgery experiments with AVFF and MRDF conducted on open-set full.

Generative method	AVFF			MRDF
Generative method	mAP	AUC	Acc	mAP	AUC	Acc
Echomimic (EM)	0.99	0.99	94.36	0.96	0.99	96.48
Memo	0.99	0.99	93.69	0.93	0.98	93.13
LivePortrait (LP)	0.98	0.98	90.85	0.46	0.41	50.16
Inswapper	0.84	0.83	74.96	0.89	0.90	83.17
VC1	0.99	0.99	93.97	0.62	0.59	50.50

TABLE VII : Results on MAVOS-DD per generative method. In the upper part, results correspond to in-domain methods. In the lower part, results correspond to cross-domain methods.

Language	Correct audio			Missing audio			Mismatched audio
Language	mAP	AUC	acc	mAP	AUC	acc	mAP	AUC	acc
Arabic	0.98	0.98	94.76	0.69	0.69	37.81	0.75	0.77	64.97
English	0.94	0.94	84.83	0.61	0.62	48.56	0.64	0.66	61.53
Mandarin	0.93	0.96	90.48	0.70	0.74	67.26	0.69	0.74	69.94
Romanian	0.99	0.99	97.87	0.63	0.64	44.52	0.81	0.82	70.90
Russian	0.86	0.96	91.23	0.69	0.80	78.06	0.65	0.77	74.14

TABLE VIII : Language-level results of AVFF on MAVOS-DD, under three scenarios. In the first scenario, videos contain the original audio. In the second scenario, audio is removed by replacing it with zero values. In the third scenario, audio is randomly replaced with that of another video.

Fig. 5 : Open-set full performance of AVFF by progressively adding more languages to the training set. Best viewed in color.

Deepfake Detection

Deepfake detection identifies manipulated or AI-generated media to support trustworthy communication, identity verification, and forensic analysis.

See all 241 papers

BabelFake: A Multilingual Audio-Visual DeepFake Benchmark
Oct 5, 2026Carlotta Segna, Joel Tschesche, Anna RohrbachAudio Deepfake Detection Deepfake Detection
TU Darmstadt & Hessian.AI, Germany
Leveraging large multimodal models for audio-video deepfake detection: a pilot study
Feb 25, 2026Songjun Cao, Yuqi Li, Yunpeng Luo +2Multimodal Deepfake Detection Deepfake Detection
Tencent YouTu Lab, China · Tencent Youtu Lab, China · Fudan University, China
ML-ITW: A Multilingual in-the-wild Benchmark for Speech Deepfake Detection
Mar 6, 2026Daixian Li, Jun Xue, Zhuolin Yi +4Audio Deepfake Detection
School of Cyber Science and Engineering, Wuhan University

Oct 5, 2026J/K move · Enter open · S save

BabelFake: A Multilingual Audio-Visual DeepFake Benchmark

Carlotta Segna, Joel Tschesche, Anna Rohrbach

TU Darmstadt & Hessian.AI, Germany

Audio Deepfake Detection Deepfake Detection Multimodal Deepfake Detection

Open paper AskPDF arXiv

Reliable and practical audio-visual DeepFake detection requires benchmarks that reflect diverse linguistic contexts and modern data synthesis pipelines for visual as well as audio manipulations. However, existing datasets predominantly contain footage of English-speakers, often include outdated manipulation types, or overlook the audio modality. Further, many datasets feature individuals who did not consent to be used in DeepFake creation. We introduce BabelFake, a multilingual audio-visual DeepFake benchmark recorded with consenting participants. BabelFake contains 399k clips (1,323 hours) from 496 individuals spanning five languages (English, German, Italian, French, Spanish). Our modular data generation pipeline pairs 11 modern video manipulation methods with 4 voice cloning engines, distinguishing visual-only (face swapping) and joint audio-visual manipulations (lip synchronization and portrait animation). By benchmarking state-of-the-art detectors, we show that detection difficulty depends on the audio-visual generation pairing, with substantial performance degradation when authentic audio is preserved. Cross-language/demographic evaluation reveals sensitivity varying across detector architectures and training data, while human evaluation reveals that perceived realism and machine-detection difficulty do not necessarily align.

Abstract

Deepfake Detection

Explore similar work

BabelFake: A Multilingual Audio-Visual DeepFake Benchmark

Leveraging large multimodal models for audio-video deepfake detection: a pilot study

ML-ITW: A Multilingual in-the-wild Benchmark for Speech Deepfake Detection