DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection
Organizations: The Chinese University of Hong Kong, Shenzhen · University at Buffalo, State University of New York · Nanjing University of Posts and Telecommunications
Abstract
Audio-visual deepfakes can spread misinformation and enable impersonation, undermining trust in digital media. However, existing datasets provide limited coverage of forgery operations and their combinations, constraining detector development and systematic evaluation. We introduce Mega-MMDF, containing 100,000 real and 1.2 million forged clips across 22 pipelines spanning modality-specific manipulations, stacked forgeries, and joint audio-video generation. To assess its training value and enable fair comparisons, we develop DeepfakeBench-MM, which standardizes training and evaluation across 8 datasets and 10 detectors. Training-based detectors achieve their highest mean AUC when trained on Mega-MMDF, with gains of up to 0.0784 over the best alternative training dataset for each detector. Experiments on data scale and pipeline coverage further support the value of large, diverse training data. Cross-pipeline evaluations reveal bidirectional and asymmetric generalization. Training on composite forgeries does not necessarily enable detection of their constituent operations in isolation. Our analysis of altered inputs shows that modality masking can increase false alarms and missed detections, and temporal misalignment can raise mean predicted forgery probabilities. Together, these resources provide a foundation for understanding and improving multimodal deepfake detection, with publicly available code and dataset access subject to application and usage terms.
Figures & tables
| Dataset | Publication & Year | Modality | Real Samples | Fake Samples | # Forgery Methods | Forgery Types | Stacked Forgery | Latest Method Year | |||||||
| TTS | VC | PF | FS | EFS | FE | FR | T2AV | ||||||||
| ASVspoof2019LA [ 59 ] | CSL20 | A | 12,483 | 108,978 | 17 | 10 | 7 | - | - | - | - | - | - | ✓ | 19 |
| PartialSpoof [ 60 ] | TASLP23 | A | 12,483 | 108,978 | 17 | - | - | 17 | - | - | - | - | - | ✓ | 19 |
| ASVspoof2021LA [ 61 ] | TASLP23 | A | 16,492 * | 148,148 | 13 | 7 | 6 | - | - | - | - | - | - | ✓ | 19 |
| CD-ADD [ 62 ] | EMNLP24 | A | 25,111 | 120,459 | 5 | 5 | - | - | - | - | - | - | - | ✗ | 24 |
| ASVspoof5 (Track 1) [ 63 ] | CSL26 | A | 188,819 | 815,262 | 24 | 19 | 5 | - | - | - | - | - | - | ✓ | 24 |
| ID | Audio forgery | Visual forgery | Audio-driven FR | Type |
| P0 | Real | Real | – | RARV |
| P1 | Real | FS | – | RAFV |
| P2 | Real | – | ✓ | RAFV |
| P3 | VC | Real | – | FARV |
| P4 | VC | FS | – | FAFV |
| P5 | VC | – | ✓ | FAFV |
| Detector | Training dataset | Test datasets | ||||||||
| FakeAVCeleb | LAV-DF | PolyGlotFake | IDForge | Mega-MMDF | TalkingHeadBench | DigiFakeAV | MMDF | Average | ||
| Baseline | FakeAVCeleb [ 30 ] | 0.8260 | 0.6209 | 0.8094 | 0.6068 | 0.5449 | 0.4223 | 0.5366 | 0.4971 | 0.6080 |
| LAV-DF [ 31 ] | 0.6968 | 0.9898 | 0.7958 | 0.5296 | 0.5829 | 0.5104 | 0.5518 | 0.5487 | 0.6507 | |
| PolyGlotFake [ 32 ] | 0.5963 | 0.5891 | 0.9537 | 0.5285 | 0.6555 | 0.4844 | 0.5350 | 0.4456 | 0.5985 | |
| IDForge [ 33 ] | 0.6558 | 0.5253 | 0.7959 | 0.8358 | 0.6254 | 0.4645 | 0.5296 | 0.4552 | 0.6109 | |
| Mega-MMDF (Ours) | 0.7715 | 0.5093 | 0.9100 | 0.4854 | 0.9958 | 0.8802 | 0.5360 | 0.4925 | 0.6976 | |
| Detector | Training dataset | ||||
| FakeAVCeleb [ 30 ] | LAV-DF [ 31 ] | PolyGlotFake [ 32 ] | IDForge [ 33 ] | Mega-MMDF (Ours) | |
| Baseline | 0.6080 | 0.6507 | 0.5985 | 0.6109 | 0.6976 (+0.0469) |
| AVFF [ 23 ] | 0.6269 | 0.6787 | 0.6527 | 0.6253 | 0.7093 (+0.0306) |
| AVH [ 29 ] | 0.8338 | 0.7514 | 0.8302 | 0.8636 | 0.8721 (+0.0085) |
| FauForensics [ 56 ] | 0.6475 | 0.6981 | 0.6961 | 0.6644 | 0.7346 (+0.0365) |
| FGMDF [ 25 ] | 0.6528 | 0.6324 | 0.6623 | 0.6676 | 0.7132 (+0.0455) |
| Detector | FakeAVCeleb | LAV-DF | PolyGlotFake | IDForge | Mega-MMDF | TalkingHeadBench | DigiFakeAV | MMDF | Average rank | Average AUC |
| Baseline | 0.7715 (6) | 0.5093 (9) | 0.9100 (5) | 0.4854 (10) | 0.9958 (2) | 0.8802 (4) | 0.5360 (8) | 0.4925 (6) | 6.250 | 0.6976 (8) |
| AVFF [ 23 ] | 0.7017 (8) | 0.6833 (5) | 0.8843 (6) | 0.5561 (9) | 0.9819 (5) | 0.9082 (1) | 0.5058 (10) | 0.4533 (10) | 6.750 | 0.7093 (6) |
| AVH [ 29 ] | 0.9597 (1) | 0.8293 (1) | 0.9835 (1) | 0.8343 (1) | 0.9802 (6) | 0.8988 (3) | 0.7080 (1) | 0.7826 (2) | 2.000 | 0.8721 (1) |
| FauForensics [ 56 ] | 0.9220 (3) | 0.7748 (2) | 0.8595 (7) | 0.6518 (4) | 0.9886 (4) | 0.6927 (8) | 0.5255 (9) | 0.4621 (8) | 5.625 | 0.7346 (3) |
| FGMDF [ 25 ] | 0.8019 (5) | 0.4815 (10) | 0.9781 (2) | 0.6560 (3) | 0.9939 (3) | 0.7024 (6) | 0.6158 (3) | 0.4757 (7) | 4.875 | 0.7132 (5) |
| FRADE [ 49 ] | 0.9256 (2) | 0.6874 (4) | 0.9305 (3) | 0.6279 (6) | 0.9963 (1) | 0.9075 (2) | 0.5669 (6) | 0.4597 (9) | 4.125 | 0.7627 (2) |
| Mask audio: video retained | Mask video: audio retained | |||||||||||
| RARV | RAFV | FAFV | RARV | FARV | FAFV | |||||||
| Detector | FPR | MSE | FNR | MSE | FNR | MSE | FPR | MSE | FNR | MSE | FNR | MSE |
| Baseline | {\color[rgb]{1,0,0}+0.351} | {\color[rgb]{1,0,0}+0.278} | {\color[rgb]{1,0,0}+0.150} | {\color[rgb]{1,0,0}+0.095} | {\color[rgb]{1,0,0}+0.118} | {\color[rgb]{1,0,0}+0.093} | {\color[rgb]{1,0,0}+0.350} | {\color[rgb]{1,0,0}+0.280} | ||||
| AVFF [ 23 ] | {\color[rgb]{1,0,0}+0.957} | {\color[rgb]{1,0,0}+0.953} | {\color[rgb]{1,0,0}+0.842} | {\color[rgb]{1,0,0}+0.334} | ||||||||
| AVH [ 29 ] | {\color[rgb]{1,0,0}+0.108} | {\color[rgb]{1,0,0}+0.063} | {\color[rgb]{1,0,0}+0.179} | {\color[rgb]{1,0,0}+0.111} | {\color[rgb]{1,0,0}+0.325} | {\color[rgb]{1,0,0}+0.192} | ||||||
| FGMDF [ 25 ] | {\color[rgb]{1,0,0}+0.231} | {\color[rgb]{1,0,0}+0.140} | {\color[rgb]{1,0,0}+0.099} | {\color[rgb]{1,0,0}+0.071} | ||||||||
| Detector | DP | EO | OAE | MEO | SD AUC |
| Baseline | 0.0257 | 0.5387 | 0.0358 | 0.2064 | 0.0057 |
| AVFF [ 23 ] | 0.0634 | 0.3131 | 0.0529 | 0.0652 | 0.0102 |
| AVH [ 29 ] | 0.0207 | 0.3051 | 0.0197 | 0.0935 | 0.0027 |
| FauForensics [ 56 ] | 0.0735 | 0.2074 | 0.0571 | 0.0596 | 0.0064 |
| FGMDF [ 25 ] | 0.0688 | 0.2605 | 0.0542 | 0.0606 | 0.0061 |
| FRADE [ 49 ] | 0.0185 | 0.2867 | 0.0119 | 0.1229 | 0.0021 |
| Forgery Type | Forgery Method | Year | Publication Venue | Public Engagement | Output Data Format |
| TTS (Text-to-Speech) | YourTTS [ 2 ] | 2022 | ICML 2022 | GitHub : 1,053 stars | Audio |
| SpeechT5 [ 93 ] | 2022 | ACL 2022 | GitHub : 1,450 stars | Audio | |
| OpenVoice V1 [ 94 ] | 2023 | arXiv 2023 | GitHub : 37,532 stars | Audio | |
| XTTS [ 1 ] | 2024 | Interspeech 2024 | GitHub : 46,016 stars | Audio | |
| Vevo [ 95 ] | 2025 | ICLR 2025 | GitHub : 10,295 stars | Audio | |
| VC (Voice Conversion) | FreeVC [ 3 ] | 2023 | ICASSP 2023 | GitHub : 717 stars | Audio |
| Detector | Video | Audio | Duration | |||||||
| Color mode | Size | Mean / Std | # Frames | Stride | SR (Hz) | Remove DC offset | Spectrogram | Mean / Std | ||
| Baseline | RGB | 224 | 16 | 1 | 16,000 | No | MFCC | – | 0.64 s | |
| AVFF [ 23 ] | RGB | 224 | 16 | 5 | 16,000 | Yes | Log-Mel | 3.20 s | ||
| AVH [ 29 ] | Gray | 25 | 1 | 16,000 | No | Log-Mel | – | 1.00 s | ||
| FGMDF [ 25 ] | RGB | 128 | – | 1 | 16,000 | No | – | – | 4.00 s | |
| FRADE [ 49 ] | RGB | 224 | 16 | 1 | 16,000 | No | Log-Mel | – | 0.64 s | |
| Detector / Stage | Optimizer | Learning Rate | Epochs | Batch size |
| Baseline | Adam | 10 | 64 | |
| AVFF (Pretrain) | AdamW | G: | 80 | 16 |
| D: | ||||
| AVFF (Finetune) | AdamW | 10 | 16 | |
| AVH | Adam | 10 | 64 | |
| FGMDF | Adam | 5 | 32 |
| Operation / Parameter | Level 1 | Level 2 | Level 3 | Intended degradation |
| H.264 compression / CRF | 23 | 30 | 37 | Increasing video compression during transcoding. |
| Spatial downsampling / Ratio | Progressively reduced spatial detail during resolution adaptation. | |||
| Frame loss / Probability | 0.05 | 0.10 | 0.20 | Increasing frequency of frozen frames during interrupted delivery. |
| AAC compression / Bitrate (kb/s) | 64 | 32 | 16 | Increasing audio compression under reduced bitrate budgets. |
| Audio dropout / Probability | 0.05 | 0.10 | 0.20 | Increasing frequency of short audio interruptions. |
| Video delay / Offset (ms) | 80 | 160 | 320 | Increasing audio–video offset from unequal buffering or processing delays. |