cs.CVAug 24, 2026
SaveDF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts
Organizations: University of Bucharest Bucharest, Romania · Technical University Munich, MCML Munich, Germany · University of Tübingen, Tübingen AI Center Tübingen, Germany
Abstract
Audio-visual deepfake detection is an actively studied topic, where one of the main challenges is to develop detectors able to generalize across deepfake generation methods. We conjecture that overfitting can be mitigated by extracting multiple high-level cues from the available audio and visual modalities via pre-trained models. We therefore assemble a wide variety of pre-trained models to extract features that encode mouth movements, face parsing, facial expressions, head pose, gaze tracking, heart rate, audio emotion and speech activity. We further integrate both unimodal and multimodal cues via a Mixture-of-Experts (MoE) backbone to detect deepfakes. We perform in-domain and cross-domain experiments on five benchmarks for deepfake detection (MAVOS-DD, AVLips, PolyGlotFake, BioDeepAV, FakeAVCeleb) to compare our framework (DF-MoE) with state-of-the-art methods. Our results indicate that DF-MoE obtains superior deepfake detection results, surpassing all competing methods. We release our code at https://github.com/vladhondru25/DF-MoE.
Figures & tables
Figure 1: Overview of the proposed pipeline for deepfake detection. First, we employ multiple pre-trained models to extract high-level semantic cues. Then, a lightweight Mixture-of-Experts (MoE) transformer aggregates the extracted signals and learns to classify audio-visual inputs as real or fake. Our design prevents overfitting to a specific dataset by keeping the pre-trained models frozen, while only training the lightweight MoE on the target task.
Figure 2: Detailed overview of our DF-MoE framework for deepfake detection. Face detection and tracking are applied to obtain face tracks. Next, multiple pre-trained models (frozen) are employed to extract high-level semantic cues (features). The extracted features are encoded into tokens via specialized encoders (trainable adapters), ensuring consistent dimensionality across different cues. The resulting tokens are processed by a sparse MoE transformer (trainable). Best viewed in color.
| Method | Closed-set | Open-set model | Open-set lang. | Open-set full | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| mAP | AUC | acc | mAP | AUC | acc | mAP | AUC | acc | mAP | AUC | acc | ||||
| TALL [ 105 ] | |||||||||||||||
| MRDF [ 121 ] | |||||||||||||||
| AVFF [ 75 ] | {\color[rgb]{0.8633,0.4688,0}86.58} | ||||||||||||||
| AVH-Align [ 89 ] | |||||||||||||||
| AVH-Align [ 89 ] | |||||||||||||||
Table 1: Results on MAVOS-DD obtained by TALL [ 105 ] , MRDF [ 121 ] , AVFF [ 75 ] , AVH-Align [ 89 ] , AVH-Align [ 89 ] , DF-Linear, and our DF-MoE. The best performing method is highlighted in blue bold , and the second-best in orange . Our DF-MoE outperforms all previous state-of-the-art methods, regardless of the evaluation setup.
| Method | AVLips | ||
|---|---|---|---|
| mAP | AUC | acc | |
| RealForensics [ 31 ] | - | ||
| LipForensics [ 32 ] | - | ||
| LipFD [ 64 ] | \mathbf{{\color[rgb]{0,0.3125,0.7852}95.27}} | ||
| AVH-Align [ 89 ] | - | - | |
| AVFF [ 75 ] | |||
Table 2: In-domain results on AVLips [ 64 ] obtained by state-of-the-art models vs. DF-MoE. The best performing method is highlighted in blue bold , and the second-best in orange . Our DF-MoE outperforms all previous state-of-the-art methods in terms of mAP and AUC.
| rPPG | Face | HP+Gaze | AV | AV | AVFF+ | mAP | AUC | acc |
|---|---|---|---|---|---|---|---|---|
| segmentation | emotion | transformer | Effort | |||||
| ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | |||
| ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | |||
| ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | |||
| ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | |||
| ✗ | ✗ | ✗ | ✗ | ✓ | ✗ |
Table 3: Ablation study on the impact of each feature type on the final performance of DF-MoE on MAVOS-DD (open-set full). The best individual components are AV Transformer and AVFF. However, the complementary cues brought by the other pre-trained models (HP+Gaze, rPPG, audio-visual emotion, face segmentation) bring consistent performance gains, all contributing to the final performance of DF-MoE.
| Year | Method | PolyGlotFake [ 38 ] | BioDeepAV [ 18 ] | Vox+FakeAVCeleb [ 49 ] | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| mAP | AUC | acc | mAP | AUC | acc | mAP | AUC | acc | ||||
| 2018 | MesoNet [ 1 ] | - | - | - | - | - | - | - | - | |||
| MesoInception [ 1 ] | - | - | - | - | - | - | - | - | ||||
| DSP-FWA [ 60 ] | - | - | - | - | - | - | - | - | ||||
| 2019 | XceptionNet [ 80 ] | - | - | - | - | - | - | - | ||||
| EfficienNet-B4 [ 97 ] | - | - | - | - | - | - | - | - | ||||
Table 4: Cross-dataset evaluation on PolyGlotFake, BioDeepAV and Vox+FakeAVCeleb. The best performing method is highlighted in blue bold , and the second-best in orange . DF-MoE obtains the best performance on both datasets in terms of AUC and acc. Results of methods marked with an asterisk are reproduced using publicly available code.
Figure 3: Examples of videos correctly classified by DF-MoE, alongside the corresponding attention scores of each high-level cue. Best viewed in color.
| #GPUs | Average | Face detection & | Feature | MoE | Total |
|---|---|---|---|---|---|
| video duration | tracking | extraction | |||
| 1 | 10.97 | 3.35 | 8.47 | 1.16 | 12.98 |
| 2 | 10.97 | 3.35 | 4.65 | 1.16 | 9.16 |
Table 5: Average video duration and stage-wise inference times (in seconds) for DF-MoE. The values are estimated across 20 videos on a machine with AMD Ryzen Threadripper 9960X 24-core CPU and Nvidia RTX 5090 GPU (32GB VRAM).
| Dropout Ratio | #Experts | mAP | AUC | acc |
|---|---|---|---|---|
Table A1: Ablation study on the expert dropout ratio ( ) and the number of experts ( ) included in DF-MoE. The study is conducted on the open-set full protocol of MAVOS-DD [ 19 ] .
Figure 1: Latent space of DF-MoE before introducing the CRO loss.
| CRO Loss | mAP | AUC | acc |
|---|---|---|---|
| ✗ | |||
| ✓ |
Table A2: Results of DF-MoE on PolyGlotFake [ 38 ] , before and after introducing the proposed contractive-repulsive objective (CRO).
| AVFF+ | Face | HP+Gaze | AV | AV | rPPG | mAP | AUC | acc |
|---|---|---|---|---|---|---|---|---|
| Effort | segmentation | emotion | transformer | |||||
| ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | |||
| ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | |||
| ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | |||
| ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | |||
| ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
Table A3: Ablation study on combinations of AVFF+Effort features and other high-level features included in DF-MoE. Results are reported on PolyGlotFake [ 38 ] .
| Method | mAP | AUC | acc |
|---|---|---|---|
| DF-MoE (AVFF full) | |||
| DF-MoE (AVFF+Effort) |
Table A4: Full vs. parameter-efficient training of AVFF encoder on MAVOS-DD (open-set full). Fine-tuning based on Effort [ 107 ] surpasses full fine-tuning.
| Method | AUC |
|---|---|
| F3Net [ 76 ] | 0.789 |
| CORE [ 72 ] | 0.809 |
| RECCE [ 8 ] | 0.823 |
| UCF [ 108 ] | 0.837 |
| LSDA [ 106 ] | 0.875 |
| DF-MoE (ours) | 0.834 |
Table A5: Cross-dataset evaluation on Celeb-DF (v2) [ 61 ] , a video-only dataset. The results demonstrate that DF-MoE obtains competitive results even when the audio modality is missing.
Figure 4: Average attention scores across datasets and class labels.
Figure 5: Examples of videos incorrectly classified by DF-MoE, alongside the corresponding attention scores of each feature type. Best viewed in color.
Figure 6: rPPG.