A SUPERB-Style Benchmark of Self-Supervised Speech Models for Audio Deepfake Detection
Organizations: Electrical and Computer Engineering, University of Michigan, 4901 Evergreen Rd, Dearborn, MI 48128, USA
Abstract
Self-supervised learning (SSL) has transformed speech processing, with benchmarks such as SUPERB establishing fair comparisons across diverse downstream tasks. Despite it's security-critical importance, Audio deepfake detection has remained outside these efforts. In this work, we introduce Spoof-SUPERB, a benchmark for audio deepfake detection that systematically evaluates 20 SSL models spanning generative, discriminative, and spectrogram-based architectures. We evaluated these models on multiple in-domain and out-of-domain datasets. Our results reveal that large-scale discriminative models such as XLS-R, UniSpeech-SAT, and WavLM Large consistently outperform other models, benefiting from multilingual pretraining, speaker-aware objectives, and model scale. We further analyze the robustness of these models under acoustic degradations, showing that generative approaches degrade sharply, while discriminative models remain resilient. This benchmark establishes a reproducible baseline and provides practical insights into which SSL representations are most reliable for securing speech systems against audio deepfakes.
Figures & tables
| Model | Category | Network | #Params | Stride | Input | Corpus | Pretraining |
|---|---|---|---|---|---|---|---|
| FBANK | Baseline | – | – | 10ms | waveform | – | – |
| APC | Generative | 3-GRU | 4.11M | 10ms | FBANK | LS-360h | F-G |
| VQ-APC | Generative | 3-GRU | 4.63M | 10ms | FBANK | LS-360h | F-G + VQ |
| NPC | Generative | 4-Conv, 4-MaskedConv | 19.38M | 10ms | FBANK | LS-360h | M-G (+VQ) |
| Mockingjay | Generative | 12-Trans | 85M | 10ms | FBANK | LS-960h | time M-G |
| TERA | Generative | 3-Trans | 21M | 10ms | FBANK | LS-960h | time/freq M-G |
| Model | ASV19 LA | ASV21 LA | ASV21 DF | ASV5 Eval | In-the-Wild | DFEval 2024 | Famous Fig. | ASVspoofLD | Mean EER |
|---|---|---|---|---|---|---|---|---|---|
| FBANK | 42.828 | 43.155 | 44.789 | 49.838 | 48.393 | 47.113 | 48.427 | 47.672 | 46.527 |
| APC | 10.075 | 16.335 | 22.276 | 33.311 | 36.889 | 42.662 | 58.402 | 34.345 | 31.787 |
| VQ-APC | 12.155 | 18.872 | 20.217 | 30.581 | 34.860 | 52.173 | 58.544 | 31.799 | 32.400 |
| NPC | 15.243 | 17.619 | 25.239 | 37.868 | 40.986 | 49.843 | 51.979 | 29.758 | 33.567 |
| Mockingjay | 15.430 | 19.798 | 25.312 | 40.217 | 35.848 | 49.800 | 40.975 | 56.033 | 35.427 |
| Mockingjay-960h | 13.801 | 25.525 | 22.584 | 37.866 | 52.387 | 52.130 | 49.953 | 59.283 | 39.191 |
| Model | Avg. Codec EER |
|---|---|
| FBANK (Baseline) | 49.8 |
| APC (Generative) | 33.3 |
| XLS-R (Discriminative) | 13.5 |
| UniSpeech-SAT (Discriminative) | 14.0 |
| WavLM Large (Discriminative) | 18.1 |
| SSAST (Hybrid) | 28.8 |