Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTOR

Ajinkya Kulkarni; Sandipana Dowerah; Atharva Kulkarni; Tanel Alumäe; Mathew Magimai Doss

cs.SDMar 6, 2026

Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTOR

Authors: Ajinkya Kulkarni, Sandipana Dowerah, Atharva Kulkarni, Tanel Alumäe, Mathew Magimai Doss

Organizations: Idiap Research Institute, Switzerland · Tallinn University of Technology, Estonia · MBZUAI, UAE

Abstract

Self-supervised learning (SSL) underpins modern audio deepfake detection, yet most prior work centers on a single large wav2vec2-XLSR backbone, leaving compact under studied. We present RAPTOR, Representation Aware Pairwise-gated Transformer for Out-of-domain Recognition a controlled study of compact SSL backbones from the HuBERT and WavLM within a unified pairwise-gated fusion detector, evaluated across 14 cross-domain benchmarks. We show that multilingual HuBERT pre-training is the primary driver of cross-domain robustness, enabling 100M models to match larger and commercial systems. Beyond EER, we introduce a test-time augmentation protocol with perturbation-based aleatoric uncertainty to expose calibration differences invisible to standard metrics: WavLM variants exhibit overconfident miscalibration under perturbation, whereas iterative mHuBERT remains stable. These findings indicate that SSL pre-training trajectory, not model scale, drives reliable audio deepfake detection.

Figures & tables

Family	Model	Params	Pre-training data
HuBERT	Base	95M	LibriSpeech 960 h
HuBERT	mHuBERT-Iter1	95M	90K h, 147 lang. (iter. 1)
HuBERT	mHuBERT-Iter2	95M	90K h, 147 lang. (iter. 2)
HuBERT	mHuBERT-Final	95M	90K h, 147 lang. (final)
WavLM	Base	94M	LibriSpeech 960 h
WavLM	Base+	94M	60K h + GigaSpeech + VoxPopuli

Table 1: SSL backbone families and pre-training data. Lang. = languages.

Figure 1: RAPTOR framework. SSL layer representations are progressively fused by pairwise and hierarchical softmax gates, followed by attention pooling and a binary classifier.

	System	ITW	ASVspoof				FoR	CodecFake	ADD22		ADD23		DFADD	LibriSeVoc	SONAR
			19	21LA	21DF	24			T1	T3	R1	R2
SOTA systems	W2V2-AASIST	2.12	1.88	6.01	1.78	11.31	3.80	13.35	22.76	3.62	12.43	17.44	0.23	0.08	1.07
	W2V2-TCM	1.65	3.49	6.95	2.91	13.81	4.02	13.70	23.63	4.16	11.61	16.89	0.15	0.81	1.83
	ResembleAI-2B [P]	3.94	1.32	1.64	3.79	16.29	1.36	33.04	28.21	6.11	21.07	28.27	0.00	1.62	2.98
	MoLEX [P]	0.03	0.28	6.32	1.88	15.87	0.17	32.40	31.93	3.66	11.11	19.02	6.46	3.30	0.91
	DF-Arena 500M	1.76	1.09	4.23	3.30	12.39	2.30	6.36	23.98	2.77	7.47	12.30	0.00	0.12	1.90

Table 2: EER (%) of RAPTOR and SOTA systems trained under Protocol 2 (multi-dataset) across 14 cross-domain benchmarks. T = track, R = round, [P] = proprietary system (architecture and training details not publicly disclosed; included for reference only). Bold = best overall; underline = best among 100M systems.

System	Params	Protocol 1 (ASV19 train)			Protocol 2 (DF Arena Model)
System	Params	ITW	FoR	ASV19	Avg EER	Pooled EER
W2V2-AASIST	317M	11.19	7.46	0.22	6.99	12.46
W2V2-TCM	319M	7.79	10.68	0.18	7.54	12.88
ResembleAI-2B	2000M	-	-	-	10.83	12.74
MoLEX	376M	-	-	-	9.51	12.40
DF-Arena 500M	500M	-	-	-	5.78	10.88

Table 3: Systems trained on ASVspoof 2019 only (Protocol 1), evaluated in-domain and on out-of-domain sets; DF Arena aggregate metrics are from multi-dataset training (Protocol 2). Avg EER = mean over 14 sets; Pooled EER = single global threshold across all 14 sets.

	ITW		FoR		ASV19
System	$\Delta$ EER	$U_{\mathrm{ale}}$	$\Delta$ EER	$U_{\mathrm{ale}}$	$\Delta$ EER	$U_{\mathrm{ale}}$
W2V2-TCM	$+$ 1.70	0.299	$+$ 44.79	0.247	$+$ 1.06	0.214
W2V2-AASIST	$+$ 1.83	0.227	$+$ 45.03	0.263	$+$ 1.57	0.124
HuBERT-Base	$+$ 0.99	0.384	$+$ 42.51	0.418	$+$ 0.48	0.289
mHuBERT-Iter1	$+$ 0.89	0.367	$+$ 42.95	0.405	$+$ 0.40	0.354
mHuBERT-Iter2	$+$ 0.38	0.321	$+$ 45.72	0.321	$+$ 1.66	0.277

Table 4:

\Delta

EER (%)

\downarrow

and mean aleatoric uncertainty (

U_{\mathrm{ale}}

)

\uparrow

across

K{=}3

TTA views (VoIP codec, additive noise, speed-pitch perturbation) for all systems on ITW, FoR, and ASV19, trained on Protocol 2 .

\Delta

EER is the TTA ensemble EER minus clean-inference EER; Systems exhibiting high

\Delta

EER alongside low

U_{\mathrm{ale}}

demonstrate miscalibration under perturbation.

Figure 2: Pairwise gate maps $\alpha_{p,1}(t)$ for a spoofed utterance from ITW produced by mHuBERT-Iter2. The $x$ -axis denotes time frames (50 ms resolution), the $y$ -axis the SSL layer-pair index ( $p=1\ldots 6$ ). Spoof utterances activate lower-to-middle layer pairs (indices 2–4) more strongly, suggesting synthesis artifacts concentrate at earlier stages of the SSL hierarchy.

Explore similar work

Probing-Guided Layer Selection from Self-Supervised Speech Models for Generalizable Audio Deepfake Detection
Jun 29, 2026Marjan Beheshti, Majid Rostami, Bo ChenAudio Deepfake Detection Domain Generalization
Department of Computer Science, Michigan Technological University, Houghton, MI, USA
A SUPERB-Style Benchmark of Self-Supervised Speech Models for Audio Deepfake Detection
Mar 2, 2026Hashim Ali, Nithin Sai Adupa, Surya Subramani +1Audio Deepfake Detection Deepfake Detection
Electrical and Computer Engineering, University of Michigan, 4901 Evergreen Rd, Dearborn, MI 48128, USA
REIMU: Efficient Heterogeneous Hierarchical Reasoning for SSL-Based Speech Deepfake Detection
Aug 1, 2026Kwok-Ho Ng, Tingting Song, Bingwen Feng +1Audio Deepfake Detection ASVspoof
College of Cyber Security, Jinan University, Guangzhou, China

Jun 29, 2026J/K move · Enter open · S save

Probing-Guided Layer Selection from Self-Supervised Speech Models for Generalizable Audio Deepfake Detection

Marjan Beheshti, Majid Rostami, Bo Chen

Department of Computer Science, Michigan Technological University, Houghton, MI, USA

Audio Deepfake Detection Domain Generalization

Open paper AskPDF arXiv

Audio deepfake detection systems often fail to generalize across domains because they rely on features tied to specific attacks or recording conditions. Self-supervised speech models offer rich multi-layer representations, yet existing approaches either use a single layer or fuse all layers indiscriminately, and only reveal layer importance after training. We propose a model-agnostic, two-stage methodology that identifies informative depth zones before any task-specific model is trained. In the first stage, lightweight XGBoost probes evaluate each transformer layer's cross-domain discriminative power, producing a layer ranking. In the second stage, a compact neural classifier fuses only the selected layers through per-layer attention pooling and a shared bottleneck projection, while the backbone remains frozen. Applied across three backbones, the probing reveals two key findings. First, informative layers cluster in depth zones rather than at uniquely optimal positions: within-zone substitutions fall within multi-seed noise, while zone violations degrade performance by up to 5x. Second, the probing produces backbone-specific selections rather than a fixed layer recipe. On XLS-R-300M, four probing-selected layers with 1.34M trainable parameters achieve 4.94 +/- 0.32% equal error rate on In-The-Wild and 5.07% cross-domain average over four shared datasets, a 28% relative improvement over the best prior frozen-backbone result (Xiao and Vu, 2025) using all 25 layers with identical training data.

Abstract

Explore similar work

Probing-Guided Layer Selection from Self-Supervised Speech Models for Generalizable Audio Deepfake Detection

A SUPERB-Style Benchmark of Self-Supervised Speech Models for Audio Deepfake Detection

REIMU: Efficient Heterogeneous Hierarchical Reasoning for SSL-Based Speech Deepfake Detection