From Spectrograms to Shallow Speech SSL: Audio-Frontend Design for Generator-Disjoint and Low-Latency Audio--Visual Deepfake Detection

Qiwu Wen; Jahangir Alam; Eric Charton

cs.SDOct 16, 2026

From Spectrograms to Shallow Speech SSL: Audio-Frontend Design for Generator-Disjoint and Low-Latency Audio--Visual Deepfake Detection

Authors: Qiwu Wen, Jahangir Alam, Eric Charton

Organizations: Luqia Technologies

Abstract

Video-mediated conversational systems rely on synchronized speech and facial video, making audio--visual input integrity important for reliable perception. We study audio-frontend design in a fixed MRDF (Modality-Regularization-based DeepFake)-based pipeline with AV-HuBERT visual processing and an MSCT (multi-scale cross-modal transformer) fusion encoder. We compare spectrogram-style log-filterbank features with frozen wav2vec~~2.0 and WavLM representations, dual-SSL fusion, executed depth, and shallow-layer aggregation. Across three training runs, full wav2vec~~2.0 improves mean macro AUROC from $84.19\%$ to $87.19\%$ on generator-disjoint MMDF. Six-layer execution further improves detection, and learned non-uniform aggregation adds $2.53$ percentage points, reaching $90.61\%$ . Shallow execution reduces measured inference latency relative to full-depth wav2vec~2.0 across the tested input durations; first-second evaluation reaches $86.85\%$ macro AUROC. In a single-run comparison, dual-SSL concatenation outperforms either single stream, while one-way cross-attention yields lower detection performance and higher latency than concatenation. Layer-weight analysis shows greater emphasis on early speech layers, while removing the selector's visual input has little effect on detection performance.

Figures & tables

Figure 1: (a) Fixed AV detector. (b) W2V2-L6-weighted and comparison frontends. Audio and stop-gradient visual summaries produce weights for the original layer sequences. Q/K/V label the MHA inputs. Solid arrows carry features; dashed arrows carry weights. Alignment, frontend input/output normalization, biases, dropout, masks, and auxiliary heads are omitted. Dual-SSL comparisons appear in Appendix A.7 .

Frontend	Test mAUC $\uparrow$	Gap $\downarrow$	mEER $\downarrow$	Worst $\uparrow$
Spectrogram	$84.19{\pm}3.40$	13.93	$23.07{\pm}3.40$	$\mathbf{79.68{\pm}4.68}$
W2V2-L12	$87.19{\pm}3.84$	10.75	$19.02{\pm}3.74$	$76.12{\pm}6.96$
W2V2-L6-last	$88.08{\pm}3.82$	9.45	$18.00{\pm}3.54$	$74.28{\pm}9.18$
W2V2-L6-weighted	$\mathbf{90.61{\pm}1.81}$	$\mathbf{7.20}$	$\mathbf{15.17{\pm}1.57}$	$78.58{\pm}5.42$

Table 1: Generator-disjoint MMDF results (%). Values are mean

\pm

standard deviation over three training runs. Gap is validation mAUC minus test mAUC.

Frontend	1 s	8 s	Full	Slope ms/s $\downarrow$
Spectrogram	$\mathbf{23.15}$	33.20	52.05	1.59
W2V2-L12	32.28	35.80	52.93	1.14
W2V2-L6-last	28.61	$\mathbf{31.70}$	$\mathbf{46.70}$	0.99
W2V2-L6-weighted	29.90	33.24	47.37	$\mathbf{0.96}$

Table 2: Duration-dependent p50 model-path latency (ms) on one NVIDIA A40. Full denotes a controlled 19.095-s synchronized input; slope is a descriptive fit over all five durations.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Frontend / fusion	1-s mAUC $\uparrow$	Native mAUC $\uparrow$	Native mEER $\downarrow$	Native Worst $\uparrow$
Spectrogram	80.65	84.30	22.75	80.12
W2V2-L12	83.35	84.18	21.72	69.08
WavLM-L12	–	80.11	27.77	70.56
Dual SSL concat	–	89.00	17.46	76.97
Dual SSL cross-attn.	–	84.72	21.87	69.09
W2V2-L6-last	82.67	84.80	21.03	66.25

Appendix

Table 3: Single-run audio-frontend comparison under the shared MSCT/MRDF detector. Native denotes the available clip (median synchronized AV duration:

2.015

s); 1-s results use the same Spectrogram/W2V2 checkpoints.

Comparison	Run-wise gains	Mean	Positive runs	95% CI
L12 $-$ Spectrogram	$+5.13,+3.97,-.12$	$+2.99$	$2/3$	$[+.06,+5.14]$
L6-last $-$ L12	$+1.29,+.76,+.63$	$+.89$	$3/3$	$[+.38,+1.43]$
L6-weighted $-$ L6-last	$+2.37,+.43,+4.79$	$+2.53$	$3/3$	$[+.48,+4.65]$
L6-weighted $-$ Spectrogram	$+8.78,+5.16,+5.30$	$+6.41$	$3/3$	$[+4.86,+8.67]$

Appendix

Table 4: Paired mAUC gains in percentage points across three training runs. Run-wise gains follow the run order defined at the start of the appendix.

Run	$\alpha_{1}$	$\alpha_{2}$	$\alpha_{3}$	$\alpha_{4}$	$\alpha_{5}$	$\alpha_{6}$	$H_{\mathrm{norm}}$	$N_{\mathrm{eff}}$
1	.269	.216	.181	.120	.109	.106	.958	5.57
2	.386	.231	.146	.095	.076	.066	.878	4.84
3	.430	.208	.151	.096	.061	.054	.846	4.57
Mean	.362	.218	.159	.103	.082	.076	.894	4.99

Appendix

Table 5: Mean unseen-test weights and effective number of active layers for three training runs.

Inference rule	mAUC	$\Delta$ mAUC	Worst AUC
Normal	89.598	—	75.362
Zero visual contribution	89.613	$+.015$	75.416
Shuffled visual summary (mean)	89.615	$+.017$	75.389
Fixed validation-mean weights	89.418	$-.180$	74.949
Uniform weights	88.259	$-1.338$	72.087

Appendix

Table 6: Weighting-rule interventions on one fixed checkpoint. The shuffled result averages five fixed within-group permutations.

Frontend	1 s	2 s	4 s	8 s	19.095 s	Slope	Intercept	$R^{2}$
Spectrogram	23.15	24.92	27.93	33.20	52.05	1.59	21.43	.998
W2V2-L12	32.28	33.29	34.29	35.80	52.93	1.14	29.95	.950
W2V2-L6-last	28.61	29.45	30.67	31.70	46.70	.99	26.64	.951
W2V2-L6-weighted	29.90	30.95	32.01	33.24	47.37	.96	28.18	.957

Appendix

Table 7: Duration-dependent p50 latency (ms).

Figure 2: MSCT-style fusion in this implementation, following Wei et al. (2026) . (a) One of six fusion blocks with pre-LayerNorm residual paths. (b) MSSA applies four temporal convolutions to key-head groups. (c) Audio DCA subtracts cross-query attention from self-query attention after separate softmax operations, then applies audio values; the visual branch is symmetric. Dropout and masks are omitted.

Frontend	1-s mAUC $\uparrow$	1-s Worst $\uparrow$	Native mAUC $\uparrow$	Native Worst $\uparrow$
Spectrogram	80.65	75.07	84.30	80.12
W2V2-L12	83.35	65.54	84.18	69.08
W2V2-L6-last	82.67	63.43	84.80	66.25
W2V2-L6-weighted	86.85	68.84	89.60	75.36

Appendix

Table 8: First-second and native-clip detection using one checkpoint per frontend.

Frontend	mAUC $\uparrow$	mAP $\uparrow$	mEER $\downarrow$	Worst $\uparrow$	p50 ms $\downarrow$	Peak MB $\downarrow$
W2V2-L12	84.18	83.65	21.72	69.08	64.88	937.95
WavLM-L12	80.11	80.23	27.77	70.56	84.11	938.00
Dual concat	89.00	88.57	17.46	76.97	91.24	1317.01
Dual cross-attn.	84.72	85.34	21.87	69.09	92.30	1317.51

Appendix

Table 9: Single-run comparison of full-depth SSL frontends. mAP denotes generator-macro average precision. Latency uses a separate full-clip profiling session from Table 7 .

Fusion	AniPortrait	Hunyuan	MegActor-S
Concatenation	91.23	76.97	98.79
Cross-attention	88.94	69.09	96.15

Appendix

Table 10: Unseen-generator AUROC for the two fusion rules in the same single-run comparison.

Configuration	mAUC $\uparrow$	mEER $\downarrow$	p50 ms $\downarrow$
W2V2-L4-last	90.57	14.89	43.87
W2V2-L6-last	84.80	21.03	45.72
W2V2-L9-last	81.52	23.61	48.91
W2V2-L12	84.18	21.72	64.88
W2V2-L6-uniform	84.96	20.33	45.51
W2V2-L6-global	91.86	14.16	45.69

Appendix

Table 11: Single-run depth and weighting comparison. p50 values come from earlier full-clip profiling rather than the duration-dependent measurements in Table 7 .

Frontend	AniPortrait	Hunyuan	MegActor-S	Worst
Spectrogram	90.38	79.68	82.52	79.68
W2V2-L12	87.21	76.12	98.23	76.12
W2V2-L6-last	92.28	74.28	97.67	74.28
W2V2-L6-weighted	94.04	78.58	99.19	78.58

Appendix

Table 12: Mean AUROC by unseen generator over three training runs.

Comparison	Spearman	Disagree	Error Jaccard	Rescue
L6-weighted vs. L6-last	.852	13.35%	.514	39.24%
L12 vs. Spectrogram	.656	23.05%	.403	46.02%
Dual concat vs. L12	.838	14.11%	.532	36.65%
Dual concat vs. cross-attn.	.815	14.85%	.505	—

Appendix

Table 13: Single-run prediction overlap using validation-selected thresholds. Rescue is the probability that the first model is correct given that the reference model is wrong.

Multimodal Deepfake Detection

See all 29 papers

Multi-Backbone Self-Supervised Ensembles for Audio Deepfake Detection and a Cross-Track Analysis of Generation-Detection Asymmetry
Aug 3, 2026Seunghyun Kim, Junghyun Kim, Jiyoung WooAudio Deepfake Detection Deep Ensembles
Soonchunhyang University, Department of AI and Big Data Engineering, Asan, Republic of Korea
Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTOR
Mar 6, 2026Ajinkya Kulkarni, Sandipana Dowerah, Atharva Kulkarni +2Audio Deepfake Detection Audio Deepfake Detection Robustness
Idiap Research Institute, Switzerland · Tallinn University of Technology, Estonia · MBZUAI, UAE
Leveraging large multimodal models for audio-video deepfake detection: a pilot study
Feb 25, 2026Songjun Cao, Yuqi Li, Yunpeng Luo +2Multimodal Deepfake Detection Deepfake Detection
Tencent YouTu Lab, China · Tencent Youtu Lab, China · Fudan University, China

Aug 3, 2026J/K move · Enter open · S save

Multi-Backbone Self-Supervised Ensembles for Audio Deepfake Detection and a Cross-Track Analysis of Generation-Detection Asymmetry

Seunghyun Kim, Junghyun Kim, Jiyoung Woo

Soonchunhyang University, Department of AI and Big Data Engineering, Asan, Republic of Korea

Audio Deepfake Detection Deep Ensembles Audio Generation

Open paper AskPDF arXiv

This paper describes the participation of team "Go-To-Germany" in the ImageCLEF 2026 Audio Deepfake Detection and Generation task. Our detection system, built on a four-backbone self-supervised learning (SSL) ensemble combining WavLM-Large, Wav2Vec2-XLS-R-300M, ECAPA-TDNN, and x-vector representations, achieved a final score of 0.9522 on the official ImageCLEF 2026 evaluation, with perfect accuracy (1.0000) on participant-generated deepfakes and 0.8875 on the held-out organizer ground-truth real data. For the Generation sub-task, our official team submission, an F5-TTS v1 baseline processed with a uniform reverberation pass and submitted as a deliberate anti-forensic probe, ranked first with a final score of 0.4304 (word error rate (WER) 4.99%, character error rate (CER) 2.07%); details of our four-model program (GLM-TTS, F5-TTS, XTTS v2, CosyVoice3), from which the official entry was drawn, appear in the paper. We present a cross-track analysis revealing a pronounced asymmetry: our detection system identifies 100% of participant-generated deepfakes, while our official generation entry, despite ranking first in the Audio Generation sub-task and evading 61.4% and 56.2% of participant and organizer detectors, attains a Final Score of 0.4304 against 0.9522 on the Detection side. We further report falsification-based ablation experiments (LOSO 56-speaker cross-validation, three-region backbone geometry, bootstrap confidence intervals, and PCA analysis) that motivate our architectural-insurance hypothesis for multi-backbone SSL ensembling. We complement these results with five cross-track insights and five pre-registered falsification experiments connecting generation-side evasion to detection-side design decisions, and we openly report an 11.25% false-positive gap on held-out organizer real recordings as the principal open challenge for deployment.

Abstract

Multimodal Deepfake Detection

Explore similar work

Multi-Backbone Self-Supervised Ensembles for Audio Deepfake Detection and a Cross-Track Analysis of Generation-Detection Asymmetry

Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTOR

Leveraging large multimodal models for audio-video deepfake detection: a pilot study