Unsupervised Data Drift Monitoring for Deployed Deepfake Speech Detectors

Lukáš Prokeš; Vojtěch Staněk; Anton Firc; Kamil Malinka

cs.SDOct 12, 2026

Unsupervised Data Drift Monitoring for Deployed Deepfake Speech Detectors

Authors: Lukáš Prokeš, Vojtěch Staněk, Anton Firc, Kamil Malinka

Abstract

Deepfake speech detectors are typically evaluated as binary classifiers on fixed benchmarks, but deployed systems face evolving attacks, codecs, speakers, and recording conditions, with labels unavailable during operation. This makes it difficult to know when a detector operates in familiar conditions and can be relied on. We propose an unsupervised drift monitor that operates on the internal embeddings of a deployed detector. The monitor compares each input against bona fide and spoof references using regularized Mahalanobis distances. These distances define trusted, ambiguous, and novel regions of the detector embedding space, allowing unfamiliar inputs to be routed for review. We evaluate the monitor on four modern detectors on three datasets. The monitor raises alerts for dataset mismatches and substantially reduces the security-critical case of accepting deepfakes at the cost of reduced automatic coverage. The results also show that drift affects both classes: bona fide recordings can move away from the training reference, increasing alert rates even in in-domain settings. Our findings suggest that deployed deepfake speech detectors should be treated not only as a binary classification, but also as a reliability problem.

Figures & tables

Panel of Figure 1 — Fig. 1: Proposed deployment pipeline. A fixed speech deepfake detector processes the input recording, produces an utterance embedding $z$ , and an output score. The drift monitor compares $z$ to both bona fide and spoof references using Mahalanobis distance. Inputs assigned to known regions are trusted, while borderline or novel inputs are routed to review.

Fig. 2: Two-reference monitoring space for the MHFA detector trained on AS19. The colored regions and thresholds $\tau_{b}$ and $\tau_{s}$ are derived from AS19 train. Filled contours show AS19 eval distributions from the same benchmark, while points show AS5 eval samples from a different benchmark. AS19 eval spoof samples drift toward the bona fide side, while AS5 eval bona fide samples drift substantially across $\tau_{b}$ into spoof regions, illustrating the data drift captured by the monitor.

Train	Detector	AS19	AS5	ITW
AS19	AASIST	0.13	10.86	6.96
AS19	MHFA	1.37	17.47	7.31
AS5	AASIST	7.84	3.60	8.60
AS5	MHFA	18.20	4.36	18.18

TABLE I: Detector performance measured by EER (%) across evaluation datasets. The first column shows the training dataset of each detector. Light gray cells mark the in-domain performance.

Train	Detector	AS19			AS5			ITW
Train	Detector	CT	Alert	WT	CT	Alert	WT	CT	Alert	WT
AS19	AASIST	82.1	17.9	0.00	73.1	23.2	3.74	24.3	75.5	0.27
AS19	MHFA	78.8	21.0	0.20	76.2	20.3	3.42	41.0	58.3	0.67
AS5	AASIST	0.2	99.7	0.13	22.0	78.0	0.01	9.3	90.7	0.01
AS5	MHFA	4.1	95.1	0.76	18.9	81.1	0.00	20.5	79.5	0.01

TABLE II: Operational monitor outcomes of correctly trusted (CT), alerted, and wrongly trusted (WT) inputs. All values are percentages. Light-gray cells indicate the in-domain dataset for the corresponding detector.

Train	Detector	AS19		AS5		ITW
Train	Detector	Deepfake FAR	Reduction	Deepfake FAR	Reduction	Deepfake FAR	Reduction
AS19	AASIST	$3.20\%\rightarrow 0.00\%$	99.95%	$0.02\%\rightarrow 0.00\%$	100.00%	$0.88\%\rightarrow 0.00\%$	100.00%
AS19	MHFA	$5.49\%\rightarrow 0.22\%$	96.04%	$0.05\%\rightarrow 0.00\%$	100.00%	$1.51\%\rightarrow 0.00\%$	100.00%
AS5	AASIST	$59.43\%\rightarrow 0.15\%$	99.75%	$32.28\%\rightarrow 0.02\%$	99.94%	$55.01\%\rightarrow 0.02\%$	99.96%
AS5	MHFA	$54.08\%\rightarrow 0.85\%$	98.43%	$17.29\%\rightarrow 0.00\%$	100.00%	$61.72\%\rightarrow 0.02\%$	99.97%

TABLE III: Deepfake false acceptance rate (FAR) before and after applying the monitor. Each Deepfake FAR cell reports the detector-only failure before monitoring, followed by the uncaught failure after monitoring. Reduction is the relative reduction in deepfake FAR, computed before rounding. Light-gray cells indicate the in-domain dataset.

Train	Detector	Metric	AS19		AS5		ITW
Train	Detector	Metric	$\mathbf{\Sigma}$	$\mathbf{\widehat{\Sigma}}$	$\mathbf{\Sigma}$	$\mathbf{\widehat{\Sigma}}$	$\mathbf{\Sigma}$	$\mathbf{\widehat{\Sigma}}$
AS19	AASIST	Alert	28.1	17.9	49.3	23.2	84.6	75.5
	AASIST	FAR	0.00	0.00	0.00	0.00	0.00	0.00
	MHFA	Alert	9.4	21.0	17.7	20.3	45.2	58.3
	MHFA	FAR	0.94	0.22	0.00	0.00	0.00	0.00
AS5	AASIST	Alert	100.0	99.7	95.3	78.0	96.4	90.7

TABLE IV: Effect of covariance regularization on monitor alert rate and deepfake FAR. Empirical covariance

\mathbf{\Sigma}

is compared with the regularized covariance

\mathbf{\widehat{\Sigma}}

used in the main results. All values are percentages.

Fig. 3: Category assignment before and after covariance regularization for AASIST trained on AS19. Bars are normalized within each dataset and class. The bars show how the monitor categorized a specific evaluation subset (per class), whereas Table IV reports aggregate alert and FAR rates (per detector).

Explore similar work

From Scores to Evidence: Auditable Decisions Can Improve Speech Deepfake Detection
Sep 8, 2026Mengzhe Geng, Yujia Lu, Manuela Kunz +1Audio Deepfake Detection Explainable Deepfake Detection
National Research Council Canada · The Chinese University of Hong Kong
Probing Speaker Identity Sensitivity in Audio Deepfake Detectors
Jul 23, 2026Daniyal Kabir Dar, Arun RossAudio Deepfake Detection Shortcut Learning
Michigan State University, USA
When EER Hides Deployment Failure: Auditing Threshold Transfer and Unlabeled Score Calibration for Speech Deepfake Detectors
Jun 19, 2026Jingwen Zhou, Mingzhe WangAudio Deepfake Detection ASVspoof
Xidian University, Xi’an, China

Sep 8, 2026J/K move · Enter open · S save

From Scores to Evidence: Auditable Decisions Can Improve Speech Deepfake Detection

Mengzhe Geng, Yujia Lu, Manuela Kunz, Patrick Littell

National Research Council Canada · The Chinese University of Hong Kong

Audio Deepfake Detection Explainable Deepfake Detection ASVspoof

Open paper AskPDF arXiv

Speech deepfake detectors usually emit one score per utterance, but a borderline score does not reveal why two examples differ during retrospective error analysis. We ask whether a final score can be calibrated from component fields while keeping those fields visible for inspection. We build a decision record with a passive detector score and a score from a probe applied to a marked copy. It also includes retrieval support held out of the evaluated family, a margin from a support-set profile, and raw neighbor closeness. A cross-fit calibrator combines these fields and two differences between raw scores into one final score. On matched ASVspoof development data, the calibrated record reduces equal error rate (EER) by 3.48 percentage points relative to the fixed retrieval-augmented rule. It reaches 8.43% EER, whereas a passive WavLM baseline reaches 6.71% on the same subset. The record is therefore not the strongest detector in this comparison. Its value is to retain inspectable component fields while producing one scalar score for retrospective diagnosis.

Abstract

Explore similar work

From Scores to Evidence: Auditable Decisions Can Improve Speech Deepfake Detection

Probing Speaker Identity Sensitivity in Audio Deepfake Detectors

When EER Hides Deployment Failure: Auditing Threshold Transfer and Unlabeled Score Calibration for Speech Deepfake Detectors