Deepfake speech detectors are typically evaluated as binary classifiers on fixed benchmarks, but deployed systems face evolving attacks, codecs, speakers, and recording conditions, with labels unavailable during operation. This makes it difficult to know when a detector operates in familiar conditions and can be relied on. We propose an unsupervised drift monitor that operates on the internal embeddings of a deployed detector. The monitor compares each input against bona fide and spoof references using regularized Mahalanobis distances. These distances define trusted, ambiguous, and novel regions of the detector embedding space, allowing unfamiliar inputs to be routed for review. We evaluate the monitor on four modern detectors on three datasets. The monitor raises alerts for dataset mismatches and substantially reduces the security-critical case of accepting deepfakes at the cost of reduced automatic coverage. The results also show that drift affects both classes: bona fide recordings can move away from the training reference, increasing alert rates even in in-domain settings. Our findings suggest that deployed deepfake speech detectors should be treated not only as a binary classification, but also as a reliability problem.
Figures & tables
Fig. 1: Proposed deployment pipeline. A fixed speech deepfake detector processes the input recording, produces an utterance embedding z , and an output score. The drift monitor compares z to both bona fide and spoof references using Mahalanobis distance. Inputs assigned to known regions are trusted, while borderline or novel inputs are routed to review.
Fig. 2: Two-reference monitoring space for the MHFA detector trained on AS19. The colored regions and thresholds τb and τs are derived from AS19 train. Filled contours show AS19 eval distributions from the same benchmark, while points show AS5 eval samples from a different benchmark. AS19 eval spoof samples drift toward the bona fide side, while AS5 eval bona fide samples drift substantially across τb into spoof regions, illustrating the data drift captured by the monitor.
Train
Detector
AS19
AS5
ITW
AS19
AASIST
0.13
10.86
6.96
MHFA
1.37
17.47
7.31
AS5
AASIST
7.84
3.60
8.60
MHFA
18.20
4.36
18.18
TABLE I: Detector performance measured by EER (%) across evaluation datasets. The first column shows the training dataset of each detector. Light gray cells mark the in-domain performance.
Train
Detector
AS19
AS5
ITW
CT
Alert
WT
CT
Alert
WT
CT
Alert
WT
AS19
AASIST
82.1
17.9
0.00
73.1
23.2
3.74
24.3
75.5
0.27
MHFA
78.8
21.0
0.20
76.2
20.3
3.42
41.0
58.3
0.67
AS5
AASIST
0.2
99.7
0.13
22.0
78.0
0.01
9.3
90.7
0.01
MHFA
4.1
95.1
0.76
18.9
81.1
0.00
20.5
79.5
0.01
TABLE II: Operational monitor outcomes of correctly trusted (CT), alerted, and wrongly trusted (WT) inputs. All values are percentages. Light-gray cells indicate the in-domain dataset for the corresponding detector.
Train
Detector
AS19
AS5
ITW
Deepfake FAR
Reduction
Deepfake FAR
Reduction
Deepfake FAR
Reduction
AS19
AASIST
3.20%→0.00%
99.95%
0.02%→0.00%
100.00%
0.88%→0.00%
100.00%
MHFA
5.49%→0.22%
96.04%
0.05%→0.00%
100.00%
1.51%→0.00%
100.00%
AS5
AASIST
59.43%→0.15%
99.75%
32.28%→0.02%
99.94%
55.01%→0.02%
99.96%
MHFA
54.08%→0.85%
98.43%
17.29%→0.00%
100.00%
61.72%→0.02%
99.97%
TABLE III: Deepfake false acceptance rate (FAR) before and after applying the monitor. Each Deepfake FAR cell reports the detector-only failure before monitoring, followed by the uncaught failure after monitoring. Reduction is the relative reduction in deepfake FAR, computed before rounding. Light-gray cells indicate the in-domain dataset.
Train
Detector
Metric
AS19
AS5
ITW
Σ
Σ
Σ
Σ
Σ
Σ
AS19
AASIST
Alert
28.1
17.9
49.3
23.2
84.6
75.5
FAR
0.00
0.00
0.00
0.00
0.00
0.00
MHFA
Alert
9.4
21.0
17.7
20.3
45.2
58.3
FAR
0.94
0.22
0.00
0.00
0.00
0.00
AS5
AASIST
Alert
100.0
99.7
95.3
78.0
96.4
90.7
TABLE IV: Effect of covariance regularization on monitor alert rate and deepfake FAR. Empirical covariance Σ is compared with the regularized covariance Σ used in the main results. All values are percentages.
Fig. 3: Category assignment before and after covariance regularization for AASIST trained on AS19. Bars are normalized within each dataset and class. The bars show how the monitor categorized a specific evaluation subset (per class), whereas Table IV reports aggregate alert and FAR rates (per detector).
Speech deepfake detectors usually emit one score per utterance, but a borderline score does not reveal why two examples differ during retrospective error analysis. We ask whether a final score can be calibrated from component fields while keeping those fields visible for inspection. We build a decision record with a passive detector score and a score from a probe applied to a marked copy. It also includes retrieval support held out of the evaluated family, a margin from a support-set profile, and raw neighbor closeness. A cross-fit calibrator combines these fields and two differences between raw scores into one final score. On matched ASVspoof development data, the calibrated record reduces equal error rate (EER) by 3.48 percentage points relative to the fixed retrieval-augmented rule. It reaches 8.43% EER, whereas a passive WavLM baseline reaches 6.71% on the same subset. The record is therefore not the strongest detector in this comparison. Its value is to retain inspectable component fields while producing one scalar score for retrospective diagnosis.
Mengzhe Geng, Yujia Lu, Manuela Kunz +1
National Research Council Canada · The Chinese University of Hong Kong
Audio deepfake detectors are trained to distinguish genuine speech from synthetic speech and often perform well on standard benchmarks. Yet the same detector that achieves less than 1% error on one dataset can see its error rate increase twentyfold when evaluated on a different dataset. We argue that one contributing factor is speaker-identity reliance: standard training corpora correlate speaker identity with the genuine/synthetic label, allowing detectors to partially rely on speaker-related cues rather than synthesis artifacts alone. We propose the Identity Sensitivity Score (ISS), a per-utterance diagnostic that quantifies how much a detector's output changes across different speaker identity contexts. ISS requires no ground-truth labels at inference time and can be computed from the detector score and a pool of reference speaker examples. Across two detectors and two datasets, incorrectly classified utterances have ISS scores 29 to 52 times higher than correctly classified utterances, and ISS alone predicts misclassification with area-under-curve (AUC) up to 0.954. To test whether ISS actually captures identity-sensitive behavior rather than serving only as a proxy for prediction confidence, we apply voice conversion to 500 utterances and measure the resulting detector-score shift. Utterances flagged as identity-sensitive by ISS respond 19 to 30 times more strongly to this manipulation than utterances flagged as stable. These results position ISS as a practical inference-time diagnostic for speaker-dependent failure analysis in audio deepfake detection.
Speech deepfake countermeasures (CMs) are compared almost exclusively by equal error rate (EER), a metric computed at an oracle threshold chosen on the labeled test set. Deployed CMs enjoy no such oracle: a threshold must be fixed in advance and applied to unlabeled target data. We audit this gap with a frozen state-of-the-art SSL-AASIST detector trained on ASVspoof 2019 LA. While its in-domain EER is 0.21%, transferring its LA-calibrated threshold to the In-the-Wild corpus yields a half total error rate (HTER) of 39.5%, with 78.7% of bona fide speech rejected, even though the In-the-Wild EER (11.2%) appears moderate. We then test whether popular unlabeled test-time corrections close this gap, and first prove a simple proposition: any strictly increasing score transform, including z-norm, temperature/shift calibration, and embedding mean alignment under a frozen linear head, cannot change EER. An audit of seven corrections on In-the-Wild and ASVspoof 2021 DF confirms the proposition empirically and exposes two further failure modes: AS-norm with an unlabeled target cohort collapses (EER 11.2% to 60.2%), and pseudo-label calibration that reduces HTER by 38% relative on In-the-Wild degenerates to 50% HTER on DF21, whose spoof prior is 96%. No audited correction reduces EER by more than 1% relative. We recommend reporting HTER at a transferred threshold alongside EER.