Deepfake speech utterances can be forged by replacing one or more words in a bona fide utterance with semantically different words synthesized with speech-generative models. While a dedicated synthetic word detector could be developed, we developed a cost-effective method that fine-tunes a pre-trained Whisper model to detect synthetic words while transcribing the input utterance via next-token prediction. We further investigate using partially vocoded utterances as the fine-tuning data, thus reducing the cost of data collection. Our experiments demonstrate that, on in-domain test data, the fine-tuned Whisper yields low synthetic-word detection error rates and transcription error rates. On out-of-domain test data with synthetic words produced with unseen speech-generative models, the fine-tuned Whisper remains on par with a dedicated ResNet-based detection model; however, the overall performance degradation calls for strategies to improve its generalization capability.
Figures & tables
Figure 1: Illustration of fine-tuned Whisper for speech-to-text transcription and synthetic word detection. <TOF> and <EOF> denote tokens surrounding synthetic word.
Name
#. of utterances
Language(s)
Domain
Generator of synthetic words
Train data
Ft.Voc
60,596
en, es, fr, it, de
Audiobook
HiFi-GAN, NSF, NSF+GAN, WaveGlow, WORLD, GL
Ft.TTS
60,596
en
Audiobook
JETS, YourTTS, XTTS, SoVITS, CosyVoice, ElevenLab
Ft.V+T
60,596
en, es, fr, it, de
Audiobook
All above
Test data
E.Voc
3,000
en, es, fr, it, de
Audiobook
HiFi-GAN, NSF, NSF+GAN, WaveGlow, WORLD, GL
E.TTS
3,000
en
Audiobook
JETS, YourTTS, XTTS, SoVITS, CosyVoice, ElevenLab
E.AV1M
3,000
en
Youtube
YourTTS, VITS
Table 1: Statistics of fine-tuning ( Ft. ) and test ( E. ) data sets. Generators using full speech-generative systems are in italics.
Data set
WER(%) Whisper
FAR (%)
FRR (%)
Test
Fine-tune
Whisper
ResNet
Whisper
ResNet
E.Voc
Pre-trained
23.89
Ft.Voc
0.87
7.22
7.15
0.52
3.81
Ft.V+T
2.18
8.45
9.80
1.01
5.32
Ft.TTS
35.11
18.86
76.42
78.59
9.08
E.TTS
Pre-trained
8.13
Table 2: Results on E.Voc and E.TTS . Bold text indicates the best result in each column and each test set.
Figure 2: Analysis of synthetic-word detection error rates based on word duration. Black and red profiles correspond to fine-tuned Whisper using Ft.Voc and Ft.TTS , respectively.
FAR (%)
FRR (%)
Ft.Voc
Ft.TTS
Ft.Voc
Ft.TTS
en
8.38
46.28
0.50
40.97
fr
10.41
11.54
0.78
91.19
es
5.59
10.76
0.36
89.36
de
5.47
8.97
0.40
91.80
it
6.30
13.03
0.53
88.12
Table 3: Decomposed results over languages using pre-trained Whisper (left) or fine-tuned on Ft.Voc (right).
Data set
WER(%) Whisper
FAR (%)
FRR (%)
Test
Fine-tune
Whisper
ResNet
Whisper
ResNet
E.AV1M
Pre-trained
14.72
Ft.Voc
23.17
39.98
33.70
7.70
17.69
Ft.V+T
21.23
15.98
27.63
39.42
27.04
Ft.TTS
20.47
16.04
11.14
59.97
36.44
E.PE
Pre-trained
3.64
Table 4: Results on test sets manipulated using DNN-based speech synthesis and editing systems. Within each column, value with darker shading indicates worse performance.
Provenance watermarking is increasingly treated as a safeguard for synthetic speech, whether built directly into speech-generation models such as Chatterbox, provided through dedicated techniques such as AudioSeal, or deployed by commercial platforms such as ElevenLabs. We identify a previously uncharacterized liability: when synthetic speech is watermarked and human speech is not, detectors trained alongside latch onto the watermark as a spurious "watermark => fake" shortcut. This single feature yields three coupled failures: generalization degradation (model performance deteriorates on unseen data), strip-to-evade (a watermarked fake escapes once unwatermarked), and mark-to-frame (watermarking a real voice flags it as fake). In a controlled white-box experiment, a watermark-trained detector shows all three (for example, mark-to-frame lifts Equal Error Rate from 16% to 75%). In a black-box test of a commercial API, we show that adding a watermark to real speech disguises it as fake. However, this shortcut is fixable: retraining with the watermark on both classes decorrelates it and restores clean behavior. We release experiment data as a paired clean-versus-watermarked corpus (WASP).
Advances in speech synthesis have made deepfake audio highly realistic. Earlier studies reported 70-80% human detection accuracy, but relied primarily on older synthesizers. We compare human detection for three selected voice synthesis tools released in 2019, 2022, and 2024 with 82 IT professionals, and benchmark humans against six pretrained detectors on the same material. For fully synthetic speech (full spoofs), the F1 score drops from about 90% for RTVC and YourTTS to 48% for ElevenLabs, although listeners were explicitly warned that deepfakes were present. For partial spoofing, where only one sentence of an utterance is altered, strict accuracy falls to 9%, and listeners classify the synthetic sentence as bona fide 77% of the time. Humans and detectors fail in complementary ways, and neither reliably localizes short manipulations. Additionally, listeners increasingly mislabel bona fide speech as fake, eroding trust in unmanipulated audio. These findings show that human perception alone is unreliable for the selected modern and partial-spoof conditions and motivate procedural verification, provenance, watermarking, and segment-level detection.
Audio deepfakes generated by neural text-to-speech and voice-cloning systems threaten speaker verification and public discourse at scale. The core challenge is cross-dataset generalization: detectors trained on one synthesis pipeline collapse on unseen forgeries. We argue that this failure is primarily because of structural synthetic speech artifacts which are multi-timescale trajectory anomalies. Though every existing detector aggregates a fixed-window frame statistics, this misaligns the architecture with the signal. We propose FlowFake, a Liquid Time-Constant (LTC) architecture whose hidden state evolves via a learned ODE, with per-neuron adaptive time constants simultaneously resolving spectral (10ms) and prosodic (2s) cues. At only 34K parameters FlowFake achieves formal BIBO stability and O(dt^4) integration error. On a four-dataset cross domain benchmark (ASVspoof2019-LA, FakeOrReal, InTheWild, MLAAD), FlowFake reaches 75.29% on ASVspoof2019 trained only on FakeOrReal and 79.97% trained only on MLAAD. It outperforms RawGAT-ST and Whisper-DF on every evaluated pair and matching SSL Wav2vec2 (300x larger) at 0.01% of its parameter count. The source code is available on : https://github.com/GhostRider2023/FlowFake