Advances in speech synthesis have made deepfake audio highly realistic. Earlier studies reported 70-80% human detection accuracy, but relied primarily on older synthesizers. We compare human detection for three selected voice synthesis tools released in 2019, 2022, and 2024 with 82 IT professionals, and benchmark humans against six pretrained detectors on the same material. For fully synthetic speech (full spoofs), the F1 score drops from about 90% for RTVC and YourTTS to 48% for ElevenLabs, although listeners were explicitly warned that deepfakes were present. For partial spoofing, where only one sentence of an utterance is altered, strict accuracy falls to 9%, and listeners classify the synthetic sentence as bona fide 77% of the time. Humans and detectors fail in complementary ways, and neither reliably localizes short manipulations. Additionally, listeners increasingly mislabel bona fide speech as fake, eroding trust in unmanipulated audio. These findings show that human perception alone is unreliable for the selected modern and partial-spoof conditions and motivate procedural verification, provenance, watermarking, and segment-level detection.
Figures & tables
Figure 1: Structure of one of the nine speaker test sets. Each recording comprises four sentence-level segments that participants judged independently as bona fide or synthetic.
Training data
Front-end
Back-end
EER
ASVspoof 2019 LA
XLS-R
AASIST
0.68%
MHFA
0.57%
ASVspoof 5
XLS-R
AASIST
4.16%
MHFA
5.34%
WavLM
AASIST
4.06%
MHFA
5.26%
Table 1: Automated detector baselines. EER is reported on the respective source benchmark used for training (eval subset of ASVspoof 2019 LA or ASVspoof 5), not on our study material.
Condition
Items
Judgments
α
All sentence-level judgments
252
17,233
0.631
Full spoof
108
7,392
0.502
Partial spoof
108
7,429
0.528
Full spoof, YourTTS
36
2,569
0.035
Full spoof, RTVC
36
2,514
0.011
Full spoof, ElevenLabs
36
2,309
0.155
Table 2: Inter-rater agreement for sentence-level authenticity judgments. Uncertain and missing responses were excluded before computing Krippendorff’s α . Items denote unique sentence-level segments.
Figure 2: Human detection performance (mean F1) for full and partial spoofs from the three selected synthesizers. Detection remains high for RTVC and YourTTS and is substantially lower for ElevenLabs. The dotted line marks an F1 reference value of 0.5.
Figure 3: Human detection per speaker for (a,b) full-spoof and (c,d) partial-spoof recordings (the prefix “mix” marks one synthetically replaced sentence). Rows: recording source (bona fide Original, RTVC 2019, YourTTS 2022, ElevenLabs 2024); columns: speaker initials (AD = Adele, CB = Cate Blanchett, JB = Jeff Bridges, JA = Jennifer Aniston, MC = Miley Cyrus, MF = Morgan Freeman, SG = Selena Gomez, TH = Tom Holland, LN = Liam Neeson), with the final Mean column. Green = reliably detected, red = deceptive. Lower scores on the Original row reflect false positives (bona fide speech judged synthetic); the mixElevenLabs rows (c,d) show the partial-spoof collapse.
Figure 4: Humans versus six pretrained automated detectors (mean F1) on the same recordings, grouped by synthesizer. On ElevenLabs, detectors (0.09) fall below humans (0.48) on full spoofs (panel a), whereas on partial spoofs (panel b) detectors (0.40) exceed humans (0.29), though their strict All OK localization rate is 0.00. Detector scores are averaged across all six detectors.
Model
Deepfake part (%)
bona fide part (%)
mixRTVC
90.95
88.07
mixYourTTS
94.26
90.13
mixElevenLabs
22.65
88.59
Table 3: Human accuracy of identifying individual sentences based on their type for partial spoof recordings.