PhonemeDF: A Synthetic Speech Dataset for Audio Deepfake Detection and Naturalness Evaluation
Authors: Vamshi Nallaguntla, Aishwarya Fursule, Shruti Kshirsagar, Anderson R. Avila
Organizations: Wichita State University, Wichita, Kansas, USA · Institut national de la recherche scientifique (INRS-EMT), Université du Québec, Canada · INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Canada
The growing sophistication of speech generated by Artificial Intelligence (AI) has introduced new challenges in audio deepfake detection. Text-to-speech (TTS) and voice conversion (VC) technologies can create highly convincing synthetic speech with naturalness and intelligibility. This poses serious threats to voice biometric security and to systems designed to combat the spread of spoken misinformation, where synthetic voices may be used to disseminate false or malicious content. While interest in AI-generated speech has increased, resources for evaluating naturalness at the phoneme level remain limited. In this work, we address this gap by presenting the Phoneme-Level DeepFake dataset (PhonemeDF), comprising parallel real and synthetic speech segmented at the phoneme level. Real speech samples are derived from a subset of LibriSpeech, while synthetic samples are generated using four TTS and three VC systems. For each system, phoneme-aligned TextGrid files are obtained using the Montreal Forced Aligner (MFA). We compute the Kullback-Leibler divergence (KLD) between real and synthetic phoneme distributions to quantify fidelity and establish a ranking based on similarity to natural speech. Our findings show a clear correlation between the KLD of real and synthetic phoneme distributions and the performance of classifiers trained to distinguish them, suggesting that KLD can serve as an indicator of the most discriminative phonemes for deepfake detection.
Figures & tables
Figure 1: Overview of the dataset creation pipeline.
Figure 2: Per-phoneme evaluation pipeline for real vs. synthetic discrimination.
Dataset
# Files
Avg. (s)
Total (h)
LibriSpeech
28,539
12.69
100
MeloTTS
28,539
10.15
80
XTTS v2
28,539
10.47
82
Chatterbox TTS
28,539
10.15
80
VITS TTS
28,539
10.77
85
Chatterbox VC
28,539
12.70
100
Table 1: Statistics showing number of files, average duration of an audio file (in seconds), and total duration (in hours) for real and synthetic datasets.
Figure 3: TextGrid showing phoneme boundaries aligned using Montreal Forced Aligner (MFA).
Figure 4: KLD between synthesized and real speech using handcrafted features.
Figure 5: KLD between synthesized and real speech using self-supervised features.
Figure 6: Lowest and highest KLD scores for (a) WavLM vowel representation and (b) WavLM consonant representation.
LR
SVM
System
r
p -value
r
p -value
Melo TTS
0.68
4.99E-03
0.63
1.16E-02
XTTS v2
0.67
6.11E-03
0.69
4.07E-03
Ch-TTS
0.79
4.94E-04
0.80
3.89E-04
VITS TTS
0.49
0.065569
0.51
0.054600
Ch-VC
0.68
5.38E-03
0.69
4.05E-03
Table 2: Results for vowels based on LFCC representation.
LR
SVM
System
r
p -value
r
p -value
Melo TTS
0.79
4.28E-06
0.78
5.99E-06
XTTS v2
0.76
1.47E-05
0.76
1.69E-05
Ch-TTS
0.85
1.11E-07
0.85
1.38E-07
VITS TTS
0.65
5.79E-04
0.66
5.08E-04
Ch-VC
0.88
1.22E-08
0.84
2.92E-07
Table 3: Results for consonants based on LFCC representation.
LR
SVM
System
r
p -value
r
p -value
Melo TTS
0.29
0.298568
0.27
0.338104
XTTS v2
-0.43
0.109768
-0.39
0.155521
Ch-TTS
-0.36
0.187258
-0.50
0.059611
VITS TTS
0.75
1.30E-03
0.75
1.39E-03
Ch-VC
-0.60
0.017691
-0.61
0.016767
Table 4: Results for vowels based on Wav2vec 2.0 representation.
LR
SVM
System
r
p -value
r
p -value
Melo TTS
-0.50
0.013894
-0.59
2.19E-03
XTTS v2
0.31
0.134920
0.02
0.908695
Ch-TTS
-0.43
0.036489
-0.25
0.232014
VITS TTS
-0.56
4.25E-03
-0.16
0.443928
Ch-VC
-0.38
0.069623
-0.35
0.090622
Table 5: Results for consonants based on Wav2vec 2.0 representation.
LR
SVM
System
r
p -value
r
p -value
Melo TTS
0.87
2.48E-05
0.81
2.41E-04
XTTS v2
0.70
3.55E-03
0.76
9.30E-04
Ch-TTS
0.65
9.37E-03
0.74
1.59E-03
VITS TTS
0.07
0.803638
0.04
0.898342
Ch-VC
0.52
0.046131
0.61
0.014760
Table 6: Results for vowels based on LogSpec representation.
LR
SVM
System
r
p -value
r
p -value
Melo TTS
0.41
0.046255
0.56
4.82E-03
XTTS v2
0.62
1.29E-03
0.55
4.94E-03
Ch-TTS
0.57
3.70E-03
0.54
6.44E-03
VITS TTS
0.00
0.997485
-0.17
0.428621
Ch-VC
0.74
3.57E-05
0.69
1.86E-04
Table 7: Results for consonants based on LogSpec representation.
LR
SVM
System
r
p -value
r
p -value
Melo TTS
-0.41
0.132823
-0.31
0.264627
XTTS v2
0.18
0.527040
0.17
0.537221
Ch-TTS
-0.04
0.882777
0.01
0.965355
VITS TTS
0.40
0.145025
0.47
0.080362
Ch-VC
-0.65
8.13E-03
-0.67
6.15E-03
Table 8: Results for vowels based on WavLM representation.
LR
SVM
System
r
p -value
r
p -value
Melo TTS
-0.79
4.18E-06
-0.74
3.43E-05
XTTS v2
-0.60
1.93E-03
-0.60
1.89E-03
Ch-TTS
-0.56
4.76E-03
-0.61
1.44E-03
VITS TTS
-0.78
7.42E-06
-0.73
5.40E-05
Ch-VC
-0.58
3.19E-03
-0.53
8.13E-03
Table 9: Results for consonants based on WavLM representation.
Recent advances in emotional voice conversion (EVC) have enabled the generation of expressive synthetic speech, raising new concerns in audio deepfake detection. Existing approaches treat speech as a homogeneous signal and largely overlook its internal phonetic structure, limiting their interpretability in emotionally conditioned settings. In this work, we propose a phoneme-level framework to analyze emotionally manipulated synthetic speech using real and EVC-generated speech under matched emotional conditions with shared transcripts, phoneme-aligned TextGrids, and WavLM-based embeddings. Our results show that phoneme behavior varies across categories, with complex vowels and fricatives exhibiting higher divergence while simpler phonemes remain more stable. Phonemes with larger distributional differences are also found to be more easily detected, consistently across multiple emotions and synthesis systems. These findings demonstrate that phoneme-level analysis is an effective and interpretable approach for detecting emotionally manipulated synthetic speech.
Vamshi Nallaguntla, Shruti Kshirsagar, Anderson R. Avila
Wichita State University, Wichita, KS, USA. · Institut national de la recherche scientifique (INRS–EMT), QC, Canada · INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Canada
The rapid advancement of generative AI has made audio deepfakes increasingly indistinguishable from authentic human vocals, posing significant threats to persons-of-interest (POI) such as public figures. Current detection systems primarily rely on generic, black-box models that fail to capture speaker-specific idiosyncratic traits and lack interpretability. In this paper, we propose Phoneme-based Voice Profiling (PVP), a novel personalized defense framework. By shifting the detection paradigm from macro-utterance analysis to micro-phonetic modeling, PVP captures the unique acoustic distributions underlying a POI's habitual articulatory patterns. Specifically, our framework models speaker-specific phonetic realizations using lightweight Gaussian Mixture Models (GMMs) estimated solely from bona fide reference speech. This design enables data-efficient profiling and robust generalization to previously unseen spoofing attacks without requiring heavy spoof-specific training. Furthermore, we introduce the first large-scale Chinese POI deepfake dataset to benchmark speaker-specific detection. Experimental results demonstrate that PVP significantly outperforms state-of-the-art generic detectors in POI spoofing scenarios, achieving substantial EER reductions while providing fine-grained, phoneme-level interpretability for forensic analysis. Code and data are available at: https://github.com/JunXue-tech/PVP
Jun Xue, Tong Zhang, Zhuolin Yi +4
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education · School of Cyber Science and Engineering, Wuhan University
Audio deepfakes generated by neural text-to-speech and voice-cloning systems threaten speaker verification and public discourse at scale. The core challenge is cross-dataset generalization: detectors trained on one synthesis pipeline collapse on unseen forgeries. We argue that this failure is primarily because of structural synthetic speech artifacts which are multi-timescale trajectory anomalies. Though every existing detector aggregates a fixed-window frame statistics, this misaligns the architecture with the signal. We propose FlowFake, a Liquid Time-Constant (LTC) architecture whose hidden state evolves via a learned ODE, with per-neuron adaptive time constants simultaneously resolving spectral (10ms) and prosodic (2s) cues. At only 34K parameters FlowFake achieves formal BIBO stability and O(dt^4) integration error. On a four-dataset cross domain benchmark (ASVspoof2019-LA, FakeOrReal, InTheWild, MLAAD), FlowFake reaches 75.29% on ASVspoof2019 trained only on FakeOrReal and 79.97% trained only on MLAAD. It outperforms RawGAT-ST and Whisper-DF on every evaluated pair and matching SSL Wav2vec2 (300x larger) at 0.01% of its parameter count. The source code is available on : https://github.com/GhostRider2023/FlowFake