Phonetic error detection (PED) aims to identify phoneme-level substitutions, deletions, and insertions and is essential for educational and clinical speech assessment and intervention. Despite recent advances in speech models for phoneme recognition, their ability to reliably detect fine-grained phonetic errors remains unclear because of a lack of high-quality evaluation data. To address this gap, we introduce PEDBench, the first PED benchmark spanning a continuum from natural to controlled speech. It comprises real-world recordings of spoken errors made by children (N=334), scripted error productions by human actors (N=1,188), and synthetic error productions generated using phone-conditioned text-to-speech (N=1,083). All phonetic annotations were transcribed and verified by two phonetically trained annotators. Our evaluation of 13 state-of-the-art speech models shows that they fail to reliably detect phonetic errors, even under controlled conditions with acted and synthetic speech, where acoustic variability is reduced. The models also exhibit low precision, frequently misclassifying correctly pronounced phonemes as errors. Our speech-language models, fine-tuned on synthetic data, outperform commercial and open-source baselines on word-level PED, although a sim-to-real domain gap remains. We plan to publicly release our code and recipes, along with our acted and synthetic datasets, to support the development of more accurate PED systems.
Figures & tables
Figure 1: Overview of the inference and evaluation pipeline for phonetic error detection.
Figure 2: Phonetic Error Detection Alignment and Metric Calculation Rules Gelin et al. (2023) .
Dataset
Model
PR Metrics
PED Metrics
PFER ↓
PER ↓
FRR ↓
FAR ↓
DER ↓
Prec. ↑
Rec. ↑
F1 ↑
L2-ARCTIC
W2V2P-LV60
10.66
23.76
10.73
56.55
21.66
30.32
28.13
29.19
W2V2P-XLSR53
11.42
26.35
15.33
44.01
24.02
28.52
36.06
31.85
MultiIPA
19.21
58.00
48.57
16.19
49.97
13.37
43.17
20.42
ZIPA-CTC
10.07
22.45
6.84
72.31
20.61
28.27
16.64
20.95
ZIPA-CTC-NS
9.98
22.00
6.06
74.43
20.19
28.97
15.41
20.12
Table 1: Benchmarked results on L2-ARCTIC and CMU Kids.
Dataset
Model
PR Metrics
PED Metrics
PFER ↓
PER ↓
FRR ↓
FAR ↓
DER ↓
Prec. ↑
Rec. ↑
F1 ↑
Child
W2V2P-LV60
27.53
62.38
50.53
7.38
49.71
24.92
52.20
33.73
W2V2P-XLSR53
22.47
50.53
45.27
6.85
42.28
33.64
64.15
44.14
W2V2P-XLSR53 SC (ours)
21.30
47.27
42.42
7.38
39.10
36.74
68.19
47.75
MultiIPA
26.30
76.70
65.23
4.57
62.51
18.05
43.59
25.53
ZIPA-CTC
25.55
53.46
46.21
11.42
45.37
30.31
56.41
39.43
Table 2: Benchmarked results on our child and acted speech evaluation datasets.
Model
Word
Sentence
Passage
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
W2V2P-LV60
15.14
51.34
79.30
62.33
9.33
23.50
71.34
35.36
10.44
6.49
77.39
11.98
W2V2P-XLSR53
18.97
45.64
79.94
58.10
13.87
19.09
80.57
30.87
15.08
4.82
81.53
9.10
MultiIPA
54.53
16.62
56.69
25.70
48.86
4.85
64.33
9.01
46.83
1.21
64.33
2.38
ZIPA-CTC
11.16
57.85
78.66
66.67
9.67
25.45
81.53
38.79
9.44
7.76
84.71
14.22
ZIPA-CTC-NS
9.17
61.68
77.39
68.64
9.44
25.20
78.98
38.21
9.47
7.52
82.17
13.78
Table 3: Benchmarked results on our synthetic speech evaluation dataset.
Figure 3: Error profiles for four representative models on Authentic Child Speech and Acted Speech.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Spectrum of Ambiguity in Phoneme Production
Vowel
Consonant
(I, i)
(I, E)
(i, E)
(b, p)
(b, v)
(tS, t)
(tS, S)
(i, eI)
(E, æ)
(E, I)
(d, k)
(d, g)
(f, p)
(f, b)
(eI, E)
(eI, æ)
(æ, A)
(g, d)
(g, k)
(dZ, d)
(dZ, z)
(æ, E)
(A, oU)
(A, u)
(k, t)
(k, g)
(l, w)
(l, j)
(O, oU)
(O, u)
(oU, O)
(m, b)
(m, n)
(n, m)
(n, t)
(oU, aU)
(U, u)
(U, oU)
(N, n)
(N, k)
(p, b)
(p, f)
Appendix
Table 4: IPA substitution pairs, written as (canonical, substituted), used to generate mispronunciations in the synthetic data.
Source symbols
CMU-39
@
2
I
3
@
e / o
eI / oU
a
A
œ / ø / y
E / U / u
Appendix
Table 5: Main rules for mapping IPA to the CMU-39 inventory. Rhotic hooks on other vowels are removed. The last two rows depend on the aligned canonical phone. Only aI, aU, eI, oU, and OI are single diphthongs; other vowel sequences are split (e.g., @U becomes 2 U).
Model
Type
Multi.
Native output
W2V2P-XLSR53
CTC
✓
IPA (eSpeak)
W2V2P-LV60
CTC
✓
IPA (eSpeak)
MultiIPA
CTC
✓
IPA
ZIPA-CTC / -NS
CTC
✓
IPA
PhoneticXeus
CTC
✓
IPA
HuPER
CTC
×
ARPAbet (to IPA)
Appendix
Table 6: Models and their native output. Multi. marks models trained on more than one language. W2V2P-LV60 is pretrained on English audio and fine-tuned on multilingual phone labels. Audio Flamingo 3’s language coverage is not documented. All outputs are projected to CMU-39.
Qwen / AF3
W2V2P-XLSR53
Method
LoRA
Full fine-tuning
Trained weights
Text-LM q,k,v,o
All but CNN
Trainable params
10.1M
–
LoRA r / α / dropout
16 / 32 / 0.05
–
Loss
Cross-entropy
CTC
Optimizer
AdamW, wd 0.01
AdamW, wd 0.01
Appendix
Table 7: Fine-tuning settings.
Figure 5: System prompt and user prompts for speech-language models. Long lines are wrapped for display.
Data
Train
Dev.
Test
Split
Used in
TIMIT
3,696
400
192
Kaldi speaker split, no SA
TIMIT, TIMIT+SW, TIMIT+SC (train)
Synthetic words (SW)
396
–
–
All for training
TIMIT+SW, TIMIT+SC
Synthetic combined (SC)
550
–
–
All for training
TIMIT+SC
Acted
–
1,188
–
All for model selection
TIMIT+SC
TIMIT+SW
4,092
400
–
Dev. = TIMIT
Evaluated on Acted
TIMIT+SC
4,246
1,188
–
Dev. = Acted
Evaluated on Authentic
Appendix
Table 8: Data used for fine-tuning and model selection.
Output
Result
{"transcription":"S i t"}
Accepted.
{"answer":"S i t"}
Accepted: only one key.
S i t
Accepted: plain phone string.
The transcription is {"transcription":"S i t"}
Invalid: text before the JSON.
The word is sheet.
Invalid: English text.
Appendix
Table 9: Examples of model outputs and how they are parsed. Invalid outputs are scored as empty predictions.
Figure 6: Scripted vs Unscripted Results on Our Authentic and Acted Evaluation Datasets.
Model
Word
Sentence
Passage
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
W2V2P-LV60
15.14
51.34
79.30
62.33
6.89
67.07
71.34
69.14
7.84
67.13
77.39
71.89
W2V2P-XLSR53
18.97
45.64
79.94
58.10
10.40
62.01
80.57
70.08
8.71
65.64
81.53
72.73
MultiIPA
54.53
16.62
56.69
25.70
48.26
20.32
64.33
30.89
47.91
20.34
64.33
30.91
ZIPA-CTC
11.16
57.85
78.66
66.67
6.16
72.32
81.53
76.65
4.91
77.33
84.71
80.85
ZIPA-CTC-NS
9.17
61.68
77.39
68.64
6.72
69.66
78.98
74.03
5.21
75.44
82.17
78.66
Appendix
Table 10: Target-word evaluation on the Synthetic Speech Dataset across isolated-word, sentence, and passage contexts. The evaluation is only conducted within the target-word span, including insertions at both word boundaries.
Figure 7: Consonant confusion matrices on Authentic Child Speech for Qwen2.5-Omni-7B + TIMIT+SC (top left), W2V2P-XLSR53 (top right), BabAR (bottom left), and Gemini-3.8-Flash (bottom right).
Figure 8: Vowel confusion matrices on Authentic Child Speech for Qwen2.5-Omni-7B + TIMIT+SC (top left), W2V2P-XLSR53 (top right), BabAR (bottom left), and Gemini-3.8-Flash (bottom right).
Figure 9: Consonant confusion matrices on Acted Speech for Qwen2.5-Omni-7B + TIMIT+SW (top left), W2V2P-XLSR53 (top right), BabAR (bottom left), and Gemini-3.8-Flash (bottom right).
Figure 10: Vowel confusion matrices on Acted Speech for Qwen2.5-Omni-7B + TIMIT+SW (top left), W2V2P-XLSR53 (top right), BabAR (bottom left), and Gemini-3.8-Flash (bottom right).