Phonetic error detection (PED) aims to identify phoneme-level substitutions, deletions, and insertions and is essential for educational and clinical speech assessment and intervention. Despite recent advances in speech models for phoneme recognition, their ability to reliably detect fine-grained phonetic errors remains unclear because of a lack of high-quality evaluation data. To address this gap, we introduce PEDBench, the first PED benchmark spanning a continuum from natural to controlled speech. It comprises real-world recordings of spoken errors made by children (N=334), scripted error productions by human actors (N=1,188), and synthetic error productions generated using phone-conditioned text-to-speech (N=1,083). All phonetic annotations were transcribed and verified by two phonetically trained annotators. Our evaluation of 13 state-of-the-art speech models shows that they fail to reliably detect phonetic errors, even under controlled conditions with acted and synthetic speech, where acoustic variability is reduced. The models also exhibit low precision, frequently misclassifying correctly pronounced phonemes as errors. Our speech-language models, fine-tuned on synthetic data, outperform commercial and open-source baselines on word-level PED, although a sim-to-real domain gap remains. We plan to publicly release our code and recipes, along with our acted and synthetic datasets, to support the development of more accurate PED systems.
Figures & tables
Figure 1: Overview of the inference and evaluation pipeline for phonetic error detection.
Figure 2: Phonetic Error Detection Alignment and Metric Calculation Rules Gelin et al. (2023) .
Dataset
Model
PR Metrics
PED Metrics
PFER ↓
PER ↓
FRR ↓
FAR ↓
DER ↓
Prec. ↑
Rec. ↑
F1 ↑
L2-ARCTIC
W2V2P-LV60
10.66
23.76
10.73
56.55
21.66
30.32
28.13
29.19
W2V2P-XLSR53
11.42
26.35
15.33
44.01
24.02
28.52
36.06
31.85
MultiIPA
19.21
58.00
48.57
16.19
49.97
13.37
43.17
20.42
ZIPA-CTC
10.07
22.45
6.84
72.31
20.61
28.27
16.64
20.95
ZIPA-CTC-NS
9.98
22.00
6.06
74.43
20.19
28.97
15.41
20.12
Table 1: Benchmarked results on L2-ARCTIC and CMU Kids.
Dataset
Model
PR Metrics
PED Metrics
PFER ↓
PER ↓
FRR ↓
FAR ↓
DER ↓
Prec. ↑
Rec. ↑
F1 ↑
Child
W2V2P-LV60
27.53
62.38
50.53
7.38
49.71
24.92
52.20
33.73
W2V2P-XLSR53
22.47
50.53
45.27
6.85
42.28
33.64
64.15
44.14
W2V2P-XLSR53 SC (ours)
21.30
47.27
42.42
7.38
39.10
36.74
68.19
47.75
MultiIPA
26.30
76.70
65.23
4.57
62.51
18.05
43.59
25.53
ZIPA-CTC
25.55
53.46
46.21
11.42
45.37
30.31
56.41
39.43
Table 2: Benchmarked results on our child and acted speech evaluation datasets.
Model
Word
Sentence
Passage
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
W2V2P-LV60
15.14
51.34
79.30
62.33
9.33
23.50
71.34
35.36
10.44
6.49
77.39
11.98
W2V2P-XLSR53
18.97
45.64
79.94
58.10
13.87
19.09
80.57
30.87
15.08
4.82
81.53
9.10
MultiIPA
54.53
16.62
56.69
25.70
48.86
4.85
64.33
9.01
46.83
1.21
64.33
2.38
ZIPA-CTC
11.16
57.85
78.66
66.67
9.67
25.45
81.53
38.79
9.44
7.76
84.71
14.22
ZIPA-CTC-NS
9.17
61.68
77.39
68.64
9.44
25.20
78.98
38.21
9.47
7.52
82.17
13.78
Table 3: Benchmarked results on our synthetic speech evaluation dataset.
Figure 3: Error profiles for four representative models on Authentic Child Speech and Acted Speech.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Spectrum of Ambiguity in Phoneme Production
Vowel
Consonant
(I, i)
(I, E)
(i, E)
(b, p)
(b, v)
(tS, t)
(tS, S)
(i, eI)
(E, æ)
(E, I)
(d, k)
(d, g)
(f, p)
(f, b)
(eI, E)
(eI, æ)
(æ, A)
(g, d)
(g, k)
(dZ, d)
(dZ, z)
(æ, E)
(A, oU)
(A, u)
(k, t)
(k, g)
(l, w)
(l, j)
(O, oU)
(O, u)
(oU, O)
(m, b)
(m, n)
(n, m)
(n, t)
(oU, aU)
(U, u)
(U, oU)
(N, n)
(N, k)
(p, b)
(p, f)
Appendix
Table 4: IPA substitution pairs, written as (canonical, substituted), used to generate mispronunciations in the synthetic data.
Source symbols
CMU-39
@
2
I
3
@
e / o
eI / oU
a
A
œ / ø / y
E / U / u
Appendix
Table 5: Main rules for mapping IPA to the CMU-39 inventory. Rhotic hooks on other vowels are removed. The last two rows depend on the aligned canonical phone. Only aI, aU, eI, oU, and OI are single diphthongs; other vowel sequences are split (e.g., @U becomes 2 U).
Model
Type
Multi.
Native output
W2V2P-XLSR53
CTC
✓
IPA (eSpeak)
W2V2P-LV60
CTC
✓
IPA (eSpeak)
MultiIPA
CTC
✓
IPA
ZIPA-CTC / -NS
CTC
✓
IPA
PhoneticXeus
CTC
✓
IPA
HuPER
CTC
×
ARPAbet (to IPA)
Appendix
Table 6: Models and their native output. Multi. marks models trained on more than one language. W2V2P-LV60 is pretrained on English audio and fine-tuned on multilingual phone labels. Audio Flamingo 3’s language coverage is not documented. All outputs are projected to CMU-39.
Qwen / AF3
W2V2P-XLSR53
Method
LoRA
Full fine-tuning
Trained weights
Text-LM q,k,v,o
All but CNN
Trainable params
10.1M
–
LoRA r / α / dropout
16 / 32 / 0.05
–
Loss
Cross-entropy
CTC
Optimizer
AdamW, wd 0.01
AdamW, wd 0.01
Appendix
Table 7: Fine-tuning settings.
Figure 5: System prompt and user prompts for speech-language models. Long lines are wrapped for display.
Data
Train
Dev.
Test
Split
Used in
TIMIT
3,696
400
192
Kaldi speaker split, no SA
TIMIT, TIMIT+SW, TIMIT+SC (train)
Synthetic words (SW)
396
–
–
All for training
TIMIT+SW, TIMIT+SC
Synthetic combined (SC)
550
–
–
All for training
TIMIT+SC
Acted
–
1,188
–
All for model selection
TIMIT+SC
TIMIT+SW
4,092
400
–
Dev. = TIMIT
Evaluated on Acted
TIMIT+SC
4,246
1,188
–
Dev. = Acted
Evaluated on Authentic
Appendix
Table 8: Data used for fine-tuning and model selection.
Output
Result
{"transcription":"S i t"}
Accepted.
{"answer":"S i t"}
Accepted: only one key.
S i t
Accepted: plain phone string.
The transcription is {"transcription":"S i t"}
Invalid: text before the JSON.
The word is sheet.
Invalid: English text.
Appendix
Table 9: Examples of model outputs and how they are parsed. Invalid outputs are scored as empty predictions.
Figure 6: Scripted vs Unscripted Results on Our Authentic and Acted Evaluation Datasets.
Model
Word
Sentence
Passage
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
FRR ↓
Prec. ↑
Rec. ↑
F1 ↑
W2V2P-LV60
15.14
51.34
79.30
62.33
6.89
67.07
71.34
69.14
7.84
67.13
77.39
71.89
W2V2P-XLSR53
18.97
45.64
79.94
58.10
10.40
62.01
80.57
70.08
8.71
65.64
81.53
72.73
MultiIPA
54.53
16.62
56.69
25.70
48.26
20.32
64.33
30.89
47.91
20.34
64.33
30.91
ZIPA-CTC
11.16
57.85
78.66
66.67
6.16
72.32
81.53
76.65
4.91
77.33
84.71
80.85
ZIPA-CTC-NS
9.17
61.68
77.39
68.64
6.72
69.66
78.98
74.03
5.21
75.44
82.17
78.66
Appendix
Table 10: Target-word evaluation on the Synthetic Speech Dataset across isolated-word, sentence, and passage contexts. The evaluation is only conducted within the target-word span, including insertions at both word boundaries.
Figure 7: Consonant confusion matrices on Authentic Child Speech for Qwen2.5-Omni-7B + TIMIT+SC (top left), W2V2P-XLSR53 (top right), BabAR (bottom left), and Gemini-3.8-Flash (bottom right).
Figure 8: Vowel confusion matrices on Authentic Child Speech for Qwen2.5-Omni-7B + TIMIT+SC (top left), W2V2P-XLSR53 (top right), BabAR (bottom left), and Gemini-3.8-Flash (bottom right).
Figure 9: Consonant confusion matrices on Acted Speech for Qwen2.5-Omni-7B + TIMIT+SW (top left), W2V2P-XLSR53 (top right), BabAR (bottom left), and Gemini-3.8-Flash (bottom right).
Figure 10: Vowel confusion matrices on Acted Speech for Qwen2.5-Omni-7B + TIMIT+SW (top left), W2V2P-XLSR53 (top right), BabAR (bottom left), and Gemini-3.8-Flash (bottom right).
Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis. Despite prolonged efforts in developing PR systems, current evaluations only measure surface-level transcription accuracy. We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception through intrinsic and extrinsic evaluation of PR systems. PRiSM standardizes transcription-based evaluation and assesses downstream utility in clinical, educational, and multilingual settings with transcription and representation probes. We find that diverse language exposure during training is key to PR performance, encoder-CTC models are the most stable, and specialized PR models still outperform Large Audio Language Models. PRiSM releases code, recipes, and datasets to move the field toward multilingual speech models with robust phonetic ability: https://github.com/changelinglab/prism.
Automated phoneme-level pronunciation assessment is vital for scalable speech therapy and language learning, yet validated tools for Arabic remain scarce. We present Harf-Speech, a modular system scoring Arabic pronunciation at the phoneme level on a clinical scale. It combines an MSA phonetizer, a fine-tuned speech-to-phoneme model, Levenshtein alignment, and a blended scorer using longest common subsequence and edit-distance metrics. We fine-tune three ASR architectures on Arabic phoneme data and benchmark them with zero-shot multimodal models; the best, OmniASR-CTC-1B-v2, achieves 8.92% phoneme error rate. Three certified speech-language pathologists independently scored 40 utterances for clinical validation. Harf-Speech attains a Pearson correlation of 0.791 and ICC(2,1) of 0.659 with mean expert scores, outperforming existing end-to-end assessment frameworks. These results show Harf-Speech yields clinically aligned, interpretable scores comparable to inter-rater expert agreement.
Asif Azad, MD Sadik Hossain Shanto, Mohammad Sadat Hossain +6
Ministry of Defense, Saudi Arabia · Ability Center, Saudi Arabia · University of Rochester, USA
Synthetic speech detection benchmarks now report sub-1% error rates on some in-domain evaluations, yet performance degrades under unseen attacks, channel mismatch, and distribution shift. Based on a three-year effort with Phonexia, a commercial speaker-recognition vendor, we report barriers encountered while building and deploying a detector. Many public benchmarks are not licensed for commercial model development. Real inputs are not four-second clean clips but long, codec-degraded, sometimes partially synthetic recordings. And when a calibrated system returns a log-likelihood ratio of 2.5, no one can tell the customer what it means for their decision. Rather than proposing a new model, we connect these barriers to concrete research and coordination proposals: shared standards for commercially usable datasets, realistic deployment benchmarks, and scores that non-experts can act on. These observations come from one project and should be tested in other settings.