Evaluation of Phonetic Encoding Algorithms on Transcription Datasets

Can Özbey; Emre Kaplan; Berkin Deniz Kahya

cs.CLSep 3, 2026

Evaluation of Phonetic Encoding Algorithms on Transcription Datasets

Authors: Can Özbey, Emre Kaplan, Berkin Deniz Kahya

Organizations: Huawei Turkey R&D Center Istanbul, Turkey · Dept. of Computer Engineering, Istanbul Technical University Istanbul, Turkey

Abstract

In this work, a novel evaluation scheme built on a generalized variant of the Rand Index measure, namely, the Hüllermeier-Rifqi Index, is proposed in order to assess how well phonetic encoding algorithms conform to word-based transcriptions in IPA (International Phonetic Alphabet) notation. For this objective, the discordance score is obtained by calculating the absolute difference between the pairwise similarity values of ground-truth transcriptions and those of corresponding phonetic encodings, which are computed using normalized edit distance as a permutation dependent string metric. The resulting score is subsequently adjusted with respect to that of a random string generator incorporating the same alphabet as the encoder under consideration. A wide range of phonetic encoders were evaluated as such on multi-lingual transcription datasets along with their recall capabilities based on the collision rate. The validity of the proposed scheme is further supported by its applicability in measuring the orthographic transparency of a language when the writing system is viewed as an inherent phonetic representation.

Figures & tables

Encoder	Abbr.	$\boldsymbol{\|\Sigma\|}$	$\boldsymbol{\mu}$	$\boldsymbol{PC}$	$\boldsymbol{NC}$	$\boldsymbol{OC}$	$\boldsymbol{AOC}$
Writing System	WS	26	1.00	84.89	92.46	88.52	51.08
Russell Index	RI	8	7.48	93.93	79.72	86.24	41.02
Soundex	SX	26	4.44	90.23	88.79	89.50	55.02
Refined Soundex	RSX	26	3.27	89.20	91.25	90.21	58.09
Daitch-Mokotoff	DMF	10	4.52	89.25	87.97	88.60	51.22
Metaphone	MP	21	2.54	89.41	93.57	91.44	63.39

Table 1: Evaluation of the phonetic encoders in English

Figure 1: Performance of the phonetic encoders in English showing the

PC

and

NC

scores (left) and the

AOC

scores with collision rates (right)

Encoder	$\boldsymbol{\|\Sigma\|}$	$\boldsymbol{\mu}$	$\boldsymbol{PC}$	$\boldsymbol{NC}$	$\boldsymbol{OC}$	$\boldsymbol{AOC}$
Writing System	29	1.00	94.44	91.86	93.13	69.89
Haase	9	1.65	98.97	78.69	87.67	44.87
Kölner	9	2.49	95.38	88.17	91.63	62.58
Phonem	15	1.35	94.84	90.86	92.81	68.18
Phonet	25	1.04	95.00	93.99	94.49	75.83
Reth Schek	22	1.05	96.00	90.68	93.27	70.39

Table 2: Evaluation of the phonetic encoders in German

Encoder	$\boldsymbol{\|\Sigma\|}$	$\boldsymbol{\mu}$	$\boldsymbol{PC}$	$\boldsymbol{NC}$	$\boldsymbol{OC}$	$\boldsymbol{AOC}$
Writing System	38	1.00	91.06	89.95	90.50	59.08
FONEM	26	1.22	93.36	89.11	91.18	62.04
Early Henry	22	8.42	89.84	98.05	93.76	73.01

Table 3: Evaluation of the phonetic encoders in French

Encoder	$\boldsymbol{\|\Sigma\|}$	$\boldsymbol{\mu}$	$\boldsymbol{PC}$	$\boldsymbol{NC}$	$\boldsymbol{OC}$	$\boldsymbol{AOC}$
Writing System	29	1.00	94.30	95.96	95.12	79.31
SfinxBis	18	1.74	92.45	91.51	91.98	65.83
Wåhlin	27	1.01	95.14	96.07	95.60	81.35

Table 4: Evaluation of the phonetic encoders in Swedish

Figure 2: The

AOC

scores (y-axis) and the collision rates (x-axis) of the encoders with varying maximum lengths

Figure 3: Evaluation of the writing systems of various languages indicating the degree of orthographic transparency. Provided are the

PC

and

NC

distribution (left) and the

AOC

scores obtained for each language (right)

Measure	FI	EO	RO	SV	IS	NL	PT	DE	FR	EN
AOC	99.62 (1)	95.77 (2)	87.81 (3)	79.31 (4)	77.06 (5)	73.67 (6)	72.02 (7)	69.89 (8)	59.08 (9)	51.08 (10)
OTEANN ( READ )	0.982 (2)	0.988 (1)	0.936 (3)	0.714 (9)	0.791 (6)	0.777 (7)	0.846 (5)	0.771 (8)	0.870 (4)	0.428 (10)
OTEANN ( WRITE )	0.988 (2)	0.989 (1)	0.926 (3)	0.754 (4)	0.666 (6)	0.639 (7)	0.581 (8)	0.697 (5)	0.317 (9)	0.264 (10)

Table 5: Orthographic transparency evaluation (AOC vs. OTEANN accuracy)

An Empirical Analysis of Task-Induced Encoder Bias in Fréchet Audio Distance
Feb 27, 2026Wonwoo JeongText-to-Audio Generation Audio Representation Learning
Dept. of Computer Science and Engineering, Sogang University, South Korea
Stringalign: Moving beyond summary statistics with a transparent Unicode-aware tool for evaluating automatic transcription models
Jun 14, 2026Yngve Mardal Moe, Marie RoaldASR Evaluation
†Independent researcher · ‡The National Library of Norway
BranchShine: Compact Raw-Audio-to-IPA Transcription with a RoPE E-Branchformer Encoder
Jun 22, 2026Nikhil Navas, Sergio Chevtchenko, Talisson Damiao +1Transformer Multilingual ASR
1International Centre for Neuromorphic Systems, Western Sydney University · 2Neurabuild

Feb 27, 2026J/K move · Enter open · S save

An Empirical Analysis of Task-Induced Encoder Bias in Fréchet Audio Distance

Wonwoo Jeong

Dept. of Computer Science and Engineering, Sogang University, South Korea

Text-to-Audio Generation Audio Representation Learning

Open paper AskPDF arXiv

Fréchet Audio Distance (FAD) is the de facto standard for evaluating text-to-audio generation, yet its scores depend on the underlying encoder's embedding space. An encoder's training task dictates which acoustic features are preserved or discarded, causing FAD to inherit systematic task-induced biases. We decompose evaluation into Recall, Precision, and Alignment (split into semantic and structural dimensions), using log-scale normalization for fair cross-encoder comparison. Controlled experiments on six encoders across two datasets reveal a four-axis trade-off: reconstruction-based AudioMAE leads precision sensitivity; ASR-trained Whisper dominates structural detection but is blind to signal degradation; classification-trained VGGish maximizes semantic detection but penalizes legitimate intra-class variation. Since no single encoder is a universal evaluator, future metrics must shift toward evaluation-native encoders intrinsically aligned with human perception.

Abstract

Explore similar work

An Empirical Analysis of Task-Induced Encoder Bias in Fréchet Audio Distance

Stringalign: Moving beyond summary statistics with a transparent Unicode-aware tool for evaluating automatic transcription models

BranchShine: Compact Raw-Audio-to-IPA Transcription with a RoPE E-Branchformer Encoder