Written transcripts of the same speech can differ in ways that are not uniquely determined by the speech signal. We formulate evaluation-side transcript canonicalization (ETC), which maps multiple surface forms to a single canonical form when their differences are not determined by speech. To our knowledge, ETC-Bench is the first and most comprehensive multilingual benchmark for evaluation-side transcript canonicalization, covering 181 phenomenon types across 12 languages. Each instance is a minimal pair that shares a sentence context and differs only in a target span exhibiting one phenomenon. To evaluate both canonicalization success and semantic preservation, we propose Semantic-preserving Exact-match Normalization Accuracy (SENA), which combines exact agreement between independently canonicalized pair members with a pair-relative BERTScore constraint against information loss. We further introduce the Taxonomy-guided Agent Canonicalization Skill (TACS), which provides LLM agents with phenomenon descriptions and explicit canonicalization contracts. Experiments with five baseline configurations and three TACS-based agents show that the agents outperform the baseline systems on shared languages. ETC-Bench provides a systematic testbed for evaluation-side transcript canonicalization across languages.
Figures & tables
Figure 1: Overview of ETC-Bench: taxonomy, benchmark-item structure, and evaluation metric.
Category
Phenomenon
Example Pair
Numeric
Cardinal
The recipe calls for 2 eggs. The recipe calls for two eggs.
Homophonic
Homophonic Names
Sara called me yesterday. Sarah called me yesterday.
Abbreviation
Context-dependent Abbreviation
Ст. 5 закона … Статья 5 закона …
Orthographic
Umlaut Variants
Wir kaufen Öl . Wir kaufen Oel .
Homophonic
Third-person Pronoun
他 负责那部分任务。 她 负责那部分任务。
Table 1: Representative ETC-Bench pairs. Blue highlights distinctions not recoverable from speech.
LID
Category
Phenomenon
Canonical Form
en
Numeric Expressions
Cardinal
Quantity in ASCII digits.
en
Homophonic Substitutions
Homophonic Names
Context-appropriate name in broad American English IPA.
ru
Abbreviations
Context-dependent Abbreviation
Context-appropriate full Russian form.
de
Orthographic Variants
Umlaut Variants
Use ä, ö, ü instead of ae, oe, ue .
zh
Homophonic Substitutions
Third-person Pronoun
Broad Mandarin IPA with tone.
Table 2: TACS canonical forms for the phenomena illustrated in Table 1 .
System
ar
bn
de
en
fa
fr
hi
id
ja
ru
vi
zh
Avg.
NeMo-ITN
1.2
–
2.2
5.7
–
5.6
0.0
–
5.3
5.0
1.0
14.1
4.4
NeMo-TN
0.0
–
2.1
18.2
–
3.4
2.0
–
2.6
4.0
6.1
5.4
4.9
FunASR-ITN
–
–
2.2
5.7
–
5.6
–
3.1
5.3
5.0
1.0
2.2
3.7
FunASR-TN
–
–
2.1
19.3
–
–
–
–
–
4.0
–
12.0
9.4
Whisper Normalizer
0.0
0.0
0.0
40.9
5.0
5.6
0.0
8.2
3.9
4.0
7.1
0.0
6.2
No-Skill-GPT
0.0
24.0
0.0
0.0
0.0
0.0
18.0
0.0
0.0
0.0
0.0
0.0
3.5
Table 3: Performance comparison across languages in terms of SENA (%). – indicates unsupported languages. Avg. denotes the macro-average across supported languages.
System
Num.
Sym.
Abbr.
Orth.
Homo.
NeMo-ITN
16.0
0.6
6.8
2.8
0.0
NeMo-TN
23.7
3.2
6.8
0.0
0.0
FunASR-ITN
13.3
1.4
5.1
1.7
0.0
FunASR-TN
32.9
5.7
8.2
0.0
0.0
Whisper Normalizer
6.7
1.0
3.8
8.1
7.0
No-Skill-GPT
0.3
0.5
0.0
7.7
1.9
Table 4: Performance comparison across benchmark categories in terms of SENA (%).
LID
Whisper (Hyp. / Ref.)
TACS-DeepSeek
de
und sehr gewiß daß
und sehr gewiss dass
und sehr gewiss dass
en
thinking of you mabel and
thinking of you /”meIb@l/ and
thinking of you mable and
Table 5: TACS-DeepSeek unifies transcript variants left distinct by Whisper normalization in two real examples in ASR evaluation.