Written transcripts of the same speech can differ in ways that are not uniquely determined by the speech signal. We formulate evaluation-side transcript canonicalization (ETC), which maps multiple surface forms to a single canonical form when their differences are not determined by speech. To our knowledge, ETC-Bench is the first and most comprehensive multilingual benchmark for evaluation-side transcript canonicalization, covering 181 phenomenon types across 12 languages. Each instance is a minimal pair that shares a sentence context and differs only in a target span exhibiting one phenomenon. To evaluate both canonicalization success and semantic preservation, we propose Semantic-preserving Exact-match Normalization Accuracy (SENA), which combines exact agreement between independently canonicalized pair members with a pair-relative BERTScore constraint against information loss. We further introduce the Taxonomy-guided Agent Canonicalization Skill (TACS), which provides LLM agents with phenomenon descriptions and explicit canonicalization contracts. Experiments with five baseline configurations and three TACS-based agents show that the agents outperform the baseline systems on shared languages. ETC-Bench provides a systematic testbed for evaluation-side transcript canonicalization across languages.
Figures & tables
Figure 1: Overview of ETC-Bench: taxonomy, benchmark-item structure, and evaluation metric.
Category
Phenomenon
Example Pair
Numeric
Cardinal
The recipe calls for 2 eggs. The recipe calls for two eggs.
Homophonic
Homophonic Names
Sara called me yesterday. Sarah called me yesterday.
Abbreviation
Context-dependent Abbreviation
Ст. 5 закона … Статья 5 закона …
Orthographic
Umlaut Variants
Wir kaufen Öl . Wir kaufen Oel .
Homophonic
Third-person Pronoun
他 负责那部分任务。 她 负责那部分任务。
Table 1: Representative ETC-Bench pairs. Blue highlights distinctions not recoverable from speech.
LID
Category
Phenomenon
Canonical Form
en
Numeric Expressions
Cardinal
Quantity in ASCII digits.
en
Homophonic Substitutions
Homophonic Names
Context-appropriate name in broad American English IPA.
ru
Abbreviations
Context-dependent Abbreviation
Context-appropriate full Russian form.
de
Orthographic Variants
Umlaut Variants
Use ä, ö, ü instead of ae, oe, ue .
zh
Homophonic Substitutions
Third-person Pronoun
Broad Mandarin IPA with tone.
Table 2: TACS canonical forms for the phenomena illustrated in Table 1 .
System
ar
bn
de
en
fa
fr
hi
id
ja
ru
vi
zh
Avg.
NeMo-ITN
1.2
–
2.2
5.7
–
5.6
0.0
–
5.3
5.0
1.0
14.1
4.4
NeMo-TN
0.0
–
2.1
18.2
–
3.4
2.0
–
2.6
4.0
6.1
5.4
4.9
FunASR-ITN
–
–
2.2
5.7
–
5.6
–
3.1
5.3
5.0
1.0
2.2
3.7
FunASR-TN
–
–
2.1
19.3
–
–
–
–
–
4.0
–
12.0
9.4
Whisper Normalizer
0.0
0.0
0.0
40.9
5.0
5.6
0.0
8.2
3.9
4.0
7.1
0.0
6.2
No-Skill-GPT
0.0
24.0
0.0
0.0
0.0
0.0
18.0
0.0
0.0
0.0
0.0
0.0
3.5
Table 3: Performance comparison across languages in terms of SENA (%). – indicates unsupported languages. Avg. denotes the macro-average across supported languages.
System
Num.
Sym.
Abbr.
Orth.
Homo.
NeMo-ITN
16.0
0.6
6.8
2.8
0.0
NeMo-TN
23.7
3.2
6.8
0.0
0.0
FunASR-ITN
13.3
1.4
5.1
1.7
0.0
FunASR-TN
32.9
5.7
8.2
0.0
0.0
Whisper Normalizer
6.7
1.0
3.8
8.1
7.0
No-Skill-GPT
0.3
0.5
0.0
7.7
1.9
Table 4: Performance comparison across benchmark categories in terms of SENA (%).
LID
Whisper (Hyp. / Ref.)
TACS-DeepSeek
de
und sehr gewiß daß
und sehr gewiss dass
und sehr gewiss dass
en
thinking of you mabel and
thinking of you /”meIb@l/ and
thinking of you mable and
Table 5: TACS-DeepSeek unifies transcript variants left distinct by Whisper normalization in two real examples in ASR evaluation.
Low-latency simultaneous speech-to-speech translation must keep pace with ongoing speech while preserving key information. To meet these demands, systems use segmentation, reformulation and condensation to reorganize and rephrase information. However, metrics developed for text translation, including BLEU and COMET, may not consistently distinguish faithful adaptations from semantic errors. We propose EviSI, a large language model evaluation agent combining Multidimensional Quality Metrics (MQM) with criteria developed with professional interpreters. Shared source evidence guides assessment across four dimensions: Anchor, Event, Logic and Fluency. Verified errors are deduplicated before deterministic scoring. On human-rated English to Chinese and Chinese to English data, EviSI recovers the aggregate English to Chinese human system ranking. Mean within-dataset Kendall correlations for system rankings reach 0.707 and 0.467, respectively, exceeding evaluated BLEU and COMET baselines. A multilingual extension to five directions without human ratings retains the dimensions and scoring rule, showing positive system ranking correlations with COMET throughout.
Ben Yan, Zongyao Li, Xiaoyu Chen +6
1Huawei Translation Service Center, Beijing, China · School of Informatics, Xiamen University, China
Automatic speech recognition is usually evaluated with word error rate (WER), although voice workflows often require exact written values. VoiceCodeBench measures whether transcripts preserve identifiers, paths, commands, and other structured tokens needed by downstream software. It contains 300 human-recorded English workplace segments (5.59 hours, 85 speakers) and 1,482 audited entities across 26 types and eight domains. Under a raw-audio-only protocol, we evaluate 19 batch and streaming systems using WER, Canonical Token/Entity Match (CTEM), and strict segment-level Task Success Rate (TSR). Across systems, WER has little rank agreement with CTEM (Spearman ρ=−0.28) or TSR (ρ=−0.22). The best CTEM and TSR are 91.8% and 68.7%. Even the strongest system therefore leaves nearly one-third of recordings with an unrecovered critical value. Symbol-, separator-, and boundary-sensitive entities account for most errors.
Tyler Baumgartner, Brandon Tai, Lisa Kaelin-Martin +4
Voice agents depend on accurate automatic speech recognition (ASR) to act on what callers say, yet ASR is evaluated on read, English-centric speech with word error rate (WER), which penalizes surface rather than semantic differences. We introduce mu-bench, a dataset of 4,270 caller utterances from 250 phone calls to an AI banking agent in English, Spanish, Turkish, Vietnamese, and Mandarin, centered on form-field inputs such as names, email addresses, and confirmation codes. We release Utterance Error Rate (UER), an LLM judge of whether a transcript preserves meaning, calibrated against human raters, together with an LLM normalizer that makes WER comparable across providers' output formats. On 1,847 human-rated transcripts, UER agrees with annotators at κ = 0.78, versus 0.53 for exact-match WER on normalized text. We rank six commercial providers on a public leaderboard; the best reaches 11.9% UER, and Mandarin is hardest for all six.