Modern automatic speech recognition (ASR) systems trained on extremely large datasets can produce transcripts with numbers written in Arabic numerals. This creates a need for fair comparison with models that output verbatim texts and proper processing of reference transcripts. Popular approaches often reduce text normalization to lowercase and remove punctuation, with no additional normalization applied to languages other than English. In this work, we analyze the impact of normalization of numerical expressions in the evaluation of ASR systems in various languages, using Polish as an example of a highly inflective language. We perform experiments on VoxPopuli and The Polish Parliamentary speech datasets and estimate word error rate (WER) differences for different text normalization approaches. We show that the difference due to the lack of number normalization in WER may be substantial - more than 2 percentage points, and often higher than the differences between systems in popular multilingual benchmarks.
Figures & tables
Reference
Prediction
Normalized Reference
Normalized Prediction
WER
The meeting is at 10 am.
The meeting is at 10 a.m.
the meeting is at 10 am
the meeting is at 10 a m
29%
It’s twelve twenty-five.
It’s 12:25.
it is 1225
it is 12 25
66%
Width is 0.01 mm.
Width is one hundredth millimeter.
width is 0.01
width is 100th of a millimeter
133%
One hundred, two, one, one.
100, 2, 1, 1.
10211
100 2 one one
400%
Thousand, million, two, thousand, two.
1000, 1000000, 2, 1000, 2.
3002
1000 1000000 2 1000 2
500%
Table 1: Examples of high WER obtained for “perfect” predictions caused by imperfect normalization algorithm from [ 14 ]
Lng.
Verbatim number
Number of words
DE
neunzehnhundertvierundachtzig
1
EN
nineteen eighty-four
3
PL
tysiąc dziewięćset osemdziesiąt cztery
4
ES
mil novecientos ochenta y cuatro
5
FR
mille neuf cent quatre-vingt-quatre
6
Table 2: Example of verbatim form of number 1984 in different languages. Words are counted after normalization whith BasicTextNormalizer [ 14 ]
Dataset
Reference
Digits [%]
V
D
N(V)
N(D)
English VoxPopuli
V
0.00
–
0.98
1.21
1.32
D
0.33
0.98
–
0.70
0.40
N(V)
0.68
1.22
0.70
–
0.31
N(D)
0.62
1.33
0.40
0.31
–
Polish VoxPopuli
V
0.00
–
1.27
1.55
1.79
D
0.51
1.28
–
0.88
0.60
Table 3: WER [%] between digit (D) and verbatim (V) transcripts.
Dataset
Reference
Whisper
Whisper-IP
OWSM
Canary-v2
Canary-v1
N( ⋅ )
N( ⋅ )
N( ⋅ )
N( ⋅ )
N( ⋅ )
English VoxPopuli
V
8.46
8.77
8.38
9.03
8.02
8.34
7.50
7.98
5.95
6.73
N(V)
8.24
7.87
8.99
8.36
7.77
7.40
7.64
7.13
6.46
5.64
D
7.65
8.01
8.45
8.54
7.22
7.58
7.09
7.31
5.95
5.98
N(D)
8.01
7.64
8.79
8.19
7.57
7.21
7.45
6.95
6.29
5.59
Polish VoxPopuli
V
7.85
8.28
7.52
8.51
13.86
14.27
7.95
8.78
Table 4: WER in % obtained on various datasets with and without number normalization
Figure 1: Absolute WER reduction (percentage points) in WER when ASR outputs and reference transcripts are both number normalized for various VoxPopuli languages