Text-to-speech (TTS) is usually evaluated by transcribing its output with an automatic speech recognition (ASR) system, so the word error rate also counts recognizer errors. We measure these errors on human recordings of the test sentences. On LibriSpeech and FLEURS, the evaluation recognizer misses 33-67% of very rare words even in human speech, and all three recognizers miss 22-42%. Checking every word with two more recognizer families lowers the apparent TTS error on words that at least one recognizer hears two- to three-fold (very rare LibriSpeech words: from 10-13% to 3-5%), yet very rare words remain the hardest. Ignoring words the recognizer misses in human speech is biased: the other recognizers find TTS errors far more often on these words (4-6% vs. 0.2-0.3%) and overturn about half of the errors this mask keeps. As half of the remaining adjudicated errors may be spelling variants, a listening test is needed.
Figures & tables
word
human
TTS output (A / B / C)
(1)
(2)
(3)
Phoebe
A, B, C
fuby / pube / fuby
error
error
error
Tessa
A, B, C
= / = / tess
error
error
correct
stile
A, B
= / = / style
error
ignored
correct
Veolan
A
violan / violan / tullyviolen
error
ignored
error
Table 1: Four words from the F5-TTS LibriSpeech set: recognizers that hear the word in the human recording, the most frequent transcript of A / B / C over the 8 synthetic versions (=: correct), and the decision of estimates (1)–(3).
Figure 1: TTS word error per judged word (any-of- N ) against the share of rare words (corpus measure) in the 500 LibriSpeech sentences, split into four bins at the quartiles (124, 124, 111 and 141 sentences). Points are bin means; bars are 95% CIs.
ASR on human speech
standard TTS error
adjudicated TTS error
band
words
unjudged
C misses
all words
judged words
F5-TTS
Kokoro
LibriSpeech, 500 sentences
very rare ( z<2.5 )
380
21.6
32.9
21.8 / 25.0
10.1 / 13.4
3.4 [ 1.4 , 5.4 ]
4.7 [ 2.5 , 7.2 ]
rare ( 2.5 – 3.5 )
783
4.3
8.9
4.3 / 2.6
2.8 / 1.1
1.5 [ 0.7 , 2.4 ]
0.4 [ 0.0 , 0.9 ]
mid ( 3.5 – 4.5 )
1319
1.4
4.7
0.9 / 1.1
0.5 / 0.8
0.2 [ 0.0 , 0.4 ]
0.2 [ 0.0 , 0.4 ]
common ( z≥4.5 )
7930
0.5
1.5
0.2 / 0.3
0.1 / 0.2
0.0 [ 0 , 0.05 ] †
0.0 [ 0.0 , 0.1 ]
Table 2: ASR error on human speech and TTS error by word frequency (Zipf band z ; tokens with digits excluded). Unjudged : no recognizer hears the word in the human recording. Standard : C misses the word in all N synthetic versions (LibriSpeech: F5-TTS / Kokoro). Adjudicated : Eq. ( 1 ), with 95% CIs; † no errors, 95% Clopper–Pearson upper limit. All values in %.
LibriSpeech
FLEURS
estimate
F5-TTS
Kokoro
F5-TTS
any-of- N
standard
0.63
0.72
1.52
mask
0.30
0.36
0.66
adjudicated
0.22
0.22
0.46
N=1
standard
1.51
2.26
2.56
adjudicated
0.63
0.43
1.11
Table 3: TTS word error per judged word (%) and its correlation r with the share of rare words per sentence (corpus measure/Zipf share; ∗ : the 95% CI excludes zero). Kept : the version with the fewest words missed by C.
TTS error on words C…
kept errors
misses on human
hears on human
A/B hear
LS, F5-TTS
3.9 [ 1.5 , 6.8 ]
0.2 [ 0.1 , 0.3 ]
53 [ 35 , 71 ]
LS, Kokoro
4.9 [ 2.2 , 8.1 ]
0.2 [ 0.1 , 0.3 ]
64 [ 48 , 79 ]
FLEURS, F5-TTS
6.0 [ 2.3 , 10.4 ]
0.3 [ 0.1 , 0.6 ]
56 [ 35 , 79 ]
Table 4: Test of the single-recognizer mask with A and B as judges, counting only words that A or B hears in human speech. Left: TTS error on words that C misses or hears there. Right: share of the mask’s kept errors that A or B hears in a synthetic version. Values in % with 95% CIs.