Recent advances in speech synthesis technology have enabled the generation of speech with diverse characteristics. One such characteristic is a foreign accent. In our preliminary experiments, we found that UTMOS, a widely used automatic speech quality assessment system, tended to assign lower scores to speech with stronger foreign accents, even though it is produced by a human speaker. This observation raised a question: How do humans evaluate the naturalness of foreign-accented speech? We conducted a large-scale subjective evaluation involving both native and non-native listeners under different evaluation instructions. The results showed that humans also considered accentedness when evaluating naturalness, but that defining naturalness in terms of human-likeness and selecting raters who have greater exposure to the target accent may enable more appropriate evaluation. This study aims to provide a starting point for future discussions on fairer and more appropriate evaluation of synthetic speech with a foreign accent.
Figures & tables
Fig. 1: Relationship between the proficiency of Japanese speakers of English and the naturalness scores predicted by UTMOS [ 23 ] from their utterances: speakers with lower proficiency tend to have lower naturalness scores.
DNSMOS
UTMOS-Prof.
JE
AE
correlation
Original
3.16
3.26
0.417
Restored
3.30
3.34
0.549
TABLE I: Effects of speech restoration on ERJ corpus recordings: DNSMOS scores and Spearman’s rank correlation coefficients between UTMOS and proficiency scores.
Type
“Listen to the speech sample and rate …”
I1
-
how natural it sounds.
I2
HL
how human-like (natural as human speech) it sounds.
I3
CC
how natural it sounds as native speech.
I4
CC
how natural it sounds as non-native speech.
TABLE II: Four evaluation instructions used in the subjective listening test. HL: Human-Likeness; CC: Context Conformity.
No resyn.
BigVGAN
Encodec
WORLD
Rater
AE (N=20)
JE (N=100)
AE (N=20)
JE (N=100)
AE (N=20)
JE (N=100)
AE(N=20)
JE (N=100)
EN
I1
4.28 ± 0.22 (0.49)
2.75 ± 0.15 (0.76)
4.05 ± 0.25 (0.56)
2.71 ± 0.14 (0.70)
4.10 ± 0.23 (0.52)
2.58 ± 0.14 (0.69)
3.69 ± 0.31 (0.71)
2.44 ± 0.13 (0.69)
I2
3.95 ± 0.24 (0.55)
2.89 ± 0.14 (0.71)
4.01 ± 0.26 (0.60)
2.75 ± 0.13 (0.68)
3.80 ± 0.23 (0.52)
2.61 ± 0.14 (0.70)
3.29 ± 0.27 (0.61)
2.32 ± 0.11 (0.55)
I3
3.96 ± 0.24 (0.54)
2.51 ± 0.13 (0.65)
4.06 ± 0.29 (0.66)
2.42 ± 0.12 (0.62)
3.93 ± 0.23 (0.53)
2.32 ± 0.12 (0.59)
3.53 ± 0.26 (0.59)
2.22 ± 0.12 (0.63)
I4
3.96 ± 0.26 (0.59)
2.91 ± 0.12 (0.63)
3.83 ± 0.25 (0.57)
2.82 ± 0.12 (0.63)
3.79 ± 0.22 (0.50)
2.70 ± 0.12 (0.61)
3.34 ± 0.27 (0.62)
2.50 ± 0.12 (0.61)
JP
I1
4.23 ± 0.26 (0.59)
3.33 ± 0.12 (0.62)
4.16 ± 0.21 (0.48)
3.24 ± 0.12 (0.62)
4.06 ± 0.23 (0.52)
3.14 ± 0.13 (0.66)
3.49 ± 0.23 (0.52)
2.93 ± 0.13 (0.64)
TABLE III: Summary statistics of the constructed dataset: mean ± 95% confidence interval (standard deviation). For each rater group, system, and speech type (AE or JE), MOS values lower and higher than those of I1 are shown in red and green , respectively. For each instruction, system, and speech type, the higher MOS values between native English and Japanese raters is shown in bold .
Fig. 2: Violin plots of MOS values for JE speech by rater group and evaluation instruction. Statistical significance was assessed using Wilcoxon signed-rank tests with Holm correction (*: p<0.05 , **: p<0.01 , ***: p<0.001 , n.s.: not significant).
Rater
No resyn.
BigVGAN
Encodec
WORLD
Avg.
EN
I1
0.455
0.535
0.549
0.521
0.515
I2
0.588
0.526
0.392
0.202
0.427
I3
0.446
0.317
0.469
0.426
0.415
I4
0.381
0.259
0.308
0.484
0.358
JP
I1
0.575
0.437
0.641
0.411
0.516
I2
0.491
0.284
0.270
0.254
0.325
TABLE IV: Spearman’s rank correlation coefficients between MOS scores and proficiency scores.
Fig. 3: Relationship between proficiency scores and speaker-level mean pseudo MOS scores predicted by each system for JE speech. Stars indicate the overall mean pseudo MOS scores for AE speech.
EN
JP
I1
I2
I3
I4
I1
I2
I3
I4
UTMOS
0.26
0.42
0.28
0.33
0.49
0.51
0.28
0.24
UTMOSv2
0.15
0.33
0.26
0.17
0.28
0.34
0.23
0.07
SSL-MOS
0.14
0.28
0.17
0.19
0.27
0.34
-0.01
0.03
TABLE V: Utterance-level Spearman’s rank correlation coefficients between the scores predicted by each MOS prediction system and the actual MOS values (JE only, N=400).