CodecMOS-Accent: A MOS Benchmark of Resynthesized and TTS Speech from Neural Codecs Across English Accents
Authors: Wen-Chin Huang, Nicholas Sanders, Erica Cooper
Organizations: Nagoya University, Japan · University of Edinburgh, United Kingdom · National Institute Of Information And Communications Technology, Japan
We present the CodecMOS-Accent dataset, a mean opinion score (MOS) benchmark designed to evaluate neural audio codec (NAC) models and the large language model (LLM)-based text-to-speech (TTS) models trained upon them, especially across non-standard speech like accented speech. The dataset comprises 4,000 codec resynthesis and TTS samples from 24 systems, featuring 32 speakers spanning ten accents. A large-scale subjective test was conducted to collect 19,600 annotations from 25 listeners across three dimensions: naturalness, speaker similarity, and accent similarity. This dataset does not only represent an up-to-date study of recent speech synthesis system performance but reveals insights including a tight relationship between speaker and accent similarity, the predictive power of objective metrics, and a perceptual bias when listeners share the same accent with the speaker. This dataset is expected to foster research on more human-centric evaluation for NAC and accented TTS.
Figures & tables
System
S-NAT
S-SPK-SIM
S-ACC-SIM
O-WER
O-SPK-SIM
O-ACC-SIM
O-UTMOS
Ground Truth
4.045 ± 0.063
4.756 ± 0.042
4.678 ± 0.050
1.911
1.000
1.000
4.082
NanoCodec [ 23 ]
4.073 ± 0.062
4.558 ± 0.055
4.493 ± 0.062
0.640
0.872
0.907
4.116
FACodec [ 19 ]
4.070 ± 0.061
4.671 ± 0.048
4.609 ± 0.053
0.650
0.861
0.908
4.100
NeuCodec [ 24 ]
4.048 ± 0.064
4.116 ± 0.077
4.207 ± 0.074
3.847
0.774
0.847
3.945
SNAC [ 21 ]
3.367 ± 0.080
4.268 ± 0.069
4.252 ± 0.071
4.318
0.697
0.802
3.567
Mimi 4.4kbps [ 20 ]
3.261 ± 0.074
4.302 ± 0.066
4.276 ± 0.069
1.700
0.695
0.796
3.534
Table 1: System-level evaluation results comparing Ground Truth, resynthesis, and TTS models. The table reports subjective scores (S-NAT, S-SPK-SIM, S-ACC-SIM) as mean values ±95% confidence intervals, and objective metrics (O-WER, O-SPK-SIM, O-ACC-SIM, O-UTMOS) as mean values. Background color intensity indicates relative performance within each column, where darker blue denotes better performance (higher scores for all metrics except O-WER, for which lower is better). Systems are grouped by category and sorted by S-NAT score.
Metric
S-NAT
S-SPK-SIM
S-ACC-SIM
Utterance-level
S-NAT
1.00
S-SPK-SIM
0.31
1.00
S-ACC-SIM
0.26
0.75
1.00
O-WER
-0.15
-0.04
-0.03
O-SPK-SIM
0.31
0.39
0.34
Table 2: Pearson correlation coefficients of all metrics with respect to the subjective scores (S-NAT, S-SPK-SIM, S-ACC-SIM) in the utterance-level (top) and system-level (bottom). Color intensity denotes the magnitude.
Metric
Same Accent
Diff. Accent
p -value
Ground truth only
S-NAT
4.145
4.026
0.160
S-SPK-SIM
4.855
4.737
0.011
S-ACC-SIM
4.831
4.650
< 0.001
All
S-NAT
3.616
3.489
< 0.001
Table 3: Mean subjective scores from listeners sharing the speaker’s accent (Same Accent) and listeners with a different accent (Diff. Accent). Results are reported separately for ground truth only and the full dataset (“All”). p -values are derived from Welch’s t-test.
S-NAT
S-SPK-SIM
S-ACC-SIM
Speaker Accent
Avg
Min
Avg
Min
Avg
Min
South African
0.90
0.88
0.77
0.65
0.78
0.64
Australian
0.86
0.84
0.71
0.59
0.68
0.61
English
0.86
0.84
0.72
0.59
0.75
0.71
Northern Irish
0.85
0.76
0.79
0.72
0.74
0.62
American
0.79
0.75
0.68
0.44
0.75
0.72
Table 4: Spearman’s ρ broken down by test sample accent, with average and minimum values across all valid pairs of listener accent groups evaluating the same subset of systems. Only pairs with a statistically significant correlation ( p -value <0.05 ) are included.