Ensuring that Text-to-Speech (TTS) systems deliver human-perceived quality at scale is a central challenge for modern speech technologies. Human subjective evaluation protocols such as Mean Opinion Score (MOS) and Side-by-Side (SBS) comparisons remain the de facto gold standards, yet they are expensive, slow, and sensitive to pervasive assessor biases. This study addresses these barriers by formulating, and implementing a suite of novel neural models designed to approximate expert judgments in both relative (SBS) and absolute (MOS) settings. For relative assessment, we propose NeuralSBS, a HuBERT-backed model achieving 73.7% accuracy (on SOMOS dataset). For absolute assessment, we introduce enhancements to MOSNet using custom sequence-length batching, as well as WhisperBert, a multimodal stacking ensemble that combines Whisper audio features and BERT textual embeddings via weak learners. Our best MOS models achieve a Root Mean Square Error (RMSE) of ~0.40, significantly outperforming the human inter-rater RMSE baseline of 0.62. Furthermore, our ablation studies reveal that naively fusing text via cross-attention can degrade performance, highlighting the effectiveness of ensemble-based stacking over direct latent fusion. We additionally report negative results with SpeechLM-based architectures and zero-shot LLM evaluators (Qwen2-Audio, Gemini 2.5 flash preview), reinforcing the necessity of dedicated metric learning frameworks.
Figures & tables
Figure 1: The NeuralSBS architecture. Audio clips A and B are encoded by a shared HuBERT model, followed by temporal averaging to produce utterance embeddings. These embeddings are processed by an antisymmetric bilinear layer to produce a preference score.
Figure 2: The NeuralSBSBert architecture. It extends NeuralSBS by incorporating textual features from BERT via a cross-attention mechanism, allowing the model to fuse audio and text information before the final preference decision.
Figure 3
Figure 5: The WhisperBert multimodal stacking architecture. Independent weak learners process concatenated audio (Whisper) and text (BERT) embeddings, supervised by a meta-learner.
Metric
NeuralSBS
NeuralSBSBert
Accuracy (SOMOS)
0.737
0.727
AUC-ROC (SOMOS)
0.816
0.804
Accuracy (Std SOMOS)
0.717
0.712
AUC-ROC (Std SOMOS)
0.793
0.786
Table 1: Performance of SBS models on original and standardized SOMOS datasets.
Dataset
MOSNet
MOSNetBert
WhisperBert
RMSE (SOMOS)
0.491
0.547
0.457
RMSE (Std SOMOS)
0.422
0.496
0.402
Table 2: RMSE of absolute MOS predictors. Lower is better. Human inter-rater RMSE is ≈0.62 .
Configuration
RMSE
Vanilla (no sort, no mask, no reg.)
0.531
+ Dropout (0.3) + BatchNorm
0.491
+ Length sorting (SOMOS)
0.496
+ Weighted class sampler
0.546
CNNMOS (no BLSTM, deeper CNN)
∼ 0.50
Attention-only (no BLSTM, no text)
∼ 0.50
Table 3: MOSNet ablation: incremental effect of training enhancements. All models share the same CNN+BLSTM backbone.
Metric
SOMOS
Std SOMOS
MSE
0.209
0.161
RMSE
0.457
0.402
Linear Corr. (LCC)
0.599
0.565
Kendall’s τ
0.415
0.373
Table 4: Full evaluation metrics for WhisperBert on SOMOS and Std SOMOS.
Metric
WhisperBert
UTMOS
MSE
0.161
0.242
LCC
0.565
0.842
SRCC
0.373
0.809
Kendall’s τ
0.373
0.625
Table 5: System-level comparison of WhisperBert (Std SOMOS) and UTMOS.
Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Model (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors are more sensitive to acoustic signal-level properties at the prosodic and paralinguistic levels, while Audio-LLM judges show selective, prompt-dependent detection that favors word-level errors and does not generalize across all dimensions. Neither class reliably captures a breadth of linguistically structured, human-perceptible speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.
Modern text-to-speech (TTS) systems can clone a target speaker from a short reference clip or be fine-tuned on a target voice, yet their behaviour on morphologically rich, lower-resource languages such as Turkish remain under-characterised. We present a systematic benchmark of four contemporary systems (Chatterbox, CosyVoice, OmniVoice, and VoxCPM2) evaluated across fine-tuned and zero-shot configurations, contrasted with a conventional VITS baseline and anchored to natural gold speech. Each configuration is scored with eighteen complementary objective metrics spanning learned naturalness predictors (UTMOS v2, DNSMOS-Pro, SCOREQ, WhisQA, AudioBox-PQ, NatScore, SpeechLMScore), intelligibility and signal-quality estimators (SQUIM PESQ/SI-SDR/STOI, Brouhaha), speaker similarity, distributional fidelity (TTSDS) and low-level acoustic descriptors. We further analyse how quality varies with utterance length and quantify long-form temporal consistency through speaker-identity and naturalness drift over chunked utterances. We release our evaluation code to support reproducible TTS evaluation.
Yunus Emre Ozkose, Alperen Kahraman, Ali Haznedaroglu
Mean opinion score (MOS) prediction models are widely used as proxy metrics in text-to-speech (TTS) research, yet their ability to capture quality differences beyond acoustic fidelity remains unclear. We investigate this via controlled perturbations on speech: acoustic degradation, prosodic errors, and manipulation of speaker-specific characteristics such as pitch and speaking rate. We obtained MOS predictions for these speech samples from both human listeners and the model, and analyzed the differences in their perceptual characteristics. Results show that most models track acoustic degradation well, while all are insensitive to prosodic errors despite large subjective score drops. For speaker characteristics, models exhibit a double dissociation: strong mean fundamental frequency (F0) biases absent in human ratings, yet insensitivity to speaking rate and F0 variability that humans notice. These findings highlight limitations of scalar MOS prediction beyond acoustic fidelity.
Masato Takagi, Masaya Kawamura, Reo Shimizu +1
Nagoya Institute of Technology, Japan · LY Corporation, Japan