Ensuring that Text-to-Speech (TTS) systems deliver human-perceived quality at scale is a central challenge for modern speech technologies. Human subjective evaluation protocols such as Mean Opinion Score (MOS) and Side-by-Side (SBS) comparisons remain the de facto gold standards, yet they are expensive, slow, and sensitive to pervasive assessor biases. This study addresses these barriers by formulating, and implementing a suite of novel neural models designed to approximate expert judgments in both relative (SBS) and absolute (MOS) settings. For relative assessment, we propose NeuralSBS, a HuBERT-backed model achieving 73.7% accuracy (on SOMOS dataset). For absolute assessment, we introduce enhancements to MOSNet using custom sequence-length batching, as well as WhisperBert, a multimodal stacking ensemble that combines Whisper audio features and BERT textual embeddings via weak learners. Our best MOS models achieve a Root Mean Square Error (RMSE) of ~0.40, significantly outperforming the human inter-rater RMSE baseline of 0.62. Furthermore, our ablation studies reveal that naively fusing text via cross-attention can degrade performance, highlighting the effectiveness of ensemble-based stacking over direct latent fusion. We additionally report negative results with SpeechLM-based architectures and zero-shot LLM evaluators (Qwen2-Audio, Gemini 2.5 flash preview), reinforcing the necessity of dedicated metric learning frameworks.
Figures & tables
Figure 1: The NeuralSBS architecture. Audio clips A and B are encoded by a shared HuBERT model, followed by temporal averaging to produce utterance embeddings. These embeddings are processed by an antisymmetric bilinear layer to produce a preference score.
Figure 2: The NeuralSBSBert architecture. It extends NeuralSBS by incorporating textual features from BERT via a cross-attention mechanism, allowing the model to fuse audio and text information before the final preference decision.
Figure 3
Figure 5: The WhisperBert multimodal stacking architecture. Independent weak learners process concatenated audio (Whisper) and text (BERT) embeddings, supervised by a meta-learner.
Metric
NeuralSBS
NeuralSBSBert
Accuracy (SOMOS)
0.737
0.727
AUC-ROC (SOMOS)
0.816
0.804
Accuracy (Std SOMOS)
0.717
0.712
AUC-ROC (Std SOMOS)
0.793
0.786
Table 1: Performance of SBS models on original and standardized SOMOS datasets.
Dataset
MOSNet
MOSNetBert
WhisperBert
RMSE (SOMOS)
0.491
0.547
0.457
RMSE (Std SOMOS)
0.422
0.496
0.402
Table 2: RMSE of absolute MOS predictors. Lower is better. Human inter-rater RMSE is ≈0.62 .
Configuration
RMSE
Vanilla (no sort, no mask, no reg.)
0.531
+ Dropout (0.3) + BatchNorm
0.491
+ Length sorting (SOMOS)
0.496
+ Weighted class sampler
0.546
CNNMOS (no BLSTM, deeper CNN)
∼ 0.50
Attention-only (no BLSTM, no text)
∼ 0.50
Table 3: MOSNet ablation: incremental effect of training enhancements. All models share the same CNN+BLSTM backbone.
Metric
SOMOS
Std SOMOS
MSE
0.209
0.161
RMSE
0.457
0.402
Linear Corr. (LCC)
0.599
0.565
Kendall’s τ
0.415
0.373
Table 4: Full evaluation metrics for WhisperBert on SOMOS and Std SOMOS.
Metric
WhisperBert
UTMOS
MSE
0.161
0.242
LCC
0.565
0.842
SRCC
0.373
0.809
Kendall’s τ
0.373
0.625
Table 5: System-level comparison of WhisperBert (Std SOMOS) and UTMOS.