The continuous deployment of enterprise voice systems creates a critical operational bottleneck: evaluating the acoustic quality of synthetic speech across high-volume production updates. While human Mean Opinion Score (MOS) remains the quality gold standard, it introduces severe operational costs and turn-around delays that make continuous integration infeasible. In this work, we present a system design case study for a scalable, low-cost and offline evaluation pipeline capable of scoring large batches of synthetic speech at marginal cost. First, we use Qwen2-Audio-7B-Instruct in a zero-shot setting for granular MOS estimation, noticing perceptual blind spots and inherent biases. Then, we analyze SOMOS dataset and construct a diagnostic benchmark that operationalizes two prosodic dimensions associated with human naturalness ratings. Finally, we adapt Qwen2-Audio-7B-Instruct through a parameter-efficient finetuning (LoRA) on the task of MOS evaluation, observing improvements over base model and widely used neural baselines like UTMOS and establishing a practical blueprint for enterprise-grade speech quality assurance.
Figures & tables
Figure 1: Proposed framework design for adapting a base Audio LLM into a MOS prediction system.
Temperature
Pearson (r)
Base
-0.0312
0.5
0.1603
1.0
0.1681
1.5
0.2467
2.0
0.0422
Table 1: Pearson (r) correlation between the generated scores and ground-truth MOS values, across different temperatures.
Figure 2: Average MOS score for each characteristic.
Main
Easy
Model
Speed
Expressive
Content
Speed
Expressive
Content
Qwen2-Audio-7B-Instruct
21.9%
26.5%
29.5%
33%
33.3%
58.2%
Audio-Flamingo3
16.9%
27.4%
36.5%
39.8%
37.3%
42.3%
Table 2: Performance of each model on the benchmark, evaluating speech rate, expressiveness and content perception. The easy mode considers a 2-class classification.
mode + balance_ratio
LoRA
r pearson
SRCC
textual + 0.8
r = 16, a = 16
0.0406
0.0287
numerical + 0.8
r = 16, a = 16
0.6332
0.6438
numerical + 1.0
r = 16, a = 16
0.5517
0.5971
numerical + 1.0
r = 16, a = 32
0.5435
0.5986
numerical + 0.0
r = 16, a = 16
0.6371
0.6426
numerical + 0.0
r = 16, a = 32
0.6506
0.6582
Table 3: Pearson (r) and Spearman (SRCC) correlation between the utterance-level predictions and ground truth MOS scores across each different finetuning setting. Bold indicates the best result.
In this paper, we introduce GatherMOS, a novel framework that leverages large language models (LLM) as meta-evaluators to aggregate diverse signals into quality predictions. GatherMOS integrates lightweight acoustic descriptors with pseudo-labels from DNSMOS and VQScore, enabling the LLM to reason over heterogeneous inputs and infer perceptual mean opinion scores (MOS). We further explore both zero-shot and few-shot in-context learning setups, showing that zero-shot GatherMOS maintains stable performance across diverse conditions, while few-shot guidance yields large gains when support samples match the test conditions. Experiments on the VoiceBank-DEMAND dataset demonstrate that GatherMOS consistently outperforms DNSMOS, VQScore, naive score averaging, and even learning-based models such as CNN-BLSTM and MOS-SSL when trained under limited labeled-data conditions. These results highlight the potential of LLM-based aggregation as a practical strategy for non-intrusive speech quality evaluation.
Ryandhimas E. Zezario, Dyah A. M. G. Wisnu, Szu-Wei Fu +3
Academia Sinica · National Chengchi University · NVIDIA +1
Mean Opinion Score (MOS) is the gold standard for evaluating synthesized speech naturalness. However, current automatic MOS predictors are dominated by self-supervised learning (SSL) models that prioritize high-level semantics, potentially compromising their ability to capture critical acoustic details. In this paper, we systematically investigate representations from three paradigms: SSLs, acoustic-only neural audio codecs (NACs), and unified NACs that integrate semantics into reconstruction-based architectures. Extensive benchmarking on the standard BVCC and multiple out-of-domain (OOD) datasets demonstrates that features synergizing semantic understanding with fine-grained acoustic modeling achieve a higher performance upper bound in speech quality assessment. Ultimately, our findings highlight that semantics alone are not enough; a dual focus on semantic content and acoustic fidelity is essential for robust MOS prediction.
Tianyu Lan, Yufei Shi, Yang Ai +3
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China
Mean opinion score (MOS) prediction models are widely used as proxy metrics in text-to-speech (TTS) research, yet their ability to capture quality differences beyond acoustic fidelity remains unclear. We investigate this via controlled perturbations on speech: acoustic degradation, prosodic errors, and manipulation of speaker-specific characteristics such as pitch and speaking rate. We obtained MOS predictions for these speech samples from both human listeners and the model, and analyzed the differences in their perceptual characteristics. Results show that most models track acoustic degradation well, while all are insensitive to prosodic errors despite large subjective score drops. For speaker characteristics, models exhibit a double dissociation: strong mean fundamental frequency (F0) biases absent in human ratings, yet insensitivity to speaking rate and F0 variability that humans notice. These findings highlight limitations of scalar MOS prediction beyond acoustic fidelity.
Masato Takagi, Masaya Kawamura, Reo Shimizu +1
Nagoya Institute of Technology, Japan · LY Corporation, Japan