Scalable Audio-LLM System for Speech MOS Evaluation: A Diagnostic Case Study and Adaptation Roadmap
Organizations: Innoetics, Samsung Electronics, Greece · Mobile eXperience Business, Samsung Electronics, Republic of Korea
Abstract
The continuous deployment of enterprise voice systems creates a critical operational bottleneck: evaluating the acoustic quality of synthetic speech across high-volume production updates. While human Mean Opinion Score (MOS) remains the quality gold standard, it introduces severe operational costs and turn-around delays that make continuous integration infeasible. In this work, we present a system design case study for a scalable, low-cost and offline evaluation pipeline capable of scoring large batches of synthetic speech at marginal cost. First, we use Qwen2-Audio-7B-Instruct in a zero-shot setting for granular MOS estimation, noticing perceptual blind spots and inherent biases. Then, we analyze SOMOS dataset and construct a diagnostic benchmark that operationalizes two prosodic dimensions associated with human naturalness ratings. Finally, we adapt Qwen2-Audio-7B-Instruct through a parameter-efficient finetuning (LoRA) on the task of MOS evaluation, observing improvements over base model and widely used neural baselines like UTMOS and establishing a practical blueprint for enterprise-grade speech quality assurance.
Figures & tables
| Temperature | Pearson (r) |
|---|---|
| Base | -0.0312 |
| 0.5 | 0.1603 |
| 1.0 | 0.1681 |
| 1.5 | 0.2467 |
| 2.0 | 0.0422 |
| Main | Easy | |||||
|---|---|---|---|---|---|---|
| Model | Speed | Expressive | Content | Speed | Expressive | Content |
| Qwen2-Audio-7B-Instruct | 21.9% | 26.5% | 29.5% | 33% | 33.3% | 58.2% |
| Audio-Flamingo3 | 16.9% | 27.4% | 36.5% | 39.8% | 37.3% | 42.3% |
| mode + balance_ratio | LoRA | r pearson | SRCC |
|---|---|---|---|
| textual + 0.8 | r = 16, a = 16 | 0.0406 | 0.0287 |
| numerical + 0.8 | r = 16, a = 16 | 0.6332 | 0.6438 |
| numerical + 1.0 | r = 16, a = 16 | 0.5517 | 0.5971 |
| numerical + 1.0 | r = 16, a = 32 | 0.5435 | 0.5986 |
| numerical + 0.0 | r = 16, a = 16 | 0.6371 | 0.6426 |
| numerical + 0.0 | r = 16, a = 32 | 0.6506 | 0.6582 |
Speech Quality Assessment
Speech quality assessment measures how well speech preserves perceptual quality, intelligibility, pronunciation, naturalness, and speaker characteristics.
See all 75 papers