ConformalMOS: Uncertainty-Aware MOS Prediction with Conformal Intervals and Ordinal Modeling

Kehinde Elelu; Joshua E. Siegel; Mohammadali Saffary; Tashfain Ahmed; Simeon Babatunde; Ebuka Okpala

cs.SDOct 26, 2026

ConformalMOS: Uncertainty-Aware MOS Prediction with Conformal Intervals and Ordinal Modeling

Authors: Kehinde Elelu, Joshua E. Siegel, Mohammadali Saffary, Tashfain Ahmed, Simeon Babatunde, Ebuka Okpala

Organizations: Michigan State University, East Lansing, MI, USA · Clemson University, Clemson, SC, USA

Abstract

Accurate prediction of human Mean Opinion Scores (MOS) is crucial for evaluating synthetic speech quality in text-to-speech (TTS) and voice conversion (VC). Most MOS predictors output only point estimates and ignore uncertainty, limiting reliability in model selection and deployment. We propose ConformalMOS, a framework that augments MOS prediction with conformal prediction to produce statistically valid prediction intervals with guaranteed coverage under exchangeability, alongside point estimates. During training, MOS labels are modeled in an ordinal-aware manner by converting one-hot targets into Gaussian-smoothed distributions, preserving score ordering. This improves uncertainty calibration, particularly when softmax confidence is overconfident on out-of-distribution speech. On BVCC datasets, ConformalMOS achieves competitive accuracy (MSE = 0.08) while delivering empirically validated coverage, enhancing reliability for real-world TTS and VC deployment.

Figures & tables

Figure 1: Overview of the proposed ConformalMOS framework. Input audio representations are first processed by a frozen upstream feature extractor. These features are fed into an ordinal prediction head, which uses Gaussian label smoothing to produce point MOS estimates. Finally, post-hoc conformal calibration generates statistically valid prediction intervals, providing reliability-aware predictions.

Model	Utterance				System
Model	MSE↓	LCC↑	SRCC↑	KTAU↑	MSE↓	LCC↑	SRCC↑	KTAU↑
UTMOS	0.165	0.899	0.896	0.730	0.090	0.936	0.936	0.795
FUSE-MOS	0.191	0.887	0.887	-	0.086	0.946	0.947	-
Our Models:
M2D2 ( $\alpha=0.01$ )	0.229	0.877	0.875	0.697	0.082	0.952	0.941	0.802
M2D2 ( $\alpha=0.05$ )	0.228	0.878	0.876	0.698	0.080	0.953	0.943	0.805

Table 1: Comparison of FUSE-MOS, prior methods, and Conformal-MOS (Ours) on BVCC datasets. Best values in each column are in boldface .

Panel of Figure 2 — Figure 2: Calibration performance of Conformal-MOS on the BVCC dataset using two different backbones. (a) M2D2 achieves empirical coverage closely matching nominal targets across multiple $\alpha$ levels, with low calibration error and relatively narrow (sharp) intervals, indicating reliable uncertainty estimates. (b) Wav2Vec shows larger calibration error, empirical coverage that falls short at low $\alpha$ levels, and wider intervals, highlighting less reliable uncertainty estimation.

Speech Quality Assessment

Speech quality assessment measures how well speech preserves perceptual quality, intelligibility, pronunciation, naturalness, and speaker characteristics.

See all 69 papers

ProxyMOS: Label-Free Speech Quality Assessment by Multi-Teacher Distillation with Adaptive Routing
Sep 30, 2026Maxim Trokunov, Kirill Borodin, Nikita Vasiliev +1Ensemble Learning TTS Evaluation
Lab260, Yerevan, Armenia · BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia
Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations
Jun 18, 2026Masato Takagi, Masaya Kawamura, Reo Shimizu +1Human Preference Evaluation Speech Generation Evaluation
Nagoya Institute of Technology, Japan · LY Corporation, Japan
Is Semantics Enough for Speech Mean Opinion Score Prediction?
Sep 3, 2026Tianyu Lan, Yufei Shi, Yang Ai +3Human Preference Evaluation Neural Audio Codecs
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China

Sep 30, 2026J/K move · Enter open · S save

ProxyMOS: Label-Free Speech Quality Assessment by Multi-Teacher Distillation with Adaptive Routing

Maxim Trokunov, Kirill Borodin, Nikita Vasiliev, Grach Mkrtchian

Lab260, Yerevan, Armenia · BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia

Ensemble Learning TTS Evaluation Speech Quality Assessment Knowledge Distillation

Open paper AskPDF arXiv

Human mean opinion scores (MOS) are costly to collect, and non-intrusive MOS predictors degrade sharply outside their training domain. ProxyMOS turns a pool of public MOS predictors into a single stronger model without new human labels. Eight predictors are benchmarked against human ratings; the five most informative enter a subset search under uniform, correlation-weighted, error-weighted, MSE-optimised and adaptive per-utterance routing; and the best routed four-model ensemble labels 807k unlabeled utterances that train a wav2vec 2.0 student. On URGENT the student reaches Spearman

ρ=0.802

against

0.773

for the best teacher. On mos260, a new Russian TTS benchmark of 4,600 utterances from 38 synthesis conditions, it reaches

ρ=0.636

against

0.613

per utterance and

0.95

per condition, matching its own routed ensemble in one forward pass. Adaptive routing is the only rule that does not degrade when weak predictors are added. Model, ONNX exports and mos260 are released. It's about 950 characters; arXiv's limit is 1,920. I kept

ρ

because arXiv renders it on the abstract page. If you'd rather avoid math, replace

ρ=0.802

with rho = 0.802 and do the same for the other

...

values.

Abstract

Speech Quality Assessment

Explore similar work

ProxyMOS: Label-Free Speech Quality Assessment by Multi-Teacher Distillation with Adaptive Routing

Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations

Is Semantics Enough for Speech Mean Opinion Score Prediction?