Improving Predicted MOS Scores, Not Perceived Quality: Multi-Predictor Test-Time Optimization of Enhanced Speech

Tsubasa Ochiai; Marc Delcroix; Nahomi Kusunoki; Rintaro Ikeshita; Naohiro Tawara; Naoyuki Kamo; Tetsuji Ogawa; Shoko Araki

eess.ASSep 30, 2026

Improving Predicted MOS Scores, Not Perceived Quality: Multi-Predictor Test-Time Optimization of Enhanced Speech

Authors: Tsubasa Ochiai, Marc Delcroix, Nahomi Kusunoki, Rintaro Ikeshita, Naohiro Tawara, Naoyuki Kamo, Tetsuji Ogawa, Shoko Araki

Organizations: NTT, Inc., Japan · Waseda University, Japan

Abstract

Non-intrusive MOS predictors are widely used instead of subjective listening tests to evaluate and rank speech enhancement (SE) systems. If they accurately reflect perceived quality, raising their scores should lead to higher-quality speech. We present the first comprehensive analysis of test-time optimization for the SE task, which directly modifies the enhanced signal to raise the average of multiple MOS predictor scores. On seven systems from the URGENT 2026 challenge, we find that 1)~all the optimized predicted scores increase while reference-based metrics remain nearly unchanged, 2)~a non-optimized predicted score does not increase, and 3)~a MUSHRA listening test shows no improvement in perceived quality. These findings reveal a risk that such optimization can distort evaluations, e.g., biasing comparisons of SE systems regardless of their perceived quality. We believe these findings can inform future evaluation practices: they suggest that predictors used for optimization should not be used for evaluation, and that challenges should keep the predictors used for ranking undisclosed.

Figures & tables

		Non-intrusive (Optimized)				Unseen	Intrusive		Task-indep.		Task-dependent				Rank
System	$\lambda$	DNSMOS	NISQA	UTMOS	SCOREQ	Distill	PESQ	ESTOI	SBERT	LPS	SpkSim	EmoSim	LID	CAcc	non-int.	overall
WR	$-$	$3.06$	$3.37$	$2.77$	$3.67$	$3.68$	$2.81$	$0.86$	$0.89$	$0.83$	$0.76$	$0.99$	$0.96$	$88.16$	3	1
	$0.005$	$+0.24$	$+1.19$	$+0.15$	$+0.12$	$+0.02$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$+0.05$	3 $\to$ 1	1 $\to$ 1
	$0.05$	$+0.50$	$+1.77$	$+0.54$	$+0.28$	$+0.01$	$-0.06$	$-0.01$	$-0.01$	$0.00$	$0.00$	$0.00$	$0.00$	$+0.43$	3 $\to$ 1	1 $\to$ 1
subatom.	$-$	$2.96$	$3.38$	$2.41$	$3.32$	$3.36$	$2.71$	$0.84$	$0.88$	$0.79$	$0.76$	$0.99$	$0.95$	$87.76$	6	2
	$0.005$	$+0.49$	$+1.21$	$+0.21$	$+0.19$	$+0.03$	$-0.01$	$0.00$	$-0.01$	$0.00$	$0.00$	$0.00$	$0.00$	$-0.66$	6 $\to$ 2	2 $\to$ 2

Table 1: Scores and ranks of the enhanced signal

\hat{\mathbf{x}}

(shaded rows), and score differences and rank changes between the modified signal

\tilde{\mathbf{x}}

and

\hat{\mathbf{x}}

. The enhanced signal is jointly optimized for the four predictors, except in the rows marked (S), where it is optimized for DNSMOS alone.

Comparison	$\Delta$ MUSHRA	95% CI	$p$	Equiv. bound
RICK $\,-\,$ baird	$+8.48$	$[+2.5,+14.5]$	$0.011$	13.4
baird, $\lambda\!=\!0.005$	$-0.22$	$[-2.2,+1.7]$	$0.81$	$1.8$
baird, $\lambda\!=\!0.05$	$-1.20$	$[-5.0,+2.6]$	$0.50$	$4.3$
RICK, $\lambda\!=\!0.005$	$-0.65$	$[-2.6,+1.3]$	$0.47$	$2.2$

Table 2: Differences in the MUSHRA scores, with a

95\%

confidence interval and the smallest equivalence bound at the

5\%

level.

Explore similar work

How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?
Sep 30, 2026Nahomi Kusunoki, Tsubasa Ochiai, Naohiro Tawara +4Mean Opinion Scores Speech Enhancement
Waseda University, Japan · NTT, Inc., Japan
PrefSQA: Pairwise Preference Prediction for Speech Quality Assessment and the Critical Role of High Quality Datasets
Jun 17, 2026Junyi Fan, Donald S. WilliamsonMean Opinion Scores Speaker
Department of Computer Science and Engineering, The Ohio State University, USA
Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations
Jun 18, 2026Masato Takagi, Masaya Kawamura, Reo Shimizu +1Seed-Tts-Eval Benchmark Prosody
Nagoya Institute of Technology, Japan · LY Corporation, Japan

Sep 30, 2026, cs.SDJ/K move · Enter open · S save

How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?

Nahomi Kusunoki, Tsubasa Ochiai, Naohiro Tawara, Marc Delcroix+3

Waseda University, Japan · NTT, Inc., Japan

Mean Opinion Scores Speech Enhancement Speaker Preference Alignment Learning Reliable Domain Adaptation

Open paper AskPDF arXiv

We investigate whether predicted Mean Opinion Scores (MOS) can reliably support system-level comparisons of speech enhancement (SE) methods by introducing system-level preference accuracy (SPA). Although MOS prediction models are widely used to evaluate SE systems, their performance is typically assessed by correlation with human-rated MOS, which does not guarantee agreement on which system is better. SPA addresses this gap by directly evaluating whether predicted and human-rated MOS yield the same system preferences. Using SPA, we systematically evaluate three settings: single prediction models, ensembling, and domain adaptation. Through experiments, SPA varies substantially across single prediction models, from 9.4% to 76.8%. Even the best model disagrees with human judgments in approximately 23% of system comparisons. Ensembling yields only limited improvement, while domain adaptation tends to substantially improve SPA in the closed condition but brings only modest gains in the more practical open condition, where neither the target systems nor the speakers are known. These results suggest that SPA can reveal errors correlation-based evaluation alone does not expose, and that predicted MOS alone can lead to unreliable conclusions in practical SE system comparison.

Abstract

Explore similar work

How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?

PrefSQA: Pairwise Preference Prediction for Speech Quality Assessment and the Critical Role of High Quality Datasets

Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations