Perceptual Differences and Preferences in Speech Enhancement

Simon Dahl Jepsen; Jesper Rindom Jensen

eess.ASOct 27, 2026

Perceptual Differences and Preferences in Speech Enhancement

Authors: Simon Dahl Jepsen, Jesper Rindom Jensen

Organizations: Audio Analysis Lab, Dept. Electronic Systems, Aalborg University

Abstract

Speech enhancement (SE) systems are commonly evaluated with objective metrics, with and without references, and with categorical subjective listening tests, all measuring speech quality and intelligibility. So far, it is unclear from such evaluations whether the processed speech is perceptually different or is preferred by everyday listeners. We investigate discrimination and preference as complementary dimensions of SE evaluation through crowdsourced triangle and pairwise preference tests across four SE systems, with comparisons to clean references. Results show that both listeners' ability to distinguish and whether they exhibit systematic preference vary across systems and substantially across acoustic conditions. Pairwise evaluations of objective metrics showed variable correlation with both discriminability and preference, motivating further studies into these novel aspects in speech enhancement evaluation.

Figures & tables

Figure 1: Triangle test discrimination rate across system pairs. Each open dot is the empirically estimated discrimination rate of an utterance/system-pair condition as in 2 . The filled dots represent the average across utterances as in 3 . The error bars represent the

95\%

CI by bootstrap resampling.

Figure 2: Triangle test discrimination rate across utterances. Each open dot is the empirically estimated discrimination rate of an utterance/system-pair condition as in 2 . The filled dots represent the average across system pairs for the given utterance.

Metric	$\rho_{D}$	$\rho_{\mathrm{dec}}$	$\rho_{\mathrm{pref}}$	Metric	$\rho_{D}$	$\rho_{\mathrm{dec}}$	$\rho_{\mathrm{pref}}$
$\textrm{SQUIM}_{\textrm{STOI}}$	0.62	0.56	0.62	NISQA-MOS	0.32	0.20	0.33
SCOREQ	0.59	0.54	0.75	$\textrm{DNSMOS}_{\textrm{OVRL}}$	0.29	0.27	0.31
$\textrm{SQUIM}_{\textrm{PESQ}}$	0.57	0.53	0.62	CSIG	0.20	0.12	0.05
Distill-MOS	0.51	0.51	0.41	$\textrm{DNSMOS}_{\textrm{SIG}}$	0.18	0.18	0.08
$\textrm{SQUIM}_{\textrm{SI-SDR}}$	0.49	0.55	0.66	NISQA-Disc.	0.21	0.13	0.30
ESTOI	0.43	0.38	0.48	COVL	0.17	0.13	0.17

Table 1: Spearman rank correlations between objective metric differences and perceptual responses.

\rho_{D}

relates absolute metric differences

|\Delta m|

to triangle-test discrimination,

\rho_{\mathrm{dec}}

relates

|\Delta m|

to preference decisiveness, and

\rho_{\mathrm{pref}}

relates signed metric differences

\Delta m

to mean preference direction. Conditions containing the clean reference are excluded for intrusive metrics (

N=81

); otherwise

N=135

Figure 3: Empirical discrimination rate ( 3 ) and preference direction ( 2.2 ) a function of paired difference of PESQ and

\mathrm{SQUIM}_{\mathrm{STOI}}

. The curves in the top row show the chance constrained psychometric fit ( 4 ), and the dashed line denotes the chance performance. The bottom row shows the bidirectional logistic fit ( 8 ) Conditions including clean references are excluded for PESQ.

Figure 4: Decisiveness of the preference test across system pairs. Each open dot is the empirically estimated decisiveness rate of an utterance/system-pair condition as in 5 . The filled dots represent the average across utterances. The error bars represent the

95\%

CI by bootstrap resampling.

Figure 5: Directional preference across system pairs. Each open dot is the empirically estimated decisiveness rate of an utterance/system-pair condition as in 2.2 . The filled dots represent the average across utterances. The error bars represent the

95\%

CI by bootstrap resampling.

Speech Enhancement

Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.

See all 129 papers

How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?
Sep 30, 2026Nahomi Kusunoki, Tsubasa Ochiai, Naohiro Tawara +4Human Preference Evaluation Speech Quality Assessment
Waseda University, Japan · NTT, Inc., Japan
Improving Predicted MOS Scores, Not Perceived Quality: Multi-Predictor Test-Time Optimization of Enhanced Speech
Sep 30, 2026Tsubasa Ochiai, Marc Delcroix, Nahomi Kusunoki +5Reward Hacking Test-Time Optimization
NTT, Inc., Japan · Waseda University, Japan
Perceptual Quality Loss or Loss of Perceptual Quality?
Sep 28, 2026Danilo de Oliveira, Tal Peer, Maurício do V. M. da Costa +1Perceptual Loss Speech Quality Assessment
Signal Processing, University of Hamburg, Germany

Sep 30, 2026J/K move · Enter open · S save

How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?

Nahomi Kusunoki, Tsubasa Ochiai, Naohiro Tawara, Marc Delcroix+3

Waseda University, Japan · NTT, Inc., Japan

Human Preference Evaluation Speech Quality Assessment

Open paper AskPDF arXiv

We investigate whether predicted Mean Opinion Scores (MOS) can reliably support system-level comparisons of speech enhancement (SE) methods by introducing system-level preference accuracy (SPA). Although MOS prediction models are widely used to evaluate SE systems, their performance is typically assessed by correlation with human-rated MOS, which does not guarantee agreement on which system is better. SPA addresses this gap by directly evaluating whether predicted and human-rated MOS yield the same system preferences. Using SPA, we systematically evaluate three settings: single prediction models, ensembling, and domain adaptation. Through experiments, SPA varies substantially across single prediction models, from 9.4% to 76.8%. Even the best model disagrees with human judgments in approximately 23% of system comparisons. Ensembling yields only limited improvement, while domain adaptation tends to substantially improve SPA in the closed condition but brings only modest gains in the more practical open condition, where neither the target systems nor the speakers are known. These results suggest that SPA can reveal errors correlation-based evaluation alone does not expose, and that predicted MOS alone can lead to unreliable conclusions in practical SE system comparison.

Abstract

Speech Enhancement

Explore similar work

How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?

Improving Predicted MOS Scores, Not Perceived Quality: Multi-Predictor Test-Time Optimization of Enhanced Speech

Perceptual Quality Loss or Loss of Perceptual Quality?