Physiology-Informed Audio Quality and Listening Effort Estimation

Niall McGuire; Dimitra Emmanouilidou

cs.SDOct 27, 2026

Physiology-Informed Audio Quality and Listening Effort Estimation

Authors: Niall McGuire, Dimitra Emmanouilidou

Organizations: University of Strathclyde, Glasgow, UK · Microsoft Research, Redmond, WA, USA

Abstract

Audio systems aim to maximise the listener's perceived quality, traditionally evaluated through mean opinion scoring (MOS), a subjective process that is slow, costly, and collected only after playback. Machine learning now predicts MOS directly from the audio, but in generalising across participants it tends to discard the subject variance that defines individual perception. Neurophysiological measures may offer a complementary signal taken directly from the listener, objective, implicit, and continuous, though prior work has relied on laboratory-grade EEG and fNIRS. We ask how practical consumer-grade devices compare, and whether they might recover population-level audio-quality trends and, when calibrated per listener, track that listener's effort. Recording 32-channel laboratory EEG alongside wearable EEG, fNIRS, and eye tracking from thirty listeners, our results suggest commercial wearable and optical sensing can decode audio quality at statistically significant levels and recover listening effort per listener. These findings may point towards platforms tuned not only to perceived quality but to the effort a clip demands, read from the listener's own response as they listen.

Figures & tables

	AQ-2								EF-2
	Full				Matched				Full				Matched
Modality	$N$	Trials	LOSO	WS	$N$	Trials	LOSO	WS	$N$	Trials	LOSO	WS	$N$	Trials	LOSO	WS
Muse EEG	25	1023	$\underline{0.64^{***}_{\pm.12}}$	$\mathbf{0.67^{***}_{\pm.05}}$	8	122	$\underline{0.55_{\pm.23}}$	$0.48_{\pm.11}$	28	1613	$0.52_{\pm.13}$	$\underline{0.61^{***}_{\pm.04}}$	11	236	$\underline{0.52_{\pm.19}}$	$\mathbf{0.59_{\pm.05}}$
fNIRS	17	610	$0.60^{***}_{\pm.09}$	$0.58^{***}_{\pm.04}$	8	122	$\mathbf{0.65^{***}_{\pm.11}}$	$\underline{0.53_{\pm.08}}$	21	1235	$0.52^{*}_{\pm.04}$	$0.58^{**}_{\pm.02}$	11	236	$0.48_{\pm.18}$	$\underline{0.52_{\pm.05}}$
Eye	25	1164	$\mathbf{0.65^{***}_{\pm.17}}$	$\underline{0.66^{***}_{\pm.04}}$	8	122	$0.44_{\pm.09}$	$\mathbf{0.54_{\pm.06}}$	29	1891	$\mathbf{0.57_{\pm.17}}$	$\mathbf{0.67^{***}_{\pm.02}}$	11	236	$\underline{0.52_{\pm.07}}$	$0.50_{\pm.04}$
Lab EEG	12	218	$0.51_{\pm.09}$	$0.37^{**\downarrow}_{\pm.08}$	8	122	$0.50_{\pm.07}$	$0.47_{\pm.07}$	18	381	$\underline{0.56_{\pm.23}}$	$0.58_{\pm.04}$	11	236	$\mathbf{0.56_{\pm.17}}$	$\mathbf{0.59_{\pm.12}}$

Table 1: Binary tasks (AQ-2, EF-2), balanced accuracy against chance

0.5

, higher better, mean ±SD over folds. Best per column bold , second underlined . LOSO tests an unseen listener, WS a calibrated one. Full uses all of a modality’s trials, matched only those common to all four.

{}^{\ast}/^{\ast\ast}/^{\ast\ast\ast}

mark Holm

p{<}.05/.01/.001

vs. chance,

\downarrow

below.

	AQ-5								EF-5
	Full				Matched				Full				Matched
Modality	$N$	Trials	LOSO	WS	$N$	Trials	LOSO	WS	$N$	Trials	LOSO	WS	$N$	Trials	LOSO	WS
Muse EEG	29	3183	$\mathbf{0.38^{***}_{\pm.14}}$	$\mathbf{0.43^{***}_{\pm.03}}$	12	573	$0.07^{*}_{\pm.10}$	$0.23_{\pm.04}$	28	3193	$0.09_{\pm.23}$	$\underline{0.23^{***}_{\pm.05}}$	11	526	$\mathbf{0.02_{\pm.19}}$	$\mathbf{0.17_{\pm.08}}$
fNIRS	22	2417	$\underline{0.36^{***}_{\pm.11}}$	$\underline{0.41^{***}_{\pm.05}}$	12	573	$\mathbf{0.19^{***}_{\pm.15}}$	$\mathbf{0.34^{***}_{\pm.03}}$	21	2400	$0.02_{\pm.11}$	$0.13^{***}_{\pm.06}$	11	526	$-0.13_{\pm.31}$	$0.07_{\pm.05}$
Eye	30	3750	$0.20^{**}_{\pm.29}$	$\underline{0.41^{***}_{\pm.04}}$	12	573	$\underline{0.10^{*}_{\pm.15}}$	$\underline{0.29_{\pm.04}}$	29	3761	$\mathbf{0.14_{\pm.37}}$	$\mathbf{0.37^{**}_{\pm.03}}$	11	526	$\underline{-0.10_{\pm.19}}$	$\underline{0.11_{\pm.07}}$
Lab EEG	19	867	$0.05_{\pm.33}$	$0.15^{*}_{\pm.05}$	12	573	$-0.06_{\pm.15}$	$0.13_{\pm.22}$	18	816	$\underline{0.12_{\pm.44}}$	$0.13_{\pm.09}$	11	526	$-0.18_{\pm.35}$	$0.08_{\pm.14}$

Table 2: Regression tasks (AQ-5, EF-5), Pearson

r

against null

0

, higher better, mean ±SD over folds. Best per column bold , second underlined . Support, regimes, and markers as in Table 1 . Full-support RMSE, untested, spans 1.18–1.30 MOS units (AQ-5) and 0.83–0.96 rating units (EF-5).

Panel of Figure 1 — Figure 1: Behavioural validation against objective MOS. Left: subjective quality tracks MOS ( $r{=}0.75$ ). Right: mental effort rises as quality falls ( $r{=}{-}0.57$ ). Cells are rating counts, dashed line marks agreement.

			LOSO		WS
Modality	Trials	$P_{\mathrm{sp}}/P_{\mathrm{mu}}$	Sp	Mu	Sp	Mu
Muse EEG	334	11/23	$\mathbf{0.83_{\pm.12}}$	$\underline{0.57_{\pm.14}}$	$\mathbf{0.83_{\pm.09}}$	$0.50_{\pm.21}$
fNIRS	96	3/16	$0.75_{\pm.03}$	$0.48_{\pm.20}$	$\underline{0.73_{\pm.05}}$	$\underline{0.53_{\pm.22}}$
Eye	352	11/25	$\mathbf{0.83_{\pm.16}}$	$\mathbf{0.59_{\pm.21}}$	$\mathbf{0.83_{\pm.15}}$	$\mathbf{0.62_{\pm.19}}$
Lab EEG	34	4/9	$\underline{0.78_{\pm.06}}$	$0.38_{\pm.24}$	$0.69_{\pm.10}$	$0.52_{\pm.27}$

Table 3: Binary quality (AQ-2) by content, speech (Sp) vs. music (Mu). Participant-mean balanced accuracy against chance

0.5

, mean ±SD , best per column bold , second underlined .

P

is the speech/music listener count. Descriptive, single seed.

LOSO
			RMSE		$r$
Modality	$N$	Trials	Subj	Obj	Subj	Obj
Muse EEG	29	3183	$1.38_{\pm.05}$	$1.20_{\pm.09}$	$0.19_{\pm.13}$	$0.38_{\pm.14}$
fNIRS	22	2417	$1.38_{\pm.04}$	$1.18_{\pm.07}$	$0.18_{\pm.11}$	$0.36_{\pm.11}$
Eye	30	3750	$1.35_{\pm.16}$	$1.24_{\pm.16}$	$0.19_{\pm.30}$	$0.20_{\pm.29}$
Lab EEG	19	867	$1.47_{\pm.11}$	$1.30_{\pm.16}$	$0.01_{\pm.47}$	$0.05_{\pm.33}$

Table 4: Quality supervised by the listener’s rating (Subj) or the corpus MOS (Obj), both scored against corpus MOS. RMSE in MOS units (lower better) and Pearson

r

(higher better), mean ±SD . The Obj arm is the deployed model of Table 2 , so the gap bounds the benefit of objective supervision rather than isolating it. Descriptive, single seed.

Speech Quality Assessment

Speech quality assessment measures how well speech preserves perceptual quality, intelligibility, pronunciation, naturalness, and speaker characteristics.

See all 69 papers

Oct 5, 2026

Ensemble-Based Perceptual Audio Quality Assessment with Confidence Intervals

Objective audio quality metrics typically provide point estimates, whereas listening tests yield score distributions from which mean opinion scores (MOS), confidence intervals (CIs), and significance decisions are derived. We propose a lightweight intrusive metric that combines a PEAQ-style perceptual front-end (ITU-R BS.1387) with a bagging ensemble of regressors. Calibration with subjective data aligns ensemble outputs with listener scores. The resulting item-dependent score distributions enable uncertainty assessment, panel-size-matched CIs, and identification of less conclusive predictions. Calibration improves agreement with subjective distributions and CI coverage across all evaluated datasets while preserving MOS accuracy. Using only 11 fixed PEAQ features, low-capacity regressors, and public training data, the method performs comparably to more data-intensive end-to-end approaches. Its output can support uncertainty-aware assessment and target listening tests towards uncertain conditions. The distributions also enable approximate pairwise comparisons, but not yet reliable significance inference.

Pablo M. Delgado, Andreas Brendel, Konstantin Schmidt +1

Fraunhofer Institute for Integrated Circuits IIS, Erlangen, Germany · International Audio Laboratories Erlangen †, Germany

Uncertainty Quantification

Ask PDF

Sep 25, 2026

MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

Humans rely on gaze, head movements, and visual cues to attend to speakers in noisy environments, yet auditory attention decoding (AAD) has been studied primarily using electroencephalography (EEG). We introduce the Multimodal Auditory-attention Egocentric Speech-TRacking Open (MAESTRO) corpus, the first AAD dataset to simultaneously record EEG, eye gaze, pupillometry, egocentric video, and head inertial measurement unit (IMU) data. MAESTRO includes four competing speakers and background noise across multiple signal-to-noise ratio (SNR) conditions, enabling attention decoding under realistic listening scenarios. Through a four-speaker attention decoding benchmark, we show that combining behavioral and physiological signals improves decoding performance over EEG-only approaches, enabling future advances in multimodal auditory attention decoding. These findings open the door to new applications, analyses, and methodological advances in multimodal AAD. The complete dataset is publicly available at https://huggingface.co/datasets/aspire-osu/maestro-eeg-dataset . The official code repository is available at https://github.com/ASPIRE-OSU/MAESTRO .

K M Naimul Hassan, Ali Alavi, Donald S. Williamson

Department of Computer Science and Engineering, The Ohio State University, Columbus, OH 43210 USA · Center for Cognitive and Brain Sciences, The Ohio State University

EEG Decoding Neural Decoding Multimodal Sensor Fusion

Ask PDF

Apr 27, 2026

All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation

Large Audio-Language Models show consistent performance gains across speech and audio benchmarks, yet high scores may not reflect true auditory perception. If a model can answer questions without processing the acoustic signal, the benchmark fails as a measure of auditory understanding. We present a diagnostic framework using two axes: text prior, which measures answerability from text and general knowledge alone, and audio reliance, which assesses actual dependency on the acoustic signal. Evaluating eight LALMs across three benchmarks, we find that models retain 60-72% of their full audio scores even without any audio input. Moreover, among items that require audio, only 3.0-4.2% need the complete audio clip; the majority can be resolved using localized fragments. These findings challenge the assumption that benchmark performance equals robust audio understanding, and we conclude with practical guidelines for improving evaluation reliability and benchmark design.

Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li +2

National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)

Audio-Language Model Evaluation Modality Reliance in Multimodal Models Audio QA Audio Understanding Audio-Language Models Benchmark Validity+2

Ask PDF

Abstract

Speech Quality Assessment

Explore similar work

Ensemble-Based Perceptual Audio Quality Assessment with Confidence Intervals

MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation