Label Authority in Speech Emotion Recognition: Intended and Perceived Ground Truth in Emozionalmente

Marco Giordano; Stefano Giacomelli; Claudia Rinaldi; Fabio Graziosi

cs.SDOct 19, 2026

Label Authority in Speech Emotion Recognition: Intended and Perceived Ground Truth in Emozionalmente

Authors: Marco Giordano, Stefano Giacomelli, Claudia Rinaldi, Fabio Graziosi

Organizations: Department of Information Engineering, Computer Science and Mathematics (DISIM), University of L’Aquila, L’Aquila, 67100, ITA · National Inter-University Consortium for Telecommunications (CNIT), University of L’Aquila, L’Aquila, 67100, ITA

Abstract

Ground truth in Speech Emotion Recognition (SER) is often presented as a single categorical fact, yet an acted emotion and the emotion heard by listeners describe different aspects of the same recording. We examine this distinction in Emozionalmente, an acted Italian corpus that provides both an elicited production label and five listener judgements for every clip. Using frozen XLS-R, HuBERT, and WavLM representations with matched probes, we compare three label authorities: the expressed condition, the listeners' hard plurality, and their complete vote distribution. The results tell a consistent story. Recovering the expressed condition is easier than predicting a tie-broken listener plurality, but that gap compares two different questions rather than revealing a general loss of SER ability. As listener agreement weakens, the plurality becomes an increasingly brittle hard target. Training on the full vote distribution does not substantially change which class wins the model's hard decision, but it produces probability assignments that more closely follow the alternatives selected by listeners, especially for ambiguous clips. No authority therefore replaces the others. Expressed labels describe how speech was produced, pluralities provide an operational single-class decision, and vote distributions preserve perceptual uncertainty. For human-facing Internet of Sounds applications, evaluations should state which authority is being predicted, retain its aggregation rule and agreement information, and use distributional scores when disagreement is part of the target.

Figures & tables

Authority	Construction	Reference unit	Question answered
Expressed condition	One-hot elicitation label	One production instruction per clip	Was the requested acted condition recoverable from speech?
Recognized plurality	Most frequent of five votes; deterministic tie-break	One aggregated listener-derived class per clip	Which single class wins under the stated aggregation rule?
Recognized distribution	Seven-class vector of vote proportions	Complete five-vote histogram per clip	How should probability be allocated across the alternatives selected by listeners?
Evaluator-level view	Each recognition vote retained separately	One listener judgement, paired with the clip’s expressed label	How much individual perception is hidden by clip-level aggregation?

TABLE I: Operational label authorities. “Reference unit” identifies what one evaluation item represents; it is not a claim that one authority is universally more truthful than another.

Component	Setting
Backbones	XLS-R 300M, HuBERT Large, WavLM Large; frozen
Single layers	0, 5, 8, 10, 13, 15, 18, 21, 22, 24
Probe families	linear , dense , agg_dense
Split	Speaker-independent 60/20/20; 4,142/1,380/1,380 clips
Training	Adam, $10^{-3}$ ; 30 epochs; five runs
Selection	Validation checkpoint; reported layer/winner by mean test UAR

TABLE II: Experimental factors held in common across label authorities.

Fig. 1: Expressed condition versus the plurality of all five listener votes. Rows are expressed labels and columns are recognized labels. The left panel shows counts and row-normalized rates, the centre suppresses the diagonal, and the right ranks expressed categories by their off-diagonal rate.

Training target	Best configuration	Layer	UAR
Expressed	XLS-R Dense	13	$0.825\pm 0.009$
Recognized plurality	XLS-R Dense	15	$0.746\pm 0.006$
Recognized distribution	WavLM Dense	10	$0.749\pm 0.004$

TABLE IV: Target-specific best own-reference hard UAR. Expressed models use the expressed reference; recognized models use plurality, with the soft target read by argmax. Values are mean

\pm

sample SD over five runs.

Training target	Hard UAR	KL	Brier
Expressed	$0.739\pm 0.002$	$1.667\pm 0.156$	$0.272\pm 0.009$
Rec. plurality	$0.731\pm 0.005$	$1.223\pm 0.145$	$0.236\pm 0.008$
Rec. distribution	$0.736\pm 0.005$	$\mathbf{0.510\pm 0.002}$	$\mathbf{0.162\pm 0.001}$

TABLE V: Common-reference evaluation, mean

\pm

sample SD over five runs after equally averaging nine target-selected backbone/probe configurations. Hard UAR uses recognized plurality; KL and Brier use the five-vote histogram.

Panel of Figure 6 — Fig. 2: Authority and agreement diagnostics. (a) Hard-UAR cross-evaluation: rows are training authorities and columns are hard evaluation authorities. Cells average nine target-selected configurations within each run and report mean $\pm$ sample SD over five runs. (b) Hard accuracy by mutually exclusive listener-agreement stratum. Expressed probes use the expressed reference; recognized probes use plurality.

Training target	Mean $\pm$ SD	Config. range
Expressed	$0.660\pm 0.013$	$[0.600,0.700]$
Recognized plurality	$0.707\pm 0.017$	$[0.680,0.740]$
Recognized distribution	$\mathbf{0.733\pm 0.014}$	$[0.680,0.780]$

TABLE VI: Top-10 model–human confusion-pair overlap. The mean averages nine selected backbone/probe configurations within each run; the SD is over five runs, and the range is across configuration means.

Panel of Figure 8 — Fig. 3: Alignment with the observed five-vote distribution: (a) KL divergence and (b) multiclass Brier score. Bars equally average nine target-selected configurations within each run; error bars are sample SDs over five runs. Lower is better.

Fig. 4: Distributional error against the observed vote histogram by human agreement stratum. Panels show cross-entropy, KL divergence, Jensen–Shannon divergence, and Brier score. Each point averages nine selected configurations within each run; error bars are sample SDs over five runs. KL and Brier provide the primary proper-score evidence; the other panels are supporting diagnostics.

Subset	Training	Hard UAR	KL	Brier
Tie ( $n=145$ )	Hard	$0.469\pm 0.015$	$2.121\pm 0.256$	$0.360\pm 0.014$
	Soft	$0.435\pm 0.012$	$\mathbf{0.671\pm 0.004}$	$\mathbf{0.184\pm 0.001}$
$<0.60$ ( $n=82$ )	Hard	$0.443\pm 0.010$	$2.415\pm 0.343$	$0.396\pm 0.019$
	Soft	$0.439\pm 0.006$	$\mathbf{0.696\pm 0.008}$	$\mathbf{0.184\pm 0.004}$

TABLE VII: Shared-reference results on ambiguous test subsets. Values are mean

\pm

sample SD over five runs after equally averaging nine selected configurations. Lower KL and Brier are better.

Fig. 5: Learned agg_dense mixture weights for expressed, recognized-plurality, and recognized-distribution supervision. Lines are five-run means and bands are sample SDs. Layer 0 is the convolutional state; layers 1–24 are Transformer states.

Target	Backbone	Best single layer	Single-layer UAR	Aggregate UAR	Paired $\Delta$
Expressed	XLS-R	Dense L13	$0.825\pm 0.010$	$0.801\pm 0.006$	$-0.023\pm 0.013$
	HuBERT	Dense L13	$0.808\pm 0.007$	$0.796\pm 0.008$	$-0.012\pm 0.014$
	WavLM	Dense L18	$0.807\pm 0.004$	$0.822\pm 0.010$	$+0.015\pm 0.010$
Recognized plurality	XLS-R	Dense L15	$0.746\pm 0.006$	$0.719\pm 0.016$	$-0.026\pm 0.012$
	HuBERT	Dense L13	$0.729\pm 0.007$	$0.721\pm 0.011$	$-0.008\pm 0.009$
	WavLM	Dense L10	$0.739\pm 0.007$	$0.739\pm 0.009$	$\approx 0\pm 0.013$

TABLE VIII: Best single-layer dense probe versus agg_dense . Values are hard UAR mean

\pm

sample SD over five aligned runs;

\Delta

is aggregation minus single layer. Soft-target models are evaluated against hard plurality.

Comparison	Paired-run $\Delta$	Conditional 95% CI
Expressed $-$ hard recognized, own-reference UAR	$+0.079\pm 0.014$	$[0.058,0.099]$
Soft $-$ hard recognized, recognized UAR	$+0.003\pm 0.008$	$[-0.013,0.019]$
Soft $-$ hard recognized, KL	$-0.360\pm 0.081$	$[-0.399,-0.325]$
Soft $-$ hard recognized, Brier	$-0.056\pm 0.015$	$[-0.066,-0.046]$

TABLE IX: Best-individual paired contrasts on all 1,380 test clips. The first column gives the mean

\pm

sample SD of five aligned run differences; the interval comes from 1,000 paired clip-bootstrap replicates and excludes training and model-selection uncertainty. Each contrast is left minus right.

Speech Emotion Recognition

Speech emotion recognition uses vocal signals to infer emotional states for affect-aware interfaces, health tools, and conversational systems.

See all 69 papers

Learning from Annotation Uncertainty: Entropy-Aware Curriculum for Speech Emotion Recognition
Jun 25, 2026Zahra Omidi, John H. L. HansenAnnotator Disagreement Label Distribution Learning
Center for Robust Speech Systems, The University of Texas at Dallas, USA
From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models
Sep 30, 2026Hezhao Zhang, Thomas HainConcept Bottleneck Models Emotion Recognition in Conversations
School of Computer Science, University of Sheffield Sheffield, United Kingdom
Explainable and Trustworthy Speech Emotion Recognition Using Confidence Score and Reinforcement Learning Rectified Speech Emotion Descriptors
Jun 12, 2026Youjun Chen, Xurong Xie, Mengzhe Geng +9Explainable Artificial Intelligence Reinforcement Learning
The Chinese University of Hong Kong, Hong Kong SAR, China · Institute of Software, Chinese Academy of Sciences, China · National Research Council Canada, Canada +1

Jun 25, 2026J/K move · Enter open · S save

Learning from Annotation Uncertainty: Entropy-Aware Curriculum for Speech Emotion Recognition

Zahra Omidi, John H. L. Hansen

Center for Robust Speech Systems, The University of Texas at Dallas, USA

Annotator Disagreement Label Distribution Learning Speech Emotion Recognition

Open paper AskPDF arXiv

Speech emotion recognition (SER) often relies on hard consensus labels that collapse annotator disagreement. We study distribution-based supervision for 9-class SER on MSP-Podcast 2.0 using a WavLM-Base multitask model for categorical emotion and dimensional VAD. Hard-label training is compared with targets from primary and merged primary--secondary annotator vote distributions. Distributional objectives improve alignment with human vote distributions, reducing JSD/KLD relative to hard-label training. Analysis shows that hard supervision partly benefits from assigning ambiguous utterances to the residual Other class, whereas distributional supervision redistributes uncertainty across emotion categories. Entropy-stratified evaluation shows that high-ambiguity utterances remain challenging, but distribution-based supervision better captures perceptual uncertainty. These findings support moving beyond hard labels toward targets that reflect listener disagreement.

Abstract

Speech Emotion Recognition

Explore similar work

Learning from Annotation Uncertainty: Entropy-Aware Curriculum for Speech Emotion Recognition

From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models

Explainable and Trustworthy Speech Emotion Recognition Using Confidence Score and Reinforcement Learning Rectified Speech Emotion Descriptors