Label Authority in Speech Emotion Recognition: Intended and Perceived Ground Truth in Emozionalmente
Authors: Marco Giordano, Stefano Giacomelli, Claudia Rinaldi, Fabio Graziosi
Organizations: Department of Information Engineering, Computer Science and Mathematics (DISIM), University of L’Aquila, L’Aquila, 67100, ITA · National Inter-University Consortium for Telecommunications (CNIT), University of L’Aquila, L’Aquila, 67100, ITA
Ground truth in Speech Emotion Recognition (SER) is often presented as a single categorical fact, yet an acted emotion and the emotion heard by listeners describe different aspects of the same recording. We examine this distinction in Emozionalmente, an acted Italian corpus that provides both an elicited production label and five listener judgements for every clip. Using frozen XLS-R, HuBERT, and WavLM representations with matched probes, we compare three label authorities: the expressed condition, the listeners' hard plurality, and their complete vote distribution. The results tell a consistent story. Recovering the expressed condition is easier than predicting a tie-broken listener plurality, but that gap compares two different questions rather than revealing a general loss of SER ability. As listener agreement weakens, the plurality becomes an increasingly brittle hard target. Training on the full vote distribution does not substantially change which class wins the model's hard decision, but it produces probability assignments that more closely follow the alternatives selected by listeners, especially for ambiguous clips. No authority therefore replaces the others. Expressed labels describe how speech was produced, pluralities provide an operational single-class decision, and vote distributions preserve perceptual uncertainty. For human-facing Internet of Sounds applications, evaluations should state which authority is being predicted, retain its aggregation rule and agreement information, and use distributional scores when disagreement is part of the target.
Figures & tables
Authority
Construction
Reference unit
Question answered
Expressed condition
One-hot elicitation label
One production instruction per clip
Was the requested acted condition recoverable from speech?
Recognized plurality
Most frequent of five votes; deterministic tie-break
One aggregated listener-derived class per clip
Which single class wins under the stated aggregation rule?
Recognized distribution
Seven-class vector of vote proportions
Complete five-vote histogram per clip
How should probability be allocated across the alternatives selected by listeners?
Evaluator-level view
Each recognition vote retained separately
One listener judgement, paired with the clip’s expressed label
How much individual perception is hidden by clip-level aggregation?
TABLE I: Operational label authorities. “Reference unit” identifies what one evaluation item represents; it is not a claim that one authority is universally more truthful than another.
Validation checkpoint; reported layer/winner by mean test UAR
TABLE II: Experimental factors held in common across label authorities.
Fig. 1: Expressed condition versus the plurality of all five listener votes. Rows are expressed labels and columns are recognized labels. The left panel shows counts and row-normalized rates, the centre suppresses the diagonal, and the right ranks expressed categories by their off-diagonal rate.
Training target
Best configuration
Layer
UAR
Expressed
XLS-R Dense
13
0.825±0.009
Recognized plurality
XLS-R Dense
15
0.746±0.006
Recognized distribution
WavLM Dense
10
0.749±0.004
TABLE IV: Target-specific best own-reference hard UAR. Expressed models use the expressed reference; recognized models use plurality, with the soft target read by argmax. Values are mean ± sample SD over five runs.
Training target
Hard UAR
KL
Brier
Expressed
0.739±0.002
1.667±0.156
0.272±0.009
Rec. plurality
0.731±0.005
1.223±0.145
0.236±0.008
Rec. distribution
0.736±0.005
0.510±0.002
0.162±0.001
TABLE V: Common-reference evaluation, mean ± sample SD over five runs after equally averaging nine target-selected backbone/probe configurations. Hard UAR uses recognized plurality; KL and Brier use the five-vote histogram.
Fig. 2: Authority and agreement diagnostics. (a) Hard-UAR cross-evaluation: rows are training authorities and columns are hard evaluation authorities. Cells average nine target-selected configurations within each run and report mean ± sample SD over five runs. (b) Hard accuracy by mutually exclusive listener-agreement stratum. Expressed probes use the expressed reference; recognized probes use plurality.
Training target
Mean ± SD
Config. range
Expressed
0.660±0.013
[0.600,0.700]
Recognized plurality
0.707±0.017
[0.680,0.740]
Recognized distribution
0.733±0.014
[0.680,0.780]
TABLE VI: Top-10 model–human confusion-pair overlap. The mean averages nine selected backbone/probe configurations within each run; the SD is over five runs, and the range is across configuration means.
Fig. 3: Alignment with the observed five-vote distribution: (a) KL divergence and (b) multiclass Brier score. Bars equally average nine target-selected configurations within each run; error bars are sample SDs over five runs. Lower is better.
Fig. 4: Distributional error against the observed vote histogram by human agreement stratum. Panels show cross-entropy, KL divergence, Jensen–Shannon divergence, and Brier score. Each point averages nine selected configurations within each run; error bars are sample SDs over five runs. KL and Brier provide the primary proper-score evidence; the other panels are supporting diagnostics.
Subset
Training
Hard UAR
KL
Brier
Tie ( n=145 )
Hard
0.469±0.015
2.121±0.256
0.360±0.014
Soft
0.435±0.012
0.671±0.004
0.184±0.001
<0.60 ( n=82 )
Hard
0.443±0.010
2.415±0.343
0.396±0.019
Soft
0.439±0.006
0.696±0.008
0.184±0.004
TABLE VII: Shared-reference results on ambiguous test subsets. Values are mean ± sample SD over five runs after equally averaging nine selected configurations. Lower KL and Brier are better.
Fig. 5: Learned agg_dense mixture weights for expressed, recognized-plurality, and recognized-distribution supervision. Lines are five-run means and bands are sample SDs. Layer 0 is the convolutional state; layers 1–24 are Transformer states.
Target
Backbone
Best single layer
Single-layer UAR
Aggregate UAR
Paired Δ
Expressed
XLS-R
Dense L13
0.825±0.010
0.801±0.006
−0.023±0.013
HuBERT
Dense L13
0.808±0.007
0.796±0.008
−0.012±0.014
WavLM
Dense L18
0.807±0.004
0.822±0.010
+0.015±0.010
Recognized plurality
XLS-R
Dense L15
0.746±0.006
0.719±0.016
−0.026±0.012
HuBERT
Dense L13
0.729±0.007
0.721±0.011
−0.008±0.009
WavLM
Dense L10
0.739±0.007
0.739±0.009
≈0±0.013
TABLE VIII: Best single-layer dense probe versus agg_dense . Values are hard UAR mean ± sample SD over five aligned runs; Δ is aggregation minus single layer. Soft-target models are evaluated against hard plurality.
Comparison
Paired-run Δ
Conditional 95% CI
Expressed − hard recognized, own-reference UAR
+0.079±0.014
[0.058,0.099]
Soft − hard recognized, recognized UAR
+0.003±0.008
[−0.013,0.019]
Soft − hard recognized, KL
−0.360±0.081
[−0.399,−0.325]
Soft − hard recognized, Brier
−0.056±0.015
[−0.066,−0.046]
TABLE IX: Best-individual paired contrasts on all 1,380 test clips. The first column gives the mean ± sample SD of five aligned run differences; the interval comes from 1,000 paired clip-bootstrap replicates and excludes training and model-selection uncertainty. Each contrast is left minus right.
Speech emotion recognition (SER) often relies on hard consensus labels that collapse annotator disagreement. We study distribution-based supervision for 9-class SER on MSP-Podcast 2.0 using a WavLM-Base multitask model for categorical emotion and dimensional VAD. Hard-label training is compared with targets from primary and merged primary--secondary annotator vote distributions. Distributional objectives improve alignment with human vote distributions, reducing JSD/KLD relative to hard-label training. Analysis shows that hard supervision partly benefits from assigning ambiguous utterances to the residual Other class, whereas distributional supervision redistributes uncertainty across emotion categories. Entropy-stratified evaluation shows that high-ambiguity utterances remain challenging, but distribution-based supervision better captures perceptual uncertainty. These findings support moving beyond hard labels toward targets that reflect listener disagreement.
Zahra Omidi, John H. L. Hansen
Center for Robust Speech Systems, The University of Texas at Dallas, USA
Speech emotion recognition (SER) is the task of assigning emotion labels to utterances. Early systems relied on acoustic features, whereas recent approaches combine multiple modalities, most commonly speech and text. Still, performance remains poor on many datasets. Large language models (LLMs) have therefore attracted interest for SER, as they can process diverse inputs jointly with instructions. However, direct audio input raises questions of explainability. To address similar questions in image classification, concept bottleneck models were introduced. This work adapts concept bottlenecks to SER to examine how individual predictions depend on transcripts, acoustic descriptions and speaker attributes. Experiments test three LLMs on CREMA-D, IEMOCAP and MELD, with concepts extracted by separate tools. On scripted corpora, LLMs are strongly biased towards the transcript in the zero-shot setting, which lowers Macro-F1 from 27.8 to 5.8 on CREMA-D. Fine-tuning removes this bias, and the transcript raises Macro-F1 from 41.8 to 45.1. Removing speech rate changes 48% of Neutral predictions to Disgust on CREMA-D; removing intensity level on MELD changes predictions despite little change in Macro-F1. These findings show that aggregate performance changes alone do not capture the effects of concept removal on individual predictions.
Hezhao Zhang, Thomas Hain
School of Computer Science, University of Sheffield Sheffield, United Kingdom
Explainable and trustworthy speech emotion recognition (SER) remains a challenging task to date, largely due to the scarcity of SER data with reliable speech emotion descriptor (SED) labels, such as prosodic features and speaker traits. This paper presents a confidence score and reinforcement learning (RL) based on-the-fly SED rectification approach for post-training SER systems on automatically annotated SED labels. Experiments on IEMOCAP and MELD suggest that explainable SER systems incorporating the proposed confidence score and RL-based SED rectification approach consistently outperform baselines without data selection or SED rectification. The best performing system, which integrates both components, surpasses the baseline without data selection and SED rectification, achieving SER gains of 2.9% and 3.3% absolute (3.7% and 5.4% relative) on IEMOCAP and MELD benchmarks, respectively.
Youjun Chen, Xurong Xie, Mengzhe Geng +9
The Chinese University of Hong Kong, Hong Kong SAR, China · Institute of Software, Chinese Academy of Sciences, China · National Research Council Canada, Canada +1