Beyond the Mean: Modelling Annotation Distributions in Continuous Affect Prediction

Kosmas Pinitas; Ilias Maglogiannis

cs.LGApr 8, 2026

Beyond the Mean: Modelling Annotation Distributions in Continuous Affect Prediction

Authors: Kosmas Pinitas, Ilias Maglogiannis

Organizations: Department of Digital Systems, University of Piraeus

Abstract

Emotion annotation is inherently subjective and cognitively demanding, producing signals that reflect diverse perceptions across annotators rather than a single ground truth. In continuous affect prediction, this variability is typically collapsed into point estimates such as the mean or median, discarding valuable information about annotator disagreement and uncertainty. In this work, we propose a distribution-aware framework that models annotation consensus using the Beta distribution. Instead of predicting a single affect value, models estimate the mean and standard deviation of the annotation distribution, which are transformed into valid Beta parameters through moment matching. This formulation enables the recovery of higher-order distributional descriptors, including skewness, kurtosis, and quantiles, in closed form. As a result, the model captures not only the central tendency of emotional perception but also variability, asymmetry, and uncertainty in annotator responses. We evaluate the proposed approach on the SEWA and RECOLA datasets using multimodal features. Experimental results show that Beta-based modelling produces predictive distributions that closely match the empirical annotator distributions while achieving competitive performance with conventional regression approaches. These findings highlight the importance of modelling annotation uncertainty in affective computing and demonstrate the potential of distribution-aware learning for subjective signal analysis.

Figures & tables

Figure 1: Framework Overview: (a) annotator signals form a probabilistic consensus $(\mu,\sigma)$ ; (b) multimodal features are mapped to these parameters during training; (c) at test time, predicted $(\mu,\sigma)$ define a Beta distribution capturing annotation ambiguity.

RECOLA		Arousal		Valence
Modality	Model	$\mu$	$\sigma$	$\mu$	$\sigma$
Audio	$M_{I}$	0.19	0.04	0.54	0.25
	$M_{S}$	0.17	0.04	0.51	0.27
	$M_{F}$	0.13	0.02	0.52	0.22
Visual	$M_{I}$	0.05	0.03	0.03	0.03
	$M_{S}$	0.03	0.01	0.02	0.03

Table 1: CCC performance on RECOLA and SEWA for arousal and valence regression.

M_{I}

: independent

\mu,\sigma

networks;

M_{S}

: shared first layer with split second layers;

M_{F}

: fully shared network with two outputs. Best scores in bold; non-significant differences also marked.

RECOLA		Arousal		Valence
Modality	Model	$\mathcal{U}$	$\mathcal{B}$	$\mathcal{U}$	$\mathcal{B}$
Audio	$M_{I}$	13.59	0.64	5.97	0.53
	$M_{S}$	13.18	0.62	5.90	0.55
	$M_{F}$	13.77	0.64	5.87	0.57
Fusion	$M_{I}$	16.87	0.78	6.18	0.61
	$M_{S}$	15.73	0.75	5.89	0.60

Table 2: KL divergence of predicted distributions against ground-truth annotation Betas on RECOLA and SEWA.

\mathcal{U}

: Uniform baseline;

\mathcal{B}

: true Beta distribution. Lower values indicate closer match to annotator distributions.

RECOLA		Arousal		Valence
Modality	Descriptor	$B$	$M$	$B$	$M$
Audio	median	0.16	0.18	0.50	0.52
	q75	0.10	0.13	0.45	0.44
	q25	0.12	0.18	0.50	0.53
	skew	-0.02	0.01	-0.01	0.03
	kurt	0.05	-0.01	0.04	0.02

Table 3: CCC performance for higher-order descriptors on RECOLA and SEWA.

B

: point-regressor baselines trained separately for each descriptor.

M

: proposed Beta-based approach deriving descriptors from

\mu,\sigma

Panel of Figure 2 — Figure 2: Indicative true and predicted probability density functions for Beta-Based Regression ( $M_{B}$ ).

Affective Computing

Affective computing develops systems that recognize, model, and respond to human emotions and affective states from behavior and physiological signals.

See all 141 papers

Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition
Feb 24, 2026Ming Li, Yong-Jin Liu, Fang Liu +6Emotion Recognition Multimodal Emotion Recognition
MOE-Key Laboratory of Pervasive Computing, Department of Computer Science and Technology, Tsinghua University, Beijing, China. · the Key State Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China · the School of Computer Science and Technology, and Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China +1
Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework
Jun 19, 2026Emily Öhman, Anna KoufakouSoft-Label Learning Annotator Disagreement
Waseda University, Tokyo, Japan · Florida Gulf Coast University, Fort Myers FL, USA
Bias-Corrected Ceilings of Emotion Predictability from Human Label Variation Based on Instance-Level Fano Bounds
Aug 16, 2026Keito InoshitaEmotion Recognition
Faculty of Business and Commerce, Kansai University

Feb 24, 2026J/K move · Enter open · S save

Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition

Ming Li, Yong-Jin Liu, Fang Liu, Huankun Sheng+5

MOE-Key Laboratory of Pervasive Computing, Department of Computer Science and Technology, Tsinghua University, Beijing, China. · the Key State Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China · the School of Computer Science and Technology, and Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China+1

Emotion Recognition Multimodal Emotion Recognition Label Distribution Learning

Open paper AskPDF arXiv

Emotion recognition from multi-modal physiological and behavioral signals plays a pivotal role in affective computing, yet most existing models remain constrained to the prediction of singular emotions in controlled laboratory settings. Real-world human emotional experiences, by contrast, are often characterized by the simultaneous presence of multiple affective states, spurring recent interest in mixed emotion recognition as an emotion distribution learning problem. Current approaches, however, often neglect the valence consistency and structured correlations inherent among coexisting emotions. To address this limitation, we propose a Memory-guided Prototypical Co-occurrence Learning (MPCL) framework that explicitly models emotion co-occurrence patterns. Specifically, we first fuse multi-modal signals via a multi-scale associative memory mechanism. To capture cross-modal semantic relationships, we construct emotion-specific prototype memory banks, yielding rich physiological and behavioral representations, and employ prototype relation distillation to ensure cross-modal alignment in the latent prototype space. Furthermore, inspired by human cognitive memory systems, we introduce a memory retrieval strategy to extract semantic-level co-occurrence associations across emotion categories. Through this bottom-up hierarchical abstraction process, our model learns affectively informative representations for accurate emotion distribution prediction. Comprehensive experiments on two public datasets demonstrate that MPCL consistently outperforms state-of-the-art methods in mixed emotion recognition, both quantitatively and qualitatively.

Abstract

Affective Computing

Explore similar work

Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition

Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

Bias-Corrected Ceilings of Emotion Predictability from Human Label Variation Based on Instance-Level Fano Bounds