Color-based Emotion Representation for Speech Emotion Recognition

Ryotaro Nagase; Ryoichi Takashima; Yoichi Yamashita

eess.ASFeb 18, 2026

Color-based Emotion Representation for Speech Emotion Recognition

Authors: Ryotaro Nagase, Ryoichi Takashima, Yoichi Yamashita

Organizations: College of Information Science and Engineering, Ritsumeikan University, Osaka, Japan

Abstract

Speech emotion recognition (SER) has traditionally relied on categorical or dimensional labels. However, this technique is limited in representing both the diversity and interpretability of emotions. To overcome this limitation, we focus on color attributes, such as hue, saturation, and value, to represent emotions as continuous and interpretable scores. We annotated an emotional speech corpus with color attributes via crowdsourcing and analyzed them. Moreover, we built regression models for color attributes in SER using machine learning and deep learning, and explored the multitask learning of color attribute regression and emotion classification. As a result, we demonstrated the relationship between color attributes and emotions in speech, and successfully developed color attribute regression models for SER. We also showed that multitask learning improved the performance of each task.

Figures & tables

Fig. 1 : Interface for annotating color attributes

Fig. 2 : Histogram of hue label frequencies per emotion

Panel of Figure 4 — Fig. 5 : Outline of the models used in Experiments 1 and 2

Setting	Hue	Saturation		Value
Setting	AE $\downarrow$	PCC $\uparrow$	CCC $\uparrow$	PCC $\uparrow$	CCC $\uparrow$
SVR models
ComParE2016	$41.739\,434\,616\,631\,44$	$0.563\,770\,742\,009\,113\,6$	$0.325\,226\,184\,862\,187\,35$	$0.699\,327\,375\,393\,679\,8$	$0.480\,051\,765\,489\,791\,56$
HuBERT embeddings
$L=6$	$35.146\,035\,04$	$0.549\,290\,660\,1$	$0.370\,252\,199\,7$	$0.754\,729\,422\,2$	$0.564\,674\,050\,3$
$L=9$	$31.285\,800\,53$	$0.579\,737\,613\,9$	$0.402\,971\,658\,1$	$0.787\,536\,468\,1$	$0.658\,218\,787\,6$

TABLE I : Results of color attribute regression with SVR and DNN models

$\alpha$	Hue	Saturation		Value		Accuracy $\uparrow$ (6 emotion cls.)
$\alpha$	AE $\downarrow$	PCC $\uparrow$	CCC $\uparrow$	PCC $\uparrow$	CCC $\uparrow$	Accuracy $\uparrow$ (6 emotion cls.)
0.6	$32.665\,741\,399\,705\,95$	$0.444\,990\,118\,005\,082\,4$	$0.406\,161\,100\,483\,980\,13$	$0.775\,756\,685\,051\,954$	$0.753\,296\,690\,900\,693\,4$	$83.75$
0.7	$31.225\,794\,356\,903\,343$	$0.500\,194\,683\,398\,763\,8$	$0.463\,489\,268\,424\,972\,9$	$0.795\,607\,486\,787\,621\,8$	$0.783\,874\,314\,972\,817\,3$	$85.833\,333\,333\,333\,34$
0.8	$30.067\,799\,593\,466\,425$	$0.522\,367\,172\,590\,378\,5$	$0.503\,877\,505\,325\,686\,2$	$0.821\,426\,793\,915\,825\,6$	$0.810\,249\,197\,599\,962\,2$	$88.333\,333\,333\,333\,34$
0.9	$29.736\,212\,693\,404\,635$	$0.580\,024\,856\,910\,904\,6$	$0.560\,142\,514\,100\,588\,9$	$0.813\,705\,143\,269\,895\,2$	$0.802\,911\,931\,270\,722\,5$	$90.833\,333\,333\,333\,33$
1.0	-	-	-	-	-	$88.333\,333\,333\,333\,33$

TABLE II : Results of multitask learning of color attribute regression and categorical emotion classification

Explore similar work

Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition
Mar 24, 2026Lucas H. Ueda, João G. T. Lima, Paula D. P. CostaContrastive Learning Multimodal Emotion Recognition
Dept. of Computer Engineering and Automation (DCA), Faculdade de Engenharia El´etrica e de Computac¸˜ao, and are part of the AI Lab., Recod.ai, Institute of Computing, Universidade Estadual de Campinas, UNICAMP
Explainable and Trustworthy Speech Emotion Recognition Using Confidence Score and Reinforcement Learning Rectified Speech Emotion Descriptors
Jun 12, 2026Youjun Chen, Xurong Xie, Mengzhe Geng +9Explainable Artificial Intelligence Reinforcement Learning
The Chinese University of Hong Kong, Hong Kong SAR, China · Institute of Software, Chinese Academy of Sciences, China · National Research Council Canada, Canada +1
Enriching Speech Emotion Representations with Conversational Context
Sep 22, 2026Arthur Peuvot, Romaric Besançon, Gaël de Chalendar +2Representation Learning Emotion Recognition in Conversations
Université Paris-Saclay, CEA, List, France · LISN, CNRS, Université Paris-Saclay, France

Mar 24, 2026J/K move · Enter open · S save

Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition

Lucas H. Ueda, João G. T. Lima, Paula D. P. Costa

Dept. of Computer Engineering and Automation (DCA), Faculdade de Engenharia El´etrica e de Computac¸˜ao, and are part of the AI Lab., Recod.ai, Institute of Computing, Universidade Estadual de Campinas, UNICAMP

Contrastive Learning Multimodal Emotion Recognition Speech Emotion Recognition Multimodal Contrastive Learning Supervised Contrastive Learning

Open paper AskPDF arXiv

Speech Emotion Recognition (SER) in real-world scenarios remains challenging due to severe class imbalance and the prevalence of spontaneous, natural speech. While recent approaches leverage self-supervised learning (SSL) representations and multimodal fusion of speech and text, most existing methods apply supervision only at the final classification layer, limiting the discriminative power of intermediate representations. In this work, we propose Crab (Contrastive Representation and Multimodal Aligned Bottleneck), a bimodal Cross-Modal Transformer architecture that integrates speech representations from WavLM and textual representations from RoBERTa, together with a novel Multi Layer Contrastive Supervision (MLCS) strategy. MLCS injects multi-positive contrastive learning signals at multiple layers of the network, encouraging emotionally discriminative representations throughout the model without introducing additional parameters at inference time. To further address data imbalance, we adopt weighted cross-entropy during training. We evaluate the proposed approach on three benchmark datasets covering different degrees of emotional naturalness: IEMOCAP, MELD, and MSP-Podcast 2.0. Experimental results demonstrate that Crab consistently outperforms strong unimodal and multimodal baselines across all datasets, with particularly large gains under naturalistic and highly imbalanced conditions. These findings highlight the effectiveness of Multi Layer Contrastive Supervision as a general and robust strategy for SER. Official implementation can be found in https://github.com/AI-Unicamp/Crab.

Abstract

Explore similar work

Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition

Explainable and Trustworthy Speech Emotion Recognition Using Confidence Score and Reinforcement Learning Rectified Speech Emotion Descriptors

Enriching Speech Emotion Representations with Conversational Context