How Language-Independent Are Emotional Attributes? A Study on Training Data Scaling and Cross-Lingual Generalization
Authors: Dániel Halmai, Gábor Gosztolya
Organizations: Institute of Informatics, University of Szeged, Szeged, Hungary · HUN-REN–SZTE Research Group on Artificial Intelligence, Szeged, Hungary
Current state-of-the-art Speech Emotion Recognition (SER) systems rely on lots of training data, which is available only for a few languages such as English. For low-resource languages, using a cross-lingual model or adapting it to the target language might be an alternative to training a mono-lingual model from scratch. However, it is not clear what constitutes a "low-resource", i.e. where model adaptation loses its edge over the mono-lingual approach. We investigate this topic by training and adapting several SER models on English and Mandarin data, focusing on the emotional attributes of arousal, valence and dominance. Our results show that, for arousal, model adaptation is beneficial even with two hours of data, while for valence 10-20 hours of adaptation material is required to match the performance of a mono-lingual model trained with 100 hours of data. This indicates that task-specific model adaptation may be an effective approach in low-resource speech emotion recognition problems.
Figures & tables
No. of
Duration
Subset
chunks
Mean
Total
MSP train
65 205
5.75 s
104.19 h
BIIC train
1h
487
7.57 s
1.02 h
2h
974
7.56 s
2.05 h
5h
2 436
7.54 s
5.10 h
10h
4 873
7.55 s
10.21 h
Table 1: Main properties of the MSP Podcast and BIIC Podcast corpora used in our experiments
Figure 1: Pearson’s correlation and CCC results on the development set.
Figure 2: Pearson’s correlation and CCC results on the test set.
Attribute
Approach
Hrs.
Pearson
CCC
Arousal
Direct
100h
0.606
0.554
Transfer
10h
0.624
0.548
Significance level
p< 0.01
p= 0.547
Valence
Direct
100h
0.365
0.271
Transfer
20h
0.362
0.274
Significance level
p= 0.547
p= 0.691
Table 2: Test set performance of the models trained with the in-language data on 100 hours (‘Direct-100h’) and the best adapted model (‘Transfer’, chosen based on development set performance)
Zero-shot cross-lingual speech emotion recognition (SER) remains challenging due to distribution mismatches across languages and the lack of emotion annotations in target language. Under such conditions, models trained solely on source-language data frequently suffer from degraded generalization when evaluated on unseen target languages. To address this limitation, we propose an emotion-discriminative representation learning method that integrates supervised contrastive learning and speaker adversarial learning. The contrastive learning promotes cross-lingual emotion alignment, while speaker adversarial learning suppresses speaker-related cues to encourage speaker-invariant representations. Experimental results under a zero-shot cross-lingual SER setting demonstrate that the proposed method significantly improves SER performance over conventional training strategies.
Jinyi Mi, Ding Ma, Tomoki Toda
Graduate School of Informatics, Nagoya University, Japan · Information Technology Center, Nagoya University, Japan
Speech emotion recognition (SER) systems can exhibit gender-related performance disparities, but how such bias manifests in multilingual speech LLMs across languages and modalities is unclear. We introduce a novel multilingual, multimodal benchmark built on MELD-ST, spanning English, Japanese, and German, to quantify language-specific SER performance and gender gaps. We find bias is strongly language-dependent, and multimodal fusion does not reliably improve fairness. To address these, we propose ERM-MinMaxGAP, a fairness-informed training objective, which augments empirical risk minimization (ERM) with a proposed adaptive fairness weight mechanism and a novel MinMaxGAP regularizer on the maximum male-female loss gap within each language and modality. Building upon the Qwen2-Audio backbone, our ERM-MinMaxGAP approach improves multilingual SER performance by 5.5% and 5.0% while reducing the overall gender bias gap by 0.1% and 1.4% in the unimodal and multimodal settings, respectively.
Zi Haur Pang, Xiaoxue Gao, Tatsuya Kawahara +1
Kyoto University, Japan · Agency for Science, Technology, and Research (A*STAR), Singapore
Prosodic emphasis varies across languages, emotions, and speaking styles, yet existing emphasis detection models are largely trained and evaluated on monolingual neutral read speech. We introduce MMEE (Multilingual Multi-Emotion Emphasis), a corpus of 10,000 professionally recorded expressive utterances (14.13 hours) across 7 languages and 34 emotion/style categories, with three-level perceptual labels (10 annotations per sample). We benchmark two state-of-the-art architectures under monolingual, cross-lingual, multilingual, cross-emotion, cross-dataset, and data-scale settings. Monolingual models show limited zero-shot transfer, degrading across typologically distant languages, while multilingual training substantially improves robustness. Models transfer robustly between high- and low-arousal emotions; bidirectional transfer between synthetic and perceptual benchmarks suggests shared prosodic structure; and performance stays robust even at smaller training scales.