Semantic-Emotional Resonance Embedding: A Semi-Supervised Paradigm for Cross-Lingual Speech Emotion Recognition
Authors: Ya Zhao, Yinfeng Yu, Liejun Wang
Organizations: School of Computer Science and Technology, Xinjiang University, Urumqi, China · Pengcheng Laboratory Xinjiang Network Node · Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center · Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University
Cross-lingual Speech Emotion Recognition (CLSER) aims to identify emotional states in unseen languages. However, existing methods heavily rely on the semantic synchrony of complete labels and static feature stability, hindering low-resource languages from reaching high-resource performance. To address this, we propose a semi-supervised framework based on Semantic-Emotional Resonance Embedding (SERE), a cross-lingual dynamic feature paradigm that requires neither target language labels nor translation alignment. Specifically, SERE constructs an emotion-semantic structure using a small number of labeled samples. It learns human emotional experiences through an Instantaneous Resonance Field (IRF), enabling unlabeled samples to self-organize into this structure. This achieves semi-supervised semantic guidance and structural discovery. Additionally, we design a Triple-Resonance Interaction Chain (TRIC) loss to enable the model to reinforce the interaction and embedding capabilities between labeled and unlabeled samples during emotional highlights. Extensive experiments across multiple languages demonstrate the effectiveness of our method, requiring only 5-shot labeling in the source language.
Figures & tables
Fig. 1: Traditional cross-lingual SER methods have significant drawbacks. Method A requires the presence of relevant emotional words for recognition, while B struggles with recognition due to the lack of explicit semantic content. Additionally, C relies on the assumption that the entire speech segment maintains a static emotional state. In contrast, our method D learns human emotional experiences to capture instantaneous dynamic emotional processes.
Fig. 2: Overview of the proposed SERE semi-supervised dual-path architecture for CLSER tasks.
Method (Source → Target)
B → C
C → B
B → E
E → B
C → E
E → C
B → O
O → B
C → O
O → C
E → O
O → E
Avg.
JDAR [ 10 ]
42.70
48.97
37.95
47.80
37.58
35.60
-
-
-
-
-
-
-
JIASL [ 11 ]
38.10
53.64
38.05
48.35
37.76
36.00
-
-
-
-
-
-
-
CIAN [ 12 ]
39.60
59.37
36.34
47.40
34.75
31.90
-
-
-
-
-
-
-
LRJDA-IS10 [ 13 ]
43.10
50.09
37.57
48.93
34.88
38.40
-
-
-
-
-
-
-
ADoGT [ 14 ]
34.91
55.01
-
-
-
-
60.85
71.43
45.99
45.52
-
-
-
BYOL [ 15 ]
44.44
61.03
-
-
-
-
57.66
78.96
45.50
43.94
-
-
-
TABLE I: Compared with state-of-the-art CLSER experiments, where the best results are highlighted in bold.(%)
Method (Source → Target)
B → C
C → B
B → E
E → B
C → E
E → C
B → O
O → B
C → O
O → C
E → O
O → E
Avg.
SERE w/o Lproto & Ldual
41.77
65.05
36.00
51.45
32.14
32.40
43.12
51.35
39.54
41.60
35.66
30.03
41.68
SERE w/o Ldual
42.36
65.22
35.78
49.73
35.60
35.00
46.98
55.65
43.37
43.09
35.74
31.52
43.34
SERE w/o Lproto
44.90
67.50
37.67
51.12
38.17
37.10
46.52
55.03
45.29
46.50
36.80
32.47
44.92
SERE
48.68
69.28
40.97
54.47
40.52
40.05
49.86
58.43
48.55
51.98
37.58
32.65
47.75
TABLE II: Ablation Study of the Proposed SERE Method. ‘w/o’denotes removing the component.(%)
Fig. 3: Feature distribution of different SERE components under task C → B.
Method
Encoders(Source → Target)
Encoding Type
UA(%)
C-Hubert → C-Hubert
Isomorphic
46.21
B → C
G-Whisper → G-Whisper
Isomorphic
46.05
G-Whisper → C-Hubert
Heterogeneous
48.68
G-Whisper → G-Whisper
Isomorphic
66.28
C → B
C-Hubert → C-Hubert
Isomorphic
66.95
C-Hubert → G-Whisper
Heterogeneous
69.28
TABLE III: Heterogeneous vs. Isomorphic Encoding: Performance on Unlabeled Paths.
Zero-shot cross-lingual speech emotion recognition (SER) remains challenging due to distribution mismatches across languages and the lack of emotion annotations in target language. Under such conditions, models trained solely on source-language data frequently suffer from degraded generalization when evaluated on unseen target languages. To address this limitation, we propose an emotion-discriminative representation learning method that integrates supervised contrastive learning and speaker adversarial learning. The contrastive learning promotes cross-lingual emotion alignment, while speaker adversarial learning suppresses speaker-related cues to encourage speaker-invariant representations. Experimental results under a zero-shot cross-lingual SER setting demonstrate that the proposed method significantly improves SER performance over conventional training strategies.
Jinyi Mi, Ding Ma, Tomoki Toda
Graduate School of Informatics, Nagoya University, Japan · Information Technology Center, Nagoya University, Japan
Current state-of-the-art Speech Emotion Recognition (SER) systems rely on lots of training data, which is available only for a few languages such as English. For low-resource languages, using a cross-lingual model or adapting it to the target language might be an alternative to training a mono-lingual model from scratch. However, it is not clear what constitutes a "low-resource", i.e. where model adaptation loses its edge over the mono-lingual approach. We investigate this topic by training and adapting several SER models on English and Mandarin data, focusing on the emotional attributes of arousal, valence and dominance. Our results show that, for arousal, model adaptation is beneficial even with two hours of data, while for valence 10-20 hours of adaptation material is required to match the performance of a mono-lingual model trained with 100 hours of data. This indicates that task-specific model adaptation may be an effective approach in low-resource speech emotion recognition problems.
Dániel Halmai, Gábor Gosztolya
Institute of Informatics, University of Szeged, Szeged, Hungary · HUN-REN–SZTE Research Group on Artificial Intelligence, Szeged, Hungary
Speech Emotion Recognition (SER) in real-world scenarios remains challenging due to severe class imbalance and the prevalence of spontaneous, natural speech. While recent approaches leverage self-supervised learning (SSL) representations and multimodal fusion of speech and text, most existing methods apply supervision only at the final classification layer, limiting the discriminative power of intermediate representations. In this work, we propose Crab (Contrastive Representation and Multimodal Aligned Bottleneck), a bimodal Cross-Modal Transformer architecture that integrates speech representations from WavLM and textual representations from RoBERTa, together with a novel Multi Layer Contrastive Supervision (MLCS) strategy. MLCS injects multi-positive contrastive learning signals at multiple layers of the network, encouraging emotionally discriminative representations throughout the model without introducing additional parameters at inference time. To further address data imbalance, we adopt weighted cross-entropy during training. We evaluate the proposed approach on three benchmark datasets covering different degrees of emotional naturalness: IEMOCAP, MELD, and MSP-Podcast 2.0. Experimental results demonstrate that Crab consistently outperforms strong unimodal and multimodal baselines across all datasets, with particularly large gains under naturalistic and highly imbalanced conditions. These findings highlight the effectiveness of Multi Layer Contrastive Supervision as a general and robust strategy for SER. Official implementation can be found in https://github.com/AI-Unicamp/Crab.
Lucas H. Ueda, João G. T. Lima, Paula D. P. Costa
Dept. of Computer Engineering and Automation (DCA), Faculdade de Engenharia El´etrica e de Computac¸˜ao, and are part of the AI Lab., Recod.ai, Institute of Computing, Universidade Estadual de Campinas, UNICAMP