Semantic-Emotional Resonance Embedding: A Semi-Supervised Paradigm for Cross-Lingual Speech Emotion Recognition

Ya Zhao; Yinfeng Yu; Liejun Wang

cs.SDApr 8, 2026

Semantic-Emotional Resonance Embedding: A Semi-Supervised Paradigm for Cross-Lingual Speech Emotion Recognition

Authors: Ya Zhao, Yinfeng Yu, Liejun Wang

Organizations: School of Computer Science and Technology, Xinjiang University, Urumqi, China · Pengcheng Laboratory Xinjiang Network Node · Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center · Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University

Abstract

Cross-lingual Speech Emotion Recognition (CLSER) aims to identify emotional states in unseen languages. However, existing methods heavily rely on the semantic synchrony of complete labels and static feature stability, hindering low-resource languages from reaching high-resource performance. To address this, we propose a semi-supervised framework based on Semantic-Emotional Resonance Embedding (SERE), a cross-lingual dynamic feature paradigm that requires neither target language labels nor translation alignment. Specifically, SERE constructs an emotion-semantic structure using a small number of labeled samples. It learns human emotional experiences through an Instantaneous Resonance Field (IRF), enabling unlabeled samples to self-organize into this structure. This achieves semi-supervised semantic guidance and structural discovery. Additionally, we design a Triple-Resonance Interaction Chain (TRIC) loss to enable the model to reinforce the interaction and embedding capabilities between labeled and unlabeled samples during emotional highlights. Extensive experiments across multiple languages demonstrate the effectiveness of our method, requiring only 5-shot labeling in the source language.

Figures & tables

Fig. 1: Traditional cross-lingual SER methods have significant drawbacks. Method A requires the presence of relevant emotional words for recognition, while B struggles with recognition due to the lack of explicit semantic content. Additionally, C relies on the assumption that the entire speech segment maintains a static emotional state. In contrast, our method D learns human emotional experiences to capture instantaneous dynamic emotional processes.

Fig. 2: Overview of the proposed SERE semi-supervised dual-path architecture for CLSER tasks.

Method (Source $\rightarrow$ Target)	B $\rightarrow$ C	C $\rightarrow$ B	B $\rightarrow$ E	E $\rightarrow$ B	C $\rightarrow$ E	E $\rightarrow$ C	B $\rightarrow$ O	O $\rightarrow$ B	C $\rightarrow$ O	O $\rightarrow$ C	E $\rightarrow$ O	O $\rightarrow$ E	Avg.
JDAR [ 10 ]	42.70	48.97	37.95	47.80	37.58	35.60	-	-	-	-	-	-	-
JIASL [ 11 ]	38.10	53.64	38.05	48.35	37.76	36.00	-	-	-	-	-	-	-
CIAN [ 12 ]	39.60	59.37	36.34	47.40	34.75	31.90	-	-	-	-	-	-	-
LRJDA-IS10 [ 13 ]	43.10	50.09	37.57	48.93	34.88	38.40	-	-	-	-	-	-	-
ADoGT [ 14 ]	34.91	55.01	-	-	-	-	60.85	71.43	45.99	45.52	-	-	-
BYOL [ 15 ]	44.44	61.03	-	-	-	-	57.66	78.96	45.50	43.94	-	-	-

TABLE I: Compared with state-of-the-art CLSER experiments, where the best results are highlighted in bold.(%)

Method (Source $\rightarrow$ Target)	B $\rightarrow$ C	C $\rightarrow$ B	B $\rightarrow$ E	E $\rightarrow$ B	C $\rightarrow$ E	E $\rightarrow$ C	B $\rightarrow$ O	O $\rightarrow$ B	C $\rightarrow$ O	O $\rightarrow$ C	E $\rightarrow$ O	O $\rightarrow$ E	Avg.
SERE w/o $\mathcal{L}_{\text{proto}}$ & $\mathcal{L}_{\text{dual}}$	41.77	65.05	36.00	51.45	32.14	32.40	43.12	51.35	39.54	41.60	35.66	30.03	41.68
SERE w/o $\mathcal{L}_{\text{dual}}$	42.36	65.22	35.78	49.73	35.60	35.00	46.98	55.65	43.37	43.09	35.74	31.52	43.34
SERE w/o $\mathcal{L}_{\text{proto}}$	44.90	67.50	37.67	51.12	38.17	37.10	46.52	55.03	45.29	46.50	36.80	32.47	44.92
SERE	48.68	69.28	40.97	54.47	40.52	40.05	49.86	58.43	48.55	51.98	37.58	32.65	47.75

TABLE II: Ablation Study of the Proposed SERE Method. ‘w/o’denotes removing the component.(%)

Panel of Figure 5 — Fig. 3: Feature distribution of different SERE components under task C $\rightarrow$ B.

Method	Encoders(Source $\rightarrow$ Target)	Encoding Type	UA(%)
	C-Hubert $\rightarrow$ C-Hubert	Isomorphic	46.21
B $\rightarrow$ C	G-Whisper $\rightarrow$ G-Whisper	Isomorphic	46.05
	G-Whisper $\rightarrow$ C-Hubert	Heterogeneous	48.68
	G-Whisper $\rightarrow$ G-Whisper	Isomorphic	66.28
C $\rightarrow$ B	C-Hubert $\rightarrow$ C-Hubert	Isomorphic	66.95
	C-Hubert $\rightarrow$ G-Whisper	Heterogeneous	69.28

TABLE III: Heterogeneous vs. Isomorphic Encoding: Performance on Unlabeled Paths.

Explore similar work

Learning Emotion-discriminative Representations for Zero-Shot Cross-lingual Speech Emotion Recognition
Jun 4, 2026Jinyi Mi, Ding Ma, Tomoki TodaZero-Shot Learning Cross-Lingual Representation Alignment
Graduate School of Informatics, Nagoya University, Japan · Information Technology Center, Nagoya University, Japan
How Language-Independent Are Emotional Attributes? A Study on Training Data Scaling and Cross-Lingual Generalization
Oct 10, 2026Dániel Halmai, Gábor GosztolyaSpeech Emotion Recognition
Institute of Informatics, University of Szeged, Szeged, Hungary · HUN-REN–SZTE Research Group on Artificial Intelligence, Szeged, Hungary
Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition
Mar 24, 2026Lucas H. Ueda, João G. T. Lima, Paula D. P. CostaContrastive Learning Multimodal Emotion Recognition
Dept. of Computer Engineering and Automation (DCA), Faculdade de Engenharia El´etrica e de Computac¸˜ao, and are part of the AI Lab., Recod.ai, Institute of Computing, Universidade Estadual de Campinas, UNICAMP

Jun 4, 2026J/K move · Enter open · S save

Learning Emotion-discriminative Representations for Zero-Shot Cross-lingual Speech Emotion Recognition

Jinyi Mi, Ding Ma, Tomoki Toda

Graduate School of Informatics, Nagoya University, Japan · Information Technology Center, Nagoya University, Japan

Zero-Shot Learning Cross-Lingual Representation Alignment Speech Emotion Recognition Invariant Representation Learning

Open paper AskPDF arXiv

Zero-shot cross-lingual speech emotion recognition (SER) remains challenging due to distribution mismatches across languages and the lack of emotion annotations in target language. Under such conditions, models trained solely on source-language data frequently suffer from degraded generalization when evaluated on unseen target languages. To address this limitation, we propose an emotion-discriminative representation learning method that integrates supervised contrastive learning and speaker adversarial learning. The contrastive learning promotes cross-lingual emotion alignment, while speaker adversarial learning suppresses speaker-related cues to encourage speaker-invariant representations. Experimental results under a zero-shot cross-lingual SER setting demonstrate that the proposed method significantly improves SER performance over conventional training strategies.

Abstract

Explore similar work

Learning Emotion-discriminative Representations for Zero-Shot Cross-lingual Speech Emotion Recognition

How Language-Independent Are Emotional Attributes? A Study on Training Data Scaling and Cross-Lingual Generalization

Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition