Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition

Ming Li; Yong-Jin Liu; Fang Liu; Huankun Sheng; Yeying Fan; Yixiang Wei; Minnan Luo; Weizhan Zhang

cs.LGFeb 24, 2026

Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition

Authors: Ming Li, Yong-Jin Liu, Fang Liu, Huankun Sheng, Yeying Fan, Yixiang Wei, Minnan Luo, Weizhan Zhang, +1 more

Organizations: MOE-Key Laboratory of Pervasive Computing, Department of Computer Science and Technology, Tsinghua University, Beijing, China. · the Key State Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China · the School of Computer Science and Technology, and Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · the Department of Computer Science and Computer Engineering, Texas A&M University, College Station, TX 77840 USA

Abstract

Emotion recognition from multi-modal physiological and behavioral signals plays a pivotal role in affective computing, yet most existing models remain constrained to the prediction of singular emotions in controlled laboratory settings. Real-world human emotional experiences, by contrast, are often characterized by the simultaneous presence of multiple affective states, spurring recent interest in mixed emotion recognition as an emotion distribution learning problem. Current approaches, however, often neglect the valence consistency and structured correlations inherent among coexisting emotions. To address this limitation, we propose a Memory-guided Prototypical Co-occurrence Learning (MPCL) framework that explicitly models emotion co-occurrence patterns. Specifically, we first fuse multi-modal signals via a multi-scale associative memory mechanism. To capture cross-modal semantic relationships, we construct emotion-specific prototype memory banks, yielding rich physiological and behavioral representations, and employ prototype relation distillation to ensure cross-modal alignment in the latent prototype space. Furthermore, inspired by human cognitive memory systems, we introduce a memory retrieval strategy to extract semantic-level co-occurrence associations across emotion categories. Through this bottom-up hierarchical abstraction process, our model learns affectively informative representations for accurate emotion distribution prediction. Comprehensive experiments on two public datasets demonstrate that MPCL consistently outperforms state-of-the-art methods in mixed emotion recognition, both quantitatively and qualitatively.

Figures & tables

Fig. 1: Overview of the MPCL Framework, which consists of three stages. (a) Multimodal Feature Extraction and Fusion: the Multi‑Scale Associative Fusion (MSAF) module fuses multi-modal physiological signals, while a separate behavioral encoder extracts structured behavioral feature. (b) Prototypical Alignment and Co‑occurrence Learning: emotion prototype memory banks are constructed from both modalities. The Prototype Relation Distillation (PRD) module enforces cross‑modal structural alignment, and the Prototypical Co‑occurrence Learning (PCL) module captures semantic‑level co‑occurrence patterns through memory retrieval. (c) Hierarchical Semantic Compression and Distribution Prediction: the Hierarchical Semantic Compression (HSC) module abstracts affective representations via a bottom‑up strategy, followed by a classifier that outputs the final emotion distribution.

Fig. 2: Fusion of multimodal physiological signals. EEG serves as the primary modality to integrate complementary information from auxiliary modalities (PPG and GSR) via a multi-scale associative memory mechanism. Three distinct scaling factors $\mathcal{B}=\{\beta_{\text{low}},\beta_{\text{mid}},\beta_{\text{high}}\}$ are employed to modulate the granularity of information aggregation.

Fig. 3: Prototype alignment process. Prototype memory banks are first constructed separately for the physiological and behavioral modalities, respectively. A semantics-enriched representation is then obtained as a weighted combination of prototypes from each bank. Meanwhile, the Prototype Relation Distillation (PRD) strategy enforces semantic structural consistency across the two modalities.

Fig. 4: Architecture of Prototypical Co‑occurrence Learning (PCL). The physiological embedding $x_{i}$ and the behavioral embedding $y_{i}$ retrieve associated representations $U_{x_{i}}$ and $U_{y_{i}}$ , respectively, via the Hopfield network. Within the physiological embedding space, $U_{x_{i}}$ (retrieved via the physiological query) serves as an anchor that is contrasted with the positive sample $U_{y_{i}}$ (retrieved from the corresponding behavioral query) and with negative samples $U_{y_{j}}$ (retrieved from mismatched behavioral queries, where $j\neq i$ ). The same procedure is applied symmetrically in the behavioral embedding space $V$ .

Dataset	Mesure	LDL			MER	Single-modal EDL			MMER	Multimodal EDL
Dataset	Mesure	PT-SVM [ 76 ]	AA-KNN [ 76 ]	SA-CPNN [ 77 ]	MAET [ 78 ]	LDL-LRR [ 79 ]	TLR-DL [ 80 ]	CAD [ 81 ]	CARAT [ 33 ]	EmotionDict [ 25 ]	HeLo [ 26 ]	MPCL
DMER	Chebyshev ( $\downarrow$ )	0.0851 (7)	0.0949 (10)	0.0917 (8)	0.0522 (3)	0.0936 (9)	0.0847 (6)	0.1017 (11)	0.0752 (4)	0.0753 (5)	0.0446 (2)	0.0346 (1)
	Clark ( $\downarrow$ )	0.5787 (6)	0.6649 (10)	0.6333 (8)	0.3762 (3)	0.6529 (9)	0.5380 (4)	0.7517 (11)	0.5951 (7)	0.5463 (5)	0.3256 (2)	0.2369 (1)
	Canberra ( $\downarrow$ )	1.5294 (7)	1.2357 (3)	1.8092 (9)	0.9875 (3)	1.8224 (10)	1.3271 (5)	1.9415 (11)	1.5965 (8)	1.4771 (6)	0.8664 (2)	0.6068 (1)
	KL ( $\downarrow$ )	0.0837 (6)	0.1409 (10)	0.1048 (8)	0.0288 (2)	0.0644 (4)	0.0925 (7)	0.1460 (11)	0.1137 (9)	0.0766 (5)	0.0323 (3)	0.0177 (1)
	Cosine ( $\uparrow$ )	0.9237 (8)	0.9315 (6)	0.9058 (9)	0.9688 (3)	0.9016 (10)	0.9339 (4)	0.8923 (11)	0.9332 (5)	0.9313 (7)	0.9714 (2)	0.9874 (1)

TABLE I: Quantitative comparison with baseline methods on the DMER and WESAD datasets under the subject-dependent setting.

Fig. 5: Comparison of emotion distribution predictions between MPCL and state-of-the-art baselines on two representative samples (Subject 26). GT denotes the ground-truth distribution. Emotion indices 1–10 correspond to: inspired, alert, excited, enthusiastic, determined, afraid, upset, nervous, scared, and distressed.

Dataset	Measure	LDL			MER	Single-modal EDL			MMER	Multimodal EDL
Dataset	Measure	PT-SVM [ 76 ]	AA-KNN [ 76 ]	SA-CPNN [ 77 ]	MAET [ 78 ]	LDL-LRR [ 79 ]	TLR-DL [ 80 ]	CAD [ 81 ]	CARAT [ 33 ]	EmotionDict [ 25 ]	HeLo [ 26 ]	MPCL
DMER	Chebyshev ( $\downarrow$ )	0.1462 (11)	0.1070 (10)	0.0928 (4)	0.0939 (5)	0.1012 (8)	0.0964 (7)	0.0925 (3)	0.1023 (9)	0.0945 (6)	0.0882 (2)	0.0763 (1)
	Clark ( $\downarrow$ )	1.2877 (11)	0.6860 (9)	0.6440 (4)	0.6605 (5)	0.6856 (8)	0.6702 (7)	0.6379 (3)	0.6909 (10)	0.6676 (6)	0.6289 (2)	0.5932 (1)
	Canberra ( $\downarrow$ )	2.0459 (11)	1.8308 (7)	1.8542 (8)	1.7863 (3)	1.9071 (9)	1.8244 (6)	1.8104 (5)	1.9161 (10)	1.8092 (4)	1.7603 (2)	1.5843 (1)
	KL ( $\downarrow$ )	0.5528 (11)	0.1227 (9)	0.1089 (5)	0.1088 (4)	0.1222 (8)	0.1142 (7)	0.1076 (3)	0.1245 (10)	0.1104 (6)	0.1027 (2)	0.1009 (1)
	Cosine ( $\uparrow$ )	0.8295 (11)	0.8843 (10)	0.9022 (4)	0.9016 (5)	0.8917 (8)	0.8964 (7)	0.9036 (3)	0.8899 (9)	0.9001 (6)	0.9148 (2)	0.9273 (1)

TABLE II: Quantitative comparison with baseline methods on the DMER and WESAD datasets under the subject-independent setting.

Settings	Measure	MSAF	PRD	PCL	HSC	MPCL
Settings	Measure	w/o	w/o	w/o	w/o
Subject- Dependent	Chebyshev ( $\downarrow$ )	0.0385	0.0425	0.0396	0.0388	0.0346
	Clark ( $\downarrow$ )	0.2643	0.2952	0.2914	0.2704	0.2369
	Canberra ( $\downarrow$ )	0.6652	0.7759	0.7466	0.6861	0.6068
	KL ( $\downarrow$ )	0.0258	0.0332	0.0317	0.0284	0.0177
	Cosine ( $\uparrow$ )	0.9766	0.9714	0.9702	0.9743	0.9874

TABLE III: Ablation study of different components on the DMER dataset. “ w/o ” indicates the removal of the corresponding module.

Settings	Measure	MSAF	PRD	PCL	HSC	MPCL
Settings	Measure	w/o	w/o	w/o	w/o
Subject- Dependent	Chebyshev ( $\downarrow$ )	0.0062	0.0081	0.0083	0.0068	0.0046
	Clark ( $\downarrow$ )	0.0525	0.0692	0.0674	0.0563	0.0417
	Canberra ( $\downarrow$ )	0.1263	0.1682	0.1625	0.1342	0.1026
	KL ( $\downarrow$ )	0.0007	0.0015	0.0014	0.0009	0.0003
	Cosine ( $\uparrow$ )	0.9984	0.9961	0.9954	0.9980	0.9998

TABLE IV: Ablation study of different components on WESAD dataset. “ w/o ” indicates the removal of the corresponding module.

Settings	Measure	PPG	GSR	EEG	Video	MPCL
Settings	Measure	w/o	w/o	w/o	w/o
Subject- Dependent	Chebyshev ( $\downarrow$ )	0.0362	0.0371	0.0412	0.0465	0.0346
	Clark ( $\downarrow$ )	0.2485	0.2533	0.2847	0.3105	0.2369
	Canberra ( $\downarrow$ )	0.6250	0.6384	0.7259	0.8214	0.6068
	KL ( $\downarrow$ )	0.0198	0.0215	0.0296	0.0368	0.0177
	Cosine ( $\uparrow$ )	0.9825	0.9804	0.9735	0.9668	0.9874

TABLE V: Ablation study of different modalities on DMER. “ w/o ” means the removal of the corresponding modality from the input.

Settings	Measure	EMG	EDA	ECG	ACC	MPCL
Settings	Measure	w/o	w/o	w/o	w/o
Subject- Dependent	Chebyshev ( $\downarrow$ )	0.0052	0.0055	0.0075	0.0092	0.0046
	Clark ( $\downarrow$ )	0.0458	0.0472	0.0585	0.0734	0.0417
	Canberra ( $\downarrow$ )	0.1105	0.1142	0.1428	0.1856	0.1026
	KL ( $\downarrow$ )	0.0004	0.0005	0.0011	0.0019	0.0003
	Cosine ( $\uparrow$ )	0.9992	0.9990	0.9972	0.9954	0.9998

TABLE VI: Ablation study of different modalities on WESAD. “ w/o ” means the removal of the corresponding modality from the input.

Protocol	Measure	DMER Dataset					WESAD Dataset
Protocol	Measure	w/ CrossAttn	w/ Feat-Co	w/ InfoNCE	w/ TransEnc	MPCL	w/ CrossAttn	w/ Feat-Co	w/ InfoNCE	w/ TransEnc	MPCL
Subject- Dependent	Chebyshev ( $\downarrow$ )	0.0363	0.0382	0.0374	0.0368	0.0346	0.0054	0.0067	0.0064	0.0059	0.0046
	Clark ( $\downarrow$ )	0.2497	0.2743	0.2648	0.2541	0.2369	0.0483	0.0556	0.0537	0.0498	0.0417
	Canberra ( $\downarrow$ )	0.6358	0.7012	0.6784	0.6486	0.6068	0.1147	0.1384	0.1319	0.1217	0.1026
	KL ( $\downarrow$ )	0.0213	0.0287	0.0246	0.0238	0.0177	0.0006	0.0011	0.0009	0.0007	0.0003
	Cosine ( $\uparrow$ )	0.9823	0.9748	0.9782	0.9791	0.9874	0.9989	0.9972	0.9976	0.9984	0.9998

TABLE VII: Comparison with alternative designs on DMER and WESAD. “ w/ ” indicates that the corresponding MPCL module is replaced by the specified alternative: CrossAttn (replacing MSAF), Feat-Co (replacing prototype-level co-occurrence), InfoNCE (replacing memory-guided retrieval), and TransEnc (replacing HSC).

Configuration	Subject-Dependent						Subject-Independent
Configuration	Chebyshev ( $\downarrow$ )	Clark ( $\downarrow$ )	Canberra ( $\downarrow$ )	KL ( $\downarrow$ )	Cosine ( $\uparrow$ )	Intersection ( $\uparrow$ )	Chebyshev ( $\downarrow$ )	Clark ( $\downarrow$ )	Canberra ( $\downarrow$ )	KL ( $\downarrow$ )	Cosine ( $\uparrow$ )	Intersection ( $\uparrow$ )
$\beta_{\text{low}}$	0.0368	0.2548	0.6477	0.0234	0.9782	0.9362	0.0794	0.6153	1.6458	0.1106	0.9167	0.8384
$\beta_{\text{mid}}$	0.0364	0.2519	0.6423	0.0227	0.9793	0.9374	0.0787	0.6121	1.6376	0.1092	0.9182	0.8403
$\beta_{\text{high}}$	0.0369	0.2576	0.6546	0.0243	0.9776	0.9358	0.0799	0.6184	1.6521	0.1114	0.9156	0.8372
$\{\beta_{\text{low}},\beta_{\text{mid}}\}$	0.0353	0.2447	0.6245	0.0204	0.9824	0.9403	0.0774	0.6023	1.6102	0.1051	0.9221	0.8453
$\{\beta_{\text{mid}},\beta_{\text{high}}\}$	0.0357	0.2468	0.6281	0.0209	0.9817	0.9392	0.0779	0.6051	1.6149	0.1062	0.9213	0.8441

TABLE VIII: Ablation study of different scaling strategies (

\beta

) in the MSAF module on the DMER dataset under both Subject-Dependent and Subject-Independent protocols.

Fig. 6: Ablation study of the number of prototypes

M

on the DMER dataset under subject-dependent and subject-independent settings.

Fig. 7: Ablation study of the PCL module on the DMER dataset over different values of

\beta

and

\tau_{\text{pcl}}^{-1}

. The horizontal axis shows

\tau_{\text{pcl}}^{-1}

; distinct curves correspond to different

\beta

values.

Fig. 8: Ablation study of the number of compression blocks

L

in the HSC module on DMER under subject-dependent and subject-independent setting.

Fig. 9: Visualization of emotion‑label correlation matrices. Red intensity indicates the strength of correlation.

Explore similar work

Sep 15, 2026

Affect-Prototype Guided Fusion for Open-Vocabulary Incomplete Multi-modal Emotion Recognition

Open-vocabulary multimodal emotion recognition (OV-MER) aims to generate open natural-language emotion labels from multimodal affective cues. In real-world scenarios, however, complete and synchronized modal data are difficult to obtain due to limitations of acquisition devices and user privacy constraints. Existing OV-MER methods are largely designed for full-modal inputs, and fail to perform effective feature fusion under modal missing conditions. Meanwhile, current fusion approaches designed for incomplete modalities mainly focus on fixed-label recognition context, and cannot satisfy the demand for fuse emotional cues guided with arbitrary emotion semantics in OV-MER context. To tackle these challenges, this paper proposes an Affect-Prototype-Conditioned Fusion (APCF) framework for incomplete open-vocabulary emotion recognition. As a candidate-free generative framework, APCF extends modal contribution learning to scenarios guided by arbitrary emotional semantics. Specifically, we construct an affect-prototype library to explicitly model multimodal contribution characteristics corresponding to diverse emotions, which provides dynamic constraints for modal fusion under different emotional semantic perspectives. Conditional retrieval and feature aggregation are conducted based on available modal features. The refined fused affective representations are then fed into an LLM decoder to produce open-vocabulary emotion labels. Experiments on the OV-MERD+ and MER-FG datasets demonstrate that APCF substantially outperforms state-of-the-art baselines.

Yichi Zhang, Shenyue Wang, Jing Luo +2

School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · OPPO Research Institute, Shanghai, China

Multimodal Emotion Recognition Multimodal Fusion Missing-Modality Learning

Ask PDF

Mar 18, 2026

Memory Bear AI Memory Science Engine for Multimodal Affective Intelligence: A Technical Report

Affective judgment in real interaction is rarely a purely local prediction problem. Emotional meaning often depends on prior trajectory, accumulated context, and multimodal evidence that may be weak, noisy, or incomplete at the current moment. Although multimodal emotion recognition (MER) has improved the integration of text, speech, and visual signals, many existing systems remain optimized for short-range inference and provide limited support for persistent affective memory, long-horizon dependency modeling, and robust interpretation under imperfect input. This technical report presents the Memory Bear AI Memory Science Engine, a memory-centered framework for multimodal affective intelligence. Instead of treating emotion as a transient output label, the framework models affective information as a structured and evolving variable within a memory system. It organizes processing through structured memory formation, working-memory aggregation, long-term consolidation, memory-driven retrieval, dynamic fusion calibration, and continuous memory updating. At its core, multimodal signals are transformed into structured Emotion Memory Units (EMUs), enabling affective information to be preserved, reactivated, and revised across interaction horizons. Experimental results show consistent gains over comparison systems across benchmark and business-grounded settings, with stronger accuracy and robustness, especially under noisy or missing-modality conditions. The framework offers a practical step from local emotion recognition toward more continuous, robust, and deployment-relevant affective intelligence.

Deliang Wen, Ke Sun, Yu Wang

Affective Computing Emotion Recognition Multimodal Emotion Recognition Multimodal Memory

Ask PDF

Sep 2, 2026

Cross-Modal Emotion Understanding: A Transformer-GAT Approach for Dialogue Emotion Recognition

Multimodal emotion recognition is a key research area in affective computing, with applications in sentiment analysis, intelligent customer service, and human-computer interaction. However, existing methods often rely on single-modal features or simple multimodal fusion, failing to capture the synergy between global and local contexts, which limits model performance and emotion understanding. To address this challenge, we propose Transformer-GAT, a hybrid framework that combines Transformer and the Graph Attention Network to enable cross-modal emotion understanding. The Transformer is used to capture global semantic information, while the Graph Attention Network is employed to model fine-grained relationships between modalities, thereby enhancing the representation of emotional features. Experiments on the IEMOCAP and MELD datasets show that our model achieves weighted F1 scores of 72.45% and 77.37%, outperforming state-of-the-art methods. These results demonstrate that Transformer-GAT effectively integrates multimodal features, balances global and local contexts, and provides deeper emotional insights, offering new directions for multimodal emotion computing.

Jiaqi Qiao, Yifan Lyu, Xiujuan Xu

Dalian University of Technology

Multimodal Emotion Recognition Emotion Recognition in Conversations Multimodal Fusion

Ask PDF