Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition

Ming Li; Yong-Jin Liu; Fang Liu; Huankun Sheng; Yeying Fan; Yixiang Wei; Minnan Luo; Weizhan Zhang

cs.LGFeb 24, 2026

Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition

Authors: Ming Li, Yong-Jin Liu, Fang Liu, Huankun Sheng, Yeying Fan, Yixiang Wei, Minnan Luo, Weizhan Zhang, +1 more

Organizations: MOE-Key Laboratory of Pervasive Computing, Department of Computer Science and Technology, Tsinghua University, Beijing, China. · the Key State Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China · the School of Computer Science and Technology, and Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · the Department of Computer Science and Computer Engineering, Texas A&M University, College Station, TX 77840 USA

Abstract

Emotion recognition from multi-modal physiological and behavioral signals plays a pivotal role in affective computing, yet most existing models remain constrained to the prediction of singular emotions in controlled laboratory settings. Real-world human emotional experiences, by contrast, are often characterized by the simultaneous presence of multiple affective states, spurring recent interest in mixed emotion recognition as an emotion distribution learning problem. Current approaches, however, often neglect the valence consistency and structured correlations inherent among coexisting emotions. To address this limitation, we propose a Memory-guided Prototypical Co-occurrence Learning (MPCL) framework that explicitly models emotion co-occurrence patterns. Specifically, we first fuse multi-modal signals via a multi-scale associative memory mechanism. To capture cross-modal semantic relationships, we construct emotion-specific prototype memory banks, yielding rich physiological and behavioral representations, and employ prototype relation distillation to ensure cross-modal alignment in the latent prototype space. Furthermore, inspired by human cognitive memory systems, we introduce a memory retrieval strategy to extract semantic-level co-occurrence associations across emotion categories. Through this bottom-up hierarchical abstraction process, our model learns affectively informative representations for accurate emotion distribution prediction. Comprehensive experiments on two public datasets demonstrate that MPCL consistently outperforms state-of-the-art methods in mixed emotion recognition, both quantitatively and qualitatively.

Figures & tables

Fig. 1: Overview of the MPCL Framework, which consists of three stages. (a) Multimodal Feature Extraction and Fusion: the Multi‑Scale Associative Fusion (MSAF) module fuses multi-modal physiological signals, while a separate behavioral encoder extracts structured behavioral feature. (b) Prototypical Alignment and Co‑occurrence Learning: emotion prototype memory banks are constructed from both modalities. The Prototype Relation Distillation (PRD) module enforces cross‑modal structural alignment, and the Prototypical Co‑occurrence Learning (PCL) module captures semantic‑level co‑occurrence patterns through memory retrieval. (c) Hierarchical Semantic Compression and Distribution Prediction: the Hierarchical Semantic Compression (HSC) module abstracts affective representations via a bottom‑up strategy, followed by a classifier that outputs the final emotion distribution.

Fig. 2: Fusion of multimodal physiological signals. EEG serves as the primary modality to integrate complementary information from auxiliary modalities (PPG and GSR) via a multi-scale associative memory mechanism. Three distinct scaling factors $\mathcal{B}=\{\beta_{\text{low}},\beta_{\text{mid}},\beta_{\text{high}}\}$ are employed to modulate the granularity of information aggregation.

Fig. 3: Prototype alignment process. Prototype memory banks are first constructed separately for the physiological and behavioral modalities, respectively. A semantics-enriched representation is then obtained as a weighted combination of prototypes from each bank. Meanwhile, the Prototype Relation Distillation (PRD) strategy enforces semantic structural consistency across the two modalities.

Fig. 4: Architecture of Prototypical Co‑occurrence Learning (PCL). The physiological embedding $x_{i}$ and the behavioral embedding $y_{i}$ retrieve associated representations $U_{x_{i}}$ and $U_{y_{i}}$ , respectively, via the Hopfield network. Within the physiological embedding space, $U_{x_{i}}$ (retrieved via the physiological query) serves as an anchor that is contrasted with the positive sample $U_{y_{i}}$ (retrieved from the corresponding behavioral query) and with negative samples $U_{y_{j}}$ (retrieved from mismatched behavioral queries, where $j\neq i$ ). The same procedure is applied symmetrically in the behavioral embedding space $V$ .

Dataset	Mesure	LDL			MER	Single-modal EDL			MMER	Multimodal EDL
Dataset	Mesure	PT-SVM [ 76 ]	AA-KNN [ 76 ]	SA-CPNN [ 77 ]	MAET [ 78 ]	LDL-LRR [ 79 ]	TLR-DL [ 80 ]	CAD [ 81 ]	CARAT [ 33 ]	EmotionDict [ 25 ]	HeLo [ 26 ]	MPCL
DMER	Chebyshev ( $\downarrow$ )	0.0851 (7)	0.0949 (10)	0.0917 (8)	0.0522 (3)	0.0936 (9)	0.0847 (6)	0.1017 (11)	0.0752 (4)	0.0753 (5)	0.0446 (2)	0.0346 (1)
	Clark ( $\downarrow$ )	0.5787 (6)	0.6649 (10)	0.6333 (8)	0.3762 (3)	0.6529 (9)	0.5380 (4)	0.7517 (11)	0.5951 (7)	0.5463 (5)	0.3256 (2)	0.2369 (1)
	Canberra ( $\downarrow$ )	1.5294 (7)	1.2357 (3)	1.8092 (9)	0.9875 (3)	1.8224 (10)	1.3271 (5)	1.9415 (11)	1.5965 (8)	1.4771 (6)	0.8664 (2)	0.6068 (1)
	KL ( $\downarrow$ )	0.0837 (6)	0.1409 (10)	0.1048 (8)	0.0288 (2)	0.0644 (4)	0.0925 (7)	0.1460 (11)	0.1137 (9)	0.0766 (5)	0.0323 (3)	0.0177 (1)
	Cosine ( $\uparrow$ )	0.9237 (8)	0.9315 (6)	0.9058 (9)	0.9688 (3)	0.9016 (10)	0.9339 (4)	0.8923 (11)	0.9332 (5)	0.9313 (7)	0.9714 (2)	0.9874 (1)

TABLE I: Quantitative comparison with baseline methods on the DMER and WESAD datasets under the subject-dependent setting.

Fig. 5: Comparison of emotion distribution predictions between MPCL and state-of-the-art baselines on two representative samples (Subject 26). GT denotes the ground-truth distribution. Emotion indices 1–10 correspond to: inspired, alert, excited, enthusiastic, determined, afraid, upset, nervous, scared, and distressed.

Dataset	Measure	LDL			MER	Single-modal EDL			MMER	Multimodal EDL
Dataset	Measure	PT-SVM [ 76 ]	AA-KNN [ 76 ]	SA-CPNN [ 77 ]	MAET [ 78 ]	LDL-LRR [ 79 ]	TLR-DL [ 80 ]	CAD [ 81 ]	CARAT [ 33 ]	EmotionDict [ 25 ]	HeLo [ 26 ]	MPCL
DMER	Chebyshev ( $\downarrow$ )	0.1462 (11)	0.1070 (10)	0.0928 (4)	0.0939 (5)	0.1012 (8)	0.0964 (7)	0.0925 (3)	0.1023 (9)	0.0945 (6)	0.0882 (2)	0.0763 (1)
	Clark ( $\downarrow$ )	1.2877 (11)	0.6860 (9)	0.6440 (4)	0.6605 (5)	0.6856 (8)	0.6702 (7)	0.6379 (3)	0.6909 (10)	0.6676 (6)	0.6289 (2)	0.5932 (1)
	Canberra ( $\downarrow$ )	2.0459 (11)	1.8308 (7)	1.8542 (8)	1.7863 (3)	1.9071 (9)	1.8244 (6)	1.8104 (5)	1.9161 (10)	1.8092 (4)	1.7603 (2)	1.5843 (1)
	KL ( $\downarrow$ )	0.5528 (11)	0.1227 (9)	0.1089 (5)	0.1088 (4)	0.1222 (8)	0.1142 (7)	0.1076 (3)	0.1245 (10)	0.1104 (6)	0.1027 (2)	0.1009 (1)
	Cosine ( $\uparrow$ )	0.8295 (11)	0.8843 (10)	0.9022 (4)	0.9016 (5)	0.8917 (8)	0.8964 (7)	0.9036 (3)	0.8899 (9)	0.9001 (6)	0.9148 (2)	0.9273 (1)

TABLE II: Quantitative comparison with baseline methods on the DMER and WESAD datasets under the subject-independent setting.

Settings	Measure	MSAF	PRD	PCL	HSC	MPCL
Settings	Measure	w/o	w/o	w/o	w/o
Subject- Dependent	Chebyshev ( $\downarrow$ )	0.0385	0.0425	0.0396	0.0388	0.0346
	Clark ( $\downarrow$ )	0.2643	0.2952	0.2914	0.2704	0.2369
	Canberra ( $\downarrow$ )	0.6652	0.7759	0.7466	0.6861	0.6068
	KL ( $\downarrow$ )	0.0258	0.0332	0.0317	0.0284	0.0177
	Cosine ( $\uparrow$ )	0.9766	0.9714	0.9702	0.9743	0.9874

TABLE III: Ablation study of different components on the DMER dataset. “ w/o ” indicates the removal of the corresponding module.

Settings	Measure	MSAF	PRD	PCL	HSC	MPCL
Settings	Measure	w/o	w/o	w/o	w/o
Subject- Dependent	Chebyshev ( $\downarrow$ )	0.0062	0.0081	0.0083	0.0068	0.0046
	Clark ( $\downarrow$ )	0.0525	0.0692	0.0674	0.0563	0.0417
	Canberra ( $\downarrow$ )	0.1263	0.1682	0.1625	0.1342	0.1026
	KL ( $\downarrow$ )	0.0007	0.0015	0.0014	0.0009	0.0003
	Cosine ( $\uparrow$ )	0.9984	0.9961	0.9954	0.9980	0.9998

TABLE IV: Ablation study of different components on WESAD dataset. “ w/o ” indicates the removal of the corresponding module.

Settings	Measure	PPG	GSR	EEG	Video	MPCL
Settings	Measure	w/o	w/o	w/o	w/o
Subject- Dependent	Chebyshev ( $\downarrow$ )	0.0362	0.0371	0.0412	0.0465	0.0346
	Clark ( $\downarrow$ )	0.2485	0.2533	0.2847	0.3105	0.2369
	Canberra ( $\downarrow$ )	0.6250	0.6384	0.7259	0.8214	0.6068
	KL ( $\downarrow$ )	0.0198	0.0215	0.0296	0.0368	0.0177
	Cosine ( $\uparrow$ )	0.9825	0.9804	0.9735	0.9668	0.9874

TABLE V: Ablation study of different modalities on DMER. “ w/o ” means the removal of the corresponding modality from the input.

Settings	Measure	EMG	EDA	ECG	ACC	MPCL
Settings	Measure	w/o	w/o	w/o	w/o
Subject- Dependent	Chebyshev ( $\downarrow$ )	0.0052	0.0055	0.0075	0.0092	0.0046
	Clark ( $\downarrow$ )	0.0458	0.0472	0.0585	0.0734	0.0417
	Canberra ( $\downarrow$ )	0.1105	0.1142	0.1428	0.1856	0.1026
	KL ( $\downarrow$ )	0.0004	0.0005	0.0011	0.0019	0.0003
	Cosine ( $\uparrow$ )	0.9992	0.9990	0.9972	0.9954	0.9998

TABLE VI: Ablation study of different modalities on WESAD. “ w/o ” means the removal of the corresponding modality from the input.

Protocol	Measure	DMER Dataset					WESAD Dataset
Protocol	Measure	w/ CrossAttn	w/ Feat-Co	w/ InfoNCE	w/ TransEnc	MPCL	w/ CrossAttn	w/ Feat-Co	w/ InfoNCE	w/ TransEnc	MPCL
Subject- Dependent	Chebyshev ( $\downarrow$ )	0.0363	0.0382	0.0374	0.0368	0.0346	0.0054	0.0067	0.0064	0.0059	0.0046
	Clark ( $\downarrow$ )	0.2497	0.2743	0.2648	0.2541	0.2369	0.0483	0.0556	0.0537	0.0498	0.0417
	Canberra ( $\downarrow$ )	0.6358	0.7012	0.6784	0.6486	0.6068	0.1147	0.1384	0.1319	0.1217	0.1026
	KL ( $\downarrow$ )	0.0213	0.0287	0.0246	0.0238	0.0177	0.0006	0.0011	0.0009	0.0007	0.0003
	Cosine ( $\uparrow$ )	0.9823	0.9748	0.9782	0.9791	0.9874	0.9989	0.9972	0.9976	0.9984	0.9998

TABLE VII: Comparison with alternative designs on DMER and WESAD. “ w/ ” indicates that the corresponding MPCL module is replaced by the specified alternative: CrossAttn (replacing MSAF), Feat-Co (replacing prototype-level co-occurrence), InfoNCE (replacing memory-guided retrieval), and TransEnc (replacing HSC).

Configuration	Subject-Dependent						Subject-Independent
Configuration	Chebyshev ( $\downarrow$ )	Clark ( $\downarrow$ )	Canberra ( $\downarrow$ )	KL ( $\downarrow$ )	Cosine ( $\uparrow$ )	Intersection ( $\uparrow$ )	Chebyshev ( $\downarrow$ )	Clark ( $\downarrow$ )	Canberra ( $\downarrow$ )	KL ( $\downarrow$ )	Cosine ( $\uparrow$ )	Intersection ( $\uparrow$ )
$\beta_{\text{low}}$	0.0368	0.2548	0.6477	0.0234	0.9782	0.9362	0.0794	0.6153	1.6458	0.1106	0.9167	0.8384
$\beta_{\text{mid}}$	0.0364	0.2519	0.6423	0.0227	0.9793	0.9374	0.0787	0.6121	1.6376	0.1092	0.9182	0.8403
$\beta_{\text{high}}$	0.0369	0.2576	0.6546	0.0243	0.9776	0.9358	0.0799	0.6184	1.6521	0.1114	0.9156	0.8372
$\{\beta_{\text{low}},\beta_{\text{mid}}\}$	0.0353	0.2447	0.6245	0.0204	0.9824	0.9403	0.0774	0.6023	1.6102	0.1051	0.9221	0.8453
$\{\beta_{\text{mid}},\beta_{\text{high}}\}$	0.0357	0.2468	0.6281	0.0209	0.9817	0.9392	0.0779	0.6051	1.6149	0.1062	0.9213	0.8441

TABLE VIII: Ablation study of different scaling strategies (

\beta

) in the MSAF module on the DMER dataset under both Subject-Dependent and Subject-Independent protocols.

Fig. 6: Ablation study of the number of prototypes

M

on the DMER dataset under subject-dependent and subject-independent settings.

Fig. 7: Ablation study of the PCL module on the DMER dataset over different values of

\beta

and

\tau_{\text{pcl}}^{-1}

. The horizontal axis shows

\tau_{\text{pcl}}^{-1}

; distinct curves correspond to different

\beta

values.

Fig. 8: Ablation study of the number of compression blocks

L

in the HSC module on DMER under subject-dependent and subject-independent setting.

Fig. 9: Visualization of emotion‑label correlation matrices. Red intensity indicates the strength of correlation.

Explore similar work

Affect-Prototype Guided Fusion for Open-Vocabulary Incomplete Multi-modal Emotion Recognition
Sep 15, 2026Yichi Zhang, Shenyue Wang, Jing Luo +2Multimodal Emotion Recognition Multimodal Fusion
School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · OPPO Research Institute, Shanghai, China
Memory Bear AI Memory Science Engine for Multimodal Affective Intelligence: A Technical Report
Mar 18, 2026Deliang Wen, Ke Sun, Yu WangAffective Computing Emotion Recognition
Cross-Modal Emotion Understanding: A Transformer-GAT Approach for Dialogue Emotion Recognition
Sep 2, 2026Jiaqi Qiao, Yifan Lyu, Xiujuan XuMultimodal Emotion Recognition Emotion Recognition in Conversations
Dalian University of Technology

Sep 15, 2026J/K move · Enter open · S save

Affect-Prototype Guided Fusion for Open-Vocabulary Incomplete Multi-modal Emotion Recognition

Yichi Zhang, Shenyue Wang, Jing Luo, Chunyang Yu+1

School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · OPPO Research Institute, Shanghai, China

Multimodal Emotion Recognition Multimodal Fusion Missing-Modality Learning

Open paper AskPDF arXiv

Open-vocabulary multimodal emotion recognition (OV-MER) aims to generate open natural-language emotion labels from multimodal affective cues. In real-world scenarios, however, complete and synchronized modal data are difficult to obtain due to limitations of acquisition devices and user privacy constraints. Existing OV-MER methods are largely designed for full-modal inputs, and fail to perform effective feature fusion under modal missing conditions. Meanwhile, current fusion approaches designed for incomplete modalities mainly focus on fixed-label recognition context, and cannot satisfy the demand for fuse emotional cues guided with arbitrary emotion semantics in OV-MER context. To tackle these challenges, this paper proposes an Affect-Prototype-Conditioned Fusion (APCF) framework for incomplete open-vocabulary emotion recognition. As a candidate-free generative framework, APCF extends modal contribution learning to scenarios guided by arbitrary emotional semantics. Specifically, we construct an affect-prototype library to explicitly model multimodal contribution characteristics corresponding to diverse emotions, which provides dynamic constraints for modal fusion under different emotional semantic perspectives. Conditional retrieval and feature aggregation are conducted based on available modal features. The refined fused affective representations are then fed into an LLM decoder to produce open-vocabulary emotion labels. Experiments on the OV-MERD+ and MER-FG datasets demonstrate that APCF substantially outperforms state-of-the-art baselines.

Abstract

Explore similar work

Affect-Prototype Guided Fusion for Open-Vocabulary Incomplete Multi-modal Emotion Recognition

Memory Bear AI Memory Science Engine for Multimodal Affective Intelligence: A Technical Report

Cross-Modal Emotion Understanding: A Transformer-GAT Approach for Dialogue Emotion Recognition