Multiscale Temporal-Spectral-Spatial EEG Learning With Modality-Specific and Modality-Invariant Representations for Emotion Recognition

Jian Zhang; Manrong Liu; Yilei Huang; Wenhua Liu; Xiaohui Yang; Lijun Yang

cs.CVOct 20, 2026

Multiscale Temporal-Spectral-Spatial EEG Learning With Modality-Specific and Modality-Invariant Representations for Emotion Recognition

Authors: Jian Zhang, Manrong Liu, Yilei Huang, Wenhua Liu, Xiaohui Yang, Lijun Yang

Organizations: Henan Engineering Research Center for Artificial Intelligence Theory and Algorithms, School of Mathematics and Statistics, Henan University, Kaifeng, 475004, China. · The Second Clinical College, Henan University, Kaifeng, 475004, China.

Abstract

Multimodal physiological signals provide complementary cues for emotion recognition. They also raise two related challenges. Emotion-related EEG patterns vary across time, frequency bands, electrodes, and brain regions. Different physiological modalities also contain both modality-specific and shared affective information. This paper presents GMSI-Net, an EEG-centered multimodal emotion recognition framework. It has three key components. The temporal-spatial-spectral-region (TSSR) encoder models multiscale temporal-channel patterns, differential entropy features, and local-to-global brain-region relations. The modality-specific representation (MSR) branch uses bidirectional cross-modal attention and learnable gates. It controls information transfer while retaining modality-specific cues. The modality-invariant representation (MIR) branch learns a shared affective representation. An adaptively weighted contrastive loss aligns this representation across modalities. Independent multiscale Conv-Transformer encoders process the peripheral physiological signals. GMSI-Net is evaluated on DEAP and MAHNOB-HCI using within-subject random ten-fold cross-validation. The model reaches 96.21 (1.47) and 96.41 (1.60) for valence and arousal on DEAP. It reaches 94.10 (2.86) and 94.48 (3.77) on MAHNOB-HCI. These results indicate that combining EEG features with complementary physiological information can improve emotion recognition.

Figures & tables

Fig. 1: Architecture of GMSI-Net. The EEG and peripheral encoders feed gated MSR interactions and an adaptive contrastive MIR branch. The resulting representations are fused for emotion classification.

Fig. 2: TSSR encoder. Temporal-channel and spectral features are fused before local and global brain-region modeling.

Setting	DEAP and MAHNOB-HCI
Brain partition	DL
Batch size	60
Learning rate	0.002
Adam $(\beta_{1},\beta_{2})$	$(0.5,0.9)$
Weight decay	0.005
$\lambda,\tau$	$0.2,\ 0.2$

TABLE I: Training and Reproducibility Settings

Method	Modality	Accuracy (%)
		Valence	Arousal
Ding et al. [ 11 ]	EEG+PPS	94.88	95.26
GBV-Net [ 17 ]	EEG+PPS+Face	95.10	95.65
Att-CNN-GRU [ 18 ]	EEG+ECG	92.50	91.30
Diff-MT [ 19 ]	EEG+EOG+EMG+GSR	94.83	93.95
HCNN [ 20 ]	EEG+Face	87.84	88.09

TABLE II: Classification Accuracy on DEAP

Method	Modality	Accuracy (%)
		Valence	Arousal
HCNN [ 20 ]	EEG+Face	91.40	90.72
STST [ 21 ]	EEG+micro-expression	86.10	85.70
DMMVAE-LSTM ∗ [ 22 ]	EEG+Face	$93.92\pm 1.80$	$93.18\pm 1.90$
DeepVADNet † [ 23 ]	EEG+bio+Face	89.98	88.60
Liu et al. ‡ [ 24 ]	EEG+Speech	81.09	80.11

TABLE III: Classification Accuracy on MAHNOB-HCI

	DEAP		MAHNOB-HCI
Setting	Valence	Arousal	Valence	Arousal
w/o MCA encoder	$94.73\pm 1.86$	$94.96\pm 2.21$	$93.09\pm 3.48$	$92.81\pm 5.10$
w/o DE features	$93.61\pm 2.11$	$93.98\pm 2.17$	$73.24\pm 5.29$	$76.12\pm 9.54$
w/o brain-region graph	$94.54\pm 1.83$	$94.70\pm 2.21$	$92.50\pm 3.39$	$92.28\pm 5.23$
w/o peripheral Conv-Transformer	$93.33\pm 3.10$	$93.38\pm 4.07$	$92.82\pm 3.75$	$92.44\pm 5.19$
w/o cross-modal attention	$94.89\pm 1.93$	$95.22\pm 2.09$	$92.82\pm 3.36$	$92.72\pm 4.79$

TABLE IV: Ablation Study on DEAP and MAHNOB-HCI

Combination	ACC $\pm$ Std (%)
	Valence	Arousal
EEG	$89.45\pm 4.65$	$90.45\pm 4.49$
EEG+EOG	$91.82\pm 3.18$	$92.18\pm 3.06$
EEG+EMG	$92.17\pm 3.73$	$92.45\pm 5.32$
EEG+GSR	$88.99\pm 4.81$	$90.44\pm 3.96$
EEG+EOG+EMG	$93.83\pm 1.70$	$94.07\pm 1.80$

TABLE V: Modality Combinations on DEAP

Combination	ACC $\pm$ Std (%)
	Valence	Arousal
EEG	$91.38\pm 3.99$	$91.38\pm 5.27$
EEG+ECG	$91.88\pm 4.36$	$92.93\pm 4.53$
EEG+GSR	$91.77\pm 3.65$	$92.14\pm 4.83$
EEG+ECG+GSR	$94.10\pm 2.86$	$94.48\pm 3.77$

TABLE VI: Modality Combinations on MAHNOB-HCI

Partition	DEAP		MAHNOB-HCI
	Valence	Arousal	Valence	Arousal
H	94.52	94.51	91.49	91.22
RL	95.16	95.27	91.63	91.13
DL	95.37	95.79	93.76	93.72
F	95.19	95.37	92.83	93.07
T	95.27	95.25	91.49	91.26

TABLE VII: Brain-Region Partition Analysis (Accuracy, %)

Fig. 3: Band-wise EEG topographies for valence on DEAP. The first two rows show relative DE maps for low- and high-valence states across the $\delta$ , $\theta$ , $\alpha$ , $\beta$ , and $\gamma$ bands. The bottom row shows the high-minus-low contrast, revealing spatial and spectral differences between emotional states.

Fig. 4: EEG channel importance on DEAP obtained by channel perturbation. Panels (a)–(d) represent low valence, high valence, low arousal, and high arousal, respectively. The normalized importance maps illustrate task-dependent spatial contributions of EEG electrodes.

Fig. 5: t-SNE visualization of feature distributions before and after GMSI-Net fusion. The top and bottom rows correspond to valence and arousal, respectively. The left panels show pre-fusion features, while the right panels show fused representations. Silhouette scores increase from 0.084 to 0.567 for valence and from 0.094 to 0.535 for arousal.

EEG Emotion Recognition

Electroencephalography (EEG) emotion recognition uses brain signals to classify or track affective states across people and recording settings.

See all 31 papers

Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition
Sep 25, 2026Philip H. Lee, Shreeram Suresh Chandra, John H. L. HansenEEG Artifact Removal Differential Attention
Independent Researcher, USA · Center for Language and Speech Processing (CLSP), Johns Hopkins University, Baltimore, MD, USA · Center for Robust Speech Systems (CRSS), University of Texas at Dallas, Richardson, TX, USA
A Multi-Scale Temporal Framework with Dynamic Fusion for EEG-Based Emotion Recognition
Aug 10, 2026Stefanos Gkikas, Yang Guo, Guangliang Li +3Multi-Scale Feature Fusion EEG Emotion Recognition
Honda Research Institute Japan Wako City, Japan · Faculty of Information Science and Engineering Ocean University of China Qingdao, China · BioSIS (Biosensing & Intelligent Systems) Lab Centre for Intelligent Computing and Systems University of Canberra Canberra, Australia +1
Context-aware tokenization for Cross-subject Emotion Decoding from EEG
May 30, 2026Jiaxin Qing, Lexin LiCross-Subject EEG Decoding EEG Decoding
University of California, Berkeley

Sep 25, 2026J/K move · Enter open · S save

Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition

Philip H. Lee, Shreeram Suresh Chandra, John H. L. Hansen

Independent Researcher, USA · Center for Language and Speech Processing (CLSP), Johns Hopkins University, Baltimore, MD, USA · Center for Robust Speech Systems (CRSS), University of Texas at Dallas, Richardson, TX, USA

EEG Artifact Removal Differential Attention Multimodal Emotion Recognition EEG Emotion Recognition Multimodal Fusion Electroencephalography

Open paper AskPDF arXiv

Multimodal emotion recognition (MER) increasingly pairs EEG with speech, treating internal neural signals and external vocal expression as informative views of affect. In practice, naive fusion underperforms the stronger single modality, because EEG artifacts inject noise that corrupts the shared representation. We introduce EmoSpeechBrain, a multimodal framework built on the insight that noise suppression is a precondition for effective fusion. Its EEG encoder uses differential attention, taking the difference between two attention maps to cancel shared noise and isolate discriminative neural activity. An attention-based gating adapter aligns both modalities in a shared space and weights each one's contribution to the prediction. On two datasets - PME4 and EAV, EmoSpeechBrain improves MER accuracy by up to 12.9% over other state-of-the-art (SOTA) EEG encoders, and surpasses unimodal speech and EEG baselines by up to 13.1% and 23.1%. These results show that once EEG noise is suppressed, fusion delivers gains that naive combination cannot.

Abstract

EEG Emotion Recognition

Explore similar work

Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition

A Multi-Scale Temporal Framework with Dynamic Fusion for EEG-Based Emotion Recognition

Context-aware tokenization for Cross-subject Emotion Decoding from EEG