Context-Aware Topological Distillation Graph Network for Asymmetric EEG-Eye Movement Emotion Recognition

Peijun Shi; Xiujin Zhu; Zhenguo Lao; Chee-Onn Chow; Wei Ru Wong; Zhen Liu; Xiaobing Jiang

cs.CVOct 19, 2026

Context-Aware Topological Distillation Graph Network for Asymmetric EEG-Eye Movement Emotion Recognition

Authors: Peijun Shi, Xiujin Zhu, Zhenguo Lao, Chee-Onn Chow, Wei Ru Wong, Zhen Liu, Xiaobing Jiang

Organizations: Department of Electrical Engineering, Faculty of Engineering, Universiti Malaya, 50603 Lembah Pantai, Kuala Lumpur, Malaysia. · Department of Neurosurgery, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, Hubei 430022, China · Hubei Provincial Clinical Research Center for Brain Computer Interface, Huazhong University of Science and Technology, Wuhan, Hubei 430022, China.

Abstract

EEG-eye movement emotion recognition can support mental-health monitoring and adaptive interfaces, but most fusion models treat the two signals as parallel features, classify each window alone and degrade when eye data are lost. We propose the Context-Aware Topological Distillation Graph Network (CTD-GNet), in which EEG features form the nodes of an electrode graph, and eye movements and preceding windows modulate its edges, nodes and readout, the edges through electrode-specific prompts. Bidirectional distillation of the graph, node relations and predictions couples this multimodal teacher with a single-window EEG student, improving the teacher and aligning the two graphs 35-fold. In leave-one-subject-out tests on SEED and SEED-IV against 17 baselines, CTD-GNet has the highest mean accuracy, 84.91% and 76.17%, significantly above 9 and 6 of 13 eye-based baselines after Holm correction, the smallest cross-subject spread among multimodal models and the highest worst-subject accuracy. It is the most accurate model at every rate of missing eye data on SEED-IV, keeps 66.1% on SEED without eye data, more than any EEG-only model, and stays 9.3 and 11.9 points above all single-window fusion models under eye-movement noise of one standard deviation. On film clips unseen in training, it leads on SEED with 77.1%.

Figures & tables

Fig. 1: Two ways to combine EEG and eye movements. (a) Symmetric fusion encodes both signals as feature vectors and merges them by concatenation or cross-attention; missing eye data change the fused feature, and each window is classified independently. (b) CTD-GNet: EEG gives the nodes of an electrode graph, and eye movements (green) and the preceding windows (purple) modulate its edges, nodes and readout. Bidirectional topology distillation couples the graph with that of an EEG-only student, which serves devices without an eye tracker.

Fig. 2: Architecture of CTD-GNet. Teacher (blue): a GRU context bank encodes the $K$ preceding windows into $\mathbf{c}_{t}$ ; a gated node pathway adds the eye movements to the EEG node embeddings; the generator adds the context prompt $\boldsymbol{\Pi}^{c}$ to the queries, the eye prompt $\boldsymbol{\Pi}^{e}$ to the keys and $\log\mathbf{A}^{\mathrm{phy}}$ to the scores, and a row-wise softmax gives $\mathbf{A}_{T}$ ; a GCN propagates over $\mathbf{A}_{T}$ , and a gated multimodal unit fuses the graph readout with the eye movements and $\mathbf{c}_{t}$ . Student (orange): the same operations with prompts from the current EEG window. Red dashed arrows: distillation losses, with gradients to both networks. Right: deployment and training objective.

Fig. 3: Dual-prompt graph generator. (a) Proposition 1 : with shared prompts $\boldsymbol{\gamma}$ and $\boldsymbol{\eta}$ , the term $(\mathbf{a}_{i}+\boldsymbol{\gamma})\boldsymbol{\eta}^{\top}$ is constant within each row (red frame) and is removed by the row-wise softmax. (b) Electrode-specific prompts make $\mathbf{q}_{i}\boldsymbol{\pi}^{e\top}_{j}$ vary with $i$ and $j$ , so the eye movements change the graph. (c) The physical prior acts as a soft mask: $\mathrm{softmax}_{\mathrm{row}}(\mathbf{S})\odot\mathbf{A}^{\mathrm{phy}}$ normalized per row equals $\mathrm{softmax}_{\mathrm{row}}(\mathbf{S}+\log\mathbf{A}^{\mathrm{phy}})$ .

Fig. 4: Bidirectional multi-level topology distillation. (a) $\mathcal{L}_{\mathrm{topo}}$ compares rows of the softened topologies, $\mathcal{L}_{\mathrm{rel}}$ rows of the $N\times N$ electrode relation matrices, which need no projection between networks of different width, and $\mathcal{L}_{\mathrm{logit}}$ the class distributions. (b) Proposition 2 : the topology difference $\Delta\mathbf{A}$ bounds the message difference. (c) Training: $\omega(e)$ ramps to 1 over $E_{w}$ epochs, and the distillation gradients reach both networks.

Fig. 5: Accuracy on each test subject for the 14 models that use eye movements. Boxes span the quartiles over the test subjects, whiskers reach the worst and the best subject, dots mark the individual subjects and diamonds the mean.

Method	SEED				SEED-IV
Method	Negative	Neutral	Positive	Macro	Neutral	Sad	Fear	Happy	Macro
SVM	54.8	57.3	79.3	63.8	45.9	43.7	38.7	45.6	43.5
EEGNet [ 21 ]	38.1	43.1	60.6	47.3	34.2	29.1	26.5	38.2	32.0
DGCNN [ 25 ]	53.4	57.5	79.1	63.3	43.7	42.0	38.5	45.6	42.5
RGNN [ 26 ]	44.5	51.6	78.3	58.2	40.3	33.0	29.5	40.5	35.9
MLP (eye only)	71.2	80.6	73.5	75.1	76.0	63.2	78.3	67.0	71.1

TABLE II: F1 score of each class (%), mean over the test subjects. Macro: mean over the classes, as in Table I . The best value in each column is in bold .

Method	SEED		SEED-IV
Method	Acc	F1	Acc	F1
Student without distillation	63.85	63.22	41.93	41.58
Vanilla KD [ 39 ]	64.46	63.84	41.85	41.54
FitNets [ 40 ]	64.00	63.55	41.99	41.77
RKD [ 41 ]	63.92	63.25	41.55	41.17
SP [ 42 ]	63.61	63.04	42.08	41.78

TABLE III: Prediction without eye movements (accuracy and macro-F1, %). Upper block: student trained with different distillation losses in the base configuration. Middle block: multimodal models tested with all eye movements removed. Lower block: CTD-GNet in the final configuration.

Model	Input	Params	FLOPs	CPU	GPU
DGCNN [ 25 ]	EEG	131.5k	0.79M	0.09	0.20
RGNN [ 26 ]	EEG	2.6k	0.12M	0.08	0.20
DCCA [ 6 ]	EEG+Eye	120.7k	0.24M	0.08	0.15
TFN [ 8 ]	EEG+Eye	204.6k	0.41M	0.09	0.16
LMF [ 9 ]	EEG+Eye	65.3k	0.13M	0.10	0.21
MISA [ 36 ]	EEG+Eye	125.5k	0.44M	0.16	0.29

TABLE IV: Computational cost per window on SEED-IV (

K=5

). FLOPs count a multiply-accumulate as two operations, with recurrent layers counted analytically; latency (ms) is measured at batch size one on one core of an Intel Xeon Gold 6430 CPU and on an NVIDIA RTX 4090 GPU. Hist.:

K

past windows.

Method	SEED	SEED-IV
DGCNN [ 25 ]	63.4	39.8
MLP (eye only)	63.2	57.6
Feature concatenation	70.2	61.7
Bimodal-LSTM [ 14 ]	72.6	62.3
MMIN [ 16 ]	71.2	58.9
CTD-GNet student	64.2	38.9

TABLE V: Generalization to new subjects and new stimuli: accuracy (%) on film clips that no training or validation subject has seen. In each fold, one clip per emotion, on SEED-IV one per emotion and session, is held out from all training and validation subjects.

Variant	SEED		SEED-IV
Variant	Acc	F1	Acc	F1
Final (GRU, bidirectional)	84.91	84.67	76.17	76.01
TCN instead of GRU	83.92 †	83.68	75.65	75.43
One-directional distillation	83.71 †	83.52	75.90	75.74
No distillation	83.71 †	83.53	75.82	75.65
Bidirectional, logit loss only	84.57	84.35	76.25	76.08

TABLE VI: Ablation of the teacher (accuracy and macro-F1, %). Upper block: the final configuration and variants of it. Lower block: components removed from the base configuration, which uses a TCN context encoder and one-directional distillation. † :

p<0.05

against the first row of the block, two-sided Wilcoxon signed-rank test on per-fold accuracies, uncorrected; with Holm correction within each block, no difference reaches the 0.05 level.

Fig. 6: Test accuracy versus the position of the window within the trial, in 20-s bins plotted at the bin centre, over all folds and seeds. Bins with fewer than 1000 test windows are omitted.

Fig. 7: Validation accuracy over the hyperparameter grid, relative to the selected configuration, which lies at zero. Each panel varies one hyperparameter and keeps the others at their selected values. The temperature and the loss weights are searched in the base configuration, whose stop-gradient leaves the teacher unaffected by them, so only the student is shown in these panels.

Fig. 8: Accuracy under missing (a, b) and noisy (c, d) eye-movement data, mean over the test subjects. Blue band: teacher

\pm

one standard deviation over the test subjects. Grey band: range of the other seven models that use eye movements, namely the eye-only MLP, feature concatenation, DCCA-AM, TFN, LMF, MISA and ModDrop. The dual mode uses the student for windows without eye data; under noise, it differs from the teacher only on the few windows whose eye data are invalid in the recordings.

Dataset	Method	Missing rate $r$					Noise level $\sigma_{n}$
Dataset	Method	20%	40%	60%	80%	100%	0.25	0.5	1	2
SEED	MLP (eye only)	66.7	58.5	50.0	41.7	33.4	73.0	67.8	57.7	47.2
	Feature concatenation	78.1	73.5	69.0	64.4	60.0	81.6	78.9	71.2	58.5
	DCCA-AM [ 6 ]	76.4	71.8	67.3	62.8	58.3	79.6	76.5	69.4	59.9
	TFN [ 8 ]	75.9	70.7	65.5	60.4	55.3	79.6	75.8	67.3	56.6
	LMF [ 9 ]	76.9	71.6	66.3	61.1	55.9	80.1	76.0	67.2	56.5

TABLE VII: Accuracy (%) under missing and noisy eye-movement data, mean over the test subjects and five random draws. The results without corruption are those of Table I . The best value in each column is in bold .

Fig. 9: Learned graphs per emotion on SEED-IV. Rows: teacher topology $\mathbf{A}_{T}$ , distilled student topology $\mathbf{A}_{S}$ , student topology without distillation, and eye-movement attribution $\Delta\mathbf{A}=\mathbb{E}_{t}[\mathbf{A}_{T}-\mathbf{A}_{T}^{(\mathbf{x}_{e}=\mathbf{0})}]$ , where the eye features of all windows are set to their calibrated mean of zero; red edges gain and blue edges lose weight through the eye movements. Each panel shows the 5% strongest edges averaged over the test windows of the class.

Fig. 10: Learned graphs per emotion on SEED, drawn as in Fig. 9 .

Fig. 11: Band-wise topology sensitivity $I_{b,i}$ of the teacher, defined in Eq. ( 20 ), for the $\alpha$ , $\beta$ and $\gamma$ bands and each emotion, averaged over the test windows of all folds and seeds. The values at the 62 electrodes (dots) are interpolated with a thin-plate spline; nose up, left hemisphere on the left, one colour scale for both datasets.

Distillation	SEED		SEED-IV
Distillation	KL $\downarrow$	Jaccard $\uparrow$	KL $\downarrow$	Jaccard $\uparrow$
None	7.55 $\,\pm\,$ 1.42	0.209 $\,\pm\,$ 0.047	7.90 $\,\pm\,$ 1.47	0.093 $\,\pm\,$ 0.030
Bidirectional	0.21 $\,\pm\,$ 0.09	0.414 $\,\pm\,$ 0.131	0.23 $\,\pm\,$ 0.06	0.226 $\,\pm\,$ 0.063

TABLE VIII: Agreement between the teacher graph

\mathbf{A}_{T}

and the student graph

\mathbf{A}_{S}

on the test windows: mean row-wise KL divergence

\mathrm{KL}(\mathbf{A}_{T,i}\|\mathbf{A}_{S,i})

and Jaccard overlap of the 10% strongest edges, mean

\pm

standard deviation over the 36 (SEED) and 45 (SEED-IV) runs of all folds and seeds.

Dataset	Emotion	Frontal	Central	Temporal	Parietal	Occipital
SEED	Negative	$-$ 0.70	$-$ 0.33	$-$ 2.05	$-$ 0.48	$-$ 0.54
	Neutral	$-$ 0.95	$-$ 0.14	$-$ 2.43	$-$ 0.54	$-$ 0.80
	Positive	$-$ 0.49	$-$ 0.14	$-$ 1.30	$-$ 0.16	$-$ 0.51
SEED-IV	Fear	$-$ 0.38	$-$ 0.01	$-$ 0.55	$-$ 0.31	$-$ 0.50
	Happy	$-$ 0.34	$-$ 0.16	$-$ 0.24	$-$ 0.52	$-$ 0.25
	Neutral	$-$ 0.58	$-$ 0.00	$-$ 0.27	$-$ 0.27	$-$ 0.04

TABLE IX: Change of the attention received by each scalp region through the eye movements,

\frac{100}{N}\sum_{i}\sum_{j\in\mathcal{R}}\Delta A_{ij}

(points), for the teacher graph averaged over the test windows of each emotion. Frontal: FP, AF and F electrodes (14); central: FC and C (14); temporal: FT7, FT8, T7, T8, TP7 and TP8 (6); parietal: CP and P (16); occipital: PO, O and CB (12). Each row sums to zero.

Multimodal Emotion Recognition

Multimodal emotion recognition combines signals such as speech, text, facial expressions, video, and physiology to infer affect.

See all 150 papers

Graph-Regularized Deep Learning for EEG-Based Emotion Recognition with Psychologically-Grounded Label Structure
Jul 8, 2026Dongyang Kuang, Zizheng Ma, Yushan Zhang +1EEG Emotion Recognition
School of Mathematics (Zhuhai), Sun Yat-sen University, Guangzhou, China
GL-LFGNN:A Global-Local Dual-branch Causal Graph Neural Network Based on Liang-Kleeman Information Flow for EEG Emotion Recognition
May 24, 2026Ziyi Wang, Dongyang KuangGraph Neural Networks EEG Emotion Recognition
A Multi-Scale Temporal Framework with Dynamic Fusion for EEG-Based Emotion Recognition
Aug 10, 2026Stefanos Gkikas, Yang Guo, Guangliang Li +3Multi-Scale Feature Fusion EEG Emotion Recognition
Honda Research Institute Japan Wako City, Japan · Faculty of Information Science and Engineering Ocean University of China Qingdao, China · BioSIS (Biosensing & Intelligent Systems) Lab Centre for Intelligent Computing and Systems University of Canberra Canberra, Australia +1

Jul 8, 2026J/K move · Enter open · S save

Graph-Regularized Deep Learning for EEG-Based Emotion Recognition with Psychologically-Grounded Label Structure

Dongyang Kuang, Zizheng Ma, Yushan Zhang, Xiaocong Zeng

School of Mathematics (Zhuhai), Sun Yat-sen University, Guangzhou, China

EEG Emotion Recognition

Open paper AskPDF arXiv

EEG-based emotion recognition is critical for mental health monitoring and affective brain-computer interfaces, yet existing deep learning approaches often treat emotion classes as isolated labels, ignoring their psychological interdependencies. We propose a graph-regularized learning framework that conceptualizes emotions as nodes in a graph where edges encode proximity based on dimensional emotion theories. We adapt three complementary regularization strategies--Graph Label Smoothing (intuitive soft labeling), Commuting distance on graph via Graph Laplacian (spectral graph theory), and Sliced Wasserstein Distance (optimal transport on graph)--ordered by increasing computational complexity. These strategies penalize model predictions that deviate from the established emotion topology. Our framework is evaluated across three representative backbone architectures: AudioTransformer (pure transformer), Conformer (CNN-transformer hybrid), and DCGNN (causal graph neural network), demonstrating architecture-agnostic benefits. Experiments on SEED-IV (4 classes) and SEED-V (5 classes) datasets show consistent improvements: best case up to +5.42% accuracy and 39% reduction in psychologically implausible misclassifications. Ultimately, our framework help raise the upper bound of performance achievable with standard approaches. Code will be released.

Abstract

Multimodal Emotion Recognition

Explore similar work

Graph-Regularized Deep Learning for EEG-Based Emotion Recognition with Psychologically-Grounded Label Structure

GL-LFGNN:A Global-Local Dual-branch Causal Graph Neural Network Based on Liang-Kleeman Information Flow for EEG Emotion Recognition

A Multi-Scale Temporal Framework with Dynamic Fusion for EEG-Based Emotion Recognition