EEG-Based Multimodal Learning via Hyperbolic Mixture-of-Curvature Experts

Runhe Zhou; Shanglin Li; Guanxiang Huang; Xinliang Zhou; Qibin Zhao; Motoaki Kawanabe; Yi Ding; Cuntai Guan

cs.LGApr 14, 2026

EEG-Based Multimodal Learning via Hyperbolic Mixture-of-Curvature Experts

Authors: Runhe Zhou, Shanglin Li, Guanxiang Huang, Xinliang Zhou, Qibin Zhao, Motoaki Kawanabe, Yi Ding, Cuntai Guan

Organizations: Nanyang Technological University, Singapore · BIFOLD, Berlin Institute for the Foundations of Learning and Data, Berlin, Germany · ATR, Kyoto, Japan · Riken AIP, Tokyo, Japan ·

Abstract

Electroencephalography (EEG)-based multimodal learning integrates brain signals with complementary modalities to improve mental state assessment, providing great clinical potential. The effectiveness of such paradigms largely depends on the representation learning on heterogeneous modalities. For EEG-based paradigms, one promising approach is to leverage their hierarchical structures, as recent studies have shown that both EEG and associated modalities (e.g., facial expressions) exhibit hierarchical structures reflecting complex cognitive processes. However, Euclidean embeddings struggle to represent these hierarchical structures due to their flat geometry, while hyperbolic spaces, with their exponential growth property, are naturally suited for them. In this work, we propose EEG-MoCE, a novel hyperbolic mixture-of-curvature experts framework designed for multimodal neurotechnology. EEG-MoCE assigns each modality to an expert in a learnable-curvature hyperbolic space, enabling adaptive modeling of its intrinsic geometry. A curvature-aware fusion strategy then dynamically weights experts, emphasizing modalities with richer hierarchical information. Extensive experiments on benchmark datasets demonstrate that EEG-MoCE achieves state-of-the-art performance, including emotion recognition, sleep staging, and cognitive assessment. Code is available at https://github.com/zhourunhe/EEG-MoCE.

Figures & tables

Figure 1: Euclidean vs. hyperbolic geometry for hierarchical data. Euclidean space is flat and tends to under-represent hierarchical branching; hyperbolic space exhibits exponential volume growth and better preserves tree-like separation. Hyperbolic geometry is informative for multimodal learning, where modalities may differ in how strongly hierarchical their underlying structure is.

Figure 2: Architecture of EEG-MoCE on EAV dataset. Other datasets use the same overall architecture, with only modality encoders adapted. (a) Modality-specific hyperbolic experts : each modality (e.g., EEG, audio, video) is encoded by an expert that embeds inputs in its own learnable-curvature hyperbolic space. (b) Curvature-oriented fusion : expert representations are aggregated by a curvature-aware scheme, combining curvature-scaled temperature and a curvature prior bias, that favors modalities with richer hierarchical structures.

Dataset	Modality	Raw	Encoded
EAV ( $n=42$ )	EEG	$0.097\pm 0.021$	$0.160\pm 0.013$
	Audio	$0.217\pm 0.023$	$0.293\pm 0.028$
	Video	$0.280\pm 0.019$	$0.278\pm 0.011$
ISRUC ( $n=10$ )	EEG	$0.107\pm 0.010$	$0.202\pm 0.019$
	EMG	$0.072\pm 0.007$	$0.202\pm 0.008$
	EOG	$0.137\pm 0.013$	$0.253\pm 0.028$

Table 1: Hyperbolicity

\delta_{\mathrm{rel}}

measured for raw data and encoded features following ( Khrulkov et al., 2020 ) . Lower values indicate stronger hierarchical tree-like structure and better suitability for hyperbolic embedding. All datasets’ modalities show hyperbolic structure with

\delta_{\text{rel}}<0.3

. Modality-wise differences motivate modality-specific learnable curvature.

Modality	Encoded $\delta_{\text{rel}}$	$K$	Contribution
EEG	0.160	-2.34	36.0
Vision	0.278	-2.29	33.6
Audio	0.293	-1.91	30.5

Table 2: Encoded

\delta_{\mathrm{rel}}

, learned curvature values

K

and attention-based modality contributions on EAV dataset, evaluated using a model with learnable curvatures but without curvature-oriented fusion. A negative association between

\delta_{\mathrm{rel}}

and

|K|

and a positive correlation between

|K|

and attention contribution support the curvature-oriented fusion hypothesis.

Dataset	$\lambda_{\text{init}}$	$\lambda_{\text{learned}}$	$\Delta\lambda$
EAV	0.300	$0.464\pm 0.036$	$+0.164$
ISRUC	0.300	$0.533\pm 0.034$	$+0.233$
Cognitive	0.300	$0.332\pm 0.009$	$+0.032$

Table 3: Learned curvature prior

\lambda

, initialized at 0.30 on full model with curvature-oriented fusion. Values are mean

\pm

std across cross-validation folds.

\lambda

consistently increases during training across datasets, indicating growing reliance on curvature information for attention weighting.

Method	Acc (%)	F1 (%)
MM-DFN	$51.13\pm 6.81$	$50.17\pm 6.95$
CTMWA	$52.86\pm 5.63$	$49.92\pm 6.33$
AGF-IB	$54.59\pm 12.45$	$54.20\pm 15.03$
LMF	$57.29\pm 12.37$	$56.80\pm 14.16$
CMERC	$58.57\pm 11.95$	$57.70\pm 14.30$
MMML	$59.60\pm 11.83$	$56.55\pm 14.31$

Table 4: Performance Comparison with State-of-the-Art Methods on EAV Emotion Recognition Dataset. We evaluate all methods using balanced accuracy (Acc) and macro-averaged F1-score (F1) under cross-subject 10-fold cross-validation protocol (

n=42

subjects).

Method	Acc (%)	F1 (%)
CMST	$66.68\pm 8.31$	$64.06\pm 10.32$
MMML	$73.33\pm 3.93$	$72.80\pm 4.55$
CFSNet	$74.30\pm 4.82$	$72.71\pm 5.31$
SSNet	$74.56\pm 4.10$	$74.13\pm 4.83$
MMNet	$74.73\pm 4.02$	$72.90\pm 5.45$
LMF	$75.02\pm 4.54$	$74.31\pm 5.63$

Table 5: Performance Comparison with State-of-the-Art Methods on ISRUC-S3 Sleep Staging Dataset. We evaluate all methods using balanced accuracy (Acc) and macro-averaged F1-score (F1) under leave-one-subject-out cross-validation protocol (

n=10

subjects).

Method	Acc (%)	F1 (%)
MMML	$45.32\pm 9.37$	$42.67\pm 11.81$
STA-Net	$46.63\pm 11.67$	$43.12\pm 15.32$
EFDFNet	$49.15\pm 12.85$	$45.00\pm 15.13$
CTMWA	$49.86\pm 11.72$	$45.71\pm 12.23$
ST2A	$51.28\pm 11.74$	$46.08\pm 13.17$
LMF	$52.71\pm 11.77$	$48.70\pm 13.04$

Table 6: Performance Comparison with State-of-the-Art Methods on Cognitive N-back Task. We evaluate all methods using balanced accuracy (Acc) and macro-averaged F1-score (F1) under cross-subject 10-fold cross-validation protocol (

n=26

subjects).

Encoder	Fusion	Acc (%)	F1 (%)
–	–	$60.33\pm 11.68$	$57.24\pm 14.18$
–	✓	$61.48\pm 12.63$	$58.79\pm 14.47$
✓	–	$74.17\pm 9.83$	$73.41\pm 10.27$
✓	✓ (Ours)	$\mathbf{75.88}\pm 8.31$	$\mathbf{75.47}\pm 8.66$

Table 7: Architecture ablation on EAV comparing Euclidean and hyperbolic choices for encoder and fusion. Four variants are shown: neither hyperbolic; hyperbolic fusion only; hyperbolic encoder only; both encoder and fusion. The combination of hyperbolic encoder and hyperbolic fusion produces the best result, indicating complementary benefits.

Panel of Figure 3 — Figure 3: t-SNE of fused features on EAV dataset. Hyperbolic encoder and fusion variant produces more compact and better-separated emotion clusters than the Euclidean baseline variant, illustrating improved class separability by hyperbolic geometry.

Figure 4: Ablation of hyperbolic components on the EAV dataset, focusing on learnable curvature and curvature-oriented multimodal fusion (COMF). All fusion variants use learnable curvatures to isolate the effect of the fusion mechanism. Our proposed method achieves the best performance, demonstrating the complementary benefits of learnable curvatures and COMF.

Method	Train sec/epoch	Test msec/trial
MMML	10.86	1.17
Hyper-MML	12.95	1.24
GA2MIF	8.56	0.86
Ours	26.07	2.71

Table 8: Computational cost on EAV measured on the same RTX 4090 GPU. Train reports seconds per epoch and Test reports milliseconds per trial. EEG-MoCE is slower due to hyperbolic operations, but its absolute inference latency remains low for real-time deployment settings.

Dataset	Method	Acc (%)	F1 (%)
EAV	Ours	$\mathbf{75.79}\pm 0.78$	$\mathbf{75.25}\pm 0.76$
EAV	HEEGNet	$61.54\pm 0.76$	$60.37\pm 0.77$
ISRUC	Ours	$\mathbf{78.42}\pm 0.31$	$\mathbf{75.10}\pm 0.29$
ISRUC	XSleepFusion	$75.02\pm 0.67$	$73.30\pm 0.71$
Cognitive	Ours	$\mathbf{61.86}\pm 1.75$	$\mathbf{59.23}\pm 1.64$
Cognitive	EF-Net	$54.28\pm 2.41$	$51.36\pm 2.35$

Table 9: Seed-level robustness over 5 random seeds under the same cross-subject protocol as the main experiments. Values are mean

\pm

std over seeds. Across EAV, ISRUC, and Cognitive, EEG-MoCE consistently exceeds the strongest baseline with low seed-level variance, supporting initialization robustness.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Modality	Acc (%)	F1 (%)
Video only	$53.75\pm 10.74$	$53.46\pm 11.02$
Audio only	$60.52\pm 8.14$	$60.23\pm 8.42$
EEG only	$62.74\pm 9.04$	$62.13\pm 9.42$
All modalities	$\mathbf{75.88}\pm 8.31$	$\mathbf{75.47}\pm 8.66$

Appendix

Table 10: Ablation study on single-modality performance on EAV dataset.

Dataset	Fusion	Acc (%)	F1 (%)
EAV	Mean	$\mathbf{75.88}\pm 8.31$	$\mathbf{75.47}\pm 8.66$
EAV	Learnable	$75.55\pm 8.57$	$75.10\pm 9.13$
ISRUC	Mean	$\mathbf{78.53}\pm 2.95$	$\mathbf{75.38}\pm 4.05$
ISRUC	Learnable	$78.08\pm 3.01$	$74.97\pm 3.96$
Cognitive	Mean	$\mathbf{62.39}\pm 13.07$	$\mathbf{59.67}\pm 13.08$
Cognitive	Learnable	$60.96\pm 13.48$	$57.68\pm 13.61$

Appendix

Table 11: Ablation of fusion-curvature parameterization in the shared manifold. Mean uses the arithmetic mean of modality curvatures, while Learnable treats the fusion curvature as an extra trainable parameter. Mean curvature performs comparably or slightly better across all datasets, supporting the stable and effective design choice.

Dataset	Mean $K$	Optimized $K$
EAV	$-2.2544\pm 0.1020$	$-2.2509\pm 0.1013$
ISRUC	$-1.3511\pm 0.0777$	$-1.3547\pm 0.0760$
Cognitive	$-1.9834\pm 0.0630$	$-1.9821\pm 0.0623$

Appendix

Table 12: Mean fusion curvature vs. optimization-based fusion curvature (Optimized

K

) derived using the objective in ( Wei et al., 2026 ) . Values are mean

\pm

std across folds. The close agreement across datasets supports the practical use of mean fusion curvature.

Method	Acc (%)	F1 (%)
MMML ( Wu et al., 2024 )	$56.35\pm 8.22$	$51.76\pm 12.07$
EFDFNet ( Xu et al., 2025 )	$58.85\pm 7.63$	$53.83\pm 11.13$
LMF ( Liu et al., 2018 )	$59.74\pm 6.18$	$59.16\pm 6.38$
CTMWA ( Zhang et al., 2024 )	$59.87\pm 8.15$	$58.68\pm 8.63$
STA-Net ( Liu et al., 2025 )	$62.50\pm 8.72$	$58.08\pm 12.65$
TSMMF ( Si et al., 2025 )	$63.82\pm 9.47$	$62.87\pm 10.36$

Appendix

Table 13: Performance comparison with state-of-the-art methods on the Cognitive dataset, Word Generation task (

n=26

). We report balanced accuracy (Acc) and macro-averaged F1 (F1) under cross-subject 10-fold cross-validation.

Multimodal Learning

Multimodal learning studies how models combine and reason over different data types, such as images, text, audio, signals, and structured records.

See all 335 papers

GeoMER: A Joint Euclidean-Hyperbolic Geometry Framework for Multimodal Emotion Recognition
Oct 13, 2026Ting Zhu, Huizhi Liang, Boguslaw Obara +2Multimodal Emotion Recognition Multimodal Fusion
Newcastle University, Newcastle upon Tyne, United Kingdom · Taiyuan University of Technology, Shanxi, China
Separating Diagnosis from Disease Representation: Dual-View EEG Learning with Neural-Dynamics-Guided Deformation
Sep 26, 2026Jiaying Wang, Shouqian Shi, Yutong Chen +5Electroencephalography EEG Decoding
Nanjing University · Tianjin University
Multiscale Temporal-Spectral-Spatial EEG Learning With Modality-Specific and Modality-Invariant Representations for Emotion Recognition
Oct 20, 2026Jian Zhang, Manrong Liu, Yilei Huang +3EEG Emotion Recognition Multimodal Emotion Recognition
Henan Engineering Research Center for Artificial Intelligence Theory and Algorithms, School of Mathematics and Statistics, Henan University, Kaifeng, 475004, China. · The Second Clinical College, Henan University, Kaifeng, 475004, China.

Oct 13, 2026J/K move · Enter open · S save

GeoMER: A Joint Euclidean-Hyperbolic Geometry Framework for Multimodal Emotion Recognition

Ting Zhu, Huizhi Liang, Boguslaw Obara, Shufei Duan+1

Newcastle University, Newcastle upon Tyne, United Kingdom · Taiyuan University of Technology, Shanxi, China

Multimodal Emotion Recognition Multimodal Fusion Geometric Representation Learning Hierarchical Representation Learning Hyperbolic Representation Learning

Open paper AskPDF arXiv

Multimodal Emotion Recognition (MER) aims to understand human emotions by integrating information from multiple modalities such as speech and text. Psychological studies suggest that emotions exhibit latent hierarchical structured relationships in their semantic representations. However, most MER approaches rely on Euclidean representations, which capture fine-grained cross-modal interactions but do not explicitly provide a geometric prior for organizing hierarchical emotional structures. To mitigate this gap, we propose GeoMER, a joint Euclidean-Hyperbolic modeling framework that introduces hyperbolic emotion-core routing to capture higher-level, abstract emotional representations for MER. The proposed framework jointly models token-level cross-modal semantic interactions and latent emotion representations in two complementary geometric spaces. Specifically, GeoMER consists of a Euclidean Multimodal Fusion Module (EuclFusion) for multimodal representation fusion and a Hyperbolic Geometric Core Routing (HypGeoRoute) module that performs geometry-aware core assignment and aggregation in a hyperbolic manifold to capture more abstract feature representations. The HypGeoRoute module complements the Euclidean token-level modeling. We experiment on the IEMOCAP multimodal MER benchmark and demonstrate that the proposed method achieves the best performance, with 83.54% weighted accuracy and 83.35% unweighted accuracy, outperforming Euclidean baselines, including MemoCMT.

Abstract

Multimodal Learning

Explore similar work

GeoMER: A Joint Euclidean-Hyperbolic Geometry Framework for Multimodal Emotion Recognition

Separating Diagnosis from Disease Representation: Dual-View EEG Learning with Neural-Dynamics-Guided Deformation

Multiscale Temporal-Spectral-Spatial EEG Learning With Modality-Specific and Modality-Invariant Representations for Emotion Recognition