Multimodal Emotion Recognition (MER) aims to understand human emotions by integrating information from multiple modalities such as speech and text. Psychological studies suggest that emotions exhibit latent hierarchical structured relationships in their semantic representations. However, most MER approaches rely on Euclidean representations, which capture fine-grained cross-modal interactions but do not explicitly provide a geometric prior for organizing hierarchical emotional structures. To mitigate this gap, we propose GeoMER, a joint Euclidean-Hyperbolic modeling framework that introduces hyperbolic emotion-core routing to capture higher-level, abstract emotional representations for MER. The proposed framework jointly models token-level cross-modal semantic interactions and latent emotion representations in two complementary geometric spaces. Specifically, GeoMER consists of a Euclidean Multimodal Fusion Module (EuclFusion) for multimodal representation fusion and a Hyperbolic Geometric Core Routing (HypGeoRoute) module that performs geometry-aware core assignment and aggregation in a hyperbolic manifold to capture more abstract feature representations. The HypGeoRoute module complements the Euclidean token-level modeling. We experiment on the IEMOCAP multimodal MER benchmark and demonstrate that the proposed method achieves the best performance, with 83.54% weighted accuracy and 83.35% unweighted accuracy, outperforming Euclidean baselines, including MemoCMT.
Figures & tables
Figure 1: Overview of the GeoMER framework. Text and audio features extracted by pre-trained encoders are first aligned by a Euclidean adaptor. EuclFusion models multimodal interactions in Euclidean space, while HypGeoRoute captures abstract emotional features via geometric core routing in a hyperbolic manifold. The resulting hyperbolic representations are mapped back to Euclidean space and concatenated with Euclidean features for the final emotion classification.
Emotion
Train
Val
Test
Total
#Utterances
(%)
#Utterances
(%)
#Utterances
(%)
#Utterances
(%)
Angry
902
20.1
99
19.9
102
18.4
1103
19.9
Happy
1321
29.5
144
28.9
171
30.9
1636
29.6
Sad
857
19.1
98
19.7
129
23.3
1084
19.6
Neutral
1399
31.2
157
31.5
152
27.4
1708
30.9
Total
4479
100
498
100
554
100
5531
100
Table 1: Emotion distribution of IEMOCAP under the four-class setting. Percentages are computed within each split.
Model
WA (%)
UA (%)
Audio-only Modality
CNN
65.30
64.20
LSTM
66.80
65.90
MemoCMT (Audio-only)
66.56
65.34
GeoMER (Audio-only)
70.80
70.36
Text-only Modality
Table 2: Comparison with representative multimodal methods on IEMOCAP (4-class). Results marked with † are reproduced under our experimental setting and reported as mean ± standard deviation over five runs; other results are taken from the corresponding published papers. WA and UA indicated the weighted and unweighted accuracies.
Model
WA (%)
UA (%)
Macro-F1 (%)
Δ WA/ Δ UA
MemoCMT(reproduced)
81.05
81.33
80.84
–
+EuclFusion
81.95
82.38
81.77
+0.90/+1.05
+Hyperbolic mean pooling
82.16
82.53
81.95
+1.11/+1.20
+Hyp self-attention (w/o Routing)
82.51
82.99
82.31
+1.46/+1.66
+Hyp cross-attention (w/o Routing)
82.54
82.60
82.42
+1.49/+1.27
GeoMER (+HypGeoRoute) (Ours)
83.54
83.35
83.17
+2.49/+2.02
Table 3: Ablation study of different components on IEMOCAP (4-class). WA, UA, and Macro-F1 denote weighted accuracy, unweighted accuracy, and macro-averaged F1-score, respectively. Δ WA/ Δ UA reports the absolute gain over the reproduced MemoCMT baseline.
Figure 2: Per-class performance analysis of GeoMER on the IEMOCAP benchmark. (a) Confusion matrix, where rows denote ground-truth labels and columns denote predicted labels. The values indicate the classification percentage for each emotion category. (b) Precision–recall distribution of each emotion category.
Figure 3: Effect of the number of emotion cores Nc in HypGeoRoute on weighted accuracy(%). Performance is best when Nc=4 , but larger values also yield good results. This suggests that the learned emotion cores are not restricted to a strict one-to-one mapping to emotion classes, but instead serve as latent affective anchors for flexible representation learning.