Multimodal Emotion Recognition (MER) aims to understand human emotions by integrating information from multiple modalities such as speech and text. Psychological studies suggest that emotions exhibit latent hierarchical structured relationships in their semantic representations. However, most MER approaches rely on Euclidean representations, which capture fine-grained cross-modal interactions but do not explicitly provide a geometric prior for organizing hierarchical emotional structures. To mitigate this gap, we propose GeoMER, a joint Euclidean-Hyperbolic modeling framework that introduces hyperbolic emotion-core routing to capture higher-level, abstract emotional representations for MER. The proposed framework jointly models token-level cross-modal semantic interactions and latent emotion representations in two complementary geometric spaces. Specifically, GeoMER consists of a Euclidean Multimodal Fusion Module (EuclFusion) for multimodal representation fusion and a Hyperbolic Geometric Core Routing (HypGeoRoute) module that performs geometry-aware core assignment and aggregation in a hyperbolic manifold to capture more abstract feature representations. The HypGeoRoute module complements the Euclidean token-level modeling. We experiment on the IEMOCAP multimodal MER benchmark and demonstrate that the proposed method achieves the best performance, with 83.54% weighted accuracy and 83.35% unweighted accuracy, outperforming Euclidean baselines, including MemoCMT.
Figures & tables
Figure 1: Overview of the GeoMER framework. Text and audio features extracted by pre-trained encoders are first aligned by a Euclidean adaptor. EuclFusion models multimodal interactions in Euclidean space, while HypGeoRoute captures abstract emotional features via geometric core routing in a hyperbolic manifold. The resulting hyperbolic representations are mapped back to Euclidean space and concatenated with Euclidean features for the final emotion classification.
Emotion
Train
Val
Test
Total
#Utterances
(%)
#Utterances
(%)
#Utterances
(%)
#Utterances
(%)
Angry
902
20.1
99
19.9
102
18.4
1103
19.9
Happy
1321
29.5
144
28.9
171
30.9
1636
29.6
Sad
857
19.1
98
19.7
129
23.3
1084
19.6
Neutral
1399
31.2
157
31.5
152
27.4
1708
30.9
Total
4479
100
498
100
554
100
5531
100
Table 1: Emotion distribution of IEMOCAP under the four-class setting. Percentages are computed within each split.
Model
WA (%)
UA (%)
Audio-only Modality
CNN
65.30
64.20
LSTM
66.80
65.90
MemoCMT (Audio-only)
66.56
65.34
GeoMER (Audio-only)
70.80
70.36
Text-only Modality
Table 2: Comparison with representative multimodal methods on IEMOCAP (4-class). Results marked with † are reproduced under our experimental setting and reported as mean ± standard deviation over five runs; other results are taken from the corresponding published papers. WA and UA indicated the weighted and unweighted accuracies.
Model
WA (%)
UA (%)
Macro-F1 (%)
Δ WA/ Δ UA
MemoCMT(reproduced)
81.05
81.33
80.84
–
+EuclFusion
81.95
82.38
81.77
+0.90/+1.05
+Hyperbolic mean pooling
82.16
82.53
81.95
+1.11/+1.20
+Hyp self-attention (w/o Routing)
82.51
82.99
82.31
+1.46/+1.66
+Hyp cross-attention (w/o Routing)
82.54
82.60
82.42
+1.49/+1.27
GeoMER (+HypGeoRoute) (Ours)
83.54
83.35
83.17
+2.49/+2.02
Table 3: Ablation study of different components on IEMOCAP (4-class). WA, UA, and Macro-F1 denote weighted accuracy, unweighted accuracy, and macro-averaged F1-score, respectively. Δ WA/ Δ UA reports the absolute gain over the reproduced MemoCMT baseline.
Figure 2: Per-class performance analysis of GeoMER on the IEMOCAP benchmark. (a) Confusion matrix, where rows denote ground-truth labels and columns denote predicted labels. The values indicate the classification percentage for each emotion category. (b) Precision–recall distribution of each emotion category.
Figure 3: Effect of the number of emotion cores Nc in HypGeoRoute on weighted accuracy(%). Performance is best when Nc=4 , but larger values also yield good results. This suggests that the learned emotion cores are not restricted to a strict one-to-one mapping to emotion classes, but instead serve as latent affective anchors for flexible representation learning.
Emotional expression underpins natural communication and effective human-computer interaction. We present Emotion Collider (EC-Net), a hyperbolic hypergraph framework for multimodal emotion and sentiment modeling. EC-Net represents modality hierarchies using Poincare-ball embeddings and performs fusion through a hypergraph mechanism that passes messages bidirectionally between nodes and hyperedges. To sharpen class separation, contrastive learning is formulated in hyperbolic space with decoupled radial and angular objectives. High-order semantic relations across time steps and modalities are preserved via adaptive hyperedge construction. Empirical results on standard multimodal emotion benchmarks show that EC-Net produces robust, semantically coherent representations and consistently improves accuracy, particularly when modalities are partially available or contaminated by noise. These findings indicate that explicit hierarchical geometry combined with hypergraph fusion is effective for resilient multimodal affect understanding.
Multimodal emotion recognition aims to integrate text, audio, and video sources to understand human affective states. Although multimodal large language models excel at multimodal reasoning, they typically treat emotion categories as independent labels, ignoring the rich hierarchical taxonomy of human psychology. Moreover, lacking external contextual knowledge makes them highly susceptible to over-interpreting noisy cues, further complicating fine-grained emotion classification. To address these issues, we propose HyperEmo-RAG, a retrieval-augmented generation framework that leverages a structured emotional knowledge base. Our framework introduces two key innovations. 1) Hierarchical hyperbolic grounding. Recognizing the inherent hierarchical tree structure of emotion taxonomies, we jointly embed hierarchical emotion labels and multimodal samples into a continuous hyperbolic space (Poincaré ball) and design a hierarchical beam-search deliberation process that progressively retrieves samples from coarse to fine-grained levels. 2) Structured evidence injection. Based on the retrieved evidence, we construct an evidence graph and inject the structured knowledge as explicit cognitive context into the LLM through a Tree-Aware Attention mechanism and an EmotionGraphFormer, preserving the integrity of graph-structured information. Experiments on multiple datasets demonstrate that HyperEmo-RAG significantly outperforms existing methods.
Zeheng Wang, Bo Zhao, Yijie Zhu +6
Great Bay University Guangdong, Dongguan, China · Tencent Youtu Lab Shanghai, China · CUHK, shenzhen Guangdong, Shenzhen, China
Emotion Recognition in Conversations (ERC) requires integrating heterogeneous textual, audio, and visual cues while accounting for conversational context and emotional dynamics. We extend the Self-Distillation Transformer architecture for ERC with appearance+geometry visual representations, class-wise adaptive modality fusion, and a valence-arousal prior for affective transitions. On the MELD and IEMOCAP datasets, geometry-enhanced visual representations improve weighted F1 by 0.27 and 4.36 points over appearance-only features, respectively, while class-wise adaptive fusion provides further gains of 0.17 and 0.25 points over the original softmax gate. The valence-arousal prior yields targeted improvements of 0.30 and 0.74 accuracy points on emotionally shifted utterances while preserving performance on stable turns. These results indicate that structured facial cues, emotion-dependent modality weighting, and affective geometry provide complementary benefits for multimodal ERC.