Emotional expression underpins natural communication and effective human-computer interaction. We present Emotion Collider (EC-Net), a hyperbolic hypergraph framework for multimodal emotion and sentiment modeling. EC-Net represents modality hierarchies using Poincare-ball embeddings and performs fusion through a hypergraph mechanism that passes messages bidirectionally between nodes and hyperedges. To sharpen class separation, contrastive learning is formulated in hyperbolic space with decoupled radial and angular objectives. High-order semantic relations across time steps and modalities are preserved via adaptive hyperedge construction. Empirical results on standard multimodal emotion benchmarks show that EC-Net produces robust, semantically coherent representations and consistently improves accuracy, particularly when modalities are partially available or contaminated by noise. These findings indicate that explicit hierarchical geometry combined with hypergraph fusion is effective for resilient multimodal affect understanding.
Figures & tables
Figure 1: Overview of the Emotion Collider (EC-Net) architecture for sentiment recovery. The framework projects multimodal features ( xL,xA,xV ) into paired hyperbolic embeddings across the Emotion Manifold ( ME ) and the Anti-Emotion Manifold ( MA ), both modeled as Poincaré balls. A Differentiable Mirror Layer, realized as a learnable involution ( gϕ,fψ ), facilitates bidirectional mapping between the dual manifolds while enforcing geometric consistency through Riemannian importance re-weighting. For recovery of missing modalities, the system utilizes mirror-space implicit score matching ( sθ ) to reconstruct the emotion vector field V in the tangent space. Available and recovered embeddings are integrated via a SetTransformer-based fusion network to produce the final prediction y^ . Additionally, the geometric discrepancy between the paired manifolds is captured as an asymmetry deception cue ( sasym ) for auxiliary task enhancement.
Figure 2: Radar summary across six missing patterns and three metrics (Acc2 / F1 / MAE). EC-Net shows consistent advantage.
Figure 3: Training trajectories for principal losses (mean across three seeds). Task loss, reconstruction loss, property-alignment loss and involution loss all decline stably.
Figure 4: Stacked bar plot showing Acc2 drops for each ablation across FIX and MR regimes.
Figure 5: Histogram of principal angles θ(Σ,μ) after training (50 bins). The distribution concentrates near small angles (mean ≈3.8∘ ).
Figure 6: Single-factor hyperparameter scans showing Acc2 versus the swept factor. The default operating point is marked.
Figure 7: Heatmap of Acc2 as a function of curvature ratio and orthogonality penalty, revealing a stable plateau.
Figure 8: Training curve under 8 random seeds (shaded area = min/max envelope). The small envelope confirms stable convergence.
Figure 9: Two test samples with the highest geometric-asymmetry score sasym . Input modalities, EC-Net prediction, ground-truth label. Large sasym correlates with sentiment mismatch, illustrating the cue’s decision boundary.
Figure 10: Poincaré-disk scatter of randomly sampled emotion embeddings (1,000 points) colored by label; superimposed geodesics illustrate geometric separation.
Figure 11: Mirror-space t-SNE: left original hE , right mapped fψ(gϕ(hE)) ; gray lines connect corresponding points. Small cycle distances indicate good involution behaviour.
Figure 12: Corruption robustness: bars show performance under clean, light and heavy corruption conditions for several methods.
Figure 13: Peak GPU memory vs. Acc2 Pareto plot across batch sizes. EC-Net occupies a favourable memory-accuracy region.
Figure 14: Training curve with mean and 95% confidence band from three seeds. Low variance indicates stable training dynamics.
Method
CMU-MOSI
CMU-MOSEI
Acc7
Acc2
F1
MAE ↓
Corr
Acc7
Acc2
F1
MAE ↓
Corr
HyCon [ 3 ]
46.6
85.2
85.1
0.741
0.779
52.8
85.4
85.6
0.554
0.751
UniMSE [ 34 ]
48.7
86.9
86.4
0.691
0.809
54.4
87.5
87.5
0.523
0.773
ConFEDE [ 35 ]
42.3
85.5
85.5
0.742
0.782
54.9
85.8
85.8
0.522
0.780
MGCL [ 36 ]
49.3
86.7
86.7
0.685
0.707
53.9
86.4
86.4
0.535
0.772
HyDiscGAN [ 37 ]
43.2
86.7
86.3
0.749
0.782
54.4
86.3
86.2
0.533
0.761
Table 1: CMU-MOSI [ 31 ] and CMU-MOSEI [ 32 ] full-modality comparison. Metrics: Acc7 (%), Acc2 (%), F1 (%), MAE (lower better), Corr. Best entries are bold . Mean ± std over 3 runs.
Model
WA (%) ↑
UA (%) ↑
TwoStageFT [ 42 ]
74.9
76.1
AdaptiveMixup [ 43 ]
75.4
76.0
EmoAug [ 44 ]
72.7
73.8
MoMKE [ 45 ]
77.9
77.1
APIN [ 46 ]
77.8
78.2
IAM [ 47 ]
74.8
75.6
Table 2: IEMOCAP [ 33 ] full-modality comparison. WA: weighted accuracy; UA: unweighted accuracy. Best entries are bold . Mean ± std over 3 runs.
Available
GCNet [ 50 ]
IMDer [ 51 ]
MoMKE [ 45 ]
LNLN [ 52 ]
EUAR [ 53 ]
CIDer [ 54 ]
EC-Net (ours)
{t}
83.7/83.6/42.3
84.8/84.7/44.8
86.2/86.1/38.1
84.9/84.7/45.1
86.0/86.0/46.1
83.7/83.6/41.3
90.0/90.0/50.0
{v}
56.1/55.7/16.9
61.3/60.8/22.2
54.1/53.7/17.0
52.2/58.9/18.8
64.9/64.9/23.6
57.8/42.3/15.5
68.5/68.5/24.5
{a}
56.1/54.5/16.6
62.0/62.2/22.0
59.3/59.0/18.4
52.2/58.9/18.0
63.0/62.3/23.2
57.8/43.2/15.2
69.0/69.0/25.0
{t,v}
84.3/84.2/43.4
85.5/85.4/45.3
86.5/86.4/37.5
84.3/84.6/44.6
86.2/86.2/45.5
83.8/83.8/42.1
91.0/91.0/51.5
{t,a}
84.3/84.2/43.4
85.4/85.3/45.0
86.5/86.4/38.6
84.9/85.2/45.1
86.1/86.1/44.7
83.8/83.8/41.7
91.5/91.5/52.0
{v,a}
62.0/61.9/17.2
63.6/63.4/23.8
59.6/59.6/20.1
52.2/58.9/18.8
66.1/65.8/24.2
57.8/44.0/15.5
70.5/70.5/26.0
Table 3: Fixed missing-modality results on CMU-MOSI [ 31 ] . Each cell reports Acc2 / F1 / Acc7. Column ‘Available’ indicates which modalities are available (t: text, a: audio, v: visual). Best results are bold . Mean ± std over 3 runs.
Missing rate η
GCNet [ 50 ]
IMDer [ 51 ]
MoMKE [ 45 ]
LNLN [ 52 ]
EUAR [ 53 ]
CIDer [ 54 ]
EC-Net (ours)
0.1
82.4/82.2/41.9
83.3/83.2/43.0
83.6/83.6/35.5
81.1/82.0/42.0
84.1/84.1/43.8
81.1/79.6/39.4
90.0/90.0/48.5
0.2
79.6/79.3/38.9
80.9/80.8/40.7
80.7/80.7/33.7
78.0/79.5/39.5
81.9/81.9/41.5
78.5/75.6/36.7
87.5/87.5/46.5
0.3
76.7/76.5/35.9
78.5/78.4/38.4
77.8/77.7/31.9
74.8/77.0/36.9
79.8/79.7/39.2
76.0/71.6/34.0
85.5/85.5/44.5
0.4
73.6/73.2/33.0
76.0/75.9/36.0
74.7/74.6/29.8
71.6/74.4/34.3
77.4/77.3/36.9
73.4/67.4/31.2
83.0/83.0/41.5
0.5
70.4/69.9/30.1
73.5/73.4/33.6
71.6/71.4/27.8
68.4/71.8/31.7
75.1/74.9/34.7
70.8/63.3/28.5
80.5/80.5/38.8
0.6
67.3/66.7/27.2
71.0/70.9/31.2
68.5/68.3/25.8
65.2/69.2/29.0
72.8/72.6/32.5
68.2/59.1/25.8
78.0/78.0/36.5
Table 4: Varying global missing rate η on CMU-MOSI [ 31 ] . Each cell reports Acc2 / F1 / Acc7. Best results are bold . Mean ± std over 3 runs.
Variant
FIX (Acc2 / F1 / Acc7)
MR (Acc2 / F1 / Acc7)
w/o property embedding pathway
76.5 / 76.3 / 33.0
79.2 / 79.0 / 37.8
w/o reconstruction module
77.7 / 77.5 / 34.8
80.4 / 80.3 / 39.1
w/o mirror involution
78.0 / 77.8 / 35.1
80.6 / 80.5 / 39.3
w/o fusion module
79.1 / 79.0 / 36.3
81.5 / 81.4 / 40.4
w/o property pathway + reconstruction
75.4 / 75.2 / 32.2
78.1 / 77.9 / 36.1
w/o all modules
73.6 / 73.4 / 30.5
76.3 / 76.1 / 33.0
Table 5: Ablation study of EC-Net on CMU-MOSI [ 31 ] . Each entry reports Acc2 / F1 / Acc7 for FIX and MR regimes. Best results are bold .
Condition
FIX
MR
With corruption
79.4 / 79.4 / 37.4
80.7 / 80.7 / 39.7
Clean (no corruption)
80.1 / 80.1 / 38.3
81.4 / 81.4 / 40.6
Table 6: Robustness to synthetic corruption on CMU-MOSI [ 31 ] (Acc2 / F1 / Acc7). Best results are bold .
Hyperparameter
Range
Acc2 (%)
Default
Significance
Curvature ratio ( cE/cA )
[0.5, 2.0]
80.0–80.2
0.8
p<0.01
Orthogonal penalty ( λorth )
[0.05, 0.2]
79.8–80.3
0.1
p<0.01
Mask rate schedule
fixed vs anneal
−1.3%
anneal
p<0.01
EWMA decay
[0.95, 0.999]
79.9–80.2
0.99
p<0.01
Gradient clip norm
[1.0, 5.0]
80.0–80.1
1.0
p<0.01
Mirror layer depth
[1, 3]
79.7–80.2
2
p<0.01
Table 7: Hyperparameter sensitivity analysis of EC-Net on CMU-MOSI. Each parameter is varied while keeping others at default values. Best results are bold .
Test Condition
Δ Acc2
Peak Memory (GB)
Notes
Batch size 128 → 64
−0.4%
18 → 13.2
Minor performance change together with large memory reduction
Curvature ratio = 10
−6.2%
—
Gradient explosion observed, motivates ratio clipping in practice
Multimodal Emotion Recognition (MER) aims to understand human emotions by integrating information from multiple modalities such as speech and text. Psychological studies suggest that emotions exhibit latent hierarchical structured relationships in their semantic representations. However, most MER approaches rely on Euclidean representations, which capture fine-grained cross-modal interactions but do not explicitly provide a geometric prior for organizing hierarchical emotional structures. To mitigate this gap, we propose GeoMER, a joint Euclidean-Hyperbolic modeling framework that introduces hyperbolic emotion-core routing to capture higher-level, abstract emotional representations for MER. The proposed framework jointly models token-level cross-modal semantic interactions and latent emotion representations in two complementary geometric spaces. Specifically, GeoMER consists of a Euclidean Multimodal Fusion Module (EuclFusion) for multimodal representation fusion and a Hyperbolic Geometric Core Routing (HypGeoRoute) module that performs geometry-aware core assignment and aggregation in a hyperbolic manifold to capture more abstract feature representations. The HypGeoRoute module complements the Euclidean token-level modeling. We experiment on the IEMOCAP multimodal MER benchmark and demonstrate that the proposed method achieves the best performance, with 83.54% weighted accuracy and 83.35% unweighted accuracy, outperforming Euclidean baselines, including MemoCMT.
Ting Zhu, Huizhi Liang, Boguslaw Obara +2
Newcastle University, Newcastle upon Tyne, United Kingdom · Taiyuan University of Technology, Shanxi, China
Multimodal fusion must simultaneously refine modality-specific signals and model cross-modal interactions; two competing objectives typically entangled within the same operation. We propose SeRIn (Segregate, Refine, Integrate), a multimodal LM fusion scheme that enforces this separation as an architectural prior. Modality-specific representations evolve along isolated pathways, each refined against its respective encoder context, while a dedicated cross-modal pathway accumulates their joint evolution without contaminating unimodal streams. Full cross-modal interaction is deferred to a final prediction step - ablations confirm that structured interactions, not added capacity, drive the gains; gate analysis under visual corruption reveals emergent modality reweighting without explicit supervision. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI, improving all metrics on both benchmarks.
Alexios Filippakopoulos, Elias Kallioras, Nikolaos Xiros +2
National Technical University of Athens, Greece · Athena Research Center, Greece · University of Bern, Switzerland +2
Multimodal emotion recognition aims to integrate text, audio, and video sources to understand human affective states. Although multimodal large language models excel at multimodal reasoning, they typically treat emotion categories as independent labels, ignoring the rich hierarchical taxonomy of human psychology. Moreover, lacking external contextual knowledge makes them highly susceptible to over-interpreting noisy cues, further complicating fine-grained emotion classification. To address these issues, we propose HyperEmo-RAG, a retrieval-augmented generation framework that leverages a structured emotional knowledge base. Our framework introduces two key innovations. 1) Hierarchical hyperbolic grounding. Recognizing the inherent hierarchical tree structure of emotion taxonomies, we jointly embed hierarchical emotion labels and multimodal samples into a continuous hyperbolic space (Poincaré ball) and design a hierarchical beam-search deliberation process that progressively retrieves samples from coarse to fine-grained levels. 2) Structured evidence injection. Based on the retrieved evidence, we construct an evidence graph and inject the structured knowledge as explicit cognitive context into the LLM through a Tree-Aware Attention mechanism and an EmotionGraphFormer, preserving the integrity of graph-structured information. Experiments on multiple datasets demonstrate that HyperEmo-RAG significantly outperforms existing methods.
Zeheng Wang, Bo Zhao, Yijie Zhu +6
Great Bay University Guangdong, Dongguan, China · Tencent Youtu Lab Shanghai, China · CUHK, shenzhen Guangdong, Shenzhen, China