I2-KAN: Interpretable Intra- and Inter-Modal Multi-Pathway Representation Learning for Multimodal Sentiment Analysis
Authors: Xiaoting Xie, Feng Qiao, Xianwei Zhang, Hailong Chen, Kaiwen Zheng, Joemon M. Jose, Xuri Ge, Yujun Li
Organizations: School of Information Science and Engineering, Shandong University, Qingdao, Shandong, China · Department of Computer Science, University of Exeter, Exeter, UK · School of Computing Science, University of Glasgow, Glasgow, UK · School of Artificial Intelligence, Shandong University, Jinan, Shandong, China
Multimodal sentiment analysis (MSA) aims to infer sentiment by integrating linguistic, acoustic, and visual information. However, many fusion methods integrate modality-specific representations and cross-modal interactions into joint representations, making it difficult to determine whether predictive utility arises from a modality itself or from its interactions with others. More fundamentally, a modality's standalone predictive utility may not faithfully reflect its interaction-conditioned predictive utility. To address this issue, we propose I2-KAN, an interpretable multi-pathway framework that explicitly maintains intra-modal and pairwise inter-modal information in separate structural pathways. Each pathway is refined by a Task-Decoupled Linear-Gated Denoising Bottleneck (TDL-GDB) to learn compact and task-relevant representations, followed by structured Kolmogorov-Arnold Network fusion that retains explicit pathway associations and supports multi-level interpretation. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS v2 demonstrate strong overall performance. Ablation and interpretability analyses consistently show across the three benchmarks that low standalone predictive utility does not necessarily imply low interaction-conditioned predictive utility: a modality that appears weak in isolation can remain informative through pairwise interactions. These findings highlight the interaction-dependent nature of multimodal utility and suggest that modality utility should be characterized from both standalone and interaction-conditioned perspectives.
Figures & tables
Fig. 1: Motivation of I 2 -KAN. Standalone predictive utility does not necessarily reflect interaction-conditioned predictive utility: a modality that exhibits limited utility in isolation can still provide substantial predictive utility through pairwise interactions. By explicitly maintaining intra-modal and pairwise inter-modal information within a multi-pathway structure, I 2 -KAN supports three complementary levels of interpretation: where predictive sensitivity is concentrated across pathways, what compact features exhibit higher sensitivity within each pathway, and how those features are nonlinearly transformed toward the final decision.
Fig. 2: Overview of the proposed I 2 -KAN framework. Modality-specific features are first organized into three intra-modal and three pairwise inter-modal semantic pathways. TDL-GDB independently refines each pathway into compact task-adaptive representations, which are subsequently integrated by the Interaction-Aware KAN Fusion module for sentiment prediction and multi-level interpretation.
Model
CMU-MOSI
CMU-MOSEI
Acc7 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
Acc7 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
Graph-MFN [ 35 ]
34.4
80.2
80.1
0.939
0.656
51.9
84.0
83.8
0.569
0.725
MMIM [ 11 ]
45.0
85.1
85.0
0.738
0.781
53.1
85.1
85.0
0.547
0.752
UniMSE [ 36 ]
48.7
86.9
86.4
0.691
0.809
54.4
87.5
87.5
0.523
0.773
HGraph-CL [ 22 ]
–
86.2
86.2
0.717
0.799
–
85.9
85.8
0.527
0.769
HyCon [ 37 ]
46.6
85.2
85.1
0.741
0.779
52.8
85.4
85.6
0.554
0.751
TABLE I: Comparison With Baselines on CMU-MOSI and CMU-MOSEI.
Model
CH-SIMS v2
Acc3 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
EF-LSTM [ 46 ]
73.5
80.1
80.0
0.309
0.700
LF-DNN [ 47 ]
71.2
77.8
77.9
0.322
0.668
TFN [ 6 ]
70.9
78.1
78.1
0.322
0.662
LMF [ 7 ]
70.0
77.8
77.8
0.327
0.651
MFN [ 48 ]
72.7
79.4
79.4
0.301
0.712
TABLE II: Comparison With Baselines on the CH-SIMS v2 Dataset.
Interactions
CMU-MOSI
CMU-MOSEI
Acc7 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
Acc7 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
None
44.9
86.5
86.5
0.704
0.808
49.2
86.7
86.6
0.578
0.753
VA
47.4
86.2
86.2
0.699
0.811
52.6
87.7
87.7
0.539
0.763
TV
47.0
88.2
88.1
0.648
0.841
54.2
88.1
88.1
0.516
0.789
TA
47.8
88.7
88.6
0.631
0.856
54.8
87.1
87.0
0.523
0.784
TV+VA
48.1
89.4
89.4
0.628
0.856
54.2
87.7
87.7
0.521
0.787
TABLE III: Analysis of Pairwise Interaction Pathways on CMU-MOSI and CMU-MOSEI. All configurations retain the intra-modal pathways T , A , and V .
Interactions
CH-SIMS v2
Acc3 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
None
66.1
81.4
81.4
0.316
0.703
VA
70.4
83.8
83.8
0.300
0.736
TV
71.0
84.6
84.6
0.298
0.734
TA
70.4
84.3
84.3
0.301
0.738
TV+VA
71.0
84.6
84.5
0.308
0.727
TABLE IV: Analysis of Pairwise Interaction Pathways on CH-SIMS v2.
Fig. 3: Data-conditioned KAN functional-edge visualization on the three datasets. The left column of each subfigure contains the compressed features from the six intra- and inter-modal pathways, the middle column contains the KAN hidden nodes, and the right node denotes the final sentiment prediction. Edge width and opacity indicate the normalized functional-edge strength.
Fig. 4: Normalized masking-based predictive sensitivity of the six semantic pathways, obtained by masking all three compressed dimensions belonging to one pathway at a time.
Fig. 5: Normalized masking-based predictive sensitivity of individual compressed dimensions within each pathway.
Configuration
Acc7 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
w/o Pairwise Interactions
49.2
86.7
86.6
0.578
0.753
w/o TDL-GDB
53.8
85.6
85.6
0.525
0.778
w/o KAN
53.0
86.7
86.6
0.524
0.780
I 2 -KAN
54.4
88.2
88.1
0.515
0.798
TABLE V: Leave-one-component-out ablation results on CMU-MOSEI.
Multimodal fusion must simultaneously refine modality-specific signals and model cross-modal interactions; two competing objectives typically entangled within the same operation. We propose SeRIn (Segregate, Refine, Integrate), a multimodal LM fusion scheme that enforces this separation as an architectural prior. Modality-specific representations evolve along isolated pathways, each refined against its respective encoder context, while a dedicated cross-modal pathway accumulates their joint evolution without contaminating unimodal streams. Full cross-modal interaction is deferred to a final prediction step - ablations confirm that structured interactions, not added capacity, drive the gains; gate analysis under visual corruption reveals emergent modality reweighting without explicit supervision. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI, improving all metrics on both benchmarks.
Alexios Filippakopoulos, Elias Kallioras, Nikolaos Xiros +2
National Technical University of Athens, Greece · Athena Research Center, Greece · University of Bern, Switzerland +2
Multimodal sentiment analysis relies on language, visual, and acoustic cues, but utterance-level modality quality may vary due to occlusion, background noise, motion blur, or imperfect transcripts, causing conventional fusion to over-trust unreliable modalities. We propose MRUF, a reliability-aware fusion method that combines multi-granularity routing with uncertainty-aware calibration. MRUF summarizes sentiment-relevant representations, performs subspace- and modality-level routing, and supervises modality routing with leave-one-out error increases to estimate utterance-level modality importance. It further predicts modality-wise uncertainty and refines modality gates through inverse-variance reweighting, while modality-invariant contrastive alignment stabilizes the shared representation space. Experiments on CMU-MOSI and CMU-MOSEI under aligned and unaligned settings show consistent improvements over strong baselines, and mechanism analysis verifies that modalities with higher predicted uncertainty receive lower fusion weights.
Haoran Ma, Yinfeng Yu, Liejun Wang
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China. · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing. · Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center. +2
Multimodal Sentiment Analysis (MSA) integrates language, visual, and acoustic modalities to infer human sentiment. Most existing methods either focus on globally shared representations or modality-specific features, while overlooking signals that are shared only by certain modality pairs. This limits the expressiveness and discriminative power of multimodal representations. To address this limitation, we propose a Tri-Subspace Disentanglement (TSD) framework that explicitly factorizes features into three complementary subspaces: a common subspace capturing global consistency, submodally-shared subspaces modeling pairwise cross-modal synergies, and private subspaces preserving modality-specific cues. To keep these subspaces pure and independent, we introduce a decoupling supervisor together with structured regularization losses. We further design a Subspace-Aware Cross-Attention (SACA) fusion module that adaptively models and integrates information from the three subspaces to obtain richer and more robust representations. Experiments on CMU-MOSI and CMU-MOSEI demonstrate that TSD achieves state-of-the-art performance across all key metrics, reaching 0.691 MAE on CMU-MOSI and 54.9% ACC-7 on CMU-MOSEI, and also transfers well to multimodal intent recognition tasks. Ablation studies confirm that tri-subspace disentanglement and SACA jointly enhance the modeling of multi-granular cross-modal sentiment cues.
Chunlei Meng, Jiabin Luo, Zhenglin Yan +4
Fudan University · Peking University · University of Macau