I2-KAN: Interpretable Intra- and Inter-Modal Multi-Pathway Representation Learning for Multimodal Sentiment Analysis
Authors: Xiaoting Xie, Feng Qiao, Xianwei Zhang, Hailong Chen, Kaiwen Zheng, Joemon M. Jose, Xuri Ge, Yujun Li
Organizations: School of Information Science and Engineering, Shandong University, Qingdao, Shandong, China · Department of Computer Science, University of Exeter, Exeter, UK · School of Computing Science, University of Glasgow, Glasgow, UK · School of Artificial Intelligence, Shandong University, Jinan, Shandong, China
Multimodal sentiment analysis (MSA) aims to infer sentiment by integrating linguistic, acoustic, and visual information. However, many fusion methods integrate modality-specific representations and cross-modal interactions into joint representations, making it difficult to determine whether predictive utility arises from a modality itself or from its interactions with others. More fundamentally, a modality's standalone predictive utility may not faithfully reflect its interaction-conditioned predictive utility. To address this issue, we propose I2-KAN, an interpretable multi-pathway framework that explicitly maintains intra-modal and pairwise inter-modal information in separate structural pathways. Each pathway is refined by a Task-Decoupled Linear-Gated Denoising Bottleneck (TDL-GDB) to learn compact and task-relevant representations, followed by structured Kolmogorov-Arnold Network fusion that retains explicit pathway associations and supports multi-level interpretation. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS v2 demonstrate strong overall performance. Ablation and interpretability analyses consistently show across the three benchmarks that low standalone predictive utility does not necessarily imply low interaction-conditioned predictive utility: a modality that appears weak in isolation can remain informative through pairwise interactions. These findings highlight the interaction-dependent nature of multimodal utility and suggest that modality utility should be characterized from both standalone and interaction-conditioned perspectives.
Figures & tables
Fig. 1: Motivation of I 2 -KAN. Standalone predictive utility does not necessarily reflect interaction-conditioned predictive utility: a modality that exhibits limited utility in isolation can still provide substantial predictive utility through pairwise interactions. By explicitly maintaining intra-modal and pairwise inter-modal information within a multi-pathway structure, I 2 -KAN supports three complementary levels of interpretation: where predictive sensitivity is concentrated across pathways, what compact features exhibit higher sensitivity within each pathway, and how those features are nonlinearly transformed toward the final decision.
Fig. 2: Overview of the proposed I 2 -KAN framework. Modality-specific features are first organized into three intra-modal and three pairwise inter-modal semantic pathways. TDL-GDB independently refines each pathway into compact task-adaptive representations, which are subsequently integrated by the Interaction-Aware KAN Fusion module for sentiment prediction and multi-level interpretation.
Model
CMU-MOSI
CMU-MOSEI
Acc7 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
Acc7 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
Graph-MFN [ 35 ]
34.4
80.2
80.1
0.939
0.656
51.9
84.0
83.8
0.569
0.725
MMIM [ 11 ]
45.0
85.1
85.0
0.738
0.781
53.1
85.1
85.0
0.547
0.752
UniMSE [ 36 ]
48.7
86.9
86.4
0.691
0.809
54.4
87.5
87.5
0.523
0.773
HGraph-CL [ 22 ]
–
86.2
86.2
0.717
0.799
–
85.9
85.8
0.527
0.769
HyCon [ 37 ]
46.6
85.2
85.1
0.741
0.779
52.8
85.4
85.6
0.554
0.751
TABLE I: Comparison With Baselines on CMU-MOSI and CMU-MOSEI.
Model
CH-SIMS v2
Acc3 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
EF-LSTM [ 46 ]
73.5
80.1
80.0
0.309
0.700
LF-DNN [ 47 ]
71.2
77.8
77.9
0.322
0.668
TFN [ 6 ]
70.9
78.1
78.1
0.322
0.662
LMF [ 7 ]
70.0
77.8
77.8
0.327
0.651
MFN [ 48 ]
72.7
79.4
79.4
0.301
0.712
TABLE II: Comparison With Baselines on the CH-SIMS v2 Dataset.
Interactions
CMU-MOSI
CMU-MOSEI
Acc7 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
Acc7 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
None
44.9
86.5
86.5
0.704
0.808
49.2
86.7
86.6
0.578
0.753
VA
47.4
86.2
86.2
0.699
0.811
52.6
87.7
87.7
0.539
0.763
TV
47.0
88.2
88.1
0.648
0.841
54.2
88.1
88.1
0.516
0.789
TA
47.8
88.7
88.6
0.631
0.856
54.8
87.1
87.0
0.523
0.784
TV+VA
48.1
89.4
89.4
0.628
0.856
54.2
87.7
87.7
0.521
0.787
TABLE III: Analysis of Pairwise Interaction Pathways on CMU-MOSI and CMU-MOSEI. All configurations retain the intra-modal pathways T , A , and V .
Interactions
CH-SIMS v2
Acc3 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
None
66.1
81.4
81.4
0.316
0.703
VA
70.4
83.8
83.8
0.300
0.736
TV
71.0
84.6
84.6
0.298
0.734
TA
70.4
84.3
84.3
0.301
0.738
TV+VA
71.0
84.6
84.5
0.308
0.727
TABLE IV: Analysis of Pairwise Interaction Pathways on CH-SIMS v2.
Fig. 3: Data-conditioned KAN functional-edge visualization on the three datasets. The left column of each subfigure contains the compressed features from the six intra- and inter-modal pathways, the middle column contains the KAN hidden nodes, and the right node denotes the final sentiment prediction. Edge width and opacity indicate the normalized functional-edge strength.
Fig. 4: Normalized masking-based predictive sensitivity of the six semantic pathways, obtained by masking all three compressed dimensions belonging to one pathway at a time.
Fig. 5: Normalized masking-based predictive sensitivity of individual compressed dimensions within each pathway.
Configuration
Acc7 ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
w/o Pairwise Interactions
49.2
86.7
86.6
0.578
0.753
w/o TDL-GDB
53.8
85.6
85.6
0.525
0.778
w/o KAN
53.0
86.7
86.6
0.524
0.780
I 2 -KAN
54.4
88.2
88.1
0.515
0.798
TABLE V: Leave-one-component-out ablation results on CMU-MOSEI.
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China. · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing. · Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center. +2