Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis

Menghua Jiang; Haokai Gao; Xiangui Kang; Haifeng Hu; Sijie Mai

cs.LGSep 24, 2026

Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis

Authors: Menghua Jiang, Haokai Gao, Xiangui Kang, Haifeng Hu, Sijie Mai

Organizations: School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Science, South China Normal University · School of Electronics and Information Technology, Sun Yat-sen University

Abstract

Multimodal Sentiment Analysis (MSA) aims to infer human emotions from multiple modalities such as text, audio, and vision. In practice, inputs are often corrupted by noise and missing modalities, which degrades performance. Existing methods typically address these challenges in isolation, limiting their effectiveness in realistic settings. To address this limitation, we propose a Reliability-aware Cross-sample Enhancement (RCE) framework. Specifically, RCE first introduces an adaptive variational information bottleneck to model modality-wise uncertainty and perform quality-aware information compression, thereby suppressing redundant noise in unreliable modalities. Furthermore, we design a reliability-aware cross-sample enhancement strategy that retrieves high-confidence, semantically consistent neighbors from a large candidate pool to enrich and calibrate current representations, effectively alleviating information deficiency caused by missing modalities. Building upon this, RCE integrates cross-modal interactions with a multilevel reliability-aware fusion mechanism to adaptively aggregate information across modalities and enhancement stages, leading to more robust multimodal representations. Extensive experiments demonstrate that RCE consistently outperforms state-of-the-art methods across full, noisy, and missing-modality settings.

Figures & tables

Figure 1: Overview of the RCE framework. For clarity, we provide detailed descriptions and definitions of all notations used in the figure in Appendix A .

Method	CMU-MOSI					CMU-MOSEI
Method	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
MODS Yang et al. (2026)	49.27	85.83	85.96	0.688	0.798	54.32	85.88	86.14	0.527	0.772
PSA-MF Xie et al. (2026)	46.50	86.43	86.19	0.686	0.807	55.00	86.30	86.28	0.521	0.774
DecAlign Qian et al. (2026)	45.07	85.75	85.82	0.735	0.811	55.02	86.48	86.07	0.543	0.768
C-MIB Mai et al. (2023b)	47.01	87.33	87.28	0.650	0.836	52.47	86.08	86.05	0.551	0.773
ITHP Xiao et al. (2024)	46.57	88.40	88.40	0.650	0.848	52.44	86.80	86.83	0.529	0.789

Table 1: Comparison on CMU-MOSI and CMU-MOSEI. Best and Second results are highlighted.

Figure 2: Comparison on CMU-MOSEI under the random missing protocol.

Dataset	RCE	w/o AVIB	w/o RME	w/o MB	w/o CP	w/o HMG	w/o RF
CMU-MOSI	50.22 / 0.589	48.03 / 0.646	50.36 / 0.610	48.03 / 0.616	47.30 / 0.610	46.72 / 0.637	48.47 / 0.626
CMU-MOSEI	55.44 / 0.503	54.60 / 0.518	54.10 / 0.519	53.50 / 0.511	55.14 / 0.504	54.77 / 0.506	55.52 / 0.507

Table 2: Ablation results of RCE components on CMU-MOSI and CMU-MOSEI. We report Acc7 and MAE . Full results are provided in Appendix E.3 .

Dataset	Method	Text	Audio	Vision
CMU-MOSI	HME	46.29%	46.90%	46.90%
CMU-MOSI	RCE	2.60%	5.90%	6.48%
CMU-MOSEI	HME	38.40%	38.87%	38.87%
CMU-MOSEI	RCE	10.57%	31.25%	26.08%

Table 3: Training negative enhancement rates on CMU-MOSI and CMU-MOSEI.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Definition
$X_{m}^{i}$	Input feature sequence of modality $m$ for sample $i$ .
$L_{m}$	Sequence length of modality $m$ .
$d_{m}$	Original feature dimension of modality $m$ .
$d$	Shared embedding dimension across all modalities.
$\hat{X}_{m}^{i}$	Projected modality feature after 1D convolution.
$S_{m}^{i}$	Sequence representation after modality-specific Transformer encoder.

Appendix

Table 4: Notation and Definitions

Hyper-parameter	CMU-MOSI	CMU-MOSEI	UR-FUNNY	MUStARD
Batch Size	48	128	256	32
Epochs	50	10	10	20
Warm-up	✓	✓	✓	✓
Initial Learning Rate	$2\times 10^{-5}$	$3\times 10^{-5}$	$1\times 10^{-5}$	$3\times 10^{-6}$
Optimizer	AdamW	AdamW	AdamW	AdamW
Dropout Rate	0.4	0.4	0.1	0.1

Appendix

Table 5: Hyper-parameter settings of RCE across different datasets.

Testing Condition: { t }
Method	CMU-MOSI					CMU-MOSEI
Method	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
C-MIB [ 33 ]	45.69	87.33	87.28	0.644	0.842	51.76	85.66	85.62	0.550	0.771
ITHP [ 51 ]	43.50	86.72	86.71	0.678	0.838	52.40	86.96	86.94	0.541	0.791
KAN-MCP [ 30 ]	45.69	88.24	88.21	0.635	0.851	53.87	86.80	86.80	0.531	0.774
OMIB [ 49 ]	49.49	87.63	87.51	0.632	0.844	53.37	87.18	87.14	0.529	0.786

Appendix

Table 6: Performance comparison of RCE and baseline methods under fixed missing protocol.

Method	CMU-MOSI					CMU-MOSEI
Method	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
Gaussian Noise Intensity: 1.0
C-MIB [ 33 ]	42.34	85.65	85.68	0.750	0.782	51.07	84.50	84.65	0.560	0.766
ITHP [ 51 ]	45.55	86.11	86.10	0.697	0.822	52.10	84.03	84.27	0.564	0.781
KAN-MCP [ 30 ]	45.99	85.80	85.76	0.710	0.805	52.04	86.88	86.84	0.543	0.775
OMIB [ 49 ]	42.34	83.51	83.51	0.751	0.785	50.36	84.42	84.63	0.562	0.770

Appendix

Table 7: Comparison under different Gaussian noise intensities on CMU-MOSI and CMU-MOSEI.

Method	CMU-MOSI					CMU-MOSEI
Method	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
Salt-pepper Noise Intensity: 1.0
C-MIB	42.04	85.80	85.80	0.751	0.786	51.93	85.52	85.41	0.555	0.765
ITHP	15.47	42.29	25.14	1.471	-0.030	52.49	85.75	85.71	0.552	0.777
KAN-MCP	48.47	87.02	87.02	0.657	0.835	52.98	86.85	86.90	0.531	0.777
OMIB	37.52	78.32	78.43	0.903	0.691	53.24	86.16	86.24	0.533	0.776

Appendix

Table 8: Comparison under different Salt-pepper noise intensities on CMU-MOSI and CMU-MOSEI.

Method	CMU-MOSI					CMU-MOSEI
Method	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
w/o AVIB	48.03	87.79	87.77	0.646	0.842	54.60	86.35	86.47	0.518	0.785
w/o RME	50.36	88.09	88.02	0.610	0.857	54.10	87.04	87.04	0.519	0.794
w/o MB	48.03	88.24	88.20	0.616	0.853	53.50	87.18	87.15	0.511	0.794
w/o LP	49.34	88.70	88.62	0.603	0.859	56.06	86.60	86.67	0.505	0.795
w/o CP	47.30	87.94	87.81	0.610	0.856	55.14	87.18	87.17	0.504	0.798

Appendix

Table 9: Ablation details.

Dataset	Method	Modality	Samples	Negative Enhancement Samples	Negative Enhancement Rate
CMU-MOSI	HME	Text	61,400	28,420	46.29%
		Audio	61,400	28,797	46.90%
		Vision	61,400	28,797	46.90%
	RCE	Text	61,400	1,595	2.60%
		Audio	61,400	3,621	5.90%
		Vision	61,400	3,976	6.48%

Appendix

Table 10: Detailed training-stage negative enhancement statistics on CMU-MOSI and CMU-MOSEI.

Panel of Figure 5 — Figure 5: Parameter sensitivity analysis of $\beta$ and $N_{b}$ on the CMU-MOSI dataset.

Panel of Figure 6 — Figure 6: Parameter sensitivity analysis of $\alpha$ , $\lambda$ , and $\rho$ on the CMU-MOSI dataset.

Panel of Figure 7 — Figure 7: Parameter sensitivity analysis of $\beta$ and $N_{b}$ on the CMU-MOSEI dataset.

Panel of Figure 8 — Figure 8: Parameter sensitivity analysis of $\alpha$ , $\lambda$ , and $\rho$ on the CMU-MOSEI dataset.

Dataset	$Acc7\%$	$Acc2\%$	$F1\%$	$MAE$	$Corr$
CMU-MOSI	$49.81\pm 0.56$	$88.98\pm 0.97$	$88.93\pm 0.97$	$0.598\pm 0.009$	$0.859\pm 0.004$
CMU-MOSEI	$55.14\pm 0.37$	$87.25\pm 0.35$	$87.26\pm 0.31$	$0.510\pm 0.006$	$0.792\pm 0.006$

Appendix

Table 11: Testing stability with five random seeds under the complete-modality setting (95% confidence intervals).

Method	FLOPs	Memory	Parameters	Per-Epoch Training Time
C-MIB [ 33 ]	11.773 G	8.48 GB	190.93 M	2.93 s
ITHP [ 51 ]	11.523 G	8.36 GB	184.88 M	3.43 s
KAN-MCP [ 30 ]	11.501 G	13.36 GB	184.76 M	10.62 s
OMIB [ 49 ]	11.615 G	8.10 GB	189.05 M	3.70 s
QMF [ 65 ]	12.529 G	8.41 GB	196.47 M	4.09 s
PML [ 14 ]	12.144 G	8.42 GB	188.27 M	3.99 s

Appendix

Table 12: Comparison of model complexity on the CMU-MOSI dataset with batch size 48.

Explore similar work

Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
Apr 14, 2026Kang He, Yuzhe Ding, Xinrong Wang +3Multimodal Sentiment Analysis Modality Imbalance
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University. · Shanghai Innovation Institute
SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data
Sep 24, 2026Wenhao Li, Zhibin Wu, Chong Xiao +1Missing-Modality Learning Multimodal Robustness
Software School, Shandong University · Shenzhen Loop Area Institute
Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities
Mar 10, 2026Jindi Bao, Jianjun Qian, Mengkai Yan +1Multimodal Sentiment Analysis Missing-Modality Learning
Nanjing University of Science and Technology, Nanjing 210094, China · Hohai University, Nanjing 211100, China

Apr 14, 2026J/K move · Enter open · S save

Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis

Kang He, Yuzhe Ding, Xinrong Wang, Fei Li+2

Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University. · Shanghai Innovation Institute

Multimodal Sentiment Analysis Modality Imbalance Multimodal Fusion Multimodal Robustness

Open paper AskPDF arXiv

Multimodal sentiment analysis (MSA) integrates heterogeneous text, audio, and visual signals to infer human emotions. While recent approaches leverage cross-modal complementarity, they often struggle to fully utilize weaker modalities. In practice, dominant modalities tend to overshadow non-verbal ones, inducing modality competition and limiting overall contributions. This imbalance degrades fusion performance and robustness under noisy or missing modalities. To address this, we propose a novel model, Enhance-then-Balance Modality Collaboration framework (EBMC). EBMC improves representation quality via semantic disentanglement and cross-modal enhancement, strengthening weaker modalities. To prevent dominant modalities from overwhelming others, an Energy-guided Modality Coordination mechanism achieves implicit gradient rebalancing via a differentiable equilibrium objective. Furthermore, Instance-aware Modality Trust Distillation estimates sample-level reliability to adaptively modulate fusion weights, ensuring robustness. Extensive experiments demonstrate that EBMC achieves state-of-the-art or competitive results and maintains strong performance under missing-modality settings.

Abstract

Explore similar work

Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities