Uncertainty-Guided Multimodal Consensus for Multimodal Sentiment Analysis

Guowei Zhong; Junjie Li; Huaiyu Zhu; Ruohong Huan; Yun Pan

cs.CVOct 23, 2025

Uncertainty-Guided Multimodal Consensus for Multimodal Sentiment Analysis

Authors: Guowei Zhong, Junjie Li, Huaiyu Zhu, Ruohong Huan, Yun Pan

Organizations: College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, 310027, China · Zhejiang University Jinhua Research Institute, Jinhua, 321000, China · College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, 310023, China

Abstract

Multimodal sentiment analysis (MSA) has made considerable progress in recent years. Existing MSA methods mainly learn shared multimodal representations or train unimodal predictors with a common multimodal label. However, assigning the shared label to every modality can obscure modality-specific sentiment, while direct fusion may overemphasize unreliable predictions when modalities disagree. To address these issues, we propose Uncertainty-Guided Multimodal Consensus (UMC), which first recovers continuous unimodal pseudo-labels and then forms a reliability-aware consensus across modalities. Our method consists of two key components: i) Continuous Prototype-Guided Label Recovery (CPLR), which combines heteroscedastic predictions with ordered sentiment prototypes and uses confidence and label conflict to refine continuous unimodal pseudo-labels, and ii) Reliability and Precision Consensus (RPC), which estimates sample-dependent modality reliability from predictive statistics and cross-modal disagreement and combines it with predictive precision through a closed-form consensus solver. Extensive experiments conducted on four publicly available MSA datasets demonstrate the effectiveness and superiority of our proposed model.

Figures & tables

Figure 1. Cross-modal semantic conflict in two CH-SIMS examples. Text-only and multimodal predictions from EMT-DLFR ( Sun et al., 2023 ) retain the polarity of the textual annotation despite conflicting nonverbal annotations. Labels range from $-1$ (negative) to $1$ (positive). Two CH-SIMS video examples compare independent modality annotations with text-only and multimodal predictions from EMT-DLFR.

Figure 2. Overview of UMC with CPLR for continuous unimodal pseudo-label recovery and RPC for reliability and precision consensus. Overview of UMC, including multimodal inputs, CPLR, and RPC.

	CMU-MOSI						CMU-MOSEI
Method	Acc-7 $\uparrow$	Acc-5 $\uparrow$	Acc-2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc-7 $\uparrow$	Acc-5 $\uparrow$	Acc-2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
MulT ( Tsai et al., 2019 )	40.00	42.68	83.00	82.00	0.8710	0.6980	51.80	54.18	82.50	82.30	0.5800	0.7030
Self-MM ∗ ( Yu et al., 2021 )	47.67	54.37	83.23	83.25	0.7267	0.7813	54.20	55.87	85.17	85.16	0.5356	0.7690
SUGRM ∗ ( Hwang and Kim, 2023 )	46.21	53.21	81.71	81.82	0.7357	0.7822	53.55	55.29	85.25	84.91	0.5389	0.7560
DMD ( Li et al., 2023 )	46.06	-	83.23	83.29	0.7520	-	52.78	-	84.62	84.62	0.5430	-
EUAR ∗ ( Gao et al., 2024 )	46.06	53.21	83.54	83.59	0.7167	0.7929	52.48	54.39	83.52	83.66	0.5363	0.7590

Table 1. Comparative experiments with the best-performing MSA models conducted on the CMU-MOSI, CMU-MOSEI, CH-SIMS, and CH-SIMS v2 datasets. Except for MAE (

\downarrow

), higher values (

\uparrow

) indicate better performance. Bold and underlined values are best and second best. Starred results are reproduced from released code in the same environment as UMC. Unstarred results for CMU-MOSI and CMU-MOSEI come from DLF ( Wang et al., 2025 ) . Those for CH-SIMS and CH-SIMS v2 come from the MMSA repository ( Mao et al., 2022 ) .

	CMU-MOSI						CMU-MOSEI
Variant	Acc-7 $\uparrow$	Acc-5 $\uparrow$	Acc-2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc-7 $\uparrow$	Acc-5 $\uparrow$	Acc-2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
UMC	47.81	55.10	86.59	86.52	0.7067	0.8074	54.73	56.58	86.10	86.05	0.5248	0.7761
w/o CPLR	40.23	51.60	84.91	84.85	0.7850	0.7742	52.41	55.46	85.20	85.16	0.5403	0.7690
w/o RPC	22.89	23.62	82.62	82.70	0.7511	0.7177	49.24	47.93	81.37	80.91	0.6428	0.6973
w/o CPLR and RPC	24.93	23.91	80.49	80.52	0.7719	0.7006	49.26	47.39	81.73	80.91	0.6265	0.7228
	CH-SIMS						CH-SIMS v2

Table 2. The ablation experiments on the CMU-MOSI, CMU-MOSEI, CH-SIMS and CH-SIMS v2 datasets. Bold and underlined values are best and second within each dataset.

Dataset	M	Target	A5 $\uparrow$	A3 $\uparrow$	A2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
CH-SIMS	T	Raw	27.98	39.45	65.60	66.93	0.4697	0.2071
		CPLR	30.28	39.91	62.84	64.97	0.3915	0.3156
	A	Raw	38.07	55.05	72.94	72.69	0.3413	0.6217
		CPLR	39.45	55.05	71.10	71.29	0.3183	0.6299
	V	Raw	38.99	55.50	72.02	71.52	0.3872	0.5861
		CPLR	34.40	56.88	73.39	73.10	0.3879	0.6319

Table 3. Pseudo-label recovery on

D_{\mathrm{msi}}

. M denotes modality.

A_{k}

denotes Acc-

k

. Raw denotes the multimodal label. T, A, and V denote text, audio, and vision. Bold values rank first.

Dataset	Variant	A7 $\uparrow$	A5 $\uparrow$	A2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
CMU-MOSI	Continuous	47.81	55.10	86.59	86.52	0.7067	0.8074
	Discrete	43.73	51.90	83.38	83.47	0.7533	0.7931
CMU-MOSEI	Continuous	54.73	56.58	86.10	86.05	0.5248	0.7761
	Discrete	54.02	54.45	84.65	84.67	0.5575	0.7595
Dataset	Variant	A5 $\uparrow$	A3 $\uparrow$	A2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
CH-SIMS	Continuous	50.11	74.18	85.78	85.35	0.3632	0.7052

Table 4. Comparison of continuous and discrete CPLR. A

k

denotes Acc-

k

. Bold values rank first.

Dataset		MOSI						MOSEI
Set	Fuse	A7 $\uparrow$	A5 $\uparrow$	A2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	A7 $\uparrow$	A5 $\uparrow$	A2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
All	Mean	22.30	23.47	78.05	78.13	1.0723	0.7526	47.61	48.47	81.10	80.81	0.6448	0.6988
	RPC	47.81	55.10	86.59	86.52	0.7067	0.8074	54.73	56.58	86.10	86.05	0.5248	0.7761
Q4	Mean	10.47	10.47	80.12	81.00	1.6538	0.7594	31.33	30.56	86.01	85.93	0.9327	0.7831
	RPC	48.84	68.02	92.40	91.91	0.7184	0.8614	51.50	55.36	90.51	90.43	0.5793	0.8600
Dataset		SIMS						SIMS v2

Table 5. RPC versus the arithmetic mean on the full test set and highest-disagreement quartile (Q4). A

k

denotes Acc-

k

. Bold values rank first.

Figure 3. Modality corruption analysis. Normalized MAE is computed by dividing MAE by the dataset’s sentiment-label span. Higher Acc-2 and lower normalized MAE are better. Audio and visual noise uses

\sigma_{\mathrm{noise}}=0.5

. Two horizontal grouped-bar charts compare arithmetic-mean and RPC fusion in absolute Acc-2 and label-span-normalized MAE under eight controlled corruptions.

Figure 4. CPLR pseudo-label MSE trajectories. Solid lines show the MSE of CPLR pseudo labels, while dashed lines show the MSE of raw multimodal labels. Lower is better. Line plots compare text, audio, and visual pseudo-label mean squared error over CPLR epochs with the fixed MSE of raw multimodal labels on CH-SIMS and CH-SIMS v2.

Dataset	Confidence	Conflict	$n$	$\bar{c}$	$\bar{\eta}$	$\bar{\lambda}$	Raw MAE $\downarrow$	CPLR MAE $\downarrow$	$\Delta$ MAE
CH-SIMS	Low	Low	142	0.7157	0.0694	0.0497	0.3352	0.3361	-0.0009
	Low	High	185	0.7116	0.3564	0.2531	0.4768	0.4231	+0.0537
	High	Low	185	0.7498	0.0732	0.0549	0.2605	0.2606	-0.0001
	High	High	142	0.7496	0.3031	0.2270	0.5437	0.4583	+0.0854
CH-SIMS v2	Low	Low	336	0.7160	0.0792	0.0567	0.2131	0.2125	+0.0006
	Low	High	485	0.7105	0.2845	0.2018	0.3287	0.2983	+0.0304

Table 6. CPLR update strength on

D_{\mathrm{msi}}

. The term

n

is the number of modality-level instances, and

\bar{c}

\bar{\eta}

, and

\bar{\lambda}

are mean confidence, conflict, and update strength. Both MAEs use held-out unimodal annotations, and

\Delta

MAE is raw minus CPLR. Bold values denote the lower MAE.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5. Fixed-expert fusion performance. Fine and mid accuracy denote Acc-7 and Acc-5 on CMU datasets and Acc-5 and Acc-3 on CH datasets. Higher is better except for MAE. Six grouped bar charts compare arithmetic mean, precision-only fusion, reliability-only fusion, and full RPC with fixed experts on four datasets.

Figure 6. Fixed-expert arithmetic-mean and RPC MAE across prediction-disagreement quartiles. Q1 and Q4 denote the lowest and highest disagreement. Lower is better. Grouped plots compare RPC and arithmetic-fusion mean absolute error from the lowest to the highest cross-modal disagreement quartile on four datasets.

	Top-1 (%)			Reliability correlation
Dataset	$r$	$\bar{\tau}$	$\omega$	$\rho_{t}$	$\rho_{a}$	$\rho_{v}$
CMU-MOSI	69.53	69.53	69.53	0.0633	0.5253	0.4779
CMU-MOSEI	53.34	53.14	53.32	-0.0855	0.3979	0.3978
CH-SIMS	56.46	54.27	56.02	0.2688	0.3534	0.1122
CH-SIMS v2	56.87	56.87	55.51	0.3488	0.2244	0.2732

Appendix

Table 7. RPC signal analysis. Top-1 measures whether the largest reliability

r

, normalized precision

\bar{\tau}

, or final weight

\omega

selects the lowest-error expert. The term

\rho_{k}

is the reliability correlation with negative modality-

k

error. Bold and underlined Top-1 values are best and second.

	Dataset	CMU-MOSI						CMU-MOSEI						CH-SIMS						CH-SIMS v2
	Metric	Acc-7	Acc-5	Acc-2	F1	MAE	Corr	Acc-7	Acc-5	Acc-2	F1	MAE	Corr	Acc-5	Acc-3	Acc-2	F1	MAE	Corr	Acc-5	Acc-3	Acc-2	F1	MAE	Corr
Full	UMC	47.81	55.10	86.59	86.52	0.7067	0.8074	54.73	56.58	86.10	86.05	0.5248	0.7761	50.11	74.18	85.78	85.35	0.3632	0.7052	58.90	77.27	85.20	85.17	0.2627	0.7878
CPLR variants	Head only	46.21	48.25	82.93	82.94	0.7456	0.7592	53.68	55.53	85.36	85.24	0.5407	0.7657	40.92	66.96	80.09	80.57	0.3867	0.6345	55.80	73.40	78.14	78.27	0.2939	0.7456
	Proto. only	39.65	47.81	79.88	80.00	0.7522	0.7637	53.40	54.99	84.81	84.72	0.5359	0.7684	45.08	68.71	79.65	80.01	0.3762	0.6176	54.64	74.27	81.53	81.64	0.2772	0.7616
	No unc.	41.40	45.63	83.38	83.36	0.7329	0.7818	53.94	54.43	85.33	85.31	0.5335	0.7723	42.89	63.68	78.12	78.20	0.3738	0.6300	56.67	75.73	81.24	81.34	0.2848	0.7569
	No conf.	23.76	23.76	85.21	85.16	0.7338	0.7541	46.02	50.91	84.20	83.81	0.6232	0.7211	40.48	61.93	77.46	77.69	0.3912	0.6606	50.97	66.15	80.95	81.00	0.2737	0.7475

Appendix

Table 8. Internal-signal ablations. Higher is better except for MAE. Bold and underlined values are best and second. Proto., unc., conf., disag., and sup. abbreviate prototype, uncertainty, conflict, disagreement, and supervision.

Dataset	Params	FLOPs	Mean ms	RPC ms	$\Delta t$	Sample/s
CMU-MOSI	2,305	13,080	29.2024	29.8395	2.18%	2144.8
CMU-MOSEI	2,305	13,080	28.0514	28.5647	1.83%	2240.5
CH-SIMS	2,305	13,080	32.2526	32.8795	1.94%	1946.5
CH-SIMS v2	2,305	13,080	29.7458	30.4773	2.46%	2099.9

Appendix

Table 9. Inference efficiency of UMC. Mean uses arithmetic-mean fusion with the same experts, while RPC ms and Sample/s measure complete UMC inference. Params, FLOPs, and

\Delta t

report the incremental RPC cost. Latency is the median of 100 batches on one RTX 4090D. Bold values denote the lower latency.

Dataset	M	Target	A5 $\uparrow$	A3 $\uparrow$	A2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
CH-SIMS	T	Raw	71.55	100.00	100.00	100.00	0.1908	0.9267
		CPLR	70.71	99.58	99.16	99.17	0.2000	0.9328
	A	Raw	75.73	100.00	100.00	100.00	0.1506	0.9411
		CPLR	62.76	100.00	97.91	97.93	0.2335	0.9429
	V	Raw	80.33	100.00	100.00	100.00	0.1238	0.9557
		CPLR	73.22	99.16	100.00	100.00	0.2226	0.9520

Appendix

Table 10. Pseudo-label recovery on

D_{\mathrm{msc}}

. Raw denotes the multimodal label. T, A, and V denote text, audio, and vision. Bold values rank first.

	Dataset	CH-SIMS												CH-SIMS v2
	Subset	$D_{\mathrm{msc}}$						$D_{\mathrm{msi}}$						$D_{\mathrm{msc}}$						$D_{\mathrm{msi}}$
	Metric	Acc-5	Acc-3	Acc-2	F1	MAE	Corr	Acc-5	Acc-3	Acc-2	F1	MAE	Corr	Acc-5	Acc-3	Acc-2	F1	MAE	Corr	Acc-5	Acc-3	Acc-2	F1	MAE	Corr
Full	UMC	56.07	87.45	92.47	92.20	0.3338	0.8368	43.58	59.63	78.44	77.92	0.3954	0.4165	58.73	88.30	94.05	94.05	0.2161	0.8749	59.05	67.46	77.33	77.27	0.3043	0.6281
w/o	CPLR	52.30	79.08	89.96	90.01	0.3377	0.7389	34.86	52.29	69.72	70.24	0.4028	0.3114	54.83	85.01	92.81	92.80	0.2270	0.8665	56.67	65.45	75.32	75.23	0.3060	0.5895
	RPC	46.86	82.01	88.70	88.17	0.3856	0.7790	34.86	52.29	74.77	73.65	0.3795	0.3595	50.92	81.72	90.76	90.82	0.2733	0.8338	52.65	59.23	71.66	71.70	0.3320	0.5050
	CPLR and RPC	43.93	79.50	89.12	89.00	0.4008	0.7684	39.45	55.96	68.81	69.18	0.3890	0.3955	51.13	79.06	89.73	89.78	0.2642	0.8337	51.55	57.04	72.94	72.96	0.3198	0.5156

Appendix

Table 11. Component ablations on modality-semantically consistent (

D_{\mathrm{msc}}

) and inconsistent (

D_{\mathrm{msi}}

) test subsets. Higher is better except for MAE. Bold and underlined values are best and second. Proto., unc., conf., disag., and sup. abbreviate prototype, uncertainty, conflict, disagreement, and supervision.

Figure 7. Acc-2 sensitivity to prototype count

K

, prototype momentum

\zeta

, router target temperature

T_{r}

, router-loss weight

\alpha

, and unimodal-loss weight

\beta

on four datasets. The

\alpha

and

\beta

axes show multipliers of their selected values. Five plots show binary classification accuracy when prototype count, prototype momentum, router target temperature, router-loss weight, or unimodal-loss weight is varied independently on four datasets.

Multimodal Sentiment Analysis

Multimodal sentiment analysis combines language, voice, and visual cues to infer sentiment, polarity, and emotional intensity.

See all 54 papers

MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis
Jul 12, 2026Haoran Ma, Yinfeng Yu, Liejun WangMultimodal Sentiment Analysis Multimodal Robustness
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China. · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing. · Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center. +2
ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis
Oct 22, 2026Pan Wang, Lipeng Ke, Huajun Ying +5Multimodal Sentiment Analysis Multimodal Learning
Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, USA · Amazon, Sunnyvale, USA
SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data
Sep 24, 2026Wenhao Li, Zhibin Wu, Chong Xiao +1Missing-Modality Learning Multimodal Robustness
Software School, Shandong University · Shenzhen Loop Area Institute

Jul 12, 2026J/K move · Enter open · S save

MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis

Haoran Ma, Yinfeng Yu, Liejun Wang

School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China. · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing. · Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center.+2

Multimodal Sentiment Analysis Multimodal Robustness Uncertainty-Aware Multimodal Fusion Multimodal Fusion Multimodal Model Routing Reliability-Aware Multimodal Fusion+1

Open paper AskPDF arXiv

Multimodal sentiment analysis relies on language, visual, and acoustic cues, but utterance-level modality quality may vary due to occlusion, background noise, motion blur, or imperfect transcripts, causing conventional fusion to over-trust unreliable modalities. We propose MRUF, a reliability-aware fusion method that combines multi-granularity routing with uncertainty-aware calibration. MRUF summarizes sentiment-relevant representations, performs subspace- and modality-level routing, and supervises modality routing with leave-one-out error increases to estimate utterance-level modality importance. It further predicts modality-wise uncertainty and refines modality gates through inverse-variance reweighting, while modality-invariant contrastive alignment stabilizes the shared representation space. Experiments on CMU-MOSI and CMU-MOSEI under aligned and unaligned settings show consistent improvements over strong baselines, and mechanism analysis verifies that modalities with higher predicted uncertainty receive lower fusion weights.

Abstract

Multimodal Sentiment Analysis

Explore similar work

MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis

ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data