ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis

Pan Wang; Lipeng Ke; Huajun Ying; Pritish Mohapatra; Rohan Sarkar; Suresh Lakhani; Sankar Venkataraman; Jingtong Hu

cs.LGOct 22, 2026

ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis

Authors: Pan Wang, Lipeng Ke, Huajun Ying, Pritish Mohapatra, Rohan Sarkar, Suresh Lakhani, Sankar Venkataraman, Jingtong Hu

Organizations: Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, USA · Amazon, Sunnyvale, USA

Abstract

Multimodal sentiment analysis (MSA) aims to predict human sentiments by integrating signals from different modalities such as text, video, and audio. However, raw multimodal sequences often suffer from semantic inconsistency--exhibiting redundancy or conflicts within and across modalities--which hinders robust understanding and increases computational cost. To this end, we introduce ConsMSA, which explicitly formalizes semantic distribution consistency across both $\textit{intra}$ - and $\textit{inter}$ -modality, providing a consistency-aware mechanism for robust and efficient multimodal sentiment prediction. Specifically, ConsMSA projects multimodal token features into a shared sentiment space to compute an Intra- and Inter-modality Consistency Score ( $I^2CS$ ). By coupling this score with predictive relevance, we formulate consistency-aware importance signals that are utilized: (i) as a consistency regularizer to align latent distributions during training, (ii) to derive semantic-aware weights for adaptive multimodal token reweighting, and (iii) as a practical criterion to prune redundant or conflicting tokens. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate that ConsMSA achieves state-of-the-art performance while remaining robust under aggressive token compression--retaining only 10% of tokens yields comparable accuracy. These results establish semantic distribution consistency as a promising foundation for synergizing predictive robustness with computational efficiency.

Figures & tables

Figure 1: Motivation of ConsMSA. We explicitly model token-level intra- and inter- modality consistency by measuring semantic alignment in the prediction space, enabling multimodal token reweighting and compression .

Figure 2: Overview of the ConsMSA framework. Modality-specific tokens are projected into a shared semantic space where the Consensus module computes intra - and inter -modality consistency ( $I^{2}CS$ ). By coupling these scores with predictive relevance ( $Rel$ ), we derive practical importance signals that drive consistency-aware learning and token optimization. The optimized tokens are finally concatenated in the Fusion Module for the final prediction. The system is optimized via a joint loss to achieve a robust balance between task accuracy ( $\mathcal{L}_{task}$ ), semantic consistency ( $\mathcal{L}_{I^{2}CS}$ ), and modality-specific losses ( $\mathcal{L}_{MS}$ ).

Figure 3: Details of our Consensus module. It projects text and audio tokens into a semantic space, and then computes IntraCS, InterCS, $I^{2}CS$ , and relevance, resulting in $\mathcal{L}_{I^{2}CS}$ .

\vskip-5.69054pt\text{I}^{2}\text{CS}^{(a)},\text{I}^{2}\text{CS}^{(t)}\xrightarrow[\text{Rel}]{\text{Masking}}\begin{cases}\tilde{h}^{(a)},\tilde{h}^{(t)}\in\mathbb{R}^{B\times T\times D},\text{(soft)}\\ \tilde{h}^{(a)},\tilde{h}^{(t)}\in\mathbb{R}^{B\times K\times D},\text{(hard)}\end{cases}

Algorithm 1 End-to-end ConsMSA Pipeline

BERT-based
Method	Modality	CMU-MOSI				CMU-MOSEI
Method	Modality	Acc-7( $\uparrow$ )	Acc-2( $\uparrow$ )	F1( $\uparrow$ )	MAE( $\downarrow$ )	Acc-7( $\uparrow$ )	Acc-2( $\uparrow$ )	F1( $\uparrow$ )	MAE( $\downarrow$ )
TFN*	T+A+V	34.90	80.08	80.07	0.901	50.20	82.50	82.10	0.593
LMF*	T+A+V	33.20	82.50	82.40	0.917	48.00	82.00	82.10	0.623
EF-LSTM †	T+A+V	35.39	78.48	78.51	0.949	50.01	80.79	80.67	0.601
LF-DNN †	T+A+V	34.52	78.63	78.63	0.955	50.83	82.74	82.52	0.580

Table 1: Comparison of our method with prior SOTA methods on MOSI and MOSEI datasets. The best performance is highlighted in bold. T: Text, A: Audio, and V: Video. Note: † denotes the result from THUIAR’s GitHub page, ∗ denotes the result from ( Hazarika et al., 2020 ) , - denotes that the result from the original paper is not provided, and others are all from the original papers. For fair comparison, the BERT-based models utilize T+A+V following DLF, while the RoBERTa-based model utilizes T+A following MMML.

Method	Modality	Acc-2 $\uparrow$	Acc-3 $\uparrow$	Acc-5 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$
MMML	T+A	82.93	69.37	49.38	82.90	0.332
Ours	T+A	83.59	70.68	50.77	83.64	0.331

Table 2: Comparison on CH-SIMS dataset.

Method	Acc-7( $\uparrow$ )	Acc-2( $\uparrow$ )	F1( $\uparrow$ )	MAE( $\downarrow$ )
ConsMSA (Ours)	51.46	90.09	90.08	0.557
w/o IntraCS	47.52	87.20	87.21	0.629
w/o InterCS	46.79	88.87	88.83	0.626
w/o $Rel$ in Eq. ( 6 )	45.48	87.96	87.99	0.668
w/o $Rel$ in Eq. ( 9 )	46.36	87.80	87.82	0.647
w/o Gating	45.77	87.20	87.21	0.653

Table 3: Results of ablation studies on the MOSI benchmark.

Ratio	CH-SIMS (Acc2/F1/MAE)	MOSI (Acc2/F1/MAE)
1.0	83.59/83.64/ 0.331	90.09/90.08/0.557
0.8	82.06/82.25/0.341	90.85/90.84/0.553
0.7	83.15/83.20/0.347	89.33/89.34/0.562
0.6	83.59/83.70 /0.337	89.63/89.66/0.585
0.5	82.71/83.00/0.337	88.72/88.75/0.585
0.3	81.84/82.19/0.333	89.33/89.31/0.567

Table 4: Different token retention ratios on CH-SIMS and MOSI.

Panel of Figure 4 — Figure 4: Visualization of intra- and inter-modality semantic consistency at the token level. (a) Pre-optimization with ConsMSA framework. (b) After-optimization with ConsMSA framework. Note : Lower value, higher consistency.

Panel of Figure 5 — Figure 5: Analysis on the MOSI dataset. (a) Confusion matrix, and (b) Corresponding accuracy for each sentiment. HN: Highly Negative; N: Negative; WN: Weakly Negative; WP: Weak Positive; P: Positive; HP: Highly Positive.

Method	Acc-7( $\uparrow$ )	Acc-2( $\uparrow$ )	F1( $\uparrow$ )	MAE( $\downarrow$ )
ConsMSA (Ours)	51.46	90.09	90.08	0.557
Entropy	50.29	88.72	88.75	0.578
Attention	50.48	88.63	88.62	0.572
Random	49.27	88.33	88.31	0.575
KL-div	50.32	89.30	89.20	0.612
Cosine	47.96	88.90	88.71	0.628

Table 5: Different masking signals.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	MOSI	MOSEI	CH-SIMS
Optimizer	AdamW	AdamW	AdamW
Learning rate	5e-6	5e-6	1e-5
Batch size	8	8	8
Token length	50/96	50/96	50/96
Early stopping epochs	8	8	8
$\alpha$	1.0	1.0	1.0

Appendix

Table 6: Hyperparameter settings on all MSA benchmarks. T: Text, A: Audio. The token length is based on the BERT/RoBERTa-based backbone.

Modality	Acc-7 (%)	Acc-2 (%)	F1 (%)	MAE( $\downarrow$ )
ConsMSA (Ours)	51.46	90.09	90.08	0.557
only Audio	39.11	84.23	84.25	0.805
only Text	45.75	86.61	86.62	0.654

Appendix

Table 7: Performance of different modalities on the MOSI benchmark in the setting of A+T. A: Audio, T: Text.

Modality Pair	Acc-7 (%)	Acc-2 (%)	F1 (%)	MAE( $\downarrow$ )
All (Ours)	47.96	86.06	86.06	0.698
(A, V)	43.44	83.58	83.02	0.771
(V, T)	45.92	84.49	84.50	0.732
(T, A)	47.23	85.26	85.21	0.728

Appendix

Table 8: Ablation of I²CS pair on the MOSI dataset in the BERT-based setting. T: Text, A: Audio, V: Visual.

Method	Modality	CMU-MOSI				CMU-MOSEI
Method	Modality	Acc-7 $\uparrow$	Acc-2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Acc-7 $\uparrow$	Acc-2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$
MMML	T+A	50.34	89.69	89.67	0.580	55.74	88.02	88.15	0.492
ConsMSA (Ours)	T+A	51.46	90.09	90.08	0.557	56.00	88.41	88.39	0.485
FCD	T+A+V	-	88.48	88.44	0.599	-	86.63	86.63	0.500
ConsMSA (Ours)	T+A+V	51.60	90.48	90.38	0.544	56.08	88.89	88.80	0.483

Appendix

Table 9: Performance of RoBERTa-based ConsMSA with “T+A+V” and “T+A” on the MOSI dataset. T: Text, A: Audio, V: Visual.

Method	Modality	Acc2 $\uparrow$	Acc3 $\uparrow$	Acc5 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$
MMML	T+A	82.93	69.37	49.38	82.90	0.332
Ours	T+A	83.59	70.68	50.77	83.64	0.331
Ours	T+A+V	83.81	71.12	50.92	83.81	0.316

Appendix

Table 10: Comparison on CH-SIMS dataset.

$\alpha$	$\beta$	$\gamma$	Acc2 $\uparrow$	Acc3 $\uparrow$	Acc5 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$
1.0	1.0	20	83.59	70.68	50.77	83.64	0.331
1.0	0.5	20	83.59	70.68	49.67	83.70	0.339
0.5	1.0	10	83.59	70.68	49.67	83.70	0.339
0.5	0.5	10	83.59	70.68	49.67	83.70	0.339

Appendix

Table 11: Hyperparameter sensitivity on the CH-SIMS dataset.

BERT-based
Methods	Latency (ms)	Runtime Memory (GB)	Speedup
TFN	22.3689	0.0101	1.00x
MULT	22.5551	0.0110	0.99x
MISA	33.9139	0.0254	0.66x
MAG-BERT	33.5336	0.0555	0.67x
MMIM	78.2354	0.0312	0.29x

Appendix

Table 12: Comparison of efficiency on the MOSI dataset. 1.0 and 0.1 mean the token retention ratios.

Multimodal Sentiment Analysis

Multimodal sentiment analysis combines language, voice, and visual cues to infer sentiment, polarity, and emotional intensity.

See all 53 papers

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data
Sep 24, 2026Wenhao Li, Zhibin Wu, Chong Xiao +1Missing-Modality Learning Multimodal Robustness
Software School, Shandong University · Shenzhen Loop Area Institute
A Conflict-Aware Penalty and Statistical Loss Framework for Balancing Modalities and Enhancing Stability in Multimodal Sentiment Analysis
May 27, 2026Jianheng Dai, Jiazhang Liang, Sijie MaiModality Imbalance Gradient Interference
School of Computer Science, South China Normal University, Guangzhou, Guangdong, China
Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
Apr 14, 2026Kang He, Yuzhe Ding, Xinrong Wang +3Multimodal Sentiment Analysis Modality Imbalance
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University. · Shanghai Innovation Institute

Sep 24, 2026J/K move · Enter open · S save

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

Wenhao Li, Zhibin Wu, Chong Xiao, Qiangchang Wang

Software School, Shandong University · Shenzhen Loop Area Institute

Missing-Modality Learning Multimodal Robustness Cross-Modal Alignment Multimodal Sentiment Analysis

Open paper AskPDF arXiv

Recent research on Multimodal Sentiment Analysis (MSA) has focused on learning from language, visual, and acoustic modalities with incomplete data to infer human sentiment. Most studies typically compensate for missing information by reconstructing modality features or designing complicated fusion mechanisms. However, these methods still suffer from spurious generation and noisy guidance due to the lack of high-level semantic grounding in partially observed multimodal evidence. To address these issues, we propose SemMSA, a latent semantic-aided framework that constructs rich sentiment-relevant semantics with LLMs, fully integrating with all modalities via anchor-free spectral alignment. It mainly consists of Cross-modal Semantic Refinement (CSR) and Cross-modal Spectral Alignment (CSA). Specifically, CSR first adaptively extracts visual and acoustic representations by corresponding adapters to form a unified multimodal prefix with language in the frozen LLM embedding space. It then iteratively produces continuous discriminative semantic states through a token-efficient latent refinement process without decoding explicit text. Next, CSA simultaneously aligns the refined semantics with all modalities by enhancing the dominant spectral component of their kernel Gram matrix. This captures global nonlinear dependencies among all representations without relying on a predefined anchor modality. In addition, an instance-level spectral separation constraint preserves cross-sample discriminability and mitigates representation collapse. Extensive experiments on SIMS, MOSI, and MOSEI benchmarks demonstrate that SemMSA achieves state-of-the-art performance.

Abstract

Multimodal Sentiment Analysis

Explore similar work

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

A Conflict-Aware Penalty and Statistical Loss Framework for Balancing Modalities and Enhancing Stability in Multimodal Sentiment Analysis

Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis