Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysis

Shanshan Lin; Yuesheng Wu; Chao Chen; Yizhe Yang; Zhihao Chen; Zexian Yang; Xiangwen Liao

cs.LGAug 17, 2026

Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysis

Authors: Shanshan Lin, Yuesheng Wu, Chao Chen, Yizhe Yang, Zhihao Chen, Zexian Yang, Xiangwen Liao

Organizations: Fuzhou University, Fuzhou, China · Harbin Institute of Technology (Shenzhen), Shenzhen, China · Jiangxia University, Fuzhou, China

Abstract

Multimodal sentiment analysis (MSA) aims to predict sentiment polarity and intensity from heterogeneous inputs such as text, audio, and vision. While large language models (LLMs) offer strong semantic priors for MSA, effectively incorporating audio and visual signals effectively remains challenging. A key challenge is that audio and visual sentiment cues evolve over different temporal scales, yet many LLM-based methods compress these signals through shallow projection or coarse pooling before fusing them with text, which can weaken cross-modal alignment and erase fine-grained affective information. We propose MGSI, a multi-granularity sentiment integration framework for LLM-based MSA. MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends. It then refines non-text features through text-guided alignment, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples. The resulting multimodal representation is finally compressed into a small set of pseudo-tokens for efficient conditioning of a frozen LLM. Experiments on four public benchmarks show that MGSI substantially outperforms frozen-LLM baselines and remains competitive with strong multimodal methods. Further ablation and sensitivity analyses support the effectiveness of multi-granularity temporal modeling, text-guided refinement, and adaptive sentiment calibration.

Figures & tables

Figure 1: Overview of the proposed MGSI framework. Audio and visual streams are first encoded with multi-granularity temporal encoders. Their representations are then refined via (a) text-guided alignment, (b) auxiliary non-neutral classifier, (c) adaptive sentiment calibrator, and (d) multi-scale fusion adapter. Finally, the generated pseudo-tokens are injected into the frozen LLM for sentiment prediction.

Method	MOSI				MOSEI
Method	Acc-2 $\uparrow$	F1 $\uparrow$	Acc-7 $\uparrow$	MAE $\downarrow$	Acc-2 $\uparrow$	F1 $\uparrow$	Acc-7 $\uparrow$	MAE $\downarrow$
TFN	79.08	79.11	34.46	0.947	81.89	81.74	51.60	0.573
LMF	79.18	79.15	33.82	0.950	84.48	83.36	51.59	0.576
MulT	80.98	80.95	36.91	0.880	84.63	84.52	52.84	0.559
MISA	83.54	83.58	41.37	0.777	84.67	84.66	52.05	0.558
CENet	85.21	85.22	44.90	0.725	86.38	86.32	54.26	0.526

Table 1: Results on MOSI and MOSEI . Best results are in bold.

\ast

and

\dagger

indicate statistical significance at the 0.05 level compared with the strongest non-LLM method and MSE-Adapter baseline under the same LLM backbone, respectively.

Method	SIMS				SIMS-V2
Method	Acc-2 $\uparrow$	F1 $\uparrow$	Acc-5 $\uparrow$	MAE $\downarrow$	Acc-2 $\uparrow$	F1 $\uparrow$	Acc-5 $\uparrow$	MAE $\downarrow$
TFN	78.38	78.62	39.30	0.432	80.14	80.14	52.55	0.303
LMF	77.77	77.88	40.53	0.441	74.18	73.88	47.79	0.367
MulT	78.56	79.66	37.94	0.453	80.68	80.73	54.81	0.291
Self-MM	80.04	80.44	41.53	0.425	79.69	79.76	52.77	0.311
TETFN	81.18	80.24	41.79	0.420	79.73	79.81	54.47	0.310

Table 2: Results on SIMS and SIMS-V2 . Best results are in bold.

\ast

and

\dagger

indicate statistical significance at the 0.05 level compared with the strongest non-LLM method and MSE-Adapter under the same LLM backbone, respectively.

Variant	MOSI				SIMS
	Acc-2 $\uparrow$	F1 $\uparrow$	Acc-7 $\uparrow$	MAE $\downarrow$	Acc-2 $\uparrow$	F1 $\uparrow$	Acc-5 $\uparrow$	MAE $\downarrow$
Core components
MGSI	89.60	89.57	49.15	0.603	82.67	81.81	46.92	0.359
w/o MGT	89.30	89.28	48.42	0.617	80.48	79.56	46.08	0.357
w/o NNC	89.54	89.49	48.78	0.613	82.05	80.89	47.13	0.361
w/o ASC	88.78	88.71	49.10	0.614	81.66	79.89	43.76	0.382

Table 3: Ablation results using the frozen ChatGLM3-6B backbone on MOSI and SIMS. The upper block evaluates the main components, and the lower block analyzes the temporal branches in the MGT encoder.

Figure 2: Sensitivity and design analysis on MOSI and SIMS. All values denote absolute improvements in percentage points over MSE-Adapter under the same backbone. From left to right: sensitivity to the visual short-term kernel size, comparison of branch-fusion strategies, and effect of the non-neutral loss weight

\lambda_{nnc}

Explore similar work

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data
Sep 24, 2026Wenhao Li, Zhibin Wu, Chong Xiao +1Missing-Modality Learning Multimodal Robustness
Software School, Shandong University · Shenzhen Loop Area Institute
Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis
Jul 30, 2026Wei Chen, Junkai Li, Tongguan Wang +4Multimodal Large Language Models Multimodal Classification
Huazhong Agricultural University Wuhan, China · Hubei University Wuhan, China · Engineering Research Center of Intelligent Technology for Agriculture, Hubei Engineering Technology Research Center of Agricultural Big Data, Key Laboratory of Smart Farming for Agricultural Animals.
Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
Apr 14, 2026Kang He, Yuzhe Ding, Xinrong Wang +3Multimodal Sentiment Analysis Modality Imbalance
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University. · Shanghai Innovation Institute

Sep 24, 2026J/K move · Enter open · S save

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

Wenhao Li, Zhibin Wu, Chong Xiao, Qiangchang Wang

Software School, Shandong University · Shenzhen Loop Area Institute

Missing-Modality Learning Multimodal Robustness Cross-Modal Alignment Multimodal Sentiment Analysis

Open paper AskPDF arXiv

Recent research on Multimodal Sentiment Analysis (MSA) has focused on learning from language, visual, and acoustic modalities with incomplete data to infer human sentiment. Most studies typically compensate for missing information by reconstructing modality features or designing complicated fusion mechanisms. However, these methods still suffer from spurious generation and noisy guidance due to the lack of high-level semantic grounding in partially observed multimodal evidence. To address these issues, we propose SemMSA, a latent semantic-aided framework that constructs rich sentiment-relevant semantics with LLMs, fully integrating with all modalities via anchor-free spectral alignment. It mainly consists of Cross-modal Semantic Refinement (CSR) and Cross-modal Spectral Alignment (CSA). Specifically, CSR first adaptively extracts visual and acoustic representations by corresponding adapters to form a unified multimodal prefix with language in the frozen LLM embedding space. It then iteratively produces continuous discriminative semantic states through a token-efficient latent refinement process without decoding explicit text. Next, CSA simultaneously aligns the refined semantics with all modalities by enhancing the dominant spectral component of their kernel Gram matrix. This captures global nonlinear dependencies among all representations without relying on a predefined anchor modality. In addition, an instance-level spectral separation constraint preserves cross-sample discriminability and mitigates representation collapse. Extensive experiments on SIMS, MOSI, and MOSEI benchmarks demonstrate that SemMSA achieves state-of-the-art performance.

Abstract

Explore similar work

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis

Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis