MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition
Authors: Pengfei Shao, Jisheng Dang, Jiawen Fang, Ning Liu, Wencan Zhang, Bimei Wang, Jingwen Zhao, Jianhuang Lai, +2 more
Organizations: School of Information Science and Technology, Guangdong University of Foreign Studies, Guangdong, China · School of Information Science and Engineering, Lanzhou University, Lanzhou, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangdong, China · School of Computing, National University of Singapore, Singapore 119077 · Cloud and AI BU, Huawei, Shenzhen, Guangdong 518129, China
Multimodal sentiment analysis and emotion recognition in conversations demand effective modeling of heterogeneous interactions across textual, acoustic, and visual modalities. Although large language models (LLMs) offer powerful language understanding, adapting them to multimodal affective computing remains challenging: full-model fine-tuning is computationally prohibitive, while many existing lightweight adapters fail to preserve rich textual cues during cross-modal fusion. To address these limitations, we propose the multi-view text-guided multimodal fusion adapter (MVFA), a parameter-efficient framework that augments frozen LLMs with strong multimodal reasoning capability. MVFA first constructs complementary text views via max pooling, mean pooling, and attention pooling; these views then guide cross-modal interactions with audio and visual features. The fused multimodal representations are subsequently compressed into a compact set of learnable pseudo-tokens through an Enhanced Q-Former Fusion Module. Using ChatGLM3-6B-base as the primary backbone, we further validate MVFA on LLaMA2-7B and Qwen3-8B to examine its portability across multiple frozen LLM backbones. MVFA is evaluated on three challenging datasets: CH-SIMS V2.0, MELD, and CHERMA. Experimental results demonstrate that MVFA achieves state-of-the-art performance on key metrics while updating only a small fraction of parameters. Specifically, it attains 84.62% Acc2 and 84.59% F1 on CH-SIMS V2.0, 67.36% Acc and 66.03% WF1 on MELD, and 74.66% Acc on CHERMA. These findings establish multi-view text-guided fusion as an effective and scalable paradigm for parameter-efficient multimodal LLM adaptation in affective computing. The code is publicly available at https://github.com/Overwhelm1208/MVFA.
Figures & tables
Fig. 1: An example illustrating that a single compressed text view may fail to preserve complementary affective cues (e.g., salient lexical expressions, discourse-level contrast, and context-relevant tokens) during multimodal fusion.
Fig. 2: Overview of the proposed MVFA framework. Lightweight encoders first extract representations from the audio and visual modalities. The MVFA then constructs complementary text views through max pooling, mean pooling, and attention pooling, progressively fuses them to guide cross-modal interaction, and compresses the resulting multimodal representation into a small set of pseudo-tokens via the Enhanced Q-Former Fusion Module (ENQF). These pseudo-tokens are concatenated with the raw text embeddings T and the prompt-augmented text embeddings Tp to form the input sequence Xinput=[P;T;Tp] , which is fed into a frozen LLM for autoregressive sentiment or emotion prediction. Only the MVFA is trained, while the LLM backbone remains frozen.
Model
Acc2 ↑
F1 ↑
Acc2-Weak ↑
Corr ↑
R 2 ↑
MAE ↓
TFN [ 3 ]
76.51
76.31
66.27
0.667
35.90
0.323
LMF [ 4 ]
77.05
77.02
69.34
0.638
40.64
0.343
MFN [ 25 ]
75.27
75.24
66.46
0.606
32.26
0.355
MULT [ 26 ]
79.50
79.59
69.61
0.703
47.15
0.317
Self-MM [ 14 ]
79.01
78.89
71.87
0.640
29.36
0.335
MISA [ 15 ]
80.53
80.63
70.50
0.725
50.59
0.314
TABLE I: MSA results on CH-SIMS V2.0. Acc2 and F1 are computed under the non-positive/positive binarization rule, and Acc2-Weak is evaluated on the subset of samples whose ground-truth sentiment scores fall in the range [-0.4, 0.4]. Results marked with * are cited from [ 10 ] , results marked with † are reproduced on our experimental platform, and the remaining results are taken from the corresponding original papers.
Model
Happiness ↑
Sadness ↑
Fear ↑
Anger ↑
Surprise ↑
Disgust ↑
Neutrality ↑
Acc ↑
TFN [ 3 ]
74.91
75.56
66.15
74.41
66.29
43.34
65.60
68.37
LMF [ 4 ]
74.52
75.83
66.73
74.55
65.08
45.70
65.64
68.23
EFT [ 11 ]
74.98
76.88
67.32
74.85
66.73
47.48
64.60
68.72
LFT [ 11 ]
75.07
76.29
66.80
74.88
66.67
47.74
65.97
69.05
LFMIM [ 11 ]
76.60
77.83
69.44
75.32
69.83
50.20
68.24
70.54
MULT [ 26 ]
76.18
76.88
67.36
74.85
68.18
46.96
65.26
69.24
TABLE II: Fine-grained emotion recognition results on the CHERMA dataset. We report F1 scores for each emotion category together with overall accuracy (Acc). Results marked with † are reproduced on our experimental platform.
Fig. 3: Row-normalized confusion matrices on CHERMA for (a) MSE-Adapter and (b) MVFA, both based on ChatGLM3-6B-base. MVFA yields a more concentrated diagonal pattern, especially for difficult classes such as Disgust, Fear, and Surprise.
Model
Acc ↑
WF1 ↑
LMF [ 4 ]
61.15
58.30
TFN [ 3 ]
60.70
57.74
GraphCFC [ 7 ]
61.42
58.86
MM-DFN [ 29 ]
62.49
59.46
EmoCaps [ 30 ]
–
64.00
GA2MIF [ 31 ]
61.65
58.94
TABLE III: Emotion recognition results on the MELD dataset. We report overall accuracy (Acc) and weighted F1 (WF1). Results marked with † are reproduced on our experimental platform, and the remaining results are taken from the corresponding original papers.
Model
Acc2 ↑
F1 ↑
Acc2-Weak ↑
Acc3 ↑
Acc5 ↑
Corr ↑
R 2 ↑
MAE ↓
w/o MTF
82.92
82.75
75.20
77.45
57.68
0.734
42.77
0.289
w/o ENQF
82.65
82.46
75.44
75.90
55.96
0.710
43.07
0.296
w/o MTF, ENQF
80.33
79.99
73.50
73.42
57.86
0.765
42.77
0.305
MVFA-ChatGLM3-6B (Ours)
84.62
84.59
77.27
79.01
59.19
0.736
45.34
0.284
TABLE IV: Module-level ablation results on CH-SIMS V2.0, where w/o denotes removing a module. The full model achieves the best results on the main classification-oriented metrics and on R 2 , while some ablated variants remain competitive on individual regression-related metrics such as Corr or MAE.
Model
MELD
CHERMA
Acc ↑
WF1 ↑
Acc ↑
WF1 ↑
w/o MTF
66.24
65.04
73.52
73.21
w/o ENQF
63.85
62.73
74.36
74.41
w/o MTF, ENQF
62.01
60.62
72.37
72.30
MVFA-ChatGLM3-6B (Ours)
67.36
66.03
74.66
74.62
TABLE V: Module-level ablation results on MELD and CHERMA, where w/o denotes removing a module. The setting w/o MTF, ENQF corresponds to an early-fusion baseline in which features from all modalities are symmetrically concatenated with textual prompts before being fed into the frozen LLM.
Model
Acc2 ↑
F1 ↑
Acc2-Weak ↑
Acc3 ↑
Acc5 ↑
Corr ↑
R 2 ↑
MAE ↓
w/o Max
82.92
82.70
75.40
77.10
57.37
0.727
46.32
0.281
w/o Mean
83.75
83.61
76.52
78.16
58.48
0.729
44.80
0.284
w/o Attn
82.63
82.33
75.15
76.42
56.73
0.728
45.00
0.286
MVFA-ChatGLM3-6B (Ours)
84.62
84.59
77.27
79.01
59.19
0.736
45.34
0.284
TABLE VI: Leave-one-out branch ablation of the proposed multi-view text modeling module on CH-SIMS V2.0. We remove the max-pooling, mean-pooling, and attention-pooling branches individually, denoted as w/o Max, w/o Mean, and w/o Attn.
Model
Acc ↑
WF1 ↑
Δ Acc
Δ WF1
w/o Max
66.73
65.58
-0.63
-0.45
w/o Mean
66.31
64.89
-1.05
-1.14
w/o Attn
66.47
65.31
-0.89
-0.72
MVFA-ChatGLM3-6B (Ours)
67.36
66.03
–
–
TABLE VII: Leave-one-out branch ablation of the proposed multi-view text modeling module on MELD. We remove the max-pooling, mean-pooling, and attention-pooling branches individually, denoted as w/o Max, w/o Mean, and w/o Attn, respectively. Results are averaged over five random seeds. Δ denotes the performance difference relative to the full MVFA model.
Modality
Acc ↑
WF1 ↑
Δ Acc
Δ WF1
V+A
47.21
35.35
-20.15
-30.68
T+V
66.62
65.62
-0.74
-0.41
T+A
66.81
65.77
-0.55
-0.26
T+V+A
67.36
66.03
–
–
TABLE VIII: Modality ablation results on MELD. T , V , and A denote the textual, visual, and audio modalities, respectively. Δ denotes the performance difference relative to the full tri-modal model.
n
Acc2 ↑
F1 ↑
Acc3 ↑
Acc5 ↑
Corr ↑
R 2 ↑
MAE ↓
2
83.15
83.07
77.87
57.85
0.725
43.02
0.291
4
84.62
84.59
79.01
59.19
0.736
45.34
0.284
8
82.61
82.49
77.18
56.56
0.728
46.46
0.286
16
81.20
80.71
75.57
55.45
0.725
45.34
0.291
TABLE IX: Sensitivity analysis of the pseudo-token number n in ENQF on CH-SIMS V2.0, where n denotes the number of pseudo-tokens. Results are averaged over five random seeds.
α
Acc2 ↑
F1 ↑
Acc3 ↑
Acc5 ↑
Corr ↑
R 2 ↑
MAE ↓
0.1
83.14
83.05
77.33
57.45
0.723
46.41
0.283
0.2
83.66
83.62
78.41
59.15
0.727
44.53
0.285
0.3
84.62
84.59
79.01
59.19
0.736
45.34
0.284
0.4
83.19
82.97
77.25
57.00
0.728
45.67
0.287
0.5
83.19
83.09
77.93
57.49
0.723
45.33
0.286
TABLE X: Sensitivity analysis of the enhancement factor α in ENQF on CH-SIMS V2.0. Results are averaged over five random seeds, and all other settings are kept unchanged.
Fig. 4: Qualitative cases from CH-SIMS V2.0. Each example shows the utterance, predicted sentiment score, and modality-wise labels. MVFA handles both explicit and contrastive/figurative sentiment cases.
Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in understanding complex multimodal content. However, their performance in sentiment analysis exhibits acute sensitivity to prompt design, rendering static, uniformly applied prompts inherently suboptimal for capturing the nuanced multimodal cues that vary across inputs. To address this limitation, we propose a Multimodal Adaptive Few-Shot Prompting (MAF) framework, which dynamically retrieves and integrates query-relevant demonstrations to elicit the sentiment reasoning capabilities of MLLMs in a context-sensitive manner. MAF constructs a demonstration retrieval module that holistically encodes facial expressions, scene context, and textual semantics, with a lip movement amplitude detection mechanism introduced for accurate speaker identification in multi-person scenarios. Departing from conventional fixed-weight fusion, a lightweight coefficient generation network is trained to output query-conditioned fusion weights in real time, enabling weighted aggregation of multimodal similarity scores to retrieve the top-K most informative demonstrations. Prediction stability is further enhanced through majority voting over multiple candidate outputs generated by the MLLM. Extensive experiments on public benchmark datasets demonstrate that MAF achieves substantial and consistent performance improvements over the corresponding backbone variants and remains competitive with strong multimodal sentiment-analysis baselines.
Hangling Xie
Nanjing University of Posts and Telecommunications, Wenyuan Road No.9, Xixia District, Nanjing City, Jiangsu Province, China
Multimodal sentiment analysis integrates textual, acoustic and visual cues, yet current language-model-based fusion methods typically leave prediction-layer branch allocation implicit. We introduce Branch-Calibrated Multimodal Language Fusion (BC-MLF), which explicitly models prediction-layer branch allocation through a Branch-Calibrated Task Head (BCHead), complemented by Fusion Token Contrastive Learning (FTCL) for sentiment-aware fusion-token regularization. FTCL organizes mean-pooled fusion-token representations according to continuous sentiment affinity, while BCHead combines fusion, text and audiovisual predictions through a lightweight sample-adaptive constrained mixture. Without modifying the fusion backbone, BC-MLF consistently improves the reproduced DeepMLF baseline and achieves the strongest results among the compared methods on CMU-MOSEI and CH-SIMS across classification and regression metrics. The controlled ablations show that sample-adaptive prediction-layer branch allocation consistently outperforms static branch aggregation. Code is available at https://github.com/sunyulin0421/BC-MLF.
Multimodal sentiment analysis (MSA) aims to predict sentiment polarity and intensity from heterogeneous inputs such as text, audio, and vision. While large language models (LLMs) offer strong semantic priors for MSA, effectively incorporating audio and visual signals effectively remains challenging. A key challenge is that audio and visual sentiment cues evolve over different temporal scales, yet many LLM-based methods compress these signals through shallow projection or coarse pooling before fusing them with text, which can weaken cross-modal alignment and erase fine-grained affective information. We propose MGSI, a multi-granularity sentiment integration framework for LLM-based MSA. MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends. It then refines non-text features through text-guided alignment, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples. The resulting multimodal representation is finally compressed into a small set of pseudo-tokens for efficient conditioning of a frozen LLM. Experiments on four public benchmarks show that MGSI substantially outperforms frozen-LLM baselines and remains competitive with strong multimodal methods. Further ablation and sensitivity analyses support the effectiveness of multi-granularity temporal modeling, text-guided refinement, and adaptive sentiment calibration.
Shanshan Lin, Yuesheng Wu, Chao Chen +4
Fuzhou University, Fuzhou, China · Harbin Institute of Technology (Shenzhen), Shenzhen, China · Jiangxia University, Fuzhou, China