MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition

Pengfei Shao; Jisheng Dang; Jiawen Fang; Ning Liu; Wencan Zhang; Bimei Wang; Jingwen Zhao; Jianhuang Lai

cs.AISep 5, 2026

MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition

Authors: Pengfei Shao, Jisheng Dang, Jiawen Fang, Ning Liu, Wencan Zhang, Bimei Wang, Jingwen Zhao, Jianhuang Lai, +2 more

Organizations: School of Information Science and Technology, Guangdong University of Foreign Studies, Guangdong, China · School of Information Science and Engineering, Lanzhou University, Lanzhou, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangdong, China · School of Computing, National University of Singapore, Singapore 119077 · Cloud and AI BU, Huawei, Shenzhen, Guangdong 518129, China

Abstract

Multimodal sentiment analysis and emotion recognition in conversations demand effective modeling of heterogeneous interactions across textual, acoustic, and visual modalities. Although large language models (LLMs) offer powerful language understanding, adapting them to multimodal affective computing remains challenging: full-model fine-tuning is computationally prohibitive, while many existing lightweight adapters fail to preserve rich textual cues during cross-modal fusion. To address these limitations, we propose the multi-view text-guided multimodal fusion adapter (MVFA), a parameter-efficient framework that augments frozen LLMs with strong multimodal reasoning capability. MVFA first constructs complementary text views via max pooling, mean pooling, and attention pooling; these views then guide cross-modal interactions with audio and visual features. The fused multimodal representations are subsequently compressed into a compact set of learnable pseudo-tokens through an Enhanced Q-Former Fusion Module. Using ChatGLM3-6B-base as the primary backbone, we further validate MVFA on LLaMA2-7B and Qwen3-8B to examine its portability across multiple frozen LLM backbones. MVFA is evaluated on three challenging datasets: CH-SIMS V2.0, MELD, and CHERMA. Experimental results demonstrate that MVFA achieves state-of-the-art performance on key metrics while updating only a small fraction of parameters. Specifically, it attains 84.62% Acc2 and 84.59% F1 on CH-SIMS V2.0, 67.36% Acc and 66.03% WF1 on MELD, and 74.66% Acc on CHERMA. These findings establish multi-view text-guided fusion as an effective and scalable paradigm for parameter-efficient multimodal LLM adaptation in affective computing. The code is publicly available at https://github.com/Overwhelm1208/MVFA.

Figures & tables

Fig. 1: An example illustrating that a single compressed text view may fail to preserve complementary affective cues (e.g., salient lexical expressions, discourse-level contrast, and context-relevant tokens) during multimodal fusion.

Fig. 2: Overview of the proposed MVFA framework. Lightweight encoders first extract representations from the audio and visual modalities. The MVFA then constructs complementary text views through max pooling, mean pooling, and attention pooling, progressively fuses them to guide cross-modal interaction, and compresses the resulting multimodal representation into a small set of pseudo-tokens via the Enhanced Q-Former Fusion Module (ENQF). These pseudo-tokens are concatenated with the raw text embeddings $T$ and the prompt-augmented text embeddings $T_{p}$ to form the input sequence $X_{\text{input}}=[P;T;T_{p}]$ , which is fed into a frozen LLM for autoregressive sentiment or emotion prediction. Only the MVFA is trained, while the LLM backbone remains frozen.

Model	Acc2 $\uparrow$	F1 $\uparrow$	Acc2-Weak $\uparrow$	Corr $\uparrow$	R 2 $\uparrow$	MAE $\downarrow$
TFN [ 3 ]	76.51	76.31	66.27	0.667	35.90	0.323
LMF [ 4 ]	77.05	77.02	69.34	0.638	40.64	0.343
MFN [ 25 ]	75.27	75.24	66.46	0.606	32.26	0.355
MULT [ 26 ]	79.50	79.59	69.61	0.703	47.15	0.317
Self-MM [ 14 ]	79.01	78.89	71.87	0.640	29.36	0.335
MISA [ 15 ]	80.53	80.63	70.50	0.725	50.59	0.314

TABLE I: MSA results on CH-SIMS V2.0. Acc2 and F1 are computed under the non-positive/positive binarization rule, and Acc2-Weak is evaluated on the subset of samples whose ground-truth sentiment scores fall in the range [-0.4, 0.4]. Results marked with * are cited from [ 10 ] , results marked with

\dagger

are reproduced on our experimental platform, and the remaining results are taken from the corresponding original papers.

Model	Happiness $\uparrow$	Sadness $\uparrow$	Fear $\uparrow$	Anger $\uparrow$	Surprise $\uparrow$	Disgust $\uparrow$	Neutrality $\uparrow$	Acc $\uparrow$
TFN [ 3 ]	74.91	75.56	66.15	74.41	66.29	43.34	65.60	68.37
LMF [ 4 ]	74.52	75.83	66.73	74.55	65.08	45.70	65.64	68.23
EFT [ 11 ]	74.98	76.88	67.32	74.85	66.73	47.48	64.60	68.72
LFT [ 11 ]	75.07	76.29	66.80	74.88	66.67	47.74	65.97	69.05
LFMIM [ 11 ]	76.60	77.83	69.44	75.32	69.83	50.20	68.24	70.54
MULT [ 26 ]	76.18	76.88	67.36	74.85	68.18	46.96	65.26	69.24

TABLE II: Fine-grained emotion recognition results on the CHERMA dataset. We report F1 scores for each emotion category together with overall accuracy (Acc). Results marked with

\dagger

are reproduced on our experimental platform.

Fig. 3: Row-normalized confusion matrices on CHERMA for (a) MSE-Adapter and (b) MVFA, both based on ChatGLM3-6B-base. MVFA yields a more concentrated diagonal pattern, especially for difficult classes such as Disgust, Fear, and Surprise.

Model	Acc $\uparrow$	WF1 $\uparrow$
LMF [ 4 ]	61.15	58.30
TFN [ 3 ]	60.70	57.74
GraphCFC [ 7 ]	61.42	58.86
MM-DFN [ 29 ]	62.49	59.46
EmoCaps [ 30 ]	–	64.00
GA2MIF [ 31 ]	61.65	58.94

TABLE III: Emotion recognition results on the MELD dataset. We report overall accuracy (Acc) and weighted F1 (WF1). Results marked with

\dagger

are reproduced on our experimental platform, and the remaining results are taken from the corresponding original papers.

Model	Acc2 $\uparrow$	F1 $\uparrow$	Acc2-Weak $\uparrow$	Acc3 $\uparrow$	Acc5 $\uparrow$	Corr $\uparrow$	R 2 $\uparrow$	MAE $\downarrow$
w/o MTF	82.92	82.75	75.20	77.45	57.68	0.734	42.77	0.289
w/o ENQF	82.65	82.46	75.44	75.90	55.96	0.710	43.07	0.296
w/o MTF, ENQF	80.33	79.99	73.50	73.42	57.86	0.765	42.77	0.305
MVFA-ChatGLM3-6B (Ours)	84.62	84.59	77.27	79.01	59.19	0.736	45.34	0.284

TABLE IV: Module-level ablation results on CH-SIMS V2.0, where w/o denotes removing a module. The full model achieves the best results on the main classification-oriented metrics and on R 2 , while some ablated variants remain competitive on individual regression-related metrics such as Corr or MAE.

Model	MELD		CHERMA
Model	Acc $\uparrow$	WF1 $\uparrow$	Acc $\uparrow$	WF1 $\uparrow$
w/o MTF	66.24	65.04	73.52	73.21
w/o ENQF	63.85	62.73	74.36	74.41
w/o MTF, ENQF	62.01	60.62	72.37	72.30
MVFA-ChatGLM3-6B (Ours)	67.36	66.03	74.66	74.62

TABLE V: Module-level ablation results on MELD and CHERMA, where w/o denotes removing a module. The setting w/o MTF, ENQF corresponds to an early-fusion baseline in which features from all modalities are symmetrically concatenated with textual prompts before being fed into the frozen LLM.

Model	Acc2 $\uparrow$	F1 $\uparrow$	Acc2-Weak $\uparrow$	Acc3 $\uparrow$	Acc5 $\uparrow$	Corr $\uparrow$	R 2 $\uparrow$	MAE $\downarrow$
w/o Max	82.92	82.70	75.40	77.10	57.37	0.727	46.32	0.281
w/o Mean	83.75	83.61	76.52	78.16	58.48	0.729	44.80	0.284
w/o Attn	82.63	82.33	75.15	76.42	56.73	0.728	45.00	0.286
MVFA-ChatGLM3-6B (Ours)	84.62	84.59	77.27	79.01	59.19	0.736	45.34	0.284

TABLE VI: Leave-one-out branch ablation of the proposed multi-view text modeling module on CH-SIMS V2.0. We remove the max-pooling, mean-pooling, and attention-pooling branches individually, denoted as w/o Max, w/o Mean, and w/o Attn.

Model	Acc $\uparrow$	WF1 $\uparrow$	$\Delta$ Acc	$\Delta$ WF1
w/o Max	66.73	65.58	-0.63	-0.45
w/o Mean	66.31	64.89	-1.05	-1.14
w/o Attn	66.47	65.31	-0.89	-0.72
MVFA-ChatGLM3-6B (Ours)	67.36	66.03	–	–

TABLE VII: Leave-one-out branch ablation of the proposed multi-view text modeling module on MELD. We remove the max-pooling, mean-pooling, and attention-pooling branches individually, denoted as w/o Max, w/o Mean, and w/o Attn, respectively. Results are averaged over five random seeds.

\Delta

denotes the performance difference relative to the full MVFA model.

Modality	Acc $\uparrow$	WF1 $\uparrow$	$\Delta$ Acc	$\Delta$ WF1
$V+A$	47.21	35.35	-20.15	-30.68
$T+V$	66.62	65.62	-0.74	-0.41
$T+A$	66.81	65.77	-0.55	-0.26
$T+V+A$	67.36	66.03	–	–

TABLE VIII: Modality ablation results on MELD.

T

V

, and

A

denote the textual, visual, and audio modalities, respectively.

\Delta

denotes the performance difference relative to the full tri-modal model.

$n$	Acc2 $\uparrow$	F1 $\uparrow$	Acc3 $\uparrow$	Acc5 $\uparrow$	Corr $\uparrow$	R 2 $\uparrow$	MAE $\downarrow$
2	83.15	83.07	77.87	57.85	0.725	43.02	0.291
4	84.62	84.59	79.01	59.19	0.736	45.34	0.284
8	82.61	82.49	77.18	56.56	0.728	46.46	0.286
16	81.20	80.71	75.57	55.45	0.725	45.34	0.291

TABLE IX: Sensitivity analysis of the pseudo-token number

n

in ENQF on CH-SIMS V2.0, where

n

denotes the number of pseudo-tokens. Results are averaged over five random seeds.

$\alpha$	Acc2 $\uparrow$	F1 $\uparrow$	Acc3 $\uparrow$	Acc5 $\uparrow$	Corr $\uparrow$	R 2 $\uparrow$	MAE $\downarrow$
0.1	83.14	83.05	77.33	57.45	0.723	46.41	0.283
0.2	83.66	83.62	78.41	59.15	0.727	44.53	0.285
0.3	84.62	84.59	79.01	59.19	0.736	45.34	0.284
0.4	83.19	82.97	77.25	57.00	0.728	45.67	0.287
0.5	83.19	83.09	77.93	57.49	0.723	45.33	0.286

TABLE X: Sensitivity analysis of the enhancement factor

\alpha

in ENQF on CH-SIMS V2.0. Results are averaged over five random seeds, and all other settings are kept unchanged.

Fig. 4: Qualitative cases from CH-SIMS V2.0. Each example shows the utterance, predicted sentiment score, and modality-wise labels. MVFA handles both explicit and contrastive/figurative sentiment cases.

Explore similar work

MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs
Jun 14, 2026Hangling XieMultimodal ICL Multimodal Large Language Models
Nanjing University of Posts and Telecommunications, Wenyuan Road No.9, Xixia District, Nanjing City, Jiangsu Province, China
Prediction-Layer Branch Calibration for Multimodal Sentiment Analysis
Sep 29, 2026Yulin Sun, Kele Xu, Yong DouMultimodal Sentiment Analysis Multimodal Fusion
National University of Defense Technology
Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysis
Aug 17, 2026Shanshan Lin, Yuesheng Wu, Chao Chen +4Sentiment Analysis Multimodal Sentiment Analysis
Fuzhou University, Fuzhou, China · Harbin Institute of Technology (Shenzhen), Shenzhen, China · Jiangxia University, Fuzhou, China

Jun 14, 2026J/K move · Enter open · S save

MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs

Hangling Xie

Nanjing University of Posts and Telecommunications, Wenyuan Road No.9, Xixia District, Nanjing City, Jiangsu Province, China

Multimodal ICL Multimodal Large Language Models Multimodal Sentiment Analysis Multimodal Fusion

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in understanding complex multimodal content. However, their performance in sentiment analysis exhibits acute sensitivity to prompt design, rendering static, uniformly applied prompts inherently suboptimal for capturing the nuanced multimodal cues that vary across inputs. To address this limitation, we propose a Multimodal Adaptive Few-Shot Prompting (MAF) framework, which dynamically retrieves and integrates query-relevant demonstrations to elicit the sentiment reasoning capabilities of MLLMs in a context-sensitive manner. MAF constructs a demonstration retrieval module that holistically encodes facial expressions, scene context, and textual semantics, with a lip movement amplitude detection mechanism introduced for accurate speaker identification in multi-person scenarios. Departing from conventional fixed-weight fusion, a lightweight coefficient generation network is trained to output query-conditioned fusion weights in real time, enabling weighted aggregation of multimodal similarity scores to retrieve the top-K most informative demonstrations. Prediction stability is further enhanced through majority voting over multiple candidate outputs generated by the MLLM. Extensive experiments on public benchmark datasets demonstrate that MAF achieves substantial and consistent performance improvements over the corresponding backbone variants and remains competitive with strong multimodal sentiment-analysis baselines.

Abstract

Explore similar work

MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs

Prediction-Layer Branch Calibration for Multimodal Sentiment Analysis

Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysis