Vision-Guided Text Prompt Tuning for Multimodal Sentiment Analysis
Authors: Xiaoran Kou, Jingyi Wu, Peng Sun, Yang Liu, Hong Chen
Organizations: College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai 200433, China · Division of Natural and Applied Sciences, Duke Kunshan University, Suzhou 215316, China
Multimodal sentiment analysis requires effective modeling of both verbal semantics and non-verbal affective cues. A central challenge is to calibrate text-centered sentiment understanding with visual facial evidence in a controlled, adaptive, and parameter-efficient manner. Text usually serves as the semantic anchor, whereas visual cues provide complementary evidence for ambiguous or implicit expressions; however, indiscriminate fusion may introduce visual noise and distort textual semantics. Moreover, fully fine-tuning large visual and textual encoders is costly and prone to overfitting on limited and scenario-dependent MSA benchmarks. To address these issues, we propose Vision-Guided Text Prompt Tuning (VG-TPT), which formulates visual-text sentiment modeling as controllable visual calibration of frozen text representations. VG-TPT injects visual affective cues into a frozen BERT encoder through layer-wise adaptive prompts, rather than relying on late-stage feature fusion or full backbone tuning. A co-guided router composes prompts from a trainable prompt bank according to both the evolving text state and the visual guidance feature, enabling sample-specific and layer-specific modulation. Experiments on CMU-MOSEI and CMU-MOSI show that VG-TPT consistently improves over text-only baselines and achieves competitive or superior performance compared with several full-modality methods, while updating only 2.4M trainable parameters. The code is available at https://github.com/ma-tubu/VG-TPT.
Figures & tables
Fig. 1: Motivation of VG-TPT. Text alone may lead to an ambiguous or incorrect sentiment prediction, while facial expressions provide fine-grained affective cues to refine and disambiguate text-centered sentiment interpretation.
Fig. 2: Overall architecture of VG-TPT. A frozen visual encoder extracts the visual guidance feature gV , which is injected into the BERT-based textual encoder through layer-wise adaptive prompts. The final vision-guided text representation is used for sentiment prediction.
Fig. 3: Illustration of the co-guided adaptive prompt generation mechanism. The current text state and the visual guidance feature are jointly used to produce routing scores over a trainable prompt basis bank. The routing scores softly activate different prompt bases and compose a layer-specific adaptive prompt.
Method
Mod.
Acc-7 ↑
Acc-2 ↑
F1 ↑
MAE ↓
BERT [ 27 ]
T
49.22
82.12
82.81
0.557
XLNet [ 27 ]
T
51.16
83.32
83.09
0.569
V-Transformer [ 2 ]
V
43.5
66.4
69.3
0.759
EF-LSTM [ 26 ]
A+V+T
46.3
76.1
75.9
0.594
MulT [ 2 ]
A+V+T
50.7
81.6
81.6
0.591
MMIM [ 13 ]
A+V+T
52.6
81.5
81.3
0.578
TABLE I: Main results on CMU-MOSEI. A, V, and T denote acoustic, visual, and textual modalities, respectively. ↑ / ↓ indicates that higher/lower values are better. Boldface denotes the best performance, and underlining denotes the second-best performance.
Method
Mod.
Acc-7 ↑
Acc-2 ↑
F1 ↑
MAE ↓
BERT [ 27 ]
T
42.17
83.60
84.80
0.731
XLNet [ 27 ]
T
44.89
84.52
86.74
0.723
V-Transformer [ 2 ]
V
–
–
–
–
EF-LSTM [ 26 ]
A+V+T
31.0
73.6
74.5
1.420
MulT [ 2 ]
A+V+T
39.1
81.1
81.0
0.889
MMIM [ 13 ]
A+V+T
45.9
83.4
83.4
0.777
TABLE II: Main results on CMU-MOSI. Notation is consistent with Table I .
Method
Acc-7 ↑
Acc-2 ↑
F1 ↑
MAE ↓
w/o Adaptive Prompt
50.6
83.4
83.2
0.586
w/o Layer-wise Updating
51.1
83.8
83.7
0.578
w/o Text State
50.9
83.6
83.4
0.582
w/o Prompt Selection
51.4
84.0
83.8
0.574
Full Model
52.2
84.6
84.5
0.565
TABLE III: Ablation study on CMU-MOSEI. Boldface denotes the best result, and underlining denotes the second-best result.
Method
Params (M)
Ratio
Acc-7 ↑
Acc-2 ↑
BERT full fine-tuning [ 27 ]
109
100%
49.22
82.12
MulT [ 2 ]
–
100%
50.7
81.6
VG-TPT
2.4
1.2%
52.2
84.6
TABLE IV: Parameter efficiency comparison on CMU-MOSEI. Params denotes the number of trainable parameters in millions (M), and Trainable Ratio denotes the percentage of trainable parameters relative to the total parameters of each model. Bold indicates VG-TPT results.
School of Information Science and Technology, Guangdong University of Foreign Studies, Guangdong, China · School of Information Science and Engineering, Lanzhou University, Lanzhou, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangdong, China +2