Organizations: School of Computer Science, South China Normal University Guangzhou, Guangdong, China · School of Electronic and Information Engineering, South China University of Technology Guangzhou, Guangdong, China
Multimodal sentiment analysis aims to integrate textual, acoustic, and visual information for deep emotional understanding. Despite the progress of multimodal large language models (MLLMs) via supervised fine-tuning, their "black-box" nature hinders interpretability. While Chain-of-Thought (CoT) reasoning offers a potential remedy, it is constrained by high manual annotation costs and the inherent challenges of reinforcement learning (RL), such as reward sparsity and low exploration efficiency on hard samples. This paper presents C2F-Thinker, a framework that harmonizes coarse-to-fine structured reasoning with hint-guided RL through a two-stage progressive training pipeline. In the first stage, we conduct cold-start supervised fine-tuning using high-quality CoT data distilled from a larger teacher model, consisting of three distinct phases: polarity judgment, intermediate analysis, and fine-grained scoring. This equips the base model with a structured emotional reasoning paradigm. In the second stage, we introduce a hint-guided Group Relative Policy Optimization (GRPO) algorithm. By injecting correct initial polarity predictions as hints during the sampling process, the model is guided toward accurate reasoning paths, effectively mitigating cascading errors and enhancing the utilization of hard samples. Furthermore, a multi-faceted reward function incorporating classification, regression, and formatting constraints is designed to refine prediction accuracy while preserving interpretability. Experimental results demonstrate that C2F-Thinker achieves competitive performance on fine-grained sentiment regression tasks while significantly outperforming baselines in cross-domain generalization. This highlights its potential in building trustworthy and robust sentiment analysis systems for real-world applications.
Figures & tables
Fig. 1: Illustration of the gradient vanishing problem in GRPO when handling hard samples in MSA.
Fig. 2: The overall training pipeline for the proposed C2F-Thinker.
Model
CH-SIMS (OOD)
CH-SIMSv2 (OOD)
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
PandaGPT
35.2
60.6
75.1
75.1
0.488
0.488
35.6
63.9
72.9
72.4
0.453
0 545
Emotion-LLaMA
31.1
42.0
71.8
64.1
0.481
0.485
23.7
37.0
61.7
51.7
0.475
0.464
MiniCPM-o
33.7
62.6
70.5
71.6
0.481
0.575
51.9
70.8
75.4
75.5
0.335
0.636
Ola
38.9
69.8
77.5
78.2
0.431
0.629
48.6
76.1
80.0
80.1
0.331
0.682
VideoLLaMA2
37.9
65.0
70.2
71.4
0.442
0.610
48.2
74.3
77.3
77.3
0.332
0.678
TABLE I: Performance comparison on CH-SIMS (OOD) and CH-SIMS v2 (OOD) datasets. All models are trained on the CMU-MOSI dataset. The best results are highlighted in bold, and the runner-up results are indicated with underlines.
Model
CMU-MOSI (ID)
CMU-MOSEI (OOD)
Acc7↑
Acc2↑
F1↑
MAE↓
Corr↑
Acc7↑
Acc2↑
F1↑
MAE↓
Corr↑
PandaGPT
52.1
90.2
90.2
0.536
0.899
50.3
85.0
85.1
0.587
0.758
Emotion-LLaMA
40.7
86.1
86.2
0.800
0.764
41.1
83.4
83.2
2.481
0.026
MiniCPM-o
49.8
89.5
89.5
0.636
0.853
33.6
86.4
86.2
0.862
0.723
Ola
48.3
89.3
89.3
0.620
0.860
40.4
85.2
85.4
0.670
0.743
VideoLLaMA2
50.4
90.5
90.5
0.571
0.877
44.0
84.8
84.6
0.682
0.630
TABLE II: Performance comparison on CMU-MOSI (ID) and CMU-MOSEI (OOD) datasets. All models are trained on the CMU-MOSI dataset. The best results are highlighted in bold, and the runner-up results are indicated with underlines.
Model
CH-SIMS (ID)
CH-SIMSv2 (ID)
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
PandaGPT
38.3
58.4
77.2
74.7
0.431
0.537
40
57.6
70.4
67.9
0.403
0.550
Emotion-LLaMA
41.1
59.3
77.2
75.4
0.403
0.628
39.4
57.4
72.3
70.5
0.402
0.610
MiniCPM-o
48.8
68.7
82.5
80.5
0.350
0.695
39.1
68.5
76.6
75.0
0.420
0.700
Ola
48.4
67.8
81.6
80.2
0.406
0.646
39.8
70.7
77.6
76.6
0.460
0.645
VideoLLaMA2
52.1
74.4
81.6
82.3
0.388
0.733
43.3
76.4
80.9
80.9
0.494
0.726
TABLE III: Performance comparison on CH-SIMS (ID) and CH-SIMS v2 (ID) datasets. All models are trained on the CH-SIMS dataset. The best results are highlighted in bold, and the runner-up results are indicated with underlines.
Model
CMU-MOSI (OOD)
CMU-MOSEI (OOD)
Acc7↑
Acc2↑
F1↑
MAE↓
Corr↑
Acc7↑
Acc2↑
F1↑
MAE↓
Corr↑
PandaGPT
20.9
85.3
85.4
1.130
0.670
44.6
85.5
85.3
0.723
0.610
Emotion-LLaMA
25.5
79.2
79.4
1.078
0.593
38.5
80.7
80.5
1.224
0.025
MiniCPM-o
32.6
87.0
87.1
0.874
0.756
47.1
85.3
84.9
0.641
0.688
Ola
35.2
82.7
82.8
0.870
0.728
47.4
83.2
83.2
0.666
0.672
VideoLLaMA2
25.4
69.9
69.3
1.142
0.561
37.7
74.4
72.0
0.835
0.448
TABLE IV: Performance comparison on CMU-MOSI (OOD) and CMU-MOSEI (OOD) datasets. All models are trained on the CH-SIMS dataset. The best results are highlighted in bold, and the runner-up results are indicated with underlines.
Fig. 3: Comparison of reward curves during training for different configurations.
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
CoT(Zero-shot)
36.5
55.1
70.0
70.8
0.462
0.558
CoT(SFT)
52.7
76.4
86.7
86.9
0.345
0.752
CoT(SFT+RL)
54.3
77.2
87.5
87.8
0.331
0.780
TABLE V: Performance comparison of different training stages (Zero-shot, SFT, SFT+RL) on the CH-SIMS dataset.
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
w/o Hint
46.2
67.0
82.5
81.4
0.368
0.701
w/ Hint
54.3
77.2
87.5
87.8
0.331
0.780
TABLE VI: Ablation study on the effectiveness of the Hint mechanism on the CH-SIMS dataset.
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
w/o Hard
47.9
77.0
87.3
87.6
0.447
0.715
w/ Hard
54.3
77.2
87.5
87.8
0.331
0.780
TABLE VII: Ablation study on the necessity of hard samples during training on the CH-SIMS dataset.