Organizations: School of Computer Science, South China Normal University Guangzhou, Guangdong, China · School of Electronic and Information Engineering, South China University of Technology Guangzhou, Guangdong, China
Multimodal sentiment analysis aims to integrate textual, acoustic, and visual information for deep emotional understanding. Despite the progress of multimodal large language models (MLLMs) via supervised fine-tuning, their "black-box" nature hinders interpretability. While Chain-of-Thought (CoT) reasoning offers a potential remedy, it is constrained by high manual annotation costs and the inherent challenges of reinforcement learning (RL), such as reward sparsity and low exploration efficiency on hard samples. This paper presents C2F-Thinker, a framework that harmonizes coarse-to-fine structured reasoning with hint-guided RL through a two-stage progressive training pipeline. In the first stage, we conduct cold-start supervised fine-tuning using high-quality CoT data distilled from a larger teacher model, consisting of three distinct phases: polarity judgment, intermediate analysis, and fine-grained scoring. This equips the base model with a structured emotional reasoning paradigm. In the second stage, we introduce a hint-guided Group Relative Policy Optimization (GRPO) algorithm. By injecting correct initial polarity predictions as hints during the sampling process, the model is guided toward accurate reasoning paths, effectively mitigating cascading errors and enhancing the utilization of hard samples. Furthermore, a multi-faceted reward function incorporating classification, regression, and formatting constraints is designed to refine prediction accuracy while preserving interpretability. Experimental results demonstrate that C2F-Thinker achieves competitive performance on fine-grained sentiment regression tasks while significantly outperforming baselines in cross-domain generalization. This highlights its potential in building trustworthy and robust sentiment analysis systems for real-world applications.
Figures & tables
Fig. 1: Illustration of the gradient vanishing problem in GRPO when handling hard samples in MSA.
Fig. 2: The overall training pipeline for the proposed C2F-Thinker.
Model
CH-SIMS (OOD)
CH-SIMSv2 (OOD)
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
PandaGPT
35.2
60.6
75.1
75.1
0.488
0.488
35.6
63.9
72.9
72.4
0.453
0 545
Emotion-LLaMA
31.1
42.0
71.8
64.1
0.481
0.485
23.7
37.0
61.7
51.7
0.475
0.464
MiniCPM-o
33.7
62.6
70.5
71.6
0.481
0.575
51.9
70.8
75.4
75.5
0.335
0.636
Ola
38.9
69.8
77.5
78.2
0.431
0.629
48.6
76.1
80.0
80.1
0.331
0.682
VideoLLaMA2
37.9
65.0
70.2
71.4
0.442
0.610
48.2
74.3
77.3
77.3
0.332
0.678
TABLE I: Performance comparison on CH-SIMS (OOD) and CH-SIMS v2 (OOD) datasets. All models are trained on the CMU-MOSI dataset. The best results are highlighted in bold, and the runner-up results are indicated with underlines.
Model
CMU-MOSI (ID)
CMU-MOSEI (OOD)
Acc7↑
Acc2↑
F1↑
MAE↓
Corr↑
Acc7↑
Acc2↑
F1↑
MAE↓
Corr↑
PandaGPT
52.1
90.2
90.2
0.536
0.899
50.3
85.0
85.1
0.587
0.758
Emotion-LLaMA
40.7
86.1
86.2
0.800
0.764
41.1
83.4
83.2
2.481
0.026
MiniCPM-o
49.8
89.5
89.5
0.636
0.853
33.6
86.4
86.2
0.862
0.723
Ola
48.3
89.3
89.3
0.620
0.860
40.4
85.2
85.4
0.670
0.743
VideoLLaMA2
50.4
90.5
90.5
0.571
0.877
44.0
84.8
84.6
0.682
0.630
TABLE II: Performance comparison on CMU-MOSI (ID) and CMU-MOSEI (OOD) datasets. All models are trained on the CMU-MOSI dataset. The best results are highlighted in bold, and the runner-up results are indicated with underlines.
Model
CH-SIMS (ID)
CH-SIMSv2 (ID)
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
PandaGPT
38.3
58.4
77.2
74.7
0.431
0.537
40
57.6
70.4
67.9
0.403
0.550
Emotion-LLaMA
41.1
59.3
77.2
75.4
0.403
0.628
39.4
57.4
72.3
70.5
0.402
0.610
MiniCPM-o
48.8
68.7
82.5
80.5
0.350
0.695
39.1
68.5
76.6
75.0
0.420
0.700
Ola
48.4
67.8
81.6
80.2
0.406
0.646
39.8
70.7
77.6
76.6
0.460
0.645
VideoLLaMA2
52.1
74.4
81.6
82.3
0.388
0.733
43.3
76.4
80.9
80.9
0.494
0.726
TABLE III: Performance comparison on CH-SIMS (ID) and CH-SIMS v2 (ID) datasets. All models are trained on the CH-SIMS dataset. The best results are highlighted in bold, and the runner-up results are indicated with underlines.
Model
CMU-MOSI (OOD)
CMU-MOSEI (OOD)
Acc7↑
Acc2↑
F1↑
MAE↓
Corr↑
Acc7↑
Acc2↑
F1↑
MAE↓
Corr↑
PandaGPT
20.9
85.3
85.4
1.130
0.670
44.6
85.5
85.3
0.723
0.610
Emotion-LLaMA
25.5
79.2
79.4
1.078
0.593
38.5
80.7
80.5
1.224
0.025
MiniCPM-o
32.6
87.0
87.1
0.874
0.756
47.1
85.3
84.9
0.641
0.688
Ola
35.2
82.7
82.8
0.870
0.728
47.4
83.2
83.2
0.666
0.672
VideoLLaMA2
25.4
69.9
69.3
1.142
0.561
37.7
74.4
72.0
0.835
0.448
TABLE IV: Performance comparison on CMU-MOSI (OOD) and CMU-MOSEI (OOD) datasets. All models are trained on the CH-SIMS dataset. The best results are highlighted in bold, and the runner-up results are indicated with underlines.
Fig. 3: Comparison of reward curves during training for different configurations.
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
CoT(Zero-shot)
36.5
55.1
70.0
70.8
0.462
0.558
CoT(SFT)
52.7
76.4
86.7
86.9
0.345
0.752
CoT(SFT+RL)
54.3
77.2
87.5
87.8
0.331
0.780
TABLE V: Performance comparison of different training stages (Zero-shot, SFT, SFT+RL) on the CH-SIMS dataset.
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
w/o Hint
46.2
67.0
82.5
81.4
0.368
0.701
w/ Hint
54.3
77.2
87.5
87.8
0.331
0.780
TABLE VI: Ablation study on the effectiveness of the Hint mechanism on the CH-SIMS dataset.
Acc5↑
Acc3↑
Acc2↑
F1↑
MAE↓
Corr↑
w/o Hard
47.9
77.0
87.3
87.6
0.447
0.715
w/ Hard
54.3
77.2
87.5
87.8
0.331
0.780
TABLE VII: Ablation study on the necessity of hard samples during training on the CH-SIMS dataset.
Multimodal Large Language Models (MLLMs) have shown remarkable progress in visual reasoning and understanding tasks but still struggle to capture the complexity and subjectivity of human emotions. Existing approaches based on supervised fine-tuning often suffer from limited generalization and poor interpretability, while reinforcement learning methods such as Group Relative Policy Optimization fail to align with the intrinsic characteristics of emotional cognition. To address these challenges, we propose Reflective Reinforcement Learning for Emotional Reasoning (EMO-R3), a framework designed to enhance the emotional reasoning ability of MLLMs. Specifically, we introduce Structured Emotional Thinking to guide the model to perform step-by-step emotional reasoning in a structured and interpretable manner, and design a Reflective Emotional Reward that enables the model to re-evaluate its reasoning based on visual-text consistency and emotional coherence. Extensive experiments demonstrate that EMO-R3 significantly improves both the interpretability and emotional intelligence of MLLMs, achieving superior performance across multiple visual emotional understanding benchmarks.
Yiyang Fang, Wenke Huang, Pei Fu +5
School of Computer Science, Wuhan University. · MiLM Plus, Xiaomi Inc.
While Aspect-based Sentiment Analysis (ABSA) systems have achieved high accuracy in identifying sentiment polarities, they often operate as "black boxes," lacking the explicit reasoning capabilities characteristic of human affective cognition. Humans do not merely categorize sentiment; they construct causal explanations for their judgments. To bridge this gap, we propose ABSA-R1, a large language model framework designed to mimic this ``reason-before-predict" cognitive process. By leveraging reinforcement learning (RL), ABSA-R1 learns to articulate the why behind the what, generating natural language justifications that ground its sentiment predictions. We introduce a Cognition-Aligned Reward Model (formerly sentiment-aware reward model) that enforces consistency between the generated reasoning path and the final emotional label. Furthermore, inspired by metacognitive monitoring, we implement a performance-driven rejection sampling strategy that selectively targets hard cases where the model's internal reasoning is uncertain or inconsistent. Experimental results on four benchmarks demonstrate that equipping models with this explicit reasoning capability not only enhances interpretability but also yields superior performance in sentiment classification and triplet extraction compared to non-reasoning baselines.
Shihao Zhang, Ziwei Wang, Jie Zhou +6
School of Computer Science and Technology, East China Normal University · Shanghai Qiji Zhifeng Co., Ltd. · Ocean University of China
Current approaches for Multimodal Sentiment Analysis (MSA) primarily leverage the knowledge and reasoning capabilities of parameter-heavy (Multimodal) LLMs for classification, overlooking autonomous multimodal sentiment reasoning generation in resource-constrained environments. In this paper, we focus on the Resource-Limited Joint Multimodal Sentiment Reasoning and Classification task, JMSRC, which simultaneously performs multimodal sentiment reasoning chain generation and sentiment classification only with a lightweight model. We propose a Multimodal Chain-of-Thought Reasoning Distillation model, MulCoT-RD, designed for JMSRC that employs a "Teacher-Assistant-Student" distillation paradigm to address deployment constraints in resource-limited environments. We first leverage a high-performance Multimodal Large Language Model (MLLM) to generate the initial reasoning dataset and train a medium-sized assistant model with a multi-task learning mechanism. A lightweight student model is jointly trained to perform efficient multimodal sentiment reasoning generation and classification. Extensive experiments on four datasets demonstrate that MulCoT-RD, with only 3B parameters, achieves strong performance on JMSRC while exhibiting robust generalization and enhanced interpretability.
Haonan Shangguan, Xiaocui Yang, Shi Feng +3
School of Computer Science and Engineering, Northeastern University, Shenyang, China