Robust Image Emotion Classification via Multi-Level Affective Captioning
Authors: Zibo Zhou, Zhengjun Zhai, Huimin Chen
Organizations: School of Computer Science, Northwestern Polytechnical University, No.127 Youyi Xilu, Xi’an, 710072, Shaanxi, China · School of Cybersecurity, Northwestern Polytechnical University, No.127 Youyi Xilu, Xi’an, 710072, Shaanxi, China
Image emotion classification (IEC) has been studied for many years and has recently attracted increasing attention with the rapid development of deep learning. Although recent studies have benefited from knowledge embedded in pre-trained visual models, the performance is still limited by the "affective gap" between low-level visual features and high-level affective semantics. Psychological studies suggest that language can convey fine-grained and emotion-rich information. Inspired by this observation, we attempt to narrow the "affective gap" by introducing affective text. Specifically, we design a multi-level contrastive loss to detect emotional concepts from images and introduce an emotional-attribute-guided large language model approach to generate affective sentences. Then, a pre-trained language model is employed to combine emotional concepts with affective sentences for IEC. In addition, we introduce a contrastive loss with a semantic-similarity-based sampling strategy to address large intra-class differences and limited inter-class differences in affective datasets. Moreover, unlike many previous studies, we explicitly consider images containing embedded text. Extensive experiments on multiple benchmarks show that the proposed method helps narrow the "affective gap" and achieves competitive performance.
Figures & tables
Fig. 1: The pipeline of the proposed ACIEC.
Fig. 2: Example of the hierarchical structure of the VSO dataset.
Fig. 3: A template of an EA-CoT prompt.
Fig. 4: The sampling procedure of the proposed loss function.
Method
FI
EmotionROI
Twitter I
Twitter II
Number of classes
8
2
6
2
2
2
WSCNet
70.07
–
58.25
–
84.25
81.35
SOLVER
72.33
–
62.12
–
85.43
83.19
MSRCA
72.60
87.40
55.60
83.00
–
–
OEAN
73.44
–
64.14
–
85.59
–
DeepSentiBank
51.54
61.54
42.53
70.11
71.25
70.23
TABLE I: Comparison with state-of-the-art IEC methods on different IEC datasets.
Dataset
Category
Precision
Recall
F1
Specificity
FI
amusement
90.86
88.07
89.44
97.52
anger
78.14
72.09
74.99
98.86
awe
79.25
85.29
82.16
96.36
contentment
81.95
81.48
81.71
94.28
disgust
81.56
87.28
84.32
98.45
excitement
76.55
80.12
78.29
96.72
TABLE II: Class-wise evaluation metrics on different IEC datasets.
Fig. 5: Zero-shot MLLM prompt template for image emotion classification.
Model
FI(8)
EmotionROI(6)
LLaVA-v1.5
53.64
49.83
LLaVA-Next
61.59
53.37
Qwen2-VL-7B
58.16
51.68
Qwen2.5-VL-7B
67.12
57.74
GPT-4o
54.87
64.99
ACIEC
81.98
74.66
TABLE III: Comparison with zero-shot MLLM baselines on different IEC datasets.
Method
FI
EmotionROI
Twitter I
Twitter II
Number of classes
8
2
6
2
2
2
ANP
68.48
84.17
59.25
81.32
91.07
90.83
Sentence
69.74
85.12
57.89
79.41
89.96
89.75
w/o OCR
78.87
88.91
69.48
85.31
91.54
89.14
Obj
63.16
72.19
50.52
72.94
45.47
44.47
Sce
53.48
57.52
41.47
70.66
39.78
43.06
TABLE IV: Ablation results of different ACIEC variants on different IEC datasets.
Method
FI
EmotionROI
Twitter I
Twitter II
Number of classes
8
2
6
2
2
2
LLaVA-v1.5
79.96
94.12
71.85
90.02
92.11
90.24
Qwen2.5-VL-7B
80.84
94.63
72.92
90.67
92.56
90.71
ACIEC (GPT-4o)
81.98
95.68
74.66
92.14
94.54
91.86
TABLE V: Effect of different reasoning backbones on ACIEC.
Fig. 6: Confusion matrices for the FI and EmotionROI datasets.
Emotion label
Valid sentences
Sentences with emotion label
Amusement
4,539
1,178 (25.95%)
Anger
1,143
43 (3.76%)
Awe
2,914
509 (17.47%)
Contentment
5,069
368 (7.26%)
Disgust
1,488
10 (0.67%)
Excitement
2,490
466 (18.71%)
TABLE VI: Occurrence of exact emotion-label words in affective sentences on FI.
Model variant
Accuracy (%)
Macro-F1 (%)
ACIEC
81.98
79.27
ACIEC w/o emotion-label terms
81.86
79.18
TABLE VII: Emotion-label removal ablation on FI.
Test on
Accuracy (%)
mAP (%)
m-F1 (%)
w-F1 (%)
EmotionROI
79.84
83.96
78.98
80.39
Twitter I
79.38
84.06
78.64
79.63
Twitter II
80.47
85.28
80.39
81.25
TABLE VIII: Cross-dataset generalization results of ACIEC.
Dataset
ECE ↓
SCE ↓
NLL ↓
Brier Score ↓
FI
0.0265
0.0213
0.7513
0.3566
EmotionROI
0.0450
0.0381
0.7854
0.4033
Twitter I
0.0366
0.0502
0.3061
0.0911
Twitter II
0.0258
0.0369
0.2673
0.0847
TABLE IX: Confidence calibration results on different IEC datasets.
Method
Mean (%)
StdDev
t -test p
Wilcoxon p
Baseline
80.63
0.15
1.79×10−6
3.13×10−2
ACIEC
81.98
0.14
TABLE X: Statistical significance testing between ACIEC and the baseline.
Model
Params (M)
FLOPs (G)
Memory (GB)
ACIEC
85.65
21.76
11.84
TABLE XI: Computational cost of the proposed ACIEC framework.
TABLE XII: Qualitative comparison of affective sentences generated by EA-CoT and zero-shot prompting.
Method
Visual Consistency
Emotional Consistency
Overall MOS
Zero-shot
3.40 ± 0.37
3.28 ± 0.35
3.34 ± 0.36
Ours
4.10 ± 0.36
4.22 ± 0.34
4.16 ± 0.35
TABLE XIII: Human evaluation results of affective sentences.
Image
ANP
Affective Sentence
Ground Truth
Great adventure
A distant view of a beachfront amusement area with a large Ferris wheel and a tall tower ride, set against city buildings under a clear sky.
Amusement
Comfortable room
A warm-lit bedroom with two single beds, matching bedside lamps, and a curtained window, with a small table and chair set near the light.
Contentment
Dirty streets
Food scraps and litter are scattered across cracked pavement near a curb and a parked truck, creating a messy street-corner scene.
Disgust
Heavy clouds
A quiet shoreline reflecting the sunset, with dark clouds and distant mountains, calling forth serene reflection.
Awe
Talented student
Three graduates laugh together outdoors in their gowns and stoles, conveying joyful celebration in a milestone moment.
Excitement
TABLE XIV: Case studies with ANPs and affective sentences.
Fig. 7: t-SNE visualization of intermediate semantic features.
Dataset
Accuracy (%)
mAP (%)
m-F1 (%)
W-F1 (%)
Abstract
48.13
49.99
44.43
45.91
ArtPhoto
31.57
28.41
26.23
27.60
TABLE XV: Performance on abstract and artistic emotion datasets.
Dataset
Image
ANP
Affective Sentence
ArtPhoto
bumpy water
The bubbles catch the light in an interesting way, making the scene feel curious and a little magical.
bloody hands
The fiery colors and rough texture make the image feel full of energy and strong emotion.
Abstract
lonely road
Solitary journey on a vibrant path, bridging uncertainty with hope amidst an expansive, introspective horizon.
amazing paintings
An abstract dance of fiery reds and vibrant greens, where bold textures and dynamic strokes evoke passion, vitality, and the chaos of raw emotion.
TABLE XVI: Failure cases on abstract and artistic emotion datasets.
Emotion recognition uses language, voice, face, body movement, and physiological signals to infer emotional categories or dimensions such as valence and arousal.
Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Poznań, Poland · Faculty of Computer Science and Information Technology, Universiti Malaya, Malaysia · Department of Computer Science and Engineering, Korea University, Korea +2