Robust Image Emotion Classification via Multi-Level Affective Captioning
Authors: Zibo Zhou, Zhengjun Zhai, Huimin Chen
Organizations: School of Computer Science, Northwestern Polytechnical University, No.127 Youyi Xilu, Xi’an, 710072, Shaanxi, China · School of Cybersecurity, Northwestern Polytechnical University, No.127 Youyi Xilu, Xi’an, 710072, Shaanxi, China
Image emotion classification (IEC) has been studied for many years and has recently attracted increasing attention with the rapid development of deep learning. Although recent studies have benefited from knowledge embedded in pre-trained visual models, the performance is still limited by the "affective gap" between low-level visual features and high-level affective semantics. Psychological studies suggest that language can convey fine-grained and emotion-rich information. Inspired by this observation, we attempt to narrow the "affective gap" by introducing affective text. Specifically, we design a multi-level contrastive loss to detect emotional concepts from images and introduce an emotional-attribute-guided large language model approach to generate affective sentences. Then, a pre-trained language model is employed to combine emotional concepts with affective sentences for IEC. In addition, we introduce a contrastive loss with a semantic-similarity-based sampling strategy to address large intra-class differences and limited inter-class differences in affective datasets. Moreover, unlike many previous studies, we explicitly consider images containing embedded text. Extensive experiments on multiple benchmarks show that the proposed method helps narrow the "affective gap" and achieves competitive performance.
Figures & tables
Fig. 1: The pipeline of the proposed ACIEC.
Fig. 2: Example of the hierarchical structure of the VSO dataset.
Fig. 3: A template of an EA-CoT prompt.
Fig. 4: The sampling procedure of the proposed loss function.
Method
FI
EmotionROI
Twitter I
Twitter II
Number of classes
8
2
6
2
2
2
WSCNet
70.07
–
58.25
–
84.25
81.35
SOLVER
72.33
–
62.12
–
85.43
83.19
MSRCA
72.60
87.40
55.60
83.00
–
–
OEAN
73.44
–
64.14
–
85.59
–
DeepSentiBank
51.54
61.54
42.53
70.11
71.25
70.23
TABLE I: Comparison with state-of-the-art IEC methods on different IEC datasets.
Dataset
Category
Precision
Recall
F1
Specificity
FI
amusement
90.86
88.07
89.44
97.52
anger
78.14
72.09
74.99
98.86
awe
79.25
85.29
82.16
96.36
contentment
81.95
81.48
81.71
94.28
disgust
81.56
87.28
84.32
98.45
excitement
76.55
80.12
78.29
96.72
TABLE II: Class-wise evaluation metrics on different IEC datasets.
Fig. 5: Zero-shot MLLM prompt template for image emotion classification.
Model
FI(8)
EmotionROI(6)
LLaVA-v1.5
53.64
49.83
LLaVA-Next
61.59
53.37
Qwen2-VL-7B
58.16
51.68
Qwen2.5-VL-7B
67.12
57.74
GPT-4o
54.87
64.99
ACIEC
81.98
74.66
TABLE III: Comparison with zero-shot MLLM baselines on different IEC datasets.
Method
FI
EmotionROI
Twitter I
Twitter II
Number of classes
8
2
6
2
2
2
ANP
68.48
84.17
59.25
81.32
91.07
90.83
Sentence
69.74
85.12
57.89
79.41
89.96
89.75
w/o OCR
78.87
88.91
69.48
85.31
91.54
89.14
Obj
63.16
72.19
50.52
72.94
45.47
44.47
Sce
53.48
57.52
41.47
70.66
39.78
43.06
TABLE IV: Ablation results of different ACIEC variants on different IEC datasets.
Method
FI
EmotionROI
Twitter I
Twitter II
Number of classes
8
2
6
2
2
2
LLaVA-v1.5
79.96
94.12
71.85
90.02
92.11
90.24
Qwen2.5-VL-7B
80.84
94.63
72.92
90.67
92.56
90.71
ACIEC (GPT-4o)
81.98
95.68
74.66
92.14
94.54
91.86
TABLE V: Effect of different reasoning backbones on ACIEC.
Fig. 6: Confusion matrices for the FI and EmotionROI datasets.
Emotion label
Valid sentences
Sentences with emotion label
Amusement
4,539
1,178 (25.95%)
Anger
1,143
43 (3.76%)
Awe
2,914
509 (17.47%)
Contentment
5,069
368 (7.26%)
Disgust
1,488
10 (0.67%)
Excitement
2,490
466 (18.71%)
TABLE VI: Occurrence of exact emotion-label words in affective sentences on FI.
Model variant
Accuracy (%)
Macro-F1 (%)
ACIEC
81.98
79.27
ACIEC w/o emotion-label terms
81.86
79.18
TABLE VII: Emotion-label removal ablation on FI.
Test on
Accuracy (%)
mAP (%)
m-F1 (%)
w-F1 (%)
EmotionROI
79.84
83.96
78.98
80.39
Twitter I
79.38
84.06
78.64
79.63
Twitter II
80.47
85.28
80.39
81.25
TABLE VIII: Cross-dataset generalization results of ACIEC.
Dataset
ECE ↓
SCE ↓
NLL ↓
Brier Score ↓
FI
0.0265
0.0213
0.7513
0.3566
EmotionROI
0.0450
0.0381
0.7854
0.4033
Twitter I
0.0366
0.0502
0.3061
0.0911
Twitter II
0.0258
0.0369
0.2673
0.0847
TABLE IX: Confidence calibration results on different IEC datasets.
Method
Mean (%)
StdDev
t -test p
Wilcoxon p
Baseline
80.63
0.15
1.79×10−6
3.13×10−2
ACIEC
81.98
0.14
TABLE X: Statistical significance testing between ACIEC and the baseline.
Model
Params (M)
FLOPs (G)
Memory (GB)
ACIEC
85.65
21.76
11.84
TABLE XI: Computational cost of the proposed ACIEC framework.
TABLE XII: Qualitative comparison of affective sentences generated by EA-CoT and zero-shot prompting.
Method
Visual Consistency
Emotional Consistency
Overall MOS
Zero-shot
3.40 ± 0.37
3.28 ± 0.35
3.34 ± 0.36
Ours
4.10 ± 0.36
4.22 ± 0.34
4.16 ± 0.35
TABLE XIII: Human evaluation results of affective sentences.
Image
ANP
Affective Sentence
Ground Truth
Great adventure
A distant view of a beachfront amusement area with a large Ferris wheel and a tall tower ride, set against city buildings under a clear sky.
Amusement
Comfortable room
A warm-lit bedroom with two single beds, matching bedside lamps, and a curtained window, with a small table and chair set near the light.
Contentment
Dirty streets
Food scraps and litter are scattered across cracked pavement near a curb and a parked truck, creating a messy street-corner scene.
Disgust
Heavy clouds
A quiet shoreline reflecting the sunset, with dark clouds and distant mountains, calling forth serene reflection.
Awe
Talented student
Three graduates laugh together outdoors in their gowns and stoles, conveying joyful celebration in a milestone moment.
Excitement
TABLE XIV: Case studies with ANPs and affective sentences.
Fig. 7: t-SNE visualization of intermediate semantic features.
Dataset
Accuracy (%)
mAP (%)
m-F1 (%)
W-F1 (%)
Abstract
48.13
49.99
44.43
45.91
ArtPhoto
31.57
28.41
26.23
27.60
TABLE XV: Performance on abstract and artistic emotion datasets.
Dataset
Image
ANP
Affective Sentence
ArtPhoto
bumpy water
The bubbles catch the light in an interesting way, making the scene feel curious and a little magical.
bloody hands
The fiery colors and rough texture make the image feel full of energy and strong emotion.
Abstract
lonely road
Solitary journey on a vibrant path, bridging uncertainty with hope amidst an expansive, introspective horizon.
amazing paintings
An abstract dance of fiery reds and vibrant greens, where bold textures and dynamic strokes evoke passion, vitality, and the chaos of raw emotion.
TABLE XVI: Failure cases on abstract and artistic emotion datasets.
Emotion recognition uses language, voice, face, body movement, and physiological signals to infer emotional categories or dimensions such as valence and arousal.
Vision-Language Models (VLMs) have demonstrated strong capabilities in perception, yet holistic Affective Image Content Analysis (AICA), which integrates perception, reasoning, and generation into a unified framework, remains underexplored. To address this gap, we introduce AICA-Bench, a comprehensive benchmark with three core tasks: Emotion Understanding (EU), Emotion Reasoning (ER), and Emotion-Guided Content Generation (EGCG). We evaluate 23 VLMs and identify two major limitations: weak intensity calibration and shallow open-ended descriptions. To address these issues, we propose Grounded Affective Tree (GAT) Prompting, a training-free framework that combines visual scaffolding with hierarchical reasoning. Experiments show that GAT reduces intensity errors and improves descriptive depth, providing a strong baseline for future research on affective multimodal understanding and generation.
Dong She, Xianrong Yao, Liqun Chen +3
School of Future Technology,South China University of Technology,Guangzhou
Affective image editing (AIE) aims to evoke a target emotion while preserving the content and structure of a source image. Existing methods improve emotional expression or visual fidelity, but the target emotion alone does not reveal which source regions should be changed and which should remain intact, especially in complex scenes. We introduce Interpretable Emotion Contribution Learning for Affective Image Editing (IECLA), a framework that separates source-emotion attribution from target-aware editing. IECLA constructs signed object-level supervision through counterfactual interventions and trains a context-conditioned contribution predictor to localize evidence that supports or opposes the source emotion. At inference, an image-level evaluator estimates the source emotion directly from the input, while the predictor identifies editable and preservation evidence without repeated interventions. A target-aware editing agent then translates this evidence and the desired emotion into localized, content-aware operations for existing editors. Extensive evaluations show that IECLA predicts counterfactual contributions more faithfully on both in-domain and cross-dataset tests, while external VLM and independent human references support the quality of its localized evidence. Across six editing backbones, IECLA consistently strengthens target-emotion expression while preserving image content and structure, and user studies further show improved perceived target expression and naturalness.
Yijie Guo, Dexiang Hong, Weidong Chen +2
University of Science and Technology of China · Harbin Institute of Technology
Vision-language models (VLMs) show promise for large-scale affect inference from images, but their alignment with human affective ratings remains unclear. We benchmarked 16 proprietary and open-source VLMs on three validated affective image datasets: the International Affective Picture System, the Nencki Affective Picture System, and the Library of AI-Generated Affective Images. In a zero-shot setting, models performed two tasks: (i) top-emotion classification and (ii) prediction of human ratings on 1--7/9 Likert scales for discrete emotions and affective dimensions. We also evaluated rater-conditioned prompting on the LAI-GAI dataset using de-identified participant metadata. Best models showed strong discrete emotion classification (60%--80% on six-emotion tasks; 60%--75% on a 12-category task), with anger and surprise predicted least accurately. For continuous ratings, models aligned moderately to strongly with human judgments (r>0.75) but showed consistent biases, especially weaker performance on arousal and overestimation of response intensity. Rater-conditioned prompting had only small, inconsistent effects. Overall, VLMs captured broad affective patterns but sometimes lacked the nuance, although newer model generations performed better.
Filip Nowicki, Hubert Marciniak, Jakub Łączkowski +6
Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Poznań, Poland · Faculty of Computer Science and Information Technology, Universiti Malaya, Malaysia · Department of Computer Science and Engineering, Korea University, Korea +2