Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models
Authors: Filip Nowicki, Hubert Marciniak, Jakub Łączkowski, Bartosz Wilczek, Krzysztof Jassem, Tomasz Górecki, Vimala Balakrishnan, Desmond C. Ong, Maciej Behnke
Organizations: Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Poznań, Poland · Faculty of Computer Science and Information Technology, Universiti Malaya, Malaysia · Department of Computer Science and Engineering, Korea University, Korea · Department of Psychology, University of Texas at Austin, USA · Cognitive Neuroscience Center, Adam Mickiewicz University, Poznań, Poland
Vision-language models (VLMs) show promise for large-scale affect inference from images, but their alignment with human affective ratings remains unclear. We benchmarked 16 proprietary and open-source VLMs on three validated affective image datasets: the International Affective Picture System, the Nencki Affective Picture System, and the Library of AI-Generated Affective Images. In a zero-shot setting, models performed two tasks: (i) top-emotion classification and (ii) prediction of human ratings on 1--7/9 Likert scales for discrete emotions and affective dimensions. We also evaluated rater-conditioned prompting on the LAI-GAI dataset using de-identified participant metadata. Best models showed strong discrete emotion classification (60%--80% on six-emotion tasks; 60%--75% on a 12-category task), with anger and surprise predicted least accurately. For continuous ratings, models aligned moderately to strongly with human judgments (r>0.75) but showed consistent biases, especially weaker performance on arousal and overestimation of response intensity. Rater-conditioned prompting had only small, inconsistent effects. Overall, VLMs captured broad affective patterns but sometimes lacked the nuance, although newer model generations performed better.
Figures & tables
Fig. 1: Comparison of mean emotion ratings across datasets. Each panel displays a dot plot comparing the mean predicted emotion ratings from the models with the human baseline. The y -axis lists the emotions, and the x -axis represents the mean rating. Each marker shape and color corresponds to a specific model. Panels show results for (a) LAI-GAI, (b) NAPS, and (c) IAPS.
Fig. S1: The exact prompt template used for the zero-shot emotion classification task.
Fig. S2: The prompt template used for the affective dimension rating task. The scale variable was adjusted (e.g., 1-7 or 1-9) to match the ground truth of the specific dataset being evaluated.
Fig. S3: The prompt templates for the three Context-Aware experimental conditions. The core instructions remained constant, while the Context Description and Rater Context Block varied by condition.
Emotion
Definition
Amusement
An emotional experience associated with opportunities for humor and events that violate expectations.
Anger
An emotional experience characterized by antagonism toward someone or something perceived as deliberately harmful or unfair.
Attachment love
An emotional experience tied to opportunities for affiliation, interdependence, and intimacy.
Awe
An emotional experience triggered by stimuli that are vast and beyond the ordinary frame of reference, often involving wonder, admiration, and a sense of being small in the presence of something greater than oneself.
Craving
An emotional experience associated with specific sensory pleasure that aids in distinguishing nutritional foods from nutritionally neutral objects and toxins.
Disgust
An emotional experience involving strong aversion or repulsion toward something perceived as offensive, contaminating, or unpleasant.
TABLE SI: Emotion definitions used for the LAI-GAI dataset in the classification task (12 discrete emotion labels).
Dimension
Definition
Positive
Feelings of pleasure, satisfaction, or well-being, often associated with favorable situations.
Negative
Feelings of discomfort, displeasure, or distress, often associated with unfavorable situations.
Calm
A sense of tranquility and composure, with low levels of emotional arousal or agitation.
Aroused
Heightened physiological and psychological activity, such as increased heart rate, energy, or alertness.
Motivated to approach
An emotional drive to pursue a desired goal, object, or situation, often associated with curiosity, attraction, or reward.
Motivated to avoid
An emotional drive to move away from an undesired or threatening goal, object, or situation, often associated with fear, discomfort, or aversion.
TABLE SII: Affective dimension definitions for the LAI-GAI regression task, rated alongside the 12 discrete emotions in Table SI .
Label
Dataset
Definition
Discrete emotions (shared across IAPS and NAPS)
Happiness
Both
A positive state characterized by feelings of joy, contentment, and wellbeing. It often arises in response to pleasant experiences and is typically accompanied by a sense of lightness, warmth, and optimism.
Surprise
Both
The response to a sudden, unexpected event.
Sadness
Both
Feelings of disappointment, grief, or hopelessness.
Anger
Both
Antagonism toward someone or something perceived as deliberately harmful or unfair.
Disgust
Both
Strong aversion or repulsion toward something perceived as offensive, contaminating, or unpleasant.
TABLE SIII: Scales definitions for the IAPS and NAPS datasets. The discrete emotions were used in both classification and regression, whereas valence and arousal were used only in regression (adapted to match the original dataset instructions).
#
Category
Modification
Category 1: Definition Presentation (User Prompt)
1
Definition removed
The definition line (“Definition: {emotion_name} is {definition} ”) is removed from the user prompt entirely. The model receives only the emotion name and scale.
2
Synonyms instead of definition
The definition line is replaced with “Synonyms/related terms are {definition} ”, reframing the definitional cue as a set of related terms.
Category 2: Assigned Persona (System Prompt)
3
Study participant
The persona “You are an affective scientist” is replaced with “You are a study participant”.
4
Psychologist
Replaced with “You are a psychologist”.
TABLE SIV: Summary of the 13 prompt variations tested for robustness of the affective dimension prediction task. Each variation modifies a single aspect of the base prompt (Figure S2 ). All other prompt components remain unchanged.
Gemma 3 27B
Qwen 2.5 VL 7B
Prompt
IAPS
LAI-GAI
NAPS
IAPS
LAI-GAI
NAPS
Base
1.409
0.753
1.066
1.336
0.830
0.823
Definition presentation
1. No definition
1.377
0.777
1.021
1.330
0.843
0.826
2. Synonyms
1.402
0.756
1.046
1.336
0.865
0.834
Assigned persona
TABLE SV: Mean MAE (across all affective dimensions) for each prompt variant on three datasets. Lower is better. “Base” denotes the original prompt from Figure S2 . Variant numbers correspond to Table SIV .
Gemma 3 27B
Qwen 2.5 VL 7B
Prompt
IAPS
LAI-GAI
NAPS
IAPS
LAI-GAI
NAPS
Base
.706
.824
.762
.629
.809
.606
Definition presentation
1. No definition
.695
.822
.763
.605
.803
.601
2. Synonyms
.699
.824
.759
.625
.799
.608
Assigned persona
TABLE SVI: Mean Pearson correlation ( r ) across all affective dimensions for each prompt variant on three datasets. Higher is better. “Base” denotes the original prompt from Figure S2 . Variant numbers correspond to Table SIV .
Model
Accuracy (%)
LAI-GAI
NAPS
IAPS
Average
Proprietary
GPT-4.1 *
65.42
77.98
63.37
68.92
Gemini-2.5-Flash *
69.89
74.03
63.72
69.21
Proprietary (Thinking)
GPT-5.4
64.09
83.13
67.05
71.42
TABLE SVII: Overall classification accuracy (%). Bold indicates the best model within each compute tier.
Table 12
Pearson’s Correlation ( r ) ↑
Mean Absolute Error (MAE) ↓
Scale
Gemini 2.5
Qwen 72B
Gemma 27B
Intern 8B
Qwen 7B
Average
Gemini 2.5
Qwen 72B
Gemma 27B
Intern 8B
Qwen 7B
Average
IAPS
Anger
0.76
0.62
0.69
0.57
0.55
0.64
1.17
1.06
1.33
1.01
1.05
1.12
Disgust
0.82
0.78
0.82
0.75
0.79
0.79
0.93
1.06
1.37
1.20
1.01
1.11
Fear
0.83
0.82
0.81
0.75
0.77
0.80
1.42
1.41
1.91
1.62
1.21
1.51
Happiness
0.85
0.83
0.84
0.79
0.77
0.82
1.30
1.13
0.89
1.46
1.20
1.20
Sadness
0.82
0.76
0.80
0.77
0.67
0.76
1.22
1.14
1.51
1.17
1.06
1.22
TABLE SXIV: Side-by-Side Comparison of Pearson’s Correlation ( r ) and MAE (GPT-4.1 Subset)
Model
Accuracy (%)
LAI-GAI
NAPS
IAPS
Average
Proprietary
GPT-4.1
65.16
77.71
64.32
69.06
On H100
Qwen-VL-2.5-72B
67.31
79.35
67.19
71.28
Intern-VL-3.5-38B
67.96
79.14
60.88
69.33
TABLE SXV: Comparison of Model Accuracy (Gemini Subset)
Pearson’s Correlation ( r ) ↑
Mean Absolute Error (MAE) ↓
Scale
GPT-4.1
Qwen 72B
Gemma 27B
Intern 8B
Qwen 7B
Average
GPT-4.1
Qwen 72B
Gemma 27B
Intern 8B
Qwen 7B
Average
LAI–GAI
Amusement
0.75
0.79
0.78
0.73
0.73
0.76
1.27
0.92
0.93
1.30
1.06
1.10
Anger
0.89
0.76
0.81
0.64
0.56
0.73
0.58
0.58
0.55
0.65
0.71
0.61
Attach. Love
0.86
0.88
0.87
0.88
0.87
0.87
1.35
1.01
0.90
1.36
1.28
1.18
Awe
0.57
0.72
0.70
0.71
0.73
0.69
1.17
0.92
0.71
1.52
0.98
1.06
Craving
0.89
0.88
0.90
0.82
0.90
0.88
0.91
0.68
0.45
0.74
0.50
0.66
TABLE SXVII: Per-Emotion F1 Scores (%) on LAI-GAI Dataset (Gemini Subset)
Fig. S4: Effect of participant background information on prediction accuracy for Gemini-2.5-Flash (a, c, e) and Gemma-3 27B (b, d, f). Rows denote background type. Plots report Cohen’s d for the changes in MAE (positive = improved accuracy); error bars show 95% CIs, with effects significant when the CI excludes zero.
Scale
Human ceiling
Gemini-3-Flash r
% of ceiling
LAI-GAI
Amusement
.94
.65
69
Anger
.97
.88
91
Attach. Love
.93
.86
93
Awe
.89
.75
85
Craving
.94
.89
95
TABLE SXIX: Human split-half inter-rater ceiling per scale, the best model’s correlation (Gemini-3-Flash), and the percentage of the ceiling attained. LAI-GAI ceilings are the official values [ 19 ] ; IAPS ceilings are estimated by variance decomposition. NAPS is omitted (our norms provide only means).
Model
LAI-GAI
NAPS
IAPS
GPT-4.1 *
65.2 (60.6–69.7)
79.4 (75.9–82.8)
65.6 (62.0–69.1)
GPT-5.4
63.0 (58.1–67.8)
83.1 (79.8–86.4)
67.3 (63.7–70.8)
Gemini-2.5-Flash *
70.5 (66.2–74.8)
76.2 (72.4–79.9)
66.9 (63.4–70.5)
Gemini-3-Pro
61.8 (57.0–66.6)
82.9 (79.3–86.2)
66.2 (62.6–69.8)
Gemini-3-Flash
61.0 (56.2–65.9)
84.2 (80.8–87.5)
68.5 (64.9–72.1)
Qwen-VL-2.5-72B
68.3 (64.0–72.6)
80.4 (76.9–83.9)
65.3 (61.7–68.9)
TABLE SXX: Weighted-F1 ( % ) with bootstrap 95% confidence intervals per dataset.
Data
Comparison
Δ F1
95% CI
p
NAPS
Qwen-72B vs GPT-4.1
+1.0
[−1.9,3.9]
.49
NAPS
Gemini-3-Flash vs Qwen-72B
+3.8
[1.0,6.6]
.009
NAPS
GPT-5.4 vs GPT-4.1
+3.7
[1.2,6.4]
.003
IAPS
Gemini-3-Flash vs Gemini-2.5 †
+4.9
[2.2,7.6]
<.001
IAPS
GPT-5.4 vs GPT-4.1
+3.2
[0.9,5.6]
.007
LAI-GAI
Gemma-27B vs Gemini-2.5
+3.9
[−0.2,8.1]
.065
TABLE SXXI: Classification weighted-F1 pairwise comparisons (paired permutation test on shared images).
Data
vs. Model
Δrˉ
95% CI
p
IAPS
Gemini-2.5-Flash
+.058
[.042,.075]
<.001
IAPS
GPT-4.1
+.013
[.002,.023]
.029
IAPS
Qwen-VL-2.5-72B
+.117
[.100,.136]
<.001
IAPS
Gemma-3-27B
+.089
[.075,.103]
<.001
IAPS
Intern-VL-3.5-8B
+.126
[.106,.146]
<.001
IAPS
Qwen-VL-2.5-7B
+.166
[.145,.186]
<.001
TABLE SXXII: Task 2 mean-correlation comparisons of Gemini-3-Flash against each Task 2 model (paired bootstrap on shared images).
Fig. S5: Visualization of the affective stimulus space via t-SNE embeddings. The figure compares ground-truth human emotion ratings with predictions from the Qwen-VL-2.5-72B model across three datasets. The panels are arranged as follows: (a) LAI-GAI, (c) IAPS, and (e) NAPS show the space colored by human-assigned emotion labels. Panels (b) LAI-GAI, (d) IAPS, and (f) NAPS show the same space colored by the model’s predicted emotion labels. Each point represents an image, with its position determined by t-SNE applied to its L2-normalized CLIP vision embedding.
Affective computing develops systems that recognize, model, and respond to human emotions and affective states from behavior and physiological signals.
Vision-Language Models (VLMs) have demonstrated strong capabilities in perception, yet holistic Affective Image Content Analysis (AICA), which integrates perception, reasoning, and generation into a unified framework, remains underexplored. To address this gap, we introduce AICA-Bench, a comprehensive benchmark with three core tasks: Emotion Understanding (EU), Emotion Reasoning (ER), and Emotion-Guided Content Generation (EGCG). We evaluate 23 VLMs and identify two major limitations: weak intensity calibration and shallow open-ended descriptions. To address these issues, we propose Grounded Affective Tree (GAT) Prompting, a training-free framework that combines visual scaffolding with hierarchical reasoning. Experiments show that GAT reduces intensity errors and improves descriptive depth, providing a strong baseline for future research on affective multimodal understanding and generation.
Dong She, Xianrong Yao, Liqun Chen +3
School of Future Technology,South China University of Technology,Guangzhou
Understanding emotions is a fundamental ability for intelligent systems to be able to interact with humans. Vision-language models (VLMs) have made tremendous progress in the last few years for many visual tasks, potentially offering a promising solution for understanding emotions. However, it is surprising that even the most sophisticated contemporary VLMs struggle to recognize human emotions or to outperform even specialized vision-only classifiers. In this paper we ask the question "Why do VLMs struggle to recognize human emotions?", and observe that the inherently continuous and dynamic task of facial expression recognition (DFER) exposes two critical VLM vulnerabilities. First, emotion datasets are naturally long-tailed, and the web-scale data used to pre-train VLMs exacerbates this head-class bias, causing them to systematically collapse rare, under-represented emotions into common categories. We propose alternative sampling strategies that prevent favoring common concepts. Second, temporal information is critical for understanding emotions. However, VLMs are unable to represent temporal information over dense frame sequences, as they are limited by context size and the number of tokens that can fit in memory, which poses a clear challenge for emotion recognition. We demonstrate that the sparse temporal sampling strategy used in VLMs is inherently misaligned with the fleeting nature of micro-expressions (0.25-0.5 seconds), which are often the most critical affective signal. As a diagnostic probe, we propose a multi-stage context enrichment strategy that utilizes the information from "in-between" frames by first converting them into natural language summaries. This enriched textual context is provided as input to the VLM alongside sparse keyframes, preventing attentional dilution from excessive visual data while preserving the emotional trajectory.
Madhav Agarwal, Sotirios A. Tsaftaris, Laura Sevilla-Lara +1
Vision-language models (VLMs) increasingly act as judges that pick the best of several generated images, so their choices decide what users see. Such judges are usually validated by score agreement with human ratings, not by the images they return. We audit VLM judges as decision-makers: on 300 culturally situated prompts, we compare the returned image with human ratings the judge never sees and with random choice from the same candidates, and repeat every decision with the candidates reordered. A 4B-parameter judge barely beats random and falls short of a CLIP similarity baseline. It picks the first image shown in 49% of calls (chance: 28%), and reordering changes its choice on 60% of prompts. For this judge, agreement across orders is informative: decisions that survive reordering are much better than random, whereas agreement with a weaker second judge keeps the wrong ones. An 8B judge shows almost no position bias and outperforms CLIP, yet for it the same filter mostly discards good decisions. Agreement helps only when it targets the judge's failure mode, so filters must be re-audited whenever the judge changes. The 4B judge's slight rise in stereotype ratings is no longer detectable after aggregating across orders or with the larger judge.