Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models
Organizations: Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Poznań, Poland · Faculty of Computer Science and Information Technology, Universiti Malaya, Malaysia · Department of Computer Science and Engineering, Korea University, Korea · Department of Psychology, University of Texas at Austin, USA ·
Abstract
Vision-language models (VLMs) show promise for large-scale affect inference from images, but their alignment with human affective ratings remains unclear. We benchmarked 16 proprietary and open-source VLMs on three validated affective image datasets: the International Affective Picture System, the Nencki Affective Picture System, and the Library of AI-Generated Affective Images. In a zero-shot setting, models performed two tasks: (i) top-emotion classification and (ii) prediction of human ratings on 1--7/9 Likert scales for discrete emotions and affective dimensions. We also evaluated rater-conditioned prompting on the LAI-GAI dataset using de-identified participant metadata. Best models showed strong discrete emotion classification (60%--80% on six-emotion tasks; 60%--75% on a 12-category task), with anger and surprise predicted least accurately. For continuous ratings, models aligned moderately to strongly with human judgments () but showed consistent biases, especially weaker performance on arousal and overestimation of response intensity. Rater-conditioned prompting had only small, inconsistent effects. Overall, VLMs captured broad affective patterns but sometimes lacked the nuance, although newer model generations performed better.
Figures & tables
| Emotion | Definition |
|---|---|
| Amusement | An emotional experience associated with opportunities for humor and events that violate expectations. |
| Anger | An emotional experience characterized by antagonism toward someone or something perceived as deliberately harmful or unfair. |
| Attachment love | An emotional experience tied to opportunities for affiliation, interdependence, and intimacy. |
| Awe | An emotional experience triggered by stimuli that are vast and beyond the ordinary frame of reference, often involving wonder, admiration, and a sense of being small in the presence of something greater than oneself. |
| Craving | An emotional experience associated with specific sensory pleasure that aids in distinguishing nutritional foods from nutritionally neutral objects and toxins. |
| Disgust | An emotional experience involving strong aversion or repulsion toward something perceived as offensive, contaminating, or unpleasant. |
| Dimension | Definition |
|---|---|
| Positive | Feelings of pleasure, satisfaction, or well-being, often associated with favorable situations. |
| Negative | Feelings of discomfort, displeasure, or distress, often associated with unfavorable situations. |
| Calm | A sense of tranquility and composure, with low levels of emotional arousal or agitation. |
| Aroused | Heightened physiological and psychological activity, such as increased heart rate, energy, or alertness. |
| Motivated to approach | An emotional drive to pursue a desired goal, object, or situation, often associated with curiosity, attraction, or reward. |
| Motivated to avoid | An emotional drive to move away from an undesired or threatening goal, object, or situation, often associated with fear, discomfort, or aversion. |
| Label | Dataset | Definition |
|---|---|---|
| Discrete emotions (shared across IAPS and NAPS) | ||
| Happiness | Both | A positive state characterized by feelings of joy, contentment, and wellbeing. It often arises in response to pleasant experiences and is typically accompanied by a sense of lightness, warmth, and optimism. |
| Surprise | Both | The response to a sudden, unexpected event. |
| Sadness | Both | Feelings of disappointment, grief, or hopelessness. |
| Anger | Both | Antagonism toward someone or something perceived as deliberately harmful or unfair. |
| Disgust | Both | Strong aversion or repulsion toward something perceived as offensive, contaminating, or unpleasant. |
| # | Category | Modification |
| Category 1: Definition Presentation (User Prompt) | ||
| 1 | Definition removed | The definition line (“Definition: {emotion_name} is {definition} ”) is removed from the user prompt entirely. The model receives only the emotion name and scale. |
| 2 | Synonyms instead of definition | The definition line is replaced with “Synonyms/related terms are {definition} ”, reframing the definitional cue as a set of related terms. |
| Category 2: Assigned Persona (System Prompt) | ||
| 3 | Study participant | The persona “You are an affective scientist” is replaced with “You are a study participant”. |
| 4 | Psychologist | Replaced with “You are a psychologist”. |
| Gemma 3 27B | Qwen 2.5 VL 7B | |||||
| Prompt | IAPS | LAI-GAI | NAPS | IAPS | LAI-GAI | NAPS |
| Base | 1.409 | 0.753 | 1.066 | 1.336 | 0.830 | 0.823 |
| Definition presentation | ||||||
| 1. No definition | 1.377 | 0.777 | 1.021 | 1.330 | 0.843 | 0.826 |
| 2. Synonyms | 1.402 | 0.756 | 1.046 | 1.336 | 0.865 | 0.834 |
| Assigned persona | ||||||
| Gemma 3 27B | Qwen 2.5 VL 7B | |||||
| Prompt | IAPS | LAI-GAI | NAPS | IAPS | LAI-GAI | NAPS |
| Base | .706 | .824 | .762 | .629 | .809 | .606 |
| Definition presentation | ||||||
| 1. No definition | .695 | .822 | .763 | .605 | .803 | .601 |
| 2. Synonyms | .699 | .824 | .759 | .625 | .799 | .608 |
| Assigned persona | ||||||
| Model | Accuracy (%) | |||
|---|---|---|---|---|
| LAI-GAI | NAPS | IAPS | Average | |
| Proprietary | ||||
| GPT-4.1 * | 65.42 | 77.98 | 63.37 | 68.92 |
| Gemini-2.5-Flash * | 69.89 | 74.03 | 63.72 | 69.21 |
| Proprietary (Thinking) | ||||
| GPT-5.4 | 64.09 | 83.13 | 67.05 | 71.42 |
| Pearson’s Correlation ( ) | Mean Absolute Error (MAE) | ||||||||||||
| Scale | Gemini 2.5 | Qwen 72B | Gemma 27B | Intern 8B | Qwen 7B | Average | Gemini 2.5 | Qwen 72B | Gemma 27B | Intern 8B | Qwen 7B | Average | |
| IAPS | Anger | 0.76 | 0.62 | 0.69 | 0.57 | 0.55 | 0.64 | 1.17 | 1.06 | 1.33 | 1.01 | 1.05 | 1.12 |
| Disgust | 0.82 | 0.78 | 0.82 | 0.75 | 0.79 | 0.79 | 0.93 | 1.06 | 1.37 | 1.20 | 1.01 | 1.11 | |
| Fear | 0.83 | 0.82 | 0.81 | 0.75 | 0.77 | 0.80 | 1.42 | 1.41 | 1.91 | 1.62 | 1.21 | 1.51 | |
| Happiness | 0.85 | 0.83 | 0.84 | 0.79 | 0.77 | 0.82 | 1.30 | 1.13 | 0.89 | 1.46 | 1.20 | 1.20 | |
| Sadness | 0.82 | 0.76 | 0.80 | 0.77 | 0.67 | 0.76 | 1.22 | 1.14 | 1.51 | 1.17 | 1.06 | 1.22 | |
| Model | Accuracy (%) | |||
|---|---|---|---|---|
| LAI-GAI | NAPS | IAPS | Average | |
| Proprietary | ||||
| GPT-4.1 | 65.16 | 77.71 | 64.32 | 69.06 |
| On H100 | ||||
| Qwen-VL-2.5-72B | 67.31 | 79.35 | 67.19 | 71.28 |
| Intern-VL-3.5-38B | 67.96 | 79.14 | 60.88 | 69.33 |
| Pearson’s Correlation ( ) | Mean Absolute Error (MAE) | ||||||||||||
| Scale | GPT-4.1 | Qwen 72B | Gemma 27B | Intern 8B | Qwen 7B | Average | GPT-4.1 | Qwen 72B | Gemma 27B | Intern 8B | Qwen 7B | Average | |
| LAI–GAI | Amusement | 0.75 | 0.79 | 0.78 | 0.73 | 0.73 | 0.76 | 1.27 | 0.92 | 0.93 | 1.30 | 1.06 | 1.10 |
| Anger | 0.89 | 0.76 | 0.81 | 0.64 | 0.56 | 0.73 | 0.58 | 0.58 | 0.55 | 0.65 | 0.71 | 0.61 | |
| Attach. Love | 0.86 | 0.88 | 0.87 | 0.88 | 0.87 | 0.87 | 1.35 | 1.01 | 0.90 | 1.36 | 1.28 | 1.18 | |
| Awe | 0.57 | 0.72 | 0.70 | 0.71 | 0.73 | 0.69 | 1.17 | 0.92 | 0.71 | 1.52 | 0.98 | 1.06 | |
| Craving | 0.89 | 0.88 | 0.90 | 0.82 | 0.90 | 0.88 | 0.91 | 0.68 | 0.45 | 0.74 | 0.50 | 0.66 | |
| Scale | Human ceiling | Gemini-3-Flash | % of ceiling |
| LAI-GAI | |||
| Amusement | .94 | .65 | 69 |
| Anger | .97 | .88 | 91 |
| Attach. Love | .93 | .86 | 93 |
| Awe | .89 | .75 | 85 |
| Craving | .94 | .89 | 95 |
| Model | LAI-GAI | NAPS | IAPS |
|---|---|---|---|
| GPT-4.1 * | 65.2 (60.6–69.7) | 79.4 (75.9–82.8) | 65.6 (62.0–69.1) |
| GPT-5.4 | 63.0 (58.1–67.8) | 83.1 (79.8–86.4) | 67.3 (63.7–70.8) |
| Gemini-2.5-Flash * | 70.5 (66.2–74.8) | 76.2 (72.4–79.9) | 66.9 (63.4–70.5) |
| Gemini-3-Pro | 61.8 (57.0–66.6) | 82.9 (79.3–86.2) | 66.2 (62.6–69.8) |
| Gemini-3-Flash | 61.0 (56.2–65.9) | 84.2 (80.8–87.5) | 68.5 (64.9–72.1) |
| Qwen-VL-2.5-72B | 68.3 (64.0–72.6) | 80.4 (76.9–83.9) | 65.3 (61.7–68.9) |
| Data | Comparison | F1 | 95% CI | |
|---|---|---|---|---|
| NAPS | Qwen-72B vs GPT-4.1 | |||
| NAPS | Gemini-3-Flash vs Qwen-72B | |||
| NAPS | GPT-5.4 vs GPT-4.1 | |||
| IAPS | Gemini-3-Flash vs Gemini-2.5 | |||
| IAPS | GPT-5.4 vs GPT-4.1 | |||
| LAI-GAI | Gemma-27B vs Gemini-2.5 |
| Data | vs. Model | 95% CI | ||
|---|---|---|---|---|
| IAPS | Gemini-2.5-Flash | |||
| IAPS | GPT-4.1 | |||
| IAPS | Qwen-VL-2.5-72B | |||
| IAPS | Gemma-3-27B | |||
| IAPS | Intern-VL-3.5-8B | |||
| IAPS | Qwen-VL-2.5-7B |
Affective Computing
Affective computing develops systems that recognize, model, and respond to human emotions and affective states from behavior and physiological signals.
See all 149 papers