cs.HCJan 27, 2026

Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models

Authors: Filip Nowicki, Hubert Marciniak, Jakub Łączkowski, Bartosz Wilczek, Krzysztof Jassem, Tomasz Górecki, Vimala Balakrishnan, Desmond C. Ong, 

Organizations: Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Poznań, Poland · Faculty of Computer Science and Information Technology, Universiti Malaya, Malaysia · Department of Computer Science and Engineering, Korea University, Korea · Department of Psychology, University of Texas at Austin, USA · 

Abstract

Vision-language models (VLMs) show promise for large-scale affect inference from images, but their alignment with human affective ratings remains unclear. We benchmarked 16 proprietary and open-source VLMs on three validated affective image datasets: the International Affective Picture System, the Nencki Affective Picture System, and the Library of AI-Generated Affective Images. In a zero-shot setting, models performed two tasks: (i) top-emotion classification and (ii) prediction of human ratings on 1--7/9 Likert scales for discrete emotions and affective dimensions. We also evaluated rater-conditioned prompting on the LAI-GAI dataset using de-identified participant metadata. Best models showed strong discrete emotion classification (60%--80% on six-emotion tasks; 60%--75% on a 12-category task), with anger and surprise predicted least accurately. For continuous ratings, models aligned moderately to strongly with human judgments (r>0.75r > 0.75) but showed consistent biases, especially weaker performance on arousal and overestimation of response intensity. Rater-conditioned prompting had only small, inconsistent effects. Overall, VLMs captured broad affective patterns but sometimes lacked the nuance, although newer model generations performed better.

Figures & tables

Explore similar work

CardsList
  1. AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis

    Apr 7, 2026Dong She, Xianrong Yao, Liqun Chen +3VLM EvaluationVLM Calibration

  2. Why Do Vision Language Models Struggle To Recognize Human Emotions?

    Apr 16, 2026Madhav Agarwal, Sotirios A. Tsaftaris, Laura Sevilla-Lara +1VLM EvaluationMultimodal Emotion Recognition