Evaluating Vision-Language Models for Image Quality Assessment using Psychophysical Data

Imran Mehmood; Imad Ali Shah; Ming Ronnier Luo; Brian Deegan

cs.CVMar 25, 2026

Evaluating Vision-Language Models for Image Quality Assessment using Psychophysical Data

Authors: Imran Mehmood, Imad Ali Shah, Ming Ronnier Luo, Brian Deegan

Organizations: School of Engineering, University of Galway, Ireland · Lero the Research Ireland Centre for Software, Ireland · State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University, Hangzhou, China.

Abstract

Psychophysical experiments remain the most reliable approach for perceptual image quality assessment (IQA), yet their cost and limited scalability motivate automated alternatives. This paper investigates whether Vision-Language Models (VLMs) can assist in assessing perceived image appearance and quality. We introduce a psychophysics-inspired framework to probe VLM perceptual sensitivity through controlled pairwise image comparisons of contrast, colorfulness, and overall preference. Six VLMs (four proprietary and two open-weight models) are compared against psychophysical data. Results reveal strong attribute-dependent variability: Claude exhibits the highest internal consistency, whereas GPT achieves the strongest agreement for overall preference. Claude and Qwen show the strongest alignment for colorfulness, while Qwen performs best for contrast. However, no model consistently matches human perception across all attributes. High self-consistency does not necessarily imply perceptual validity, and VLM--human agreement generally improves when perceptual differences among renderings are more pronounced.

Figures & tables

Scale	Claude	Gemini	GPT	Grok	Intern	Qwen
Contrast	1.90 (2.38, 0.82)	2.54 (2.86, 0.55)	4.44 (5.24, 0.73)	6.67 (7.62, 0.95)	17.78 (20.00, 1.98)	2.54 (2.86, 0.27)
Colorfulness	0.95 (1.43, 0.48)	6.03 (7.14, 1.53)	8.89 (10.00, 0.99)	7.30 (8.10, 1.37)	14.92 (17.14, 2.40)	2.86 (4.29, 1.26)
Overall	1.27 (1.43, 0.27)	10.48 (13.81, 4.97)	7.62 (8.10, 0.82)	8.25 (10.48, 2.15)	13.02 (14.29, 1.45)	6.35 (7.62, 1.45)

Table 1: Intra-model variability performance comparison (VR %): Mean (Max, SD).

Scale	Model	Claude	GPT	Gemini	Grok	Intern
Contrast	GPT	$25.87\ (26.67,\ 0.67)$	—	—	—	—
	Gemini	$25.71\ (26.67,\ 0.79)$	$10.00\ (11.90,\ 1.56)$	—	—	—
	Grok	$46.03\ (46.67,\ 0.53)$	$43.02\ (43.81,\ 0.98)$	$41.38\ (42.86,\ 1.08)$	—	—
	Intern	$40.79\ (40.95,\ 0.24)$	$52.06\ (52.38,\ 0.24)$	$48.73\ (49.05,\ 0.48)$	$52.22\ (52.86,\ 0.48)$	—
	Qwen	$26.35\ (26.67,\ 0.24)$	$24.92\ (25.24,\ 0.24)$	$21.90\ (22.86,\ 0.82)$	$38.41\ (39.05,\ 0.48)$	$49.05\ (49.05,\ 0.00)$
Colorfulness	GPT	$21.90\ (23.33,\ 1.17)$	—	—	—	—

Table 2: Cross-model variability rate comparison (VR %): Mean (Max, SD)

Scale	Primary Criteria (Rewarded)	Secondary Criteria	Penalty Cues (Downgraded)
Colorfulness	Color saturation; vividness and vibrancy of hues	Tonal richness; exposure balance; natural color rendering	Muted, dull, washed-out, or desaturated colors; hazy/milky film; flat appearance
Contrast	Highlight–shadow separation; dynamic/tonal range	Deep blacks and bright highlights; exposure balance; depth of shadows	Flat or low-contrast appearance; washed-out tones; hazy/milky veil; lifted black levels
Overall Quality	Sharpness and detail; exposure balance; depth and clarity; naturalness	Tonal range; highlight/shadow detail preservation; color fidelity	Haze/milky film; washed-out or flat look; over-processed/HDR-like artifacts; loss of fine detail

Table 3: Summary of the qualitative rationale criteria used by the six evaluated VLMs.

Figure 2: Correlation between VLM and human mean rankings across the Contrast, Colorfulness, and Overall Preference scales.

Figure 3: Attribute weighting for overall preference. The x-axis represents the contrast weight (

\alpha

) and the y-axis represents the colorfulness weight (

\beta

Panel of Figure 4 — Figure 4: Mean scene agreement (Spearman $\rho$ ) as a function of perceptual separability for (a) Colorfulness, (b) Contrast, and (c) Overall Preference.

Figure 5: Relationship between intra-model variability (VR%) and psychophysical correlation (Spearman

\rho

Explore similar work

ME-IQA: Memory-Enhanced Image Quality Assessment via Re-Ranking
Mar 21, 2026Kanglong Fan, Tianhe Wu, Wen Wen +6Memory-Augmented VLMs Vision-Language Models
City University of Hong Kong, Hong Kong SAR, China · ByteDance Inc., China
Q-Tacit: Image Quality Assessment via Latent Visual Reasoning
Mar 23, 2026Yuxuan Jiang, Yixuan Li, Hanwei Zhu +3Image Quality Assessment Latent Visual Reasoning
Visual Information Lab, University of Bristol · Nanyang Technological University
Can Vision Language Models Judge Action Quality? An Empirical Evaluation
Apr 9, 2026Miguel Monte e Freitas, Rui Henriques, Ricardo Rei +1Action Quality Assessment Vision-Language Models
Sword Health · Instituto Superior T´ecnico, Universidade de Lisboa · INESC-ID

Mar 21, 2026J/K move · Enter open · S save

ME-IQA: Memory-Enhanced Image Quality Assessment via Re-Ranking

Kanglong Fan, Tianhe Wu, Wen Wen, Jianzhao Liu+5

City University of Hong Kong, Hong Kong SAR, China · ByteDance Inc., China

Memory-Augmented VLMs Vision-Language Models Image Quality Assessment

Open paper AskPDF arXiv

Reasoning-induced vision-language models (VLMs) advance image quality assessment (IQA) with textual reasoning, yet their scalar scores often lack sensitivity and collapse to a few values, so-called discrete collapse. We introduce ME-IQA, a plug-and-play, test-time memory-enhanced re-ranking framework. It (i) builds a memory bank and retrieves semantically and perceptually aligned neighbors using reasoning summaries, (ii) reframes the VLM as a probabilistic comparator to obtain pairwise preference probabilities and fuse this ordinal evidence with the initial score under Thurstone's Case V model, and (iii) performs gated reflection and consolidates memory to improve future decisions. This yields denser, distortion-sensitive predictions and mitigates discrete collapse. Experiments across multiple IQA benchmarks show consistent gains over strong reasoning-induced VLM baselines, existing non-reasoning IQA methods, and test-time scaling alternatives.

Abstract

Explore similar work

ME-IQA: Memory-Enhanced Image Quality Assessment via Re-Ranking

Q-Tacit: Image Quality Assessment via Latent Visual Reasoning

Can Vision Language Models Judge Action Quality? An Empirical Evaluation