Multidimensional Observer Model and Perceptual Dimensions of Human Image Quality Assessment

Sheng Zhao; Weikai Lin; Yuhao Zhu

cs.CVSep 29, 2026

Multidimensional Observer Model and Perceptual Dimensions of Human Image Quality Assessment

Authors: Sheng Zhao, Weikai Lin, Yuhao Zhu

Organizations: Department of Computer Science University of Rochester

Abstract

Judging image quality is not only ecologically relevant to everyday human tasks, but also underpins many machine vision tasks such as image generation. This paper proposes a framework to understand the inherent perceptual space underlying image quality judgment in humans. We propose a multi-dimensional observer model that represents images as distributions in a latent perceptual space and that models human judgment as comparing noisy samples. Being constrained by neural representations in the primate ventral stream and fit to large-scale behavioral data, the model enables analysis of perceptual structure while matching the predictive power of existing metrics. Using this model, we find that the perceptual spaces needed to account for image quality judgment in humans are extremely low-dimensional compared to the image space even when considering its sparsity. The exact structure of the space (e.g., dimensionalities, information encoded) varies between low-level and high-level quality judgments, suggesting that, despite a shared retinal encoding in the beginning, humans selectively construct task-dependent perceptual spaces in visual decision making.

Figures & tables

Figure. 1 : Overview of the observer model. (a): Example triplets (reference $r$ and the two alternatives $x_{0}$ and $x_{1}$ ) from the three 2AFC datasets we evaluate. (b): Illustration of the observer model. Each image is modeled as an $N$ -dimensional Gaussian random variable $Z(x)\sim\mathcal{N}\!\big(\bm{\mu}(x),\bm{\Sigma}(x)\big)$ in an internal perceptual space. Each observation amounts of drawing a sample from the three distributions, and the candidate image that is closer to the reference is picked in that trial (visualized with a red bounding box). (c): The computational framework for the observer model.

Dataset	Ceiling $\pm$ 95% CI	CORnet-S-Multi	CVVDP [ 23 ]	PieAPP [ 12 ]	DISTS [ 13 ]	TOPIQ-FR [ 20 ]	MILO [ 19 ]	DreamSim [ 11 ]	OpenCLIP [ 60 ]	DINOv3 [ 61 ]	CLIP [ 62 ]	LPIPS [ 10 ]	SSIM [ 9 ]	PSNR
BAPPS (Agr $\uparrow$ )	.797 $\pm$ .183	.688	.645	.629	.678	.667	.634	.683	.653	.640	.624	.662	.641	.633
cnn	.885 $\pm$ .112	.831	.809	.765	.813	.808	.800	.825	.800	.796	.773	.814	.809	.801
color	.764 $\pm$ .212	.634	.610	.612	.615	.616	.596	.627	.578	.572	.560	.588	.624	.624
deblur	.740 $\pm$ .224	.596	.594	.547	.599	.599	.584	.591	.560	.572	.539	.573	.594	.590
f.int	.760 $\pm$ .208	.621	.590	.606	.626	.610	.558	.632	.611	.613	.593	.627	.555	.543
s.res	.801 $\pm$ .181	.697	.658	.619	.692	.686	.645	.695	.670	.624	.627	.690	.649	.642

Table 1 : Performance of our observer model and comparison with other perceptual metrics. The baseline metrics are grouped as follows: methods that have the capability to predict choice probabilities in 2AFC tasks (Ours, CVVDP [ 23 ] , and PieAPP [ 12 ] ), methods that predict quality scores in the form of DMOS/MOS (DISTS [ 13 ] , TOPIQ-FR [ 20 ] , and MILO [ 19 ] ) and in the form of distance in a learned embedding space (LPIPS [ 10 ] , DreamSim [ 11 ] , OpenCLIP [ 60 ] , DINOv3 [ 61 ] , and CLIP [ 62 ] ), and classical, parameter-free metrics (SSIM [ 9 ] and PSNR). The KL divergence can only be calculated for models that could output choice probabilities. Ceilings are followed by their per-triplet 95% confidence interval (CI) averaged over all triplets. Bold = best, underline = second & third best.

Table 3

Figure. 2 : Prediction performance of our observer model under different

N

. We show six variants of our model, each using a different feature encoder. Low-level IQA datasets (BAPPS, PieAPP) plateau within a handful of dimensions while the high-level IQA dataset NIGHTS requires more dimensionalities. For BAPPS and PieAPP, we show the 95% CI along with the theoretical ceiling derived the sample distribution; the

y

-axes are capped for clarity. NIGHTS ceiling is 1 and CI is 0.

Figure. 3 : Interpretation of the learned $N=2$ model (CORnet-S-Multi, BAPPS). (a) The smoothly varying covariance matrix field over the two learned dimensions $z_{1}$ and $z_{2}$ sampled in a 8 $\times$ 8 grid; each sampled covariance contour is visualized as an ellipse (log-scale) bounded by two standard deviations. (b) A complementary visualization of the covariance matrix field showing how $\sigma^{2}_{z1}$ , $\sigma^{2}_{(z1,z2)}$ , $\sigma^{2}_{(z2,z1)}$ , and $\sigma^{2}_{z2}$ vary individually in the learned 2D space. (c) $9\times 9$ images that, when mapped to the perceptual space, vary vertically along the $z_{1}$ dimension from low to high and vary horizontally along the $z_{2}$ dimension from low to high. Reading along these two axes reveals that $z_{1}$ captures texture and $z_{2}$ captures color. More results are in Appendix B .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Trainable	Frozen	Total
DISTS	2,950	14.7M (VGG-16)	14.7M
MILO	45K	—	45K
TOPIQ-FR (CFANet, ResNet-50 end-to-end)	36.0M	53K (Normalization)	36.0M
DreamSim (LoRA $r{=}16$ on 3 $\times$ ViT-B/16)	1.77M	264.2M	266.0M
PieAPP	68.4M	—	68.4M
CORnet-S-Multi ( $N{=}3$ )	7.7K	52.9M	52.9M

Appendix

Table 4: Parameter counts comparison.

Panel of Figure 7 — Figure. 4 : Visualizing the learned $N{=}2$ perceptual space under CORnet-S Multi across datasets. Each cell of the $9\times 9$ grid is an image from the dataset situated near that point in the learned space, so traversing horizontally and vertically shows how the model’s representation morphs as one of the two latent dimensions varies low to high. Across the three datasets, the two axes consistently expose a texture/structure direction crossed with a color (PieAPP) or scene-content (NIGHTS) direction. See Figure 3 (c) for results on BAPPS.

Figure. 5 : BRODEN [ 69 ] concept distribution per CORnet-S layer at the ratio

\geq 5\times

alignment threshold. Top: fraction of channels whose top concept lies in each pixel-level category. Bottom: per-layer top concepts ranked by the number of channels claiming them.

Figure. 6 : Projection structure of the learned perceptual dimensions. We visualize the projection matrix for BAPPS ( $N=4$ ), PieAPP ( $N=4$ ), and NIGHTS ( $N=10$ ). The heatmaps show row-normalized projection magnitudes $|W_{k,j}|$ , where each row corresponds to one learned perceptual dimension and each column to one of the $960$ CORnet-S-Multi channels. Dashed vertical lines mark the boundaries between V1, V2, V4, and IT channels. For each row, the bar on the right summarizes the BRODEN category composition of the top- $20$ weighted channels. The five segments, color-coded according to the bottom legend, correspond to color, material, texture, part, and object, and sum to one. Together, the heatmaps indicate where each dimension reads from in the visual hierarchy, while the side bars summarize what visual content those channels encode.

Figure. 7 : Pairwise Pearson correlation between learned dimensions on the validation set. Rows are BAPPS, PieAPP and NIGHTS. Columns are $N=2,4,8,16$ . The number above each cell is the average off-diagonal coefficient.

	BAPPS Agr / $n^{*}$		BAPPS KL / $n^{*}$		NIGHTS Acc / $n^{*}$		PieAPP Agr / $n^{*}$
Encoder	L	P	L	P	L	P	L	P
VGG-conv3 ( $4{\times}4$ )	.688/7	.667/10	.174/12	.186/10	.797/32	.771/14	.711/9	.674/4
VGG-conv3	.687/4	.666/9	.170/5	.185/9	.802/16	.786/16	.709/6	.677/3
VGG-conv5 ( $2{\times}2$ )	.666/7	.649/8	.189/6	.203/5	.801/48	.769/28	.689/20	.678/12
VGG-conv5	.670/7	.648/7	.184/7	.201/7	.831/96	.794/48	.687/8	.675/10
VGG-multi	.692/4	.671/9	.164/7	.181/12	.848/24	.811/24	.709/6	.677/3

Appendix

Table 5 : Full in-domain results across three benchmarks under two projection configurations (L = Learned, P = PCA). BAPPS: agreement/

n^{*}

and KL/

n^{*}

(

0

= oracle). NIGHTS: accuracy/

n^{*}

. PieAPP: agreement/

n^{*}

. Theoretical ceilings reported in main-text Table 1 .

Figure. 8 : Cross-dataset transfer of a BAPPS-trained observer across six feature encoders. (a): BAPPS

\rightarrow

PieAPP using per-annotator agreement on PieAPP as the accuracy metric. (b): BAPPS

\rightarrow

NIGHTS using 1-BER on the NIGHTS test set. The brown dashed line in each panel is the best in-domain ceiling on the test-side dataset.

Encoder	B $\to$ N: NIGHTS Acc / $n^{*}$	N $\to$ B: BAPPS Agr / $n^{*}$	B $\to$ P: PieAPP Agr / $n^{*}$
VGG-conv3 ( $4{\times}4$ )	.735/32	.669/10	.702/9
VGG-conv3	.770/32	.653/28	.697/14
VGG-conv5 ( $2{\times}2$ )	.714/24	.638/20	.685/20
VGG-conv5	.752/48	.626/24	.684/12
VGG-multi	.811/48	.670/12	.688/5
CORnet-S V1	.688/16	.639/8	.631/64

Appendix

Table 6 : Full cross-domain results under the default Learned projection with Cov

\leftarrow\bm{\mu}(x)

. B

\to

N: trained on BAPPS, tested on NIGHTS (accuracy/

n^{*}

). N

\to

B: trained on NIGHTS, tested on BAPPS (agreement/

n^{*}

). B

\to

P: trained on BAPPS, tested on PieAPP (agreement/

n^{*}

). Theoretical ceilings reported in main-text Table 1 . Shaded rows are the encoders used in the main text.

Method	TID2013	CSIQ	LIVE
CORnet-S Multi	0.860 / 0.870 / 0.666	0.946 / 0.948 / 0.796	0.937 / 0.927 / 0.781
MILO	0.876 / 0.876 / 0.692	0.959 / 0.942 / 0.817	0.953 / 0.913 / 0.817
TOPIQ-FR	0.917 / 0.921 / 0.744	0.967 / 0.964 / 0.838	0.898 / 0.901 / 0.717
CVVDP	0.853 / 0.864 / 0.672	0.896 / 0.883 / 0.728	0.917 / 0.902 / 0.766
DreamSim	0.809 / 0.830 / 0.612	0.932 / 0.941 / 0.768	0.899 / 0.902 / 0.729
PieAPP	0.850 / 0.846 / 0.652	0.845 / 0.829 / 0.666	0.925 / 0.915 / 0.765

Appendix

Table 7: The observer model extends to rating-based IQA datasets with absolute (D)MOS labels. Each cell reports SRCC / PLCC / KRCC; higher is better for all three. Bold = best, underline = second & third best, ranked independently per (dataset, metric) column.

Method	BAPPS	NIGHTS	PieAPP	TID2013	CSIQ	LIVE	avg. rank
CORnet-S Multi	1	3	3	3	3	4	2.83
DreamSim	2	1	1	6	4	9	3.83
DISTS	3	2	4	8	5	1	3.83
TOPIQ-FR	4	6	5	1	1	10	4.50
MILO	8	9	7	2	2	2	5.00
LPIPS	5	4	6	10	7	3	5.83

Appendix

Table 8: Per-dataset rank of each method across the six datasets covered by Tables 1 and 7 , restricted to the nine baselines that appear in both. Score per dataset: BAPPS & PieAPP use per-annotator agreement; NIGHTS uses accuracy; TID2013 & CSIQ & LIVE use SRCC. Lower rank is better; the right-most column is the mean across the six datasets.

Figure. 9 : Saturation of the KADID fitted CORnet-Multi observer model (

N{=}24

in Table 7 ) along the sweep schedule. (A) KADID-10k val. (B) TID2013 (zero-shot). (C) CSIQ (zero-shot). The vertical band marks the 95%-of-peak saturation point

n^{\star}

, computed against an SRCC chance baseline of

0

Backbone	$D$	BT	Obs	$\Delta$	BT $n^{*}$	Obs $n^{*}$	BT	Obs	$\Delta$	BT $n^{*}$	Obs $n^{*}$
		BAPPS (Agr)					NIGHTS (Acc)
CORnet-S V1	64	.686	.685	$+.001$	4	4	.764	.747	$+.017$	12	8
CORnet-S V2	128	.691	.689	$+.002$	7	3	.783	.793	$-.010$	12	14
CORnet-S V4	256	.687	.688	$-.001$	6	4	.831	.855	$-.024$	48	48
CORnet-S IT	512	.677	.677	$+.000$	8	6	.862	.853	$+.009$	96	96
CORnet-S Multi	960	.685	.688	$-.003$	6	3	.859	.859	$+.000$	256	96

Appendix

Table 9 : BT vs. full observer model, in-domain.

\Delta

= BT

-

Obs. BAPPS: per-annotator agreement. NIGHTS: 2AFC accuracy.

Backbone	$D$	BT	Obs	$\Delta$	BT	Obs	$\Delta$
		B $\to$ N (Acc)			N $\to$ B (Agr)
CORnet-S V1	64	.712	.688	$+.024$	.661	.639	$+.022$
CORnet-S V2	128	.763	.757	$+.006$	.656	.659	$-.003$
CORnet-S V4	256	.823	.796	$+.027$	.661	.663	$-.002$
CORnet-S IT	512	.847	.813	$+.034$	.664	.666	$-.002$
CORnet-S Multi	960	.837	.825	$+.012$	.665	.670	$-.005$

Appendix

Table 10 : BT vs. full observer model, cross-domain.

\Delta

= BT

-

Obs. B

\to

N: 2AFC accuracy. N

\to

B: per-annotator agreement.

Backbone	$D$	MLE	$\lambda{=}.05$	$\Delta$	$\lambda{=}1$	$\Delta$	MLE	$\lambda{=}.05$	$\Delta$	$\lambda{=}1$	$\Delta$
		BAPPS (Agr)					NIGHTS (Acc)
CORnet-S V1	64	.685	.668	$-.017$	.670	$-.015$	.747	.737	$-.010$	.732	$-.015$
CORnet-S V2	128	.689	.683	$-.006$	.683	$-.006$	.793	.788	$-.005$	.782	$-.011$
CORnet-S V4	256	.688	.686	$-.002$	.686	$-.002$	.855	.839	$-.016$	.835	$-.020$
CORnet-S IT	512	.677	.676	$-.001$	.676	$-.001$	.853	.855	$+.002$	.845	$-.008$
CORnet-S Multi	960	.688	.688	$+.000$	.688	$+.000$	.859	.868	$+.009$	.861	$+.002$

Appendix

Table 11 : Wishart MAP vs. MLE in-domain.

\Delta

= MAP

-

MLE. BAPPS: per-annotator agreement. NIGHTS: 2AFC accuracy.

Backbone	$D$	MLE	$\lambda{=}.05$	$\Delta$	$\lambda{=}1$	$\Delta$	MLE	$\lambda{=}.05$	$\Delta$	$\lambda{=}1$	$\Delta$
		B $\to$ N (Acc)					N $\to$ B (Agr)
CORnet-S V1	64	.688	.695	$+.007$	.700	$+.012$	.639	.635	$-.004$	.631	$-.008$
CORnet-S V2	128	.757	.765	$+.008$	.758	$+.001$	.659	.649	$-.010$	.649	$-.010$
CORnet-S V4	256	.796	.799	$+.003$	.802	$+.006$	.663	.644	$-.019$	.659	$-.004$
CORnet-S IT	512	.813	.813	$+.000$	.805	$-.008$	.666	.667	$+.001$	.664	$-.002$
CORnet-S Multi	960	.825	.831	$+.006$	.826	$+.001$	.670	.671	$+.001$	.670	$+.000$

Appendix

Table 12 : Wishart MAP vs. MLE cross-domain.

\Delta

= MAP

-

MLE. B

\to

N: 2AFC accuracy. N

\to

B: per-annotator agreement.

	BAPPS		NIGHTS
Encoder	$p$ ( $n^{*}$ )	$\bar{\alpha}\!\pm\!\sigma_{\alpha}$	$p$ ( $n^{*}$ )	$\bar{\alpha}\!\pm\!\sigma_{\alpha}$
CORnet-S V1	1.01 (4)	.08 $\pm$ .03	1.66 (8)	.67 $\pm$ .03
CORnet-S V2	1.09 (3)	.12 $\pm$ .02	1.66 (14)	.72 $\pm$ .04
CORnet-S V4	1.28 (4)	.18 $\pm$ .03	1.34 (48)	.82 $\pm$ .05
CORnet-S IT	2.29 (6)	.40 $\pm$ .02	1.63 (96)	.71 $\pm$ .01
CORnet-S Multi	1.36 (3)	.22 $\pm$ .02	1.51 (96)	.75 $\pm$ .02

Appendix

Table 13 : Learned weighted Minkowski parameters

p

and per-channel weights

\bm{\alpha}

at the saturation point

n^{*}

(in parentheses after

p

). For

\bm{\alpha}

we report both the mean and standard deviation across the

n^{*}

learned channels. Underline indicates the best-performing model in each case.

Explore similar work

MAME: Multidimensional Adaptive Metamer Exploration with Human Perceptual Feedback
Mar 17, 2025Mina Kamao, Hayato Ono, Ayumu Yamashita +2Visual Perception Human Brain Networks
Graduate School of Information Science and Technology The University of Tokyo, Tokyo, Japan · Graduate School of Information Science and Technology Hokkaido University, Hokkaido, Japan · Prometech CG Research, Tokyo, Japan
IQA-T1: Tool-based Visual Evidence Reasoning for Image Quality Assessment
Jul 14, 2026Jinjian Wu, Jiaqi Tang, Wei Wei +5Image Quality Assessment Visual Evidence
School of Computer Science, Northwestern Polytechnical University, Xi’an, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception
Jun 26, 2026Yana Wei, Hongbo Peng, Yanlin Lai +14Multimodal Evaluation Perception
Johns Hopkins University · StepFun · Tsinghua University +1

Mar 17, 2025, cs.LGJ/K move · Enter open · S save

MAME: Multidimensional Adaptive Metamer Exploration with Human Perceptual Feedback

Mina Kamao, Hayato Ono, Ayumu Yamashita, Kaoru Amano+1

Graduate School of Information Science and Technology The University of Tokyo, Tokyo, Japan · Graduate School of Information Science and Technology Hokkaido University, Hokkaido, Japan · Prometech CG Research, Tokyo, Japan

Visual Perception Human Brain Networks

Open paper AskPDF arXiv

Alignment between human brain networks and artificial models has become an active research area in vision science and machine learning. A widely adopted approach is identifying "metamers," stimuli physically different yet perceptually equivalent within a system. However, conventional methods lack a direct approach to searching for the human metameric space. Instead, researchers first develop biologically inspired models and then infer about human metamers indirectly by testing whether model metamers also appear as metamers to humans. Here, we propose the Multidimensional Adaptive Metamer Exploration (MAME) framework, enabling direct, high-dimensional exploration of human metameric spaces through online image generation guided by human perceptual feedback. MAME modulates reference images across multiple dimensions based on hierarchical neural network responses, adaptively updating generation parameters according to participants' perceptual discriminability. Using MAME, we successfully measured multidimensional human metameric spaces within a single psychophysical experiment. Experimental results using a biologically plausible CNN model showed that human discrimination sensitivity was lower for metameric images based on Gram-matrix representations derived from low-level CNN features than for those derived from high-level CNN features. The finding suggests a relatively worse alignment between the metameric spaces of humans and the CNN model for low-level processing compared to high-level processing. Counterintuitively, given recent discussions on alignment at higher representational levels, our results highlight the importance of early visual computations in shaping biologically plausible models. Our MAME framework can serve as a future scientific tool for directly investigating the functional organization of human vision.

Abstract

Explore similar work

MAME: Multidimensional Adaptive Metamer Exploration with Human Perceptual Feedback

IQA-T1: Tool-based Visual Evidence Reasoning for Image Quality Assessment

PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception