Fidelity Preference, Not Demographic Preference: A Pixel-Level Attribute-Sensitivity Audit of Image Aesthetic/Preference Scorers

Mingyang Xu

cs.CVAug 17, 2026

Fidelity Preference, Not Demographic Preference: A Pixel-Level Attribute-Sensitivity Audit of Image Aesthetic/Preference Scorers

Authors: Mingyang Xu

Organizations: Peking University Beijing, China

Abstract

Text-to-image systems use learned aesthetic scorers to filter training data and guide generation, but whether these scores encode demographic attributes as objective quality is unclear. We audit four scorers (LAION-Aesthetics, PickScore, ImageReward, HPSv2) using pixel-level interventions on skin tone and body type in synthetic and real images. Our key finding is that along skin-lightness, the dominant effect is fidelity preference: unaltered images score highest, and perturbations in either direction are penalized (inverted-U). Placebo arms show this penalty is not an artifact of the skin operator, as applying the same CIELAB L* shift to non-skin regions yields similar penalty magnitudes. However, the penalty is operator-dependent and holds for all operators only for LAION-Aes. Critically, audits on synthetic images alone are misleading: LAION-Aes shows strong preference for darker skin on synthetic faces, but on 1470 real faces the preference reverses and becomes much smaller, and amplification becomes non-significant. Across scorers, synthetic results do not transfer -- reversing for LAION-Aes and HPSv2, attenuating for PickScore. We contribute a reproducible benchmark with artifact control and synthetic/real cross-validation, and an auditability criterion for pixel-level causal isolation (valid for skin tone, not for body type due to deformation). Population-stratified analysis shows fidelity-penalty asymmetry is not robust across groups after FDR correction except for HPSv2. Our findings show naive synthetic audits misjudge bias direction and magnitude, and only within-image causal isolation on real data can distinguish true demographic bias from fidelity preference.

Figures & tables

Dimension	Corpus/provenance audits ( ( 13 ) )	Categorical counterfactual audits ( ( 1 ) )	This work
Unit of analysis	Corpus of existing images	Matched counterfactual image sets	Single image, graded perturbation
Attribute treatment	Caption keywords, artist/culture metadata	Categorical identity (race, gender)	One continuous axis (CIELAB $L^{*}$ ), measured per image via ITA ∘
Confound control	Large- $n$ distributional contrast	Attributes vary, context held constant across sets	Within-image pixel/geometric intervention + intervention-artifact covariate calibration
Mechanism evidence	Provenance (trace ethnography) $\rightarrow$ “measures photorealism”	Score disparity between groups	Response shape to dose (inverted U) + placebo operators
Synthetic-vs-real	Real images only	Synthetic counterfactual sets	Both arms side by side , showing synthetic effects do not transfer
Downstream	Filtering rates at a score threshold	Reward-guided optimization outcomes	best-of- $n$ amplification curve + random null calibration

Table 1. Positioning of this work relative to the two nearest prior audits.

Scorer	Fidelity index [95% CI]	Asymmetry [95% CI]	Inv. U	best-of- $n$ net drift ( $n{=}4$ )
LAION-Aes	$-0.005$ [ $-0.037$ , $+0.028$ ]	$\mathbf{-0.211}$ [ $-0.269$ , $-0.153$ ] ✓	No	$\mathbf{-26.1^{\circ}}$ [ $-32.5$ , $-19.1$ ] ✓
PickScore	$+0.256$ [ $+0.222$ , $+0.291$ ] ✓	$-0.051$ [ $-0.102$ , $+0.004$ ]	Yes	$-5.2^{\circ}$ [ $-9.6$ , $-1.1$ ] ✓
HPSv2	$+0.0013$ [ $+0.0002$ , $+0.0023$ ] ✓	$-0.0027$ [ $-0.0041$ , $-0.0014$ ] ✓	No	$-14.3^{\circ}$ [ $-21.2$ , $-7.3$ ] ✓
ImageReward	$-0.019$ [ $-0.037$ , $-0.002$ ] $\dagger$	$+0.014$ [ $-0.021$ , $+0.052$ ]	No	$-0.2^{\circ}$ [ $-9.4$ , $+8.7$ ] $\times$

Table 2. Within-image intervention, synthetic faces (

n=54

base images

\times

\Delta l

|\Delta l|\leq 15

, clip_frac constant 0). Fidelity curve referenced to the original (

\Delta l=0

Scorer	Fidelity index [95% CI]	Asymmetry [95% CI]	Inv. U	best-of- $n$ net drift ( $n{=}4$ )
LAION-Aes	$\mathbf{+0.074}$ [ $+0.057$ , $+0.090$ ] ✓	$+0.032$ [ $+0.005$ , $+0.059$ ] ✓	Yes	$+0.2^{\circ}$ [ $-2.3$ , $+2.6$ ] $\times$
PickScore	$+0.088$ [ $+0.070$ , $+0.107$ ] ✓	$-0.008$ [ $-0.031$ , $+0.015$ ]	Yes	$\mathbf{-3.2^{\circ}}$ [ $-5.5$ , $-0.9$ ] ✓
HPSv2	$-0.0008$ [ $-0.0014$ , $-0.0003$ ] ✓	$+0.0022$ [ $+0.0014$ , $+0.0029$ ] ✓	No	$+4.3^{\circ}$ [ $+1.7$ , $+7.1$ ] ✓
ImageReward	$+0.012$ [ $-0.0001$ , $+0.025$ ]	$+0.016$ [ $-0.005$ , $+0.038$ ]	No	$-1.3^{\circ}$ [ $-3.8$ , $+1.2$ ] $\times$

Table 3. Within-image intervention, real faces (FairFace,

n=210

base images

\times

\Delta l

= 1470 total images). Same intervention, same analysis.

Scorer	FI: synth $\rightarrow$ real	Asymmetry: synth $\rightarrow$ real	Amplification: synth $\rightarrow$ real
LAION-Aes	$-0.005$ (ns) $\rightarrow$ $+0.074$ (sig)	$\mathbf{-0.211\rightarrow+0.032}$ , both sig: reversal	$-26.1^{\circ}$ (sig) $\rightarrow$ $+0.2^{\circ}$ (ns): collapse
PickScore	$+0.256\rightarrow+0.088$ , both sig, same sign	$-0.051$ (ns) $\rightarrow$ $-0.008$ (ns)	$-5.2^{\circ}\rightarrow-3.2^{\circ}$ , both sig, same sign
HPSv2	$+0.0013\rightarrow-0.0008$ , both sig: reversal	$-0.0027\rightarrow+0.0022$ , both sig: reversal	$-14.3^{\circ}\rightarrow+4.3^{\circ}$ , both sig: reversal
ImageReward	$-0.019$ (sig) $\rightarrow$ $+0.012$ (ns): collapse	$+0.014$ (ns) $\rightarrow$ $+0.016$ (ns)	$-0.2^{\circ}$ (ns) $\rightarrow$ $-1.3^{\circ}$ (ns)

Table 4. How general is the divergence? On a strict criterion ( both arms FDR-significant and opposite in sign) only 4 of the 12 scorer

\times

reading cells qualify.

Scorer	Skin-tone FI	bg FI [95% CI]	hue FI [95% CI]	blur FI [95% CI]
LAION-Aes	$+0.0735$	$\mathbf{+0.0596}$ [ $+0.0409$ , $+0.0785$ ]	$+0.0319$ [ $+0.0169$ , $+0.0468$ ]	$\mathbf{+0.4359}$ [ $+0.4067$ , $+0.4654$ ]
PickScore	$+0.0883$	$\mathbf{+0.1075}$ [ $+0.0850$ , $+0.1307$ ]	$+0.0951$ [ $+0.0809$ , $+0.1092$ ]	$-0.3542$ [ $-0.3798$ , $-0.3293$ ]
HPSv2	$-0.0008$	$-0.0020$ [ $-0.0026$ , $-0.0013$ ]	$-0.0039$ [ $-0.0044$ , $-0.0034$ ]	$-0.0132$ [ $-0.0140$ , $-0.0124$ ]
ImageReward	$+0.0125$	$+0.0104$ [ $-0.0044$ , $+0.0266$ ]	$-0.0404$ [ $-0.0550$ , $-0.0257$ ]	$+0.1616$ [ $+0.1376$ , $+0.1856$ ]

Table 5. Placebo arms on the same 210 FairFace bases. Same estimator, bootstrap over base images,

B=2000

, seed 0.

Scorer	Fidelity index [95% CI]	Asymmetry [95% CI]	Inverted U
LAION-Aes	$+0.110$ [ $+0.079$ , $+0.144$ ] ✓	$-0.010$ [ $-0.060$ , $+0.038$ ]	Yes
PickScore	$+0.078$ [ $+0.058$ , $+0.098$ ] ✓	$+0.015$ [ $-0.030$ , $+0.060$ ]	Yes
ImageReward	$+0.040$ [ $+0.024$ , $+0.057$ ] ✓	$-0.010$ [ $-0.046$ , $+0.024$ ]	Yes
HPSv2	$+0.0019$ [ $+0.0014$ , $+0.0025$ ] ✓	$+0.0037$ [ $+0.0027$ , $+0.0047$ ] ✓	No (magnitude negligible)

Table 6. Fidelity curve, synthetic body (

n=54

base images

\times

\gamma

, referenced to the

\gamma=1.0

original; endpoints

\gamma=0.8/1.2

). Bootstrap over base images,

B=2000

, seed 0. ✓ = FDR-significant in the 24-test FI/asymmetry grid of Section 7 ; all four fidelity indices survive, and no asymmetry does except HPSv2’s, whose magnitude is negligible.

Scorer	$\beta$ (width $\|$ control warp)	Sig	$\gamma$ ( warp_artifact )	Sig
LAION-Aes	$-0.111$ [ $-0.48$ , $+0.09$ ]	No	$\mathbf{-0.813}$ [ $-1.14$ , $-0.54$ ]	✓
PickScore	$-0.252$ [ $-0.63$ , $-0.03$ ]	Yes	$\mathbf{-0.680}$ [ $-0.89$ , $-0.51$ ]	✓
ImageReward	$-0.131$ [ $-0.38$ , $+0.04$ ]	No	$\mathbf{-0.307}$ [ $-0.49$ , $-0.15$ ]	✓
HPSv2	$+0.0001$ [ $-0.005$ , $+0.009$ ]	No	$\mathbf{-0.023}$ [ $-0.027$ , $-0.018$ ]	✓

Table 7. Discriminative regression separating the pure width effect from the deformation artifact:

\mathrm{score}=\alpha+\beta\cdot\mathrm{width}+\gamma\cdot\texttt{warp\_artifact}

Condition	Meaning	Skin lightness	Body width
C1 composition-freezable	Can only the attribute be changed without touching composition/pose/background	✓ same image $L^{*}$ shift	$\times$ changing body type pushes pixel geometry, falls back to cross-image
C2 reversible low-artifact intervention	Is the attribute-changing operation reversible without introducing an attribute-confounded artifact	✓ clip_frac =0 ( $\|\Delta l\|\leq 15$ )	$\times$ warp_artifact dominates the inverted U ( $\gamma$ significant for all 4 scorers)
C3 valid continuous measurement	Is there a monotonic, calibrated continuous objective measure of the attribute	✓ ITA ∘ $R^{2}=0.88$ , monotonic rate 98%	$\times$ width_ratio 90% saturated, $R^{2}=0.02$

Table 8. From case to criterion: the three conditions of attribute auditability.

Arm / axis	Instrument	Known dose	Slope	$R^{2}$	Monotonic	Spearman	Verdict
Real face / skin	ITA ∘	$\Delta L^{*}$	$+2.05$	$\mathbf{0.790}$	$\mathbf{97.6\%}$	$+0.499$	Valid
Synth. face / skin	ITA ∘	$\Delta L^{*}$	$+3.35$	$\mathbf{0.876}$	$\mathbf{98.1\%}$	$+0.734$	Valid
Synthetic body	width_ratio	warp $\gamma$	$+0.066$	$\mathbf{0.021}$	66.7%	$+0.037$	Invalid

Table 9. Measurement-validity calibration: within-base regression of the measured value on the known intervention dose.

Scorer	Fidelity index (mean $\pm$ std, sig rate)	Discrim. $\beta$ (sig rate)	Net drift (mean $\pm$ std, sig rate, same-sign)
LAION-Aes	$+0.0735\pm 0.0000$ (100%)	$+0.0004$ (100%)	$+0.19^{\circ}\pm 0.09$ ( 0% , 100%)
PickScore	$+0.0883\pm 0.0000$ (100%)	$-0.0003$ (100%)	$-3.18^{\circ}\pm 0.12$ (100%, 100%)
ImageReward	$+0.0125\pm 0.0000$ (100%)	$+0.0003$ (0%)	$-1.22^{\circ}\pm 0.11$ (0%, 100%)
HPSv2	$-0.0008\pm 0.0000$ (100%)	$+0.0000$ (100%)	$+4.33^{\circ}\pm 0.10$ (100%, 100%)

Table 10. Robustness over 5 random seeds (real-face skin axis). Same-sign rate is 100% on every reading.

Scorer	Penalty-strength omnibus (fdr)	Directional-bias omnibus (fdr)	Conclusion
LAION-Aes	0.422 (0.483)	0.155 (0.249)	population differences not robust
PickScore	0.021 (0.057)	0.347 (0.462)	strength marginal (fails), no direction
ImageReward	0.613 (0.613)	0.050 (0.100)	direction marginal, no strength
HPSv2	0.006 (0.024 ✓)	0.004 (0.024 ✓)	population differences robust

Table 11. Population stratification: selection-free omnibus permutation tests, BH-FDR over 4 scorers

\times

2 metrics. Only HPSv2 survives.

Path	Role
config.py	Single source of truth: 4 scorers $\times$ 2 architecture families, 2 axes, 3 arms, paths/environment
orchestrator.py	One-click pipeline: scoring $\rightarrow$ three probes $\rightarrow$ report ( --analyze-only for second-scale reproduction)
report.py	Orchestration of three probes $\rightarrow$ audit_report.json
PROTOCOL.md	Methodology: within-image intervention + artifact calibration + best-of- $n$ null + fidelity curve

Table 12. Structure of the released benchmark.

Figure 1. Fidelity curves showing within-base score change (

\Delta

score) as a function of skin lightness intervention (

\Delta l

). (a) Synthetic faces show LAION-Aes monotonic preference for darker skin (asymmetry

-0.21

). (b) Real faces show symmetric inverted-U for LAION-Aes and PickScore, demonstrating fidelity preference.

Figure 2. Best-of-

n

amplification comparison between synthetic and real arms. The synthetic arm shows

-26.1^{\circ}

ITA drift for LAION-Aes, while the real arm shows

+0.2^{\circ}

(not significant): a

{\sim}129\times

magnitude collapse rather than a sign flip.

Figure 3. Body axis intervention artifact decomposition. The deformation artifact (

\gamma

) dominates the inverted-U pattern while the pure width effect (

\beta

) is weak after controlling for warp.

Figure 4. Population stratification omnibus test results. Comparing disparity (largest pairwise contrast) against selection-free omnibus

p

-values demonstrates that most apparent population differences are cherry-picking artifacts. Only HPSv2 survives FDR correction.

Algorithmic Bias

Algorithmic bias studies how automated systems produce or amplify unfair outcomes across demographic groups, and how to measure and reduce them.

See all 100 papers

Oct 20, 2026

Attractive Faces Skew Text-to-Image Reward Models and Leaderboards

Human preference is now central to the evaluation, guidance, finetuning, and ranking of text-to-image models. However, the susceptibility of human preferences to bias in image generation remains underexplored. We identify a key observation: for human faces, perceived attractiveness overrides image quality in preference judgments, even when comparing real photographs of comparable fidelity. In a controlled user study, annotators rank more attractive faces as higher quality, even when unattractive real faces are compared with attractive AI-generated ones. We examine the downstream consequences of this bias. Firstly, simulating Elo rankings from user study data shows attractiveness can distort a leaderboard's ranking, elevating synthetic attractive faces over real, unattractive ones. Secondly, the same bias reappears in the human preference models. Editing 40 annotated facial attributes in isolation, we find that attractiveness is the strongest correlate of score for four of five human preference models (PickScore, MPS, HPSv2.1, HPSv3), and the strongest predictor of score in regression analysis. We further show that preference aligned generation shifts faces toward younger, more feminine faces, and can override explicit prompts for unattractive faces, thereby limiting user control. Finally, we explore mitigation strategies, both for data collection and post training on human preference models. These findings highlight the need for preference models that disentangle image quality from human cognitive bias, enabling more reliable evaluation.

Harvey Mannering, Zhiwu Huang, Adam Prügel-Bennett

University of Southampton

T2I Generation Evaluation Human Preference Evaluation Human Preference Modeling Preference Optimization for Diffusion Models

Ask PDF

Oct 4, 2026

PortraitAes: Intent-Conditioned Structured Portrait Aesthetic Assessment

Portrait aesthetic assessment assigns comparable scores according to how effectively human-centered images fulfill their photographic intent. These scores support data filtering, candidate selection, and preference modeling in image-generation pipelines. Existing methods typically predict a single aesthetic score or use general-purpose MLLMs without conditioning on photographic intent. This omission matters because the same blur, pose, lighting, or framing choice may serve one photographic intent but undermine another. These models thus learn context-agnostic aesthetic priors and yield inconsistent, inaccurate, misleading judgments for portraits with distinct photographic objectives. We introduce PortraitAes-Bench, an 11K-scale benchmark that decomposes this task into intent-conditioned subjudgments. Expert-authored rubrics define nine photographic intents, six first-level dimensions, and 22 secondary criteria. They support a structured pipeline for intent routing, specialist assessment, verification, and score fusion. Following this structure, we train PortraitAes with multi-task supervision. We then improve score comparability through Gaussian score calibration and within-dimension cross-image ranking. On the standard benchmark, PortraitAes achieves a Pearson correlation of 0.924 and a Spearman rank correlation of 0.934. On the hard-case set, its Pearson correlation is 0.829 and its Spearman rank correlation is 0.795. Across both sets, PortraitAes outperforms the evaluated general-purpose MLLMs and specialized aesthetic baselines.

Junzhou Xie, Haozhong Xiong, Xunyun Tian +8

Southeast University · Qwen Business Unit of Alibaba

Image Quality Assessment

Ask PDF

May 20, 2026

TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design

Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison. Designers evaluate designs along several distinct axes (e.g., typography, layout, color harmony) that a single preference label collapses. We release TASTE (Typography, Aesthetics, Spatial, Tone, Etc.), a multi-dimensional preference dataset in which two disjoint cohorts of five professional designers each ranked outputs from four current text-to-image models across nine criteria along with per-image hallucination flags. We pair the dataset with two contributions. First, a criterion-agnostic signal-validation framework based on Kendall's

τ

, majority-vote probability, and Condorcet cycles against exact iid-uniform nulls; the analysis reveals significant but moderate designer agreement, with every TASTE criterion rejecting the random-rater null. Second, we benchmark preference models on TASTE and find that off-the-shelf VLM judges and dedicated T2I scorers fail to reach majority agreement with the designer panel, while a small MLP head trained directly on TASTE substantially narrows the gap to the single-rater ceiling, setting a baseline for future TASTE-trained preference models.

Haonan Zhu, Elad Hirsch, Alexandria Minetti +3

Lica World, San Francisco, USA. · Contra.Work Inc., Brooklyn, NY, USA.

VLM Evaluation T2I Generation Evaluation

Ask PDF

Abstract

Algorithmic Bias

Explore similar work

Attractive Faces Skew Text-to-Image Reward Models and Leaderboards

PortraitAes: Intent-Conditioned Structured Portrait Aesthetic Assessment

TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design