Measuring Preferred Representation in AI-generated Images with Community Scorers
Authors: Melanie F. Pradier, Camilla Longden, Anja Thieme, Rita F. Marques, Nari Johnson, Sidhika Balachandar, Xinnuo Xu, Linda Y. Wen, Martin Grayson, Katja Hofmann, Cecily Morrison, Daniela Massiceti
Organizations: Microsoft Research, Cambridge, United Kingdom · Carnegie Mellon University; Pittsburgh, PA USA · University of California; Berkeley, CA USA · Canva Research, Sydney, Australia
AI-generated images often misrepresent historically marginalised communities, with real-world consequences beyond the images themselves. However, conventional metrics fail to measure how communities want to be represented in AI-generated images. To address this, we introduce the concept of preferred representation. Building on previous work, we propose a measurement instrument that operationalises this concept through a Community Scorer - a model that evaluates a specific community's preferred representation in AI-generated images. We interrogate the validity of Community Scorers on three diverse disability communities that span the Global North and South. The results show that Community Scorers align significantly better with community ratings compared to other widely used text-to-image metrics. We then demonstrate the usefulness of these scorers in a downstream content filtering task. We release the Community Preference Dataset to enable others to optimise how Community Scorers can be scaled for culturally specific evaluation of AI-generated visual media.
Figures & tables
Figure 1: Our contributions. We define, operationalise, and validate an automated metric of preferred representation. The Community Library corresponds to DCL and the Community Preference Dataset to DCP . The Preferred Representation box (bottom left) represents the target function we seek to approximate via a scalable proxy (bottom middle), the Community Scorer ϕ .
Method
BLV
DWF-S
DWF-N
CLIPScore
0.10±0.06
0.02±0.08
0.13±0.08
HPSv2
0.14±0.07
0.02±0.09
0.17±0.08
LLaVAScore
0.15±0.07
0.20±0.09
0.34±0.07
VQ2
0.00±0.06
−0.09±0.08
0.09±0.07
CAIRE
0.17±0.07
0.08±0.11
0.09±0.02
Zero-Shot Scorer
−0.05±0.02
−0.06±0.02
0.00±0.14
Table 1: Metric–human agreement across communities. Kendall’s τ correlation between automated metrics and community ratings on the held-out set, reported per community. Values are mean ± standard deviation estimated via 1000 bootstrap resamples with replacement.
Figure 2: Alignment improves with community supervision. We report QWK between Community Scorers and community ratings on the held-out set as a function of training data size. Community Scorers improve consistently with more data, while zero-shot Generic Scorers remain poorly aligned. Experiments are run with 5 random seeds controlling variability due to model parameter initialisation. The dashed BLV reference is within-lead repeat agreement (QWK 0.54).
Figure 3: Community Scorers improve retained-pool composition. Proportion of good and bad images retained by an automated evaluator after manual screening. Good images correspond to designated community lead ratings of 4 or 5, and bad images to ratings of 1 or 2, computed over all images that pass the automated filtering stage. GS and CS stand for Generic and Community Scorer.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: An example from the BLV Community Library (left) and its synthetic counterpart from the Community Preference Dataset (right). Generating prompt: “ A photo of a VI (with total sight loss) shop owner arranging products on shelves with the help of a sighted assistant, showcasing her entrepreneurship and effective organizational skills for managing inventory. Make sure to include: A middle aged woman shop owner who is blind holding two beauty products; Sighted assistant who is helping in arranging products in the shop systematically; Tagged and systematically arranged products on the shop shelves. The scene is in Kenya. ”
Figure 5: Web-based rating interface for data collection. This was used to collect community ratings for the Community Preference Dataset. Community annotators are shown a text prompt, a corresponding AI-generated image, and a discrete rating scale with optional free-text comments.
Figure 6: Number of samples generated from different prompt sources.
Figure 7: Intra-rater agreement across two annotation rounds conducted six months apart.
Figure 8: Representative off-diagonal examples where ratings differed between annotation rounds.
Table 3: Vision LoRA adapter configuration. Target modules are all self-attention ( qkv_proj , o_proj ) and MLP ( gate_up_proj , down_proj ) projections across all transformer layers. The image encoder parameters are additionally unfrozen.
Hyperparameter
Value
Optimisation
Optimiser
AdamW ( adamw_torch )
Learning rate
4×10−5
LR scheduler
Linear decay
Warmup steps
50
Weight decay
0.01
Appendix
Table 4: Training hyperparameters for Phi-4 multimodal fine-tuning.
Experiment
Training Time
BLV
3 h 20 min
DWF-S
2 h 15 min
DWF-N
1 h 50 min
Appendix
Table 5: Approximate training times per experiment on a single A100 80 GB GPU.
Figure 9: Alignment between QWK and downstream quality metrics. Higher Quadratic Weighted Kappa (QWK) corresponds to a higher fraction of images rated positively and a lower fraction rated negatively by the community.
Method
BLV
DWF-N
DWF-S
CAIRE
-0.00
0.07
-0.04
Generic Scorer
-0.02
0.05
-0.00
Community Scorer
0.24
0.44
0.46
Appendix
Table 6: Quadratic Weighted Kappa (QWK) on a held-out set by community and method. QWK is only reported for methods producing ordinal scores. Community Scorers achieve the highest agreement across all communities.
Figure 10: Confusion matrices for the best Community Scorers. Predictions concentrate along the diagonal, with most disagreements limited to neighbouring rating levels.
Figure 11: Marginal score distributions. Predicted and human rating distributions are closely aligned, indicating good coverage of the community rating scale.
Figure 12: Calibration curves for Community Scorers. Predicted scores closely track empirical human ratings, indicating good calibration across most of the scale.
Figure 13: Qualitative impact of Community Scorers in content filtering. Example outputs from the filtering pipeline. Images in the first row are identified as poor representations (low alignment with community ratings) and filtered out, while images in the second row are retained as better aligned with the community’s preferred representation.
Figure 14: Threshold ablation for content filtering. Intermediate thresholds balance removal of low-rated content with retention of high-quality images.
Figure 15: Qualitative results for community-aligned content filtering across communities. For each community, the Community Scorer filters out images with subtle representational issues or misalignment, yielding final outputs that better reflect community preferences and norms.
Text-to-image systems use learned aesthetic scorers to filter training data and guide generation, but whether these scores encode demographic attributes as objective quality is unclear. We audit four scorers (LAION-Aesthetics, PickScore, ImageReward, HPSv2) using pixel-level interventions on skin tone and body type in synthetic and real images. Our key finding is that along skin-lightness, the dominant effect is fidelity preference: unaltered images score highest, and perturbations in either direction are penalized (inverted-U). Placebo arms show this penalty is not an artifact of the skin operator, as applying the same CIELAB L* shift to non-skin regions yields similar penalty magnitudes. However, the penalty is operator-dependent and holds for all operators only for LAION-Aes. Critically, audits on synthetic images alone are misleading: LAION-Aes shows strong preference for darker skin on synthetic faces, but on 1470 real faces the preference reverses and becomes much smaller, and amplification becomes non-significant. Across scorers, synthetic results do not transfer -- reversing for LAION-Aes and HPSv2, attenuating for PickScore. We contribute a reproducible benchmark with artifact control and synthetic/real cross-validation, and an auditability criterion for pixel-level causal isolation (valid for skin tone, not for body type due to deformation). Population-stratified analysis shows fidelity-penalty asymmetry is not robust across groups after FDR correction except for HPSv2. Our findings show naive synthetic audits misjudge bias direction and magnitude, and only within-image causal isolation on real data can distinguish true demographic bias from fidelity preference.
State-of-the-art toxicity detectors for text-to-image generation adopt a one-size-fits-all approach: a single universal model applying fixed safety guidelines to all users. Our empirical evidence shows that these detectors fail to shield marginalized communities: approximately 35% of generated images labeled safe are considered harmful by disability communities. In this position paper, we argue for community-specific toxicity detection (CTD). To demonstrate its feasibility, we collaborate with disability experts to develop safety guidelines for two communities: dwarfism and blind/low vision. Using a dataset of 2,400 annotated T2I-generated images we demonstrate that both large vision-language models and existing general-purpose toxicity detectors catastrophically fail to recognize harmful content under these guidelines in zero-shot settings with F1 score lower than random guessing (F1 0.32 and 0.37). Promisingly, prompt-based adaptation methods (ICL, VQA) substantially improve harm detection performance (GPT-4o: F1 0.50 and 0.78), while parameter-efficient fine-tuning improves smaller models (0.5b-7b with best F1 0.48 and 0.59) with less than 100 demonstrations, but remains sensitive to evolving guidelines. Despite these gains, CTD performance remains far below F1 ≈0.9 achieved for general-purpose toxicity detection, highlighting the challenge and the need for sustained research effort.
Xinnuo Xu, Anja Thieme, Daniela Massiceti +6
Microsoft Research Cambridge, UK · Microsoft Paris, France
Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison. Designers evaluate designs along several distinct axes (e.g., typography, layout, color harmony) that a single preference label collapses. We release TASTE(Typography, Aesthetics, Spatial, Tone, Etc.), a multi-dimensional preference dataset in which two disjoint cohorts of five professional designers each ranked outputs from four current text-to-image models across nine criteria along with per-image hallucination flags. We pair the dataset with two contributions. First, a criterion-agnostic signal-validation framework based on Kendall's τ, majority-vote probability, and Condorcet cycles against exact iid-uniform nulls; the analysis reveals significant but moderate designer agreement, with every TASTE criterion rejecting the random-rater null. Second, we benchmark preference models on TASTE and find that off-the-shelf VLM judges and dedicated T2I scorers fail to reach majority agreement with the designer panel, while a small MLP head trained directly on TASTE substantially narrows the gap to the single-rater ceiling, setting a baseline for future TASTE-trained preference models.
Haonan Zhu, Elad Hirsch, Alexandria Minetti +3
Lica World, San Francisco, USA. · Contra.Work Inc., Brooklyn, NY, USA.