cs.LGNov 11, 2026

Measuring Preferred Representation in AI-generated Images with Community Scorers

Authors: Melanie F. Pradier, Camilla Longden, Anja Thieme, Rita F. Marques, Nari Johnson, Sidhika Balachandar, Xinnuo Xu, Linda Y. Wen, 

Organizations: Microsoft Research, Cambridge, United Kingdom · Carnegie Mellon University; Pittsburgh, PA USA · University of California; Berkeley, CA USA · Canva Research, Sydney, Australia

Abstract

AI-generated images often misrepresent historically marginalised communities, with real-world consequences beyond the images themselves. However, conventional metrics fail to measure how communities want to be represented in AI-generated images. To address this, we introduce the concept of preferred representation. Building on previous work, we propose a measurement instrument that operationalises this concept through a Community Scorer - a model that evaluates a specific community's preferred representation in AI-generated images. We interrogate the validity of Community Scorers on three diverse disability communities that span the Global North and South. The results show that Community Scorers align significantly better with community ratings compared to other widely used text-to-image metrics. We then demonstrate the usefulness of these scorers in a downstream content filtering task. We release the Community Preference Dataset to enable others to optimise how Community Scorers can be scaled for culturally specific evaluation of AI-generated visual media.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed

    Jul 27, 2026Xinnuo Xu, Anja Thieme, Daniela Massiceti +6T2I GenerationSafe T2I Generation

  2. TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design

    May 20, 2026Haonan Zhu, Elad Hirsch, Alexandria Minetti +3VLM EvaluationT2I Generation Evaluation