A Proxy-Based Method for Measuring Racial Disparities inText-to-ImageGenerators
Organizations: University of Virginia
Abstract
We present an automated method for auditing text-to-image generators for representational bias against ground-truth demographic benchmarks. Our approach adapts the racial proxy variables used in U.S. antidiscrimination law to provide a fine-grained comparative analysis of variations in racial representation in generated images. We classify each generated image by race and then compare the model's distribution to publicly available demographic benchmark data to identify patterns of over- and under-representation. We also measure skin tone using CIELAB values and calculate cosine similarity amongst groups of images in order to capture biases that don't show up in demographic comparisons such as differences in image quality across races and skin tones. We report on experiments testing three models (Imagen 4 Fast, Stable Diffusion 3.5 Large Turbo, and Flux Schnell) using three categories of proxies (geographic, occupational, and surname) and 50 variants within each category. We find notable variation across the models as well as by proxy category. For example, Imagen 4 Fast overrepresents white classified images more than the other models tested and produces images with lighter, more homogeneous skin tones. Imagen 4 Fast also had the largest representational disparity for Black demographics among surname and geographical prompts. Stable Diffusion 3.5 Large Turbo produces results that overall diverge most from census data but yield the most diverse range of skin tones. Our method yields a generalizable and legally-grounded auditing tool that surfaces detailed information about model outputs without relying on implicit normative definitions of fairness.
Figures & tables
| Paper | Prompts | Images | Occupation | Geography | Surname | Tone | Embed | Comparative | Automated |
| Bianchi et al. (2023) | 10 | 100 | ✓ | ✓ | × | × | × | × | ✓ |
| Castleman and Korolova (2025) | 18 | 20 | × | × | × | × | × | × | × |
| AlDahoul et al. (2025) | 40 | 10 K | ✓ | × | × | × | ✓ | × | ✓ |
| Wilson et al. (2025) | 93 | 50 | ✓ | × | × | ✓ | × | × | ✓ |
| Zhou et al. (2024) | 1016 | 6 | ✓ | × | × | × | × | × | ✓ |
| Cheong et al. (2024) | 105 | 10 | ✓ | × | × | × | × | × | × |
| Flux Schnell | Imagen - 4 Fast | Stable Diffusion | |||||||
| Mean | 95% CI | Std | Mean | 95% CI | Std | Mean | 95% CI | Std | |
| Overall | 0.310 | [0.309, 0.320] | 0.174 | 0.312 | [0.311, 0.322] | 0.155 | 0.452 | [0.448, 0.462] | 0.158 |
| Geography | 0.299 | [0.295, 0.315] | 0.110 | 0.325 | [0.319, 0.339] | 0.112 | 0.412 | [0.405, 0.430] | 0.105 |
| Occupation | 0.330 | [0.322, 0.345] | 0.160 | 0.313 | [0.308, 0.330] | 0.119 | 0.420 | [0.411, 0.436] | 0.137 |
| Surname | 0.302 | [0.298, 0.311] | 0.231 | 0.299 | [0.293, 0.310] | 0.215 | 0.523 | [0.513, 0.537] | 0.196 |
Demographic Bias in T2I Models
Demographic bias in text-to-image models is the unequal or stereotyped representation of people across generated images.
See all 33 papers