Organizations: Department of Information Technology, Rajkiya Engineering College Banda, 210201, Uttar Pradesh, India. · School of AI and Data Science, Indian Institute of Technology Jodhpur, 342030, Rajasthan, India.
Text-to-image (T2I) generative models achieve impressive visual fidelity but inherit and amplify demographic imbalances and cultural biases embedded in training data. We introduce T2I-BiasBench, a unified evaluation framework of thirteen complementary metrics that jointly captures demographic bias, element omission, and cultural collapse in diffusion models - the first framework to address all three dimensions simultaneously. We evaluate three open-source models - Stable Diffusion v1.5, BK-SDM Base, and Koala Lightning - against Gemini 2.5 Flash (RLHF-aligned) as a reference baseline. The benchmark comprises 1,574 generated images across five structured prompt categories. T2I-BiasBench integrates six established metrics with seven additional measures: four newly proposed (Composite Bias Score, Grounded Missing Rate, Implicit Element Missing Rate, Cultural Accuracy Ratio) and three adapted (Hallucination Score, Vendi Score, CLIP Proxy Score). Three key findings emerge: (1) Stable Diffusion v1.5 and BK-SDM exhibit bias amplification (>1.0) in beauty-related prompts; (2) contextual constraints such as surgical PPE substantially attenuate professional-role gender bias (Doctor CBS = 0.06 for SD v1.5); and (3) all models, including RLHF-aligned Gemini, collapse to a narrow set of cultural representations (CAS: 0.54-1.00), confirming that alignment techniques do not resolve cultural coverage gaps. T2I-BiasBench is publicly released to support standardized, fine-grained bias evaluation of generative models. The project page is available at: https://gyanendrachaubey.github.io/T2I-BiasBench/
Figures & tables
Figure 1: Overview of the proposed Bias and Fairness Evaluation Framework for text-to-image models. The pipeline consists of prompt generation (demographic and contextual), controlled image synthesis across multiple models, attribute extraction via vision-language methods, and a multi-metric evaluation module comprising 13 fairness, diversity, and alignment metrics. The framework produces composite bias and diversity scores, enabling comparative analysis of model behavior across demographic attributes and contextual scenarios.
Model
Type
Scale
#Images
Stable Diffusion v1.5 Rombach et al. (2022)
Open-source
∼860 M params
496
BK-SDM Base Kim et al. (2024)
Open-source (distilled)
Compact
499
Koala Lightning Lee et al. (2024)
Open-source (distilled)
Compact
500
Gemini 2.5 Flash Imran and Almusharraf (2024)
Proprietary (RLHF)
Large-scale
75 (baseline)
Table 1: Models selected for evaluation.
Prompt ID
Prompt Text
Bias Dimension
Beauty
“A fashion magazine cover featuring the most beautiful person in the world.”
“An insect resting on a flower in soft morning sunlight.”
Non-human baseline — lighting and species diversity
Table 2: Prompt design. Rows marked (*) are non-human contextual baselines used to isolate capability gaps from demographic bias.
Metric
Formula / Range
Interpretation
Reference
Representation Parity
pg=Ng/N
Raw group proportion; foundation metric
Friedrich et al. (2024)
Parity Difference
∣pa−pb∣∈[0,1]
0 = equal groups; 1 = only one group
Friedrich et al. (2024)
Bias Amplification
∑∣pi−1/k∣
>1.0 amplifies beyond training data
Zhao et al. (2017)
Shannon Entropy
H=−∑plog2p
Higher = more diverse output distribution
Info. theory
KL Divergence
KL(P∥U)
0 = perfectly fair distribution
Li et al. (2025)
CAS Score
S/(S+D+ε)
0 = diverse; 1 = fully stereotyped
Vice et al. (2023)
Table 3: Thirteen-metric evaluation framework. Rows marked † (blue-shaded in colour) are newly proposed metrics; rows marked ‡ (grey-shaded in colour) are adapted from existing work. PD = Parity Difference; H = normalised entropy; k = group count; S , D = stereotype / diverse term counts; ε = smoothing constant. 0 0 footnotetext: †Newly proposed metrics (this work): CBS = Composite Bias Score; GMR = Grounded Missing Rate; IEMR = Implicit Element Missing Rate; CAR = Cultural Accuracy Ratio. ‡Adapted from existing work and applied to T2I bias evaluation.
Table 5: 5×4 Generated Image Gallery (continued, rows 4–5). Colour-coding as per Table 4 : Low ( ≤0.30 , green-shaded), Moderate ( 0.31 – 0.55 , amber-shaded), High ( >0.55 , red-shaded). * * footnotetext: Gemini Doctor CBS =1.00 reflects counter-stereotyping (100% female); BA GEM,Doctor=0.00 confirms no active stereotype reinforcement (see Section 4.4 ). VAOP = Visual Attribute Occlusion Prompting (Section 4.4 ).
Model
Beauty
Doctor
Animal
Nature
Culture
μˉ
SD v1.5
0.50
0.06
0.30
0.32
0.66
0.37
Koala Lightning
0.51
0.76
0.43
0.23
0.48
0.48
BK-SDM Base
0.59
0.20
0.28
0.30
0.79
0.43
Gemini 2.5 Flash
0.33
1.00∗
0.20
0.20
0.60
0.47
Best model
Gemini
SD v1.5
Gemini
Gemini/Koala
Koala
SD v1.5
Table 6: Composite scores by model and prompt. Cells are colour-coded: low bias ( ≤0.30 , green-shaded), moderate ( 0.31 – 0.55 , amber-shaded), high ( >0.55 , red-shaded). ∗ Gemini Doctor CBS reflects female over-representation (counter-stereotyping); BA GEM,Doctor=0.00 . μˉ denotes the per-model mean across all prompts. Boldface marks the per-column optimum.
Figure 2: Composite bias and diversity scores ( 0 = fair, 1 = biased) across all five prompt categories for all four models. Horizontal threshold bands demarcate low ( ≤0.30 ), moderate ( 0.31 – 0.55 ), and high ( >0.55 ) severity regions. The Culture axis is consistently elevated across all models, providing quantitative evidence of systemic cultural representation collapse.
Figure 3: Ethnic composition P^(e∣Beauty) for e∈E across all four models. White-dominant distributions in SD v1.5 and BK-SDM contrast sharply with Gemini’s near-uniform ethnic coverage.
Figure 4: Beauty prompt — KL Divergence from the uniform reference KL(P^∥U) (Eq. 8 , left) and normalised Shannon Entropy H (Eq. 7 , right).
Figure 5: Bias Amplification (BA, Eq. 9 ) for the Beauty and Doctor prompts. Bars exceeding the dashed threshold BA =1.0 (red) indicate active stereotype reinforcement beyond the training distribution.
Figure 6: Gender distribution P^(g∣Doctor) for g∈{M,F,U} across all four models. Koala exhibits severe male dominance ( P^(M)=0.91 ); Gemini over-corrects to full female dominance ( P^(F)=1.00 ).
Figure 7: Animal baseline — Grounded Missing Rate (GMR, Eq. 17 ) for puzzle fidelity and Implicit Element Missing Rate (IEMR, Eq. 19 ) for laboratory context across all four models. Lower bars indicate better prompt fidelity.
Figure 8: Culture prompt — Contextual Association Score CAS (Eq. 12 , stereotype breadth, red bars) versus Cultural Accuracy Ratio CAR (Eq. 22 , green bars). The accuracy–breadth paradox is most visible in SD v1.5 (high CAR, high CAS) and Gemini (maximal CAS, maximal CAR).
Figure 9: Vendi Score (VS, Eq. 14 , caption lexical diversity) and CLIP Proxy Score (CPS, Eq. 15 , prompt–caption semantic alignment) across all models and prompts. High VS with variable CPS reveals a diversity–alignment decoupling in T2I models.
Figure 10: Bias profile radar chart across all five prompts for all four models. Smaller enclosed area Am indicates an overall fairer model. The universally extended Culture axis confirms systemic cultural representation collapse across all architectures, including the RLHF-aligned Gemini baseline.
Finding
Metric
Value
Model
Prompt
Largest ethnic skew
KL ↓
0.765
SD v1.5
Beauty
Smallest ethnic skew
KL ↓
0.063
Gemini
Beauty
Max bias amplification
BA ↓
1.15
Koala
Doctor
Min bias amplification
BA ↓
0.00
Gemini
Doctor
Worst gender parity
PD ↓
0.85
Koala
Doctor
Best composite score
CBS ↓
0.06
SD v1.5
Doctor
Table 7: Summary of key quantitative findings. ↓ = lower is better; ↑ = higher is better. All metrics defined in Section 3.5 . 0 0 footnotetext: ∗ Gemini Doctor CBS =1.00 reflects the metric polarity artefact (Section 4.4 ): female dominance, not male-dominant bias reinforcement.