We investigate how successful bias mitigation reshapes the embedding space of encoder-only and decoder-only foundation models, offering an internal audit of model behaviour through representational analysis. Using BERT and Llama2 as representative architectures, we assess the shifts in associations between gender and occupation terms by comparing baseline and bias-mitigated variants of the models. Our findings show that bias mitigation reduces gender-occupation disparities in the embedding space, leading to more neutral and balanced internal representations. These representational shifts are consistent across both model types, suggesting that fairness improvements can manifest as interpretable and geometric transformations. These results position embedding analysis as a valuable tool for understanding and validating the effectiveness of debiasing methods in foundation models. To further promote the assessment of decoder-only models, we introduce WinoDec, a dataset consisting of 4,000 sequences with gender and occupation terms, and release it to the general public. (https://github.com/winodec/wino-dec)
Figures & tables
Figure 1: Self-attention between gender-occupation term pairs for decoder-only models.
Figure 2: An input sample from the real-world job shortlisting dataset with job description and candidate information.
Figure 3: Sample instances from the synthetic dataset consisting of stereotypical occupations and gender terms.
Data
Comparison
D-statistic
p-value
HR
“Female” vs “Male” (Baseline)
0.2600
< 0.0001
“Female” vs “Male” (Mitigated)
0.1290
0.0003
“Female”: Baseline vs Mitigated
0.1556
< 0.0001
“Male”: Baseline vs Mitigated
0.4554
< 0.0001
Plumber
“Female” vs “Male” (Baseline)
0.5200
< 0.0001
“Female” vs “Male” (Mitigated)
0.1600
0.5487
Table 1: Kolmogorov–Smirnov test results for cosine similarity distributions between gender terms and occupation terms across BERT models (baseline vs bias-mitigated).
Figure 4: Distributions of cosine similarities between embeddings of male gender terms and the “HR” occupation term and embeddings of female gender terms and the “HR” occupation for the baseline BERT model (left) and the bias-mitigated BERT model (right).
Figure 5: Distributions of cosine similarities between embeddings of male gender term and the job “plumber” and embeddings for female gender terms and “plumber” for the baseline BERT model (left) and the bias-mitigated BERT model (right).
Figure 6: Distributions of cosine similarities for embeddings of male and female gender terms split across stereotypically male (left) and female (right) occupations for the baseline and bias-mitigated BERT models.
Figure 7: Distributions of cosine similarities for embeddings of male and female gender terms for configuration “Gender 2 - Occupation 2”, split across stereotypically male jobs for the baseline and bias-mitigated Llama2 models.
Job Type
Comparison
D-statistic
p-value
Stereotypically Male Jobs
Female Terms vs Male Terms (Baseline)
0.2230
< 0.0001
Female Terms vs Male Terms (Mitigated)
0.1670
< 0.0001
Female Terms: Baseline vs Mitigated
0.3150
< 0.0001
Male Terms: Baseline vs Mitigated
0.3570
< 0.0001
Stereotypically Female Jobs
Female Terms vs Male Terms (Baseline)
0.2580
< 0.0001
Female Terms vs Male Terms (Mitigated)
0.1510
< 0.0001
Table 2: Kolmogorov–Smirnov test results for cosine similarity distributions for configuration “Gender 2 - Occupation 2” with Llama2 models.
Figure 8: Distributions of cosine similarities for embeddings of male and female gender terms for configuration “Gender 1 - Occupation 2”, split across stereotypically male and female jobs for the baseline and bias-mitigated Llama2 models.
Job Type
Comparison
D-statistic
p-value
Stereotypically Male Jobs
Female Terms vs Male Terms (Baseline)
0.0920
0.0004
Female Terms vs Male Terms (Mitigated)
0.0780
0.0045
Female Terms: Baseline vs Mitigated
0.1770
< 0.0001
Male Terms: Baseline vs Mitigated
0.1670
< 0.0001
Stereotypically Female Jobs
Female Terms vs Male Terms (Baseline)
0.0490
0.1812
Female Terms vs Male Terms (Mitigated)
0.0840
0.0017
Table 3: Kolmogorov–Smirnov test results for cosine similarity distributions for configuration “Gender 1 - Occupation 2” with Llama2 models.