Standard fairness audits of foundation models quantify that a model is biased, but not where inside the network the bias resides. We propose a mechanistic fairness audit that combines projected residual-stream decomposition, zero-shot Concept Activation Vectors, and bias-augmented TextSpan analysis to locate demographic bias at the level of individual attention heads in vision transformers. As a feasibility case study, we apply this pipeline to the CLIP ViT-L-14 encoder on 42 profession classes of the FACET benchmark, auditing both gender and age bias. For gender, the pipeline identifies four terminal-layer heads whose ablation reduces global bias (Cramer's V: 0.381 -> 0.362) while marginally improving accuracy (+0.42%); a layer-matched random control confirms that this effect is specific to the identified heads. A single head in the final layer contributes to the majority of the reduction in the most stereotyped classes, and class-level analysis shows that corrected predictions shift toward the correct occupation. For age, the same pipeline identifies candidate heads, but ablation produces weaker and less consistent effects, suggesting that age bias is encoded more diffusely than gender bias in this model. These results provide preliminary evidence that head-level bias localisation is feasible for discriminative vision encoders and that the degree of localisability may vary across protected attributes. keywords: Bias . CLIP . Mechanistic Interpretability . Vision Transformer . Fairness
Figures & tables
Concept Prototypes for Demographic and Occupation Attributes
Concept
Prototype Texts (5 per concept)
Demographic Attributes (Bias Prototypes)
Male
Male person, Man, Masculine face, Male individual, He
Female
Female person, Woman, Feminine face, Female individual, She
Young person, Youth, Young adult, Youthful face, Teenager
Table 1: Text prototypes used for zero-shot CAV analysis. For occupations, 5 synonyms per class are defined (representative examples shown). For demographics, the full set of prototypes is listed.
GENDER BIAS
AGE BIAS
Class
V
→ Pred.
Male
Fem.
Class
V
→ Pred.
Yng.
Old.
Nurse
0.450
Nurse
44.9
83.1
Guard
0.238
Guard
89.3
44.8
Doctor
20.1
3.1
Laborer
0.6
10.3
Repairman
7.7
0.4
Seller
1.3
6.9
Doctor
0.399
Nurse
39.4
78.2
Ballplayer
0.186
Ballplayer
68.2
83.3
Doctor
37.2
13.4
Basket. pl.
10.3
0.0
Table 2: Top-5 gender- and age-biased occupation classes. The left half shows Gender Bias and the right half shows Age Bias ranked by Cramér’s V , including top-3 predicted classes with per-group rates (%). The correct class is highlighted.
Global Bias Reduction ( ΔV ): Suspected vs. Random Control
Condition
Accuracy
Cramér’s V
ΔV
Gender-Head Experiment ( V measured on gender)
Baseline
64.30%
0.381
—
Suspected (4)
64.72%
0.362
−0.019
Random avg (4)
64.16% ± 0.27
0.381 ± 0.002
−0.000
Age-Head Experiment ( V measured on age)
Table 3: Global ablation results for gender-head and age-head experiments, each vs. a layer-matched random control ( ± std over 10 runs). V refers to the target attribute of each experiment.
Figure 1: Per-class accuracy change (in percentage points) after combined gender-head ablation across all 19 classes with statistically significant gender bias (BH-corrected p<0.05 ). Green bars indicate improved accuracy; red bars indicate declines. Sample sizes ( n ) are shown below each class label.
Doctor Class — Per-Head Ablation and Prediction Redistribution (Gender)
Male ( N=180 )
Female ( N=179 )
Condition
V
ΔV
→ Doctor
→ Nurse
→ Doctor
→ Nurse
Baseline
0.399
—
37.2
39.4
13.4
78.2
L23H4
0.237
−0.162
38.3
37.8
26.3
60.9
L21H2
0.355
−0.044
36.7
39.4
14.0
74.3
L21H10
0.356
−0.043
37.8
38.9
16.2
73.7
Table 4: Doctor class: per-head and combined ablation results (gender). Left columns show bias metrics; right columns show prediction redistribution by gender. Heads sorted by ΔV .
Craftsman Class — Per-Head Ablation and Prediction Redistribution (Gender)
Male ( N=598 )
Female ( N=158 )
Condition
V
ΔV
→ Craftsman
→ Seller
→ Craftsman
→ Seller
Baseline
0.279
—
41.5
14.2
31.6
38.6
L23H4
0.226
−0.053
38.6
15.4
39.2
32.3
L21H2
0.273
−0.007
41.3
14.2
32.3
38.0
L21H10
0.273
−0.007
41.6
14.4
31.0
38.6
Table 5: Craftsman class: per-head and combined ablation results (gender). Left columns show bias metrics; right columns show prediction redistribution by gender. Heads sorted by ΔV .
Figure 2: Qualitative examples of correction after gender-bias head ablation. All images were misclassified at baseline (e.g., nurse for doctor, machinist or seller for craftsman) and correctly reclassified after ablating the top-10 identified gender heads. The images are chosen from cases where the demographic-parity gap was most pronounced.
Guard Class — Per-Head Ablation and Prediction Redistribution (Age)
Young ( N=159 )
Middle ( N=460 )
Older ( N=29 )
Condition
V
ΔV
→ Guard
→ Garden.
→ Guard
→ Report.
→ Guard
→ Seller
Baseline
0.238
—
89.3
1.3
69.1
3.5
44.8
6.9
L21H5
0.234
−0.004
88.7
1.3
68.7
3.9
44.8
10.3
L22H4
0.246
+0.008
90.6
1.3
70.0
3.0
44.8
10.3
L23H4
0.252
+0.015
89.3
1.3
67.6
3.9
41.4
6.9
Table 6: Guard class: per-head and combined ablation results (age). Left columns show bias metrics; right columns show top-2 predictions by age group (top confusion per group: Gardener for Young, Reporter for Middle, Seller for Older). Heads sorted by ΔV .