What Does a Bias Head Measure? Non-Identifiability in Head-Level Bias Attribution
Organizations: Bangladesh University of Engineering and Technology, Dhaka
Abstract
Methods that localise social bias in large language models to individual attention heads and remove it by ablation are validated by their effect on a fairness metric, never by evidence that the statistic selecting the heads measures bias. We audit this on Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct. The head map correlates at and with a label-blind response-length map, apparently identifying a surface confound. An intervention that breaks the length-label coupling rejects this reading: with the prompt held fixed, response length explains 1% of the statistic's variance. What does explain it is prompt identity, at 64% of variance, in an ordering that is not monotonic in length. Within any fixed prompt the fairness label, response length and the presence of a protected-group token are collinear at , partly by definition of the labelling rule, so no observational analysis can attribute the map to any of them, and no method in this family performs an experiment that could. The substantive claims fail independently: bias selectivity is not late-layer-concentrated on either model under either reference reading, and no head set we tested reduces the unfairness rate when ablated. Its complement, scored on the opposite label, reliably destroys refusal behaviour (), demonstrating that the pipeline detects effects at this scale.
Figures & tables
| primary labeller | predecessor | |
|---|---|---|
| agreement | 0.867 [0.794, 0.916] | 0.842 [0.766, 0.896] |
| false-unfair rate | 0.120 [0.042, 0.300] | 0.160 [0.064, 0.347] |
| false-fair rate | 0.044 [0.017, 0.108] | 0.044 [0.017, 0.108] |
| unclassifiable | 11/120 | 13/120 |
| arm | (score, length) | (score, unfair) | |
|---|---|---|---|
| DA | 60 | -0.679 | +0.780 |
| CoT | 40 | +0.224 | +0.615 |
| verbose | 60 | -0.277 | +0.387 |
| terse | 60 | -0.025 | +0.725 |
| model | selection | rate | u f / f u | Holm | degen / unclass. | tokens | |
|---|---|---|---|---|---|---|---|
| Qwen | final-layer (stored) | 0.775 | +0.013 | 1 / 2 | 1.000 | 0 / 0 | 21.4 47.4 |
| Qwen | layer-matched | 0.873 | +0.111 | 0 / 8 | 0.0156 | 0 / 1 | — |
| Llama | layer-matched (stored) | 0.795 | +0.086 | 3 / 11 | 0.344 | 3 / 7 | 27.2 60.0 |
| Llama | final-layer | 0.582 | -0.127 | — | — | 1 / 25 | — |
| model | sel. | lab. | u f | f u | |
|---|---|---|---|---|---|
| Qwen | stored | prim. | 0 | 8 | |
| Qwen | stored | prev. | 0 | 9 | |
| Llama | stored | prim. | 0 | 13 | |
| Llama | stored | prev. | 0 | 13 | |
| Qwen | resel. | prim. | 1 | 8 | |
| Qwen | resel. | prev. | 0 | 8 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| arm | unfair | excl. | rate | flips u f | degenerate | mean tok |
|---|---|---|---|---|---|---|
| baseline | 98 | 0 | 0.980 | — | 0 | 19.2 |
| prescribed (differ-unfair) | 98 | 0 | 0.980 | 0 | 0 | 44.8 |
| reverse set | 99 | 0 | 0.990 | 0 | 0 | 18.4 |
| random-0 | 80 | 18 | 0.976 | 19 | 34 | 30.2 |
| random-1 | 97 | 1 | 0.980 | 1 | 0 | 30.8 |
| random-2 | 94 | 5 | 0.989 | 5 | 17 | 53.6 |
| draw | refusals | u f | f u | excl. | McNemar (Holm) |
|---|---|---|---|---|---|
| 0 | 9 | 0 | 7 | 3 | 0.0781 |
| 1 | 15 | 1 | 2 | 5 | 1.0000 |
| 2 | 0 | 0 | 19 | 1 | 0.0000 |
| 3 | 27 | 9 | 0 | 1 | 0.0312 |
| 4 | 5 | 0 | 12 | 3 | 0.0044 |
| 5 | 12 | 10 | 1 | 54 (broken) | — |
| sheet | blind | source file | what it fed | |
|---|---|---|---|---|
| Qwen development | 100 | no | an earlier 4-arm evaluation file (144 responses) | initial agreement figure |
| Llama development | 50 | no | the Llama identification set | reversal-aware variant development |
| Llama held-out | 30 | yes | the Llama identification set, Direct-Answer only | variant rejection |
| finding | movement |
|---|---|
| F1 map correlations | on every row; split-half reliability identical |
| Layer distribution | centre-of-mass unchanged to 0.02 layers |
| Prescribed-set effect | shrinks: Qwen +0.025 +0.013; Llama +0.106 +0.086; both remain non-significant |
| Complement-set effect | Qwen 0/9 0/8; Llama 0/13 unchanged; zero counter-directional throughout |
| Null bands | Qwen 0.654–1.000 0.641–1.000; Llama 0.631–0.884 0.603–0.886 |
| Broken-arm set | identical under all three |
| item | source | ours |
|---|---|---|
| averaging set | all response tokens | first 64 |
| undefined; “first few tokens” contradicts the stated mean over | first 8 response positions, unit-normalised, two layer readings | |
| “the value vector of head ” | the per-head slice of the attention output | |
| head set | Eq. 4 states top- of one map; the text and figure describe a set difference | set difference |
| unspecified |
| model | persona | style | cells | reliability | (length, pooled) | top-10 length-bottom-10 |
|---|---|---|---|---|---|---|
| Qwen | full | pooled | 57u/43f | 0.950 | -0.950 | 9/10 |
| Qwen | mild | pooled | 26u/78f | 0.978 | -0.828 | 7/10 |
| Qwen | full | DA only | 48u/12f | 0.911 | -0.725 | 6/10 |
| Qwen | mild | DA only | 26u/34f | 0.965 | -0.970 | 10/10 |
| Llama | full | pooled | 48u/42f | 0.959 | -0.768 | 8/10 |
| persona | tokens, unfair | tokens, fair | gap | AUROC length label | unfair | LOO swing |
|---|---|---|---|---|---|---|
| full | 34.5 | 104.8 | +70.4 | 0.878 | 57 | 0.010 |
| mild | 13.7 | 93.3 | +79.7 | 0.956 | 26 | 0.015 |
| none | 13.0 | 85.8 | +72.8 | 0.996 | 5 | 0.004 |
| Llama, full | — | — | — | 0.836 | 48 | 0.017 |
| model | CoM layer | uniform null | (late) | (early) | final-4 share (null) | |
|---|---|---|---|---|---|---|
| Qwen | layer-matched | 11.72 | 13.5 | 0.989 | 0.011 | 0.067 (0.143) |
| Qwen | final-layer | 13.63 | 13.5 | 0.428 | 0.572 | 0.150 (0.143) |
| Llama | layer-matched | 13.67 | 15.5 | 0.979 | 0.021 | 0.073 (0.125) |
| Llama | final-layer | 13.09 | 15.5 | 0.998 | 0.002 | 0.099 (0.125) |
| elicitation | Qwen | Llama |
|---|---|---|
| Direct-Answer | 48/60 = 0.800 | 42/54 = 0.778 |
| Chain-of-Thought | 9/40 = 0.225 | 6/36 = 0.167 |
| gap | 0.575 | 0.611 |
Attention Head Analysis
Attention head analysis studies how individual Transformer heads route information, support computations, and affect model behavior.
See all 89 papers