Can We Locate and Prevent Stereotypes in LLMs?

Alex D'Souza

cs.CLMar 26, 2026

Can We Locate and Prevent Stereotypes in LLMs?

Authors: Alex D'Souza

Organizations: UC Davis

Abstract

Stereotypes in large language models (LLMs) can perpetuate harmful societal biases. Despite the widespread use of models, little is known about where these biases reside in the neural network. This study investigates the internal mechanisms of GPT 2 Small and Llama 3.2 to locate stereotype related activations. We explore two approaches: identifying individual contrastive neuron activations that encode stereotypes, and detecting attention heads that contribute heavily to biased outputs. Our experiments aim to map these "bias fingerprints" and provide initial insights for mitigating stereotypes.

Figures & tables

Category	Sentence Completion
Context	The Hispanic is usually …
Stereotype	… dirty .
Anti-stereotype	… clean .
Unrelated	… paper .

Table 1: Example of a StereoSet intrasentence triplet for the Race domain.

Figure 1: Illustration of contrastive explanations in CXAD Davidson et al. (2025) .

Figure 2: GPT-2 Architecture. Source: Heilbron and de Lange (2019)

Figure 3: Where activations are extracted in GPT-2 architecture for experiment 1

Figure 4: How activations are extracted from an input sentence in transformer architecture

Activation Type	Bias Type	Mean	Median	Max Ratio
Attn. Heads	Race	4.16	1.08	$6.9\times 10^{4}$
	Profession	3.66	0.95	$4.7\times 10^{4}$
	Gender	4.09	0.97	$4.1\times 10^{4}$
	Religion	5.12	1.03	$3.6\times 10^{4}$
FFN Outputs	Race	6.54	2.29	$4.2\times 10^{4}$
	Profession	7.12	2.07	$8.3\times 10^{4}$

Table 2: Experiment 1: Contrastive Ratio Statistics by Bias Type and Activation Component. The mean ratio reflects the average deviation of stereotype activations from both anti-stereotype and unrelated baselines. A mean ratio of 1.0 would signal there are no individual contrastive neurons. There are significant outlier that have large ratios indicating potential stereotypical encoding neurons.

Bias Type	Attn. Heads	FFN Outputs	Embeddings
Race	96	76	90
Profession	76	90	81
Gender	20	28	25
Religion	8	6	4

Table 3: Experiment 1: Bias Type Distribution (Top 200 Neurons). Note Race and Profession are most represented, though Gender and Religion show disproportionately high mean ratios relative to their feature counts.

Layer	Attention Heads	FFN Outputs
0	30	8
1	18	25
2	12	13
3	10	14
4	11	33
5	23	21

Table 4: Experiment 1: Distribution of the top 200 highest contrastive-ratio features across GPT-2 layers (0–11), for attention heads and FFN outputs. Attention heads peak early at Layer 0, while FFN outputs peak at Layer 4, suggesting that stereotypic encoding emerges at different depths across component types.

Feature Type	Mean Effect	Median	Max	Neg. Effect
Embeddings	$0.01\%$	$0.00\%$	$0.71\%$	$51\%$
Attn. Heads	$0.01\%$	$0.00\%$	$0.25\%$	$42\%$
FFN Outputs	$0.03\%$	$0.02\%$	$0.38\%$	$41\%$

Table 5: Experiment 1: Causal Effect of Single Feature Ablation (Top 100 per Type). Individual ablation demonstrates a negligible impact on model output, with a maximum reduction of

<1\%

. The high frequency of negative effects suggests that biased representations are highly distributed and resistant to single-point interventions.

Panel of Figure 5 — Figure 5: Experiment 2: Probing classifier on GPT-2 Small attention head encodings. Accuracy is plotted during iterative ablation of attention heads (top) and high-impact neurons (bottom). Top-down removal follows Shapley value rankings. Note: For attention heads, random-guess accuracy (50%) is reached at $\sim$ 20% ablation, whereas individual neuron ablation reaches 50% accuracy significantly faster at $\sim$ 5% ablation.

Panel of Figure 6 — Figure 6: Experiment 2: Probing classifier on Llama attention head encodings. Accuracy is plotted during iterative ablation of attention heads (top) and high-impact neurons (bottom). Top-down removal follows Shapley value rankings. Note: Individual neuron ablation demonstrates a similar efficiency to the GPT-2 Small model, collapsing classifier accuracy to baseline levels ( $\sim$ 50%) with significantly fewer components ( $\sim$ 5%) compared to attention head ablation.

Model	Config.	SS	LMS	iCAT
Llama 3.2	Inter (B)	51.72	84.79	81.87
(1B)	Inter (A)	51.11	84.41	82.54
	Intra (B)	64.72	97.48	68.78
	Intra (A)	64.25	97.53	69.74
	Avg (B)	58.22	91.14	75.33
	Avg (A)	57.68	90.97	76.14

Table 6: Experiment 2: Stereoset Performance Metrics. (B) represents Baseline performance, and (A) represents the Model under Ablation of high-impact neurons identified via Shapley value analysis. In almost all iterations, the intervention successfully reduced the Stereotype Score (SS) toward the ideal 50.0 parity mark. Notably, the Language Modeling Score (LMS) remained stable or showed negligible degradation, leading to a slight net increase in the Idealized Context Awareness Test (iCAT) score for both models.

Bias Category	Encoding	Positional
Race	0.780	0.506
Profession	0.711	0.528
Gender	0.663	0.576
Religion	0.634	0.512
Overall Acc.	0.7340	0.5225

Table 7: Classification Accuracy from Embeddings by Bias Type (GPT-2 Small)

Social Bias in Language Models

Social bias in language models is the study and mitigation of unfair stereotypes and disparities in generated text, judgments, and decisions.

See all 199 papers

STEREODISCO: Discovering Stereotypicality in LLMs
Jul 30, 2026Farane Jalali Farahani, Corina Dima, Mojtaba Nayyeri +2Social Bias in Language Models Representation Geometry in Language Models
Institute for Artificial Intelligence, Stuttgart, Germany · Institute for Social Science, Stuttgart, Germany · University of Southampton, Southampton, United Kingdom
The Missing Minimal Pair: Stereotype Evaluation in LLMs
Oct 6, 2026Nataliya Stepanova, Ivan Titov, Emily Allaway +1Multilingual Language Model Evaluation Social Bias in Language Models
University of Edinburgh · University of Amsterdam
Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit
May 29, 2026Jiwoo Choi, Seonwoo Ahn, Tongxin Zhang +1Gender Bias in Language Models Multilingual Language Model Evaluation
School of Digital Humanities and Computational Social Sciences, KAIST, South Korea

Jul 30, 2026J/K move · Enter open · S save

STEREODISCO: Discovering Stereotypicality in LLMs

Farane Jalali Farahani, Corina Dima, Mojtaba Nayyeri, Raphael H. Heiberger+1

Institute for Artificial Intelligence, Stuttgart, Germany · Institute for Social Science, Stuttgart, Germany · University of Southampton, Southampton, United Kingdom

Social Bias in Language Models Representation Geometry in Language Models

Open paper AskPDF arXiv

LLMs encode, convey, and perpetuate stereotypes. Prior computational research focuses on a small set of semantic axes investigated in social psychology, and operates on word embeddings produced by language models, leaving open which other semantic axes carry stereotypical associations in LLMs and how LLMs internally represent such axes. We introduce STEREODISCO, a framework that adapts the semantic differential method (Osgood et al., 1957) to the systematic study of stereotypes in LLM internal representations. STEREODISCO constructs approx. 2,000 candidate semantic axes from WordNet antonym synsets, recovers each as a geometric axis in the LLM's activation space via probing, and identifies stereotypical axes via a statistical test over concept projections. As a case study, we apply STEREODISCO to social group stereotypes with LLAMA-3-8B-INSTRUCT and MISTRAL-7B-INSTRUCT. We find that the two LLMs agree with each other on social group ratings more than with humans, suggesting that LLM-encoded stereotype content diverges from that documented in social psychology. We also discover stereotypical axes not investigated in prior work -- including humble vs. proud, narrow-minded vs. broad-minded, and cowardly vs. brave, which human annotators independently confirm.

Abstract

Social Bias in Language Models

Explore similar work

STEREODISCO: Discovering Stereotypicality in LLMs

The Missing Minimal Pair: Stereotype Evaluation in LLMs

Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit