Gender bias across LLMs is common and highly heterogeneous

Edoardo Bolzoni; Valerio Capraro

cs.CLSep 29, 2026

Gender bias across LLMs is common and highly heterogeneous

Authors: Edoardo Bolzoni, Valerio Capraro

Organizations: University of Milan-Bicocca, Milan, Italy

Abstract

Understanding gender biases in large language models (LLMs) is increasingly important as these systems become embedded in decision-support tools with real consequences. Prior research has focused only on a small set of models, leaving open the extent to which gender biases are common and heterogeneous across LLMs. We address this gap across ten models released between April 2025 and June 2026, spanning nine vendors, using two paradigms: gender attribution to stereotyped phrases (Study 1) and moral judgment of abuse or torture against a woman or a man to prevent a catastrophic outcome (Study 2). In Study 1, two of ten models attributed masculine-stereotyped phrases to female writers more often than the reverse, while three models showed the opposite pattern. In Study 2, several models converged on a male-disadvantaging asymmetry that was directionally consistent with a documented human tendency to protect female targets from harm, though the specific conditions under which this asymmetry emerged varied by model; three other models, by contrast, showed no variation across conditions. These results indicate that gender-related biases are common in LLMs. Their direction and magnitude, however, are highly heterogeneous, to the point that some models behave in diametrically opposite ways to others. Bias auditing should therefore be treated as an ongoing, multi-vendor process, rather than a one-time assessment.

Figures & tables

Label	Model	Release date	Access method
1a/2a	Meta Llama 4 Scout	Apr 2025	Web/app
1b/2b	xAI Grok 4.1 Fast	Nov 2025	Web/app
1c/2c	Anthropic Claude Sonnet 4.6	Feb 2026	Web/app
1d/2d	Google Gemini 3.1 Pro	Feb 2026	Web/app
1e/2e	Mistral Small 4	Mar 2026	API
1f/2f	OpenAI GPT-5.5	Apr 2026	Web/app

Table 1: Models tested, in release-date order.

Label	Model	$I_{F}$ (M $\pm$ SE)	$I_{M}$ (M $\pm$ SE)	$t(38)$ , $p$
1a	Llama 4 Scout	0.158 $\pm$ 0.059	0.198 $\pm$ 0.048	-0.53, .600
1b	Grok 4.1 Fast	0.075 $\pm$ 0.049	0.085 $\pm$ 0.037	-0.16, .871
1c	Claude Sonnet 4.6	0.065 $\pm$ 0.039	0.275 $\pm$ 0.087	-2.20, .034
1d	Gemini 3.1 Pro	0.311 $\pm$ 0.092	0.013 $\pm$ 0.009	3.23, .003
1e	Mistral Small 4	0.158 $\pm$ 0.044	0.391 $\pm$ 0.051	-3.49, .001
1f	GPT-5.5	0.098 $\pm$ 0.053	0.093 $\pm$ 0.034	0.08, .937

Table 2: Inclusivity indices and significance test by model (Study 1).

Figure 1: Inclusivity index by model, Study 1.

Label	Model	Abuse (woman) M $\pm$ SE	Abuse (man) M $\pm$ SE	Torture (woman) M $\pm$ SE	Torture (man) M $\pm$ SE
2a	Llama 4 Scout	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00
2b	Grok 4.1 Fast	5.32 $\pm$ 0.37	7.00 $\pm$ 0.00	6.94 $\pm$ 0.03	6.98 $\pm$ 0.02
2c	Claude Sonnet 4.6	1.48 $\pm$ 0.16	7.00 $\pm$ 0.00	7.00 $\pm$ 0.00	7.00 $\pm$ 0.00
2d	Gemini 3.1 Pro	4.87 $\pm$ 0.43	7.00 $\pm$ 0.00	6.74 $\pm$ 0.12	7.00 $\pm$ 0.00
2e	Mistral Small 4	4.12 $\pm$ 0.34	4.78 $\pm$ 0.26	4.84 $\pm$ 0.28	6.20 $\pm$ 0.12
2f	GPT-5.5	1.04 $\pm$ 0.03	6.10 $\pm$ 0.20	1.22 $\pm$ 0.06	5.44 $\pm$ 0.21

Table 3: Mean agreement by model and condition (Study 2).

Figure 2: Mean agreement with using violence to prevent a nuclear apocalypse, by model and condition, Study 2. The scale is: [1] = strongly disagree, [2] = moderately disagree, [3] = somewhat disagree, [4] = neither agree nor disagree, [5] = somewhat agree, [6] = moderately agree, [7] = strongly agree.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Label	Model	$I_{F}$ (M $\pm$ SE)	$I_{M}$ (M $\pm$ SE)	$z$ , $p$
1a	Llama 4 Scout	0.158 $\pm$ 0.059	0.198 $\pm$ 0.048	-1.26, .208
1b	Grok 4.1 Fast	0.075 $\pm$ 0.049	0.085 $\pm$ 0.037	-0.69, .488
1c	Claude Sonnet 4.6	0.065 $\pm$ 0.039	0.275 $\pm$ 0.087	-2.36, .018
1d	Gemini 3.1 Pro	0.311 $\pm$ 0.092	0.013 $\pm$ 0.009	2.94, .003
1e	Mistral Small 4	0.158 $\pm$ 0.044	0.391 $\pm$ 0.051	-3.33, .001
1f	GPT-5.5	0.098 $\pm$ 0.053	0.093 $\pm$ 0.034	-0.40, .693

Appendix

Table A.1: Inclusivity indices and rank-sum test by model (Study 1).

Label	Model	Comparison	M1	M2	$t$ -test
2a	Llama 4 Scout	torture: woman vs man	1.00	1.00	no test defined
2a	Llama 4 Scout	abuse: woman vs man	1.00	1.00	no test defined
2a	Llama 4 Scout	woman: torture vs abuse	1.00	1.00	no test defined
2a	Llama 4 Scout	man: torture vs abuse	1.00	1.00	no test defined
2b	Grok 4.1 Fast	torture: woman vs man	6.94	6.98	$t(98)=-1.02$ , $p=.312$
2b	Grok 4.1 Fast	abuse: woman vs man	5.32	7.00	$t(98)=-4.54$ , $p<.001$

Appendix

Table B.1: Pairwise comparisons by model,

t

-test (Study 2).

Label	Model	Comparison	M1	M2	Rank-sum
2a	Llama 4 Scout	torture: woman vs man	1.00	1.00	no test defined
2a	Llama 4 Scout	abuse: woman vs man	1.00	1.00	no test defined
2a	Llama 4 Scout	woman: torture vs abuse	1.00	1.00	no test defined
2a	Llama 4 Scout	man: torture vs abuse	1.00	1.00	no test defined
2b	Grok 4.1 Fast	torture: woman vs man	6.94	6.98	$z=-1.02$ , $p=.310$
2b	Grok 4.1 Fast	abuse: woman vs man	5.32	7.00	$z=-4.64$ , $p<.001$

Appendix

Table B.2: Pairwise comparisons by model, rank-sum test (Study 2).

Pattern	Models
Fixed (zero variance in all four conditions)	Llama 4 Scout, Microsoft Copilot, DeepSeek V4-Flash
Fixed, with one unstable condition	Claude Sonnet 4.6, Grok 4.1 Fast, Gemini 3.1 Pro
Consistently near the midpoint (low variance throughout)	Claude Fable 5
Unstable in one or two conditions	Qwen 3.6
Moderate variance across all conditions	GPT-5.5
Unstable across all four conditions	Mistral Small 4

Appendix

Table C.1: Response stability by model, Study 2.

Figure C.1: Response distributions, Study 2a – Llama 4 Scout.

Figure C.2: Response distributions, Study 2b – Grok 4.1 Fast.

Figure C.3: Response distributions, Study 2c – Claude Sonnet 4.6.

Figure C.4: Response distributions, Study 2d – Gemini 3.1 Pro.

Figure C.5: Response distributions, Study 2e – Mistral Small 4.

Figure C.6: Response distributions, Study 2f – GPT-5.5.

Figure C.7: Response distributions, Study 2g – Microsoft Copilot.

Figure C.8: Response distributions, Study 2h – DeepSeek V4-Flash.

Figure C.9: Response distributions, Study 2i – Qwen3.6.

Figure C.10: Response distributions, Study 2j – Claude Fable 5.

Explore similar work

Feb 14, 2025cs.CL

Man Made Language Models? Evaluating LLMs' Perpetuation of Masculine Generics Bias

Instruct-based large language models (LLMs) have been shown to propagate and even amplify gender bias when prompted with contextually constrained instructions (e.g., writing a text from a description or selecting a gendered pronoun). However, little attention has been paid to biases in responses to contextually unconstrained (generic) instructions conveyed by gendered language, particularly masculine generics (MG). MG, found in many gender-marked languages, denote the use of the masculine gender as a supposedly neutral reference to mixed-gender groups or individuals whose gender is unknown or non-binary. Yet, psycholinguistic studies demonstrate that MG are not neutral and systematically induce gender bias. This study investigates how both local and proprietary LLMs are MG-biased when responding to generic prompts in French, examining LLMs' MG bias rates and use of gender-fair language (GFL). We create a 16k+ human noun database from existing lexical resources and evaluate six LLMs on four instruction-response datasets under two conditions: prompts with and without MG. Overall, we find that

\approx

27.57% of LLMs' responses to MG-filtered generic instructions are MG-biased (

\approx

78.55% with MG-containing prompts). Moreover, we find that LLMs rarely use GFL spontaneously. These findings highlight the persistence of MG bias in LLM outputs and models' limited tendency towards GFL strategies.

Enzo Doyen, Amalia Todirascu

University of Strasbourg

Gender Bias Gender Linguistics Instruction-Tuned Models

Ask PDF

Jun 12, 2026cs.CL

Harsher on Male? Evaluating LLMs on Gender-Asymmetric Moral Framing Across Diverse Conflict Scenarios

Existing studies on gender bias in LLMs have largely focused on stereotypes, occupational associations, or explicit harmful outputs. In this work, we ask whether LLMs apply consistent response standards to the same negative behavior under matched male-actor and female-actor conditions. We introduce GAMA-Bench, a gender-mirrored benchmark of 1,298 scenarios covering intimate relationship and public social conflicts. It constructs gender-neutral misconduct templates through controlled grids and cross-model review, then compiles them into paired first-person prompts with matched actor-gender and role-reference variations. We further design a structured response-framing protocol to measure how models allocate punishment, empathy, escalation, instruction, and blame. Experiments on 10 representative LLMs reveal a consistent male-disadvantaging asymmetry: male actors receive more punitive, escalatory, and blame-centered framing, whereas female actors receive more therapeutic and empathy-oriented framing for the same misconduct. Further analyses show that this pattern persists across model families, scenario tracks, model scale, and explicit thinking-style reasoning. The official code is available at https://github.com/xufeiqiong/GAMA-Bench.

Guangzong Si, Dong Wang, Zhenhao Li +3

University of Science and Technology of China · Eastern Institute of Technology, Ningbo

Moral Reasoning Large Language Model Decisions Large Language Model Alignment Agent Behavior Modeling Hard

Ask PDF

May 29, 2026cs.CL

Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit

We audit six large language models (LLMs) for gender stereotyping across English, Korean, Chinese, and Japanese. Three were developed primarily for English-language use (Claude, GPT, Gemini) and three for East Asian use (DeepSeek, Syn-Pro, HyperCLOVA X). We adopt the HEXACO-100 personality inventory and anchor each model against a cross-cultural human dataset spanning 48 countries to ask not whether LLMs are biased, but how far their gender attributions drift from the populations they are deployed among. Our findings show that their stereotyping spans a range roughly 2.5 times wider than the entire cross-country range found in humans, and the effect can compound across languages. One English-centric model, prompted in Korean, reached 5 times the local baseline, even when the prompt stated the candidate had already been hired, which often dampens human stereotyping. To characterize such behaviors without ranking them, we introduce a four-pattern framework -- concordance, suppression, reorganization, and amplification -- across 24 (model x language) cells. Item-level analysis reveals that translation does not just rescale stereotypes, but changes the attributes tied to it, hiding significant rearrangement under the surface while appearing well-calibrated. Our results ultimately suggest that no single debiasing pipeline is likely to address bias evenly across linguistic boundaries.

Jiwoo Choi, Seonwoo Ahn, Tongxin Zhang +1

School of Digital Humanities and Computational Social Sciences, KAIST, South Korea

Large Language Model Bias Stereotype Mitigation Gender Bias Gender Cross-Lingual Consistency

Ask PDF