LLMs Mirror Country-Specific Gender Patterns If Asked, but Skew Male When Generating Media in Local Languages

Sharif Kazemi; Tanya Popli; Neil K. R. Sehgal; Sunny Rai; Niyati Malhotra; Victor Orozco-Olvera; Ana María Muñoz Boudet; Samuel P. Fraiberger

cs.CLSep 6, 2026

LLMs Mirror Country-Specific Gender Patterns If Asked, but Skew Male When Generating Media in Local Languages

Authors: Sharif Kazemi, Tanya Popli, Neil K. R. Sehgal, Sunny Rai, Niyati Malhotra, Victor Orozco-Olvera, Ana María Muñoz Boudet, Samuel P. Fraiberger,

Organizations: World Bank Group · Cornell University · University of Pennsylvania · University of Oxford ·

Abstract

Large language models (LLMs) are increasingly used to generate media, but whether their content perpetuates gender stereotypes is unknown: standard benchmarks rely on selection-based formats rather than long-form generation, and surveyed baselines for local gender associations are scarce outside the West. We collect gender associations for 22 occupational and domestic roles from 695 respondents across the United States, India, Kenya, and Nigeria, and evaluate eight LLMs under two regimes: direct questioning and media generation. Models track the surveyed associations under direct questioning but skew substantially more male under media generation in major local-language cells, consistent with the male bias documented in human-produced media. Outside the US, the shift is much smaller and non-significant under English prompting, so English-only or country-agnostic evaluation would miss this bias in the languages where these models are most deployed. Instruction prompting reduces the shift directionally, but trades off against alignment with the surveyed associations. Evaluating LLM gender bias for global deployment therefore requires generation-format testing, local-language prompting, and locally-collected human baselines.

Figures & tables

Figure 1: The same model produces gender associations roughly in line with human survey responses when asked explicitly (top), but shifts male when asked to generate media featuring a protagonist in the same role (bottom). Across all four countries, explicit elicitation tracks local human associations from our survey while implicit elicitation shows extreme gender associations - often shifting male in local languages.

Figure 2: Country-level human-survey gender associations across $22$ roles ( $n=695$ ). Per-role directional score on $[-2,+2]$ (negative = male-associated, positive = female-associated), one panel per country.

Figure 3: Per-role male share in the human survey ( $\bullet$ ), LLM explicit elicitation ( $\blacksquare$ ), and LLM media-generation output ( $\blacklozenge$ ), pooled across five main generic models (GPT-5.1, LLaMA 4 Scout, LLaMA 3.1 8B, Llama-3-Instruct, Mistral Small). Vertical dashed lines mark the LLM explicit cross-task average ( $44\%$ ), LLM implicit cross-task average ( $66\%$ ), and the $\approx 3{:}1$ male:female ratio reported in independent film/TV content analyses ( $75\%$ ).

Figure 4: Implicit-vs-explicit shift coefficient ( $\beta$ on is_implicit ) per (country $\times$ language) cell. Each cell is fit as a separate OLS with model and task fixed effects and task-clustered standard errors; whiskers are 95% confidence intervals. * $p<.05$ , ** $p<.01$ , *** $p<.001$ .

Figure 5: Instruction prompting on the eight evaluated models. Each bar is the shift in mean directional gender score (averaged across the four nationality framings) from each model’s own no-instruction baseline. Dashed segments show where the country human survey reference sits relative to each model’s baseline.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Short name (used in figures)	Domain / Sector	Survey question wording
Domestic activities (10 items)
Educating children	Domestic	Educating and disciplining children.
Playing with children	Domestic	Playing with the children.
Caring for elderly	Domestic	Taking care of the elderly, like aging parents or in-laws.
Large household purchases	Domestic	Making decisions in the household related to large purchases, like buying a car.
Daily purchases	Domestic	Making decisions in the household related to small or daily purchases, like buying clothes or toys for the children.

Appendix

Table 1: The 22 role-association items in the human survey instrument, with domain and (for occupational items) sector classification. Domains follow standard time-use survey terminology (domestic activities vs. paid occupations). The sector classification for occupational items distinguishes formal (consistently captured by establishment surveys and labour-force statistics in our two reference countries; ILO ICSE-93 employee status ( ILO, 2025 ) ), dual-sector (substantial presence in both formal employment and informal household provision; for example, professional restaurant cooking vs. home cooking, parlour styling vs. roadside barbering), and informal (predominantly captured by labour-force surveys with substantial under-count of female participation per Bonnet et al., 2019 ). The sector split is used in the validation analysis in § 3.1 and Figure 6 , and reappears in the household-task benchmark validation in Appendix A.4 .

Figure 6: Survey-reported directional gender associations versus measured occupational gender composition for the $12$ occupational tasks, decomposed by occupational sector. US correlation $r=+0.95$ (formal $r=+0.98$ , informal $r=+0.94$ ); India formal sector tracks PLFS 2025 at $r=+0.89$ while the informal sector diverges ( $r=-0.01$ ).

Model	Category	Params	Localisation type	Languages evaluated
GPT-5.1	Frontier	undisclosed	—	EN / HI / SW / HA / YO
LLaMA 4 Scout	Generic large	17B (16 $\times$ expert)	—	EN / HI / SW / HA / YO
LLaMA 3.1 8B	Generic small	8B	—	EN / HI / SW / HA / YO
LLaMA 3.0 8B	Generic small	8B	—	EN / HI / SW / HA / YO
Mistral Small	Generic medium	24B	—	EN / HI / SW / HA / YO
Sarvam-M	Localised (India)	24B	Curated SFT (Mistral)	HI

Appendix

Table 2: Models evaluated. Localisation type indicates whether a localised fine-tune was built from curated supervised fine-tuning data (curated SFT), authentic in-language conversational data (community), or machine-translated supervision (MT-SFT).

Classifier	Coverage	Exact match	Sign+zero	Within $\pm 1$
Llama 3.1 8B Instant	$11{,}289$ / $11{,}660$ ( $96.8\%$ )	$74.2\%$	$96.8\%$	$99.0\%$
Qwen 3 32B	$11{,}076$ / $11{,}660$ ( $95.0\%$ )	$67.5\%$	$97.2\%$	$99.3\%$
Regex (deterministic)	$\phantom{0}8{,}469$ / $11{,}660$ ( $72.6\%$ )	$57.9\%$	$98.7\%$	$99.7\%$

Appendix

Table 3: Agreement between each alternative classifier and the GPT-4o-mini production judge across the

11{,}660

free-text responses (subset where both classifiers committed to a score). Exact match counts only rows where the integer scores are identical; sign-plus-zero match counts rows where both classifiers agree on direction (both male, both female, or both equal); within

\pm 1

counts rows where the integer scores differ by at most one Likert step.

	Llama 3.1 8B vs GPT-4o-mini			Qwen 3 32B vs GPT-4o-mini
Model	$\boldsymbol{n}$	Exact	Sign+0	$\boldsymbol{n}$	Exact	Sign+0
GPT-5.1	$1{,}750$	$74.6\%$	$95.7\%$	$1{,}749$	$77.2\%$	$98.3\%$
LLaMA 4 Scout	$1{,}720$	$78.4\%$	$97.6\%$	$1{,}703$	$72.2\%$	$98.2\%$
LLaMA 3.1 8B	$1{,}680$	$71.1\%$	$99.2\%$	$1{,}651$	$61.1\%$	$98.7\%$
Llama-3-Instruct	$1{,}723$	$64.4\%$	$99.9\%$	$1{,}685$	$48.3\%$	$98.9\%$
Mistral Small	$1{,}684$	$75.2\%$	$95.8\%$	$1{,}600$	$65.9\%$	$96.8\%$

Appendix

Table 4: Cross-classifier agreement broken down by evaluated model. “Llama” columns: agreement between Llama 3.1 8B Instant and GPT-4o-mini. “Qwen” columns: agreement between Qwen 3 32B and GPT-4o-mini. Both localised models (N-ATLaS, UlizaLlama3) achieve sign-plus-zero agreement

\geq 87\%

under both judges; Sarvam-M achieves the highest exact agreement of any model.

		Llama 3.1 8B		Qwen 3 32B
Sheet	$\boldsymbol{n}$	Exact	Sign+0	Exact	Sign+0
USA English	$1{,}320$	$76.7\%$	$96.1\%$	$75.4\%$	$98.1\%$
India English	$1{,}315$	$79.6\%$	$99.5\%$	$68.7\%$	$99.2\%$
India Hindi	$1{,}315$	$77.6\%$	$97.1\%$	$69.7\%$	$98.1\%$
Kenya English	$1{,}521$	$79.4\%$	$98.1\%$	$73.3\%$	$99.6\%$
Kenya Swahili	$1{,}516$	$70.7\%$	$95.0\%$	$71.2\%$	$96.5\%$

Appendix

Table 5: Per-sheet sign+zero agreement between each open-weight LLM judge and GPT-4o-mini. Sign agreement remains above

93\%

on every sheet for both classifiers; exact agreement is lower on local-language sheets, driven by the calibration boundary disagreements rather than directional misreads.

Free-text scoring rule	$\boldsymbol{n}$	$\boldsymbol{\beta}$	95% CI	$\boldsymbol{p}$
GPT-4o-mini (main analysis)	$37{,}707$	$-0.719$ ***	$[-1.10,-0.34]$	$<.001$
Llama 3.1 8B Instant	$37{,}934$	$-0.583$ ***	$[-0.90,-0.27]$	$<.001$
Regex (deterministic anchors)	$34{,}863$	$-0.513$ ***	$[-0.79,-0.24]$	$<.001$
Qwen 3 32B	$37{,}537$	$-0.418$ *	$[-0.77,-0.07]$	$.018$

Appendix

Table 6: Headline implicit-vs-explicit shift coefficient under four scoring rules for the free-text explicit responses. The point estimate compresses modestly toward zero under the more conservative classifiers, but the sign is preserved under every substitution and statistical significance is preserved under all but the strictest threshold (Qwen 3 retains

p<.05

but falls below

p<.01

). ***

p<.001

, *

p<.05

Spec	$\boldsymbol{n}$	$\boldsymbol{\beta}$	95% CI	$\boldsymbol{p}$
(a) baseline (all responses)	37,707	$-0.719$ ***	$[-1.10,-0.34]$	$<.001$
(b) drop collectivist	37,063	$-0.727$ ***	$[-1.11,-0.35]$	$<.001$
(c) drop all non-individualistic	36,263	$-0.736$ ***	$[-1.11,-0.36]$	$<.001$

Appendix

Table 7: Sensitivity of the implicit-vs-explicit shift coefficient to how non-individualistic implicit responses are handled. The same fixed-effects OLS as the headline analysis is re-fit on three nested subsets of the data: (a) the full dataset with non-individualistic implicit responses coded as

0

, (b) excluding collectivist implicit responses, and (c) excluding collectivist, format-mismatch, safety-refusal, and extraction-error implicit responses. The coefficient on is_implicit is essentially unchanged in magnitude and remains highly significant under all three handling rules.

Figure 8: Per-model alignment with the country human survey, by elicitation regime. For each of the eight evaluated models, the figure shows Pearson $r$ under explicit (darker) and implicit (lighter) elicitation. Companion view to the regression coefficients in Figure 9 and Table 8 .

Figure 9: Per-model marginal effect of switching from explicit to implicit elicitation. Each point is the OLS coefficient on is_implicit ; bars are $95\%$ CIs. Overall $\beta=-0.719$ ( $n=37{,}707$ response-level observations). Six of the eight models drift male; only UlizaLlama3 drifts female, though the effect is narrowly non-significant.

Model	$\boldsymbol{\beta}$	SE	95% CI	$\boldsymbol{p}$	$\boldsymbol{q}$	$\boldsymbol{n}$	Sig.
Overall (all models)	$-0.72$	$0.19$	$[-1.10,-0.34]$	$<.001$	—	$37{,}707$
Llama-3-Instruct	$-1.09$	$0.24$	$[-1.56,-0.63]$	$<.001$	$<.001$	$6{,}139$
LLaMA 3.1 8B	$-1.03$	$0.26$	$[-1.54,-0.53]$	$<.001$	$<.001$	$6{,}080$
Mistral Small	$-0.85$	$0.18$	$[-1.20,-0.50]$	$<.001$	$<.001$	$6{,}087$
LLaMA 4 Scout	$-0.58$	$0.22$	$[-1.01,-0.15]$	$.008$	$.012$	$6{,}120$	**
GPT-5.1	$-0.52$	$0.21$	$[-0.94,-0.11]$	$.014$	$.019$	$6{,}147$

Appendix

Table 8: Per-model marginal effect of switching from explicit to implicit elicitation, from the OLS interaction model with task and nationality fixed effects and task-clustered standard errors.

\beta

is on the

[-2,+2]

Likert scale (negative coefficient

=

shift toward male protagonist).

q

is the Benjamini–Hochberg false-discovery-rate value across the eight per-model tests; six of eight models remain significant after adjustment. Localised models appear below the rule. Significance (unadjusted): ***

p<.001

, **

p<.01

, *

p<.05

\dagger

p<.10

Country	Language	$\boldsymbol{n}$	%M exp	%M imp	$\boldsymbol{\Delta}$ %M	$\boldsymbol{\beta}$	95% CI	$\boldsymbol{q}$	M/F/=
Indian	Hindi	7,913	50.5	84.4	$+33.9$	$-1.36$ ***	$[-1.80,-0.91]$	$<.001$	6/0/0
Nigerian	Yoruba	7,983	39.1	71.9	$+32.9$	$-1.27$ ***	$[-1.63,-0.92]$	$<.001$	6/0/0
Kenyan	Swahili	8,117	46.6	62.4	$+15.8$	$-0.64$ **	$[-1.03,-0.24]$	$.004$	5/1/0
American	English	6,820	41.1	51.3	$+10.2$	$-0.44$ *	$[-0.83,-0.05]$	$.056$	4/0/1
Nigerian	Hausa	7,996	40.8	50.3	$+9.6$	$-0.39$	$[-0.81,+0.03]$	$.114$	5/1/0
Kenyan	English	7,023	44.9	51.3	$+6.4$	$-0.37$	$[-0.87,+0.14]$	$.203$	3/1/1

Appendix

Table 9: Per (country

\times

language) cell implicit-vs-explicit shift. %M is computed from per-response score (

-2

male,

+2

female) as

(2-\overline{score})/4\times 100

q

is the Benjamini–Hochberg false-discovery-rate value across the eight cells. The M/F/= column counts models in the cell whose implicit %M exceeds (M), falls below (F), or stays within

\pm 1

pp (=) of their explicit %M. Significance on

\beta

(unadjusted): ***

p<.001

, **

p<.01

, *

p<.05

Country	Local lang.	$\boldsymbol{\beta_{\text{Eng}}}$	$\boldsymbol{\beta_{\text{local}}}$	$\boldsymbol{\Delta\beta}$
Indian	Hindi	$-0.11$	$-1.36$	$-1.25$ ***
Nigerian	Yoruba	$-0.31$	$-1.31$	$-1.00$ ***
Kenyan	Swahili	$-0.24$	$-0.64$	$-0.40$ **
Nigerian	Hausa	$-0.31$	$-0.37$	$-0.06$

Appendix

Table 10: Within-country English-vs-local-language interaction.

\beta_{\text{Eng}}

is the implicit-vs-explicit OLS coefficient under English-with-country-framing;

\beta_{\text{local}}

is the same coefficient under local-language prompts;

\Delta\beta

is the interaction term (the formal test of amplification). Significance on

\Delta\beta

: ***

p<.001

, **

p<.01

Source	$\boldsymbol{n}$ groups	SD	Range
is_implicit (main effect)	2	—	0.64
between-model	8	0.11	0.31
between-country framing	4	0.27	0.65
between-language	5	0.50	1.21
between-(country $\times$ language)	8	0.44	1.21
between-task	22	0.67	2.03

Appendix

Table 11: Variance components for the directional-bias score. “SD of group means” is the standard deviation of the per-group mean score on the

[-2,+2]

scale; “range” is the difference between the largest- and smallest-mean group. “Range” for the main effect is

|\Delta\bar{\text{score}}|

. The headline implicit-vs-explicit shift exceeds the variation across models, country framings, and prompt languages each taken alone, and is comparable to (but somewhat smaller than) the variation across the eight country

\times

language cells and across the 22 tasks. Both the country

\times

language interaction and the task dimension are absorbed by the fixed-effects regression in § 5.1 .

Country $\times$ Language cell	$\boldsymbol{n}$	Mean score
Indian $\times$ Hindi	$7{,}913$	$-1.15$
Nigerian $\times$ Yoruba	$7{,}983$	$-0.65$
Kenyan $\times$ Swahili	$8{,}117$	$-0.38$
Indian $\times$ English	$6{,}815$	$-0.02$
Kenyan $\times$ English	$7{,}023$	$+0.01$
Nigerian $\times$ English	$7{,}040$	$+0.02$

Appendix

Table 12: Mean directional gender score per (country

\times

language) cell, pooled across the 8 models and both elicitation regimes. Sorted from most male-leaning to most female-leaning. The four English cells cluster near the neutral midpoint (

-0.02

+0.03

); the three local-language cells in India, Kenya, and Nigeria-Yoruba drift toward male; Nigerian

\times

Hausa is the lone local-language cell that does not amplify.

n

is the number of response-level observations contributing to each cell mean.

Task	$\boldsymbol{\bar{s}_{\text{exp}}}$	$\boldsymbol{\bar{s}_{\text{imp}}}$	$\boldsymbol{\Delta}$
Cook	$+1.16$	$-0.94$	$-2.10$
Playing with children	$+1.31$	$-0.52$	$-1.83$
Grocery shopping	$+1.57$	$-0.01$	$-1.58$
Educating children	$+1.41$	$-0.06$	$-1.48$
Subsistence farmer	$+0.13$	$-1.32$	$-1.45$
Entertaining guests	$+1.34$	$-0.08$	$-1.42$

Appendix

Table 13: Per-task implicit-vs-explicit shift in mean directional score, pooled across all eight models.

\Delta=\overline{\text{score}}_{\text{implicit}}-\overline{\text{score}}_{\text{explicit}}

; negative values indicate a shift toward male. Sorted from most male-shifted to most female-shifted. The seven female-shifted tasks (bottom of the table) are roles where the explicit baseline already sits at or below the neutral midpoint.

Most male-shifted cells
Model	Task	$\boldsymbol{\bar{s}_{\text{exp}}}$	$\boldsymbol{\bar{s}_{\text{imp}}}$	$\boldsymbol{\Delta}$
N-ATLaS	Cook	$+2.00$	$-0.56$	$-2.56$
LLaMA 3.1 8B	Cook	$+1.47$	$-1.08$	$-2.56$
LLaMA 3.1 8B	Playing with children	$+2.00$	$-0.42$	$-2.42$
GPT-5.1	Cook	$+1.26$	$-1.16$	$-2.41$
Llama-3-Instruct	Playing with children	$+1.90$	$-0.44$	$-2.34$

Appendix

Table 14: Five most male-shifted and five most female-shifted (model

\times

task) cells.

\Delta=\overline{\text{score}}_{\text{implicit}}-\overline{\text{score}}_{\text{explicit}}

; the theoretical maximum is

-4

(a perfect explicit-female to implicit-male flip on the

[-2,+2]

scale).

Quantity	Value
$\beta$ on is_implicit	$-0.719$
SE (cluster-robust CR1)	$0.195$
$t$ statistic	$-3.69$
$n$ clusters (tasks)	$22$
$n$ observations	$37{,}707$
Asymptotic two-sided $p$	$0.00022$

Appendix

Table 15: Small-cluster correction for the headline implicit-vs-explicit shift. The wild cluster bootstrap

p

-value is approximately one order of magnitude larger than the asymptotic

p

-value but the result remains significant at conventional thresholds; the point estimate is unaffected.

Explicit scoring	$\boldsymbol{\beta}$	95% CI	$\boldsymbol{p}$ (asym.)	$\boldsymbol{p}$ (wild boot.)
5-point (main analysis)	$-0.719$	$[-1.10,-0.34]$	$<.001$	$.002$
3-point (collapsed)	$-0.539$	$[-1.06,-0.01]$	$.044$	$.053$

Appendix

Table 16: Headline implicit-vs-explicit coefficient with the explicit free-text scale collapsed to the same three-point support as the implicit coding. Both rows: identical sample (

n=37{,}707

), model + task + nationality fixed effects, task-clustered standard errors; wild cluster bootstrap with

B=4{,}999

Rademacher draws by task.

Gender Bias in Language Models

Gender bias in language models examines how systems encode and express gender stereotypes and how to evaluate and reduce their effects.

See all 64 papers

Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit
May 29, 2026Jiwoo Choi, Seonwoo Ahn, Tongxin Zhang +1Gender Bias in Language Models Multilingual Language Model Evaluation
School of Digital Humanities and Computational Social Sciences, KAIST, South Korea
Man Made Language Models? Evaluating LLMs' Perpetuation of Masculine Generics Bias
Feb 14, 2025Enzo Doyen, Amalia TodirascuGender Bias in Language Models Language Model Bias Evaluation
University of Strasbourg
Gender bias across LLMs is common and highly heterogeneous
Sep 29, 2026Edoardo Bolzoni, Valerio CapraroGender Bias in Language Models LLM Auditing
University of Milan-Bicocca, Milan, Italy

May 29, 2026J/K move · Enter open · S save

Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit

Jiwoo Choi, Seonwoo Ahn, Tongxin Zhang, Seohyon Jung

School of Digital Humanities and Computational Social Sciences, KAIST, South Korea

Gender Bias in Language Models Multilingual Language Model Evaluation LLM Auditing Cultural Bias in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models

Open paper AskPDF arXiv

We audit six large language models (LLMs) for gender stereotyping across English, Korean, Chinese, and Japanese. Three were developed primarily for English-language use (Claude, GPT, Gemini) and three for East Asian use (DeepSeek, Syn-Pro, HyperCLOVA X). We adopt the HEXACO-100 personality inventory and anchor each model against a cross-cultural human dataset spanning 48 countries to ask not whether LLMs are biased, but how far their gender attributions drift from the populations they are deployed among. Our findings show that their stereotyping spans a range roughly 2.5 times wider than the entire cross-country range found in humans, and the effect can compound across languages. One English-centric model, prompted in Korean, reached 5 times the local baseline, even when the prompt stated the candidate had already been hired, which often dampens human stereotyping. To characterize such behaviors without ranking them, we introduce a four-pattern framework -- concordance, suppression, reorganization, and amplification -- across 24 (model x language) cells. Item-level analysis reveals that translation does not just rescale stereotypes, but changes the attributes tied to it, hiding significant rearrangement under the surface while appearing well-calibrated. Our results ultimately suggest that no single debiasing pipeline is likely to address bias evenly across linguistic boundaries.

Abstract

Gender Bias in Language Models

Explore similar work

Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit

Man Made Language Models? Evaluating LLMs' Perpetuation of Masculine Generics Bias

Gender bias across LLMs is common and highly heterogeneous