When Who You Are Can Change the Code You Get: A Study of Persona-Induced Bias in LLM Code Generation

Anubhav Gupta; Mayara Costa Figueiredo; Leticia Santos Machado; Tanner Wright; Ivan Beschastnikh; Cleidson R. B. de Souza; Gema Rodríguez-Pérez

cs.SEAug 12, 2026

When Who You Are Can Change the Code You Get: A Study of Persona-Induced Bias in LLM Code Generation

Authors: Anubhav Gupta, Mayara Costa Figueiredo, Leticia Santos Machado, Tanner Wright, Ivan Beschastnikh, Cleidson R. B. de Souza, Gema Rodríguez-Pérez

Abstract

Large Language Models (LLMs) are widely used as programming assistants, yet it remains unclear whether and how user's demographic information impacts the technical quality of generated code. We conduct a large-scale empirical study of persona-induced bias in LLM-based code generation, focusing a proprietary model (Gemini 2.5 Pro) and an open-weight model (GPT-OSS-120B). Using 18 demographic personas spanning nationality, gender, and experience level, we compare persona-induced prompts against a neutral baseline. Across 35,000+ generated programs, we analyze demographic marker leakage in reasoning and responses, as well as differences in functional correctness, maintainability, code style, and security. Our results show that demographic cues are frequently reflected in LLM reasoning and outputs. Demographic markers appear in up to 65% of responses and 70% of reasoning traces, despite being semantically irrelevant to the tasks. On LiveCodeBench, persona prompting were associated with lower correctness scores of the Gemini model by an average of 1.54 percentage points, with one persona exhibiting a decrease of 3.6% (odds ratio = 0.51). In contrast, the accuracy of the GPT-OSS model improved by 3.4 - 5.7% across all personas (odds ratios = 1.8 - 3.0). Maintainability and code style metrics show statistically significant but negligible effect sizes (all Cliff's δ < 0.15), and security vulnerabilities exhibit no systematic persona-specific patterns. Overall, our results show that the presence of demographic information about users is associated with measurable variation in LLM reasoning and code quality even in purely technical tasks, and that these effects hold across models. Our work highlights an under-examined risk in LLM-assisted software development.

Figures & tables

Figure 1: Illustration showing demographic persona prompts (left) and observed demographic markers in the LLM’s internal reasoning process (right).

Variant	Structure	Persona Placement / Intended Effect	Leakage Strength	Decision
A: Context Line	Neutral context block placed before the task.	Provides subtle demographic information without implying stylistic changes; tests low-intervention exposure.	Low	Rejected
B: First-Person Disclosure	First-person phrasing (“For context: I am …”).	Increases persona salience through narrative voice; tests whether self-description alters tone or style.	Medium	Rejected
C: Metadata Block	YAML-like structured profile ( age:, gender:, nationality:, ... ).	Tests whether structured context reduces style bleed while maintaining demographic salience.	Low-Medium	Rejected
D: Context After Task	Same structure as Variant A, but placed after the task.	Tests order sensitivity in planning and whether demographic cues influence late-stage decision-making.	Low	Rejected
E: Role Assignment	Persona embedded in a role-binding instruction (“You are assisting a developer with this background: …”).	Strongest identity binding; tests whether task framing with persona context increases demographic leakage.	High	Selected

Table 1: Prompt variants we explored in the sensitivity analysis.

Figure 3: The prompt construction and selection pipeline we used in the main study.

Figure 4: An illustration of the demographic persona-induced prompt structure based on the Jinja template.

Number and total percentage of omitted data by persona
Difficulty	Count	Percentage	Persona	Count	Percentage
Easy	22	0.6%
Medium	970	24.7%
Hard	2,936	74.7%
neutral	275	7.0%	br_nb_junior	209	5.3%
br_w_senior	228	5.8%	in_m_senior	207	5.3%

Table 2: Distribution of excluded GPT-OSS-120B issues.

Figure 5: RQ1 methodology: demographic marker detection.

Test	Purpose	Decision Criterion
Shapiro-Wilk	Test normality	$p<0.05$ → non-normal
Levene’s	Test equal variance	$p<0.05$ → heterogeneous
Kruskal-Wallis	Omnibus comparison	$p<0.05$ → significant
Mann-Whitney U	Pairwise comparison	$p_{\text{corrected}}<0.05$ → sign.
Bonferroni	Control FWER	$\alpha=0.05/18=0.00278$
Holm-Bonferroni	Stepwise FWER control	$p_{(i)}<0.05/(m-i+1)$

Table 3: Statistical tests and procedures summary.

Figure 6: Demographic marker prevalence analysis: (A) Comparison of marker prevalence between THOUGHTS and RESPONSE sections, (B) Overall prevalence appearing in either, and (C) Distribution of marker counts stratified by problem difficulty.

Figure 7: Persona-specific demographic marker leakage patterns: (A) Aggregate leakage scores by persona attributes, (B) Spider charts by nationality.

Figure 8: Difficulty effects on demographic marker leakage: (A) All five markers show declining prevalence from easy to hard problems, and (B) Nationality-difficulty interaction shows Brazil and Spain exhibit steeper declines than India.

Persona	Persona Acc.	Diff (%)	OR	$p$ -value
in_nb_junior	0.6398	$-3.60$	0.513	0.000659
br_nb_junior	0.6502	$-2.56$	0.591	0.011170
br_m_junior	0.6502	$-2.56$	0.630	0.017153
br_w_junior	0.6502	$-2.56$	0.640	0.019061
in_w_junior	0.6540	$-2.18$	0.635	0.030180
br_m_senior	0.6540	$-2.18$	0.676	0.043723

Table 4: McNemar’s test results. A comparison of functional correctness of demographic personas against the neutral baseline.

Persona	Mean	Mean Diff	$p_{\text{corr}}$	$\delta$	Effect Size
sp_w_junior	65.84	-9.74	$<.001$ ***	-0.520	Large
br_w_junior	66.75	-8.84	$<.001$ ***	-0.480	Large
sp_m_junior	67.04	-8.55	$<.001$ ***	-0.462	Medium
br_m_junior	67.15	-8.44	$<.001$ ***	-0.452	Medium
br_nb_junior	68.44	-7.15	$<.001$ ***	-0.407	Medium
in_m_junior	69.34	-6.25	$<.001$ ***	-0.392	Medium

Table 5: Pairwise comparisons – each persona vs. neutral baseline (maintainability index).

Significant Comparisons (Better than Neutral)
Persona	Mean	Mean Diff	$U$	$p_{\text{corr}}$	$\delta$	Sig.
sp_m_senior	4.97	+0.83	629,786	$<.001$ ***	+0.132	Yes
sp_w_senior	4.94	+0.80	624,617	$<.001$ ***	+0.125	Yes
br_w_senior	4.78	+0.65	616,731	$<.001$ ***	+0.109	Yes
sp_nb_senior	4.71	+0.57	612,218	$<.001$ ***	+0.102	Yes
sp_m_junior	4.75	+0.62	606,110	.005**	+0.090	Yes

Table 6: Pairwise comparisons – each persona vs. neutral baseline (Pylint score).

Figure 9: Demographic marker prevalence analysis for GPT-OSS-120B: (A) Comparison of marker prevalence between THOUGHTS and RESPONSE sections, (B) Overall prevalence appearing in either, and (C) Distribution of marker counts stratified by problem difficulty.

Figure 10: Persona-specific demographic marker leakage patterns for GPT-OSS-120B: (A) Aggregate leakage scores by persona attributes, (B) Spider charts by nationality.

Figure 11: Difficulty effects on demographic marker leakage for GPT-OSS-120B: (A) Most markers show declining prevalence from easy to hard problems, and (B) Nationality-difficulty interaction shows a consistent decline across all 3 nationalities.

Persona	Persona Acc.	Diff (%)	OR	p-value
in_nb_senior	0.4237	+5.69	3.000	0.000000
sp_w_senior	0.4237	+5.69	2.818	0.000000
sp_nb_senior	0.4218	+5.50	2.657	0.000000
in_w_junior	0.4180	+5.12	2.500	0.000002
sp_nb_junior	0.4180	+5.12	2.500	0.000002
sp_w_junior	0.4180	+5.12	2.459	0.000003

Table 7: McNemar’s test results (GPT-OSS-120B). A comparison of functional correctness of demographic personas against the neutral baseline.

Persona	Mean	Mean Diff	$p_{corr}$	$\delta$	Effect Size
in_m_junior	76.85	+1.21	0.187	+0.075	Negligible
br_nb_junior	76.67	+1.03	0.283	+0.071	Negligible
br_w_junior	76.75	+1.11	0.304	+0.070	Negligible
sp_w_senior	76.69	+1.05	0.414	+0.066	Negligible
sp_nb_junior	76.49	+0.86	0.437	+0.066	Negligible
br_nb_senior	76.58	+0.94	0.654	+0.061	Negligible

Table 8: Pairwise comparisons: – each persona vs. neutral baseline (maintainability index - GPT-OSS-120B).

Persona	Mean	Mean Diff	U	$p_{corr}$	$\delta$	Sig.
Significant Comparisons (Better than Neutral)
None
Non-Significant Comparisons
br_w_senior	6.65	+0.01	310,171	1.000	+0.036	No
in_m_junior	6.67	+0.03	311,716	1.000	+0.031	No
in_w_senior	6.68	+0.04	318,894	1.000	+0.028	No

Table 9: Pairwise comparisons – each persona vs. neutral baseline (PyLint score - GPT-OSS-120B).

Demographic Bias in Language Models

Demographic bias in language models studies how identity cues shape model outputs, decisions, and representations, and how to detect and reduce resulting disparities.

See all 135 papers

Jul 19, 2026

The Librarian Who Refused to Code: Model-Dependent Identity Enactment in LLM Code Generation

Biographical personas are widely used in system prompts, but their effects on code generation are rarely evaluated under controlled, pre-registered conditions. We tested four prompt conditions (no persona, two engineer personas, and a research-librarian persona), 12 code-generation tasks, two frontier models, and five runs per cell (480 completions). Persona effects differed between the two tested models. Under the pre-registered mixed-effects analysis, the condition-by-model interaction was significant for provider-reported output tokens; a post-hoc visible-character measure showed the same qualitative pattern. Six GPT-5.5 completions were length-capped and are reported separately. On Claude Opus, the minimalist engineer persona reduced visible output by 30% (33% in provider tokens) without improving correctness, while the thorough engineer persona increased output without a correctness gain. In an exploratory post-hoc analysis, the librarian persona elicited in-character disclaimers in 55 of 60 Opus responses and 12 genuine no-code responses, lowering mean correctness from 0.92 to 0.67. GPT-5.5 produced neither behavior in its 59 non-truncated responses. These results are consistent with personas acting as Model-Dependent behavioral-policy biases rather than universal quality interventions. We release raw completions, derived scores, analysis artifacts, a pre-registration document, and an execution gate log; end-to-end test-based rescoring requires an unreleased task harness.

Shayell Aharon Salomon, Noam Israel, Ido Safruti +1

Bluebear Security

Personality Modeling in Language Models Code Generation Code Generation Evaluation Persona-Conditioned Generation

Ask PDF

May 1, 2026

Social Bias in LLM-Generated Code: Benchmark and Mitigation

Large Language Models (LLMs) are increasingly deployed to generate code for human-centered applications where demographic fairness is critical. However, existing evaluations focus almost exclusively on functional correctness, leaving social bias in LLM-generated code largely unexamined. Extending our prior work on Solar, we conduct a comprehensive empirical study using SocialBias-Bench, a benchmark of 343 real-world coding tasks spanning seven demographic dimensions. We evaluate four prominent LLMs and find severe bias across all models, with Code Bias Scores reaching up to 60.58%. We further show that standard prompt-level interventions, such as Chain-of-Thought reasoning and fairness persona assignment, inadvertently amplify bias rather than reduce it. We then investigate whether structured multi-agent software process frameworks can improve fairness, finding that structured pipelines reduce bias when early roles correctly scope what the code should and should not consider. However, adding explicit fairness instructions to all agent roles produces worse outcomes than providing none, suggesting that diffused responsibility goes unaddressed. To address these limitations, we propose the Fairness Monitor Agent (FMA), a modular component that plugs into any existing code generation pipeline without modifying it. FMA analyzes the task description to determine which attributes should be considered or restricted, then detects and corrects violations through an iterative review process, without requiring an executable test suite. Evaluated on all 343 tasks, FMA reduces bias by 65.1% compared to a developer agent alone and improves functional correctness from 75.80% to 83.97%, outperforming all other studied approaches.

Fazle Rabbi, Lin Ling, Song Wang +1

Concordia University, Montreal, Canada · Lassonde School of Engineering, York University, Toronto, Canada

Social Bias in Language Models Agentic Code Generation Algorithmic Bias Code Generation Evaluation

Ask PDF

Apr 23, 2026

From If-Statements to ML Pipelines: Revisiting Bias in Code-Generation

Prior work evaluates code generation bias primarily through simple conditional statements, which represent only a narrow slice of real-world programming and reveal solely overt, explicitly encoded bias. We demonstrate that this approach dramatically underestimates bias in practice by examining a more realistic task: generating machine learning (ML) pipelines. Testing both code-specialized and general-instruction large language models, we find that generated pipelines exhibit significant bias during feature selection. Sensitive attributes appear in 87.7% of cases on average, despite models demonstrably excluding irrelevant features (e.g., including "race" while dropping "favorite color" for credit scoring). This bias is substantially more prevalent than that captured by conditional statements, where sensitive attributes appear in only 59.2% of cases. These findings are robust across prompt mitigation strategies, varying numbers of attributes, and different pipeline difficulty levels. Our results challenge simple conditionals as valid proxies for bias evaluation and suggest current benchmarks underestimate bias risk in practical deployments.

Minh Duc Bui, Xenia Heilmann, Mattia Cerrato +2

Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Ciudad de Mexico · University of Colorado Boulder, USA

Algorithmic Bias Code Generation Evaluation Demographic Bias in Language Models

Ask PDF

Abstract

Demographic Bias in Language Models

Explore similar work

The Librarian Who Refused to Code: Model-Dependent Identity Enactment in LLM Code Generation

Social Bias in LLM-Generated Code: Benchmark and Mitigation

From If-Statements to ML Pipelines: Revisiting Bias in Code-Generation