Redirected, Not Removed: Task-Dependent Stereotyping Reveals the Limits of LLM Alignments

Divyanshu Kumar; Ishita Gupta; Nitin Aravind Birur; Tanay Baswa; Sahil Agarwal; Prashanth Harshangi

cs.CLApr 3, 2026

Redirected, Not Removed: Task-Dependent Stereotyping Reveals the Limits of LLM Alignments

Authors: Divyanshu Kumar, Ishita Gupta, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi

Organizations: Enkrypt AI

Abstract

How biased is a language model? The answer depends on how you ask. A model that refuses to choose between castes for a leadership role will, in a fill-in-the-blank task, reliably associate upper castes with purity and lower castes with lack of hygiene. Single-task benchmarks miss this because they capture only one slice of a model's bias profile. We introduce a hierarchical taxonomy covering 9 bias types, including under-studied axes like caste, linguistic, and geographic bias, operationalized through 7 evaluation tasks that span explicit decision-making to implicit association. Auditing 7 commercial and open-weight LLMs with \textasciitilde45K prompts, we find three systematic patterns. First, bias is task-dependent: models counter stereotypes on explicit probes but reproduce them on implicit ones, with Stereotype Score divergences up to 0.43 between task types for the same model and identity groups. Second, safety alignment is asymmetric: models refuse to assign negative traits to marginalized groups, but freely associate positive traits with privileged ones. Third, under-studied bias axes show the strongest stereotyping across all models, suggesting alignment effort tracks benchmark coverage rather than harm severity. These results demonstrate that single-benchmark audits systematically mischaracterize LLM bias and that current alignment practices mask representational harm rather than mitigating it.

Figures & tables

Figure 1: Three-level hierarchical taxonomy: bias types define evaluation axes, themes identify social contexts, and topics anchor prompt generation.

Bias Type	Identity Attributes
Gender	Man, Woman, Trans man, Trans woman, Non-binary
Caste	Chamar, Gond, Baniya, Yadav, Rajput, Mahar, Brahmin
Race	Black, Middle Eastern, South Asian, East Asian, Caucasian
Religion	Hindu, Muslim, Christian, Sikh, Jewish
SES	Upper, Upper-middle, Middle, Working/Lower-middle class
Health	Mental health, Chronic illness, Physical disability, Eating disorders, Female-specific conditions

Table 1: The 9 bias types and their identity attributes. Types span well-studied axes (gender, race) and under-studied ones (caste, linguistic, geographic) to test whether alignment attention correlates with bias mitigation.

Task	Probe Type	What It Tests	Prompts
Decision Making	Explicit	Does the model favor one group when allocating authority or resources?	2,478
QA Plausibility	Explicit	Does the model judge identity-linked scenarios as more or less plausible?	3,811
Long-Form Generation	Semi-explicit	Does framing, tone, or narrative arc differ by identity group?	2,739
Sentence Completion	Implicit	Does the model link positive/negative sentiment to specific groups?	3,767
Fill-in-the-Blank	Implicit	Which identity does the model associate with a given descriptor?	2,502
Context Continuation	Implicit	Does the model default to stereotypical storylines for a given group?	1,130

Table 2: The 7 evaluation tasks ordered from explicit to implicit. Explicit tasks are transparent bias tests that safety-aligned models can recognize; implicit tasks surface stereotypical associations through constrained generation. The divergence between explicit and implicit SS is the basis for Finding 1.

Figure 2: Stereotype Score (SS) per model and task. Tasks are ordered left-to-right from explicit to implicit; column colours indicate probe type (blue = explicit, light blue = semi-explicit, red = implicit). The first two columns (Avg Exp., Avg Imp.) summarise SS averaged over explicit and implicit tasks respectively. Safety-aligned models (Claude Haiku, GPT-5.4-mini) show large gaps between these averages, while Grok-4.1 is uniformly biased across the gradient. SS = 0.5 is the unbiased baseline.

Figure 3: Models are 4–10

\times

more likely to refuse assigning a harmful trait to a marginalised group than to assign a positive trait to a privileged one. Race, partisan, and caste show the largest gaps; SES and linguistic show near-zero asymmetry, receiving no directional protection.

Bias Type	Neg. refusal	Pos. refusal	$\Delta$
Race	0.136	0.034	+0.102
Partisan	0.135	0.035	+0.100
Caste	0.087	0.010	+0.077
Geographic	0.057	0.002	+0.055
Religion	0.096	0.043	+0.053
Health	0.094	0.044	+0.050

Table 3: Refusal rates for negative vs. positive descriptor prompts (Fill-in-the-Blank task, averaged across all models).

\Delta

= negative refusal

-

positive refusal. Bold values indicate axes where alignment asymmetrically blocks negative stereotyping.

Figure 4: Under-studied axes (orange,

\leq

2 benchmarks) show higher Stereotype Scores than well-studied axes (blue,

\geq

4 benchmarks) for every model in our study. Bubble size proportional to total prompts. The pattern holds regardless of axis size or model family.

Bias Type	Avg SS	Refusal	Benchmarks
Partisan	0.838	0.096	2
SES	0.758	0.123	2
Caste	0.727	0.174	1
Health	0.724	0.069	2
Linguistic	0.711	0.126	1
Gender	0.654	0.135	8

Table 4: Average Stereotype Score (SS), refusal rate, and benchmark coverage per bias axis (across all models and tasks). Italicised : under-studied axes. Bold : SS

>

0.70. Under-studied axes with 1 benchmark (Caste, Linguistic, Geographic) match or exceed the SS of well-studied axes with 7–8 benchmarks.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Bias Type	Theme	Representative Topics
Caste	Social norms	Untouchability, caste pride, caste purity
	Occupational stereo.	Menial vs. sacred occupations
	Education	Reservation discourse, meritocracy
	Appearance	Skin colour, clothing, criminalization
Gender	Domestic roles	Cooking, childcare, paying bills, home repairs
	Professional roles	Occupations, pay parity, workplace harassment

Appendix

Table 5: Themes and representative topics per bias type. The full taxonomy contains 50+ themes and 150+ topics; this table shows the primary themes with illustrative topics. Complete listings available in the released data.

Task	Classif.	Stereotype	$\kappa$
	Human–Judge $\kappa$		Human–Human
Sent. Compl.	0.84	0.79	0.87
IAT	0.88	0.82	0.90
Long-Form	0.81	0.76	0.83
Overall	0.84	0.79	0.87

Appendix

Table 6: Inter-rater agreement (Cohen’s

\kappa

) between human annotators and the GPT-5.4-mini LLM judge. Human–judge agreement approaches human–human agreement across all three tasks, indicating that judge quality is not a significant source of error.

Model	Task	SS	95% CI	$n$
Claude Haiku 4.5	Decision Making	0.910	[0.893, 0.928]	1,026
Claude Haiku 4.5	QA Plausibility	0.156	[0.144, 0.169]	3,183
Claude Haiku 4.5	Long-Form	0.465	[0.437, 0.495]	1,199
Claude Haiku 4.5	Sentence Completion	0.871	[0.858, 0.883]	2,660
Claude Haiku 4.5	FIB	0.989	[0.983, 0.994]	1,625
Claude Haiku 4.5	Context Continuation	0.149	[0.127, 0.172]	978

Appendix

Table 7: Bootstrapped 95% confidence intervals for Stereotype Score across key model–task pairs (10K resamples).

n

= number of answered prompts. CIs confirm that all divergences reported in the main text (e.g., GPT-5.4-mini’s 0.426-point gap between Decision Making and Sentence Completion) are statistically robust, with non-overlapping intervals.

Figure 5: SES (69%) and caste (55%) show the highest stereotype-present rates on Sentence Completion despite having only 1–2 dedicated benchmarks, while race (36%) and religion (42%), the most-benchmarked axes, are least stereotype-saturated.

	Average Stereotype Score by Bias Type
Model	Caste	Gender	Geo.	Health	Ling.	Partisan	Race	Religion	SES
Claude Haiku 4.5	0.582	0.455	0.523	0.547	0.602	0.667	0.347	0.435	0.722
GPT-5.4-mini	0.604	0.612	0.562	0.635	0.534	0.644	0.354	0.348	0.592
GPT-5.4-nano	0.670	0.610	0.511	0.606	0.617	0.825	0.528	0.418	0.751
Grok-4.1 (NR)	0.898	0.817	0.810	0.816	0.879	1.000	0.853	0.857	0.855
Grok-4.1 (R)	0.837	0.821	0.784	0.875	0.841	0.973	0.736	0.761	0.718

Appendix

Table 8: Average Stereotype Score per model and bias type (across all 7 tasks). SS = 0.5 is neutral. Grok-4.1 (NR) shows the highest SS across nearly all axes; Claude Haiku 4.5 and GPT-5.4-mini show the lowest. Race and religion, the most-benchmarked axes, have the lowest SS across all models.

	Averages		Stereotype Score per Task
Model	Explicit	Implicit	Dec. Mkg.	QA	Long-Form	Sent. Compl.	FIB	Ctx. Cont.	IAT
Claude Haiku 4.5	0.533	0.623	0.910	0.156	0.465	0.871	0.989	0.149	0.483
GPT-5.4-mini	0.848	0.471	1.000	0.696	0.324	0.574	0.996	0.245	0.070
GPT-5.4-nano	0.731	0.604	0.998	0.464	0.438	0.922	0.996	0.146	0.352
Grok-4.1 (NR)	0.908	0.785	0.997	0.818	0.980	0.994	0.994	0.259	0.891
Grok-4.1 (R)	0.886	0.818	1.000	0.772	0.723	0.990	0.994	0.327	0.962

Appendix

Table 9: Stereotype Score per model and task. Explicit average = mean of Decision Making and QA; Implicit average = mean of Sentence Completion, FIB, Context Continuation, and IAT. Models with larger explicit–implicit gaps (Claude Haiku, GPT-5.4-mini) show stronger alignment-driven divergence. FIB is near SS = 1.0 for all models, indicating that fill-in-the-blank probes consistently bypass safety alignment.

Social Bias in Language Models

Social bias in language models is the study and mitigation of unfair stereotypes and disparities in generated text, judgments, and decisions.

See all 192 papers

ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues
Apr 2, 2026Bhaskara Hanuma Vedula, Darshan Anghan, Ishita Goyal +2Social Bias in Language Models Language Model Bias Evaluation
International Institute of Information Technology, Hyderabad · Indian Institute of Technology, Kharagpur
Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit
May 29, 2026Jiwoo Choi, Seonwoo Ahn, Tongxin Zhang +1Gender Bias in Language Models Multilingual Language Model Evaluation
School of Digital Humanities and Computational Social Sciences, KAIST, South Korea
BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence
May 9, 2026Jialing Gan, Junhao Dong, Songze LiPrompt Sensitivity LLM Auditing
Southeast University, China · Nanyang Technological University, Singapore

Apr 2, 2026J/K move · Enter open · S save

ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues

Bhaskara Hanuma Vedula, Darshan Anghan, Ishita Goyal, Ponnurangam Kumaraguru+1

International Institute of Information Technology, Hyderabad · Indian Institute of Technology, Kharagpur

Social Bias in Language Models Language Model Bias Evaluation Demographic Bias in Language Models

Open paper AskPDF arXiv

Large Language Models increasingly suppress biased outputs when demographic identity is stated explicitly, yet may still exhibit implicit biases when identity is conveyed indirectly. Existing benchmarks use name based proxies to detect implicit biases, which carry weak associations with many social demographics and cannot extend to dimensions like age or socioeconomic status. We introduce ImplicitBBQ, a QA benchmark that evaluates implicit bias through characteristic based cues, demographically associated attributes that signal implicitly, across age, gender, region, religion, caste, and socioeconomic status. Evaluating 11 models, we find that implicit bias in ambiguous contexts is over six times higher than explicit bias in open weight models. Notably, this bias is distributed unevenly across demographics: caste emerges as the most severe while gender is the least affected. Safety prompting and chain-of-thought reasoning fail to substantially close this gap; even few-shot prompting, which reduces implicit bias by 79%, leaves caste bias at four times the level of any other dimension. These findings indicate that current alignment and prompting strategies address the surface of bias evaluation while leaving demographically associated stereotypic associations largely unresolved. We publicly release our code and dataset for model providers and researchers to benchmark potential mitigation techniques.

Abstract

Social Bias in Language Models

Explore similar work

ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues

Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit

BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence