ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues

Bhaskara Hanuma Vedula; Darshan Anghan; Ishita Goyal; Ponnurangam Kumaraguru; Abhijnan Chakraborty

cs.CLApr 2, 2026

ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues

Authors: Bhaskara Hanuma Vedula, Darshan Anghan, Ishita Goyal, Ponnurangam Kumaraguru, Abhijnan Chakraborty

Organizations: International Institute of Information Technology, Hyderabad · Indian Institute of Technology, Kharagpur

Abstract

Large Language Models increasingly suppress biased outputs when demographic identity is stated explicitly, yet may still exhibit implicit biases when identity is conveyed indirectly. Existing benchmarks use name based proxies to detect implicit biases, which carry weak associations with many social demographics and cannot extend to dimensions like age or socioeconomic status. We introduce ImplicitBBQ, a QA benchmark that evaluates implicit bias through characteristic based cues, demographically associated attributes that signal implicitly, across age, gender, region, religion, caste, and socioeconomic status. Evaluating 11 models, we find that implicit bias in ambiguous contexts is over six times higher than explicit bias in open weight models. Notably, this bias is distributed unevenly across demographics: caste emerges as the most severe while gender is the least affected. Safety prompting and chain-of-thought reasoning fail to substantially close this gap; even few-shot prompting, which reduces implicit bias by 79%, leaves caste bias at four times the level of any other dimension. These findings indicate that current alignment and prompting strategies address the surface of bias evaluation while leaving demographically associated stereotypic associations largely unresolved. We publicly release our code and dataset for model providers and researchers to benchmark potential mitigation techniques.

Figures & tables

Figure 1: Illustrative example with Llama-3.1-8B Instruct demonstrating how responses vary with the mode of expressing demographic identity. When the demographic is explicitly specified (A) or indicated via a name-based proxy (B), the model declines to answer. In contrast, when the same identity is only indirectly signaled through a cultural attribute (C), the model associates the cue with the Muslim identity and returns a stereotypical response.

Figure 2: Behaviour of Llama-3.1-8B Instruct under explicit versus implicit prompting in ambiguous and disambiguated conditions. Under explicit prompting, the model correctly abstains on ambiguous inputs and follows context on disambiguated ones. Under implicit prompting, it selects stereotypical answers in both conditions, disregarding ambiguity and overriding contextual evidence.

Demographic	Sub-Demo.	Example Cue
Gender	Male	has beard
Religion	Hindu	visits temple
Region	American	celebrates July 4th
SES	Poor	is homeless
Age	Old	has wrinkles
Caste	Brahmin	wears Janeu

Table 1: Sample implicit cues per demographic. See Appendix A , Table 5 for the complete list.

Figure 3: Bias scores across demographic dimensions for all 11 models under zero-shot prompting, for all contexts (explicit/implicit $\times$ ambiguous/disambiguous).

Context	$n$	$\tilde{\Delta}$	% I $>$ E	$r$	$p_{\text{BH}}$
Ambiguous	8	$0.25$	$100$	$1.00$	$.008^{**}$
Disambig.	8	$0.08$	$75$	$0.83$	$.020^{*}$

Table 3: Paired Wilcoxon signed-rank tests of the implicit

-

explicit bias gap for open-weight models.

\tilde{\Delta}

: median paired difference; % I

>

E: share of models with implicit

>

explicit;

r

: rank-biserial effect size;

p_{\text{BH}}

: Benjamini–Hochberg corrected

p

{}^{*}p<.05

{}^{**}p<.01

Figure 4: Accuracy and bias scores across four prompting strategies (zeroshot, safety, fewshot, CoT) for 11 models. Each panel shows ambiguous or disambiguated contexts under explicit or implicit prompting.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: National distribution of the 110 annotators, spanning 25 countries.

Panel of Figure 6 — Figure 6: The annotation tool interface, showing the welcome screen, the task instructions, and a representative annotation example.

Metric	Value
Total annotators	110
General pool	100
India-specific (caste)	10
Total candidate sentences	140
Non-caste sentences	124
Caste-specific sentences	16

Appendix

Table 4: Summary statistics from the human annotation validation study.

Dimension	Category	Implicit Cues
Gender	Male	has beard; has chest hair
Gender	Female	carries a purse; wears earrings
Religion	Hindu	worships cow; worships Ganga; wears tilak; follows dharma; visits temple
	Muslim	wears burka; prays namaz; celebrates Eid; believes in Quran; has nikkah; gives zakat; fasts during Ramadan
	Christian	observes Good Friday; attends Sunday service; says grace before meals; reads the Bible
	Jewish	celebrates Hanukkah

Appendix

Table 5: Final validated implicit cues across demographic dimensions. Each cue conveys demographic identity through cultural practices or physical attributes rather than explicit labels.

Social Bias in Language Models

Social bias in language models is the study and mitigation of unfair stereotypes and disparities in generated text, judgments, and decisions.

See all 192 papers

Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models
Feb 16, 2026Lance Calvin Lim Gamboa, Yue Feng, Mark LeeSocial Bias in Language Models Language Model Bias Evaluation
School of Computer Science, University of Birmingham · Department of Information Systems and Computer Science, Ateneo de Manila University
Redirected, Not Removed: Task-Dependent Stereotyping Reveals the Limits of LLM Alignments
Apr 3, 2026Divyanshu Kumar, Ishita Gupta, Nitin Aravind Birur +3Social Bias in Language Models Language Model Bias Evaluation
Enkrypt AI
Invisible Influences: Investigating Implicit Intersectional Biases through Persona Engineering in Large Language Models
Mar 16, 2026Nandini Arimanda, Achyuth Mukund, Sakthi Balan Muthiah +1Intersectional Bias in Language Models Social Bias in Language Models
Department of Computer Science and Engineering, Shiv Nadar University Chennai, Tamil Nadu, India · School of AI and CS, Plaksha University, Mohali, Punjab, India

Feb 16, 2026J/K move · Enter open · S save

Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

Lance Calvin Lim Gamboa, Yue Feng, Mark Lee

School of Computer Science, University of Birmingham · Department of Information Systems and Computer Science, Ateneo de Manila University

Social Bias in Language Models Language Model Bias Evaluation Cultural Bias in Language Models Gender Bias in Language Models Cross-Cultural Language Model Evaluation

Open paper AskPDF arXiv

With natural language generation becoming a popular use case for language models, the Bias Benchmark for Question-Answering (BBQ) has grown to be an important benchmark format for evaluating stereotypical associations exhibited by generative models. We expand the linguistic scope of BBQ and construct FilBBQ through a four-phase development process consisting of template categorization, culturally aware translation, new template construction, and prompt generation. These processes resulted in a bias test composed of more than 10,000 prompts which assess whether models demonstrate sexist and homophobic prejudices relevant to the Philippine context. We then apply FilBBQ on models trained in Filipino but do so with a robust evaluation protocol that improves upon the reliability and accuracy of previous BBQ implementations. Specifically, we account for models' response instability by obtaining prompt responses across multiple seeds and averaging the bias scores calculated from these distinctly seeded runs. Our results confirm both the variability of bias scores across different seeds and the presence of sexist and homophobic biases relating to emotion, domesticity, stereotyped queer interests, and polygamy. FilBBQ is available via https://github.com/gamboalance/filbbq.

Abstract

Social Bias in Language Models

Explore similar work

Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

Redirected, Not Removed: Task-Dependent Stereotyping Reveals the Limits of LLM Alignments

Invisible Influences: Investigating Implicit Intersectional Biases through Persona Engineering in Large Language Models