BanglaBBQ: A Bias Benchmark for Question Answering in the Bangladeshi Context

Sumit Anthony Gomes; Yasin Rahman; Taeeba Tasnia; Faiza Nooren Suhita

cs.CLOct 17, 2026

BanglaBBQ: A Bias Benchmark for Question Answering in the Bangladeshi Context

Authors: Sumit Anthony Gomes, Yasin Rahman, Taeeba Tasnia, Faiza Nooren Suhita

Organizations: BRAC University, Dhaka, Bangladesh

Abstract

This paper introduces BanglaBBQ, a question-answering bias benchmark designed to capture stereotypes grounded in the Bangladeshi context, in both Bangla and English. Following KoBBQ and PakBBQ, BanglaBBQ adapts the original Bias Benchmark for Question Answering (BBQ) by classifying its templates into three groups: directly translated (DT), modified to align target groups or contexts with Bangladesh (TM), and removed when deemed unsuitable for the Bangladeshi context (SR). This process retains 140 templates across six categories, comprising 76 DT and 64 TM templates. Five newly authored (NA) categories, based on stereotypes common in Bangladeshi society, are then added: marital status, age and political perception, regional origin, educational background, and mental-health status. Native Bangladeshi annotators judged all 466 newly authored templates, and 372 were retained because a strict majority of English annotators identified the intended target and rated the attitude as common enough. Each newly authored template is linked to a documented source and a verbatim quote from it. Each evaluated template also has a counter-stereotypical version of its disambiguated context. The dataset was evaluated using two open-weight models (Qwen3-4B and Qwen3-8B) and two closed models (GPT-4.1-mini and Gemini 2.5 Flash). All four lean towards the stereotyped person when the context is ambiguous, in both languages, and the effect of language differs by model, so English results do not predict behaviour in Bangla. On the adapted BBQ templates, both Qwen3 models are also less accurate when the context contradicts a stereotype than when it confirms it. On the newly authored templates, the closed models follow counter-stereotypical evidence almost perfectly, and all four models are more biased on the templates that annotators validated than on those they rejected.

Figures & tables

Category	$n$	EN	BN	EN	BN	EN	BN	Retained
		AC1 [95% CI]		Fleiss’ $\kappa$		Kept
Marital status	100	0.654 [0.557, 0.748]	0.821 [0.753, 0.888]	0.351	0.360	80	89	80
Age & political perception	100	0.563 [0.463, 0.662]	0.617 [0.523, 0.711]	0.446	0.438	62	69	62
Regional origin	100	0.658 [0.564, 0.744]	0.678 [0.572, 0.772]	0.224	0.221	85	83	85
Educational background	95	0.667 [0.579, 0.752]	0.716 [0.614, 0.803]	0.286	0.177	77	85	77
Mental-health status	71	0.798 [0.702, 0.875]	0.880 [0.805, 0.942]	0.021	0.028	68	70	68

Table 1: Human annotation of BanglaBBQ templates in English (EN) and Bangla (BN).

n

: templates annotated. AC1: Gwet’s AC1 on target / non-target / can’t tell. Kept: templates on which a strict majority chose the intended target and mean relevance was at least 1.5. Retained: kept in English.

Category	Auth.	Ret.	Eval.	Inst.
Marital status	100	80	75	900
Age & political perception	100	62	62	744
Regional origin	100	85	80	960
Educational background	95	77	76	912
Mental-health status	71	68	67	804
Total	466	372	360	4,320

Table 2: Newly authored templates by category: authored (Auth.), retained after human validation (Ret., § 3.4 ), evaluated (Eval.: retained and with a usable counter-stereotypical version, § 6.2 ), and test instances per language (Inst.) for the evaluated templates (eight in version A and four in version B per template). Five further Educational-background templates were dropped before annotation because their quotes did not support the encoded stereotype.

Category	DT	TM	SR	Instances
Age	14	7	4	2,488
Disability status	20	5	0	1,432
Gender identity	13	30	7	4,808
Physical appearance	15	7	3	1,296
Religion	4	1	20	160
Socio-economic status	10	14	1	5,144

Table 3: Adapted BBQ templates by category. DT: directly translated; TM: target-modified; SR: removed. Instances per language (5,928 DT and 9,400 TM). Five further BBQ categories (168 templates) were removed.

Model		$\text{acc}_{\text{amb}}$ [95% CI]	$\text{acc}_{\text{dis}}$	$s_{\text{AMB}}$ [95% CI]	$\text{diff}_{\text{AMB}}$	$s_{\text{DIS}}$	$\text{diff}_{\text{DIS}}$ [95% CI]	OOS	Consist.
Qwen3-4B	EN	0.169 [0.14, 0.20]	0.992	$+$ 0.425 [0.36, 0.49]	$+$ 0.407	$+$ 0.012	$+$ 0.012 [0.00, 0.02]	1.2%	0.862
Qwen3-4B	BN	0.069 [0.05, 0.09]	0.963	$+$ 0.336 [0.27, 0.40]	$+$ 0.328	$+$ 0.042	$+$ 0.042 [0.03, 0.06]	0.7%	0.825
Qwen3-8B	EN	0.461 [0.42, 0.51]	0.999	$+$ 0.293 [0.24, 0.35]	$+$ 0.278	$+$ 0.001	$+$ 0.001 [0.00, 0.00]	0.9%	0.866
Qwen3-8B	BN	0.554 [0.51, 0.60]	0.988	$+$ 0.210 [0.16, 0.26]	$+$ 0.176	$+$ 0.013	$+$ 0.017 [0.01, 0.03]	2.6%	0.839
GPT-4.1-mini	EN	0.372 [0.33, 0.41]	0.999	$+$ 0.475 [0.42, 0.52]	$+$ 0.469	$+$ 0.002	$+$ 0.003 [0.00, 0.01]	0.3%	0.910
GPT-4.1-mini	BN	0.384 [0.35, 0.43]	0.993	$+$ 0.399 [0.34, 0.45]	$+$ 0.396	$+$ 0.002	$+$ 0.002 [ $-$ 0.00, 0.01]	0.2%	0.897

Table 4: Results on the newly authored templates (360 templates, versions A and B). Positive bias scores are stereotype-aligned.

s_{\text{DIS}}

and

\text{diff}_{\text{DIS}}

are computed over versions A and B together. OOS: share of all items with no valid majority answer. Consist.: share of items answered identically in all three option orders.

Model		Mar.	Age-Pol.	Reg.	Edu.	MH
Qwen3-4B	EN	.413	.347	.200	.763	.299
Qwen3-4B	BN	.347	.315	.000	.447	.575
Qwen3-8B	EN	.187	.234	.131	.493	.351
Qwen3-8B	BN	.113	.089	.025	.342	.321
GPT-4.1-mini	EN	.470	.415	.494	.612	.325
GPT-4.1-mini	BN	.453	.351	.353	.520	.284

Table 5: Ambiguous-context bias

\text{diff}_{\text{AMB}}

by category (highest per row in bold). Mar.: marital status (75 templates); Age-Pol.: age and political perception (62); Reg.: regional origin (80); Edu.: educational background (76); MH: mental-health status (67).

Model		Amb.	Dis.	$s_{\text{AMB}}$	$\text{diff}_{\text{AMB}}$	$s_{\text{DIS}}$	$\text{diff}_{\text{DIS}}$
		Accuracy
Qwen3-4B	EN	.643	.899	$+$ .224	$+$ .211	$+$ .035	$+$ .044
Qwen3-4B	BN	.323	.838	$+$ .294	$+$ .275	$+$ .058	$+$ .060
Qwen3-8B	EN	.858	.904	$+$ .106	$+$ .101	$+$ .026	$+$ .039
Qwen3-8B	BN	.707	.859	$+$ .178	$+$ .151	$+$ .056	$+$ .063

Table 6: Results on the adapted BBQ templates (140 templates, balanced sample of 3,256 instances per language).

\text{diff}_{\text{DIS}}

intervals exclude zero for both models in both languages.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

		$s_{\text{AMB}}$		$\text{acc}_{\text{amb}}$
Model		Valid.	Rej.	Valid.	Rej.
Qwen3-4B	EN	$+$ .418	$+$ .218	.172	.346
Qwen3-4B	BN	$+$ .329	$+$ .197	.071	.122
Qwen3-8B	EN	$+$ .290	$+$ .127	.458	.777
Qwen3-8B	BN	$+$ .211	$+$ .007	.556	.761
GPT-4.1-mini	EN	$+$ .475	$+$ .179	.368	.593

Appendix

Table 7: Ambiguous-context results on templates that annotators validated (372) and rejected (94), version A.

		$s_{\text{AMB}}$ by wording
Model		1	2	3
Qwen3-8B	EN	$+$ .290	$+$ .260	–
Qwen3-8B	BN	$+$ .211	$+$ .200	$+$ .132
GPT-4.1-mini	EN	$+$ .475	$+$ .315	$+$ .264
GPT-4.1-mini	BN	$+$ .402	$+$ .141	$+$ .311
Gemini 2.5 F.	EN	$+$ .148	$+$ .094	$+$ .044

Appendix

Table 8: Ambiguous-context bias under three prompt wordings (372 validated templates, version A). Wording 1 is used for the main results. The Qwen3-8B English run with wording 3 failed (57% of items without a valid answer) and is excluded. Qwen3-4B is reported on wording 1 only: its wording 2 and 3 runs gave many invalid answers and were excluded. Qwen3-8B wordings 2 and 3 were run mostly (78% of the calls scored here) through OpenRouter ( qwen/qwen3-8b , temperature 0, no reasoning tokens) and the rest locally; wording 1 was run entirely locally.

Model		Acc.	$s_{\text{DIS}}$	Acc.	$s_{\text{DIS}}$	$\text{diff}_{\text{DIS}}$
		Version A		Version B
Qwen3-4B	EN	.999	$+$ .997	.986	$-$ .972	$+$ .012
Qwen3-4B	BN	.983	$+$ .967	.942	$-$ .883	$+$ .042
Qwen3-8B	EN	1.00	$+$ 1.00	.999	$-$ .997	$+$ .001
Qwen3-8B	BN	.996	$+$ .992	.979	$-$ .972	$+$ .017
GPT-4.1-mini	EN	1.00	$+$ 1.00	.997	$-$ .997	$+$ .003

Appendix

Table 9: Disambiguated accuracy and

s_{\text{DIS}}

under version A (stereotype-aligned) and version B (counter-stereotypical), 360 templates.

s_{\text{DIS}}

near

+1

in version A reflects reading comprehension, not bias.

		Qwen3-4B		Qwen3-8B
Category	$n$	EN	BN	EN	BN
Age	21	$+$ .211	$+$ .125	$+$ .202	$+$ .098
Disability status	25	$+$ .261	$+$ .422	$+$ .072	$+$ .218
Gender identity	43	$+$ .055	$+$ .055	$+$ .015	$+$ .026
Physical appearance	22	$+$ .241	$+$ .241	$+$ .122	$+$ .244
Religion	5	$+$ .300	$+$ .800	$+$ .300	$+$ .350

Appendix

Table 10: Ambiguous-context bias

\text{diff}_{\text{AMB}}

on the adapted BBQ templates by category;

n

: templates. Religion has only five templates.

Cultural Bias in Language Models

Cultural bias in language models is the uneven representation of cultural knowledge, values, identities, and norms across groups and languages.

See all 125 papers

Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models
Feb 16, 2026Lance Calvin Lim Gamboa, Yue Feng, Mark LeeSocial Bias in Language Models Language Model Bias Evaluation
School of Computer Science, University of Birmingham · Department of Information Systems and Computer Science, Ateneo de Manila University
MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs
Jun 11, 2024Vera Neplenbroek, Arianna Bisazza, Raquel FernándezSocial Bias in Language Models Language Model Bias Evaluation
Institute for Logic, Language and Computation, University of Amsterdam · Center for Language and Cognition, University of Groningen
ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues
Apr 2, 2026Bhaskara Hanuma Vedula, Darshan Anghan, Ishita Goyal +2Social Bias in Language Models Language Model Bias Evaluation
International Institute of Information Technology, Hyderabad · Indian Institute of Technology, Kharagpur

Feb 16, 2026J/K move · Enter open · S save

Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

Lance Calvin Lim Gamboa, Yue Feng, Mark Lee

School of Computer Science, University of Birmingham · Department of Information Systems and Computer Science, Ateneo de Manila University

Social Bias in Language Models Language Model Bias Evaluation Cultural Bias in Language Models Gender Bias in Language Models Cross-Cultural Language Model Evaluation

Open paper AskPDF arXiv

With natural language generation becoming a popular use case for language models, the Bias Benchmark for Question-Answering (BBQ) has grown to be an important benchmark format for evaluating stereotypical associations exhibited by generative models. We expand the linguistic scope of BBQ and construct FilBBQ through a four-phase development process consisting of template categorization, culturally aware translation, new template construction, and prompt generation. These processes resulted in a bias test composed of more than 10,000 prompts which assess whether models demonstrate sexist and homophobic prejudices relevant to the Philippine context. We then apply FilBBQ on models trained in Filipino but do so with a robust evaluation protocol that improves upon the reliability and accuracy of previous BBQ implementations. Specifically, we account for models' response instability by obtaining prompt responses across multiple seeds and averaging the bias scores calculated from these distinctly seeded runs. Our results confirm both the variability of bias scores across different seeds and the presence of sexist and homophobic biases relating to emotion, domesticity, stereotyped queer interests, and polygamy. FilBBQ is available via https://github.com/gamboalance/filbbq.

Abstract

Cultural Bias in Language Models

Explore similar work

Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs

ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues