Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

Lance Calvin Lim Gamboa; Yue Feng; Mark Lee

cs.CLFeb 16, 2026

Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

Authors: Lance Calvin Lim Gamboa, Yue Feng, Mark Lee

Organizations: School of Computer Science, University of Birmingham · Department of Information Systems and Computer Science, Ateneo de Manila University

Abstract

With natural language generation becoming a popular use case for language models, the Bias Benchmark for Question-Answering (BBQ) has grown to be an important benchmark format for evaluating stereotypical associations exhibited by generative models. We expand the linguistic scope of BBQ and construct FilBBQ through a four-phase development process consisting of template categorization, culturally aware translation, new template construction, and prompt generation. These processes resulted in a bias test composed of more than 10,000 prompts which assess whether models demonstrate sexist and homophobic prejudices relevant to the Philippine context. We then apply FilBBQ on models trained in Filipino but do so with a robust evaluation protocol that improves upon the reliability and accuracy of previous BBQ implementations. Specifically, we account for models' response instability by obtaining prompt responses across multiple seeds and averaging the bias scores calculated from these distinctly seeded runs. Our results confirm both the variability of bias scores across different seeds and the presence of sexist and homophobic biases relating to emotion, domesticity, stereotyped queer interests, and polygamy. FilBBQ is available via https://github.com/gamboalance/filbbq.

Figures & tables

Dimension	gender	sexual orientation
Ambiguous Context	Naghatian ng trabaho kanina sina {NAME1} at {NAME2} sa pagpupulong nila. In their meeting earlier, { NAME1 } and { NAME2 } divvied up their responsibilities.	Nasa parehong mesa ang {NAME1} at {NAME2} sa reception kanina. The { NAME1 } and the { NAME2 } were at the same table during the reception dinner a while ago.
Disambiguated Context	Nagprisintang maging lider si {NAME2} , samantala handa namang makinig at sumunod sa kanya si {NAME1} . { NAME2 } volunteered to be leader, while { NAME1 } was ready to listen and follow.	Tahimik lang na pangiti-ngiti ang {NAME1} , samantala grabe naman ang lakas ng halakhak ng {NAME2} kasama ang mga kaibigan niya. The { NAME1 } was just quietly smiling at others, while the { NAME2 } laughed out loud with friends.
Negative Question	Sino ang masunuring tagagawa? Who is the obedient follower?	Sino ang maingay? Who is loud?
Non-negative Question	Sino ang dominanteng pinuno? Who is the dominant leader.	Sino ang tahimik? Who is quiet?
Choices	Si 2 2 footnotemark: 2 {NAME1} , Si {NAME2} , unknown option	Ang 3 3 3 Si is a subject marker for proper nouns in Filipino. {NAME1} , Ang {NAME2} , unknown option
Stereotype	Women are obedient. Men are dominant.	Gay people are loud.

Table 1: Sample prompt from the sexual orientation subset of the original BBQ.

Bias Dimension	Translated	Modified	Created	Total	Prompts
	Templates
gender	34	11	32	77	7952
sexual orientation	19	7	20	46	2624
TOTAL	53	18	52	123	10576

Table 2: FilBBQ statistics.

Figure 1: Jitter plot showing variable bias scores across differently seeded runs. The plot’s points reflect scores for the FilBBQ prompt on the “Women are emotional” stereotype (ambiguous context version).

Figure 2: Jitter plot showing variable bias scores across differently seeded runs. The plot’s points reflect scores for the FilBBQ prompt on the “Gay people like fashion, design, and gossip” stereotype (ambiguous context version).

Context	Dimension	Stereotype 5 5 5 Statements under the Stereotype column are author-written characterizations of stereotypes present in the most bias-inducing prompts.	Bias Score
ambiguous	gender	Women are more emotional than men.	$0.57$
ambiguous	gender	Men are more emotionally closed than women and lack empathy.	$0.42$
ambiguous	sexual orientation	Gay people like fashion, design, and gossip.	$0.31$
ambiguous	gender	Women are homemakers. Men are economic providers.	$0.29$
ambiguous	sexual orientation	Gay people like clothes and fashion.	$0.26$
disambiguated	gender	Women are nurses. Men are doctors.	$0.78$

Table 3: Llama-SEA-LION-v2-8B-IT bias profile. This table shows the five strongest biases in the model when tested in the ambiguous and disambiguated contexts respectively.

Context	Dimension	Stereotype	Bias Score
ambiguous	gender	Women are more emotional than men.	$0.22$
ambiguous	gender	Women do gymnastics. Men do weightlifting.	$0.16$
ambiguous	gender	Women are homemakers. Men are economic providers.	$0.09$
ambiguous	gender	Women are family-focused. Men are career-focused.	$0.09$
ambiguous	gender	Women are more affectionate than men.	$0.09$
disambiguated	gender	Women cook. Men carry heavy equipment.	$0.38$

Table 4: SeaLLMs-v3-7B-Chat bias profile. This table shows the five strongest biases in the model when tested in the ambiguous and disambiguated contexts respectively.

Context	Dimension	Stereotype	Bias Score
ambiguous	gender	Women are more emotional than men.	$1.00$
ambiguous	gender	Women are trusting and gullible. Men are shrewd and clever.	$0.25$
ambiguous	sexual orientation	Non-heterosexual individuals are afraid of commitment.	$0.25$
ambiguous	sexual orientation	Non-heterosexual individuals are pedophiles.	$0.16$
ambiguous	sexual orientation	Gay people are vain.	$0.13$
disambiguated	gender	Women do gymnastics. Men do weightlifting.	$1.00$

Table 5: roberta-tagalog-base bias bias profile. This table shows the five strongest biases in the model when tested in the ambiguous and disambiguated contexts respectively.

Social Bias in Language Models

Social bias in language models is the study and mitigation of unfair stereotypes and disparities in generated text, judgments, and decisions.

See all 192 papers

ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues
Apr 2, 2026Bhaskara Hanuma Vedula, Darshan Anghan, Ishita Goyal +2Social Bias in Language Models Language Model Bias Evaluation
International Institute of Information Technology, Hyderabad · Indian Institute of Technology, Kharagpur
BanglaBBQ: A Bias Benchmark for Question Answering in the Bangladeshi Context
Oct 17, 2026Sumit Anthony Gomes, Yasin Rahman, Taeeba Tasnia +1Cultural Bias in Language Models Language Model Bias Evaluation
BRAC University, Dhaka, Bangladesh
MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs
Jun 11, 2024Vera Neplenbroek, Arianna Bisazza, Raquel FernándezSocial Bias in Language Models Language Model Bias Evaluation
Institute for Logic, Language and Computation, University of Amsterdam · Center for Language and Cognition, University of Groningen

Apr 2, 2026J/K move · Enter open · S save

ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues

Bhaskara Hanuma Vedula, Darshan Anghan, Ishita Goyal, Ponnurangam Kumaraguru+1

International Institute of Information Technology, Hyderabad · Indian Institute of Technology, Kharagpur

Social Bias in Language Models Language Model Bias Evaluation Demographic Bias in Language Models

Open paper AskPDF arXiv

Large Language Models increasingly suppress biased outputs when demographic identity is stated explicitly, yet may still exhibit implicit biases when identity is conveyed indirectly. Existing benchmarks use name based proxies to detect implicit biases, which carry weak associations with many social demographics and cannot extend to dimensions like age or socioeconomic status. We introduce ImplicitBBQ, a QA benchmark that evaluates implicit bias through characteristic based cues, demographically associated attributes that signal implicitly, across age, gender, region, religion, caste, and socioeconomic status. Evaluating 11 models, we find that implicit bias in ambiguous contexts is over six times higher than explicit bias in open weight models. Notably, this bias is distributed unevenly across demographics: caste emerges as the most severe while gender is the least affected. Safety prompting and chain-of-thought reasoning fail to substantially close this gap; even few-shot prompting, which reduces implicit bias by 79%, leaves caste bias at four times the level of any other dimension. These findings indicate that current alignment and prompting strategies address the surface of bias evaluation while leaving demographically associated stereotypic associations largely unresolved. We publicly release our code and dataset for model providers and researchers to benchmark potential mitigation techniques.

Abstract

Social Bias in Language Models

Explore similar work

ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues

BanglaBBQ: A Bias Benchmark for Question Answering in the Bangladeshi Context

MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs