cs.CLNov 9, 2026

BDBBQ: A Human-AI Collaborative Benchmark for Bengali Social Bias in LLMs

Authors: Provat Saha, Ashika Habib Khan, Tahseen Tanha, Hasin Ishrak, Muhammad Iqbal Hossain

Organizations: Department of Computer Science and Engineering, BRAC University, Dhaka, Bangladesh

Abstract

Despite growing attention to social bias in large language models (LLMs), existing bias benchmarks provide limited coverage of culturally grounded evaluation in Bengali, particularly within Bangladesh's sociocultural context. To address this gap, we develop a bilingual extension of BanglaBBQ comprising 61,312 question-answering instances across 12 social categories in English and Bengali. The benchmark incorporates balanced answer permutations, contrasting question framings, and both ambiguous and disambiguated contexts to systematically examine model performance and stereotype-aligned bias across languages. We validate the dataset through structural checks, 220 statistical tests, and expert annotation of 150 instances by four annotators, achieving strong inter-annotator agreement (Fleiss' \k{appa}=0.8349). We evaluate eight LLMs across 490,496 instances and identify five key findings. First, disambiguated contexts yield substantially greater accuracy improvements in Bengali (+36.59 percentage points) than in English (+17.71). Second, all evaluated models exhibit lower accuracy in Bengali, although Mistral-Nemo shows the smallest cross-lingual accuracy gap (5.12 percentage points). Third, stereotype-aligned bias exhibits language and model dependent variation, increasing in Bengali for Gemma 2 9B but decreasing for Mistral-Nemo. Fourth, higher accuracy does not necessarily correspond to lower stereotype-aligned bias. Finally, bias varies considerably across social categories, with Qwen 2.5 7B exhibiting stronger stereotype alignment in Bengali for Politics and Physical Appearance. These findings demonstrate that cross-lingual performance and social bias are distinct dimensions of LLM behavior and underscore the importance of culturally grounded, language-sensitive benchmarks for evaluating fairness beyond English-centric settings.

Explore similar work

CardsList
  1. BanglaSocialBench: A Benchmark for Evaluating Sociopragmatic and Cultural Alignment of LLMs in Bangladeshi Social Interaction

    Mar 16, 2026Tanvir Ahmed Sijan, S. M Golam Rifat, Pankaj Chowdhury Partha +2Pragmatic Reasoning in Language ModelsCultural Bias in Language Models

  2. BanglaBBQ: A Bias Benchmark for Question Answering in the Bangladeshi Context

    Oct 17, 2026Sumit Anthony Gomes, Yasin Rahman, Taeeba Tasnia +1Cultural Bias in Language ModelsLanguage Model Bias Evaluation

  3. MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs

    Jun 11, 2024Vera Neplenbroek, Arianna Bisazza, Raquel FernándezSocial Bias in Language ModelsLanguage Model Bias Evaluation