Benchmarking Bengali Dialectal Bias: A Multi-Stage Framework Integrating RAG-Based Translation and Human-Augmented RLAIF

K. M. Jubair Sami; Dipto Sumit; Ariyan Hossain; Farig Sadeque

cs.CLMar 22, 2026

Benchmarking Bengali Dialectal Bias: A Multi-Stage Framework Integrating RAG-Based Translation and Human-Augmented RLAIF

Authors: K. M. Jubair Sami, Dipto Sumit, Ariyan Hossain, Farig Sadeque

Organizations: Department of Computer Science and Engineering BRAC University, Dhaka, Bangladesh

Abstract

Large language models (LLMs) frequently exhibit performance biases against regional dialects of low-resource languages. However, frameworks to quantify these disparities remain scarce. We propose a two-phase framework to evaluate dialectal bias, operationalized as comprehension degradation relative to standard Bengali, in LLM question-answering across nine Bengali dialects. First, we translate and gold-label standard Bengali questions into dialectal variants adopting a retrieval-augmented generation (RAG) pipeline to prepare 4,000 question sets. Since traditional translation quality evaluation metrics fail on unstandardized dialects, we evaluate fidelity using an LLM-as-a-judge, which human correlation confirms outperforms legacy metrics. Second, we benchmark 19 LLMs across these gold-labeled sets, running 68,395 RLAIF evaluations validated through multi-judge agreement and human fallback. Our findings reveal severe performance drops linked to linguistic divergence. For instance, responses to the highly divergent Chittagong dialect score 5.44/10, compared to 7.68/10 for Tangail. Furthermore, increased model scale does not consistently mitigate this bias. We contribute a validated translation quality evaluation method, a rigorous benchmark dataset, and a Critical Bias Sensitivity (CBS) metric for safety-critical applications.

Figures & tables

Figure 1: Overview of the dialectal bias measurement framework. The pipeline translates standard Bengali questions into dialectal variants via Retrieval-Augmented Generation, which are then used to probe LLMs with RLAIF-based scoring.

Metric (Weight)	Evaluation Statement
1. Dialect Comprehension (3.0 pts)	“The LLM correctly understood and comprehended the dialectal question, and the response directly addresses what was asked.”
2. Factual Correctness (2.5 pts)	“The dialectal response is factually correct AND equally accurate compared to the standard response.”
3. Content Completeness (2.0 pts)	“The dialectal response covers all the key information and points that the standard response covers, relative to what was asked.”
4. Response Clarity (1.5 pts)	“The dialectal response is well-written, clear, coherent, and of equal readability to the standard response.”
5. Appropriate Length (1.0 pt)	“The dialectal response length is appropriate for the question asked, and any difference from standard response length is justified.”

Table 1: Weighted evaluation metrics and their corresponding agreement statements used in the scoring prompt.

Dialect	N	BLEU	ChrF	WER $\downarrow$	BS-L3Cube F1	Gemini Em. Sim.	Gemini Em. BS F1	Gemma-3	GPT-OSS
Barishal	248	40.54	64.28	47.72	0.838	0.980	0.975	8.80	8.52
Chittagong	248	21.33	42.51	68.94	0.707	0.961	0.954	7.99	7.10
Mymensingh	247	40.80	67.99	43.06	0.869	0.984	0.977	8.84	9.00
Noakhali	247	24.77	50.74	58.38	0.744	0.967	0.960	8.17	7.89
Sylhet	248	22.91	46.99	62.43	0.772	0.969	0.959	8.02	7.96
Avg	1,238	30.07	54.50	56.11	0.786	0.972	0.965	8.36	8.09

Table 2: Comprehensive translation quality evaluation for the RAG pipeline with Gemma-3-27B-IT on the Vashantor validation split: BLEU/ChrF/WER (0–100), BERTScore & similarity (0–1), LLM-judge scores (0–10; judges: Gemma-3-27B-IT, GPT-OSS-120B). BS-L3Cube F1 uses the L3Cube Bengali sentence-similarity SBERT model.

Metric	Pearson $r$	Spearman $\rho$	Lin’s CCC
Gemma-3-27B-IT	0.524	0.595	0.506
GPT-OSS-120B	0.455	0.484	0.395
BS-L3Cube F1	0.379	0.420	0.358
Gemini Em. BS-F1	0.455	0.486	0.093
Gemini Em. Sim.	0.417	0.458	0.074
ChrF	0.470	0.485	0.186

Table 3: Row-level correlation between automated metrics and human judge scores for translation quality evaluation (

N=125

, 25 per dialect).

Model	Barishal	Chittagong	Kishoreganj	Mymensingh	Narail	Noakhali	Rangpur	Sylhet	Tangail	Avg
gemma-3-27b-it	8.08	7.80	9.30	9.16	8.38	9.03	8.55	8.85	9.22	8.71
gpt-oss_20b	8.13	8.32	9.14	9.19	8.11	8.60	9.14	8.72	8.99	8.70
qwen3_32b	8.51	7.74	9.01	9.03	8.24	8.47	9.42	8.21	9.35	8.67
llama-3.3-70b	8.30	7.79	8.68	9.06	8.36	8.00	9.24	8.50	9.00	8.55
ministral-3_14b	7.43	7.61	8.70	8.80	8.12	8.15	9.22	8.54	9.09	8.41
qwen-3-235b	8.20	5.60	9.17	8.90	8.20	8.40	8.92	7.92	8.89	8.25

Table 4: Dialectal bias scores (0-10 scale) across 19 LLMs and 9 Bengali dialects. Higher scores indicate better consistency with standard Bengali. Avg column shows macro-average across dialects.

Gemini vs.	CCC	CBS	Pearson	Spearman	Mean Abs
					Bias Diff
GPT-OSS	0.8614	0.7781	0.8629	0.7757	0.8986
Gemma-3	0.7769	0.4558	0.8391	0.7388	1.3482

Table 5: Multi-judge agreement metrics across 19 models evaluations. Mean Abs Bias Diff shows average absolute score deltas between judges.

Explore similar work

5-Dialects-BN: Unmasking the Impact of Transliteration on Bangla Dialectal LLMs
Sep 9, 2026Md Mahir Jawad, Galib Mahmud Jim, Rafid Ahmed +3Dialects Bangla
BRAC University · Penta Global Limited · University of Central Florida +1
How Robust Are LLMs to Vietnamese Dialects?
Aug 11, 2026Minh Tran, Trinh Chau, Thanh-Nhan Le +4Dialects Natural Language Inference
University of Science, VNU-HCM · VNU University of Engineering and Technology · New York University Abu Dhabi +4
Unified Multi-Dialectal Neural Machine Translation for Bangla Using the Dwadash Benchmark Corpus
Aug 12, 2026Rakib Ullah, Md. Ruhul Islam, Tanbir Ahmed +1Dialects Neural Machine Translation
Department of Computer Science and Engineering SYLHET ENGINEERING COLLEGE, SYLHET Sylhet, Bangladesh

Sep 9, 2026, cs.CLJ/K move · Enter open · S save

5-Dialects-BN: Unmasking the Impact of Transliteration on Bangla Dialectal LLMs

Md Mahir Jawad, Galib Mahmud Jim, Rafid Ahmed, Mir Sazzat Hossain+2

BRAC University · Penta Global Limited · University of Central Florida+1

Dialects Bangla Low-Resource Languages Multilingual Benchmark Natural Language Processing Impact Analysis

Open paper AskPDF arXiv

Large Language Models (LLMs) have achieved remarkable progress across natural language processing (NLP) tasks, yet their capabilities degrade sharply for low-resource languages and dialectally diverse settings. Bangla, the world's sixth most spoken language, exemplifies this gap: existing resources overwhelmingly target Standard Bangla, leaving its regional dialects without the benchmarks needed to develop or evaluate dialect-aware systems. We address this gap with 5-Dialects-BN, the first multi-annotation Bangla dialect benchmark to align Romanized transliteration with dialectal text, Standard Bangla, English, and subjectivity labels across five regional varieties. The dataset comprises 6,000 manually annotated entries spanning five major dialects: Chittagong, Barisal, Noakhali, Sylhet, and Rangpur (Chittagong 1,900; Noakhali 1,500; Sylhet 1,200; Barisal 700; Rangpur 700), reflecting natural online availability. Each entry is enriched with five aligned annotations: the original dialectal text, a Romanized transliteration, an English translation, a Standard Bangla translation, and a subjectivity label (subjective vs. objective). Annotations were produced and cross-validated by native speakers and undergraduate linguistics students to ensure dialectal authenticity and semantic fidelity. The resulting resource supports a diverse suite of tasks, including dialect identification, dialect-to-standard normalization, machine translation, subjectivity classification, and parameter-efficient fine-tuning (e.g., LoRA) of multilingual LLMs. By providing a standardized, multi-annotation benchmark, 5-Dialects-BN enables principled evaluation of LLMs on dialectally diverse Bangla and lays a foundation for further research in low-resource, dialect-aware NLP.

Abstract

Explore similar work

5-Dialects-BN: Unmasking the Impact of Transliteration on Bangla Dialectal LLMs

How Robust Are LLMs to Vietnamese Dialects?

Unified Multi-Dialectal Neural Machine Translation for Bangla Using the Dwadash Benchmark Corpus