Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects

Nurul Labib Sayeedi; Md. Faiyaz Abdullah Sayeedi; Shubhashis Roy Dipta; Mahbub E Sobhani; Rubaya Tabassum; Ariful Ekraj Hridoy; Mehraj Mahmood; Md. Tarek Hasan

cs.CLMar 22, 2026

Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects

Authors: Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Shubhashis Roy Dipta, Mahbub E Sobhani, Rubaya Tabassum, Ariful Ekraj Hridoy, Mehraj Mahmood, Md. Tarek Hasan,

Abstract

Bangla culture is richly expressed through region, dialect, history, food, politics, media, and everyday visual life, yet it remains underrepresented in multimodal evaluation. To address this gap, we introduce BanglaVerse, a culturally grounded benchmark for evaluating multilingual vision-language models (VLMs) on Bengali culture across historically linked languages and regional dialects. Built from 1,152 manually curated images across nine domains, the benchmark supports visual question answering and captioning, and is expanded into four languages and five Bangla dialects, yielding ~32.2K artifacts. Our experiments show that evaluating only standard Bangla overestimates true model capability: performance drops under dialectal variation, especially for caption generation, while historically linked languages such as Hindi and Urdu retain some cultural meaning but remain weaker for structured reasoning. Across domains, the main bottleneck is missing cultural knowledge rather than visual grounding alone, with knowledge-intensive categories. These findings position BanglaVerse as a more realistic test bed for measuring culturally grounded multimodal understanding under linguistic variation.

Figures & tables

Dataset	VQA	CAP.	#ART.	CA	CMA	Multi	DIAL	H. Eval
BengaliVQA ( Islam et al., 2022 )	✓	✗	5,000	✗	✗	✗	✗	✗
Bengali CLEVR ( Hasan et al., 2023 )	✓	✗	12,291	✗	✗	✗	✗	✗
Bangla VQA v2.0 ( Rafi et al., 2022 )	✓	✗	13,046	✗	✗	✗	✗	✗
CVQA ( Romero et al., 2024 )	✓	✗	286	✓	✓	✓	✗	✗
ChitroJera ( Barua et al., 2025 )	✓	✗	15,292	✓	✓	✗	✗	✓
BanglaProtha ( Fahim et al., 2026 )	✓	✗	8,034	✓	✓	✗	✗	✓

Table 1: Comparison with related Bangla and culturally grounded multimodal benchmarks. VQA = Visual Question Answering, CAP. = Captioning, #ART. = Total Number of Artifacts, CA = Cultural Awareness, CMA = Categorical Metadata Availability, Multi = Multilingual, DIAL = Dialectal Coverage, and H. Eval = Human Evaluation.

Figure 1: Overview of the BanglaVerse dataset and experimental setup. The figure shows the two task types, example annotations for each task, artifacts generation and evaluation pipeline with multiple metrics.

Category	Type	Count	Category	Type	Count	Category	Type	Count
Culture	Images	114	Media & Movies	Images	150	Personalities	Images	150
	Captions	1,026		Captions	1,350		Captions	1,350
	VQA	2,052		VQA	2,592		VQA	2,700
	Total	3,192		Total	4,092		Total	4,200
Food	Images	150	National Achievements	Images	75	Politics	Images	150
	Captions	1,350		Captions	675		Captions	1,350

Table 2: Overall statistics of BanglaVerse across 4 languages and 5 Bangla dialects. Caption and VQA counts are expanded across the 9 linguistic varieties; each domain total is the sum of its images, captions, and VQA items.

Figure 2: Geographical distribution of evaluated dialects and the historically linked languages.

Figure 3: Dialect variation across models. We compare each model’s average VQA accuracy (zero-shot, few-shot, and CoT) and caption quality on standard Bangla against its mean performance across the five regional dialects.

Model	Bangla		English (vs. BN)		Hindi (vs. BN)		Urdu (vs. BN)
Model	VQA	Cap.	VQA	Cap.	VQA	Cap.	VQA	Cap.
Gemma3-4B	50.6	52.3	51.6 ( +1.0 )	60.2 ( +7.9 )	50.5 ( -0.1 )	53.2 ( +0.9 )	46.2 ( -4.4 )	47.2 ( -5.1 )
Std. Dev.	$\pm$ 14.77	$\pm$ 3.80	$\pm$ 11.85	$\pm$ 4.24	$\pm$ 12.26	$\pm$ 3.55	$\pm$ 10.60	$\pm$ 1.76
Gemma3-12B	60.0	53.8	59.3 ( -0.7 )	58.6 ( +4.8 )	53.0 ( -7.0 )	54.0 ( +0.2 )	47.7 ( -12.3 )	53.0 ( -0.8 )
Std. Dev.	$\pm$ 14.46	$\pm$ 3.15	$\pm$ 10.65	$\pm$ 4.23	$\pm$ 10.85	$\pm$ 2.98	$\pm$ 11.30	$\pm$ 2.16
Gemma3-27B	64.8	53.9	66.2 ( +1.4 )	61.3 ( +7.4 )	60.7 ( -4.1 )	55.8 ( +1.9 )	54.6 ( -10.2 )	56.2 ( +2.3 )

Table 3: Cross-lingual preservation of Bengali cultural meaning across models. Values in parentheses denote the delta (increase in green , degradation in red ) compared to the Standard Bangla baseline. VQA is measured in average Accuracy (%) over each language, and Caption Quality (Cap.) is measured via LLM-as-a-Judge (

\times 100

R	Domain	ZS	FS	CoT	Cap.	$f(d)$	Type
1	Food	76.56	69.03	77.31	47.54	75.15	$\circ$
2	Cult.	63.42	59.01	65.48	49.23	85.29	$\triangle$
3	Nat. Achv.	55.35	50.86	62.33	49.78	87.89	$\blacktriangle$
4	Hist.	61.07	56.74	63.59	48.35	88.06	$\triangle$
5	Pers.	58.67	57.42	62.28	43.45	94.27	$\blacktriangle$
6	Sports	51.91	47.44	57.21	48.26	94.53	$\blacktriangle$

Table 4: Estimating domain-level bottlenecks. Domains are ranked from easier to harder using the

f(d)

. Based on how domains need to be answered, we labeled them into three buckets:

\circ

= visually concrete,

\triangle

= mixed,

\blacktriangle

= knowledge-intensive. R = Rank, ZS = Zero-shot VQA, FS = Few-shot VQA, CoT = CoT VQA, Cap. = Caption Quality,

f(d)

= Difficulty Score.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Source Website URL / Book Category	License / Usage Rights
Online Sources
https://allbanglanewspapersbd.com/	Copyrighted (Academic Fair Use)
https://www.wikipedia.org/	CC BY-SA 4.0
https://en.banglapedia.org/	Copyrighted (Educational Fair Use)
https://www.goodreads.com/shelf/show/history-bangladesh	Copyrighted (Academic Fair Use)
https://www.rokomari.com/book/category/1263/general-knowledge	Copyrighted (Academic Fair Use)

Appendix

Table 5: List of source websites and specific books utilized for BanglaVerse data collection, along with their respective licensing or fair use designations.

Domain	Assigned Annotators	Cohen’s $\kappa$	Sample Size ( $n$ )
Culture	Annotator 1 & Annotator 2	0.86	114
Food	Annotator 2 & Annotator 3	0.91	150
History	Annotator 3 & Annotator 4	0.85	150
Media & Movies	Annotator 4 & Annotator 1	0.84	150
National Achievements	Annotator 1 & Annotator 3	0.88	75
Nature	Annotator 2 & Annotator 4	0.89	150

Appendix

Table 6: Domain-wise Inter-Annotator Agreement (Cohen’s

\kappa

)

Hyperparameter	Value
Base Architecture	Qwen2.5-3B-Instruct
Task Type	Causal LM
Training Epochs	3
Max Sequence Length	512
Per-Device Batch Size	1
Gradient Accumulation Steps	8

Appendix

Table 7: Hyperparameters and configurations used for fine-tuning the dialect generation model.

Dialect	N	BLEU	ChrF	WER $\downarrow$	BS-L3Cube F1	Gemini Em. Sim.
Barishal	248	39.85	63.91	48.15	0.832	0.978
Chittagong	248	22.14	43.12	67.85	0.715	0.964
Rangpur	247	41.25	68.42	42.18	0.871	0.986
Noakhali	247	24.12	49.88	59.10	0.738	0.965
Sylhet	248	23.45	47.53	61.76	0.779	0.971
Average	–	30.16	54.57	55.81	0.787	0.973

Appendix

Table 8: Intrinsic Evaluation of the Fine-Tuned Dialect Generation Model on the BanglaDial Test Split. WER indicates Word Error Rate (lower is better). Note: BS-L3Cube F1 uses the L3Cube Bengali sentence-similarity SBERT model. Gemini Em. Sim. score is calculated using the gemini-embedding-001 model.

Score	Rating	Description
0	Inaccurate	The translation alters the core meaning of the original Bangla, hallucinates information, or fails to capture the target dialect’s vocabulary and grammar.
1	Acceptable	The core meaning is preserved, but the phrasing is slightly unnatural, or a cultural entity was translated too literally, requiring minor manual correction.
2	Flawless	The translation is natural, grammatically correct, culturally accurate, and (for dialects) reads as a native speaker would converse.

Appendix

Table 9: The 3-point grading scale used by human annotators to evaluate the generated artifacts. Score 1 denotes output acceptable with minor revisions.

Score Rating	Count	Percentage
2 (Flawless)	750	92.59%
1 (Acceptable)	45	5.56%
0 (Inaccurate)	15	1.85%
Total Evaluated Instances	810	100%

Appendix

Table 10: Overall distribution of human evaluation scores across 810 randomly sampled artifacts.

Cultural Domain	Total Samples	Score 2	Score 1	Score 0	% Flawless
Culture	90	83	5	2	92.2%
Food	90	85	4	1	94.4%
History	90	82	6	2	91.1%
Media & Movies	90	80	7	3	88.9%
National Achievements	90	84	5	1	93.3%
Nature	90	86	3	1	95.6%

Appendix

Table 11: Domain-wise breakdown of human evaluation scores. Each domain contains 10 samples evaluated across all 9 linguistic variants.

Figure 4: Impact of Prompting Strategies on Culturally Grounded Tasks. A unified comparison of Zero-Shot, Few-Shot, and Chain-of-Thought (CoT) prompting. (Top Row): (a) Overall performance across all 9 linguistic variants and domains by evaluated models. CoT consistently enhances VQA (b) while Few-Shot degrades it, though Few-Shot slightly improves stylistic caption generation. (Bottom Row): (c) Performance specifically isolated across the five regional Bangla dialects. Dialects universally mirror the overall trend, with CoT enhancing discriminative tasks (d) and Few-Shot aiding in generating dialectal captions.

Language / Dialect	Average Score	Average Score	Pearson	Spearman Rank
	(Gemini)	(GPT-5)	Correlation ( $r$ )	Correlation ( $\rho$ )
English	0.572	0.542	0.597	0.598
Bangla	0.504	0.505	0.374	0.353
Hindi	0.513	0.514	0.820	0.829
Urdu	0.494	0.484	0.844	0.875
Barishal	0.435	0.432	0.745	0.729

Appendix

Table 12: Cross-model validation comparing Gemini-2.5-Flash and GPT-5-mini across all nine languages.

Model	Domain	Accuracy (%)	Variance	Std. Error	95% CI
Gemma3-4B	Nat. Achievements	30.77	0.001439	$\pm$ 3.79%	[23.3%, 38.2%]
Gemma3-4B	Sports	43.55	0.001967	$\pm$ 4.43%	[34.9%, 52.2%]
Gemma3-12B	Nat. Achievements	49.23	0.001689	$\pm$ 4.11%	[41.2%, 57.3%]
Gemma3-12B	Sports	53.23	0.001992	$\pm$ 4.46%	[44.5%, 62.0%]
Gemma3-27B	Nat. Achievements	50.77	0.001689	$\pm$ 4.11%	[42.7%, 58.8%]
Gemma3-27B	Sports	69.35	0.001700	$\pm$ 4.12%	[61.3%, 77.4%]

Appendix

Table 13: Variance, Standard Error, and 95% Confidence Intervals for VQA Accuracy in the smallest domains ( National Achievements and Sports ) across all evaluated models in Standard Bangla.

Error Category	Description	Bangla (n=100)	English (n=100)	Total (n=200)	% of Failures	Most Affected Domains
Entity & Artifact Misrecognition	Hallucinating unrelated or foreign entities instead of local items.	18	22	40	42.6%	Food, Media & Movies
Surface-Level Descriptive Fallback	Providing literal visual descriptions devoid of deeper cultural context.	14	15	29	30.8%	Personalities, History
Societal & Cultural Bias	Exhibiting political skew or visual stereotyping (e.g., gender).	8	6	14	14.9%	Politics, Personalities
Language Constraint Violation	Generating English text despite strict source-language prompts.	10	1	11	11.7%	Sports, Nat. Achievements

Appendix

Table 14: Systematic breakdown of 94 notable errors identified within a 200-sample manual evaluation split evenly between Standard Bangla and English. Overall error rate across the sample is 47.0% (20.0% entity misrecognition, 14.5% descriptive fallback, 7.0% societal bias, 5.5% constraint violation).

		Zero-Shot				Few-Shot				CoT
Models	Domain	Caption			VQA	Caption			VQA	VQA
		B-F1	Gemini	GPT	Accuracy (%)	B-F1	Gemini	GPT	Accuracy (%)	Accuracy (%)
Gemma3-4B	Cult.	0.69	0.53	0.56	0.63	0.69	0.52	0.48	0.59	0.52
	Food	0.70	0.52	0.41	0.74	0.70	0.53	0.50	0.63	0.78
	Hist.	0.69	0.54	0.63	0.47	0.68	0.47	0.36	0.37	0.44
	M&M	0.66	0.50	0.50	0.30	0.84	0.52	0.57	0.32	0.38

Appendix

Table 15: Full results by model and domain for Bangla Language.

		Zero-Shot				Few-Shot				CoT
Models	Domain	Caption			VQA	Caption			VQA	VQA
		B-F1	Gemini	GPT	Accuracy (%)	B-F1	Gemini	GPT	Accuracy (%)	Accuracy (%)
Gemma3-4B	Cult.	0.87	0.65	0.57	0.72	0.86	0.56	0.54	0.58	0.50
	Food	0.85	0.61	0.54	0.75	0.86	0.54	0.47	0.70	0.68
	Hist.	0.86	0.61	0.42	0.43	0.84	0.44	0.43	0.40	0.44
	M&M	0.85	0.55	0.58	0.26	0.92	0.55	0.54	0.34	0.42

Appendix

Table 16: Full results by model and domain for English Language.

		Zero-Shot				Few-Shot				CoT
Models	Domain	Caption			VQA	Caption			VQA	VQA
		B-F1	Gemini	GPT	Accuracy (%)	B-F1	Gemini	GPT	Accuracy (%)	Accuracy (%)
Gemma3-4B	Cult.	0.71	0.57	0.47	0.72	0.69	0.52	0.51	0.53	0.58
	Food	0.71	0.53	0.57	0.70	0.71	0.52	0.48	0.57	0.65
	Hist.	0.70	0.55	0.57	0.52	0.69	0.47	0.44	0.32	0.52
	M&M	0.67	0.49	0.49	0.26	0.84	0.52	0.50	0.25	0.32

Appendix

Table 17: Full results by model and domain for Hindi Language.

		Zero-Shot				Few-Shot				CoT
Models	Domain	Caption			VQA	Caption			VQA	VQA
		B-F1	Gemini	GPT	Accuracy (%)	B-F1	Gemini	GPT	Accuracy (%)	Accuracy (%)
Gemma3-4B	Cult.	0.70	0.48	0.41	0.57	0.69	0.48	0.50	0.34	0.48
	Food	0.70	0.46	0.32	0.63	0.70	0.47	0.50	0.52	0.56
	Hist.	0.69	0.49	0.37	0.51	0.70	0.44	0.43	0.37	0.45
	M&M	0.69	0.45	0.42	0.19	0.85	0.52	0.52	0.24	0.28

Appendix

Table 18: Full results by model and domain for Urdu Language.

		Zero-Shot				Few-Shot				CoT
Models	Domain	Caption			VQA	Caption			VQA	VQA
		B-F1	Gemini	GPT	Accuracy (%)	B-F1	Gemini	GPT	Accuracy (%)	Accuracy (%)
Gemma3-4B	Cult.	0.68	0.49	0.46	0.61	0.67	0.48	0.52	0.38	0.48
	Food	0.69	0.45	0.51	0.74	0.69	0.53	0.48	0.53	0.75
	Hist.	0.65	0.47	0.43	0.48	0.66	0.43	0.37	0.33	0.48
	M&M	0.64	0.43	0.38	0.29	0.75	0.47	0.48	0.29	0.29

Appendix

Table 19: Full results by model and domain for Barishal Dialect.

		Zero-Shot				Few-Shot				CoT
Models	Domain	Caption			VQA	Caption			VQA	VQA
		B-F1	Gemini	GPT	Accuracy (%)	B-F1	Gemini	GPT	Accuracy (%)	Accuracy (%)
Gemma3-4B	Cult.	0.67	0.45	0.43	0.65	0.68	0.45	0.38	0.42	0.62
	Food	0.69	0.48	0.47	0.75	0.69	0.51	0.49	0.62	0.77
	Hist.	0.65	0.40	0.44	0.47	0.68	0.45	0.39	0.33	0.41
	M&M	0.65	0.42	0.35	0.25	0.78	0.51	0.51	0.25	0.30

Appendix

Table 20: Full results by model and domain for Chittagong Dialect.

		Zero-Shot				Few-Shot				CoT
Models	Domain	Caption			VQA	Caption			VQA	VQA
		B-F1	Gemini	GPT	Accuracy (%)	B-F1	Gemini	GPT	Accuracy (%)	Accuracy (%)
Gemma3-4B	Cult.	0.69	0.48	0.41	0.65	0.68	0.49	0.55	0.49	0.51
	Food	0.69	0.45	0.39	0.74	0.69	0.50	0.49	0.61	0.72
	Hist.	0.66	0.40	0.38	0.46	0.68	0.42	0.34	0.37	0.51
	M&M	0.65	0.41	0.39	0.23	0.79	0.48	0.47	0.28	0.31

Appendix

Table 21: Full results by model and domain for Noakhali Dialect.

		Zero-Shot				Few-Shot				CoT
Models	Domain	Caption			VQA	Caption			VQA	VQA
		B-F1	Gemini	GPT	Accuracy (%)	B-F1	Gemini	GPT	Accuracy (%)	Accuracy (%)
Gemma3-4B	Cult.	0.68	0.49	0.55	0.63	0.68	0.49	0.50	0.50	0.59
	Food	0.70	0.50	0.52	0.73	0.70	0.52	0.56	0.56	0.72
	Hist.	0.67	0.48	0.38	0.44	0.68	0.44	0.40	0.42	0.40
	M&M	0.66	0.44	0.42	0.27	0.80	0.52	0.51	0.29	0.30

Appendix

Table 22: Full results by model and domain for Rangpur Dialect.

		Zero-Shot				Few-Shot				CoT
Models	Domain	Caption			VQA	Caption			VQA	VQA
		B-F1	Gemini	GPT	Accuracy (%)	B-F1	Gemini	GPT	Accuracy (%)	Accuracy (%)
Gemma3-4B	Cult.	0.67	0.47	0.45	0.60	0.68	0.45	0.40	0.45	0.58
	Food	0.70	0.46	0.50	0.72	0.70	0.50	0.48	0.62	0.64
	Hist.	0.66	0.42	0.43	0.44	0.68	0.43	0.33	0.40	0.47
	M&M	0.65	0.42	0.35	0.27	0.80	0.50	0.50	0.35	0.36

Appendix

Table 23: Full results by model and domain for Sylhet Dialect.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 957 papers

Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
Jun 4, 2026Mohammad Mahdi Abootorabi, Omid Ghahroodi, Anas Madkoor +4VLM Evaluation Multilingual VLM Evaluation
Qatar Computing Research Institute (QCRI), Hamad Bin Khalifa University
Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languages
Mar 23, 2026Swastik RMultilingual VLM Evaluation VLM Reasoning
Indian Institute of Information Technology, Raichur
BanglaMemeX: Advancing Cultural Metaphoric Image Interpretation in Bangla with a Multimodal Explainable Dataset
Sep 7, 2026Md. Sadman Sakib, Zisan Mahmud, Md. Fahim Arefin +1Figurative Language Understanding Multilingual VLM Evaluation
University of Dhaka · University of Texas at Dallas

Jun 4, 2026J/K move · Enter open · S save

Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models

Mohammad Mahdi Abootorabi, Omid Ghahroodi, Anas Madkoor, Marzia Nouri+3

Qatar Computing Research Institute (QCRI), Hamad Bin Khalifa University

VLM Evaluation Multilingual VLM Evaluation VLM Reasoning Multimodal Understanding

Open paper AskPDF arXiv

Despite the rapid progress of Vision-Language Models (VLMs), the field lacks benchmarks that rigorously diagnose their true reasoning abilities and chart meaningful progress toward human-like multimodal intelligence. Most existing evaluations focus on piecemeal or disconnected tasks, obscuring critical cognitive weaknesses and providing little insight for targeted improvement. To address this gap, we introduce BloomBench, part of the Almieyar benchmarking series, the first cognitively human-grounded, bilingual (English-Arabic) multimodal benchmark for VLMs. Grounded in Bloom's Taxonomy, BloomBench systematically evaluates six levels of cognition (Remember, Understand, Apply, Analyze, Evaluate, Create) through carefully designed image-question-answer tasks. Built with a semi-automated pipeline and validated through a stratified hybrid quality assurance protocol, it ensures scalability, cultural inclusivity, and linguistic fidelity. Leveraging this framework, we conduct a comprehensive study of state-of-the-art VLMs to diagnose their cognitive profiles. Our analysis reveals a sharp cognitive asymmetry: while state-of-the-art models achieve strong performance ceilings in semantic understanding, they struggle substantially with factual recall and creative synthesis. This demonstrates that current general multimodal proficiency masks deeper limitations in specific cognitive layers. Furthermore, our study highlights a critical performance gap between Arabic and English, exposing limitations in current cross-lingual multimodal reasoning. These findings establish a foundation for developing more cognitively aligned and inclusive VLMs. The benchmark framework and dataset is available at: https://github.com/qcri/Almieyar-Oryx-BloomBench.

Abstract

VLM Evaluation

Explore similar work

Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models

Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languages

BanglaMemeX: Advancing Cultural Metaphoric Image Interpretation in Bangla with a Multimodal Explainable Dataset