Do Language Models Understand Igbo Culture? Beyond Multiple Choice

Blessing Ngozi Uzor; Valdini Douglace Lemofouet; Paula Chikaodinaka Anyanwu; Mmesoma Peace Orji; Danielle Blanche Kapsa; Tassallah Abdullahi; Clemencia Siro; Idris Abdulmumin

cs.CLOct 26, 2026

Do Language Models Understand Igbo Culture? Beyond Multiple Choice

Authors: Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu, Mmesoma Peace Orji, Danielle Blanche Kapsa, Tassallah Abdullahi, Clemencia Siro, Idris Abdulmumin,

Organizations: African Institute for Mathematical Sciences (AIMS), Cameroon · University of Nigeria, Nsukka · Brown University · Centrum Wiskunde & Informatica ·

Abstract

The value of language models for any community depends on whether a model actually understands the culture of the people it intends to serve. We use the Igbo language as a case study. Existing benchmarks test translated Western ideas, lacking depth. To address this, we introduce CAB-Igbo, a benchmark developed from indigenous cultural sources and verified by native speakers across six domains. We evaluated diverse models under varying conditions, incorporating regional personas. Performance was assessed using multiple-choice accuracy and human evaluations across Epistemic Fidelity, Representational Richness, Linguistic Quality, and Pragmatic Proficiency. Results reveal a stark dissociation between language fluency and cultural knowledge. Through statistical testing, we show that for most models, the severe lack of Epistemic Fidelity and Representational Richness is already present in English and does not significantly worsen when switching to Igbo. Instead, native language prompting mainly degrades Linguistic Quality and Pragmatic Proficiency. Gemini 2.0 Flash is the only model whose cultural knowledge scores improve under Igbo prompting. We find that current models struggle to demonstrate the knowledge needed for true alignment with Igbo, and multiple-choice formats allow pattern-matching to appear as understanding. Furthermore, automated judges agree with humans on linguistic quality but fail on epistemic fidelity. Ultimately, cultural evaluation must be built from community knowledge, establishing human evaluation as the gold standard to prevent misrepresenting low-resource communities globally.

Figures & tables

CULTURAL DOMAIN	ENGLISH PROMPT	IGBO PROMPT
Proverbs and Oral Tradition	What practical life lesson is contained in the proverb, "It is the height that a child's hand gets to that he hangs his bag"?	Kedu nkuzi ndụ bara uru dị n'ime ilu a: "Ebe eka nwata ru ke o kọnto ekpa"?
Family and Kinship	In Igbo kinship and hospitality rituals, how is an Ọjị Igbo (kola nut) that breaks into exactly five cotyledons interpreted by the family?	N'omenala mmekọrịta ezinụlọ na ile ọbịa ndị Igbo, kedu ka ezinụlọ si akọwa Ọjị Igbo gbara kpọmkwem ibe ise?
Funerals and Burial	In traditional Igbo cosmology, which of the following criteria defines the attainment of a "good death" (Ọnwụ chi)?	N'ọdịnala na amamihe okike ndị Igbo, olee n'ime ọnọdụ ndị a na-akọwa inweta "ezi ọnwụ" (Ọnwụ chi)?

Figure 1: Samples from the CAB-Igbo Dataset. Questions are grounded in authentic cultural domains and evaluate deep epistemic fidelity rather than mere linguistic translation.

Figure 2: Prompt Degradation: The collapse of cultural reasoning across English and Igbo prompt types.

Figure 3: The Limits of Automated Evaluation: Comparing human evaluator scores against the Gemini 2.0 automated judge. The automated judge significantly overestimates Epistemic Fidelity and Representational Richness compared to native human experts.

Figure 4: Aggregate Human Evaluation Scores. Gemini 2.0 Flash significantly outperforms all other models, while smaller open-weight models suffer near-complete breakdown in culturally grounded contexts.

Figure 5: The gap between recognising and explaining Igbo culture. For each model, on the same 59 questions in the zero-shot condition across all three prompt types (177 items per model; 176 open-ended answers for GPT-OSS 20B): the share of MCQ items answered correctly (original ordering) and the share of open-ended answers rated fully correct by the human annotator (Epistemic Fidelity

=2

Figure 6: Primary human verdict on each of the 1,415 scored open-ended answers, by prompt type. The verdict is the first error label the annotator assigned; Correct covers answers labelled Culturally Accurate or with no error recorded. Other covers Western Default, Repetition, and Incomplete Response as primary labels.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Type 1	Type 2	Type 3
Gemini 2.0 Flash	99.2	97.4	98.1
GPT-4o mini	98.2	90.2	91.1
AfriqueQwen 14B	99.5	90.7	74.4
LLaMA 3.3 70B	99.0	91.6	46.1
LLaMA 3 8B Lite	96.8	53.7	64.8
LLaMA 3.1 8B	96.3	60.0	51.5

Appendix

Table 1: MCQ accuracy (%) by linguistic prompt type, pooled over the baseline and all persona conditions.

Model	Baseline	Imo	Ebonyi
Gemini 2.0 Flash	98.2	98.2	98.3
GPT-4o mini	91.7	93.5	93.2
AfriqueQwen 14B	93.2	87.7	87.8
LLaMA 3.3 70B	81.5	78.7	78.5
LLaMA 3 8B Lite	71.6	71.6	71.9
LLaMA 3.1 8B	59.8	70.1	70.3

Appendix

Table 2: MCQ accuracy (%) for the zero-shot baseline and for the Imo and Ebonyi conditions, each pooled over the five personas.

Model	Baseline	All personas	Delta
Gemini 2.0 Flash	98.2	98.3	$+$ 0.1
GPT-4o mini	91.7	93.3	$+$ 1.6
AfriqueQwen 14B	93.2	87.7	$-$ 5.5
LLaMA 3.3 70B	81.5	78.6	$-$ 2.9
LLaMA 3 8B Lite	71.6	71.8	$+$ 0.2
LLaMA 3.1 8B	59.8	70.2	$+$ 10.4

Appendix

Table 3: MCQ accuracy (%) under the zero-shot baseline and pooled over all ten persona conditions; Delta is in percentage points.

Model	BL	YTB	MW	MM	WW	Prof
Gemini 2.0 Flash	98.2	98.3	98.5	98.1	98.1	98.3
GPT-4o mini	91.7	93.3	92.8	93.2	93.6	93.6
AfriqueQwen 14B	93.2	88.5	87.5	87.7	87.8	87.1
LLaMA 3.3 70B	81.5	80.0	78.6	78.6	77.4	78.4
LLaMA 3 8B Lite	71.6	71.7	72.6	71.8	71.7	71.1
LLaMA 3.1 8B	59.8	68.5	70.8	71.3	67.8	72.6

Appendix

Table 4: MCQ accuracy (%) by persona, pooled over Imo and Ebonyi. BL = zero-shot baseline; YTB = Young Teenage Boy; MW = Married Woman; MM = Married Man; WW = Widowed Woman; Prof = Professor.

Model	$\chi^{2}$	$p$	$\chi^{2}$	$p$	sig.
	EN $\rightarrow$ IG		Position		Persona
AfriqueQwen 14B	31.03	$<0.001$	0.12	0.731	10/10
Gemini 2.0 Flash	5.14	0.023	5.26	0.022	0/10
GPT-4o mini	24.74	$<0.001$	16.60	$<0.001$	7/10
GPT-OSS 120B	25.29	$<0.001$	0.13	0.716	0/10
GPT-OSS 20B	164.28	$<0.001$	0.67	0.413	10/10

Appendix

Table 5: McNemar’s tests (

\chi^{2}

with Yates’ continuity correction) on paired MCQ outcomes for the zero-shot baseline items. EN

\rightarrow

IG: Type 1 vs Type 2, original ordering (296 pairs). Position: original vs shuffled ordering (888 question and prompt type pairs). Persona: number of the ten persona conditions that differ significantly (

p<0.05

) from the baseline (888 pairs each, original ordering).

Model	Type 1	Type 2	Type 3
Gemini 2.0 Flash	3.559	4.441	4.949
GPT-4o mini	3.271	3.169	3.000
GPT-OSS 120B	3.102	3.153	3.017
LLaMA 3.3 70B	3.203	3.119	2.797
AfriqueQwen 14B	3.237	2.983	2.356
GPT-OSS 20B	2.729	1.915	1.741

Appendix

Table 6: Mean total human evaluation score (out of 8) by model and prompt type.

Model	EF	RR	LQ	PP
Gemini 2.0 Flash	0.531	0.531	1.994	1.260
GPT-4o mini	0.062	0.062	2.000	1.023
GPT-OSS 120B	0.074	0.074	1.938	1.006
LLaMA 3.3 70B	0.062	0.062	1.932	0.983
AfriqueQwen 14B	0.102	0.102	1.740	0.915
GPT-OSS 20B	0.040	0.040	1.351	0.698

Appendix

Table 7: Mean human evaluation score per construct (each 0 to 2), averaged over the three prompt types.

Model	$\Delta$ EF	$\Delta$ RR	$\Delta$ LQ	$\Delta$ PP
AfriqueQwen 14B	$-0.03$	$-0.03$	$0.24^{*}$	$0.08$
Gemini 2.0 Flash	$-0.34^{*}$	$-0.34^{*}$	$-0.02$	$-0.19^{*}$
GPT-4o mini	$0.05$	$0.05$	$0.00$	$0.00$
GPT-OSS 120B	$-0.08$	$-0.08$	$0.10$	$0.02$
GPT-OSS 20B	$0.05$	$0.05$	$0.46^{*}$	$0.25^{*}$
LLaMA 3.1 8B	$0.10^{*}$	$0.10^{*}$	$1.36^{*}$	$0.75^{*}$

Appendix

Table 8: Mean within-item score difference (Type 1 minus Type 2) in human evaluation. Positive values are drops under Igbo prompting; negative values are gains. Asterisks mark differences whose bootstrap 95% confidence interval (10,000 resamples) excludes zero. Each model has 59 paired items.

Model	$n$	EF	RR	LQ	PP
GPT-OSS 120B	177	1.113	1.023	1.989	1.955
GPT-4o mini	177	0.672	0.582	2.000	2.000
LLaMA 3.3 70B	177	0.599	0.531	1.938	1.927
AfriqueQwen 14B	177	0.644	0.542	1.751	1.734
GPT-OSS 20B	176	0.619	0.597	1.460	1.449
LLaMA 3.1 8B	177	0.226	0.175	1.011	1.023

Appendix

Table 9: Mean LLM-as-a-judge scores by model, ordered by total, over the responses matched to human scores (59 questions across three prompt types);

n

is the number of matched responses. Empty responses are scored 0, as in the human rubric.

	$\kappa_{w}$	$\rho$	Exact	Adj.	Human	Judge
EF	0.112	0.276	59.8	87.5	0.054	0.574
RR	0.100	0.240	59.9	92.2	0.054	0.510
LQ	0.833	0.885	90.9	97.3	1.494	1.580
PP	0.261	0.883	22.1	98.1	0.767	1.560

Appendix

Table 10: Agreement between the human annotator and the Gemini 2.0 Flash judge over 1,238 matched responses.

\kappa_{w}

= weighted Cohen’s kappa;

\rho

= Spearman rank correlation; Exact and Adj. = percentage of pairs that agree exactly or differ by at most one point; Human and Judge = mean scores.

LLaMA 3 8B Lite Type 2 · question 1 99% of 167 words · 166 repeats Onye agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha agha …

LLaMA 3.1 8B Type 2 · question 1 71% of 86 words · 61 repeats …dị ka Ọnwụma Ọjọọ, Ọnwụma Ọjọọ Ọjọọ, Ọnwụma Ọjọọ Ọjọọ Ọjọọ, Ọnwụma Ọjọọ Ọjọọ Ọjọọ Ọjọọ, Ọnwụma Ọjọọ Ọjọọ Ọjọọ Ọjọọ Ọjọọ, Ọnwụma Ọjọọ Ọjọọ Ọjọọ Ọjọọ Ọjọọ Ọjọọ, Ọnwụma …

GPT-OSS 20B Type 2 · question 5 36% of 77 words · 4 repeats …nwa ọhụrụ nwere àgwà ma ọ bụ àgwà dị iche, dịka ịma mma ma ọ bụ obi ike, nke yiri nke nna nna ahụ. Ọtụtụ mgbe, a na-ahụkwa akara anụ ahụ dịka àgwà ma ọ bụ àgwà dị iche, …

AfriqueQwen 14B Type 3 · question 5 51% of 182 words · 4 repeats …na-akwado nkwupụta a. Ụfọdụ ndị na-akọ akụkọ banyere ihe ndị dị ka ọdịdị anụ ahụ ma ọ bụ àgwà ndị yiri nke nna nna nwụrụ anwụ, ma ihe ndị a nwere ike ịbụ n'ihi ọdịdị mkpụrụ ndụ ihe nketa ma ọ bụ mmetụta ọdịbendị. Ụfọdụ ndị na-akọ akụkọ banyere ihe ndị dị ka ọdịdị anụ ahụ ma ọ bụ àgwà ndị yiri nke nna nna nwụrụ anwụ, …

LLaMA 3.1 8B Type 3 · question 1 41% of 147 words · 3 repeats …ụmụ ọhụrụ: 1. Ọnwụ ụmụ ọhụrụ bụ ihe a na-egosi mgbe ndị nne na nna gụrụ nwa aha ịma aka dị ka Ọnwụma. Aha Ọnwụma bụ aha nke a na-egosi ọnwụ ụmụ ọhụrụ, na-eme ka ndị nne na nna na-achọ ịgụ aha ịma aka dị ka Ọnwụma mgbe ụmụ ha nwụrụ. 2. Ọnwụ ụmụ ọhụrụ bụ ihe a na-egosi mgbe ndị nne na nna gụrụ nwa aha ịma aka dị ka Ọnwụma. …

LLaMA 3.3 70B Type 3 · question 4 29% of 165 words · 3 repeats …a na-egosi na mmadụ niile nwere ebe ha dị iche, ebe ha na-akpata uru, na ebe ha na-aghọọ nwa. Echiche a na-akpata ike site na echiche ọdịnala Igbo banyere ụmụ mmadụ na ebe ha dị na ụwa. Ọ na-akpọ mkpa ịhụ na mmadụ niile nwere ebe ha dị iche, ebe ha na-akpata uru, na ebe ha na-aghọọ nwa. …

Appendix

Figure 7: Repetition loops in open-ended answers under Igbo prompting. Each excerpt is quoted verbatim from a model answer in the human-evaluation sample, with omitted text marked by an ellipsis; the repeated unit is highlighted, and the counts refer to the full stored answer. The annotator labelled all six answers as Model Breakdown and Repetition.

Domain	Questions
Food and Cuisine	73
Festivals and Religion	67
Family and Kinship	58
Proverbs and Oral Tradition	38
Social Customs and Etiquette	35
Funerals and Burial	25

Appendix

Table 11: Number of questions per cultural domain.

Category	Definition	Count
Western Default	Interprets Igbo culture through Western assumptions such as individualism or nuclear family structures	292
Shallow Description	Recognises surface patterns without deeper cultural understanding	256
Ethnolinguistic Substitution	Replaces Igbo practices with Yoruba or Hausa cultural forms	174
Pan-African Generalisation	Uses broad African stereotypes that erase Igbo-specific meaning	166
Total		888

Appendix

Table 12: Distractor categories and their frequency across the 888 distractors (three per question).

Epistemic Fidelity (EF): factual accuracy and cultural reliability
2	Completely accurate and culturally specific
1	Partially accurate or surface-level
0	Completely incorrect, fabricated, or blank
Representational Richness (RR): depth and nuance
2	Rich, nuanced, and culturally grounded
1	Basic or vague cultural detail

Appendix

Table 13: Human evaluation rubric.

Persona	English (Types 1, 2)	Igbo (Type 3)
Young Teenage Boy	young boy; single; secondary school student; the youth	nwata nwoke; na-alụbeghị nwanyị; nwa akwụkwọ sekọndrị; ndị ntorobịa
Married Woman	woman; married; primary school; the local trading class	nwaanyị; lụrụ di; ụlọ akwụkwọ praịmarị; ndị ahịa obodo
Married Man	man; married; basic community; the rural farming class	nwoke; lụrụ nwaanyị; obodo; ndị ọrụ ugbo
Widowed Woman	woman; widowed; vocational; the artisan class	nwaanyị; di ya nwụrụ; aka ọrụ; ndị aka ọrụ
Professor	man; married; post-graduate (PhD); the academic middle class	nwoke; lụrụ nwaanyị; mahadum (PhD); ndị gụrụ akwụkwọ nke ọma

Appendix

Table 14: Persona attributes inserted into the persona introductions.

Cultural Knowledge in Language Models

Cultural knowledge in language models concerns how systems acquire, access, and apply culture-specific facts, norms, meanings, and practices.

See all 132 papers

The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs
Jun 5, 2026Yang Zhang, Xiao Fei, Amr Mohamed +6Multilingual Language Model Evaluation LLM Evaluation
Ecole Polytechnique · MBZUAI · ENS-PSL +1
Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth
Jun 30, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh +2Multilingual Language Model Evaluation LLM-as-a-Judge
1Perle AI
MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
Jul 1, 2026Xianru Chen, Yukai Huang, Mingxiang Chen +6Multilingual Language Model Evaluation LLM Evaluation
ByteDance Seed · Nanjing University · M-A-P

Jun 5, 2026J/K move · Enter open · S save

The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs

Yang Zhang, Xiao Fei, Amr Mohamed, Sarah Almeida Carneiro+5

Ecole Polytechnique · MBZUAI · ENS-PSL+1

Multilingual Language Model Evaluation LLM Evaluation Cross-Cultural Language Model Evaluation Cultural Knowledge in Language Models

Open paper AskPDF arXiv

Large language models are increasingly used to answer culturally grounded questions across languages, yet it remains unclear whether local cultural knowledge is better accessed through English or the local language. Existing evaluations face two key limitations: many rely on parallel template-based questions that may not reflect how cultural knowledge naturally appears, and raw accuracy conflates general language proficiency with language-conditioned knowledge access. We address these issues with a controlled framework built on real-world cultural questions collected from regional benchmarks and local sources. By crossing question type (culture-agnostic vs. culture-specific) with query language (English vs. local language), and estimating ability with a shared 1PL item response theory model, we separate proficiency from localized knowledge access. Across 13 locales and roughly 80 models, we find a consistent English advantage on culture-agnostic questions, indicating stronger English proficiency. However, after accounting for this proficiency gap, local languages show a positive knowledge-access advantage in nearly all locale-model settings. This advantage is often masked in raw accuracy but becomes more visible for frontier, regionally aligned, or language-adapted models. Our results suggest that weaker local-language performance does not necessarily imply weaker cultural knowledge; rather, local cultural knowledge may be more accessible through the local language but hidden by limited language proficiency.

Abstract

Cultural Knowledge in Language Models

Explore similar work

The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs

Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth

MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark