NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria
Organizations: AWS FDE · Obafemi Awolowo University
Abstract
Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce NaijaFacts, a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, Yorùbá, Igbo, and Hausa. Answering similar questions for the UK and US yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language). Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49-50% for the US/UK but 20% for Nigeria. The language effect is smaller and tracks web representation, degrading from English to Yorùbá (9 points on Nigeria, up to 24 on US/UK questions). We find scale does not help: the strongest models reach 66-79% on the UK and the US but none exceed 31% on Nigerian statistics. Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96% on culture and 98% on politics against 26% on statistics, so the gap is not specific to Nigeria.
Figures & tables
| Judged accuracy (%) on NaijaFacts Nigeria-stats | |||||||
| Model | Size | English | Pidgin | Hausa | Igbo | Yorùbá | Avg. (NG lang) |
| GPT-5.6 | – | 28 | 26 | 29 | 31 | 22 | 27.0 |
| Claude Opus 5 | – | 28 | 26 | 21 | 25 | 24 | 24.0 |
| Grok 4.6 | – | 29 | 23 | 25 | 25 | 19 | 23.0 |
| Kimi K2.5 | – | 15 | 17 | 13 | 0 9 | 0 5 | 11.0 |
| Qwen3 | 235B | 18 | 16 | 0 8 | 0 8 | 0 5 | 0 9.2 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Language | Question (gold answer: National Population Commission) |
| English | What does NPC stand for in Nigeria’s population statistics system? |
| Pidgin | Wetin be di NPC stand for for Naija population statistics? |
| Yorùbá | Kini NPC duro fun ninu eto iṣiro awon eniyan Naijiria? |
| Igbo | Gịnị ka NPC pụtara n’usoro ọnụ ọgụgụ ndị mmadụ nke Naịjirịa? |
| Hausa | Menene ma’anar NPC a tsarin kididdigar yawan jama’a na Najeriya? |
| Language | Question | Gold answer | Models correct |
| English | What age range was used for all Nigerian women interviewed in the 2024 NDHS? | 15 to 49 years. | 8/8 |
| Pidgin | Wetin be di age range wey dem use for all Naija wumen wey dem interview for 2024 NDHS? | 15 to 49 years. | 7/8 |
| Hausa | Wane adadin shekarun matan Nageriya aka yi wa tambayoyi a NDHS 2024? | Shekaru 15 zuwa 49. | 6/8 |
| Igbo | Kedu oke afọ e jiri maka ụmụ nwanyị Naijira niile ndị a gbara ajụjụ ọnụ na NDHS 2024? | Afọ 15 ruo 49. | 5/8 |
| Yorùbá | Kini iwọn ọjọ-ori awon obinrin ni Naijiria ti won ba sọrọ ninu NDHS 2024? | Ọdun 15 si 49. | 4/8 |
| Component | Value |
| Questions | 100, authored by native speakers |
| Question languages | Yorùbá, Igbo, Hausa, Nigerian Pidgin (native); English (translated) |
| Answer countries | Nigeria (NPC/NBS/NDHS), UK (ONS), US (Census Bureau/NCHS) |
| Answer collection | Nigeria: native speakers, in-language; UK/US: deep-research, verified |
| Evaluation cells | 100 5 languages 3 countries 1,500 per model |
| Cross-country extension | 54 African countries, 118–125 English items each (6,689 total) |
| Model | Origin | Identifier | Size |
| Frontier (proprietary, via AWS Bedrock) | |||
| Claude Opus 5 | US | anthropic.claude-opus-5 | – |
| GPT-5.6 | US | openai.gpt-5.6-sol | – |
| Grok 4.6 | US | xai.grok-4.6 | – |
| Mistral Large 3 | EU | mistral.mistral-large-3-675b-instruct | 675B |
| Qwen3 | CN | qwen.qwen3-235b-a22b-2507 | 235B |
| Country asked about | |||
| Language asked in | Nigeria | UK | US |
| English | 20.1 | 49.2 | 50.2 |
| Nigerian lang. (avg) | 14.3 | 35.5 | 34.9 |
| Pidgin | 17.5 | 44.9 | 44.2 |
| Hausa | 15.1 | 35.2 | 35.1 |
| Igbo | 13.2 | 34.2 | 33.5 |
| Benchmark | # Afr. lang. | Facts about Africa? | Ground-truth source | Country lang.? | Dominant task(s) |
| AfriQA Ogundepo et al. [2023] | 10 | partial | Wikipedia / web | ✗ | Cross-lingual QA |
| Belebele Bandarkar et al. [2024] | 29 | ✗ | translated passages | ✗ | Reading comprehension |
| IrokoBench Adelani et al. [2025] | 17 | ✗ | translated (MMLU/MGSM/NLI) | ✗ | MMLU, Math, NLI |
| AfroBench Ojo et al. [2025] | 64 | ✗ | aggregated / translated | ✗ | 15 NLP tasks |
| Uhura Bayes et al. [2026] | 6 | ✗ | translated ARC / TruthfulQA | ✗ | Science QA, truthfulness |
| Afri-MCQA Tonja et al. [2026] | 15 | ✓ (culture) | native-authored culture QA | ✗ | Multimodal cultural QA |
| Country | Source body | Items |
| Nigeria | Nigeria Demographic and Health Survey 2024 [ Federal Ministry of Health and Social Welfare of Nigeria (FMoHSW), National Population Commission (NPC) [Nigeria], and ICF, 2025 ] | 81 |
| NPC census documentation, 2006 and 2023 [ National Population Commission (Nigeria), 2026 ] | 8 | |
| NBS releases carrying no edition year [ National Bureau of Statistics (Nigeria), 2026 ] | 11 | |
| UK | Office for National Statistics [ Office for National Statistics (UK) and National Records of Scotland, 2026 ] | 56 |
| UK government departments | 11 | |
| National Records of Scotland and Scotland’s Census [ Office for National Statistics (UK) and National Records of Scotland, 2026 ] | 9 |
| Country | Acc. | Country | Acc. | Country | Acc. | |||
| Algeria | 125 | 44.8 | Eswatini | 124 | 33.9 | Niger | 123 | 42.3 |
| Angola | 124 | 34.7 | Ethiopia | 125 | 38.4 | Nigeria | 125 | 36.0 |
| Benin | 125 | 44.8 | Gabon | 125 | 43.2 | Rwanda | 125 | 41.6 |
| Botswana | 125 | 33.6 | Ghana | 125 | 45.6 | Senegal | 125 | 36.0 |
| Burkina Faso | 123 | 39.8 | Guinea | 125 | 52.8 | Seychelles | 119 | 29.4 |
| Burundi | 125 | 45.6 | Guinea-Bissau | 122 | 33.6 | Sierra Leone | 125 | 45.6 |
Factual Knowledge in Language Models
Factual knowledge in language models concerns what facts they store, how reliably they retrieve them, and how that knowledge can be tested or changed.
See all 212 papers