AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages

Israel Abebe Azime; Jesujoba Oluwadara Alabi; Crystina Zhang; Iffat Maab; Atnafu Lambebo Tonja; Tadesse Destaw Belay; Folasade Peace Alabi; Salomey Osei

cs.CLApr 1, 2026

AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages

Authors: Israel Abebe Azime, Jesujoba Oluwadara Alabi, Crystina Zhang, Iffat Maab, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Folasade Peace Alabi, Salomey Osei,

Organizations: Saarland University, Saarland Informatic Campus, Germany · Masakhane NLP · University of Waterloo, Canada · National Institute of Informatics, Japan ·

Abstract

Assessing the veracity of a claim made online is a complex and important task with real-world implications. When these claims are directed at communities with limited access to information and the content concerns issues such as healthcare and culture, the consequences intensify, especially in low-resource languages. In this work, we introduce AfrIFact, a dataset that covers the necessary steps for automatic fact-checking (i.e., information retrieval, evidence extraction, and fact checking), in ten African languages and English. Our evaluation results show that even the best embedding models lack cross-lingual retrieval capabilities, and that cultural and news documents are easier to retrieve than healthcare-domain documents, both in large corpora and in single documents. We show that LLMs lack robust multilingual fact-verification capabilities in African languages, while few-shot prompting improves performance by up to 43% in AfriqueQwen-14B, and task-specific fine-tuning further improves fact-checking accuracy by up to 26%. These findings, along with our release of the AfrIFact dataset, encourage work on low-resource information retrieval, evidence retrieval, and fact checking.

Figures & tables

Figure 1: Three steps for verifying factual claims are covered by our dataset. (A) information retrieval to find documents related to a claim, (B) extraction of sentence-level evidence supporting or refuting the claim, and (C) fact-checking using the claim, retrieved document, and extracted evidence with LLMs.

Dataset	# Lang.	# Native African Lang.	Domains	Tasks
X-FACT Gupta and Srikumar (2021)	25	0	News, Politics	Claim Verification
XFEVER Chang et al. (2023)	6	0	Wikipedia	Multilingual Claim Verification
MultiClaim Panchendrarajan et al. (2025)	78	1 (Sudanese)	Social Media, News	Claim Matching, Retrieval
Poly-FEVER Zhang et al. (2025a)	11	1 (Amharic)	Wikipedia, Scientific Articles	Multilingual Fact Verification
AfrIFact(ours)	11	10	Health, Culture(Wiki)-News	Fact Verification, Evidence Extraction, Information Retrieval

Table 1: Comparison of multilingual fact-checking datasets. We report the number of supported languages, number of African languages, domain coverage, and task types. AfrIFact focuses on African languages and introduces multilingual information retrieval, evidence extraction and claim verification across culturally grounded domains.

Figure 2: Illustration of the data construction process for AfrIFact-Health on shared health-data approach for lack of structured native health documents and AfrIFact-Culture-News culturally grounded, natively sourced data approach

Split	# Lang.		Total Split		Total
		Train	Val	Test
Health	11	1100	550	7678
Culture & News	11	1228	550	7150
		2328	1100	14828	18256

Table 2: Dataset statistics (train, validation, test) for AfrIFact-Health and AfrIFact-Culture-News across 11 languages. Detail data distribution in Appendix A .

Dataset	Metric	REFUTES	NEI	SUPPORTS
Health	Fleiss’ K.	0.74	0.79	0.82
	Cohen’s K.	0.69	0.74	0.79
	Krippendorff’s	0.74	0.79	0.82
Culture-News	Fleiss’ K.	0.82	0.83	0.83
	Cohen’s K.	0.81	0.82	0.82
	Krippendorff’s	0.85	0.85	0.85

Table 3: Agreement scores for English AfrIFact-Health and averaged per language for AfrIFact-Culture-News split. Detailed result found in the Appendix A .

Figure 3: nDCG@10 scores on the Health domain when retrieving from different corpora and evaluated on relevant documents in mono- or multi-languages. "A. Avg" shows the average score on African languages. Scores with label “mono” indicate that we only consider relevant documents from the same language as the query during evaluation.

model	English	Amharic	Hausa	Igbo	Oromo	Shona	Swahili	Twi	Wolof	Yoruba	Zulu	A-Avg.	$\Delta$
AfrIFact-Health
multilingual-e5-large	0.149	0.123	0.113	0.119	0.138	0.141	0.189	0.107	0.105	0.133	0.171	0.134	-0.015
multilingual-e5-large-inst.	0.229	0.094	0.092	0.118	0.133	0.116	0.148	0.105	0.115	0.085	0.123	0.113	-0.116
AfriE5-Large-instruct	0.298	0.235	0.221	0.248	0.215	0.224	0.287	0.183	0.173	0.172	0.257	0.222	-0.077
Qwen3-Embedding-0.6B	0.171	0.065	0.070	0.106	0.111	0.073	0.112	0.098	0.111	0.081	0.082	0.091	-0.080
Qwen3-Embedding-4B	0.184	0.087	0.077	0.100	0.103	0.073	0.132	0.103	0.126	0.095	0.102	0.100	-0.084

Table 4: nDCG@10 of Information Retrieval Scores on Multilingual Corpus; A-Avg. column shows average score on African languages (i.e., excluding English);

\Delta

column shows the score gap between A-Avg. English scores. gray marks on best A-Avg. scores.

model	English	Amharic	Hausa	Igbo	Oromo	Shona	Swahili	Twi	Wolof	Yoruba	Zulu	A-Avg.	$\Delta$
AfrIFact-Health
multilingual-e5-large	0.770	0.363	0.415	0.349	0.386	0.450	0.541	0.333	0.418	0.288	0.438	0.398	-0.372
multilingual-e5-large-inst.	0.724	0.346	0.403	0.346	0.379	0.447	0.517	0.341	0.412	0.269	0.420	0.388	-0.336
AfriE5-Large-instruct	0.742	0.371	0.419	0.361	0.395	0.455	0.527	0.363	0.443	0.290	0.442	0.407	-0.335
Qwen3-Embedding-0.6B	0.782	0.305	0.370	0.275	0.340	0.368	0.474	0.298	0.397	0.244	0.362	0.343	-0.439
Qwen3-Embedding-4B	0.788	0.325	0.371	0.295	0.339	0.373	0.510	0.305	0.405	0.249	0.381	0.355	-0.433

Table 5: nDCG@3 evidence extraction performances across languages and domains in AfrIFact. A-Avg. The column shows the average score on African languages (i.e., excluding English);

\Delta

column shows the score gap between A-Avg. English scores. gray shows the best A-Avg. scores.

Panel of Figure 4 — Figure 4: Average accuracy scores on African languages of different language models on the AfrIFact fact-checking task under zero-shot and three-shot settings, with and without evidence. Left : AfrIFact-Health Right : AfrIFact-Culture-News

Model

Amharic

English

Hausa

Igbo

Oromo

Shona

Swahili

Twi

Wolof

Yoruba

Zulu

A-Avg.

AfrIFact-Health

Baseline

33.43 / 33.24

33.29 / 34.53

33.81 / 33.29

33.29 / 33.24

33.38 / 33.24

33.29 /34.96

33.29 / 33.24

33.33 /33.24

33.52 / 33.91

33.33 / 34.53

33.40 / 33.74

LoRA

49.14 / 63.94

49.00 / 59.41

47.85 / 61.41

46.78 / 56.92

48.21 / 58.74

48.50 / 58.93

48.42 / 62.85

37.25 / 40.59

38.18 / 42.26

46.92 / 59.22

48.28 / 60.46

45.95 / 56.53

QLoRA 8-bit

49.43 / 63.66

49.86 / 65.23

48.93 / 64.23

48.17 / 61.46

49.21 / 62.89

49.21 / 63.80

49.36 / 65.57

39.83 / 45.22

39.68 / 46.90

48.50 / 62.37

49.14 / 63.75

47.15 / 59.99

QLoRA 4-bit

39.30 / 50.96

34.38 / 35.96

36.29 / 49.04

35.34 / 44.84

36.06 / 44.17

36.44 / 44.41

37.06 / 49.43

33.38 / 34.96

33.33 / 34.53

38.78 / 49.76

37.06 / 50.38

36.30 / 45.25

Full Finetuning

33.31 / 33.24

33.74 / 33.48

33.31 / 33.24

33.31 / 33.29

33.31 / 33.24

33.31 / 33.25

Table 6: AfrIFact-Culture-News and AfrIFact-Culture-News without-evidence/with-evidence finetuning experiments on Baseline( AfriqueQwen-14B ) finetuned for mixed culture and health data processed for with and without setup. Training approaches include LoRA, QLoRA and full-finetuning

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

language	Amharic	English	Hausa	Igbo	Twi	Oromo	Shona	Swahili	Wolof	Yoruba	Zulu	Total
AfrIFact-Health
SUPPORT	284	284	284	284	284	284	284	284	284	284	284	3124
REFUTES	283	283	283	283	283	283	283	283	283	283	283	3113
NEI	281	281	281	281	281	281	281	281	281	281	281	3091
Total	848	848	848	848	848	848	848	848	848	848	848	9328
AfrIFact-Culture-News

Appendix

Table 7: Detailed distribution of annotated examples in the AfrIFact dataset across languages, domains, and verification labels. The table reports the number of instances labeled as SUPPORTS , REFUTES , and NOT_ENOUGH_INFORMATION (Not Enough Information) for each language in the AfrIFact-Health and AfrIFact-Culture-News domains.

Language	Class	Fleiss’ Kappa	Cohen’s Kappa (avg)	Krippendorff’s Alpha
Amharic	REFUTES	0.80	0.80	0.80
	NOT_ENOUGH_INFORMATION	0.88	0.88	0.88
	SUPPORTS	0.88	0.88	0.88
Yoruba	REFUTES	0.92	0.92	0.92
	NOT_ENOUGH_INFORMATION	0.94	0.94	0.94
	SUPPORTS	0.96	0.96	0.96

Appendix

Table 8: Inter-Annotator Agreement Metrics by Language and Category for AfrIFact-Culture-News , showing substantial to great agreement score.

Language	Family/branch	Region	# speakers	# chars in MADLAD (MB)
Swahili (swa)	Niger-Congo / Bantu	East & Central Africa	71M–106M	2,400MB
Hausa (hau)	Afro-Asiatic / Chadic	West Africa	77M	630MB
Amharic (amh)	Afro-Asiatic / Ethio-Semitic	East Africa	57M	509MB
chiShona (sna)	Niger-Congo / Bantu	Southern Africa	11M	266MB
isiZulu (zul)	Niger-Congo / Bantu	Southern Africa	27M	257MB
Igbo (ibo)	Niger-Congo / Volta-Niger	West Africa	31M	251MB

Appendix

Table 9: Languages represented in AfrIFact dataset. The table reports the language family and branch, geographic region, countries where the language is widely spoken, and the approximate number of speakers.

Language	Country	Gender Distribution (LC/Annotator)	Qualifications	Resides in Country
Amharic	Ethiopia	M / 2M1F	Linguistics, CS, Translation	✓
Hausa	Nigeria	M / 2M1F	Linguistics and translation	✓
Igbo	Nigeria	F / 1M2F	Language studies	✓
Twi	Ghana	F / 1M2F	Linguistics, education	✓
Oromo	Ethiopia	F/ 2F1M	Linguistics and journalism	✓
Shona	Zimbabwe	F / 2M1F	Language studies and translation	✓

Appendix

Table 10: Summary of annotator teams involved in the creation of the AfriFact dataset. For each language, we report the country context, gender of the language coordinator (LC), gender distribution of the three annotators, a summary of annotator qualifications, and whether annotators reside in the corresponding language region.

Model	English	Amharic	Hausa	Igbo	Oromo	Shona	Swahili	Twi	Wolof	Yoruba	Zulu	A-Avg.	$\Delta$
Monolingual Corpus on Health
multilingual-e5-large	0.524	0.422	0.366	0.385	0.444	0.354	0.371	0.323	0.348	0.259	0.387	0.366	-0.158
multilingual-e5-large-inst.	0.629	0.426	0.372	0.438	0.476	0.384	0.517	0.420	0.464	0.313	0.433	0.424	-0.205
AfriE5-Large-instruct	0.593	0.431	0.394	0.431	0.449	0.412	0.525	0.391	0.408	0.301	0.424	0.417	-0.176
Qwen3-Embedding-0.6B	0.558	0.286	0.169	0.267	0.360	0.206	0.326	0.282	0.336	0.150	0.204	0.259	-0.299
Qwen3-Embedding-4B	0.546	0.326	0.233	0.274	0.341	0.212	0.363	0.325	0.394	0.208	0.262	0.294	-0.252

Appendix

Table 11: AfrIFact-Health nDCG@10 scores when retrieving queries from different corpus and evaluated on corresponding qrels.

Monolingual Corpus on Health
Model	English	A-Avg.	$\Delta$
multilingual-e5-large	0.524	0.366	-0.158
multilingual-e5-large-inst.	0.629	0.424	-0.205
AfriE5-Large-instruct	0.593	0.417	-0.176
Qwen3-Embedding-0.6B	0.558	0.259	-0.299
Qwen3-Embedding-4B	0.546	0.294	-0.252

Appendix

Table 12: AfrIFact-Health nDCG@10 scores when retrieving queries from different corpora and evaluated on corresponding qrels. Table 11 reports results per-language.

model	English	Amharic	Hausa	Igbo	Oromo	Shona	Swahili	Twi	Wolof	Yoruba	Zulu	A-Avg.	$\Delta$
AfrIFact-Health
multilingual-e5-large	0.820	0.408	0.460	0.403	0.439	0.509	0.586	0.396	0.484	0.325	0.495	0.450	-0.369
multilingual-e5-large-inst.	0.822	0.380	0.455	0.401	0.422	0.507	0.572	0.398	0.473	0.317	0.478	0.440	-0.382
AfriE5-Large-instruct	0.822	0.412	0.464	0.412	0.437	0.513	0.574	0.409	0.506	0.337	0.495	0.456	-0.366
Qwen3-Embedding-0.6B	0.823	0.355	0.444	0.344	0.402	0.444	0.548	0.350	0.470	0.300	0.417	0.407	-0.416
Qwen3-Embedding-4B	0.825	0.373	0.443	0.349	0.395	0.458	0.567	0.347	0.480	0.294	0.440	0.415	-0.410

Appendix

Table 13: AfrIFact Recall@3 Evidence Extraction Scores.

Model

Amharic

English

Hausa

Igbo

Oromo

Shona

Swahili

Twi

Wolof

Yoruba

Zulu

A-Avg.

AfrIFact-Health With Evidence

AfriqueGemma-12B

38.11/38.68

36.01/47.66

36.82/38.44

38.59/40.40

38.68/39.40

35.20/41.17

34.48/38.40

34.81/34.48

34.19/33.38

35.96/35.86

37.73/38.78

36.46/37.90

AfriqueQwen-14B

33.24/71.11

33.24/92.26

34.53/84.19

33.29/83.38

33.24/86.44

33.24/86.87

34.96/86.34

33.24/58.17

33.24/54.97

33.91/79.27

34.53/81.76

33.74/ 77.25

Command R v01

39.06/35.24

63.75/89.83

40.54/45.08

38.49/44.56

36.91/55.06

39.54/47.66

45.32/46.94

36.68/48.95

35.77/52.96

37.73/38.97

42.88/49.28

39.29/46.47

Gemma-3-12B-it

38.59/39.02

58.07/71.30

55.21/58.31

58.60/60.74

52.58/53.39

56.26/63.18

58.36/63.80

45.46/47.23

38.40/43.36

35.77/35.91

54.58/56.30

49.38/52.13

Gemma-3-1B-it

34.77/35.91

34.34/52.39

34.24/35.51

33.76/36.51

33.67/33.72

33.76/37.18

35.82/39.21

33.67/35.10

33.43/34.15

34.62/34.03

34.00/35.24

34.17/35.65

Appendix

Table 14: AfrIFact-Health 0-shot/3-shot Accuracy Scores. The best open source models are Bold, and the best overall models are grayed.

Model

Amharic

English

Hausa

Igbo

Oromo

Shona

Swahili

Twi

Wolof

Yoruba

Zulu

A-Avg.

AfrIFact-Culture-News With Evidence

AfriqueGemma-12B

36.10/39.10

45.70/57.70

36.10/44.50

33.70/36.70

35.60/39.00

37.30/48.20

32.00/34.20

31.10/32.30

37.40/36.70

33.60/35.50

34.10/43.40

34.70/38.96

AfriqueQwen-14B

33.50/77.00

35.40/99.90

37.60/77.10

31.80/78.40

32.80/80.80

37.50/76.80

36.60/71.80

30.60/61.10

38.60/44.70

34.00/74.60

36.80/70.10

34.98/ 71.24

Command R v01

42.80/39.20

53.60/92.80

37.50/47.10

32.60/46.30

33.00/57.50

38.70/43.50

39.10/58.20

31.30/48.90

38.60/42.60

37.60/41.70

43.80/47.10

37.50/47.21

Gemma-3-12B-it

61.60/65.10

82.60/86.40

59.50/63.10

58.10/60.70

47.60/53.60

51.00/63.50

56.40/57.80

45.60/50.70

39.40/44.50

49.80/52.70

54.40/57.00

52.34/56.87

Gemma-3-1B-it

35.00/33.70

35.20/54.70

38.10/37.00

31.80/38.70

32.80/38.00

37.70/33.60

36.30/30.80

30.70/33.80

38.70/37.20

35.10/36.90

37.50/32.70

35.37/35.24

Appendix

Table 15: AfrIFact-Culture-News 0-shot/3-shot Accuracy Scores. The best open source models are Bold, and the best overall models are grayed.

Figure 5: Why is evidence not helping improve accuracy? Evidence introduces a conservative shift in model predictions: in Health, it reduces hallucinated SUPPORTS but increases NOT_ENOUGH_INFORMATION predictions, while in Culture, it significantly improves NEI detection by reducing false Supports classifications. In the Health domain NOT_ENOUGH_INFORMATION and SUPPORTS improve while in the culture domain, the model tends to be good at identifying NOT_ENOUGH_INFORMATION when evidence is provided. In both domains, the model confusion between SUPPORTS and REFUTES improves with evidence provided.

Figure 6: Distribution of COMET scores for claim and document translations across African languages. The upper grid shows histograms of COMET scores for claim translations across ten languages, while the lower grid presents score distributions for document translations in five languages (Igbo, Oromo, Shona, Twi, and Wolof). This fixed or false alarm percentages items that has below 0.6 COMET scores colored red and addressed by annotators.

Figure 7: Distribution of COMET scores for document translations across African languages. This fixed or false alarm percentages items that has below 0.6 COMET scores colored red and addressed by annotators.

Figure 8: Example interface of the customized tool used for claim generation, showing an Amharic example.

Figure 9: Example of the customized annotation interface used for labeling claims in the claim labeling, showing an English example.

Automated Fact-Checking

Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.

See all 134 papers

ARAFA: An LLM-Generated Arabic Fact-Checking Dataset
Sep 22, 2026Christophe Khalil, Shady Elbassuoni, Rida AssafArabic NLP Automated Fact-Checking
Computer Science, American University of Beirut, Beirut, Lebanon
Multilingual Fact-Checking at Scale: Fine-Tuned Compact Models vs LLMs
Jun 7, 2026Pratuat Amatya, Vinay SettyMultilingual Language Models LLM Evaluation
Factiverse AS and University of Stavanger Stavanger, Norway
M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset
Oct 27, 2025Jiahui Geng, Jonathan Tonglet, Iryna GurevychAutomated Fact-Checking Multimodal Understanding
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany · Department of Electrical Engineering, KU Leuven, Belgium +2

Sep 22, 2026J/K move · Enter open · S save

ARAFA: An LLM-Generated Arabic Fact-Checking Dataset

Christophe Khalil, Shady Elbassuoni, Rida Assaf

Computer Science, American University of Beirut, Beirut, Lebanon

Arabic NLP Automated Fact-Checking Synthetic Data Generation for Language Models Low-Resource Language Processing

Open paper AskPDF arXiv

Automatic fact-checking poses a significant challenge in Arabic natural language processing due to the scarcity of datasets and resources. In this manuscript, we introduce Arafa, a new large-scale dataset for fact-checking in Modern Standard Arabic, constructed through an automated framework leveraging large language models (LLMs). The dataset was constructed through a three-step pipeline: (1) claim generation from Arabic Wikipedia pages with supporting textual evidence, (2) claim mutation to generate challenging counterfactual claims with refuting evidence, and (3) an automatic validation step to validate that the generated claims are either supported or refuted by their accompanying evidence, or if the evidence does not provide enough information to judge the validity of the claims. The resulting dataset comprises 181,976 claim-evidence pairs labeled as supported, refuted, or not enough information. Human evaluation carried out on a test sample from the dataset demonstrated strong inter-annotator agreement (kappa = 0.89) using Cohen's Kappa for supported claims and (kappa = 0.94) for refuted claims. Automatic validation based on a human-evaluated sample achieved 86% accuracy for supported claims and 88% for refuted ones. To showcase Arafa's value as a resource for automatic Arabic fact-checking, four open-source transformer-based models were fine-tuned using Arafa, with the top-performing model achieving a Macro F1-score of 77% on the test data. In addition to Arafa being the first large-scale dataset for Arabic fact-checking, our framework presents a scalable approach for developing similar resources for other low-resource languages.

Abstract

Automated Fact-Checking

Explore similar work

ARAFA: An LLM-Generated Arabic Fact-Checking Dataset

Multilingual Fact-Checking at Scale: Fine-Tuned Compact Models vs LLMs

M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset