African Languages AI Safety Readiness Report

Israel Abebe Azime; Jesujoba O. Alabi; Tajuddeen Gwadabe; David Ifeoluwa Adelani

cs.CLOct 19, 2026

African Languages AI Safety Readiness Report

Authors: Israel Abebe Azime, Jesujoba O. Alabi, Tajuddeen Gwadabe, David Ifeoluwa Adelani

Organizations: Masakhane · Saarland University · Mila - Quebec AI Institute, McGill University, and Canada CIFAR AI Chair

Abstract

With the widespread adoption of large language models (LLMs) across languages and domains, AI safety has emerged as a critical global concern. However, due to resource constraints, research and development efforts tend to prioritize a limited number of high-resource languages, leaving many others, especially those spoken in Africa, underrepresented. While existing research on language "resource-ness" has largely examined whether sufficient resources are available to develop language technologies, the question of whether languages are adequately represented in AI safety research has received attention only more recently. This report examines the readiness of African languages for inclusion in AI safety discussions, drawing on existing literature and data. It considers four factors: the availability of AI-relevant resources, including text data; national AI policy support; performance on established text benchmarks; and speaker population. Based on these factors, languages are grouped into four readiness categories: fully ready, mostly ready, emerging, and foundational support needed. This classification is intended to inform the inclusion of African languages in AI safety benchmarks, highlight varying levels of preparedness, and guide strategic investment in data development, policy, and community engagement for underrepresented languages.

Figures & tables

Figure 1: A layered framework for evaluating the presence and readiness of African languages in machine learning ecosystems. The analysis spans dataset coverage in pretraining corpora, representation in major AI systems, and performance across both popular multilingual benchmarks and African-specific LLM benchmarks. It also incorporates qualitative dimensions such as national AI strategies, policy landscapes, and community-centered insights, including real-world usage and feedback from related research. This comprehensive view is used to assess not only technological readiness but also to guide the inclusion of languages in safety-critical benchmarks and broader AI resource prioritization. The result is highly dependent on Language ID used.

Panel of Figure 3 — Figure 3: Top 20 African languages in publicly available multilingual unlabeled corpora: Wikipedia and FineWeb.

	LID (acc $\uparrow$ )			Topic class. (acc $\uparrow$ )			RC-QA (acc $\uparrow$ )			Machine Transl. (ChrF++ $\uparrow$ )			Ave. (class. tasks)
Region	Gemma	GPT-4o	Gemini	Gemma	GPT-4o	Gemini	Gemma	GPT-4o	Gemini	Gemma	GPT-4o	Gemini	Gemma	GPT-4o	Gemini
Africa	62.9	69.6	80.0	59.9	69.6	74.9	60.6	70.2	77.0	34.8/24.0	41.0/30.5	46.2/32.0	61.4	69.6	77.5
Americas/Oceania	79.2	99.6	99.5	69.5	80.5	82.9	62.3	78.2	80.7	41.2/32.4	49.0/40.8	52.6/42.9	74.3	90.0	91.2
Asia (S)	78.2	83.2	87.2	81.5	82.1	86.7	78.6	83.0	84.7	51.9/33.3	54.2/34.2	59.6/39.2	79.8	82.7	87.0
Asia (SE)	76.2	82.0	86.6	75.8	80.9	84.5	75.5	76.3	81.7	48.4/35.1	51.8/38.5	57.6/42.1	76.0	81.5	85.5
Asia (W, C)	99.3	99.2	99.5	85.3	86.8	87.7	83.3	87.0	87.5	56.5/38.8	58.6/44.3	60.8/45.9	92.3	93.0	93.6

Table 1: Average text-based evaluation results across languages by region, using Gemma (3 27B), Gemini-2-Flash, and GPT-4o (08/24). Results obtained from [ 65 ]

									MT
Model	Size	Lang	Intent	TC	NLI	RC	MMLU	Math	en/fr-xx	AVG
Lugha-Llama	8B	eng	16.7	43.6	46.8	22.4	31.8	6.4	51.3	31.3
Lugha-Llama	8B	africa	4.1	34.1	36.7	23.0	25.2	1.8	22.1	21.0
Gemma 2	9B	eng	36.3	82.5	70.7	93.7	69.8	68.8	67.9	70.0
Gemma 2	9B	africa	27.8	64.0	40.9	49.3	36.1	21.7	37.2	39.6
Gemma 2	27B	eng	84.0	89.3	67.8	93.4	75.6	85.6	68.5	80.6

Table 2: LLM performance on AfroBench-Lite, covering 14 African languages across 7 tasks: intent classification (Intent), topic classification (TC), natural language inference (NLI), reading comprehension (RC), knowledge base question answering (MMLU), mathematical reasoning (Math), and machine translation (MT). Results are obtained from [ 70 ] .

Panel of Figure 4 — Figure 4: Correlation Between Data Size and Performance for African Languages. (Left) Accuracy ratio on SIB-200; (Right) chrF++ ratio on FLORES-200. Both plots compare performance against the amount of pretraining data in Fineweb2.

Figure 5: Variation in language understanding accuracy of GPT-4o when prompted in different African languages from the FLORES-200 benchmark. Higher scores indicate successful comprehension and response in the correct language, while lower scores highlight failure modes such as refusal or defaulting to English. The results underscore performance disparities linked to resource availability and benchmark representation.

Panel of Figure 6 — Figure 6: Overview of African AI Policy and Language Translation Service Presence. (a) Map showing the status of national AI policies across African countries, categorized as Yes , Ongoing , and No . (b) Scatter plot illustrating the performance of NLLB and Google Translate scores against language character counts (log scale), color-coded by language (BCP-47).

Category	Languages
Fully Ready	Hausa, Egyptian Arabic, Afrikaans, Nigerian Pidgin, Zulu, Swahili, Amharic, Somali, Shona, Nyanja, Southern Sotho, Yoruba, Igbo, Northern Sotho, Kinyarwanda
Mostly Ready	Tsonga, Xhosa, West Central Oromo, Tswana, Rundi, Twi, Bemba, Tunisian Arabic, Tumbuka, Swati
Emerging	Akan, Wolof, Tigrinya, Kabyle, Mossi, Bambara, Moroccan Arabic, Nigerian Fulfulde, Central Kanuri (Latin script), Chokwe, Ewe, Kabiyè, Lingala, Central Kanuri (Arabic script), Luo, Fon, Kikuyu, Central Atlas Tamazight, Nuer, N’ko, Kamba, Kabuverdianu, Plateau Malagasy, Ganda, Luba-Kasai, Sango, Kikongo, Umbundu, Kimbundu, Dyula, Southwestern Dinka, Tamasheq (Latin script), Tamasheq (Tifinagh script)
Foundational support needed	All African languages not included in FLORES-200, such as [e.g., Tigré, Wolaytta, etc.]

Table 3: Categorization of African Languages for AI Safety. the languages are arranged in no particular order.

Language Model Safety Evaluation

Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.

See all 543 papers

TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages
May 31, 2026Victor Akinode, Senyu Li, Wassim Hamidouche +3LLM Safety Benchmarks Multilingual Language Model Evaluation
Mila - Quebec AI Institute · McGill University · Microsoft AI for Good Research Lab +1
LSR: Linguistic Safety Robustness Benchmark for Low-Resource West African Languages
Feb 27, 2026Godwin Abuh FarunaLLM Safety Benchmarks Language Model Safety Evaluation
Fagmart Lab
Language-Specific Gaps in AI Safety Training Datasets
Aug 13, 2026Chialuka Prisca-Mary Onuoha, Bright Etornam Sunu, Rashidat SikiruLLM Safety Benchmarks Multilingual Language Model Evaluation
Black in AI Safety & Ethics (BASE)

May 31, 2026J/K move · Enter open · S save

TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages

Victor Akinode, Senyu Li, Wassim Hamidouche, Waqas Zamir+2

Mila - Quebec AI Institute · McGill University · Microsoft AI for Good Research Lab+1

LLM Safety Benchmarks Multilingual Language Model Evaluation Language Model Safety Evaluation LLM Jailbreak Attacks Low-Resource Language Processing Refusal Behavior in Language Models

Open paper AskPDF arXiv

Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, critically underexplored. We introduce TUKABENCH, a jailbreak benchmark for seven African languages that extends JailbreakBench (JBB) beyond direct translation through four settings: human translation of JBB prompts, English adaptation to African contexts followed by human translation, human-curated prompts validated through interactions with GPT-5.2, and code-switched prompts combining English and African languages, isolating the effect of language, cultural grounding, and prompt evasiveness on model safety. Across closed and open models, prompting in African languages reduces refusal relative to English, with culturally adapted prompts leading to least refusal. The evaluation also surfaces two structural limitations: model comprehension failures and reduced LLM-as-a-judge reliability in LRLs. To capture the first, we introduce Deflection alongside Refused and Jailbroken; to assess the second, we validate outputs with human annotations, showing that judge-human agreement drops in lower-resource languages and less commonly supported scripts.

Abstract

Language Model Safety Evaluation

Explore similar work

TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages

LSR: Linguistic Safety Robustness Benchmark for Low-Resource West African Languages

Language-Specific Gaps in AI Safety Training Datasets