Organizations: African Institute for Mathematical Sciences (AIMS), Cameroon · Independent Researcher · Brown University · Centrum Wiskunde & Informatica · University of Pretoria · University of Hamburg · Bayero University Kano · Imperial College London
As large language models (LLMs) become more widespread in Africa, their safety in low-resource languages remains largely untested since most red-teaming work focuses on English. This study evaluates six state-of-the-art LLMs using 400 explicit adversarial prompts in Yoruba and Nigerian Pidgin, categorised into four harm categories. The responses were scored by native-speaker annotators and reliability was assessed using Cohen's κ. While models that rejected up to 69% of English prompts rejected as few as 0% in Yoruba, this was not because they were safer, but because they failed to understand the prompts. Attack success rates increased for most models in the Nigerian languages, with GPT-4o Mini demonstrating the most significant cross-lingual safety discrepancy (+0.23 in Yoruba). When models did engage with adversarial prompts, many of the responses were harmful, with rates of harmful content reaching 0.63 in English and 0.50 in Nigerian Pidgin. The most difficult harm category to detect was cultural stereotypes and discrimination (ASR = 0.44 in Nigerian Pidgin). Comprehension in Yoruba was low across most models, and this directly suppressed apparent attack success without indicating actual safety. Gemini 2.0 Flash was the most consistent model across languages. Despite its explicit training in African languages, Aya 101 had the highest attack success rate in English (0.60) and one of the lowest Yoruba comprehension rates (0.28). These results demonstrate that English-centric safety alignment does not transfer to Nigerian languages, highlighting the urgent need for language-specific safety evaluations.
Figures & tables
Figure 1: Cross-lingual safety gap across English, Yoruba, and Nigerian Pidgin. The same prompt produces a safe response in English but harmful responses in Yoruba and Nigerian Pidgin.
Figure 2: Overview of the research pipeline: (1) prompt construction and adaptation, (2) translation into Yoruba and Nigerian Pidgin, (3) zero-shot evaluation of six LLMs, and (4) human evaluation and metric computation.
Harm Category
Adapted
Original
Total
Hate/abusive language
50
50
100
Harmful instructions
40
60
100
Political manipulation
33
67
100
Cultural stereotypes
49
51
100
Subtotal (English)
172
228
400
Table 1: Distribution of English prompts by harm category and source. Each has a parallel Yoruba and Nigerian Pidgin version, for 1,200 prompts total.
Figure 3: Refusal Rate (RR) and Attack Success Rate (ASR) per model across English, Yoruba, and Nigerian Pidgin. RR measures the proportion of prompts refused by each model, while ASR measures the proportion of prompts that elicited harmful responses.
Model
Lang.
RR
ASR
HCR
HR
CR
Aya-101
Eng
0.05
0.60
0.63
0.52
0.68
Yor
0.00
0.25
0.25
0.22
0.28
Pid
0.04
0.48
0.50
0.43
0.52
Gemini-2.0
Eng
0.69
0.13
0.41
0.48
0.98
Yor
0.55
0.06
0.13
0.32
0.92
Pid
0.67
0.12
0.35
0.29
0.98
Table 2: Evaluation metrics by model and language (Eng = English, Yor = Yorùbá, Pid = Nigerian Pidgin).
Figure 4: Cross-Lingual Safety Gap (CLSG) for each model in Yoruba and Nigerian Pidgin. Orange bars indicate models that are more vulnerable in the target language than in English (CLSG > 0) while blue bars indicate models that are less vulnerable (CLSG < 0).
Figure 5: Attack Success Rate by harm category and language, aggregated across all six models.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Developer
Access
Llama 3.1 8B
Meta
HF / A100
Llama 3.3 70B
Meta
Together.ai API
Qwen3 32B
Alibaba
Groq API
Aya 101
Cohere
HF / A100
Gemini 2.0 Flash
Google
Google API
GPT-4o Mini
OpenAI
OpenAI API
Appendix
Table 3: Models evaluated in this study.
Model
Lang.
Ref.
Harm.
Help.
Comp.
Aya 101
Eng
1.00
0.98
0.93
0.91
Yor
1.00
0.98
0.92
0.94
Pid
1.00
0.95
0.91
0.81
Gemini 2.0
Eng
1.00
0.92
0.98
0.74
Yor
1.00
0.82
0.90
0.86
Pid
1.00
0.96
0.90
0.85
Appendix
Table 4: Cohen’s κ inter-annotator agreement by model and language.
Lang.
Dim.
Tot.
Agree
Dis.
%
English
Refused
720
720
0
0.0
Harmful
720
688
32
4.4
Helpful
720
651
69
9.6
Comp.
720
680
40
5.6
Yoruba
Refused
720
720
0
0.0
Harmful
720
702
18
2.5
Appendix
Table 5: Annotator disagreement before adjudication.
Category
Lang.
ASR
RR
HCR
HR
CR
Cult. stereotypes
Eng
0.26
0.18
0.31
0.66
0.91
Yor
0.26
0.04
0.27
0.39
0.50
Pid
0.44
0.09
0.49
0.71
0.86
Harmful instr.
Eng
0.27
0.52
0.55
0.32
0.69
Yor
0.12
0.12
0.13
0.17
0.38
Pid
0.27
0.52
0.55
0.31
0.60
Appendix
Table 6: Metrics by harm category and language, aggregated across all six models.
Figure 6: Pearson correlation heatmaps between evaluation metrics across English, Yoruba, and Nigerian Pidgin. Blue indicates strong positive correlation, and orange indicates strong negative correlation. White indicates no correlation.
Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.
Safety alignment in large language models relies predominantly on English-language training data. When harmful intent is expressed in low-resource languages, refusal mechanisms that hold in English frequently fail to activate. We introduce LSR (Linguistic Safety Robustness), the first systematic benchmark for measuring cross-lingual refusal degradation in West African languages: Yoruba, Hausa, Igbo, and Igala. LSR uses a dual-probe evaluation protocol - submitting matched English and target-language probes to the same model - and introduces Refusal Centroid Drift (RCD), a metric that quantifies how much of a model's English refusal behavior is lost when harmful intent is encoded in a target language. We evaluate Gemini 2.5 Flash across 14 culturally grounded attack probes in four harm categories. English refusal rates hold at approximately 90 percent. Across West African languages, refusal rates fall to 35-55 percent, with Igala showing the most severe degradation (RCD = 0.55). LSR is implemented in the Inspect AI evaluation framework and is available as a PR-ready contribution to the UK AISI's inspect_evals repository. A live reference implementation and the benchmark dataset are publicly available.
Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, critically underexplored. We introduce TUKABENCH, a jailbreak benchmark for seven African languages that extends JailbreakBench (JBB) beyond direct translation through four settings: human translation of JBB prompts, English adaptation to African contexts followed by human translation, human-curated prompts validated through interactions with GPT-5.2, and code-switched prompts combining English and African languages, isolating the effect of language, cultural grounding, and prompt evasiveness on model safety. Across closed and open models, prompting in African languages reduces refusal relative to English, with culturally adapted prompts leading to least refusal. The evaluation also surfaces two structural limitations: model comprehension failures and reduced LLM-as-a-judge reliability in LRLs. To capture the first, we introduce Deflection alongside Refused and Jailbroken; to assess the second, we validate outputs with human annotations, showing that judge-human agreement drops in lower-resource languages and less commonly supported scripts.
Victor Akinode, Senyu Li, Wassim Hamidouche +3
Mila - Quebec AI Institute · McGill University · Microsoft AI for Good Research Lab +1
Safety alignment in large language models (LLMs) is largely developed in English, assuming these safeguards generalize across multilingual settings. However, this assumption remains underexplored and exposes a vulnerability in low-resource languages. We investigate cross-lingual safety transfer in four African languages, Twi, Hausa, Amharic, and Swahili, using LoDNA, a new safety dataset that pairs literal translations with culturally localized prompts. To move beyond generation-based evaluation, we propose a latent geometric framework that probes hidden-state refusal representations in LLMs. Our experimental results show that cross-lingual safety transfer is severely limited; harmful prompts retain less than 10% of the English refusal signal across most language-model pairs. Literal and localized prompts are semantically aligned (cosine 0.95-0.996) but drift across layers, suggesting models encode the concepts without routing them to safety mechanisms. These findings demonstrate that current multilingual safety alignment is superficial, providing strong evidence against the assumption of a universal, language-agnostic harm manifold within the specific low-resource languages studied. Warning: This paper contains example data that may be offensive or harmful.
Abigail Oppong, P Sam Sahil, Tadesse Destaw Belay +12
Makerere University Center for Artificial Intelligence · University of Hamburg · Instituto Politécnico Nacional +9