Large language model providers routinely cite multilingual safety benchmarks spanning a dozen or more languages as evidence that their models are safe for non-English-speaking users. We show that these collection-level coverage claims frequently do not survive inspection at the level of an individual language. Auditing 21 resources across 25 language slices, of which 20 count as datasets under our counting rules, spanning three languages chosen to represent low- (Hausa), mid- (Swahili), and high-resource (French) tiers, we find that gaps in provenance, annotation reliability, access, harm-taxonomy coverage, and data reuse recur in patterns that partially, but not fully, track resource level. Using a controlled within-pipeline comparison, we show a Hausa-language slice falling below its own paper's translation-quality acceptance threshold while the same pipeline's Swahili output clears the same bar comfortably; this is evidence that these gaps are measurable and addressable, not inherent. We further show that self-harm and sexual-content categories have no native-language coverage in either African-language tier we studied, a total rather than gradated gap that a purely resource-level account does not predict. We connect these findings to a documented, persistent asymmetry in multilingual jailbreak robustness (single-turn attacks largely mitigated, multi-turn attacks still effective), arguing that this asymmetry is structurally consistent with where our audit finds training and evaluation data thinnest. We contribute a reusable slice-level audit methodology, a cross-tier empirical comparison, and concrete recommendations for dataset creators, model providers, and venues aiming to make ``multilingual coverage'' claims verifiable rather than merely stated. Dataset: https://huggingface.co/datasets/ChialukaOnuoha/safety-slice-audit
Figures & tables
Field group
Fields
What it establishes
Identification
Dataset name; year/version; source paper (DOI/arXiv); dataset link (repository); authors; model(s) evaluated
That the artifact is uniquely identifiable and that the paper and the data are separately locatable
Language coverage
Language(s) covered; resource level; monolingual vs. multilingual
Whether the target language is a first-class subject or one slice of a larger collection
How this language’s items were actually produced, evidenced by the paper’s own methodology section
Harm coverage
Harm categories covered; harm category gaps; native taxonomy (verbatim); global vs. local harm
Coverage against our six-category framework, and the authors’ own conceptualisation before mapping
Annotation & culture
Who annotated it; native review of this language; annotator count and background; IAA for this slice; dialect and cultural handling; script; code-switching coverage
Evidentiary strength of the labels, and whether cultural and orthographic variation is modelled
Artifact characteristics
Format/structure (prompt-only, multi-turn, prompt+response, blocklist); size for this language; domain/source; entries unit
Fitness for a given downstream use, and slice magnitude rather than collection magnitude
Table 1: The audit instrument, grouped by field family. Each of the thirty-one fields is applied to every language slice, and each field carries a one-sentence rationale in the instrument itself so that the schema is self-documenting.
Harm category
Working definition adopted for this study
Representative native labels mapped in
Hate speech
Content attacking or demeaning a person or group on the basis of a protected or group identity (ethnicity, religion, nationality, gender, disability, political-communal identity).
Content promoting violent ideological action, violent organisations, or incitement to communal or political violence.
XTREMESPEECH “dangerous speech” (incitement to communal or political violence); ALERT/Aegis extremism hazards
Sexual content
Explicit sexual material, including sexualised depictions of minors and non-consensual sexual content.
RTP-LX “sexual content”; Aya “non-consensual sexual content”; JBB “sexual/adult”; Aegis sexual hazards
Misinformation
Verifiably false or fabricated claims presented as fact, including political disinformation and health misinformation.
PolitiKweli “fake” vs. “fact”; UbuntuGuard “misinformation and disinformation” theme; JBB “disinformation”; ALERT misinformation categories
Table 2: The six-category harm framework, the working definition adopted for this study, and representative native labels mapped into each category. Mappings are recorded per slice in the audit instrument alongside the verbatim native taxonomy.
Dataset (slice)
Tier
Provenance of slice
Format
Slice size / key figure
HAUSA (low)
AfriHate ( Muhammad et al., 2025 )
HA
Native-authored
Labelled posts
6,644 tweets; κ=0.75
AfriSenti ( Muhammad et al., 2023a )
HA
Native-authored
Labelled tweets
22,155 tweets (14,173 / 2,678 / 5,304); κ=0.66
HOC ( Adam et al., 2023 )
HA
Native-authored
Labelled posts
Size not reported; no IAA
NaijaOffens ( Aliyu et al., 2024 )
HA
Native-authored
Labelled tweets
Unreleased; size not reported
TukaBench ( Akinode et al., 2026 )
HA
MT + native post-edit
Prompt-only
986 described / 300 released; judge–human agreement 69.4%
Table 3: The audited corpus by resource tier. Provenance and size are recorded for the language slice, not the parent collection. Where a paper and a live release disagree on size, both are shown. The Ubisoft/ToxBuster resource is reviewed and excluded (§ 5.4 ) and does not appear here or in any count.
Tier
Nat.
Trans.
Transl.
Synth.
Slices
Hausa (low)
5
0
1
1
7
Swahili (mid)
5
1
3
1
10
French (high)
3
1
3
1
8
Table 4: Provenance composition by tier. Counts are slices, not datasets. “Trans.” denotes transcreated (translation with native-speaker cultural adaptation), distinguished from plain human “Transl.” translation because the source papers themselves draw that distinction.
Gap category
Hausa (low)
Swahili (mid)
French (high)
Provenance gaps
Severe: all multi-harm data translated or synthetic
Moderate: native present but single-domain
Mild: native red-team data exists, but breadth is translated
Annotation / agreement gaps
Mixed: strong where reported ( κ 0.66–0.75), absent in half
Severe: lowest agreement observed ( κ 0.13–0.55)
Moderate: slice-level human agreement printed for few French slices (PolyGuard α≈0.47 ); most report none
Not measurable: the one within-pipeline comparison (§ 6.2 ) does not include French
Table 5: Gap severity by category and tier. Severity levels are assigned per slice against the anchoring criteria in § 4 and aggregated to the tier by the modal and worst-case slice.
Harm category
Hausa (low)
Swahili (mid)
French (high)
Hate speech
Native, multi-dataset
Native, multi-dataset
Native + translated
Harassment
Native (“abusive”, “offensive”)
Native (“abusive”, derogatory)
Native + translated
Self-harm
None
Transcreated only
Native (Aya) + translated
Extremism
None
Native (partial: incitement)
Translated only
Sexual content
Translated only
Transcreated only
Native (Aya) + translated
Misinformation
Synthetic / translated only
Native (PolitiKweli)
Translated only
Table 6: Harm coverage by category and tier, distinguishing native from translated and transcreated provenance. “Native” requires that items in the target language were written or organically collected in that language, not translated into it.
Large language model safety benchmarks test whether AI systems avoid harm while remaining helpful across realistic tasks, users, and deployment settings.
Large language models are increasingly deployed across professional domains, bringing hard-to-predict risks, including the generation of harmful or disrespectful content. Although substantial progress has been made in developing safety evaluation datasets, existing resources remain overwhelmingly English- and Chinese-centric. This limitation is particularly pronounced when evaluating languages that operate within shared sociocultural, legal, and ethical contexts. To address this gap, we introduce Schützen: a German--Bulgarian safety dataset designed to assess model answerability under risk, covering both a low-resource language (Bulgarian) and a high-resource language (German). Experiments with multilingual and language-specific LLMs reveal pronounced cross-language differences in safety behavior, highlighting the necessity of tailored, region-specific evaluation resources to support the responsible deployment of LLMs in Germany and Bulgaria. Datasets and code are available at https://github.com/xnlp-lab/Schutzen. Warning: this paper contains examples that may be offensive, harmful, or biased.
Safety alignment in large language models (LLMs) is largely developed in English, assuming these safeguards generalize across multilingual settings. However, this assumption remains underexplored and exposes a vulnerability in low-resource languages. We investigate cross-lingual safety transfer in four African languages, Twi, Hausa, Amharic, and Swahili, using LoDNA, a new safety dataset that pairs literal translations with culturally localized prompts. To move beyond generation-based evaluation, we propose a latent geometric framework that probes hidden-state refusal representations in LLMs. Our experimental results show that cross-lingual safety transfer is severely limited; harmful prompts retain less than 10% of the English refusal signal across most language-model pairs. Literal and localized prompts are semantically aligned (cosine 0.95-0.996) but drift across layers, suggesting models encode the concepts without routing them to safety mechanisms. These findings demonstrate that current multilingual safety alignment is superficial, providing strong evidence against the assumption of a universal, language-agnostic harm manifold within the specific low-resource languages studied. Warning: This paper contains example data that may be offensive or harmful.
Abigail Oppong, P Sam Sahil, Tadesse Destaw Belay +12
Makerere University Center for Artificial Intelligence · University of Hamburg · Instituto Politécnico Nacional +9
Safety alignment in large language models relies predominantly on English-language training data. When harmful intent is expressed in low-resource languages, refusal mechanisms that hold in English frequently fail to activate. We introduce LSR (Linguistic Safety Robustness), the first systematic benchmark for measuring cross-lingual refusal degradation in West African languages: Yoruba, Hausa, Igbo, and Igala. LSR uses a dual-probe evaluation protocol - submitting matched English and target-language probes to the same model - and introduces Refusal Centroid Drift (RCD), a metric that quantifies how much of a model's English refusal behavior is lost when harmful intent is encoded in a target language. We evaluate Gemini 2.5 Flash across 14 culturally grounded attack probes in four harm categories. English refusal rates hold at approximately 90 percent. Across West African languages, refusal rates fall to 35-55 percent, with Igala showing the most severe degradation (RCD = 0.55). LSR is implemented in the Inspect AI evaluation framework and is available as a PR-ready contribution to the UK AISI's inspect_evals repository. A live reference implementation and the benchmark dataset are publicly available.