Safety alignment research for large language models (LLMs) remains concentrated on high-resource languages, leaving open how existing alignment methods perform in low-resource, culturally distinct settings. We study this question for Hausa, a language spoken by more than 200 million people across West Africa. We introduce HausaSafeLM, one of the first Hausa safety preference dataset, built from 10,000 prompts derived from PKU-SafeRLHF: 3,000 human-corrected (gold) and 7,000 machine-translated (silver) examples spanning six harm macro-categories. Using Qwen2.5-7B-Instruct, we compare supervised fine-tuning (SFT) and SFT followed by Direct Preference Optimization (DPO) under both data regimes, evaluating on 500 harmful and 200 benign Hausa prompts.We find that the untuned base model appears highly safe by refusal-rate metrics alone, but this reflects limited Hausa generation ability rather than genuine alignment: its refusals are almost uniformly low quality. SFT improves both refusal fluency and precision and benefits substantially from the larger, noisier silver-tier data, while DPO sharpens refusal precision but comes at a steep, largely data-invariant cost to linguistic quality. No single configuration jointly maximizes safety, helpfulness, and fluency. Our results suggest that current alignment techniques teach models when to refuse more readily than how to do so fluently in underrepresented languages, and that refusal-rate metrics alone are an unreliable signal of safety in low-resource settings.
Figures & tables
Figure 1: HausaSafeLM data collection pipeline. The filtered PKU-SafeRLHF pool is split into a gold tier (3,000 examples, machine-translated and human post-edited across a four-pass fluency/meaning/register/localization protocol) and a silver tier (7,000 examples, machine-translated only), which are combined into the master dataset and exported into SFT and DPO training formats.
Macro-category
PKU-SafeRLHF sources
Violence & physical harm
Violence, Physical Harm, Disrupting Public Order
Hate speech & discrimination
Insulting Behavior, Discriminatory Behavior
Sexual content
Sexual Content
Criminal activity
Economic Crime, Cybercrime, Endangering National Security, Human Trafficking
Health & public safety
Endangering Public Health, Drugs, Environmental Damage
Psychological manipulation
Psychological Harm, Mental Manipulation
Table 1: HausaSafeLM harm taxonomy: six macro-categories derived from 15 retained PKU-SafeRLHF categories.
Figure 2: Experimental framework. The base model is evaluated zero-shot, then fine-tuned with LoRA under the gold and gold+silver data tiers (Stage 2, SFT), followed by DPO on top of each SFT checkpoint (Stage 3), producing five evaluated variants (Base, SFT-G, SFT-GS, DPO-G, DPO-GS). All variants are scored with the same evaluation protocol.
Figure 3: Base model behavior on harmful Hausa prompts. Left: Refusal Rate (RR), Attack Success Rate (ASR), False Refusal Rate (FRR), and Helpfulness Rate (HR). Right: Gemini-judged response quality distribution (HIGH/MEDIUM/LOW) across the 500 harmful Hausa safety prompts.
Figure 4: SFT vs. SFT+DPO for both data regimes. Each row represents one data regime (gold-only, top; gold+silver, bottom). Columns show Refusal Rate (RR), False Refusal Rate (FRR), Helpfulness Rate (HR), and Safety Quality Score (SafeQ). Δ values denote the change from SFT to SFT+DPO within each regime.
Figure 5: Response quality distribution across all five configurations and both prompt types. Stacked bars show the proportion of HIGH (green), MEDIUM (yellow), and LOW (red) quality responses. Left: the 500 harmful Hausa safety prompts. Right: the 200 benign Aya Hausa prompts.
Figure 6: Multi-dimensional trade-off radar across all five configurations. Every axis is oriented so that a larger polygon indicates better performance: RR (safety), HR (helpfulness), 1− FRR (refusal precision), SafeQ and AyaQ (linguistic quality on harmful and benign prompts, respectively).
Figure 7: Effect of adding Silver data, by training stage. Top : SFT stage — SFT-G (Gold only) versus SFT-GS (Gold+Silver). Bottom : DPO stage — DPO-G (Gold only) versus DPO-GS (Gold+Silver). Columns show RR, FRR, HR, and Safety Quality Score in order. Δ values are colour-coded: green indicates that adding Silver data improves the metric, red that it degrades it.
Variant
RR ↑
FRR ↓
HR ↑
SafeQ ↑
AyaQ ↑
Base
1.000
0.096
0.256
0.021
0.165
SFT-G
1.000
0.405
0.180
0.238
0.2025
DPO-G
1.000
0.110
0.255
0.023
0.1425
SFT-GS
1.000
0.245
0.260
0.350
0.2675
DPO-GS
1.000
0.110
0.235
0.036
0.1325
Table 3: Main results across all five configurations, organized around RQ1–RQ4. RR saturates at 1.000 for every configuration and is retained only to make this ceiling effect explicit (RQ1); FRR, HR, and quality scores are the discriminating metrics (RQ2–RQ4). AyaQ values for the four trained variants are pending confirmation against raw evaluation logs (§ Limitations ).
Safety alignment in large language models (LLMs) is largely developed in English, assuming these safeguards generalize across multilingual settings. However, this assumption remains underexplored and exposes a vulnerability in low-resource languages. We investigate cross-lingual safety transfer in four African languages, Twi, Hausa, Amharic, and Swahili, using LoDNA, a new safety dataset that pairs literal translations with culturally localized prompts. To move beyond generation-based evaluation, we propose a latent geometric framework that probes hidden-state refusal representations in LLMs. Our experimental results show that cross-lingual safety transfer is severely limited; harmful prompts retain less than 10% of the English refusal signal across most language-model pairs. Literal and localized prompts are semantically aligned (cosine 0.95-0.996) but drift across layers, suggesting models encode the concepts without routing them to safety mechanisms. These findings demonstrate that current multilingual safety alignment is superficial, providing strong evidence against the assumption of a universal, language-agnostic harm manifold within the specific low-resource languages studied. Warning: This paper contains example data that may be offensive or harmful.
Abigail Oppong, P Sam Sahil, Tadesse Destaw Belay +12
Makerere University Center for Artificial Intelligence · University of Hamburg · Instituto Politécnico Nacional +9
Safety alignment in large language models relies predominantly on English-language training data. When harmful intent is expressed in low-resource languages, refusal mechanisms that hold in English frequently fail to activate. We introduce LSR (Linguistic Safety Robustness), the first systematic benchmark for measuring cross-lingual refusal degradation in West African languages: Yoruba, Hausa, Igbo, and Igala. LSR uses a dual-probe evaluation protocol - submitting matched English and target-language probes to the same model - and introduces Refusal Centroid Drift (RCD), a metric that quantifies how much of a model's English refusal behavior is lost when harmful intent is encoded in a target language. We evaluate Gemini 2.5 Flash across 14 culturally grounded attack probes in four harm categories. English refusal rates hold at approximately 90 percent. Across West African languages, refusal rates fall to 35-55 percent, with Igala showing the most severe degradation (RCD = 0.55). LSR is implemented in the Inspect AI evaluation framework and is available as a PR-ready contribution to the UK AISI's inspect_evals repository. A live reference implementation and the benchmark dataset are publicly available.
Large language models (LLMs) often demonstrate strong safety performance in high-resource languages, yet exhibit severe vulnerabilities when queried in low-resource languages. We attribute this gap to a mismatch between language-agnostic semantic understanding ability and language-dominant safety alignment biased toward high-resource languages. Consistent with this hypothesis, we empirically identify the semantic bottleneck in LLMs, an intermediate layer in which the geometry of model representations is governed primarily by shared semantic content rather than language identity. Building on this observation, we propose Language-Agnostic Semantic Alignment (LASA), which anchors safety alignment directly in semantic bottlenecks. Experiments show that LASA substantially improves safety across all languages: average attack success rate (ASR) drops from 24.7% to 2.8% on LLaMA-3.1-8B-Instruct and remains around 3-4% across Qwen2.5 and Qwen3 Instruct models (7B-32B). Together, our analysis and method offer a representation-level perspective on LLM safety, suggesting that safety alignment requires anchoring safety understanding not in surface text, but in the model's language-agnostic semantic space.
Junxiao Yang, Haoran Liu, Jinzhe Tu +9
The Conversational AI (CoAI) group, DCST, Tsinghua University · Alibaba Group · Tsinghua University