Hard Negatives Reveal What Easy Negatives Hide: Cross-Lingual Harmfulness Representations Degrade with Resource Tier Under Hard Negatives
Authors: Paras Balani, Subhrakanta Panda
Organizations: Department of Mathematics and Department of Computer Science, Birla Institute of Technology and Science, Pilani, Hyderabad Campus, Jawahar Nagar, Kapra Mandal, Medchal District, Telangana 500078, India · Department of Computer Science, Birla Institute of Technology and Science, Pilani, Hyderabad Campus, Jawahar Nagar, Kapra Mandal, Medchal District, Telangana 500078, India
Safety alignment in large language models is trained primarily in English, and recent work reports that the underlying harmfulness representation survives translation: English-trained probes separate harmful from harmless prompts almost as well in low-resource languages as in English. This has been taken as evidence that cross-lingual refusal failures mainly reflect calibration rather than representation quality. We show that this conclusion depends on the choice of negative examples. Across nine languages spanning three resource tiers, we replicate near-perfect transfer (AUROC > 0.98) when harmless prompts come from an unrelated distribution (easy negatives). With XSTest contrast prompts, which are benign but surface-similar to harmful requests (hard negatives), transfer collapses in low-resource languages while remaining largely stable in high-resource languages. On Qwen2.5-7B-Instruct, mean AUROC drop increases from 0.003 in English to 0.017 in high-resource, 0.042 in mid-resource, and 0.276 in low-resource languages. The pattern replicates on Aya Expanse. Back-translation chrF controls and a matched-chrF comparison across three languages reduce the likelihood that translation quality explains the effect. The collapse remains after controlling for chrF (partial r = 0.70, p = 0.03). Tokenizer fertility correlates with the collapse and explains part of the resource-tier effect, but not all of it. The results show that easy-negative transfer can coexist with substantial degradation under hard negatives. Easy-negative evaluation alone therefore cannot establish that the harmfulness representation survives translation.
Figures & tables
Figure 1: Qwen2.5-7B-Instruct: transfer AUROC under easy negatives (AdvBench vs. Alpaca) and hard negatives (XSTest contrast prompts), by language and resource tier.
Language
Tier
chrF
Easy
Hard
Drop
English
source
100.0
1.000
0.997
0.003
French
high
72.8
1.000
0.993
0.007
Spanish
high
76.6
1.000
0.990
0.010
Vietnamese
mid
67.6
1.000
0.981
0.019
Mandarin
high
62.4
0.999
0.964
0.035
Arabic
mid
67.7
1.000
0.961
0.039
Table 1: Qwen2.5-7B-Instruct: transfer AUROC under easy negatives and hard negatives, by language and resource tier. Easy peak layer 13; hard peak layer 23.
Figure 2: Easy-to-hard AUROC drop by language, Qwen2.5-7B-Instruct and Aya-Expanse-8B. The tier ordering and approximate drop magnitude replicate across model families.
Language
Tier
chrF
Easy
Hard
Drop
English
source
100.0
1.000
0.998
0.002
French
high
72.8
0.989
0.995
-0.005
Spanish
high
76.6
0.999
0.992
0.007
Vietnamese
mid
67.6
1.000
0.990
0.010
Hindi
mid
72.0
0.999
0.980
0.019
Mandarin
high
62.4
0.997
0.974
0.023
Table 2: Aya-Expanse-8B: transfer AUROC under easy negatives and hard negatives, by language and resource tier. Easy peak layer 3; hard peak layer 20.
Figure 3: AUROC drop against back-translation chrF, Qwen (left) and Aya (right). Amharic sits alone at low chrF and high drop; the remaining eight languages show no clear relationship between translation quality and collapse.
Figure 4: AUROC drop against tokenizer fertility. High-fertility languages cluster at high drop; Hindi is an outlier, with fertility comparable to Amharic but a far smaller drop.
Language models often answer harmful requests in low-resource languages (LRLs) that they refuse in high-resource languages (HRLs). Across three instruction-tuned models and 23 languages, harmful refusal falls from 87.9% in HRLs to 43.9% in LRLs, while harmless refusal remains low. A common explanation is that models represent harmfulness weakly in LRLs. We test whether harmfulness is instead represented but does not reliably produce refusal. Across three models, a harmfulness direction learned from HRL activations still separates harmful from harmless LRL prompts, showing that complete absence of harmfulness information cannot explain many failures. However, harmfulness scores shift downward for LRL prompts, making harmful prompts less likely to reach the range associated with refusal. Motivated by this shift, we train a low-rank logistic classifier on HRL activations and calibrate its threshold with a few target-language examples. During generation, the classifier conditionally adds or ablates the HRL harmfulness direction. With the same HRL data and 32 target-language examples per class, CAST remains limited by low harmful refusal and AdaSteer by high harmless refusal, yielding mean refusal selectivity (Δ = harmful - harmless refusal) of 33.6 and 6.8, respectively. Our intervention reaches 54.5 while preserving MMLU utility. HRL-only calibration improves selectivity for Qwen and Gemma, whereas Llama benefits from target-language calibration. These results show that recalibrating existing representations can offer a training-free method for repairing low-resource safety failures.
Rashad Aziz, Ikhlasul Akmal Hanif, Fajri Koto
Mohamed bin Zayed University of Artificial Intelligence
Safety alignment in large language models (LLMs) is largely developed in English, assuming these safeguards generalize across multilingual settings. However, this assumption remains underexplored and exposes a vulnerability in low-resource languages. We investigate cross-lingual safety transfer in four African languages, Twi, Hausa, Amharic, and Swahili, using LoDNA, a new safety dataset that pairs literal translations with culturally localized prompts. To move beyond generation-based evaluation, we propose a latent geometric framework that probes hidden-state refusal representations in LLMs. Our experimental results show that cross-lingual safety transfer is severely limited; harmful prompts retain less than 10% of the English refusal signal across most language-model pairs. Literal and localized prompts are semantically aligned (cosine 0.95-0.996) but drift across layers, suggesting models encode the concepts without routing them to safety mechanisms. These findings demonstrate that current multilingual safety alignment is superficial, providing strong evidence against the assumption of a universal, language-agnostic harm manifold within the specific low-resource languages studied. Warning: This paper contains example data that may be offensive or harmful.
Abigail Oppong, P Sam Sahil, Tadesse Destaw Belay +12
Makerere University Center for Artificial Intelligence · University of Hamburg · Instituto Politécnico Nacional +9
Safety alignment in large language models relies predominantly on English-language training data. When harmful intent is expressed in low-resource languages, refusal mechanisms that hold in English frequently fail to activate. We introduce LSR (Linguistic Safety Robustness), the first systematic benchmark for measuring cross-lingual refusal degradation in West African languages: Yoruba, Hausa, Igbo, and Igala. LSR uses a dual-probe evaluation protocol - submitting matched English and target-language probes to the same model - and introduces Refusal Centroid Drift (RCD), a metric that quantifies how much of a model's English refusal behavior is lost when harmful intent is encoded in a target language. We evaluate Gemini 2.5 Flash across 14 culturally grounded attack probes in four harm categories. English refusal rates hold at approximately 90 percent. Across West African languages, refusal rates fall to 35-55 percent, with Igala showing the most severe degradation (RCD = 0.55). LSR is implemented in the Inspect AI evaluation framework and is available as a PR-ready contribution to the UK AISI's inspect_evals repository. A live reference implementation and the benchmark dataset are publicly available.