Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation
Authors: Zahraa Al Sahili, Ioannis Patras, Matthew Purver
Organizations: School of Electronic Engineering and Computer Science, Queen Mary University of London, London, UK · Department of Knowledge Technologies, Jožef Stefan Institute, Ljubljana, Slovenia
Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in English. Translating existing benchmarks offers a scalable approach to multilingual evaluation, but preserving task-specific meaning across languages presents significant challenges, particularly for harmful requests, mathematical reasoning, and conversational interactions. We introduce a reasoning-guided translation framework for constructing multilingual evaluations designed to preserve the structure and intended meaning of diverse benchmark tasks. Our approach separates translation strategy selection from large-scale generation, combining reasoning-guided configuration selection with task-aware validation, targeted repair, and independent quality screening. Using this framework, we develop a source-aligned evaluation collection spanning seven safety, conversational, and mathematical reasoning resources across 11 languages. The resulting collection achieves 99.94% coverage of intended item--language pairs in the two subsets requiring repair. Evaluations of open-source LLMs reveal distinct language-dependent vulnerabilities, including inconsistent refusal behaviour and substantial reasoning degradation in lower-resource languages. These findings highlight the importance of task-sensitive benchmark construction and systematic quality assessment for understanding multilingual model behaviour beyond English.
Figures & tables
Figure 1: Overview of the reasoning-guided multilingual benchmark construction framework. (i) Seven English resources are adapted into 11 target languages while retaining source identifiers and task-specific annotations. (ii) A reasoning model proposes and evaluates translation configurations before large-scale generation. (iii) Source-aware validation and targeted repair address structural failures, while unresolved cases are explicitly recorded.
Source
Evaluation use case
StrongREJECT
Harmful-request handling
JBB-Behaviors
Jailbreak behavior coverage
WildChat
Real-world conversations
WildJailbreak
Adversarial/benign safety prompts
XSTest
Over-refusal and unsafe contrasts
MATH-500
Mathematical reasoning
Table 1: Seven English-language resources selected for translation into 11 target languages. Each resource addresses a distinct evaluation objective; WildChat provides naturalistic conversations rather than a dedicated safety benchmark.
Translator
P1
P2
P3
Gemini 2.5 Flash
70/70
69/70
68/70
Gemini 2.5 Flash-Lite
65/70
59/70
66/70
GPT-4o
68/70
65/70
66/70
GPT-5-mini
67/70
68/70
66/70
GPT-5.1
69/70
70/70
67/70
Table 2: Structurally valid translations across 15 model–instruction configurations, evaluated on 70 Thai instances sampled equally from seven resources. Bold indicates the selected production configuration rather than the highest validity.
Source
Expected
Accepted
Coverage
WildChat
4,070
4,066
99.90%
WildJailbreak
2,750
2,750
100.00%
Total
6,820
6,816
99.94%
Table 3: Operational validation coverage for the WildChat and WildJailbreak subsets following repair. Expected counts correspond to 370 WildChat conversations and 250 WildJailbreak prompts across 11 target languages. Four unresolved WildChat translations are excluded.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Mean CometKiwi score of the evaluation field per source and target language. WildChat is omitted because its long, multi-paragraph turns fall outside the sentence-level regime of the QE model.
Source
QE
LaBSE
BT-chrF
BT-sim
Mean.
Pres.
StrongREJECT
.854
.862
68.5
.934
4.98
99.4
JBB-Behaviors
.858
.841
70.6
.931
5.00
99.9
XSTest
.858
.881
68.9
.950
4.99
99.7
WildJailbreak
.828
.884
63.9
.950
4.99
99.7
WildChat
–
.883
56.1
.921
4.92
98.6
MATH-500
.821
.923
68.6
.923
5.00
100.0
Appendix
Table 4: Fidelity signals per source, averaged over 11 languages. QE: CometKiwi; LaBSE: source–translation similarity; BT-chrF / BT-sim: chrF++ and LaBSE similarity between the NLLB back-translation and the English source; Mean.: judge-rated meaning preservation (1–5); Pres.: % of translations judged to preserve intent (safety and conversation) or all mathematical content (math).
Language
Agree w/ EN
Harmful → benign
Gold acc.
English
–
–
88.7
Arabic
91.5
8.1
86.0
Bengali
91.3
8.5
85.9
Spanish
94.2
5.6
88.4
Persian
91.2
8.6
86.5
French
93.4
5.5
87.6
Appendix
Table 5: Label preservation (%) under independent harmful/benign judgments by Gemma-3-12B-it. Gold acc.: agreement with the original dataset labels, which indicates the judge’s own error rate.
Figure 3: Change relative to English (percentage points) on identical items. Colour encodes the desirable direction (red: worse than English); ∗ and bold mark p<0.05 (paired bootstrap).
Metric
Model
en
ar
bn
es
fa
fr
hi
pl
ru
th
tr
zh
StrongREJECT score ↓
Llama-3.1-8B
1.5
1.9
2.5
1.8
4.9 ∗
0.8
2.6
4.2
1.2
4.9 ∗
8.0 ∗
2.5
Qwen2.5-7B
6.5
5.0
1.4 ∗
5.0
6.5
5.1
2.4
4.6
3.6 ∗
5.1
4.0
5.3
JBB harmful refusal ↑
Llama-3.1-8B
94.0
85.0 ∗
82.0 ∗
90.9
82.0 ∗
87.0 ∗
84.0 ∗
87.0
87.0 ∗
82.0 ∗
74.0 ∗
87.0 ∗
Qwen2.5-7B
91.0
88.0
84.0
88.9
86.0
87.0
82.0 ∗
87.0
88.0
88.0
83.0 ∗
92.0
XSTest unsafe refusal ↑
Llama-3.1-8B
97.5
88.4 ∗
80.6 ∗
88.3 ∗
84.8 ∗
93.9
86.4 ∗
84.4 ∗
93.7
84.1 ∗
69.5 ∗
91.2 ∗
Qwen2.5-7B
81.5
82.8
83.2
81.7
76.6
82.7
83.9
80.7
83.1
82.1
75.1 ∗
80.8
Appendix
Table 6: Downstream results on identical items in every language. StrongREJECT score is the mean harmfulness score ×100 ; all other rows are percentages. ∗ : significant difference from English (paired bootstrap, p<0.05 ). Arrows give the desirable direction. Items: 100 StrongREJECT, 100 harmful JBB, 200 unsafe and 250 safe XSTest, 90 adversarial-harmful WildJailbreak, and 100 MATH-500 problems; item–language pairs flagged as untranslated (Appendix A ) are excluded.
Language
WildChat
WildJailbreak
Arabic
370/370
250/250
French
370/370
250/250
Hindi
370/370
250/250
Chinese
370/370
250/250
Russian
370/370
250/250
Thai
370/370
250/250
Appendix
Table 7: Accepted/source items per language after repair. Four WildChat items are unresolved.
Translator
Med. s
Out
Reas.
$/1k
Full $
Valid
Gem. 2.5 Flash-Lite
1.2
389
0
0.21
5
84.3
GPT-5-mini
10.9
2,034
1,495
4.20
97
97.1
Gem. 2.5 Flash
4.3
1,666
1,177
4.33
103
98.6
GPT-5.1
3.4
520
0
5.85
145
100.0
GPT-4o
2.4
519
0
6.51
163
92.9
Appendix
Table 8: Speed and cost under P2 (14 items per translator, Thai). Med. s: median latency per item; Out / Reas.: mean output tokens per item and the reasoning tokens included in them; /1k:costper1,000items;Full: projected cost of translating the full collection into 11 languages; Valid: % structurally valid translations in the configuration study ( n=70 ).
Safety alignment in large language models remains brittle across languages: prompts reliably refused in English can elicit harmful compliance in non-English and low-resource settings. We introduce Minionese, a multilingual jailbreak benchmark spanning 18 languages, 4 resource tiers, and 4 perturbation types (standard translation, code-switching, transliteration, and translationese), paired with a geometric mechanistic analysis of refusal failure across language tiers. We show that each attack type produces a distinct vulnerability profile: transliteration vulnerability is mediated by script identity, code-switching maintains effectiveness through the lowest-resource tier, and a sharp safety regime transition between Tiers 2 and 3 is consistent across all models. Mechanistically, low-resource jailbreaks succeed by routing harmful content through a geometrically misaligned subspace that projects insufficiently onto the refusal directions, leaving the refusal mechanism intact but untriggered. These findings show that English-only safety evaluations are insufficient; they require accounting for script family, perturbation type, and per-language alignment coverage. The benchmark and analysis code is at https://github.com/Brentkong/Minionese-Comprehensive-Benchmark-and-Mechanistic-Study-of-Multilingual-LLM-Safety.git.
Safety alignment in large language models (LLMs) is largely developed in English, assuming these safeguards generalize across multilingual settings. However, this assumption remains underexplored and exposes a vulnerability in low-resource languages. We investigate cross-lingual safety transfer in four African languages, Twi, Hausa, Amharic, and Swahili, using LoDNA, a new safety dataset that pairs literal translations with culturally localized prompts. To move beyond generation-based evaluation, we propose a latent geometric framework that probes hidden-state refusal representations in LLMs. Our experimental results show that cross-lingual safety transfer is severely limited; harmful prompts retain less than 10% of the English refusal signal across most language-model pairs. Literal and localized prompts are semantically aligned (cosine 0.95-0.996) but drift across layers, suggesting models encode the concepts without routing them to safety mechanisms. These findings demonstrate that current multilingual safety alignment is superficial, providing strong evidence against the assumption of a universal, language-agnostic harm manifold within the specific low-resource languages studied. Warning: This paper contains example data that may be offensive or harmful.
Abigail Oppong, P Sam Sahil, Tadesse Destaw Belay +12
Makerere University Center for Artificial Intelligence · University of Hamburg · Instituto Politécnico Nacional +9
Large language models are increasingly deployed across professional domains, bringing hard-to-predict risks, including the generation of harmful or disrespectful content. Although substantial progress has been made in developing safety evaluation datasets, existing resources remain overwhelmingly English- and Chinese-centric. This limitation is particularly pronounced when evaluating languages that operate within shared sociocultural, legal, and ethical contexts. To address this gap, we introduce Schützen: a German--Bulgarian safety dataset designed to assess model answerability under risk, covering both a low-resource language (Bulgarian) and a high-resource language (German). Experiments with multilingual and language-specific LLMs reveal pronounced cross-language differences in safety behavior, highlighting the necessity of tailored, region-specific evaluation resources to support the responsible deployment of LLMs in Germany and Bulgaria. Datasets and code are available at https://github.com/xnlp-lab/Schutzen. Warning: this paper contains examples that may be offensive, harmful, or biased.