Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation

Zahraa Al Sahili; Ioannis Patras; Matthew Purver

cs.CLOct 14, 2026

Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation

Authors: Zahraa Al Sahili, Ioannis Patras, Matthew Purver

Organizations: School of Electronic Engineering and Computer Science, Queen Mary University of London, London, UK · Department of Knowledge Technologies, Jožef Stefan Institute, Ljubljana, Slovenia

Abstract

Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in English. Translating existing benchmarks offers a scalable approach to multilingual evaluation, but preserving task-specific meaning across languages presents significant challenges, particularly for harmful requests, mathematical reasoning, and conversational interactions. We introduce a reasoning-guided translation framework for constructing multilingual evaluations designed to preserve the structure and intended meaning of diverse benchmark tasks. Our approach separates translation strategy selection from large-scale generation, combining reasoning-guided configuration selection with task-aware validation, targeted repair, and independent quality screening. Using this framework, we develop a source-aligned evaluation collection spanning seven safety, conversational, and mathematical reasoning resources across 11 languages. The resulting collection achieves 99.94% coverage of intended item--language pairs in the two subsets requiring repair. Evaluations of open-source LLMs reveal distinct language-dependent vulnerabilities, including inconsistent refusal behaviour and substantial reasoning degradation in lower-resource languages. These findings highlight the importance of task-sensitive benchmark construction and systematic quality assessment for understanding multilingual model behaviour beyond English.

Figures & tables

Figure 1: Overview of the reasoning-guided multilingual benchmark construction framework. (i) Seven English resources are adapted into 11 target languages while retaining source identifiers and task-specific annotations. (ii) A reasoning model proposes and evaluates translation configurations before large-scale generation. (iii) Source-aware validation and targeted repair address structural failures, while unresolved cases are explicitly recorded.

Source	Evaluation use case
StrongREJECT	Harmful-request handling
JBB-Behaviors	Jailbreak behavior coverage
WildChat	Real-world conversations
WildJailbreak	Adversarial/benign safety prompts
XSTest	Over-refusal and unsafe contrasts
MATH-500	Mathematical reasoning

Table 1: Seven English-language resources selected for translation into 11 target languages. Each resource addresses a distinct evaluation objective; WildChat provides naturalistic conversations rather than a dedicated safety benchmark.

Translator	P1	P2	P3
Gemini 2.5 Flash	70/70	69/70	68/70
Gemini 2.5 Flash-Lite	65/70	59/70	66/70
GPT-4o	68/70	65/70	66/70
GPT-5-mini	67/70	68/70	66/70
GPT-5.1	69/70	70/70	67/70

Table 2: Structurally valid translations across 15 model–instruction configurations, evaluated on 70 Thai instances sampled equally from seven resources. Bold indicates the selected production configuration rather than the highest validity.

Source	Expected	Accepted	Coverage
WildChat	4,070	4,066	99.90%
WildJailbreak	2,750	2,750	100.00%
Total	6,820	6,816	99.94%

Table 3: Operational validation coverage for the WildChat and WildJailbreak subsets following repair. Expected counts correspond to 370 WildChat conversations and 250 WildJailbreak prompts across 11 target languages. Four unresolved WildChat translations are excluded.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Mean CometKiwi score of the evaluation field per source and target language. WildChat is omitted because its long, multi-paragraph turns fall outside the sentence-level regime of the QE model.

Source	QE	LaBSE	BT-chrF	BT-sim	Mean.	Pres.
StrongREJECT	.854	.862	68.5	.934	4.98	99.4
JBB-Behaviors	.858	.841	70.6	.931	5.00	99.9
XSTest	.858	.881	68.9	.950	4.99	99.7
WildJailbreak	.828	.884	63.9	.950	4.99	99.7
WildChat	–	.883	56.1	.921	4.92	98.6
MATH-500	.821	.923	68.6	.923	5.00	100.0

Appendix

Table 4: Fidelity signals per source, averaged over 11 languages. QE: CometKiwi; LaBSE: source–translation similarity; BT-chrF / BT-sim: chrF++ and LaBSE similarity between the NLLB back-translation and the English source; Mean.: judge-rated meaning preservation (1–5); Pres.: % of translations judged to preserve intent (safety and conversation) or all mathematical content (math).

Language	Agree w/ EN	Harmful $\rightarrow$ benign	Gold acc.
English	–	–	88.7
Arabic	91.5	8.1	86.0
Bengali	91.3	8.5	85.9
Spanish	94.2	5.6	88.4
Persian	91.2	8.6	86.5
French	93.4	5.5	87.6

Appendix

Table 5: Label preservation (%) under independent harmful/benign judgments by Gemma-3-12B-it. Gold acc.: agreement with the original dataset labels, which indicates the judge’s own error rate.

Figure 3: Change relative to English (percentage points) on identical items. Colour encodes the desirable direction (red: worse than English); ∗ and bold mark $p<0.05$ (paired bootstrap).

Metric	Model	en	ar	bn	es	fa	fr	hi	pl	ru	th	tr	zh
StrongREJECT score $\downarrow$	Llama-3.1-8B	1.5	1.9	2.5	1.8	4.9 ∗	0.8	2.6	4.2	1.2	4.9 ∗	8.0 ∗	2.5
StrongREJECT score $\downarrow$	Qwen2.5-7B	6.5	5.0	1.4 ∗	5.0	6.5	5.1	2.4	4.6	3.6 ∗	5.1	4.0	5.3
JBB harmful refusal $\uparrow$	Llama-3.1-8B	94.0	85.0 ∗	82.0 ∗	90.9	82.0 ∗	87.0 ∗	84.0 ∗	87.0	87.0 ∗	82.0 ∗	74.0 ∗	87.0 ∗
JBB harmful refusal $\uparrow$	Qwen2.5-7B	91.0	88.0	84.0	88.9	86.0	87.0	82.0 ∗	87.0	88.0	88.0	83.0 ∗	92.0
XSTest unsafe refusal $\uparrow$	Llama-3.1-8B	97.5	88.4 ∗	80.6 ∗	88.3 ∗	84.8 ∗	93.9	86.4 ∗	84.4 ∗	93.7	84.1 ∗	69.5 ∗	91.2 ∗
XSTest unsafe refusal $\uparrow$	Qwen2.5-7B	81.5	82.8	83.2	81.7	76.6	82.7	83.9	80.7	83.1	82.1	75.1 ∗	80.8

Appendix

Table 6: Downstream results on identical items in every language. StrongREJECT score is the mean harmfulness score

\times 100

; all other rows are percentages. ∗ : significant difference from English (paired bootstrap,

p<0.05

). Arrows give the desirable direction. Items: 100 StrongREJECT, 100 harmful JBB, 200 unsafe and 250 safe XSTest, 90 adversarial-harmful WildJailbreak, and 100 MATH-500 problems; item–language pairs flagged as untranslated (Appendix A ) are excluded.

Language	WildChat	WildJailbreak
Arabic	370/370	250/250
French	370/370	250/250
Hindi	370/370	250/250
Chinese	370/370	250/250
Russian	370/370	250/250
Thai	370/370	250/250

Appendix

Table 7: Accepted/source items per language after repair. Four WildChat items are unresolved.

Translator	Med. s	Out	Reas.	$/1k	Full $	Valid
Gem. 2.5 Flash-Lite	1.2	389	0	0.21	5	84.3
GPT-5-mini	10.9	2,034	1,495	4.20	97	97.1
Gem. 2.5 Flash	4.3	1,666	1,177	4.33	103	98.6
GPT-5.1	3.4	520	0	5.85	145	100.0
GPT-4o	2.4	519	0	6.51	163	92.9

Appendix

Table 8: Speed and cost under P2 (14 items per translator, Thai). Med. s: median latency per item; Out / Reas.: mean output tokens per item and the reasoning tokens included in them;

/1k: cost per 1,000 items; Full

: projected cost of translating the full collection into 11 languages; Valid: % structurally valid translations in the configuration study (

n=70

Explore similar work

Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety
Jul 11, 2026Chigozirim Ifebi, Brent Kong, Ayushi MehrotraLLM Safety Benchmarks Multilingual Language Model Evaluation
California Institute of Technology.
The Illusion of Cross-Lingual Safety in Low-Resource Languages
Aug 11, 2026Abigail Oppong, P Sam Sahil, Tadesse Destaw Belay +12LLM Safety Alignment Low-Resource Language Processing
Makerere University Center for Artificial Intelligence · University of Hamburg · Instituto Politécnico Nacional +9
Schützen: Evaluating LLM Safety in Bulgarian and German Contexts
Jun 9, 2026Kiril Georgiev, Yuxia Wang, Dimitar Iliyanov Dimitrov +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski” · MBZUAI

Jul 11, 2026J/K move · Enter open · S save

Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety

Chigozirim Ifebi, Brent Kong, Ayushi Mehrotra

California Institute of Technology.

LLM Safety Benchmarks Multilingual Language Model Evaluation Language Model Safety Evaluation Jailbreak Attacks Mechanistic Interpretability

Open paper AskPDF arXiv

Safety alignment in large language models remains brittle across languages: prompts reliably refused in English can elicit harmful compliance in non-English and low-resource settings. We introduce Minionese, a multilingual jailbreak benchmark spanning 18 languages, 4 resource tiers, and 4 perturbation types (standard translation, code-switching, transliteration, and translationese), paired with a geometric mechanistic analysis of refusal failure across language tiers. We show that each attack type produces a distinct vulnerability profile: transliteration vulnerability is mediated by script identity, code-switching maintains effectiveness through the lowest-resource tier, and a sharp safety regime transition between Tiers 2 and 3 is consistent across all models. Mechanistically, low-resource jailbreaks succeed by routing harmful content through a geometrically misaligned subspace that projects insufficiently onto the refusal directions, leaving the refusal mechanism intact but untriggered. These findings show that English-only safety evaluations are insufficient; they require accounting for script family, perturbation type, and per-language alignment coverage. The benchmark and analysis code is at https://github.com/Brentkong/Minionese-Comprehensive-Benchmark-and-Mechanistic-Study-of-Multilingual-LLM-Safety.git.

Abstract

Explore similar work

Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety

The Illusion of Cross-Lingual Safety in Low-Resource Languages

Schützen: Evaluating LLM Safety in Bulgarian and German Contexts