Component Translation and Judge-Labelled Refusal: A Multilingual Safety Evaluation

Yongxi Zhou; Junwei Yao; Wenbo Ye; Jiahui Ge

cs.CLOct 28, 2026

Component Translation and Judge-Labelled Refusal: A Multilingual Safety Evaluation

Authors: Yongxi Zhou, Junwei Yao, Wenbo Ye, Jiahui Ge

Organizations: Independent Researcher

Abstract

Whole-prompt translation evaluates a request in different languages but does not compare which component carries a language. We introduce a paired reversal: translate the complete request component or the complete operation component into the same target language, holding behaviour, English frame and translated-span count fixed. Component identity and non-English text quantity change together. Assigning English, Chinese and Spanish independently to the two components yields nine configurations on 190 of 300 sampled harmful behaviours retained after consensus annotation and automatic gating. Refusal scored by language-model judges is the primary outcome; unsafe compliance is reported separately. On Gemini 2.5 Flash and DeepSeek, three refusal swaps are about 5.3 percentage points and the fourth is -0.5; none survives the pre-specified six-test Holm correction. In an exploratory extension, Mistral Small 4 has a 13.7-point Spanish swap (95% interval [8.4, 18.9]), which survives the twelve-test Holm sensitivity correction under the primary judge, and 12.1 points under a second judge. GPT-5.4-mini estimates are within 2 points of zero. Component assignment supplies an additional evaluation dimension. Evaluations should record component languages and translated text quantity, comparing matched behaviours at equal translated-span count while interpreting refusal and unsafe compliance separately.

Figures & tables

source-native group	sampled	consensus	analysed
HarmBench: copyright	45	28	0
chemical / biological	20	17	3
harmful	15	13	12
illegal	48	42	40
misinformation / disinformation	34	25	20
harassment / bullying	21	20	20

TABLE I: Behaviours retained at each stage. HarmBench rows use its original categories; AdvBench is a source group. Counts describe selection, not a unified harm ontology.

Panel of Figure 2 — Fig. 1: Primary-judge refusal rates (%) on the same 190 harmful behaviours: primary analysis above, exploratory extension below, with one colour scale. Rows assign the request slot’s language and columns the operation slot’s; E , S , Z denote English, Spanish, Chinese. Cell denominators are 190 except six Gemini cells with empty generations, where $n=187$ –189. Wilson 95% intervals are omitted; their maximum half-width across all 36 cells is $6.2\,pp$ .

model	Spanish swap	$p_{\mathrm{Holm}}$	Chinese swap	$p_{\mathrm{Holm}}$	language difference
gemini	$+5.3$ $[+1.6,+9.1]$	$0.065$	$+5.3$ $[+1.1,+9.5]$	$0.065$	$+0.0$ $[-4.9,+5.4]$
deepseek	$+5.3$ $[+1.6,+8.9]$	$0.065$	$-0.5$ $[-2.6,+1.6]$	$1.000$	$+5.8$ $[+2.1,+9.5]$
gpt-5.4-mini	$+0.5$ $[+0.0,+1.6]$	$1.000$	$+1.6$ $[-0.5,+4.2]$	$1.000$	$-1.1$ $[-3.7,+1.6]$
mistral-s4	$+13.7$ $[+8.4,+18.9]$	$<0.001$	$+2.6$ $[-3.2,+8.4]$	$0.473$	$+11.1$ $[+4.7,+17.9]$

TABLE II: The component-swap contrasts on four models, primary judge: refusal in the request-switched minus the operation-switched configuration at dose 1, in percentage points with 95% bootstrap intervals, and the language-difference contrast (Spanish swap minus Chinese swap). A positive swap means the configuration whose request slot is non-English refuses more.

p_{\mathrm{Holm}}

uses the pre-specified six-test family for the two primary models and a separate three-test family for each extension model. All on the same 190 behaviours;

n

is 188 and 187 for the two Gemini swaps and 185 for its language-difference contrast. All-English refusal is 95.8%, 97.4%, 99.5% and 88.4% on the four models in table order.

check	observed sensitivity or exclusion
Segment before translating; tabulate coverage by source-native group.	A qualifier occurs on 39/53 copyright and 6/21 generic harmful behaviours in HarmBench.
Count translated spans per configuration; match this count for component swaps.	In different pilot strata, the unsafe swap was $+13.2\,pp$ where span counts differed and $0.0\,pp$ where they matched, using any-positive labels from the first two responses.
Pin unmanipulated spans to the base language and count realised dose on the built dataset.	Under the inheriting convention the contrast was dose-balanced on 52% of behaviours; under pinning, on 305 of 305.
Treat a generation with no text as missing, not as a non-refusal.	Scoring 10 empty draws as refusal: no inflated the headline contrast by $1.0$ and $1.5\,pp$ .
Separate the naturalness floor (rating 1) from a fluency threshold; report exclusions and missing ratings by configuration.	A naturalness floor of 3 keeps 95% of the monolingual configuration and 44% of a switched one.
Use two judges from vendors supplying no evaluated model; check whether their positive sets nest or cross.	In the pilot a vendor swap moved the overall unsafe rate from 6.4% to 17.4%; here it moved the four unsafe-compliance component-swap contrasts by at most $1.1\,pp$ .

TABLE III: Measurement checks adopted in this study. Each check is one contingency table or one paired difference, and each observation is from this corpus or its two-language pilot; they are not a validated general protocol.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

source-native group	pool	two slots	qualifier	frame content word
HarmBench: copyright	53	53	39	0
chemical / biological	28	27	22	21
harmful	21	17	6	15
illegal	59	58	43	49
misinformation / disinformation	41	41	27	33
harassment / bullying	25	24	8	18

Appendix

TABLE IV: Slot coverage by source-native group over the 370-behaviour pool. Two slots: the rule-based chunker recovers both the request and the operation slot. Qualifier: an explicit qualifier span is present. Frame content word: at least one content word lies outside the two slots.

inherited stratum	source-native group	sampled	consensus	analysed
cyber_abuse	cybercrime / intrusion	48	40	39
cyber_abuse	AdvBench	33	32	29
fraud_deception	illegal	48	42	40
fraud_deception	misinformation / disinformation	34	25	20
physical_harm	HarmBench: copyright	44	27	0
physical_harm	chemical / biological	20	17	3

Appendix

TABLE V: Inherited sampling stratum against source-native group for the 300 sampled, 245 consensus and 190 analysed harmful behaviours.

configuration	intent: negative	intent: missing	naturalness 1	naturalness unrated
EE	24	15	5	15
EZ	20	13	0	3
ES	27	11	1	3
ZE	25	12	0	2
ZS	30	14	0	4
ZZ	17	16	12	16

Appendix

TABLE VI: Intent verdicts and naturalness ratings by configuration. Intent columns: harmful factorial items of the 245 consensus behaviours rated as losing intent (negative) or returned without a verdict (missing). Naturalness 1: items excluded as not interpretable as a sentence, all pools. Naturalness unrated: analysed harmful factorial prompts (190 behaviours) without a rating.

run	pairs	sentence	spans	difference	95% interval	Wilcoxon $p$	spans worse
replay, zh	305	0.794	0.787	$-0.007$	$[-0.024,+0.011]$	0.93	137
replay, es	296	0.895	0.887	$-0.008$	$[-0.020,+0.003]$	0.25	142
original, zh	305	0.794	0.787	$-0.007$	$[-0.025,+0.009]$	0.93	137
original, es	305	0.894	0.887	$-0.007$	$[-0.019,+0.005]$	0.30	145

Appendix

TABLE VII: Whole-sentence against isolated-span round-trip similarity, paired on behaviour. Sentence and spans are mean similarities; the interval is a 95% bootstrap interval for the paired difference; spans worse counts behaviours whose isolated spans score below their sentence.

comparison	YY	YN	NY	NN	agree (%)	$\kappa$ [cluster 95%]	weighted $\kappa$
rater A vs rater B, refusal	163	1	1	20	98.9	0.946 [0.86, 1.00]	0.946
rater A vs rater B, unsafe compliance	14	0	1	168	99.5	0.963 [0.85, 1.00]	0.963
reference vs primary judge, refusal	162	2	1	20	98.4	0.921 [0.82, 1.00]	0.920
reference vs second judge, refusal	163	1	1	20	98.9	0.946 [0.86, 1.00]	0.946
reference vs primary judge, unsafe compliance	14	1	6	164	96.2	0.780 [0.59, 0.91]	0.783
reference vs second judge, unsafe compliance	14	1	7	163	95.7	0.755 [0.57, 0.89]	0.758

Appendix

TABLE VIII: Agreement on the 185-response audit. Counts are two-by-two tables (YY both positive, YN first positive and second negative, NY the reverse, NN both negative); the unsafe-compliance rater comparison has 183 pairs after excluding two uncertain ratings.

\kappa

is Cohen’s kappa with an approximate behaviour-cluster 95% percentile interval; weighted

\kappa

uses the design weights.

config.	Gemini 2.5 Flash	DeepSeek	GPT-5.4-mini	Mistral Small 4
EE	$3.2$ [ $1.5$ , $6.7$ ]	$2.1$ [ $0.8$ , $5.3$ ]	$0.0$ [ $0.0$ , $2.0$ ]	$11.1$ [ $7.3$ , $16.3$ ]
ES	$9.6$ [ $6.1$ , $14.6$ ]	$10.0$ [ $6.5$ , $15.1$ ]	$0.5$ [ $0.1$ , $2.9$ ]	$22.6$ [ $17.3$ , $29.1$ ]
EZ	$9.1$ [ $5.8$ , $14.1$ ]	$3.7$ [ $1.8$ , $7.4$ ]	$1.1$ [ $0.3$ , $3.8$ ]	$19.5$ [ $14.5$ , $25.7$ ]
SE	$3.7$ [ $1.8$ , $7.4$ ]	$5.3$ [ $2.9$ , $9.4$ ]	$0.0$ [ $0.0$ , $2.0$ ]	$11.6$ [ $7.8$ , $16.9$ ]
ZE	$5.3$ [ $2.9$ , $9.4$ ]	$3.7$ [ $1.8$ , $7.4$ ]	$0.0$ [ $0.0$ , $2.0$ ]	$18.9$ [ $14.0$ , $25.1$ ]
SS	$10.1$ [ $6.5$ , $15.2$ ]	$8.9$ [ $5.7$ , $13.9$ ]	$0.0$ [ $0.0$ , $2.0$ ]	$21.6$ [ $16.3$ , $28.0$ ]

Appendix

TABLE IX: Unsafe-compliance rates (%) per configuration under the primary judge, with Wilson 95% intervals; 190 harmful behaviours per cell except the Gemini cells with empty generations.

config.	Gemini 2.5 Flash	DeepSeek	GPT-5.4-mini	Mistral Small 4
EE	$96.3$ [ $92.6$ , $98.2$ ]	$97.4$ [ $94.0$ , $98.9$ ]	$99.5$ [ $97.1$ , $99.9$ ]	$88.9$ [ $83.7$ , $92.7$ ]
ES	$89.4$ [ $84.1$ , $93.0$ ]	$89.5$ [ $84.3$ , $93.1$ ]	$99.5$ [ $97.1$ , $99.9$ ]	$76.3$ [ $69.8$ , $81.8$ ]
EZ	$88.2$ [ $82.8$ , $92.1$ ]	$96.3$ [ $92.6$ , $98.2$ ]	$98.9$ [ $96.2$ , $99.7$ ]	$79.5$ [ $73.2$ , $84.6$ ]
SE	$95.3$ [ $91.2$ , $97.5$ ]	$94.7$ [ $90.6$ , $97.1$ ]	$99.5$ [ $97.1$ , $99.9$ ]	$88.4$ [ $83.1$ , $92.2$ ]
ZE	$93.2$ [ $88.6$ , $96.0$ ]	$95.8$ [ $91.9$ , $97.9$ ]	$98.9$ [ $96.2$ , $99.7$ ]	$81.6$ [ $75.5$ , $86.4$ ]
SS	$88.4$ [ $83.0$ , $92.2$ ]	$90.0$ [ $84.9$ , $93.5$ ]	$99.5$ [ $97.1$ , $99.9$ ]	$77.9$ [ $71.5$ , $83.2$ ]

Appendix

TABLE X: Refusal rates (%) per configuration under the second judge ( qwen3.8-flash ), with Wilson 95% intervals.

refusal, second judge
model	Spanish swap	Chinese swap	language difference
Gemini 2.5 Flash	$+5.9$ $[+2.6,+9.6]$	$+4.8$ $[+1.1,+8.7]$	$+1.1$ $[-3.8,+5.9]$
	$p_{\mathrm{Holm}}$ $0.017$	$p_{\mathrm{Holm}}$ $0.105$
DeepSeek	$+5.3$ $[+2.1,+8.9]$	$-0.5$ $[-2.6,+1.6]$	$+5.8$ $[+2.6,+9.5]$
	$p_{\mathrm{Holm}}$ $0.025$	$p_{\mathrm{Holm}}$ $1.000$
GPT-5.4-mini	$+0.0$ $[+0.0,+0.0]$	$+0.0$ $[-2.1,+2.1]$	$+0.0$ $[-2.1,+2.1]$

Appendix

TABLE XI: Component-swap contrasts (request-switched minus operation-switched at dose 1, in percentage points with 95% bootstrap intervals) and the language-difference contrast (Spanish swap minus Chinese swap) under the second judge for refusal and under both judges for unsafe compliance.

p_{\mathrm{Holm}}

uses the same families as Table II : the six pre-specified tests for the two primary models and a three-test family for each extension model, each outcome and judge corrected separately.

Language Model Safety Evaluation

Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.

See all 549 papers

Jul 11, 2026

Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety

Safety alignment in large language models remains brittle across languages: prompts reliably refused in English can elicit harmful compliance in non-English and low-resource settings. We introduce Minionese, a multilingual jailbreak benchmark spanning 18 languages, 4 resource tiers, and 4 perturbation types (standard translation, code-switching, transliteration, and translationese), paired with a geometric mechanistic analysis of refusal failure across language tiers. We show that each attack type produces a distinct vulnerability profile: transliteration vulnerability is mediated by script identity, code-switching maintains effectiveness through the lowest-resource tier, and a sharp safety regime transition between Tiers 2 and 3 is consistent across all models. Mechanistically, low-resource jailbreaks succeed by routing harmful content through a geometrically misaligned subspace that projects insufficiently onto the refusal directions, leaving the refusal mechanism intact but untriggered. These findings show that English-only safety evaluations are insufficient; they require accounting for script family, perturbation type, and per-language alignment coverage. The benchmark and analysis code is at https://github.com/Brentkong/Minionese-Comprehensive-Benchmark-and-Mechanistic-Study-of-Multilingual-LLM-Safety.git.

Chigozirim Ifebi, Brent Kong, Ayushi Mehrotra

California Institute of Technology.

LLM Safety Benchmarks Multilingual Language Model Evaluation Language Model Safety Evaluation Jailbreak Attacks Mechanistic Interpretability

Ask PDF

Oct 19, 2026

Global Security Blind Spots: Translation as a Safety Bypass in Low-Resource Languages

Large language models are deployed globally, yet safety alignment and evaluation remain concentrated in English: harmful requests refused in English can elicit unsafe completions when translated into low-resource languages. We evaluate ten widely deployed LLMs across 79 languages (English plus 78 low-resource) under a translation-bypass threat model, using a metric conditioned on English safety, which separates translation bypass from unsafety already present in English. English safety does not transfer reliably: conditioned on safe English completions, mean bypass rates range from 21.9% to 44.4%. Bypass risk concentrates in a subset of language families, a pattern broadly shared across models. The shared ordering is confounded with translation degeneracy, but its variation across models is significant and survives every control we apply. We also show that per-model bypass rates do not identify a stable ranking: nine of ten models change rank across three conditioning schemes that estimate the same quantity, with no change to any underlying response.

Jalil Alva, David Harper, Abigail E. Thornton +8

Welo Data

LLM Safety Language Model Safety Evaluation Multilingual Language Model Evaluation Adversarial Attacks on LLMs Jailbreak Attacks

Ask PDF

Oct 20, 2026

Is Refusal Direction Universal across Low-Resourced Languages?

Refusal directions have been shown to transfer cross-lingually with near-perfect effectiveness across safety-aligned languages (Wang et al., 2025), reportedly regardless of resource level. However, the only truly low-resourced language evaluated is Yoruba, which is strongly misaligned and shows limited transfer, leaving open how refusal directions behave for other low-resourced languages. We extend PolyRefuse to three additional low-resourced languages (Belarusian, Bashkir, and Tajik) and replicate the pipeline of (Wang et al., 2025) on Qwen2.5-14B-Instruct. We find that universality is graded rather than binary: directions from safety-aligned and partially-aligned languages transfer broadly, while those from strongly misaligned languages fail to reliably modulate refusal behavior on any target tested, including other misaligned languages, once ceiling effects are accounted for. Geometric alignment and behavioral transfer are further shown to be partially dissociated. These findings complicate the universality claim and suggest that users of the least-resourced, most strongly misaligned languages may face weaker safety guarantees.

Anastasiia Ivanova

MaiNLP, LMU Munich

Refusal Mechanisms in Language Models Cross-Lingual Transfer Low-Resource Language Processing

Ask PDF

Abstract

Language Model Safety Evaluation

Explore similar work

Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety

Global Security Blind Spots: Translation as a Safety Bypass in Low-Resource Languages

Is Refusal Direction Universal across Low-Resourced Languages?