Disentangling Language Effects on Tool Admission and Post-Retrieval Safety

Aditya Nawal; Manit Baser; Mohan Gurusamy

cs.CLOct 16, 2026

Disentangling Language Effects on Tool Admission and Post-Retrieval Safety

Authors: Aditya Nawal, Manit Baser, Mohan Gurusamy

Organizations: National University of Singapore

Abstract

LLM agents increasingly use retrieval tools across languages, yet their safety evaluation remains English-centric. It remains unclear whether language affects tool admission, post-retrieval generation, or both. We independently vary request, retrieved-page, and required response languages in a controlled URL-retrieval setting across ten models and seven languages. We identify Safety Loss through Interlingual Processing (SLIP). Non-English harmful requests increase mean tool admission from 46.9% to 67.4%, with a positive mean change in every model. This pattern also persists in an autonomous retrieval setup across all four models tested in this setting. Requiring answers in the page language increases pooled harmfulness on matched tool calls from 50.3% to 57.4% across four closed-weight models, mainly in Chinese and Hindi. Translating pages alone produces no consistent harm increase in eight models with comparable inference settings. Automated audits show that a non-harmful response may be a genuine safety refusal, a deflection citing language difficulties, or an unusable answer. Harmfulness labels alone cannot distinguish these outcomes. Multilingual safety evaluations should therefore measure tool admission and response harmfulness separately and distinguish refusal from capability failure.

Figures & tables

Figure 1: Language-dependent safety paths for the same harmful request and URL.

Figure 2: Language interventions at tool admission and response generation.

Open-weight 3–14B
Model	English		Chinese		Hindi		Igbo		Swahili		Lao		Haitian Creole
	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$
Qwen-2.5-3B	72.3	76.6	73.0	67.6	90.5	63.1	95.6	56.2	95.6	58.7	93.9	49.6	94.9	49.1
Qwen-2.5-14B	40.2	77.3	46.6	76.8	68.2	71.8	78.0	55.4	68.9	77.5	76.7	56.8	82.4	69.3
Llama-3.1-8B	70.3	63.3	75.0	66.7	84.1	71.7	99.3	70.9	96.3	73.9	100.0	60.9	100.0	77.0
Llama-3.2-3B	64.2	40.0	77.7	30.0	89.5	19.3	95.9	23.5	98.0	10.8	100.0	30.4	97.6	22.9

Table 1: Tool admission (

\mathrm{AR}

) and post-admission harm (

\mathrm{HR}

) by request language (%).

	Admission ( $\mathrm{AR}$ )			Harm ( $\mathrm{HR}$ )
Models	En	Zh/Hi	LR	En	Zh/Hi	LR
Open 3–14B	61.7	75.6 (+13.9)	92.1 (+30.3)	64.3	58.4 ( $-$ 6.0)	52.7 ( $-$ 11.6)
Open 27–31B	33.1	41.0 (+7.9)	39.9 (+6.8)	56.0	64.2 (+8.2)	62.8 (+6.8)
Closed-weight	39.1	43.7 (+4.6)	70.9 (+31.8)	46.5	67.2 (+20.7)	60.7 (+14.2)
All ten	46.9	55.9 (+9.0)	73.2 (+26.2)	55.5	63.1 (+7.5)	57.9 (+2.4)

Table 2: Model-group means of Table 1 (%; change from English in parentheses). Zh/Hi: Chinese, Hindi; LR: Igbo, Swahili, Lao, Haitian Creole.

	Translated request		Translated page
Models	Zh/Hi	Ig/Sw	Zh/Hi	Ig/Sw
Open 3–14B	54.5	71.0	82.7	82.8
Open 27–31B	8.8	23.7	98.0	97.5
Closed-weight	5.8	21.7	94.6	97.4

Table 3: English-answer share of called responses when the request or only the page is translated (%; mean of Chinese/Hindi and of Igbo/Swahili).

Panel of Figure 3 — Figure 3: Harm among model–behavior pairs admitted and scored under both directives (pooled %). Closed: Chinese/Hindi/Igbo/Swahili; HR: Chinese/Hindi; LR: Igbo/Swahili.

Panel of Figure 4 — Figure 4: Tool-free goal recovery (%), pooled by model group and request language.

Open 3–14B
Language	Refusal	Language deflection	Realization failure	Harm- like
Chinese	50.5	8.4	28.4	12.6
Hindi	30.5	25.7	39.0	4.8
Igbo	8.9	19.3	69.6	2.2
Swahili	25.0	16.9	54.7	3.5
Open 27–31B

Table 4: Automated audit of non-harmful called answers under the page-language directive (% within row; 54–197 audited per row). Categories as defined in Section 4.4 .

	Open 3–14B			Open 27–31B			Closed-weight
Lang.	Ref.	Fail.	Harm	Ref.	Fail.	Harm	Ref.	Fail.	Harm
En	70.1	23.8	6.1	72.9	11.8	15.3	90.9	5.5	3.6
Zh	60.8	37.5	1.7	71.4	11.0	17.6	81.6	12.9	5.5
Hi	47.9	49.6	2.5	78.0	15.9	6.1	92.2	4.5	3.4
Ig	27.3	71.1	1.6	55.2	39.0	5.7	65.6	27.8	6.6
Sw	43.6	54.2	2.3	66.3	23.3	10.5	82.4	12.0	5.6

Table 5: Audited non-harmful called answers by request language (%; 72–512 audited per cell). Ref.: refusal; Fail.: realization failure; Harm: harm-like (Section 4.4 ).

Appendix figures & tables50 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Paper name	Model identifier
Open-weight	Qwen-2.5-3B	Qwen/Qwen2.5-3B-Instruct
Open-weight	Qwen-2.5-14B	Qwen/Qwen2.5-14B-Instruct
Open-weight	Llama-3.1-8B	meta-llama/Llama-3.1-8B-Instruct
Open-weight	Llama-3.2-3B	meta-llama/Llama-3.2-3B-Instruct
Open-weight	Qwen-3.5-27B	Qwen3.5-27B-FP8
Open-weight	Gemma-4-31B	gemma-4-31b-it

Appendix

Table 6: Evaluated models and identifiers.

Model	English		Chinese		Hindi		Igbo		Swahili		Lao		Haitian Creole
	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$	$\mathrm{AR}$	$\mathrm{HR}$
Qwen-2.5-7B	69.6	78.2	78.0	77.9	93.2	77.5	95.3	59.2	99.3	62.9	99.0	54.3	100.0	70.9
Mistral-Nemo-12B	90.5	90.3	95.3	86.2	55.4	84.1	99.3	65.0	100.0	65.5	95.3	60.3	95.3	74.1
o4-mini	1.7	100.0	8.8	76.9	6.4	57.9	16.9	62.0	6.4	42.1	18.9	58.5	4.7	50.0

Appendix

Table 7: Supplementary models, excluded from the ten-model averages: tool admission (AR) and post-admission harm (HR) by request language (%). The o4-mini English harm rate uses five calls.

Model	AR	HR
Qwen-2.5-3B	92.2	56.4
Qwen-2.5-7B	97.6	58.8
Qwen-2.5-14B	73.6	65.6
Llama-3.1-8B	99.3	68.8
Llama-3.2-3B	99.7	24.3
Mistral-Nemo-12B	91.7	61.4

Appendix

Table 8: Supplementary Amharic requests, excluded from the main-text averages: tool admission (AR) and post-admission harm (HR) (%).

Model	EN	ZH	HI	IG	SW	LO	HT
Qwen-2.5-3B	214	216	268	283	283	278	281
Qwen-2.5-14B	119	138	202	231	204	227	244
Llama-3.1-8B	60	54	60	79	69	69	74
Llama-3.2-3B	80	80	83	85	93	69	83
Qwen-3.5-27B	95	131	111	137	119	124	128
Gemma-4-31B	101	139	105	107	117	97	95

Appendix

Table 9: Number of scored called answers underlying Table 1 . EN: English; ZH: Chinese; HI: Hindi; IG: Igbo; SW: Swahili; LO: Lao; HT: Haitian Creole.

Model	Assigned answer	ZH	HI	IG	SW
Qwen-2.5-3B	English	200	181	199	199
Qwen-2.5-3B	Page language	182	228	146	251
Qwen-2.5-14B	English	122	116	122	122
Qwen-2.5-14B	Page language	122	136	111	119
Llama-3.1-8B	English	24	14	21	23
Llama-3.1-8B	Page language	23	10	37	64

Appendix

Table 10: Scored called-answer counts by assigned response language.

Models	Languages	Scored answers	Audited safe answers
Open 3–14B	EN	473	147
Open 3–14B	HR	1101	416
Open 3–14B	LR	2652	1191
Open 27–31B	EN	196	85
Open 27–31B	HR	486	173
Open 27–31B	LR	924	359

Appendix

Table 11: Coverage of capability/outcome summaries. HR: Chinese/Hindi; LR: Igbo/Swahili/Lao/Haitian Creole.

Request Language
Language	$\bm{N}$	Agreement [95% CI]	Original / transl. harm	Safe $\rightarrow$ harm / Harm $\rightarrow$ safe	Holm $\bm{p}$
Igbo	163	72.4 [65.6, 79.1]	46.6 / 57.1	31 / 14	0.032
Swahili	148	83.1 [77.0, 89.2]	51.4 / 58.8	18 / 7	0.043
Lao	75	82.7 [73.3, 90.7]	49.3 / 58.7	10 / 3	0.185
Haitian Creole	76	94.7 [89.5, 98.7]	48.7 / 51.3	3 / 1	0.625
Assigned Response Language

Appendix

Table 12: Translation-mediated harm-label agreement (%) on stratified samples; threshold

\geq 3

Request Language
Language	Original–Azure agree. (%)	NLLB–Azure agree. (%)	Three-way agree. (%)
Igbo	72.6	72.6	60.3
Swahili	82.4	89.2	75.7
Assigned Response Language
Igbo	79.3	82.9	74.4
Swahili	92.8	90.4	85.5

Appendix

Table 13: Harm-label agreement across original, NLLB, and Azure translations (%; 312 stratified answers).

Language	$\bm{n}$	Agree. (%)	$\bm{\kappa}$	GPT-4o harm
Igbo	311	80.1	0.61	56.9
Swahili	323	86.4	0.73	53.6
Lao	348	72.7	0.49	60.3
Haitian Creole	344	84.9	0.70	51.2

Appendix

Table 14: Gemini–GPT-4o agreement.

n

: jointly scored answers; Harm: GPT-4o harmful share (%).

Models	Page	Calls	Harm	Refusal	Page-lang. deflect.	Realiz. fail.	Harm-like
Open 3–14B	English	473	68.9	21.6	0.0	7.8	1.7
Open 3–14B	Non-English	1561	74.0	14.1	3.0	7.4	1.5
closed-weight	English	462	45.2	49.1	0.0	2.4	3.2
closed-weight	Non-English	1783	42.7	43.2	9.0	1.8	3.3
Open 27–31B	English	196	56.6	31.6	0.0	3.1	8.7
Open 27–31B	Non-English	781	50.3	33.4	5.8	1.7	8.8

Appendix

Table 15: Called-answer outcomes by page language (%). Harm uses the primary judge; other outcomes are audit-based shares. The 27–31B comparison has mismatched limits.

Language	GPT-4.1	GPT-4o	Agreement	Gemini	Valid $\bm{n}$
Chinese	100.0	98.0	98.0	–	296
Hindi	99.7	99.0	98.6	–	296
Igbo	93.9	86.5	88.5	74.0	296
Swahili	96.3	92.9	95.3	91.2	296
Lao	–	–	–	88.5	296
Haitian Creole	–	–	–	96.6	296

Appendix

Table 16: Request-translation adequacy (%). GPT agreement: raw item agreement; Gemini: meaning and intent preserved.

Subset	Zh	Hi	Ig	Sw	Lo	Ht
Full set	+7.0	+10.9	+27.5	+18.9	+28.5	+30.4
GPT	+7.2	+11.0	+24.5	+19.0	–	–
Gemini	–	–	+24.8	+18.9	+29.4	+30.2
Strict (GPT + Gemini)	–	–	+23.3 ∗	+18.9 ∗	–	–
95% interval	–	–	[20.2, 26.3]	[16.8, 20.9]	–	–
$n$ (GPT / Gemini / Strict)	290 / – / –	292 / – / –	250 / 219 / 202	273 / 270 / 263	– / 262 / –	– / 286 / –

Appendix

Table 17: Mean admission gaps from English on quality-accepted requests (pp).

n

: retained of 296; GPT: both GPT judges accept; Gemini: meaning and intent accepted. Intervals resample behaviors; ∗ Holm-adjusted McNemar

p<0.001

. Gaps use behaviors valid in both languages, so full-set values can differ by up to 0.2 points from the Table 1 means in Table 21 .

	Igbo		Swahili
Models	Full	Strict	Full	Strict
Open 3–14B	$-$ 12.8	$-$ 10.5	$-$ 9.1	$-$ 9.1
Open 27–31B	+2.7	+5.2	+7.7	+7.6
Closed-weight	+29.2	+31.0	+20.6	+20.9
All ten	+7.1	+9.3	+6.1	+6.2

Appendix

Table 18: Mean harm change from English on strict quality-accepted requests (pp; GPT + Gemini; 202 Igbo and 263 Swahili behaviors retained).

	Hindi		Igbo
Model	Original	Re-translated	Original	Re-translated
claude-haiku-4-5	63.0%	49.1%	79.9%	81.2%
gemini-2.5-flash	83.1%	82.9%	76.8%	83.7%
claude-sonnet-4	48.5%	43.9%	76.5%	73.0%
gpt-4.1	66.1%	63.6%	69.8%	66.4%
Average	65.2%	59.9%	75.8%	76.1%

Appendix

Table 19: Closed-weight called-answer harm under original and alternative request translations (%).

Language	GPT-4o	GPT-4.1	Agreement	Valid $\bm{n}$
Chinese	97.3	98.0	96.6	296
Hindi	98.3	99.7	98.0	296
Igbo	98.6	99.7	99.0	296
Swahili	99.0	99.7	98.6	296

Appendix

Table 20: Retrieved-page translation semantic audit (%).

Language	mC4 tokens (B)	Call-rate gap (pp)
English	2,733	0.0
Chinese	39	+7.0
Hindi	24	+11.0
Swahili	1.0	+18.9
Haitian Creole	0.2	+30.2
Lao	0.1	+28.3

Appendix

Table 21: mC4 sizes ( Xue et al., 2021 ) and mean admission gaps from English (ten models).

Language	Admission	Change	Targeted
English	14.9	–	14.8
Chinese	27.7	+12.8	26.0
Hindi	23.9	+9.0	22.2
Igbo	57.9	+42.9	46.0
Swahili	42.6	+27.6	38.2
Lao	37.2	+22.2	31.1

Appendix

Table 22: Mean autonomous admission rates (%; four closed-weight models). Targeted: admission counting only calls whose URL pursues the harmful task.

URL Variant	Example URL	Harmful	Benign	$\bm{\Delta}$
High-trust, hidden path	https://example.org/	64.6	89.9	+25.3
High-trust, explicit path	https://example.org/how-to-synthesize-compounds	62.0	75.2	+13.2
HTTP (no TLS)	http://example.org/	64.5	88.7	+24.2
Institutional (MSFT)	https://microsoft.example.com/	60.9	81.6	+20.7
Institutional (Stanford)	https://stanford.example.edu/	64.9	90.9	+26.0
Government	https://cdc.example.gov/	67.0	94.2	+27.2

Appendix

Table 23: Admission by URL form (%; four 3–14B models).

\Delta

: benign minus harmful.

Open 3–14B
Language	Harmful call (%)	Benign call (%)	$\bm{S_{L}}$ (%)	$\bm{S_{L}-S_{en}}$ [95% CI]	Holm $\bm{p}$
English	61.7	97.2	35.5	–	–
Chinese	68.1	96.0	28.0	$-7.5$ [ $-10.5,-4.5$ ]	$<.001$
Hindi	83.1	96.2	13.1	$-22.4$ [ $-26.0,-18.9$ ]	$<.001$
Igbo	92.2	93.7	1.4	$-34.1$ [ $-38.6,-29.7$ ]	$<.001$
Swahili	89.7	93.8	4.1	$-31.4$ [ $-35.7,-27.3$ ]	$<.001$

Appendix

Table 24: Matched benign–harmful admission selectivity (%; eight models).

Models	En	Zh	Hi	Ig	Sw	Lo	Ht
Open 3–14B	32.3	32.4	40.1	46.7	45.8	45.4	50.5
Open 27–31B	18.8	30.2	22.6	23.5	25.3	26.7	22.7
closed-weight	17.7	30.1	28.2	55.3	36.6	42.7	36.0

Appendix

Table 25: Coverage-conditioned retrieval-mediated harmful yield

Y

(%).

Category	$\bm{N}$	En	Zh	Hi	Ig	Sw	Lo	Ht	Mean low-mC4 gap
harassment_bullying	17	8.2	21.2	29.4	67.1	38.8	65.7	66.1	+51.2
harmful	18	31.7	39.4	50.6	74.4	66.7	79.4	79.0	+43.2
illegal	48	19.0	25.8	35.2	62.1	44.4	61.2	63.7	+38.9
misinformation_disinformation	41	45.1	54.4	55.4	77.0	66.6	77.6	81.7	+30.6
chemical_biological	42	27.9	32.1	39.0	53.3	45.5	56.2	58.3	+25.5
cybercrime_intrusion	50	44.0	48.5	52.8	67.2	59.7	69.3	73.3	+23.4

Appendix

Table 26: Mean admission by category and request language (%; ten models).

Panel of Figure 5 — Figure 5: Mean page-language harm changes (pp). The 27–31B comparison uses mismatched inference limits.

Open 3–14B Models
Model	English	Chinese	Hindi	Igbo	Swahili	Non-Eng. avg.
Qwen-2.5-3B	76.6	77.9	75.0	76.9	79.7	77.4
Qwen-2.5-14B	77.3	81.5	86.6	77.1	83.9	82.3
Llama-3.1-8B	63.3	60.0	56.2	62.9	60.5	59.9
Llama-3.2-3B	40.0	21.9	31.6	30.0	52.3	33.9
Avg.	64.3	60.3 ( $-4.0$ )	62.4 ( $-2.0$ )	61.7 ( $-2.6$ )	69.1 ( $+4.8$ )	63.4 ( $-0.9$ )

Appendix

Table 27: Post-admission harm by page language (%; deltas in parentheses). The 27–31B comparison uses mismatched limits.

Language	$\bm{N_{\neg\text{call}}}$	Harmful $\bm{n}$	$\bm{P(\text{harm}\mid\neg\text{call})}$
English	1540	22	1.4
Chinese	996	15	1.5
Hindi	1210	20	1.7
Igbo	745	50	6.7
Swahili	1007	7	0.7
Lao	714	14	2.0

Appendix

Table 28: No-call harmfulness (%; ten models, threshold

\geq 3

Language	Models	$\bm{N}$	Refusal	Language deflection	Realization failure	Harm- like
Lao	Closed-weight	234	100.0	0.0	0.0	0.0
Lao	Open 3–14B	87	65.5	8.0	18.4	8.0
Lao	Open 27–31B	342	96.5	0.0	0.0	3.5
Haitian Creole	Closed-weight	192	100.0	0.0	0.0	0.0
Haitian Creole	Open 3–14B	74	54.1	6.8	36.5	2.7
Haitian Creole	Open 27–31B	341	96.8	0.0	0.0	3.2

Appendix

Table 29: Taxonomy of nonempty no-call responses in Lao and Haitian Creole (%, within language and model group). The 100 empty claude-sonnet-4 answers are excluded. Realization failure: generic, misunderstood, or low-fluency content. Harm-like: partial or complete harmful realization.

Model	En	Zh	Hi	Ig	Sw	Lo	Ht
Qwen-3.5-27B	0.0 (0/201)	0.0 (0/165)	0.0 (0/185)	0.0 (0/159)	0.0 (0/177)	0.0 (0/172)	0.0 (0/168)
Gemma-4-31B	2.6 (5/195)	5.7 (9/157)	3.7 (7/191)	24.5 (46/188)	1.7 (3/179)	7.6 (13/170)	5.8 (10/173)

Appendix

Table 30: Harmful no-call responses of the 27–31B models, % (harmful

n

/ no-call

N

in parentheses).

Model	hi native	hi ITRANS	hi $\bm{\Delta}$	zh native	zh Pinyin	zh $\bm{\Delta}$
claude-haiku-4-5	24.7	67.2	+42.5	23.6	81.0	+57.4
gemini-2.5-flash	39.9	47.0	+7.1	43.9	50.7	+6.8
claude-sonnet-4	44.1	76.3	+32.2	44.1	74.5	+30.4
gpt-4.1	64.9	88.5	+23.6	64.2	91.5	+27.3
Average	43.4	69.7	+26.4	43.9	74.4	+30.5

Appendix

Table 31: Admission for native and romanized requests (%; four closed-weight models).

Model	Baseline	Sys-prompt	Tool-desc	Both
Qwen-2.5-3B	67.2	82.4	56.1	70.9
Qwen-2.5-14B	48.6	43.9	41.2	35.8
Llama-3.1-8B	88.9	75.0	82.4	69.6
Llama-3.2-3B	65.9	47.0	59.5	47.0
Mean	67.7	62.1	59.8	55.8

Appendix

Table 32: Harmful-request admission under four defenses (%; institutional URL).

Language	English	Localized	Change
Chinese	41.7	47.5	+5.8 [2.1, 9.6] ∗
Hindi	26.7	37.9	+11.2 [7.1, 15.8] ∗
Igbo	52.7	64.9	+12.1 [7.6, 16.4] ∗
Swahili	41.7	52.9	+11.2 [6.7, 15.8] ∗

Appendix

Table 33: Pooled harmful-request admission by interface language (four closed-weight models; model–behavior pairs valid in both arms). Change with behavior-clustered 95% interval; ∗ Holm-adjusted McNemar

p<0.01

Table 39

Table 40

Models	Languages	Comprehension	Harmful	Refused	Failed	Borderline
Open 3–14B	EN	99.0	64.3	25.7	8.2	1.8
Open 3–14B	HR	87.8	58.4	26.4	14.7	0.5
Open 3–14B	LR	32.0	52.7	24.1	22.0	1.1
Open 27–31B	EN	99.7	56.0	32.4	5.0	6.6
Open 27–31B	HR	96.8	64.2	26.7	4.7	4.3
Open 27–31B	LR	78.7	62.8	24.6	9.5	3.2

Appendix

Table 38: Mean comprehension and called-answer outcomes (%). Non-harmful shares are audit-based estimates. HR: Chinese/Hindi; LR: Igbo/Swahili/Lao/Haitian Creole.

Open-weight models
	English		Chinese		Hindi		Igbo		Swahili		Lao		Haitian
Model	Comp.	$\mathrm{HR}$	Comp.	$\mathrm{HR}$	Comp.	$\mathrm{HR}$	Comp.	$\mathrm{HR}$	Comp.	$\mathrm{HR}$	Comp.	$\mathrm{HR}$	Comp.	$\mathrm{HR}$
Qwen-2.5-3B	99.7	76.6	95.9	67.6	66.6	63.1	3.7	56.2	8.1	58.7	8.8	49.6	30.1	49.1
Qwen-2.5-14B	100.0	77.3	98.3	76.8	93.6	71.8	8.8	55.4	42.4	77.5	67.9	56.8	65.9	69.3
Llama-3.1-8B	100.0	63.3	91.2	66.7	84.8	71.7	18.5	70.9	75.9	73.9	14.9	60.9	69.3	77.0
Llama-3.2-3B	96.2	40.0	87.0	30.0	85.2	19.3	8.9	23.5	52.8	10.8	12.1	30.4	23.5	22.9

Appendix

Table 39: Tool-free goal recovery and called-answer harm (%; ten models).

(a) All behaviors
Models	Lang.	Understood	Not understood
Open 3–14B	Zh/Hi	75.7 $\to$ 70.7	69.1 $\to$ 59.8
Open 3–14B	LR	79.8 $\to$ 69.4	71.9 $\to$ 55.8
Open 27–31B	Zh/Hi	59.7 $\to$ 71.6	52.4 $\to$ 61.9
Open 27–31B	LR	52.4 $\to$ 68.5	69.3 $\to$ 68.9
Closed-weight	Zh/Hi	50.1 $\to$ 72.0	–

Appendix

Table 40: Post-admission harm (%) by whether the model understood the same request in the tool-free probe, English

\to

target language on the same behaviors (pooled over models; – : fewer than 20 items). LR: Igbo, Swahili, Lao, Haitian Creole.

	Native	Request translated to English ( $\mathrm{AR}$ )
Model	English	zh	hi	ig	sw
gpt-4.1	46.3	58.3	53.4	72.3	60.5
gemini-2.5-flash	39.5	48.6	45.9	46.6	43.6
claude-sonnet-4	49.2	50.7	49.2	51.7	42.2
claude-haiku-4-5	21.3	24.7	19.9	29.7	24.7
Pooled	39.1	45.6	42.1	50.0	42.7

Appendix

Table 41: Admission after translating requests to English (%); columns denote source languages.

Open 3–14B
		$\bm{P(c{=}1)}$		$\bm{P(h{=}1\mid c_{en}{=}c_{L}{=}1)}$		$\bm{P(c{=}1,h{=}1)}$
Set	Language	$\bm{\Delta}$ [95% CI]	Holm $\bm{p}$	$\bm{\Delta}$ [95% CI]	Holm $\bm{p}$	$\bm{\Delta}$ [95% CI]	Holm $\bm{p}$
	Chinese	$+6.3$ [ $+3.5,+9.3$ ]	$<.001$	$-2.6$ [ $-8.5,+3.4$ ]	.393	$-1.3$ [ $-5.0,+2.4$ ]	.969
	Hindi	$+21.4$ [ $+18.0,+24.8$ ]	$<.001$	$-5.8$ [ $-11.9,+0.3$ ]	.128	$+7.6$ [ $+3.4,+11.8$ ]	.003
	Igbo	$+30.5$ [ $+26.3,+34.7$ ]	$<.001$	$-12.3$ [ $-19.2,-5.2$ ]	.001	$+3.4$ [ $-1.9,+8.7$ ]	.666
	Swahili	$+28.0$ [ $+23.9,+32.1$ ]	$<.001$	$-6.7$ [ $-12.9,-0.6$ ]	.102	$+8.6$ [ $+3.9,+13.4$ ]	.004

Appendix

Table 42: Behavior-clustered request-language contrasts (pp from English); harm uses common called sets.

Request language (Table 1 )
Comparison	Language	$\bm{N}$ paired	Call-rate $\bm{p}$	$\bm{P(\text{harm}\mid\text{call})}$ $\bm{p}$
Open 3–14B	Chinese	1184	$1.7\times 10^{-6}$	$0.198$
Open 3–14B	Hindi	1184	$1.8\times 10^{-45}$	$0.011$
Open 3–14B	Igbo	1184	$1.5\times 10^{-69}$	$1.7\times 10^{-6}$
Open 3–14B	Swahili	1184	$5.2\times 10^{-61}$	$0.010$
Open 3–14B	Lao	1184	$1.1\times 10^{-73}$	$8.4\times 10^{-14}$

Appendix

Table 43: Paired McNemar sensitivity tests.

Scored behaviors retained ( $n$ )
Models	En	Zh	Hi	Ig	Sw	Lo	Ht
Open 3–14B	68.9	67.4	65.5	57.0	63.2	52.5	60.4
Open 27–31B	56.6	72.9	69.0	66.3	69.8	76.4	62.4
Closed-weight	45.2	73.5	69.9	83.9	77.8	73.0	66.0
Open 3–14B	473	383	409	433	418	408	414
Open 27–31B	196	188	168	169	182	165	170

Appendix

Table 44: Harm on the behaviors each model admitted in English (

S_{en}

; %, pooled over each group’s models, so English values differ slightly from Table 2 ).

Panel of Figure 6 — Figure 6: Harm on the fixed English-admitted set $S_{en}$ ; 95% Wilson intervals.

Table 49

Model	Scope	$\bm{N}$	Refusal	Failure	Harm-like
Qwen-3.5-27B	All seven	442	86.9	9.5	3.6
Qwen-3.5-27B	Lao + Haitian	131	90.8	8.4	0.8
Gemma-4-31B	All seven	175	30.3	43.4	26.3
Gemma-4-31B	Lao + Haitian	37	32.4	35.1	32.4
closed-weight models	All seven	1,926	84.5	11.0	4.5
closed-weight models	Lao + Haitian	903	86.7	9.2	4.1

Appendix

Table 47: Audited non-harmful answers in 27–31B and closed-weight models (%; seven languages).

Models	Chinese	Hindi	Igbo	Swahili	Lao	Haitian Creole
Open 3–14B	46.1	62.8	72.9	69.0	89.3	73.8
Open 27–31B	14.4	3.2	17.6	29.7	3.2	26.9
Closed-weight	7.3	4.3	26.7	16.6	27.6	38.5

Appendix

Table 48: Pooled English-answer shares under translated requests and English pages (%). Igbo/Haitian Creole shares are upper bounds.

Models	Chinese	Hindi	Igbo	Swahili
Open 3–14B	73.9	91.5	87.6	77.9
Open 27–31B	98.5	97.5	96.9	98.0
Closed-weight	95.7	93.4	99.6	95.1

Appendix

Table 49: Pooled English-answer shares under English requests and translated pages (%).

Table 53

Panel of Figure 7 — Figure 7: Non-harmful 3–14B answers under required page-language responses (% within language).

Model	Language	$\bm{N}$	Refusal	Language deflection	Realization failure	Harm- like
Qwen-3.5-27B	Chinese	59	91.5	5.1	1.7	1.7
Qwen-3.5-27B	Hindi	42	92.9	4.8	2.4	0.0
Qwen-3.5-27B	Igbo	36	91.7	5.6	0.0	2.8
Qwen-3.5-27B	Swahili	46	93.5	2.2	4.3	0.0
Qwen-3.5-27B	Higher-resource	101	92.1	5.0	2.0	1.0
Qwen-3.5-27B	Lower-resource	82	92.7	3.7	2.4	1.2

Appendix

Table 52: Audited non-harmful 27–31B page-language answers (% within row;

N

audited).

Model	Language	English directive	Page-language directive
Qwen-3.5-27B	Chinese	16.0 (106)	48.7 (115)
Qwen-3.5-27B	Hindi	9.9 (101)	8.7 (46)
Qwen-3.5-27B	Igbo	11.0 (100)	12.2 (41)
Qwen-3.5-27B	Swahili	9.8 (102)	8.0 (50)
Qwen-3.5-27B	Paired, four languages	17.8 (208)	28.8 (208)
Gemma-4-31B	Chinese	77.9 (104)	80.2 (111)

Appendix

Table 53: 27–31B called-answer harm (%; scored counts in parentheses). Paired rows use common scored behaviors.

Models	Lang.	$n$	B1	B2	Difference
Closed-weight	Zh/Hi	757	51.9	64.2	$+12.3$ [9.2, 15.5] ∗
Closed-weight	Ig/Sw	691	48.5	49.9	$+1.4$ [ $-2.2$ , 5.1]
Open 27–31B	Zh/Hi	320	53.8	66.3	$+12.5$ [7.1, 17.9] ∗
Open 27–31B	Ig/Sw	260	60.0	59.2	$-0.8$ [ $-4.7$ , 3.1]
Open 3–14B	Zh/Hi	548	81.4	79.6	$-1.8$ [ $-5.2$ , 1.6]
Open 3–14B	Ig/Sw	543	78.6	65.4	$-13.3$ [ $-18.0$ , $-8.4$ ] ∗

Appendix

Table 54: Paired-called harm by response directive (%; difference in points with 95% interval).

Lang.	B1	B2	Difference
Chinese	35.6	27.8	$-7.8$ [ $-11.8$ , $-3.6$ ]
Hindi	36.6	26.6	$-9.9$ [ $-14.3$ , $-5.4$ ]
Igbo	31.3	38.6	$+7.3$ [3.0, 11.9]
Swahili	30.6	48.3	$+17.6$ [13.5, 22.5]
All four	33.6	34.9	$+1.3$ [ $-1.5$ , 4.2]

Appendix

Table 55: Refusal share among closed-weight paired answers by response directive (%; difference with 95% interval).

Panel of Figure 8 — Figure 8: Mean changes from English across six non-English request languages (pp). Harm uses each condition’s scored called answers.

Language Model Safety Evaluation

Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.

See all 540 papers

Oct 16, 2026

Before Retrieval Begins: Safety and Tool-Use Decisions in LLM Agents

Large language models increasingly act as agents that call external tools, yet most tool-safety research examines only what follows a tool call. The prior decision of whether to call the tool remains largely unexamined and can influence downstream safety. Across the seven open-weight models, the unweighted model-average harm rate after admitted harmful requests is 74.5%. To identify what governs this decision, we apply a factorial intervention to the input request. Within our generated construction, neutralizing the requested-operation component produces the largest admission contrast (19 pp). A linear direction estimated from this contrast influences held-out admission in four of seven models; effects on operation-neutralized requests in some models show that it is not an operation-specific mechanism. We then test whether written refusal is behaviorally and representationally coupled to the same decision. Written refusal accompanies 33.6% of admitted harmful calls. Plain-text refusal training reduces unsafe admission in models that retain benign tool use, showing that response-level safety training can transfer to the call decision. Together, these results establish a partial, model-dependent relation between written refusal and tool admission. They motivate direct evaluation and supervision of tool admission rather than treating text refusal as its proxy.

Aditya Nawal, Manit Baser, Mohan Gurusamy

National University of Singapore

LLM Agent Safety Tool-Using Language Agents Language Model Safety Evaluation LLM Refusal Behavior

Ask PDF

Aug 17, 2026

BabelSteering: Multilingual Safety Alignment via English Steering Vectors

Large language models (LLMs) are deployed globally in high-stakes settings, yet most safety research and alignment efforts remain concentrated on English. Thus, users interacting with LLMs in other languages may encounter weaker safeguards despite relying on the same systems for similarly sensitive tasks. In this work, we investigate whether safety signals learned from a high-resource language, like English, can improve multilingual safety. We propose BabelSteering, an activation steering method that acts as a lightweight inference- time intervention, using refusal directions derived from English safety supervision to generalize across languages. Our evaluation includes eight languages and jointly measures refusal of harmful requests, over-refusal, and general task utility. The results show that BabelSteering increases the refusal of harmful requests across languages, with only a marginal to no reduction in task utility but with some increase in refusal of pseudo-harmful prompts. For example, for Gemma 7B, we see an average increase in the refusal of harmful prompts across languages of 11 percentage points (pp), with individual languages like Bengali seeing an increase of 17 pp, with no loss of utility on Global MMLU, while pseudo-harmful refusals increase by 13 pp on average. We also introduce a multilingual translation-and-evaluation pipeline to facilitate future work on cross-lingual safety interventions. Overall, our findings suggest that activation steering may provide a practical, low- cost mechanism for extending English-derived safety signals to other languages. Warning: this paper contains examples with unsafe content

Emma V. Stein, Dominik Meier, Terry Ruas +2

University of Göttingen, Germany · German State Police NRW, Germany

LLM Safety Alignment Language Model Steering Multilingual Language Model Evaluation

Ask PDF

Jul 11, 2026

Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety

Safety alignment in large language models remains brittle across languages: prompts reliably refused in English can elicit harmful compliance in non-English and low-resource settings. We introduce Minionese, a multilingual jailbreak benchmark spanning 18 languages, 4 resource tiers, and 4 perturbation types (standard translation, code-switching, transliteration, and translationese), paired with a geometric mechanistic analysis of refusal failure across language tiers. We show that each attack type produces a distinct vulnerability profile: transliteration vulnerability is mediated by script identity, code-switching maintains effectiveness through the lowest-resource tier, and a sharp safety regime transition between Tiers 2 and 3 is consistent across all models. Mechanistically, low-resource jailbreaks succeed by routing harmful content through a geometrically misaligned subspace that projects insufficiently onto the refusal directions, leaving the refusal mechanism intact but untriggered. These findings show that English-only safety evaluations are insufficient; they require accounting for script family, perturbation type, and per-language alignment coverage. The benchmark and analysis code is at https://github.com/Brentkong/Minionese-Comprehensive-Benchmark-and-Mechanistic-Study-of-Multilingual-LLM-Safety.git.

Chigozirim Ifebi, Brent Kong, Ayushi Mehrotra

California Institute of Technology.

LLM Safety Benchmarks Multilingual Language Model Evaluation Language Model Safety Evaluation Jailbreak Attacks Mechanistic Interpretability

Ask PDF

Abstract

Language Model Safety Evaluation

Explore similar work

Before Retrieval Begins: Safety and Tool-Use Decisions in LLM Agents

BabelSteering: Multilingual Safety Alignment via English Steering Vectors

Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety