cs.CROct 15, 2026

Language Is a Security Surface for Agent Guards

Authors: Hasan Demirkiran, Jens Ernstberger

Organizations: Kontext

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Step-level guards decide, before each tool call, whether an agent's next action is safe given the user's request and the trajectory so far. Users write those requests in many languages, while guard prompts and the benchmarks on which purpose-built guards are trained and evaluated are English. We ask whether a guard's verdict depends on the language of the request. We build LinguaGuardBench from the 1,220 labelled steps of TS-Bench AgentDojo-Traj: we translate only the user instruction into German, French, Turkish, Swahili and Amharic, and keep every tool call, observation and label byte-identical, so any change in a verdict is attributable to language alone. Six English rewordings and a whitespace placebo serve as controls, and managed guards are run three times to measure their noise. Across 19 guards, from frontier APIs to 4B open models and four purpose-built agent guards, most open, small and purpose-built guards change verdicts under translation beyond their own noise floor, and the effect grows with the rarity of the language: among the nine guards below 13B parameters, the mean placebo-adjusted flip rate rises from +1.5 pp on French to +4.4 pp on Amharic. Purpose-built guards are the most sensitive. The direction of the damage tends to follow guard size: open guards of 12-30B parameters block more benign steps (up to +5.0 pp false positives), and every significant loss of recall is in a guard below 10B (up to -7.4 pp). Frontier guards shift by at most about one point; the strongest, Gemini 3.8 Flash, returns identical verdicts with and without the request. The damage is concentrated in a few instructions and, in a post-hoc analysis, falls on steps where the agent's reasoning restates the English request, consistent with guards reacting to inconsistency between channels. English paraphrase moves every guard less than translation does. We release the benchmark and all 489k verdicts.

Figures & tables

Figure 1: Change in attack recall (left) and false-positive rate (right) when the user request is translated, relative to the original English, per guard and language. Points with 95% trajectory-clustered bootstrap intervals; faded points have intervals that include zero. Marker: high- (circle), mid- (square), low-resource (triangle) language. Colour: frontier (blue), open 17–120B (orange), small << 13B (red), specialised (green).
 ΔFPR\Delta\mathrm{FPR} (pp)ΔTPR\Delta\mathrm{TPR} (pp)
Guarddefrtrswamdefrtrswam
ShieldAgent 7B−7.6\mathbf{-7.6}−5.4\mathbf{-5.4}−3.8\mathbf{-3.8}−5.3\mathbf{-5.3}−3.8\mathbf{-3.8}−0.9-0.9+2.3−1.4-1.4−0.6-0.6+1.7
Nemotron Nano 9B−0.7-0.7−1.2\mathbf{-1.2}−0.6-0.6−1.0\mathbf{-1.0}−1.4\mathbf{-1.4}−3.1\mathbf{-3.1}−4.5\mathbf{-4.5}−4.3\mathbf{-4.3}−3.7\mathbf{-3.7}−7.4\mathbf{-7.4}
Nemotron 3.5 30B-A3B+4.7+0.1+4.1−0.8-0.8+5.00.0−0.6-0.6+0.3+0.3+1.7
TS-Guard 7.6B+3.6+1.0+2.8−2.2\mathbf{-2.2}+1.8−2.6-2.6−0.9-0.9−3.1-3.1−0.6-0.6+0.3
Gemma 4 26B-A4B (self-hosted)+0.6+0.2+2.4+1.2+3.60.00.00.00.00.0
Table 1: Translation-induced error shifts (percentage points vs. original) for every guard and language; bold: 95% trajectory-clustered interval excludes zero. Positive ΔFPR\Delta\mathrm{FPR} : more benign steps blocked; negative ΔTPR\Delta\mathrm{TPR} : more attacks missed. Guards sorted by total significant shift. Partner-hosted guards were not run on Swahili/Amharic (–).
Figure 2: Mean shift under translation (pooled over the languages each guard was run on) against guard size. Filled triangles pointing down: ΔTPR\Delta\mathrm{TPR} (negative means attacks missed); open triangles pointing up: ΔFPR\Delta\mathrm{FPR} (positive means benign steps blocked); a vertical line joins the two markers of one guard. The degenerate Gemma 4 E2B is omitted. Over-blocking is a 12–30B phenomenon; missed attacks appear below 10B.
GuardTierBACCFPRTPRPlacebo flip
Gemini 3.8 Flash (scanner control)frontier0.9830.9830.0120.0120.9770.9770.0010.001
Gemini 3.8 Flashfrontier0.9830.9830.0120.0120.9770.9770.0000.000
Claude Sonnet 5frontier0.9580.9580.0720.0720.9880.9880.0170.017
TS-Guard (7.6B, specialised)specialised0.9100.9100.0710.0710.8920.8920.0450.045
gpt-oss-120bopen0.8990.8990.1790.1790.9760.9760.1270.127
Gemma 4 12Bsmall0.8490.8490.3030.3031.0001.0000.0100.010
Table 2: Original-wording operating point and placebo flip rate (the guard’s own noise floor) for all guards, sorted by balanced accuracy. Managed guards K=3K=3 (3,660 verdicts), self-deployed K=1K=1 (1,220). Gemma 4 E2B is degenerate (blocks 6.5% of steps) and excluded from interpretation. ShieldAgent, AgentDoG and Safiron judge the trajectory prefix, so their FPR/TPR against step labels are not comparable with the step-level guards.
Figure 3: Placebo-adjusted verdict flip rate, split by whether the step’s history restates the request (leaky) or not, pooled over translation conditions and, for comparison, over paraphrase conditions. For the strong guards and the recall-losers the rate is higher on leaky steps; for the over-blockers it is higher on non-leaky steps.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Guardplacebotyposlexicalterseverbosecasualhumandefrtrswam
Gemini 3.8 Flash (scanner)0.10.00.00.00.00.00.00.00.00.00.00.0
Gemini 3.8 Flash0.00.00.00.00.00.00.00.00.00.00.00.0
Claude Sonnet 51.71.92.12.22.92.32.42.32.72.8––
GLM 5.26.15.86.25.76.45.76.36.66.26.5––
gpt-oss-120b12.712.413.412.912.713.114.9–––––
Nemotron 3.5 30B-A3B4.24.33.94.64.85.25.36.04.86.66.48.3
Appendix
Table 3: Raw verdict flip rate (%) vs. the original wording, per guard and condition. The placebo column is each guard’s own noise floor.
Guardtyposlexicalterseverbosecasualhuman
Gemini 3.8 Flash (scanner)0.00.00.00.00.00.0
Gemini 3.8 Flash0.00.00.00.00.00.0
Claude Sonnet 50.0−0.2-0.2−0.2-0.2+0.5−0.2-0.2+0.9
GLM 5.2−0.6-0.6+0.2+0.2+1.0+0.1+0.2
gpt-oss-120b−4.4\mathbf{-4.4}−3.8\mathbf{-3.8}−5.6\mathbf{-5.6}−4.0\mathbf{-4.0}−3.2\mathbf{-3.2}−5.1\mathbf{-5.1}
Nemotron 3.5 30B-A3B+1.3−0.1-0.1−1.3-1.3−1.5-1.5−0.3-0.3−0.9-0.9
Appendix
Table 4: ΔFPR\Delta\mathrm{FPR} (pp) under the six paraphrase conditions; bold: 95% interval excludes 0.
Guardtyposlexicalterseverbosecasualhuman
Gemini 3.8 Flash (scanner)0.00.00.00.00.00.0
Gemini 3.8 Flash0.00.00.00.00.00.0
Claude Sonnet 5−0.5-0.5−0.2-0.2−0.5-0.50.0+0.1−0.1-0.1
GLM 5.20.0+0.10.0+0.2+0.1−0.1-0.1
gpt-oss-120b−0.9-0.9−0.9-0.9−1.3\mathbf{-1.3}+0.5+0.1−0.7-0.7
Nemotron 3.5 30B-A3B0.0−0.3-0.3−0.3-0.3+0.3+0.9−0.3-0.3
Appendix
Table 5: ΔTPR\Delta\mathrm{TPR} (pp) under the six paraphrase conditions; bold: 95% interval excludes 0.
 ΔFPR\Delta\mathrm{FPR}ΔTPR\Delta\mathrm{TPR}adj. flip
Guardhighmidlowhighmidlowhighmidlow
Gemini 3.8 Flash (scanner)0.00.00.00.00.00.0+-0.0−0.1-0.1−0.1-0.1
Gemini 3.8 Flash0.00.00.00.00.00.0+0.0+0.0+0.0
Claude Sonnet 5−0.2-0.2+0.5–−0.1-0.1−0.4-0.4–+0.8+1.1–
GLM 5.2−1.1\mathbf{-1.1}−1.0-1.0–0.00.0–+0.3+0.4–
Nemotron 3.5 30B-A3B+2.4+4.1+2.1−0.3-0.3+0.3+1.0+1.2+2.5+3.2
Appendix
Table 6: Translation shifts pooled by language-resource tier (high: German, French, Joshi et al. class 5; mid: Turkish, class 4; low: Swahili, Amharic, class 2), percentage points vs. original; bold: 95% trajectory-clustered interval excludes 0. Flip: placebo-adjusted verdict flip rate.
Guarddefrtrswam
Gemini 3.8 Flash (scanner)0.00.00.00.00.0
Gemini 3.8 Flash0.00.00.00.00.0
Claude Sonnet 5−0.1-0.1−0.3-0.3+0.5––
GLM 5.2−1.0-1.0−1.1\mathbf{-1.1}−1.0-1.0––
gpt-oss-120b–––––
Nemotron 3.5 30B-A3B+4.7+0.1+4.1−0.8-0.8+5.0
Appendix
Table 7: ΔFPR\Delta\mathrm{FPR} (pp) under translation; bold: 95% interval excludes 0.
Guarddefrtrswam
Gemini 3.8 Flash (scanner)0.00.00.00.00.0
Gemini 3.8 Flash0.00.00.00.00.0
Claude Sonnet 5−0.2-0.20.0−0.4-0.4––
GLM 5.20.0−0.1-0.10.0––
gpt-oss-120b–––––
Nemotron 3.5 30B-A3B0.0−0.6-0.6+0.3+0.3+1.7
Appendix
Table 8: ΔTPR\Delta\mathrm{TPR} (pp) under translation; bold: 95% interval excludes 0.
Guardpara @0.5para @1trans @0.5trans @1
Gemini 3.8 Flash (scanner)-0.0+0.1-0.0+0.1
Gemini 3.8 Flash+0.0+0.0+0.0-0.0
Claude Sonnet 5+0.6+0.2+0.9+0.2
GLM 5.2-0.1+0.1+0.3+0.1
gpt-oss-120b+0.6+0.9––
Nemotron 3.5 30B-A3B+0.5+2.0+2.2+5.7
Appendix
Table 9: Placebo-adjusted flip rate (pp), pooled over paraphrase and over translation conditions, under the default rule (block if composite ≥0.5\geq 0.5 ) and the strict rule (composite =1=1 ).
More in this topic

LLM Guardrails

Large language model guardrails detect or constrain harmful outputs, prompt attacks, and unsafe agent actions before they cause harm.

See all 195 papers

Explore similar work

CardsList
Oct 16, 2026

ToolRosetta: Surface-Form Invariance of Tool-Call Guards

Step-level guards for tool-using agents are evaluated on tidy JSON tool calls, but deployed coding agents act through execute_bash and execute_code tools whose payloads are strings. We introduce ToolRosetta, which re-renders every candidate action in the AgentDojo and AgentHarm splits of TS-Bench as a Python call, as a shell command, and without the agent's rationale. The renderings are produced by code and verified by round-trip parsing, so the user request, the history, the label and the action's effect stay fixed while only the surface form changes. Across eleven guards with two placebo controls, frontier and strong open-weight guards are stable: no frontier error rate moves by more than 3.4 points. Four of five specialized agent guards are not: under the shell form their false-positive rates rise by 21 to 47 points on AgentDojo, and TS-Guard's by 38 points on its own home benchmark, blocking between a third and nine in ten benign actions; the fifth, StepGuard, does not move. A small open guard drifts five points more permissive. Removing the agent's rationale moves most guards whose input carries it, in guard-specific directions. A request-blind control on one guard shows that the rationale is where intent is read from. A parser that restores the canonical call in front of the guard removes the effect for these renderings, by construction. Benchmark accuracy on JSON calls does not predict a guard's operating point behind a code-acting agent.

Hasan Demirkiran, Jens Ernstberger

Kontext

LLM GuardrailsRuntime Enforcement for AI AgentsAI Agent Safety Benchmarks
AskPDF
Jul 23, 2026

GuardianAgentBench: Where Agents Fail and How to Guard Them

As large language model agents increasingly operate autonomously with access to tools and external environments, ensuring their safe and reliable behavior becomes critical. We present GuardianAgentBench (GABench), a benchmark of 580 scenarios across six domains evaluated on three production-ready frameworks: LangChain, LlamaIndex, and Vectara. The benchmark incorporates rigorous multi-stage validation and five adversarial attack modes. Experiments with six state-of-the-art models reveal that even the strongest configuration achieves only 74.8% overall accuracy and expose two distinct failure regimes: stronger models under-call required tools, while weaker models mis-select and over-call tools. Performance degrades monotonically with both tool-set size and sequential turn depth, with long-horizon planning proving the steeper bottleneck. Our guardrail implementation consistently outperforms system-prompt-based defenses across all models, recovering 19.9% of failures at a false positive rate of just 0.5%. These results demonstrate that execution-time structural intervention improves safety without disrupting correct agent behavior.

Vishal Ishwar Naik, Chenyu Xu, Donna Dong +5

Vectara, Inc., Palo Alto, CA, USA · Iowa State University, Ames, IA, USA

AI Agent ReliabilityLLM GuardrailsAI Agent SafetyRuntime Enforcement for AI AgentsAI Agent Safety BenchmarksTool-Using Language Agents+1
AskPDF
Oct 16, 2026

Disentangling Language Effects on Tool Admission and Post-Retrieval Safety

LLM agents increasingly use retrieval tools across languages, yet their safety evaluation remains English-centric. It remains unclear whether language affects tool admission, post-retrieval generation, or both. We independently vary request, retrieved-page, and required response languages in a controlled URL-retrieval setting across ten models and seven languages. We identify Safety Loss through Interlingual Processing (SLIP). Non-English harmful requests increase mean tool admission from 46.9% to 67.4%, with a positive mean change in every model. This pattern also persists in an autonomous retrieval setup across all four models tested in this setting. Requiring answers in the page language increases pooled harmfulness on matched tool calls from 50.3% to 57.4% across four closed-weight models, mainly in Chinese and Hindi. Translating pages alone produces no consistent harm increase in eight models with comparable inference settings. Automated audits show that a non-harmful response may be a genuine safety refusal, a deflection citing language difficulties, or an unusable answer. Harmfulness labels alone cannot distinguish these outcomes. Multilingual safety evaluations should therefore measure tool admission and response harmfulness separately and distinguish refusal from capability failure.

Aditya Nawal, Manit Baser, Mohan Gurusamy

National University of Singapore

Language Model Safety EvaluationLLM Agent SafetyMultilingual Language Model Evaluation
AskPDF