The Same Zero: Why Identical ASR Can Imply Different Guarantees in LLM-Agent Security

YaJie Yin

cs.CROct 3, 2026

The Same Zero: Why Identical ASR Can Imply Different Guarantees in LLM-Agent Security

Authors: YaJie Yin

Abstract

LLM-agent security has produced a dense landscape of defenses - prompt hardening, content filters, permission gates, sandboxes - yet no framework tells a deployer what a defense actually guarantees, or where that guarantee comes from. We apply Verification Autonomy Levels (VAL) - L0: LLM self-declaration; L1: deterministic rules; L2: objective ground truth; L3/L4: decidable completeness; L5: impossible - to 22 agent-security defenses; the taxonomy is falsifiable (10/10 prediction hits on frozen cards, flagged). We run the first controlled deployment-value comparison: at equal budget, a VAL-guided stack (confirmation gate + schema sandbox) versus a mainstream intuition stack (prompt hardening + keyword filter), 50 scenarios, 12 attack variants, adaptive/white-box/PAIR escalation (~7,000 testbed calls; ~10,000 harness calls on AgentDojo/JADE). The VAL stack holds 0.000 attack success at 1.000 benign success (0.5% ASR at 79.7% utility on AgentDojo banking vs 4.3% undefended); the intuition stack reaches 0.000 ASR but kills all benign actions - security by model-behavior luck, not structure. Across testbeds of rising attack-surface hardness the intuition stack's zero drifts (0->1.9%->6.2%, n=16 on JADE) while the VAL stack's holds within its ODD (0->0->0), its only breach a disclosed out-of-ODD password gap (0.5%). The same zero, two different guarantees: zero is an outcome, not a guarantee.

Figures & tables

Stack	ASR (both victims)	Benign (both victims)	What the zero is

ND (no defense)	0.333 / 0.253	1.000 / 1.000	no defense
V (VAL: gate + sandbox)	0.000 / 0.000	1.000 / 1.000	structure - platform record + schema, victim-independent
N (intuition: hardening + filter)	0.000 / 0.000	0.000 / 0.000	luck - the model’s refusal behavior, model-dependent

Table 1

Config	ASR	Compliance	Benign	gain vs ND

ND (no defense)	0.333	0.492	1.000 [0.84, 1.00]	—
D1 prompt hardening (L0)	0.000	0.219	0.000 [0.00, 0.16]	0.333, benign destroyed
D2 keyword filter (L1)	0.200	0.492	1.000 [0.84, 1.00]	0.133
D3 confirmation gate (L1/L2)	0.000	0.497	1.000 [0.84, 1.00]	0.333
D4 parameter sandbox (L3)	0.067	0.494	1.000 [0.84, 1.00]	0.266

Table 2

The same zero two defenses both report 0.000 ASR

\longrightarrow

Escalation

Z(\alpha)

the zeros separate by failure morphology

\longrightarrow

Provenance funnel same zero, different guarantee

Figure 1: Framework overview. Identical observed zeros do not imply identical guarantees: escalation separates the zeros by failure morphology, and provenance decides which of them is structural.

Comparison	Raters	Overlap	Exact	$\kappa$	Weakest-anchor $\kappa$	Scope

Author vs LLM	A vs B / B2 / B3	28	50.0–60.7%	0.434–0.549	0.513–0.642	v2.0–v2.2; not validity evidence
LLM vs LLM (frozen protocol)	five LLM raters (B2–B6), seven archived pairs	48–54	81.2–90.7%	0.766–0.882	0.811–0.926	v2.2/v2.3
Full corpus, latest pair	B5 vs B6	54	90.7%	0.882	0.926	domains: sys 11/11, agentsec 6/6
Security subset	B6 vs B7	22	20/22 = 90.9%	0.880	0.870	agent-security schemes; boundaries L2/L0 and L4/L3
Level	Defenses	Anchor / semantics	Predicted behavior

Table 4

Config	DeepSeek victim ASR / compl. / benign	Llama victim ASR / compl. / benign

ND (no defense)	0.333 / 0.492 / 1.000	0.253 / 0.328 / 1.000
N (D1+D2)	0.000 / 0.233 / 0.000	0.000 / 0.014 / 0.000
V (D3+D4)	0.000 / 0.497 / 1.000	0.000 / 0.317 / 1.000

Table 5

Config	ASR (n = 576, 4 attack families, full banking)	Utility

ND (no defense)	4.3% [3.0, 6.3]	86.6% [83.6, 89.2]
N (prompt hardening + keyword filter)	1.9% [1.1, 3.4]	87.7% [84.7, 90.1]
V (confirmation gate + parameter sandbox)	0.5% [0.2, 1.5]	79.7% [76.2, 82.8]
AgentDojo tool_filter	0.0% [0.0, 0.7]	25.0% [21.6, 28.7]
AgentDojo repeat_user_prompt	3.5% [2.3, 5.3]	85.9% [82.9, 88.5]

Table 6

Suite	V ASR	ND ASR	V utility	ND utility

travel (n=560)	4.11%	4.64%	78.6%	76.4%
workspace (n=960)	0.42%	0.31%	87.0%	85.9%
slack (n=420)	8.81%	7.86%	74.8%	82.6%

Table 7

Config	Static ASR	Adaptive ASR	White-box ASR	White-box compliance	PAIR-lite ASR

D1	0.000	0.000	0.000	0.333	—
D2	0.200	0.467	—	—	—
D3	0.000	0.000	0.000	0.567	—
D4	0.067	0.200	—	—	—
V	0.000	0.000	0.000	0.567	0.000

Table 8

Config	Tool-layer attacks (task 1 + task 2)	Output-layer mentions (task 6)	Utility

ND (no defense)	3	23	76.4%
V (original gate)	1	22	78.6%
A (enumeration-completed)	1	19	72.0%
Cfail (fail-secure)	0	23	54.6%
D3 (intent-anchor)	0	20	66.6%

Table 9

Judge	Pool (end / min)	Contract	D parse	D repetition

llama3.1:8b (stopped at round 16)	41 / 1	2 rounds citing out-of-pool ids; 5/15 internally inconsistent	clean	max 0.39
qwen2.5:14b (30 rounds, completed)	10 / 0 (emptied at round 12)	0 out-of-pool ids; 12/30 internally inconsistent	clean	max 0.86

Table 10

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 332 papers

Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
Jun 9, 2026Yuchen Ling, Shengcheng Yu, Zhenyu Chen +1Prompt Injection LLM Agent Security
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Technical University of Munich, Heilbronn, Germany
Kairos: Transactional Security Enforcement at the System Boundary for LLM Agents
Oct 13, 2026Kristoffer Hempel, Nils Loose, Lorenzo Cavallaro +1LLM Agent Security Runtime Enforcement for AI Agents
Universität zu Lübeck · University College London · Mohamed bin Zayed University of Artificial Intelligence
Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deployment
May 18, 2026S. Bensalem, Y. Dong, M. Franzle +6AI Agent Reliability LLM Guardrails
1CSX-AI, France · University of Liverpool, UK · 3Carl von Ossietzky Universität Oldenburg, Germany +2

Jun 9, 2026J/K move · Enter open · S save

Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation

Yuchen Ling, Shengcheng Yu, Zhenyu Chen, Chunrong Fang

State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Technical University of Munich, Heilbronn, Germany

Prompt Injection LLM Agent Security AI Agent Security Benchmarks

Open paper AskPDF arXiv

Large language model (LLM) agents are rapidly moving from conversational interfaces to software components that plan, invoke tools, maintain memory, and act on external environments. This transition changes the nature of security risk. In agentic settings, failures are no longer limited to unsafe text generation. Untrusted content may redirect control flow, misuse tool privileges, corrupt persistent state, leak sensitive information, or trigger harmful external actions. At the same time, research on LLM agent security is expanding quickly but remains fragmented across attack families, defense layers, application domains, and evaluation settings. This paper synthesizes 247 papers through a lifecycle-based, systems-oriented framework that models agent security around the interaction of information flow, delegated authority, and persistent state. We organize the literature around four questions: how LLM agent security should be modeled, which threat surfaces and attack families dominate, what defenses have been proposed and with what tradeoffs, and how security claims are evaluated. We find that prompt injection and tool-mediated control-flow hijacking still dominate the field, while persistent state corruption and multi-agent propagation are becoming central emerging concerns. We further find that current defenses provide useful building blocks but remain weakly compositional, and that existing benchmarks still underrepresent long-horizon, stateful, and deployment-sensitive risks. We argue that secure LLM agents require explicit trust boundaries, principled privilege control, provenance-aware state management, and evaluation practices aligned with realistic operational settings.

Abstract

LLM Agent Security

Explore similar work

Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation

Kairos: Transactional Security Enforcement at the System Boundary for LLM Agents

Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deployment