The Same Zero: Why Identical ASR Can Imply Different Guarantees in LLM-Agent Security
Abstract
LLM-agent security has produced a dense landscape of defenses - prompt hardening, content filters, permission gates, sandboxes - yet no framework tells a deployer what a defense actually guarantees, or where that guarantee comes from. We apply Verification Autonomy Levels (VAL) - L0: LLM self-declaration; L1: deterministic rules; L2: objective ground truth; L3/L4: decidable completeness; L5: impossible - to 22 agent-security defenses; the taxonomy is falsifiable (10/10 prediction hits on frozen cards, flagged). We run the first controlled deployment-value comparison: at equal budget, a VAL-guided stack (confirmation gate + schema sandbox) versus a mainstream intuition stack (prompt hardening + keyword filter), 50 scenarios, 12 attack variants, adaptive/white-box/PAIR escalation (~7,000 testbed calls; ~10,000 harness calls on AgentDojo/JADE). The VAL stack holds 0.000 attack success at 1.000 benign success (0.5% ASR at 79.7% utility on AgentDojo banking vs 4.3% undefended); the intuition stack reaches 0.000 ASR but kills all benign actions - security by model-behavior luck, not structure. Across testbeds of rising attack-surface hardness the intuition stack's zero drifts (0->1.9%->6.2%, n=16 on JADE) while the VAL stack's holds within its ODD (0->0->0), its only breach a disclosed out-of-ODD password gap (0.5%). The same zero, two different guarantees: zero is an outcome, not a guarantee.
Figures & tables
| Stack | ASR (both victims) | Benign (both victims) | What the zero is |
|---|---|---|---|
| ND (no defense) | 0.333 / 0.253 | 1.000 / 1.000 | no defense |
| V (VAL: gate + sandbox) | 0.000 / 0.000 | 1.000 / 1.000 | structure - platform record + schema, victim-independent |
| N (intuition: hardening + filter) | 0.000 / 0.000 | 0.000 / 0.000 | luck - the model’s refusal behavior, model-dependent |
| Config | ASR | Compliance | Benign | gain vs ND |
|---|---|---|---|---|
| ND (no defense) | 0.333 | 0.492 | 1.000 [0.84, 1.00] | — |
| D1 prompt hardening (L0) | 0.000 | 0.219 | 0.000 [0.00, 0.16] | 0.333, benign destroyed |
| D2 keyword filter (L1) | 0.200 | 0.492 | 1.000 [0.84, 1.00] | 0.133 |
| D3 confirmation gate (L1/L2) | 0.000 | 0.497 | 1.000 [0.84, 1.00] | 0.333 |
| D4 parameter sandbox (L3) | 0.067 | 0.494 | 1.000 [0.84, 1.00] | 0.266 |
| The same zero two defenses both report 0.000 ASR | Escalation the zeros separate by failure morphology | Provenance funnel same zero, different guarantee |
| Comparison | Raters | Overlap | Exact | Weakest-anchor | Scope | |
|---|---|---|---|---|---|---|
| Author vs LLM | A vs B / B2 / B3 | 28 | 50.0–60.7% | 0.434–0.549 | 0.513–0.642 | v2.0–v2.2; not validity evidence |
| LLM vs LLM (frozen protocol) | five LLM raters (B2–B6), seven archived pairs | 48–54 | 81.2–90.7% | 0.766–0.882 | 0.811–0.926 | v2.2/v2.3 |
| Full corpus, latest pair | B5 vs B6 | 54 | 90.7% | 0.882 | 0.926 | domains: sys 11/11, agentsec 6/6 |
| Security subset | B6 vs B7 | 22 | 20/22 = 90.9% | 0.880 | 0.870 | agent-security schemes; boundaries L2/L0 and L4/L3 |
| Level | Defenses | Anchor / semantics | Predicted behavior |
| Config | DeepSeek victim ASR / compl. / benign | Llama victim ASR / compl. / benign |
|---|---|---|
| ND (no defense) | 0.333 / 0.492 / 1.000 | 0.253 / 0.328 / 1.000 |
| N (D1+D2) | 0.000 / 0.233 / 0.000 | 0.000 / 0.014 / 0.000 |
| V (D3+D4) | 0.000 / 0.497 / 1.000 | 0.000 / 0.317 / 1.000 |
| Config | ASR (n = 576, 4 attack families, full banking) | Utility |
|---|---|---|
| ND (no defense) | 4.3% [3.0, 6.3] | 86.6% [83.6, 89.2] |
| N (prompt hardening + keyword filter) | 1.9% [1.1, 3.4] | 87.7% [84.7, 90.1] |
| V (confirmation gate + parameter sandbox) | 0.5% [0.2, 1.5] | 79.7% [76.2, 82.8] |
| AgentDojo tool_filter | 0.0% [0.0, 0.7] | 25.0% [21.6, 28.7] |
| AgentDojo repeat_user_prompt | 3.5% [2.3, 5.3] | 85.9% [82.9, 88.5] |
| Suite | V ASR | ND ASR | V utility | ND utility |
|---|---|---|---|---|
| travel (n=560) | 4.11% | 4.64% | 78.6% | 76.4% |
| workspace (n=960) | 0.42% | 0.31% | 87.0% | 85.9% |
| slack (n=420) | 8.81% | 7.86% | 74.8% | 82.6% |
| Config | Static ASR | Adaptive ASR | White-box ASR | White-box compliance | PAIR-lite ASR |
|---|---|---|---|---|---|
| D1 | 0.000 | 0.000 | 0.000 | 0.333 | — |
| D2 | 0.200 | 0.467 | — | — | — |
| D3 | 0.000 | 0.000 | 0.000 | 0.567 | — |
| D4 | 0.067 | 0.200 | — | — | — |
| V | 0.000 | 0.000 | 0.000 | 0.567 | 0.000 |
| Config | Tool-layer attacks (task 1 + task 2) | Output-layer mentions (task 6) | Utility |
|---|---|---|---|
| ND (no defense) | 3 | 23 | 76.4% |
| V (original gate) | 1 | 22 | 78.6% |
| A (enumeration-completed) | 1 | 19 | 72.0% |
| Cfail (fail-secure) | 0 | 23 | 54.6% |
| D3 (intent-anchor) | 0 | 20 | 66.6% |
| Judge | Pool (end / min) | Contract | D parse | D repetition |
|---|---|---|---|---|
| llama3.1:8b (stopped at round 16) | 41 / 1 | 2 rounds citing out-of-pool ids; 5/15 internally inconsistent | clean | max 0.39 |
| qwen2.5:14b (30 rounds, completed) | 10 / 0 (emptied at round 12) | 0 out-of-pool ids; 12/30 internally inconsistent | clean | max 0.86 |
LLM Agent Security
Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.
See all 332 papers