Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing

Israt Moyeen Noumi; Tarannum Ahmed Nowshin; Md. Mehedi Hasan Nipu; Mohammad Sakib Mahmood; Md. Jakir Hossain; M. F. Mridha

cs.CLAug 16, 2026

Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing

Authors: Israt Moyeen Noumi, Tarannum Ahmed Nowshin, Md. Mehedi Hasan Nipu, Mohammad Sakib Mahmood, Md. Jakir Hossain, M. F. Mridha

Organizations: Department of Computer Science and Engineering, Ahsanullah University of Science and Technology, Dhaka, Bangladesh · Department of Computer Science and Engineering, BRAC University, Dhaka, Bangladesh · Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh · Department of Computer Science, Missouri State University, Springfield, MO 65897 USA ·

Abstract

Agentic security uses large-language-model (LLM) agents to plan, dispatch, and interpret security tools. As these systems move from demonstrations to deployed products, practitioners repeatedly encounter the same operational failures. We systematize these failures through a hands-on evaluation of ten widely used static, dynamic, cloud, orchestration, and AI red-teaming tools for unattended pipelines. We introduce a four-dimensional Integration Friction Index that separates one-time engineering cost from recurring organisational, legal, and maintenance cost. We then derive quantitative regularities that explain recurring failure modes. Modelling an agentic security system as stochastic LLM policies wrapped by a deterministic mediator, we show that long-lived sessions lose resident evidence with phase count, while short-lived sub-agents extend the usable horizon according to the compression ratio between raw evidence and its summary. We show that a two-stage verdict cascade multiplies scorer likelihood ratios, but provides little benefit when scorer errors correlate. We show that treating unevaluable outcomes as attack failures biases downstream measurements toward evasive and severe responses. We formulate planner-versus-worker model routing as a knapsack problem and derive a closed-form execution cap for heavy-tailed tools, eta* = alpha v/c. Finally, we show why scope and budget enforcement cannot be delegated to system prompts: prompts do not constrain what actually executes. Inspectra, our implemented platform, serves as a worked instantiation, with mechanisms labelled shipped, partial, or planned, including those that did not work.

Figures & tables

Concern	What it does	Cost of skipping it
SAST	Reads source without executing it: injection sinks, weak crypto, hard-coded secrets, unsafe deserialisation.	Misses vulnerabilities visible only in code structure, and secrets sitting in Git history.
DAST	Exercises the running application: crawls, authenticates, fuzzes, observes.	Never confirms a static finding is reachable; misses auth bypass, IDOR, and runtime misconfiguration.
PT	Chains and escalates: turns low-severity findings into a demonstrated exploit with business impact.	Leaves a wall of severity scores with no narrative for a developer to act on.
AIRT	Attacks the model inside the product: jailbreaks, prompt exfiltration, indirect injection, tool abuse.	Leaves an injection surface invisible to a conventional web-application firewall.

TABLE I: The four concerns an agentic security system must cover.

Fig. 1: Reference architecture, drawn to expose the trust boundary rather than the data flow. (0) A planner-tier orchestrator plans and dispatches but never re-ingests raw evidence; a separate adversarial policy and judge drive AI red-teaming. (1)–(4) Triage, target intelligence, parallel reconnaissance (stacked cards denote fan-out), and a planner merge into composite findings. (5)–(7) Exploitation, AI red-teaming, and report synthesis. Every stage hands its result to the artefact store rather than a conversation turn (Section IV ); every invocation is mediated by the deterministic layer beneath, which owns scope, budgets, severity, and audit (Section VIII ).

Fig. 2: The context economy. (a) Retention fidelity

\Phi_{n}

of ( 2 ) over the design plane spanned by the phase count

n

and the compression ratio

\rho=s/\delta

, for

W=200

w_{0}=12

k, and

\delta=28

k. The heavy red curve is the coherence frontier

n=W_{\mathrm{eff}}/(\rho\delta)

: everything below and to its left retains all prior phase results, everything above and to its right has begun evicting them. Moving from a monolithic session (

\rho=1

) to

1.5

k summaries (star) shifts the frontier right by a factor of

\delta/s

. (b) Geometric decay of carry-forward detail across bin-packed batches. Both panels are model predictions under the stated parameters, not measurements.

Tool	Class	Licence / cost	Scores	$\mathrm{IFI}$	Verdict and governing reason
Gitleaks [ 19 ]	Secret scanning	MIT, free	1/0/0/1	0.50	Trivial to call; regex false positives need an allow-list from day one. Superseded here by content-level secret scoring.
Semgrep [ 39 ]	SAST	LGPL core, paid cloud	1/0/0/1	0.50	Deterministic JSON output; default rule set is noisy. Complementary to, not replaced by, LLM triage.
Playwright [ 26 ]	Browser automation	Apache 2.0, free	2/0/0/2	1.00	The default answer to authenticated crawling; the work is in making login agent-driven, not in installing it.
sqlmap [ 12 ]	SQLi exploitation	GPLv2, free	1/1/2/1	1.25	Deterministic exit codes; one of the few tools whose output an agent can trust without an interpretation pass. Needs hard caps.
Temporal [ 41 ]	Durable workflow	Apache 2.0 self-host; paid cloud	3/1/0/1	1.25	Not a security tool; supplies the durability that agent SDKs do not. Own database and mental model.
Promptfoo [ 34 ]	LLM eval / red-team	OSS core; free tier capped	1/3/1/2	1.75	Free tier capped at $10^{4}$ probes/month; heavy catalogue overlap with PyRIT; ownership change is a neutrality consideration.

TABLE II: Tool landscape with friction decomposition. Scores are

d_{\mathrm{eng}}/d_{\mathrm{org}}/d_{\mathrm{leg}}/d_{\mathrm{mnt}}

\{0,\dots,4\}

;

\mathrm{IFI}

uses uniform weights. Fit for an unattended pipeline: usable as-is, usable behind a wrapper or an explicit gate, not usable in the default pipeline. “Verdict” records the disposition in the case study of Section IX and the reasoning behind it.

Fig. 3: Integration friction across the evaluated tool set. (a) Composite

\mathrm{IFI}(\tau)

from ( 3 ), ranked, with the in-bar quadruple

d_{\mathrm{eng}}/d_{\mathrm{org}}/d_{\mathrm{leg}}/d_{\mathrm{mnt}}

; hatched bars need no adapter at all. (b) The same data resolved into technical friction (paid once by engineering) against non-technical friction (paid per customer by compliance and legal); marker area is proportional to

\mathrm{IFI}

, and the star is the lowest-friction pair. Most tools sit well off the equal-cost diagonal, which is why a single scalar rating misleads.

Panel of Table 6 — TABLE III: Client Spider versus browser automation for authenticated dynamic testing; the older AJAX Spider is omitted as ZAP itself no longer recommends it [ 31 ] . supported, partial, not supported.

Fig. 4: The verification calculus. (a) Iso-precision contours of ( 4 ) over prevalence

\pi

and likelihood ratio

\Lambda

; the three markers are the

\pi=0.05

operating points discussed in the text—a single heuristic scorer, a correlated cascade (

\rho=0.6

), and an independent cascade (star). (b) The same result as a population flow: of

1000

attempts the stage-1 scorer passes

330

, of which only

45

are genuine, and the judge removes

256

false positives. Unscoreable outcomes leave by a third exit rather than the rejected stream. Model predictions, not measurements.

Fig. 5: Budget economics under heavy-tailed tool runtimes (

x_{m}=2

min). (a) Survival function on log–log axes, where a Pareto tail is a straight line of slope

-\alpha

; the shaded area right of the cap is the mass a per-call ceiling removes, turning a divergent expected runtime into one growing as

\log\beta

. (b) Utility loss relative to the best attainable cap, over

\beta

and

\alpha

, at

v/c=60

; the heavy curve is the optimum

\beta^{\star}=\alpha v/c

of Equation ( 9 ), and the star marks the configured

30

-minute SQL-injection cap. Over-capping is far cheaper than under-capping, so a generous cap is the safer error when

\alpha

is uncertain. Model predictions, not measurements.

Role / tool	$\beta_{c}$	$\tau_{\max}$	Tier
Orchestrator	180 min (global)	50	planner
Pre-recon (target intel.)	30 min	60	planner
Recon, per class ( $\times 6$ )	30 min	60	planner
Exploit, per class ( $\times 5$ )	30 min	15	worker
SQL-injection tool	30 min	—	—
AI red-team, per attack call	10 min	—	—

TABLE IV: Inspectra configuration. Budgets are enforced in the mediation layer, outside policy control.

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 327 papers

Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection
Sep 26, 2026Animesh ShawAI Agent Security Benchmarks Prompt Injection Attacks on AI Agents
Independent Researcher
Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
Jun 9, 2026Yuchen Ling, Shengcheng Yu, Zhenyu Chen +1Prompt Injection LLM Agent Security
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Technical University of Munich, Heilbronn, Germany
Agent Audit: A Security Analysis System for LLM Agent Applications
Mar 24, 2026Haiyue Zhang, Yi Nian, Yue ZhaoAI Agent Auditing LLM Agent Security
University of Southern California Los Angeles, California, USA

Sep 26, 2026J/K move · Enter open · S save

Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection

Animesh Shaw

Independent Researcher

AI Agent Security Benchmarks Prompt Injection Attacks on AI Agents LLM Agent Harnesses AI Agent Auditing

Open paper AskPDF arXiv

LLM agents that invoke privileged tools are vulnerable to indirect prompt injection (IPI), in which adversarial instructions embedded in retrieved data hijack the agent's actions. A growing body of work evaluates defenses against IPI, but the validity of that evaluation is rarely examined. We audit an IPI benchmark and its harness and identify four defect classes -- silent payload non-delivery, attack success scored by tool identity rather than arguments, false-rejection rate conflated with model incapacity, and the absence of an audit trail -- each of which yields a plausible, publishable, and incorrect number. We quantify the distortion by re-scoring identical execution traces under the defective and corrected definitions: on real agent behaviour, the tool-identity scorer reports a 21.7% attack-success rate where the true argument-level rate is 1.2%. In the sharpest case, an open model previously reported at 62.8% registers 0% under the corrected harness -- the prior figure largely an artifact of undelivered payloads and identity-level scoring. We release a harness whose construction makes each defect unrepresentable -- machine-checkable payload placement, argument-level attacker predicates, per-scenario environments, and mandatory trace persistence -- and use it to report three quantities the field does not: whether a compromised agent discloses the attack, the full security/utility operating curve of an LLM-judge defense, and tool-calling capability disentangled from defensive over-blocking. A corrected harness further overturns a reported "capability barrier": a model deemed incapable of tool use is in fact fully capable, its earlier result an artifact of environment mismatch. We argue that evaluation validity is a prerequisite for, not a footnote to, defense claims in agentic security, and provide an instrument that enforces it.

Abstract

LLM Agent Security

Explore similar work

Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection

Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation

Agent Audit: A Security Analysis System for LLM Agent Applications