Agent-Sentry: Bounding LLM Agents via Execution Provenance

Rohan Sequeira; Stavros Damianakis; Umar Iqbal; Konstantinos Psounis

cs.CRMar 24, 2026

Agent-Sentry: Bounding LLM Agents via Execution Provenance

Authors: Rohan Sequeira, Stavros Damianakis, Umar Iqbal, Konstantinos Psounis

Organizations: University of Southern California, Los Angeles, California, USA · Washington University in St. Louis, St. Louis, Missouri, USA

Abstract

Agentic computing systems, while immensely capable, raise serious security, privacy, and safety concerns. A key issue is that the full set of functionalities offered by these systems, combined with their probabilistic execution flows, is not known beforehand. Given this lack of characterization, it is challenging to validate whether a system has successfully carried out the user's intended task or instead executed irrelevant actions, potentially as a consequence of compromise. We present Agent Sentry, a runtime defense that learns a bound on an agent's benign execution from prior legitimate executions and flags any action that falls outside this bound. Agent Sentry layers three complementary checks: a structural classifier over the sequence of actions and the provenance of each function's arguments; a deterministic allowlist check over sensitive argument values; and an LLM judge, invoked only on the residual of actions where the first two checks cannot safely decide between a legitimate new request and a carefully crafted injection. We demonstrate the effectiveness of Agent Sentry in AgentDojo and AgentDyn by blocking 94.3% of successful injections while allowing 95.1% of benign executions, without modifying the agent, its tools, or the LLM.

Figures & tables

Figure 1. Agent-Sentry in action: The user submits a request to the agent, which issues each tool call together with its provenance metadata (the tool’s argument sources) and execution history so far, to Agent-Sentry. Layer 1’s structural classifier (XGBoost or rule-based) labels the call as Allow, Block, or Ambiguous using features of the provenance graph; only ambiguous calls move to next layer. Layer 2, a provenance-aware allowlist, allows known argument values that come from trusted sources, while new or untrusted-sourced values move to the next layer. Layer 3’s LLM judge resolves the residual cases by comparing the proposed action against the user’s prompt over a bounded excerpt of any relevant, potentially untrusted, content. Allowed calls run and the agent loops back for its next step until the task completes. A legitimate request completes successfully, while an injection-induced action whose intent deviates from the user’s request is blocked.

Agent	Benign	Attack	Total
Banking	681	1,038	1,719
Slack	572	348	920
Travel	609	167	776
Workspace	505	95	600
GitHub	376	165	541
DailyLife	237	93	330

Table 1. Agent-Sentry Bench trace counts per agent. A trace is benign if the agent satisfied the user’s task without intervention; a malicious trace is a run where the attacker’s objective was realized in the execution state.

	Layer 1: XGBoost							Layer 1: Rule-Based
Agent	Util.	ABR	FPR	FNR	L1%	L2%	L3%	Util.	ABR	FPR	FNR	L1%	L2%	L3%
Banking	97.9	98.7	2.1	1.3	98.8	0.2	1.0	96.0	100.0	4.0	0.0	82.2	0.7	17.2
Slack	97.0	99.7	3.0	0.3	96.5	1.2	2.3	95.2	97.5	4.8	2.5	44.5	17.6	37.8
Travel	97.5	99.4	2.5	0.6	98.5	0.0	1.5	95.4	100.0	4.6	0.0	59.5	9.1	31.4
Workspace	99.4	95.8	0.6	4.2	96.7	0.7	2.6	91.0	89.5	9.0	10.5	58.4	14.2	27.3
GitHub	91.8	95.8	8.2	4.2	95.7	0.1	4.2	85.0	84.5	15.0	15.5	38.1	8.7	53.2

Table 2. Full pipeline performance on Agent-Sentry Bench under two Layer 1 versions, with per-layer routing breakdown.

	Layer 1: XGBoost			Layer 1: Rule-Based
Agent	Util.	ABR	Ambig. %	Util.	ABR	Ambig. %
Banking	96.9	98.2	1.0	94.5	100.0	17.2
Slack	95.4	100.0	2.3	91.9	97.6	37.8
Travel	96.2	100.0	1.5	94.4	100.0	31.4
Workspace	97.6	95.2	2.6	88.9	82.8	27.3
GitHub	90.9	94.3	4.2	78.5	93.6	53.2

Table 3. Layer 1 + Layer 2 only on Agent-Sentry Bench, under two Layer 1 variants.

Configuration	L3 invocation rate (%)	Latency/trace (s)
Agent-Sentry	3.1	0.073
Layer 1 + Layer 2	0.0	$<$ 0.01
Layer 3 alone	100.0	2.40

Table 4. Per-trace latency on an average agent execution

	Layer 1: XGBoost		Layer 1: Rule-Based
Agent	Util.	ABR	Util.	ABR
Banking	77.8	91.9	77.8	93.5
Slack	63.2	90.0	79.0	90.0
Travel	83.3	85.7	100.0	88.9
Workspace	73.7	98.5	89.5	98.5
GitHub	53.9	84.1	61.5	95.2

Table 5. Generalization to Native-Trace Bench under two Layer 1 versions. No Native-Trace Bench traces were seen during training.

	Layer 1: XGBoost		Layer 1: Rule-Based
Agent	Util.	ABR	Util.	ABR
Banking	66.7	90.3	100.0	91.9
Slack	84.2	81.4	84.2	87.1
Travel	83.3	90.5	100.0	88.9
Workspace	78.9	100.0	89.5	98.5
GitHub	53.8	90.5	76.9	90.5

Table 6. Performance of Agent-Sentry on Native-Trace Bench under two Layer 1 versions, using 5-fold cross-validation.

	Bench		/ AgentDyn		Time
	Agent-Sentry		AgentDojo
Method	Util.	ABR	Util.	ABR	(s)
Agent-Sentry, XGBoost (Partial)	96.4	96.8	69.1	91.2	4.30
Agent-Sentry, Rules (Partial)	89.3	95.3	87.1	93.2	5.46
Agent-Sentry, Rules (Full)	99.8	94.4	95.1	94.3	5.46
Progent	69.2	97.7	93.7	88.4	4.26

Table 7. Comparison on Agent-Sentry Bench and AgentDojo/AgentDyn benchmarks. Dashes indicate the method was not evaluated on that benchmark.

Delivery mechanism	Attacks	ABR
ImportantInstructions	532	91.5
DirectAttack	78	92.3
InjecAgentAttack	65	90.8

Table 8. Delivery mechanism invariance test on Native-Trace Bench

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Control flow
Feature	Description
$f_{1}$ ordinal position	Position among action tools in the execution trace
$f_{2}$ preceding retrievals	Retrieval calls before this tool call
$f_{3}$ preceding actions	Action calls before this tool call
$f_{4}$ retrieval gap	Retrieval calls made since the last preceding action tool call
$f_{5}$ tool diversity	Counts the distinct tool calls made in the preceding chain

Appendix

Table 9. The 17-dimensional feature vector, grouped into four families. Information-flow features are over the sensitive set

\mathcal{A}_{SA}

(Appendix A.2 ).

Argument	Description	$\mathcal{A}_{GA}$
recipient	money or message recipient	✓
recipients	list of recipients	✓
password	account credential	✓
otp	one time password
ssh_key	SSH public key	✓
collaborator_email	repository collaborator address	✓

Appendix

Table 10. Sensitive argument set

\mathcal{A}_{GA}

and its groundable subset

\mathcal{A}_{GA}

. The five names excluded from

\mathcal{A}_{GA}

are either per-session-dynamic ( amount , otp ) or free-text payloads whose values are always unique ( body , content , url ); neither can be allowlisted from benign history.

Anomaly detectors (8)
Detector	What it checks
CFG Retrieval Action	The set of retrievals immediately before this action was never observed in benign traces.
CFG Action Action	The action that immediately precedes this action was never observed in benign traces.
Source Integrity	A sensitive argument’s reported source is untrusted and but does not contain the argument’s value verbatim (cannot verify).
DFG Argument Source	A sensitive argument is reported as coming from a tool that has never supplied this argument in observed benign traces.
Multi Hop Chain	An action’s argument value reaches the action through more consecutive untrusted retrievals than any benign chain produced observed.

Appendix

Table 11. The 11 structural detectors manually defined for the Rule-Based Classifier

Structural signal	Feature side	Detector side
Preceding retrieval pattern	$f_{2},f_{4},f_{5}$	CFG Retrieval Action
Action-to-action transition novelty	$f_{3}$	CFG Action Action
Missing precondition retrieval	implicit in $f_{2},f_{5}$	Retrieval Necessity (detector only)
Untrusted source on arguments	$f_{6},f_{10},f_{12}$	Source Integrity, DFG Argument Source
Prompt anchoring of argument values	$f_{7},f_{13},f_{14}$	User Prompt Anchor, Trusted Provenance
Untrusted hop depth / chain length	$f_{8}$	Multi Hop Chain (uses the benign maximum as its Block threshold)

Appendix

Table 12. Correspondence between the 17 features and the 11 detectors, grouped by shared structural signal. Rows marked “feature only” or “detector only” indicate signals present in only one representation.

Configuration	Layer 3	Latency
	invocation rate	/trace (s)
Layer 1 + Layer 2 (XGBoost)	0.0%	$<$ 0.01
Layer 1 + Layer 2 (Rule-based)	0.0%	$<$ 0.01
Agent-Sentry (XGBoost)	3.0%	0.07
Agent-Sentry (Rule-based)	31.0%	0.74
Layer 3 alone	100.0%	2.40

Appendix

Table 13. Per-trace latency across all five configurations on an average agent execution (2 action tool calls).

Attack Block Rate
Agent	50/50	67/33	75/25	80/20
Banking	93.5	91.9	88.7	90.3
Slack	80.0	82.9	82.9	81.4
Travel	88.9	93.7	90.5	90.5
Workspace	97.1	98.5	100.0	100.0
GitHub	88.9	87.3	90.5	90.5

Appendix

Table 14. Performance of Agent-Sentry with Layer 1 (XGBoost) on Native-Trace Bench,

K

-fold cross validation with

K\in\{2,3,4,5\}

Attack Block Rate
Agent	50/50	67/33	75/25	80/20
Banking	91.9	91.9	91.9	91.9
Slack	90.0	85.7	87.1	87.1
Travel	88.9	88.9	88.9	88.9
Workspace	98.5	98.5	98.5	98.5
GitHub	95.2	90.5	92.1	90.5

Appendix

Table 15. Performance of Agent-Sentry with Layer 1 (Rule-based) on Native-Trace Bench,

K

-fold cross validation with

K\in\{2,3,4,5\}

Agent	Util.	ABR
Banking	50.0	95.2
Slack	65.0	100.0
Travel	5.9	100.0
Workspace	52.8	100.0
GitHub	0.0	100.0
DailyLife	0.0	99.3

Appendix

Table 16. CaMeL per-agent performance on Native-Trace Bench.

Agent	Util.	ABR
Banking	83.3	100.0
Slack	95.0	100.0
Travel	94.1	60.3
Workspace	94.4	100.0
GitHub	86.7	88.9
DailyLife	100.0	87.3

Appendix

Table 17. Progent per-agent performance on Native-Trace Bench.

Agent	Util.	ABR
Banking	89.8	100.0
Slack	81.0	100.0
Travel	9.0	97.0
Workspace	38.9	96.8
GitHub	84.9	91.5
DailyLife	96.7	100.0

Appendix

Table 18. Progent per-agent performance on Agent-Sentry Bench (policies refit on Agent-Sentry Bench benign traces).

Agent	Util.	ABR
Banking	100.0	91.9
Slack	94.7	90.0
Travel	100.0	96.8
Workspace	100.0	100.0
GitHub	92.3	92.1
DailyLife	86.7	100.0

Appendix

Table 19. Agent Sentry full coverage, Layer 1 as rule based on Native-Trace Bench.

Agent	Util.	ABR	FPR	FNR	F1
Banking	99.6	98.7	0.4	1.3	99.1
Slack	97.2	98.1	2.8	1.9	96.0
Travel	99.8	100.0	0.2	0.0	99.7
Workspace	99.5	90.5	0.5	9.5	94.0
GitHub	92.0	96.4	8.0	3.6	89.8
DailyLife	97.9	92.5	2.1	7.5	93.5

Appendix

Table 20. Performance of Layer 3 as a standalone system on Agent-Sentry Bench

	Qwen3.5-35B		Qwen2.5-7B
Agent	Util.	ABR	Util.	ABR
Banking	88.4	61.2	74.9	87.2
Slack	91.1	97.7	66.6	87.4
Travel	91.1	100.0	85.6	88.0
Workspace	88.5	96.8	72.9	53.7
GitHub	52.4	77.6	55.6	70.9

Appendix

Table 21. Per-agent utility and ABR on Agent-Sentry Bench when Layer 3 is run as a standalone defense, under the same untuned system prompt on two Qwen models.

Explore similar work

Kairos: Transactional Security Enforcement at the System Boundary for LLM Agents
Oct 13, 2026Kristoffer Hempel, Nils Loose, Lorenzo Cavallaro +1LLM Agent Security Runtime Enforcement for AI Agents
Universität zu Lübeck · University College London · Mohamed bin Zayed University of Artificial Intelligence
AgentSpy: Making AI Agent Behavior Observable
Oct 5, 2026Christoph Bühler, Matteo Biagiola, Luca Di Grazia +1AI Agent Reliability AI Agent Auditing
University of St. Gallen, St. Gallen, SG, Switzerland · University of St. Gallen and Università della Svizzera italiana (USI), St. Gallen and Lugano, SG and TI, Switzerland
SafeAgent: A Runtime Protection Architecture for Agentic Systems
Apr 19, 2026Hailin Liu, Eugene Ilyushin, Jie Ni +1Adversarial Attacks on LLMs AI Agent Security
Lomonosov Moscow State University · Central University

Oct 13, 2026J/K move · Enter open · S save

Kairos: Transactional Security Enforcement at the System Boundary for LLM Agents

Kristoffer Hempel, Nils Loose, Lorenzo Cavallaro, Thomas Eisenbarth

Universität zu Lübeck · University College London · Mohamed bin Zayed University of Artificial Intelligence

LLM Agent Security Runtime Enforcement for AI Agents Tool Access Control for LLM Agents Prompt Injection Attacks on AI Agents

Open paper AskPDF arXiv

Large language model (LLM) agents are becoming an integral part of the software development lifecycle, and the capabilities granted to them allow an agent to act with the user's authority on any resource it can access. The reach that gives an agent its utility is equally available to an attacker who can steer it. Anything an agent ingests can carry an injected instruction, and since the model does not reliably distinguish instructions from data, indirect prompt injection should be mitigated by mediating the effects the agent produces. Such mediation faces a structural difficulty. Legitimate work and the first step of an attack often look alike when an individual action is observed, and what separates them lies later in the trace, so a monitor limited to allow and deny is forced to decide before the evidence accumulates. We present Kairos, a reference monitor at the isolation boundary of an unmodified, opaque agent, where a tool call, a shell command, and a script the agent writes all cross the boundary as the same file and network operations, so that confinement holds regardless of the agent's behavior. A declarative policy that states which sources are trusted decides most effects deterministically and defers the rest, which are held while the agent continues speculatively. A decision oracle with bounded authority, such as a language model or a human, releases or refuses each deferred effect on the trace. Across three benchmarks and three agent models, Kairos outperforms existing defenses in the trade-off between attack success and utility. In particular, on AgentDojo, it reduces attack success from 47.0% to 2.7% while its benign utility stays at the level of the undefended agent.

Abstract

Explore similar work

Kairos: Transactional Security Enforcement at the System Boundary for LLM Agents

AgentSpy: Making AI Agent Behavior Observable

SafeAgent: A Runtime Protection Architecture for Agentic Systems