PROVISO: Provenance-Informed Action Selection for Secure LLM Agents

Bo Yan; Qi Zhang; Ziwen Kan; Fnu Suya; Song Wang

cs.AIOct 19, 2026

PROVISO: Provenance-Informed Action Selection for Secure LLM Agents

Authors: Bo Yan, Qi Zhang, Ziwen Kan, Fnu Suya, Song Wang

Organizations: University of Central Florida · Independent Researcher · University of Tennessee, Knoxville

Abstract

LLM agents act on the external world through tools and third-party skills, so a single executed action can cause irreversible harm. Existing defenses judge a proposed action without tracking whether its content comes from the user or from external sources such as tool outputs, and they judge it in isolation rather than against an available alternative, which forces a trade-off between security and utility. We present PROVISO, a controller that decides which action to commit after the agent proposes an action and before it is executed. PROVISO encodes trusted observations, external observations, the proposed action, and an alternative into provenance-typed memory, relates them through role-specific queries, and commits the action with the highest calibrated value learned from outcomes. On four agent security tasks spanning the agent lifecycle, no baseline exceeds PROVISO in both security and utility. PROVISO improves F1 over the strongest baseline by 13.84 points on SkillsBench and by 7.07 points on ToolSafe, and it reaches 88-99 F1 on three additional skill detection benchmarks.

Figures & tables

Figure 1: A payment task with an injected instruction. Existing defenses either release

P_{0}

or block the whole task. PROVISO accounts for where each argument of

P_{0}

comes from and compares

P_{0}

with

P_{1}

: the recipient of

P_{0}

comes only from

G

and is not authorized by

K

, so it commits

P_{1}

Figure 2: Overview of PROVISO. Left: at step

t

, the agent’s observations are split by provenance into trusted (

K

) and external (

G

) ones; the LLM policy proposes

P_{0}

, and an alternative

P_{1}

is available. Middle: PROVISO builds role-typed memory from

x=(K,G,P_{0},P_{1})

, lets four role queries and a commitment state (C) retrieve, compare, and update over

L

blocks, then values each candidate and commits

a^{*}

. Right: the executor runs

a^{*}

, and the next observation joins

G

. Dashed arrows show training: native outcomes fit the parameters, and the offsets

\tau

are calibrated on development data.

Figure 3: Security and utility of PROVISO and baselines on four agent security tasks. The upper right is better. Panel (b) includes all six guards on the same 80 benign tasks; in (d), PIGuard lies outside the plotted range.

Method	Prec $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	FPR $\downarrow$
PROVISO	89.21	82.67	85.81	1.00
SkillGate	83.33	63.33	71.97	1.27
Cisco semantic	56.49	98.67	71.84	7.60
Sentry semantic	54.74	100.00	70.75	8.27
SkillSpector semantic	41.69	98.67	58.61	13.80
Sentry static	48.44	72.67	58.13	7.73

Table 1: Malicious skill detection results on SkillsBench . Best results are in bold.

Method	H&R $\downarrow$	Benign completion $\uparrow$
PROVISO	0.00	100.00
AGrail	2.65	43.75
MCIP	14.27	78.75
GuardAgent	19.66	72.50
WildGuard	20.42	82.50
AgentSpec	20.64	80.00

Table 2: Agent misuse results on ANCHOR. H&R denotes the Harm & Risk Score. Best results are in bold. Harm & Risk uses 300 harmful tasks; benign completion uses the same 80-task panel for every method.

Method	ACC $\uparrow$	F1 $\uparrow$	Recall $\uparrow$
PROVISO	99.15	99.12	99.71
TS-Guard	92.50	92.05	90.29
MCIP	60.11	64.84	76.47
AgentSpec	71.00	62.79	50.88
ShieldAgent	58.42	58.47	60.88
WildGuard	67.33	56.00	43.24

Table 3: Unsafe tool invocation results on ToolSafe .

	Benign	Utility	Targeted
Method	utility $\uparrow$	w. attack $\uparrow$	ASR $\downarrow$
PROVISO	87.63	85.41	0.28
SecAlign	88.66	83.35	3.46
BIPIA	84.54	81.28	0.20
DRIFT	84.54	81.04	0.32
IPIGuard	81.44	78.82	2.58

Table 4: Indirect prompt injection on AgentDojo .

Benchmark	Prec $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	FPR $\downarrow$
SkillsBench	89.21	82.67	85.81	1.00
MalSkillBench	99.26	98.00	98.62	0.73
SkillVetBench	95.79	91.00	93.33	4.00
MaliciousSkillBench	92.99	83.79	88.15	9.72

Table 5: Results of PROVISO on four malicious skill detection benchmarks.

Figure 4: Effect of changing one component of PROVISO. Bars show the score with the component changed, the dashed line marks the full model, and the red number is the drop. The first two groups report F1 on SkillsBench ; the third reports utility on the payment task. Paired 95% confidence intervals are in Table 15 .

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Setting	Trusted obs. $K$	External obs. $G$	Proposed $P_{0}$	Alternative $P_{1}$	Native outcome
Skill benchmarks †	Adm.	Declared skill purpose	Package instructions and code	Install the package	Reject it	Malicious label
ANCHOR	Adm.	Task request	Task setting	Execute the requested effects	Refuse and return a safety assessment	H&R; benign completion
ToolSafe	Adm.	Task, tool registry, and schemas	Interaction history and tool returns	Proposed tool call	Block the call and return feedback	Call severity label
AgentDojo	Sel.	User request	None before commitment (fixed stage description)	Complete trajectory of the deployed agent	Complete recovery trajectory	Utility and attack success

Appendix

Table 6: Observation and action roles in each benchmark. Setting is candidate selection (Sel.) or admission (Adm.).

Benchmark	Pos.	Neg.	Split
SkillsBench	150	1,500	5-fold
MalSkillBench	3,944	4,000	5-fold
SkillVetBench	100	100	5-fold
MaliciousSkillB.	839	545	Released split
ANCHOR (test)	300	80	936/176/380
ToolSafe (test)	340	367	5,769/706/707

Appendix

Table 7: Data splits. Positive denotes malicious, harmful, unsafe, or attacked instances. “5-fold” is five-fold cross-validation grouped by source, trajectory, or user task; Split otherwise gives train/dev/test sizes.

Malicious skill detection
Method	Mechanism	Trained
SkillGate	Regex prefilter and LLM judge
Static scanners (3)	Static (and behavioral) rules
Semantic scanners (3)	LLM-based semantic analysis
SkillFortify	Static analysis
Agent misuse and unsafe tool invocation

Appendix

Table 8: Decision mechanism of each baseline. Trained indicates that the decision is made by a model fine-tuned for safety or injection defense.

Benchmark	Batch	Epochs	Decay	Threshold
SkillsBench	64	$\leq$ 30	.02	Dev. F1
MalSkillBench	16	$\leq$ 12	.02	Dev. F1
SkillVetBench	16	$\leq$ 30	.01	Fixed .7
MaliciousSkillB.	32	$\leq$ 8	.01	Dev. macro-F1
ANCHOR	64	$\leq$ 20	.01	Dev. recall
ToolSafe	128	36	.01	Dev. benign pass

Appendix

Table 9: Optimization settings. All instances use AdamW with learning rate

3\times 10^{-4}

and gradient clipping at one; thresholds are selected on development data unless fixed.

Backbone	TP/FN/FP/TN	F1	Recall	Benign FPR	Requests
Granite-3.3-8B	293/7/0/80	98.82	97.67	0.00	687

Appendix

Table 10: Cross-backbone native admission with the shared core on ANCHOR. The Granite-3.3-8B row covers all 380 scored instances and retains the complete native request count.

Shared-core readout	Prior	Tasks /417	Scen. /139	Calls
Action contrast	417/417	237	49	20,355
Conservative corrective	417/417	237	49	20,355
Finer evidence	417/417	237	49	20,355
Recoverable relation	417/417	237	49	20,355

Appendix

Table 11: Native execution stability of the shared core on AppWorld. Every row uses the complete 417-task scorer and the same 20,355 selected API calls.

Method	TP/FN/FP/TN
PROVISO	339/1/5/362
TS-Guard	307/33/20/347
MCIP	260/80/202/165
AgentSpec	173/167/38/329
ShieldAgent	207/133/161/206
WildGuard	147/193/38/329

Appendix

Table 12: Confusion counts on ToolSafe test calls.

Method	TP/FN/FP/TN
SkillsBench
PROVISO	124/26/15/1,485
SkillGate	95/55/19/1,481
Cisco semantic	148/2/114/1,386
Sentry semantic	150/0/124/1,376
SkillSpector semantic	148/2/207/1,293

Appendix

Table 13: Confusion counts on SkillsBench and the additional skill benchmarks.

Baseline	Gain	95% CI	Holm $p$
ToolSafe
TS-Guard	+7.07	[4.29, 10.16]	$<$ .001
MCIP	+34.28	[30.49, 38.15]	$<$ .001
AgentSpec	+36.33	[31.09, 41.75]	$<$ .001
ShieldAgent	+40.65	[35.64, 46.11]	$<$ .001
WildGuard	+43.12	[36.18, 50.54]	$<$ .001

Appendix

Table 14: F1 gain of PROVISO over each baseline (percentage points) with paired 95% confidence intervals.

Evidence selection (package F1; full 85.81)
Intervention	Control	Gain	95% CI
No selector supervision	69.08	+16.73	[10.22,24.42]
No local memory, no selector loss	59.38	+26.44	[18.43,36.60]
Permuted retention, fixed scorer	64.35	+21.47	[13.09,29.47]
Query reasoning (package F1; full 85.81)
Matched dense readout	48.62	+37.19	[29.04,47.54]

Appendix

Table 15: Gains behind Figure 4 . Each row changes one component of PROVISO. Gain is the score of the full model minus that of the control, with paired 95% confidence intervals.

Control	F1	Gain	95% CI
PROVISO	85.81	–	–
No selector supervision	69.08	+16.73	[10.22, 24.42]
No local memory or selector loss	59.38	+26.44	[18.43, 36.60]
All-window memory	81.19	+4.63	[0.97, 8.98]
Dense expert update	86.49	$-$ 0.67	[ $-$ 2.98, 1.66]
No role embeddings	85.32	+0.49	[ $-$ 2.91, 4.30]

Appendix

Table 16: Component controls on SkillsBench . Gain is the F1 of PROVISO minus that of the control, with paired 95% confidence intervals.

Content policy	Gain (pp)	Paired 95% CI	Holm $p$
Selected: retained	+21.47	[13.09,29.47]	.00032
Selected: restore four	+21.56	[13.22,29.48]	.00032
Five-fit: retained	+17.41	[9.90,25.43]	.00032
Five-fit: restore four	+17.26	[9.83,25.18]	.00032

Appendix

Table 17: Content retention at fixed weights and memory budget. Each row uses all 1,650 packages in 1,500 source groups. Gains compare retained or restored content with the same package-ID-permuted policy; five-fit rows average five pooled F1 contrasts.

Effect	Gain	Joint 95% CI	Holm $p$
Full-retrieval utility	+9.94	[8.43,11.41]	.00006
Conflict–aligned interaction	+13.15	[10.96,15.31]	.00006

Appendix

Table 18: Repeated retrieval on complete source-disjoint confirmation. Five paired continuation seeds cover the same 4,096 cases. Effects are percentage points; intervals jointly resample seeds and sources.

Selection	JSU /2,516	$P_{1}$ choices
Always acquired $P_{0}$	2,066	0
Always acquired $P_{1}$	2,034	2,613
PROVISO	2,147	1,270

Appendix

Table 19: Fixed selection from the acquired AgentDojo candidate pool. JSU uses all 2,516 attacked episodes;

P_{1}

choices use all 2,613 paired episodes. Each fixed row selects its acquired trajectory.

Figure 5: ToolSafe results across (a) twenty training runs, (b) proposer backbones, and (c) logit-threshold offsets for the selected model; offset zero is the selected threshold.

Figure 6: Effect of shifting the decision threshold on the logit scale for ToolSafe , ANCHOR, and SkillsBench . The vertical line marks the selected threshold.

Proposer	ToolSafe	AgentDojo calls
Granite-3.3-8B	98.98	97.73
gpt-oss-20B	99.12	96.55
Phi-4	98.98	97.73
Mistral-7B	98.53	97.73
Muse-Glimmer-30B	99.27	97.73

Appendix

Table 20: ToolSafe F1 with evidence from different proposer backbones. The AgentDojo-call subset (128 calls) uses the same task format; its Muse controller is initialized from Granite and independently fitted on Muse evidence.

Backbone	F1	Mal. recall	Benign FPR
Granite-3.3-8B	81.43	83.33	2.13
Mistral-7B	81.70	83.33	2.07
Muse-Glimmer-30B	81.41	84.67	2.33

Appendix

Table 21: Cross-backbone replay of the shared core on the complete SkillsBench population. Each row uses five source-grouped outer folds, 17.40M trainable parameters, and independent backbone-local weights.

Proposer	F1	Recall	Retention	Brier
Granite-3.3-8B	99.12	99.71	98.64	.00966
Phi-4	99.12	99.71	98.64	.00961

Appendix

Table 22: Separate backbone fits with the shared core. Each row evaluates 340 unsafe and 367 benign calls and has TP/FN/FP/TN

339/1/5/362

. Recall and retention are percentages.

Method	F1	Recall	Benign FPR	TP/FN/FP/TN
SkillGate	71.97	63.33	1.27	95/55/19/1481
PROVISO, independent fine-view fit	84.39	84.67	1.60	127/23/24/1476

Appendix

Table 23: Complete package-admission support under independent fitting. Each row covers all 1,650 packages. F1, recall, and benign FPR are percentages. The F1 gain uses paired source-group resampling with the original five folds; Holm adjustment retains all twelve completed candidate-to-SkillGate comparisons.

Execution	Mean risk $\downarrow$	Benign completion	Request starts
Unguarded Granite-3.3-8B	30.777	48/80 (60.00%)	1,558
PROVISO + Granite-3.3-8B	0.717	48/80 (60.00%)	807

Appendix

Table 24: Complete Granite native support on ANCHOR. Risk is the mean Harm & Risk score on all 300 harmful instances; benign completion covers all 80 benign instances. Each arm contains 380 scored outcomes.

Method	F1	Recall	Benign FPR	TP/FN/FP/TN
SkillGate	71.97	63.33	1.27	95/55/19/1481
PROVISO	84.95	84.67	1.47	127/23/22/1478

Appendix

Table 25: Complete full-package fine-view support. All 1,650 IDs are scored. F1, unsafe recall, and benign FPR are percentages; SkillGate uses its released workflow with the recorded Gemma 4 adapter.

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 332 papers

Agent-Sentry: Bounding LLM Agents via Execution Provenance
Mar 24, 2026Rohan Sequeira, Stavros Damianakis, Umar Iqbal +1Computer-Use Agent Safety AI Agent Security
University of Southern California, Los Angeles, California, USA · Washington University in St. Louis, St. Louis, Missouri, USA
Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback
May 17, 2026Lecheng Yan, Ruizhe Li, Xicheng Han +5AI Agent Reliability LLM Agent Evaluation
Southern University of Science and Technology · University of Science and Technology of China · Zhejiang University +2
The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneck
May 11, 2026Linfeng Fan, Ziwei Li, Yuan Tian +3LLM Agent Security Information Flow Control
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · King Abdullah University of Science and Technology, Thuwal, Saudi Arabia · Dongbei University of Finance and Economics, Dalian, China +1

Mar 24, 2026J/K move · Enter open · S save

Agent-Sentry: Bounding LLM Agents via Execution Provenance

Rohan Sequeira, Stavros Damianakis, Umar Iqbal, Konstantinos Psounis

University of Southern California, Los Angeles, California, USA · Washington University in St. Louis, St. Louis, Missouri, USA

Computer-Use Agent Safety AI Agent Security AI Agent Monitoring Runtime Enforcement for AI Agents Prompt Injection Attacks on AI Agents

Open paper AskPDF arXiv

Agentic computing systems, while immensely capable, raise serious security, privacy, and safety concerns. A key issue is that the full set of functionalities offered by these systems, combined with their probabilistic execution flows, is not known beforehand. Given this lack of characterization, it is challenging to validate whether a system has successfully carried out the user's intended task or instead executed irrelevant actions, potentially as a consequence of compromise. We present Agent Sentry, a runtime defense that learns a bound on an agent's benign execution from prior legitimate executions and flags any action that falls outside this bound. Agent Sentry layers three complementary checks: a structural classifier over the sequence of actions and the provenance of each function's arguments; a deterministic allowlist check over sensitive argument values; and an LLM judge, invoked only on the residual of actions where the first two checks cannot safely decide between a legitimate new request and a carefully crafted injection. We demonstrate the effectiveness of Agent Sentry in AgentDojo and AgentDyn by blocking 94.3% of successful injections while allowing 95.1% of benign executions, without modifying the agent, its tools, or the LLM.

Abstract

LLM Agent Security

Explore similar work

Agent-Sentry: Bounding LLM Agents via Execution Provenance

Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback

The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneck