STARS: Skill-Triggered Audit for Request-Conditioned Invocation Safety in Agent Systems

Guijia Zhang; Shu Yang; Xilin Gong; Di Wang

cs.AIApr 11, 2026

STARS: Skill-Triggered Audit for Request-Conditioned Invocation Safety in Agent Systems

Authors: Guijia Zhang, Shu Yang, Xilin Gong, Di Wang

Organizations: Shenzhen University · King Abdullah University of Science & Technology · PRADA Lab · University of Georgia

Abstract

Autonomous language-model agents increasingly rely on installable skills and tools to complete user tasks. Static skill auditing can expose capability surface before deployment, but it cannot determine whether a particular invocation is unsafe under the current user request and runtime context. We therefore study skill invocation auditing as a continuous-risk estimation problem: given a user request, candidate skill, and runtime context, predict a score that supports ranking and triage before a hard intervention is applied. We introduce STARS, which combines a static capability prior, a request-conditioned invocation risk model, and a calibrated risk-fusion policy. To evaluate this setting, we construct SIA-Bench, a benchmark of 3,000 invocation records with group-safe splits, lineage metadata, runtime context, canonical action labels, and derived continuous-risk targets. On a held-out split of indirect prompt injection attacks, calibrated fusion reaches 0.439 high-risk AUPRC, improving over 0.405 for the contextual scorer and 0.380 for the strongest static baseline, while the contextual scorer remains better calibrated with 0.289 expected calibration error. On the locked in-distribution test split, gains are smaller and static priors remain useful. The resulting claim is therefore narrower: request-conditioned auditing is most valuable as an invocation-time risk-scoring and triage layer rather than as a replacement for static screening. Code is available at https://github.com/123zgj123/STARS.

Figures & tables

Figure 1: Overview of the STARS audit stack. The primary path combines a static capability prior and a request-conditioned invocation scorer into a calibrated continuous-risk output. Thresholded allow / escalate / block actions are treated as a deployment compatibility layer placed on top of that score, and post-hoc remediation remains a secondary downstream branch rather than a co-equal primary objective.

Figure 2: Construction pipeline for SIA-Bench . The benchmark is built as an invocation-level resource rather than a prompt collection: seed skill calls are expanded, normalized into a common runtime-aware schema, paired with canonical action labels and continuous-risk targets, and finally assigned to leakage-free splits at the source-group level.

Split	$N$	Skills	Groups	Allow	Esc.	Block	D0	D1	Ben.	Dir.	Exf.	Hij.	Abus.	MTE	IPI
Train	1600	421	897	888	443	269	897	703	663	166	156	130	155	330	0
Val	500	138	163	276	139	85	163	337	207	52	49	41	48	103	0
Test	450	132	151	248	126	76	151	299	187	46	44	37	43	93	0
OOD	450	95	102	234	52	164	102	348	0	0	0	0	0	0	450
Total	3000	476	1313	1646	760	594	1313	1687	1057	264	249	208	246	526	450

Table 1: Dataset statistics for SIA-Bench . The table shows both the split geometry used in the paper and the label composition behind each split. D0 and D1 denote lineage depths 0 and 1. Ben., Dir., Exf., Hij., Abus., MTE, and IPI abbreviate benign, direct malicious, data exfiltration, tool-selection hijack, capability abuse, multi-turn escalation, and indirect prompt injection.

Figure 3: Main continuous-risk results on the locked test split and the held-out indirect prompt injection split of SIA-Bench . Figure 3 compares familiar and held-out threat settings under the same six continuous-risk metrics. Each metric column uses its own local scale. Best values are highlighted in orange, second-best values in teal, and faint connectors emphasize the main trade-offs between static screening, contextual scoring, and trigger-dominant fusion.

Method	HR-AUPRC $\uparrow$	Rec@10% $\uparrow$	Prec@10% $\uparrow$	Spearman $\uparrow$	ECE $\downarrow$	W-MAE $\downarrow$
Locked test
No Audit	$0.169$	$0.100$	$0.169$	$0.000$	$0.321$	$0.321$
Static Prior	$0.188$	0.122	0.206	$0.031$	0.149	$0.291$
Denylist	$0.188$	0.122	0.206	$0.031$	0.149	$0.291$
Agent-Audit	$0.169$	$0.090$	$0.152$	$0.015$	$0.318$	$0.319$
Text-Only	$0.189$	$0.066$	$0.111$	$0.073$	$0.217$	0.288

Table 2: Full numeric main-results table for the locked in-distribution test split of SIA-Bench . Bold denotes the best value in each column, and underlining denotes the second-best value.

Figure 4: Validation sweep for calibrated risk fusion. Each point corresponds to one static–trigger fusion weight after score normalization. The frontier visualizes the retrieval–calibration trade-off used to choose the deployment-facing operating point.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting	Value
Stage A	Permission-surface weights	code execution $=1.00$ ; database $=1.00$ ; file read $=0.65$ ; file write $=0.65$ ; network $=0.6318$ ; email $=0.2366$ ; file system $=0.0612$
Stage A	Provenance-trust weights	unverified $=0.3396$ ; official $=0.0896$ ; community $=0.0630$
Stage A	Static-score cap	$R_{\text{static}}\leq 0.40$
Stage B	Scorer family	rule-based contextual scorer with feature profile text_prov_graph_traj
Stage B	Feature weights	intent cues $=0.1046$ ; argument sensitivity $=0.0493$ ; provenance trust $=0.2990$ ; trajectory state $=0.3944$ ; taint propagation $=0.1526$
Stage B	Context interaction defaults	context-gain scale $=0.25$ ; gate floor $=0.08$ ; cross-check boost disabled

Appendix

Table 3: Frozen implementation defaults used by the paper configuration. These values define the static prior, contextual scorer, and default Stage C calibration policy used in the main benchmark runs.

Target construction	Method	HR-AUPRC	Rec@10%	Prec@10%	Spearman $\rho$	ECE $\downarrow$	W-MAE $\downarrow$
Vote only	P2	0.1876	0.1184	0.2000	0.0494	0.2350	0.3183
	P3	0.1935	0.1184	0.2000	0.0529	0.0769	0.3481
Blend 80/20	P2	0.1876	0.1184	0.2000	0.2095	0.2396	0.2995
	P3	0.1935	0.1184	0.2000	0.2147	0.0791	0.3292
Blend 65/35 (paper default)	P2	0.1876	0.1184	0.2000	0.2094	0.2439	0.2864
	P3	0.1935	0.1184	0.2000	0.2146	0.0807	0.3151

Appendix

Table 4: Locked-test sensitivity of the paper’s two primary continuous-risk methods to alternative target-construction mixtures. The ranking metrics are stable across mixtures, while calibration-oriented metrics shift gradually as the heuristic component becomes more dominant.

Target construction	Method	HR-AUPRC	Rec@10%	Prec@10%	Spearman $\rho$	ECE $\downarrow$	W-MAE $\downarrow$
Vote only	P2	0.4429	0.1524	0.5556	0.0837	0.3163	0.4209
	P3	0.4493	0.1524	0.5556	0.0866	0.0866	0.4333
Blend 80/20	P2	0.4429	0.1524	0.5556	0.1945	0.3246	0.3921
	P3	0.4493	0.1524	0.5556	0.1959	0.0932	0.4040
Blend 65/35 (paper default)	P2	0.4429	0.1524	0.5556	0.1945	0.3307	0.3753
	P3	0.4493	0.1524	0.5556	0.1959	0.0981	0.3821

Appendix

Table 5: Held-out indirect prompt injection sensitivity under the same alternative target-construction mixtures. The main paper pattern is preserved: the contextual scorer and the fused policy keep the same high-risk retrieval ordering, while calibration and error trade-offs shift smoothly with the target definition.

Selector	$w_{\text{static}}$	$w_{\text{trigger}}$	$\tau_{\text{esc}}$	$\tau_{\text{block}}$	HR-AUPRC	Rec@10%	Prec@10%	Spearman $\rho$	ECE $\downarrow$	W-MAE $\downarrow$	Macro-F1
Continuous-risk-first	0.55	0.45	0.60	0.75	0.3054	0.2000	0.3400	0.3585	0.0324	0.3062	0.2793
Threshold-first	0.35	0.65	0.30	0.35	0.3054	0.2000	0.3400	0.3585	0.1182	0.2970	0.4468

Appendix

Table 6: Validation-selected operating points from the same

2{,}940

-candidate Stage C search space. The two selectors preserve identical validation ranking metrics but choose materially different thresholded policies. Validation task-completion and false-block rates are

0.9384/0.0000

for continuous-risk-first and

0.7174/0.2029

for threshold-first, respectively.

Split	Selector	HR-AUPRC	Spearman $\rho$	ECE $\downarrow$	W-MAE $\downarrow$	Macro-F1	Malicious Recall	False Block	Task Completion
Locked test	Continuous-risk-first	0.1935	0.2146	0.0807	0.3151	0.2303	0.0248	0.0000	0.9194
Locked test	Threshold-first	0.1935	0.2146	0.1518	0.3098	0.2884	0.2376	0.1532	0.7258
Held-out OOD	Continuous-risk-first	0.4493	0.1959	0.0981	0.3821	0.2699	0.1574	0.0000	0.8889
Held-out OOD	Threshold-first	0.4493	0.1959	0.1735	0.3781	0.3749	0.5185	0.3376	0.5128

Appendix

Table 7: Evaluation of the two validation-selected policies on the locked test split and the held-out indirect prompt injection split. The continuous-risk-first selector preserves the same ranking quality while maintaining substantially lower intervention cost, which is why it is used as the paper’s primary model-selection rule.

Method	HR-AUPRC $\uparrow$	Rec@10% $\uparrow$	Prec@10% $\uparrow$	Spearman $\uparrow$	ECE $\downarrow$	W-MAE $\downarrow$
Text-Only Invocation Audit	$0.2160$	$0.1647$	$0.2800$	$0.0847$	$0.1982$	$0.2812$
Contextual Invocation Audit	$0.2305$	0.1882	0.3200	$0.1299$	$0.1969$	$0.2802$
No provenance signal	$0.2304$	0.1882	0.3200	$0.1302$	0.1967	$0.2804$
No trajectory signal	0.2342	0.1882	0.3200	$0.0978$	$0.1970$	0.2800
No taint propagation signal	$0.2309$	0.1882	0.3200	0.1321	$0.1969$	$0.2803$

Appendix

Table 8: Validation ablation of structured context signals in Stage B. Removing any single context channel changes validation performance only slightly, suggesting cumulative rather than single-feature gains.

Method / split	Req.-patch Acc.	Patch Acc.	Zero-reg. Pass	Triggered
Contextual Invocation Audit / test	$0.6844$	$0.3487$	$0.5000$	$31$
Calibrated Fusion Policy / test	$0.5356$	$0.1544$	$0.2547$	$230$
Contextual Invocation Audit / OOD	$0.5556$	$0.0388$	$0.5185$	$50$
Calibrated Fusion Policy / OOD	$0.5111$	$0.0579$	$0.2177$	$326$

Appendix

Table 9: Stage D remediation quality. Req.-patch Acc. denotes whether the generated patch family matches the flagged failure type, and Zero-reg. Pass measures whether generated regression tests pass without obvious regressions.

Comparison	Split	Method	HR-AUPRC	Recall@10%
IPIGuard replay	test	IPIGuard	$0.1522$	$0.0000$
		Static Denylist	0.3273	0.2857
	ood	IPIGuard	$0.3637$	$0.0671$
		Contextual Invocation Audit	0.4052	0.1463
SkillJect -derived threat slice	test	Static Denylist	0.3273	0.2857
		Contextual Invocation Audit	$0.1955$	$0.0000$

Appendix

Table 10: Supplementary continuous-risk comparisons. These analyses align subsets of SIA-Bench with prior threat models, but are not protocol-matched replacements for the main benchmark.

Agent Skill Security Auditing

Agent skill security auditing examines reusable agent instructions, code, and tools for malicious behavior, unsafe permissions, and supply-chain risks.

See all 65 papers

Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security
Jun 10, 2026Tu Lan, Chaowei XiaoAI Agent Auditing LLM Agent Security
Johns Hopkins University
SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment
Jun 21, 2026Dexu Yu, Youhua Li, Zhaoyang Guan +12AI Agent Auditing Automated Evaluation
Northeastern University · City University of Hong Kong · DeciLix Lab +8
Structured Security Auditing and Robustness Enhancement for Untrusted Agent Skills
Apr 28, 2026Lijia Lv, Xuehai Tang, Jie Wen +2AI Agent Auditing AI Agent Security
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences

Jun 10, 2026J/K move · Enter open · S save

Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security

Tu Lan, Chaowei Xiao

Johns Hopkins University

AI Agent Auditing LLM Agent Security Agent Skill Security Auditing

Open paper AskPDF arXiv

Agent skills let LLM agents reuse instructions, resources, tools, and workflows, but they also create a new place for malicious behavior to hide. A skill may look benign in its documentation or code while becoming harmful only when it is invoked with particular user requests, local assets, persistent state, or multi-step tool interactions. This makes purely static vetting brittle. We present Runtime Skill Audit (RSA), a dynamic analysis method that audits skills by asking what the skill-mediated agent actually does under targeted runtime conditions. Instead of testing every skill with the same generic tasks, RSA profiles risk-relevant interfaces, prepares the execution context needed to exercise them, and assigns security labels from the resulting trace evidence. We instantiate RSA on OpenClaw and evaluate it on 100 skills against representative static baselines. RSA achieves 90.0% accuracy with an 88.0% true positive rate and an 8.0% false positive rate, improving accuracy by 13.0 percentage points over the best static baseline. Under self-evolving attacks, static detectors collapse after one or two rounds, while RSA continues to detect 19--20 out of 20 malicious skills across rounds.

Abstract

Agent Skill Security Auditing

Explore similar work

Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

Structured Security Auditing and Robustness Enhancement for Untrusted Agent Skills