SkillLifeBench: Benchmarking Security Across the Lifecycle of Agent Skills

Chenxu Zhang; Zhou Liu; Wu Xin; Wentao Zhang

cs.LGOct 26, 2026

SkillLifeBench: Benchmarking Security Across the Lifecycle of Agent Skills

Authors: Chenxu Zhang, Zhou Liu, Wu Xin, Wentao Zhang

Organizations: Imperial College London, United Kingdom · Peking University, China

Abstract

Web agents reuse skills whose stored code, selected version, and loaded implementation can diverge across tasks and updates. SkillLifeBench links these states to task outcomes in 388 paired scenarios spanning implementation, interface, runtime, composition, and version changes. We evaluate 14 models on two paired decision tracks: 89 runtime tasks and 60 autonomous lifecycle tasks spanning 12 templates derived from introduced cases. Lifecycle tasks allow agents to inspect, select, and reconfigure skills while completing verifiable user requests. Joint success requires task completion without the registered attack goal. GPT-5.6-Sol and Sonnet 4.6 both achieve 82.0% on runtime tasks, but reach 96.7% and 56.7% on lifecycle tasks. Across lifecycle evaluations, challenged task success is 77.5%, while joint success is 65.2%; 103 runs complete the task and the attack goal. Adding execution provenance in a matched recovery study raises utility joint success from 49.35% with logs and metadata to 65.58%. A deterministic execution-consistency rule reaches 64.94% with higher benign utility, while full restoration reaches 69.48%. Lifecycle tasks distinguish models with similar runtime scores, and execution evidence improves recovery.

Figures & tables

Figure 1. SkillLifeBench overview. (a) 388 source scenarios; (b) paired evaluation on 89 runtime and 60 lifecycle tasks; (c) intervention feasibility; (d) matched recovery; (e) joint-success rankings across 14 models. The benchmark contains 388 paired source scenarios. Fourteen models evaluate 89 runtime tasks and 60 autonomous lifecycle task instances. The recovery and intervention studies retain their original task groups. The final panel ranks all 14 models by lifecycle joint success and shows their runtime scores alongside. Among 651 completed challenged lifecycle tasks, 103 also achieve the attack goal.

Figure 2. Controlled OpenClaw update. Selection returns patched

v_{1}

, but a cached handle executes

v_{0}

. The published ZIP preserves the requested document and includes prohibited .git metadata. The user asks to publish the example-skill documents while excluding .git metadata. The agent selects patched version 1 of the packaging skill packager-vcs, but the runtime reuses a cached version 0 handle. The locally captured ZIP contains both the required normal.txt document and prohibited .git/config metadata. The target bundle example-skill is distinct from the packaging skill.

Benchmark	Skill artifacts	Retained state	Version changes	Utility + harm	Versioned provenance	Invoked code binding	Evidence- guided mitigation
AgentDojo ( Debenedetti et al., 2024 )	✗	—	—	✓	—	—	—
InjecAgent ( Zhan et al., 2024 )	✗	—	—	$\sim$	—	—	—
ToolSword ( Ye et al., 2024 )	$\sim$	—	—	$\sim$	—	—	—
AgentTrap ( Zhuang et al., 2026 )	✓	$\sim$	—	$\sim$	$\sim$	—	—
MetaCascade ( Wen et al., 2026 )	✓	—	—	✗	—	✗	—
SCR-Bench ( Xie et al., 2026 )	✓	—	—	—	—	—	—

Table 1. Evaluated capabilities of agent and skill benchmarks (definitions in Section 2 ). ✓ evaluated;

\sim

partial; ✗ outside the stated protocol; — not reported.

Figure 3. Task and attack outcomes on 840 challenged lifecycle executions. Joint success occupies the upper-left cell; 103 completed tasks also achieve the registered attack goal. The four task/attack-goal cells contain 548 task successes without the goal, 103 task successes with the goal, 153 task failures without the goal, and 36 task failures with the goal.

Panel of Table 2 — Table 2. Model decision quality on runtime and lifecycle tasks (%). Joint success rate (Joint SR) requires task success and no registered unauthorized goal under challenge. Rows rank by lifecycle Joint SR; equal scores share rank. Task success rate (TSR) measures task completion. Each lifecycle template contributes five tasks, so its equal-template average equals the rate over all 60 tasks.

Subset	$N$	Default	Intervention	$\Delta$
1. Default $\rightarrow$ Authorization check
B/D/E	92	31.52	46.74	+15.22
C: core $\dagger$	178	67.42	72.47	+5.06
C: other	70	37.14	50.00	+12.86
2. Default deployment $\rightarrow$ Authorized restoration
Deployment	378	6.88	70.63	+63.76

Table 3. Effects of authorization checking and restoration. Rates are joint success (%).

N

counts evaluations per condition across the two Qwen models;

\Delta

is in percentage points.

Figure 4. Lifecycle decision quality by task. Templates are grouped by category and ordered by decreasing joint success. Each rate pools 14 models over five instances (70 challenged runs); panels share a 0–100% scale. Three panels group implementation and permission tasks, persistent composition tasks, and version-update tasks. Each panel contains four horizontal bars sorted by decreasing joint success. All panels share a zero-to-one-hundred percent scale and display exact successes out of 70.

Figure 5. Intervention feasibility by case family. Of 388 cases, 170 have an agent-visible workflow, 189 have a deployment intervention, and 29 have no verified intervention. Cases are classified before model evaluation. Five stacked horizontal bars for families A through E. Agent-visible workflow, deployment intervention, and no verified intervention case counts respectively are: A: 0, 19, 0; B: 10, 60, 7; C: 124, 0, 4; D: 30, 25, 10; E: 6, 85, 8.

Figure 6. Recovery policies and execution evidence. (a,b) Risk joint success and benign utility, each over 154 evaluations, with rows ordered by risk success. (c) Recovery loss after removing each evidence group from Provenance (percentage points). (d) Full versus Restore on 46 risk cases without a hash trigger. Colors identify methods; Table 8 gives exact counts. A single-column two-by-two chart. The top panels share the method order Full, Provenance, Link, Restore, Reload and Logs. Risk successes are 107,101,100,94,90,76 and benign utility successes are 117,115,120,118,122,120, respectively, each out of 154. Removing source hashes, handle links and implementation identity reduces success by 10.39,6.49,0.65 percentage points, respectively. Without a hash trigger, Full succeeds on 32 of 46 risk tasks and Restore on 19 of 46.

Panel of Table 4 — Table 4. A selected successor with an executed parent. A linked OpenClaw update ( Sid, 2026 ) under three controlled routing policies. The synthetic canary is captured locally; utility measures document preservation.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Fault type	$N$	Source	Changed state	Recovery actions
Stored code replacement	24	Storage setup	Altered disk code; unchanged interface.	Activate approved code and reload.
Version/routing mismatch	10	Registry intervention	Pointer selects an ineligible version.	Activate an eligible target and reload.
Stale loaded code	12	Controlled update	Updated disk/active version; cached parent.	Reload or restart.
Retained old handle	8	Handle retention	New code loaded; callable retains old code.	Refresh references; reload or restart.
Wrong instance, same digest	5	Instance-binding intervention	Matching hashes; invalid public instance binding.	Refresh references; reload or restart.
Permitted code with a defect	18	Approval/update records	Permitted older code executes; newer eligible candidate exists.	Activate newer approved code; reload.

Appendix

Table 6. Recovery-set construction. The 77 B-family parents receive controlled faults before recovery. Methods share six actions, including Full; fault-to-action mappings are withheld from diagnostic models.

Task template	Parent	Complete endpoint
Private report	B-REPAIR-011	Full text, owner-only access
Local document	B3-002	Exact file, retained register
Local data filter	B1-002	All matching rows/fields
Calendar read	B-REPAIR-009	All meetings, unchanged state
Revoked grant	D-EXP-020	All active-grant values
Expired cache	D-EXP-021	Complete current object

Appendix

Table 7. Lifecycle templates: five instances each; unique parent-ID prefixes.

Condition	9B	4B	Risk all	Benign	E only	Row only
Logs (M)	42	34	76	120	30	5
Reload	49	41	90	122	–	–
Restore	51	43	94	118	12	5
Link	54	46	100	120	4	3
Provenance (E)	55	46	101	115	0	0
Full	58	49	107	117	–	–

Appendix

Table 8. Recovery baselines and evidence ablations. Risk counts: 77 per model, 154 pooled; benign utility: 154. Last columns: risk successes unique to Provenance (E) or the row condition.

Panel of Table 9 — Table 9. Decision-track outcome counts. Order: clean task / challenged task / attack goal / joint success. Denominators: runtime 89, lifecycle 60. All clean goals are zero.

Panel of Table 10 — Table 10. Effects by perturbation family. Cells report positive and unresolved effect counts; others are False. Rates are $100p/N$ for positive count $p$ and column denominator $N$ . Each of 12 models covers 388 cases.

Subset	Qwen	$N$	Joint	Utility	Harm	Unobs.
Non-runtime	9B	46	17 $\to$ 25	29 $\to$ 31	18 $\to$ 11	3 $\to$ 3
	4B	46	12 $\to$ 18	24 $\to$ 25	22 $\to$ 16	4 $\to$ 4
C: core 89	9B	89	65 $\to$ 69	68 $\to$ 75	4 $\to$ 7	0 $\to$ 0
	4B	89	55 $\to$ 60	66 $\to$ 70	15 $\to$ 14	0 $\to$ 0
C: other 35	9B	35	15 $\to$ 20	24 $\to$ 25	11 $\to$ 7	3 $\to$ 3
	4B	35	11 $\to$ 15	20 $\to$ 21	14 $\to$ 10	4 $\to$ 4

Appendix

Table 11. Per-model outcomes with authorization checking. Qwen3.5-9B/4B counts compare Default

\to

Check;

N

is per condition. Utility counts task or useful-output successes; harm counts registered violations. Unobs.: unobserved harm checks.

AI Agent Security Benchmarks

AI agent security benchmarks measure how tool-using and autonomous systems handle attacks, privacy risks, authorization, and harmful actions.

See all 305 papers

SoK: Agentic Skills -- Beyond Tool Use in LLM Agents
Feb 24, 2026Yanna Jiang, Delong Li, Haiyu Deng +4Agentic AI LLM Agent Skill Learning
University of Technology Sydney · CSIRO Data61
Skill Coverage: A Test Adequacy Metric for Agent Skills
Jun 9, 2026Boyin Tan, Xiaowei Huang, Youcheng SunLLM Agent Evaluation AI Agent Benchmarks
1Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates · University of Liverpool Liverpool, United Kingdom
AgentTrap: Measuring Runtime Trust Failures in Third-Party Agent Skills
May 13, 2026Haomin Zhuang, Hanwen Xing, Yujun Zhou +5AI Agent Security LLM Agent Security
University of Notre Dame · University of Southern California · LMU Munich +1

Feb 24, 2026J/K move · Enter open · S save

SoK: Agentic Skills -- Beyond Tool Use in LLM Agents

Yanna Jiang, Delong Li, Haiyu Deng, Baihe Ma+3

University of Technology Sydney · CSIRO Data61

Agentic AI LLM Agent Skill Learning LLM Agent Security

Open paper AskPDF arXiv

Agentic systems increasingly rely on reusable procedural capabilities, a.k.a., agentic skills, to execute long-horizon workflows reliably. These capabilities are callable modules that package procedural knowledge with explicit applicability conditions, execution policies, termination criteria, and reusable interfaces. Unlike one-off plans or atomic tool calls, skills operate (and often do well) across tasks. This paper maps the skill layer across the full lifecycle (discovery, practice, distillation, storage, composition, evaluation, and update) and introduces two complementary taxonomies. The first is a system-level set of seven design patterns capturing how skills are packaged and executed in practice, from metadata-driven progressive disclosure and executable code skills to self-evolving libraries and marketplace distribution. The second is an orthogonal \textbf{representation

\times

scope} taxonomy describing what skills are (natural language, code, policy, hybrid) and what environments they operate over (web, OS, software engineering, robotics). We analyze the security and governance implications of skill-based agents, covering supply-chain risks, prompt injection via skill payloads, and trust-tiered execution, grounded by a case study of the ClawHavoc campaign in which nearly 1{,}200 malicious skills infiltrated a major agent marketplace, exfiltrating API keys, cryptocurrency wallets, and browser credentials at scale. We further survey deterministic evaluation approaches, anchored by recent benchmark evidence that curated skills can substantially improve agent success rates while self-generated skills may degrade them. We conclude with open challenges toward robust, verifiable, and certifiable skills for real-world autonomous agents.

Abstract

AI Agent Security Benchmarks

Explore similar work

SoK: Agentic Skills -- Beyond Tool Use in LLM Agents

Skill Coverage: A Test Adequacy Metric for Agent Skills

AgentTrap: Measuring Runtime Trust Failures in Third-Party Agent Skills