Autonomous agent systems increasingly depend on reusable skill abstractions for consolidating experiential knowledge and domain expertise. These artifacts typically bundle free-form instructions with heterogeneous resources. However, ensuring their correctness remains challenging. Their failure modes transcend conventional code defects to subtle semantic inconsistencies such as intent conflicts, which manifest as silent failures masked by the underlying model. Moreover, skill correctness must be grounded in intended task boundaries and generalizability. We propose SkillSpec, a Hoare-style framework that formulates skill correctness as a specification reasoning problem. It transforms a heterogeneous skill repository into a unified graph representation that aligns descriptions, instructions and code artifacts. For each node, SkillSpec derives an ExpectSpec from the surrounding declared intent, and infers FactSpecs from encoded behavior under partially disclosed intent. An intent mask regulates access to holistic, lineage, neighborhood, and local views to balance the bias introduced by excessive context against unsupported inference caused by insufficient context. SkillSpec jointly reasons over these views to flag candidate defects, and automatically validates them in an isolated sandbox. On 515 real-world skills from SkillsBench and widely downloaded repositories, SkillSpec identified 763 manually confirmed defects across 239 skills, achieving 61.2% precision. The node-level analysis across multiple model families shows that specification reasoning is consistently reliable for code nodes, whereas plain-text nodes remain a major bottleneck. Most defects arise at the boundaries between declared intent and implementation, demonstrating that explicit specifications provide a practical foundation for skill quality assurance in real-world agent ecosystems.
Figures & tables
Fig. 1: Anatomy of popular repositories on skills.sh: (a) distribution of SKILL.md lengths, (b) number of files per skill, (c) distribution of file types, and (d) number of distinct file types per skill.
Fig. 2: Semantic structure and similarity alignment between SKILL.md text and script in the skill-creator .
Fig. 3: The Example for Skill Specification
Fig. 4: Overview of SkillSpec.
Fig. 5: Intent-mask Visibility Layers
Layer
Workflow Graph Node
Code Graph Node
Holistic
The skill-level objective and global constraints, represented by the overview, root, and context units.
The same skill-level objective and global constraints, shared by all definitions in the code graph.
Lineage
Upstream workflow units, dependencies, and artifacts along the execution path leading to the target step.
Ancestor definitions along the resolved call path leading to the target definition.
Neighbors
The parent stage and sibling operations that establish the local role and coordination constraints of the target step.
The immediate caller and sibling callees that establish the local calling context of the target definition.
Self
The target workflow step. For FactSpec, the implementation branch reached through the intent / implementation binding is additionally exposed.
The target definition and its linked workflow unit. The target source is visible in all factual views and masked in the expectation view.
TABLE I: Visibility-layer semantics for workflow and code graph nodes.
Fig. 6: Specification with Multiple Views.
Dataset
Type
# Skills
# Files
Skill lines
Code lines
Other lines
SkillsTop
Text
160
1,032
26,533
–
96,169
Script
160
2,873
39,624
172,402
453,030
SkillsBench
Text
152
280
28,518
–
25,318
Script
43
303
8,330
26,002
61,339
Total
515
4,488
103,005
198,404
635,856
TABLE II: Statistics of the evaluation datasets.
Dataset
Type
Skills
Workflow Defects
Code Defects
Total
Defective
Rate
Avg. Defects
Valid.
Conf. Manual
Prec.
Valid.
Conf. Manual
Prec.
SkillsTop
Text
160
29
18.1%
1.4
103
41
39.8%
–
–
–
Script
160
111
69.4%
4.2
262
141
53.8%
482
320
66.4%
SkillsBench
Text
152
72
47.4%
2.0
221
143
64.7%
–
–
–
Script
43
27
62.8%
4.4
39
21
53.8%
139
97
69.8%
Overall
515
239
46.4%
3.2
625
346
55.4%
621
417
67.1%
TABLE III: Overall effectiveness of SkillSpec.
Fig. 7: Per-skill confirmed defect distributions in SkillsTop and SkillsBench.
Category
Workflow Defects
Code Defects
Num.
Prop.
Num.
Prop.
Requirement Error
149
43.1%
7
1.7%
Constraint Omitted
46
13.3%
28
6.7%
Semantic Conflict
61
17.6%
0
0.0%
Implementation Deviation
90
26.0%
382
91.6%
Total
346
–
417
–
TABLE IV: Taxonomy of manually confirmed skill defects.
Node Type
Distribution
Eligible
Defect Nodes
Num.
Prop.
Num.
Cand.
Valid.
Conf.
Rate
Workflow graph
root
515
4.5%
–
–
–
–
–
stage
2,110
18.4%
–
–
–
–
–
context
2,261
19.7%
–
–
–
–
–
plain
3,351
29.1%
2633
246
142
64
1.9%
TABLE V: Distribution and validation of graph nodes.
Fig. 8: Per-skill graph element distributions in SkillsTop and SkillsBench.
Model
Dataset
Skills
Workflow Defects
Code Defects
Candidate
Validated
Rate
Candidate
Validated
Rate
Candidate
Validated
Rate
GPT-5.6-Sol
SkillsTop-T
76
50
65.8%
183
103
56.3%
–
–
–
SkillsTop-S
138
127
92.0%
409
262
64.1%
532
482
90.6%
SkillsBench-T
105
89
84.8%
334
221
66.2%
–
–
–
SkillsBench-S
38
36
94.7%
55
39
70.9%
152
139
91.4%
Total
357
302
84.6%
981
625
63.7%
684
621
90.8%
TABLE VI: Robustness of SkillSpec across LLM backends and datasets.
Method
Type
Candidates
Sandbox Validated
Manually Confirmed
Vanilla
Workflow
102
67 (65.7%)
29 (28.4%)
Code
179
149 (83.2%)
101 (56.4%)
SkillSpec
Workflow
55
39 (70.9%)
21 (38.2%)
Code
152
139 (91.4%)
97 (63.8%)
TABLE VII: Specification ablation on SkillsBench-S with GPT-5.6-Sol.
LLM-powered agents can silently delete documents, leak credentials, or transfer funds on a routine user request, not because the agent was attacked, but because the skill it invoked broke its own declared safety rules. We call these specification violations: benign inputs cause a skill to breach the natural-language guardrails in its own specification, typically because the guardrail's semantics are undefined for autonomous execution, or because the implementation silently ignores the documented constraint. These violations are invisible to static analyzers, traditional fuzzers, and prompt-injection defenses alike, yet they undermine the very contract a user trusts when installing a skill. We present Sefz, a goal-directed semantic fuzzing framework that automatically discovers specification violations in agent skills. Sefz translates each guardrail into a reachability goal over an annotated execution trace, reducing violation checking to a deterministic graph query. An LLM-based mutator generates benign inputs whose traces progressively approach the violation patterns, guided by a multi-armed bandit that uses goal-proximity as its reward signal. On 402 real-world skills from the largest public agent-skill marketplace, Sefz finds specification violations in 120 (29.9%), including 26 previously unknown exploitable guardrail violations in deployed skills. Six recurring specification pitfalls explain the bulk of the failures, suggesting concrete principles for safer skill design.
Ying Li, Hongbo Wen, Yanju Chen +3
University of California, Los Angeles · University of California, Santa Barbara · University of California, San Diego
Under the current standard, Agent Skills are SKILL.md files that combine instructions with supporting resources, enabling Large Language Model (LLM) agents to reuse procedures beyond a single conversation. Yet many public skills appear to originate from a single task, repository, or conversation, even when they are shared as reusable components. We analyze this gap across 138,133 public SKILL.md files from 20,556 repositories using a two-tier defect taxonomy grounded in the official specification and best-practice guidance. We find that 91.8% of skills contain at least one detected defect, with stable estimates across lenient and strict thresholds (88.8-94.6%). The dominant failures are ordinary packaging problems rather than exotic attacks: weak routing metadata, bloated or non-actionable bodies, and poor resource organization. A deterministic routing stress test over 20,000 skills shows the functional impact: skills with valid routing metadata are retrieved more reliably from startup descriptions than skills with routing defects. Defect rates vary by platform and provenance: specification-aware skills contain fewer defects, while AI-marked skills show more safety and portability problems. Lightweight enforcement and repair experiments support a quality-assured generation workflow combining spec-aware prompting, lightweight linting, automated repair, and safety gating.
Chi Zhang, Yimin Liu, Xinze Chen +1
The Graduate Center, City University of New York · The Ohio State University · Hunter College, City University of New York
Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.
Dexu Yu, Youhua Li, Zhaoyang Guan +12
Northeastern University · City University of Hong Kong · DeciLix Lab +8