Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem
Authors: Florian Holzbauer, David Schmidt, Gabriel Gegenhuber, Sebastian Schrittwieser, Johanna Ullrich
Organizations: Interdisciplinary Transformation University (IT:U), Linz, Austria · University of Vienna, CDL AsTra, Faculty of Computer Science, Vienna, Austria
Agent skills extend local AI agents, such as Claude Code and OpenClaw, with additional functionality. Their growing popularity has led to dedicated marketplaces resembling mobile app stores, as well as automated scanners that assess whether skills are benign or malicious. However, scanner reports from individual marketplaces classify up to 46.8% of skills as malicious, raising concerns about false positives. We present the largest empirical security analysis of the AI agent skill ecosystem to date. We collect 238,180 unique skills from three major distribution platforms and GitHub, and analyze their contents, behavior, and repository context. Unlike existing scanner-based assessments, which evaluate skills largely in isolation, our repository-aware analysis checks whether a flagged skill is consistent with its surrounding GitHub project. This context substantially reduces the number of suspicious skills: only 0.52% remain suspicious after repository-aware analysis. Our results show that existing scanners can substantially overestimate maliciousness when repository context is ignored. At the same time, we identify previously undocumented real-world attack vectors, including the hijacking of skills hosted in abandoned GitHub repositories. Overall, our findings provide a more robust view of the agent-skill ecosystem's current risk surface and highlight the need for context-aware security evaluation.
Figures & tables
Figure 1. Overview of our repository-aware skill analysis to reduce the high number of false positives. Our approach consists of three stages, encompassing skill collection, malicious classification, and repository context analysis. The figure illustrates our repository-aware skill analysis pipeline designed to reduce malicious claims. It consists of three sequential stages. First, the system collects skills from multiple platforms in a cross-platform aggregation step. Second, it applies a classification component to identify potentially malicious skills based on predefined criteria. Third, it performs repository context analysis by examining associated code repositories to validate claims and detect inconsistencies.
Skill Metric
ClawHub
SkillsDir.
Skills.sh
GitHub
Indexed
16,755
32,896
79,735
142,824
Retrieved
16,755
17,611
125,928
136,095
Added
16,755
17,611
112,231
91,583
Owners
n/a
709
7,950
14,197
Repositories
n/a
766
9,431
16,413
#Total 238,180 (distinct analyzed skills)
Table 1. Overview of collected agent skills from ClawHub Steinberger, , SkillDirectory Skills Directory, , Skills.sh Vercel Labs, , and GitHub. Retrieved denotes successfully downloaded skills retained for analysis; Added denotes skills retained after cross-source deduplication. For Skills.sh, the crawl retrieved 55,366 listed skills and 77,456 additional skills extracted from referenced repositories.
Figure 3
Scanner
Scanned
Pass
Fail
Fail Rate
Clawhub
VirusTotal
12,213
7,792
4,421
36.20%
OpenClaw Scanner
14,244
8,271
5,973
41.93%
GPT 5.3-based
16,424
10,050
6,374
38.8%
Cisco Skill Scan
16,745
13,941
2,804
16.74%
Skills.sh
agent-trust-hub
62,163
53,611
8,552
13.76%
snyk
46,414
42,843
3,571
7.69%
Table 2. Comparison of security scanners used in the skill ecosystem. The table contrasts scanners deployed on ClawHub and Skills.sh (highlight in gray) with Cisco’s skill scanner and our LLM-based feature set.
Figure 5
Category
Total
Flagged
Rate
Skills (overall)
62,219
12,004
19.29%
Skills (repo stars > 1000)
7,725
1,656
21.44%
Skills (installs > 1000)
755
122
16.16%
Repositories (overall)
8,451
3,878
45.89%
Repositories (stars > 1000)
528
268
50.76%
Repositories (installs > 1000)
171
105
61.40%
Table 3. Skill.sh malicious detection rates for skills and repositories (skill flagged if at least one scanner flagged, repository malicious if at least one skill is malicious)
Figure 6. Metadata scores for unique repositories. The figure presents the metadata scores for unique repositories.
Figure 7. Codebase score category results. Categories marked with * indicate that lower values are better. The figure shows the codebase score results per category.
Figure 8. Repository context score and its weighted subcomponents, with 70% codebase score and 30% metadata score. The figure illustrates the repository context score together with its weighted subcomponents.
Figure 9. Comparison of maturity scores assigned by manual reviewers and our repository aware scoring approach. The figure compares maturity scores assigned by manual reviewers with those produced by our repository aware scoring approach.
Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.
Agent skills combine natural-language instructions with executable code while inheriting an agent's filesystem, credential, and network access. Attacks can span prose and files, whereas regex and code-only analyzers cover only one modality. SkillSieve applies three progressively deeper layers: recall-oriented regex, AST, and metadata triage; four parallel LLM security sub-tasks; and an independent three-model jury with debate on disagreement. We evaluate 49,592 real ClawHub skills, a 390-skill labeled benchmark, and 100 adversarial samples across five evasion techniques on a 440 USD ARM board. The full pipeline achieves F1 = 0.929 (precision 0.912, recall 0.945) at an average cost of $0.006 per skill. An optional XGBoost fast path reduces Layer-2/3 calls by 32% with a 1.7-point F1 decrease while preserving recall. On 52 Feishu/Lark packages, Layer 2 reclassifies 13 of 14 Layer-1 flags as safe after contextual analysis; we also deploy the system as a Feishu chat bot. Code, labels, and aggregate results are open-sourced.
Yinghan Hou, Zongyou Yang
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · Dyson School of Design Engineering, Imperial College London, London, United Kingdom
Agent skills let LLM agents reuse instructions, resources, tools, and workflows, but they also create a new place for malicious behavior to hide. A skill may look benign in its documentation or code while becoming harmful only when it is invoked with particular user requests, local assets, persistent state, or multi-step tool interactions. This makes purely static vetting brittle. We present Runtime Skill Audit (RSA), a dynamic analysis method that audits skills by asking what the skill-mediated agent actually does under targeted runtime conditions. Instead of testing every skill with the same generic tasks, RSA profiles risk-relevant interfaces, prepares the execution context needed to exercise them, and assigns security labels from the resulting trace evidence. We instantiate RSA on OpenClaw and evaluate it on 100 skills against representative static baselines. RSA achieves 90.0% accuracy with an 88.0% true positive rate and an 8.0% false positive rate, improving accuracy by 13.0 percentage points over the best static baseline. Under self-evolving attacks, static detectors collapse after one or two rounds, while RSA continues to detect 19--20 out of 20 malicious skills across rounds.
AI agents increasingly act through agent skills, i.e., natural-language instructions, that direct a host agent toward shell, network, credential, file, and process actions, and public registries distribute them at scale. In the first half of 2026, the OpenClaw AI agent went viral, and its public skill registry boomed: the observable stock nearly doubled in 91 days, and a majority of the listings visible in June were created in just two months. By the end of our study window, the wave had crested, and monthly listing creation and core-repository activity were falling from their spring peaks. This paper measures what the boom left behind, drawing on the OpenClaw Git history, its GitHub issues and pull requests, and three ClawHub registry snapshots. Attention is concentrated: the top 10% of skills received 46.93% of all downloads. No simple skill features (like size or download counts) remained a stable predictor of continued listing once creation cohort and skill age were controlled. Human scrutiny did not stay: 77.86% have zero stars and zero comments, while 85.06% of the readable skills carry privilege evidence. And automated cleanup is not ready: the three security scanners disagreed on 23,702 of the 61,990 skills they all cover. After human adjudication, weighted scanner sensitivity against the reference standard ranged from 21.67% to 61.06%. Governing fast-growing agent-skill registries cannot rely on simple metadata or single scanner scores; it requires robust, transparent measurement and independent validation.