Red Skills or Blue Skills? A Dive Into Skills Published on ClawHub

Haichuan Hu; Ye Shang; Quanjun Zhang

cs.CLMar 19, 2026

Red Skills or Blue Skills? A Dive Into Skills Published on ClawHub

Authors: Haichuan Hu, Ye Shang, Quanjun Zhang

Organizations: The Hong Kong Polytechnic University · Nanjing University · Nanjing University of Science and Technology

Abstract

Skill ecosystems have emerged as an increasingly important layer in Large Language Model (LLM) agent systems, enabling reusable task packaging, public distribution, and community-driven capability sharing. However, despite their rapid growth, the functionality, ecosystem structure, and security risks of public skill registries remain underexplored. In this paper, we present an empirical study of ClawHub, a large public registry of agent skills. We build and normalize a dataset of 26,502 skills, and conduct a systematic analysis of their language distribution, functional organization, popularity, and security signals. Our clustering results show clear cross-lingual differences: English skills are more infrastructure-oriented and centered on technical capabilities such as APIs, automation, and memory, whereas Chinese skills are more application-oriented, with clearer scenario-driven clusters such as media generation, social content production, and finance-related services. We further find that more than 30% of all crawled skills are labeled as suspicious or malicious by available platform signals, while a substantial fraction of skills still lack complete safety observability. To study early risk assessment, we formulate submission-time skill risk prediction using only information available at publication time, and construct a balanced benchmark of 11,010 skills. Across 12 classifiers, the best Logistic Regression achieves a accuracy of 72.62% and an AUROC of 78.95%, with primary documentation emerging as the most informative submission-time signal. Our findings position public skill registries as both a key enabler of agent capability reuse and a new surface for ecosystem-scale security risk.

Figures & tables

Figure 1: Weekly accumulated creation, update and install on ClawHub.

Figure 2: Data collection pipeline.

Figure 3: Domain distribution of skills.

Panel of Figure 4 — Figure 4: Cluster analysis.

Figure 5: Download distribution of skills.

Panel of Figure 6 — Figure 6: Risk overview.

Figure 7: Comparison between risk detection by LLMs and security scanning tools.

Panel of Figure 8 — Figure 8: Risk distribution by domain and function.

Figure 9: List of owners aggregating high-risk skills.

Method	Feature	ACC	PRE	REC	F1	AUC	AVG
Logistic Regression	40910	72.62	72.68	72.48	72.58	78.95	73.86
MLP	128	71	68.81	76.75	72.57	77.49	73.32
Random Forest	256	70.53	68.08	77.29	72.39	77.01	73.06
Extra Trees	256	68.89	66.46	76.3	71.04	76.48	71.83
Hist Gradient Boosting	256	69.98	68.68	73.48	70.99	77.63	72.15
SGD	40910	68.94	68.9	69.03	68.97	75.19	70.21

Table 1: Risk detection results of different classifiers.

Feature	ACC	PRE	REC	F1	AUC	AVG
drop changelog	70.21	69.71	71.48	70.58	78.12	72.02
drop file paths	70.57	70.61	70.48	70.55	78.33	72.11
drop name	71.48	71.33	71.84	71.58	79.12	73.07
drop structure	71.66	71.43	72.21	71.82	78.29	73.08
drop summary	72.84	72.68	73.21	72.94	78.91	74.12
drop tags	72.25	72.31	72.12	72.21	78.87	73.55

Table 2: Ablation of risk detection inputs on Logistic Regression.

Agent Skill Security Auditing

Agent skill security auditing examines reusable agent instructions, code, and tools for malicious behavior, unsafe permissions, and supply-chain risks.

See all 65 papers

Apr 16, 2026

HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?

Large language models (LLMs) have evolved into autonomous agents that rely on open skill ecosystems (e.g., ClawHub and Skills.Rest), hosting numerous publicly reusable skills. Existing security research on these ecosystems mainly focuses on vulnerabilities within skills, such as prompt injection. However, there is a critical gap regarding skills that may be misused for harmful actions (e.g., cyber attacks, fraud and scams, privacy violations, and sexual content generation), namely harmful skills. In this paper, we present the first large-scale measurement study of harmful skills in agent ecosystems, covering 98,440 skills across two major registries. Using an LLM-driven scoring system grounded in our harmful skill taxonomy, we find that 4.93% of skills (4,858) are harmful, with ClawHub exhibiting an 8.84% harmful rate compared to 3.49% on Skills.Rest. We then construct HarmfulSkillBench, the first benchmark for evaluating agent safety against harmful skills in realistic agent contexts, comprising 200 harmful skills across 20 categories and four evaluation conditions. By evaluating six LLMs on HarmfulSkillBench, we find that presenting a harmful task through a pre-installed skill substantially lowers refusal rates across all models, with the average harm score rising from 0.27 without the skill to 0.47 with it, and further to 0.76 when the harmful intent is implicit rather than stated as an explicit user request. We responsibly disclose our findings to the affected registries and release our benchmark to support future research (see https://github.com/TrustAIRLab/HarmfulSkillBench).

Yukun Jiang, Yage Zhang, Michael Backes +2

CISPA Helmholtz Center for Information Security

LLM Agent Security AI Agent Security Benchmarks AI Agent Safety Benchmarks LLM Agent Safety

Ask PDF

Apr 8, 2026

SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills

Agent skills combine natural-language instructions with executable code while inheriting an agent's filesystem, credential, and network access. Attacks can span prose and files, whereas regex and code-only analyzers cover only one modality. SkillSieve applies three progressively deeper layers: recall-oriented regex, AST, and metadata triage; four parallel LLM security sub-tasks; and an independent three-model jury with debate on disagreement. We evaluate 49,592 real ClawHub skills, a 390-skill labeled benchmark, and 100 adversarial samples across five evasion techniques on a 440 USD ARM board. The full pipeline achieves F1 = 0.929 (precision 0.912, recall 0.945) at an average cost of $0.006 per skill. An optional XGBoost fast path reduces Layer-2/3 calls by 32% with a 1.7-point F1 decrease while preserving recall. On 52 Feishu/Lark packages, Layer 2 reclassifies 13 of 14 Layer-1 flags as safe after contextual analysis; we also deploy the system as a Feishu chat bot. Code, labels, and aggregate results are open-sourced.

Yinghan Hou, Zongyou Yang

Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · Dyson School of Design Engineering, Imperial College London, London, United Kingdom

AI Agent Security Agent Skill Security Auditing AI Agent Security Benchmarks

Ask PDF

Feb 24, 2026

SoK: Agentic Skills -- Beyond Tool Use in LLM Agents

Agentic systems increasingly rely on reusable procedural capabilities, a.k.a., agentic skills, to execute long-horizon workflows reliably. These capabilities are callable modules that package procedural knowledge with explicit applicability conditions, execution policies, termination criteria, and reusable interfaces. Unlike one-off plans or atomic tool calls, skills operate (and often do well) across tasks. This paper maps the skill layer across the full lifecycle (discovery, practice, distillation, storage, composition, evaluation, and update) and introduces two complementary taxonomies. The first is a system-level set of seven design patterns capturing how skills are packaged and executed in practice, from metadata-driven progressive disclosure and executable code skills to self-evolving libraries and marketplace distribution. The second is an orthogonal \textbf{representation

\times

scope} taxonomy describing what skills are (natural language, code, policy, hybrid) and what environments they operate over (web, OS, software engineering, robotics). We analyze the security and governance implications of skill-based agents, covering supply-chain risks, prompt injection via skill payloads, and trust-tiered execution, grounded by a case study of the ClawHavoc campaign in which nearly 1{,}200 malicious skills infiltrated a major agent marketplace, exfiltrating API keys, cryptocurrency wallets, and browser credentials at scale. We further survey deterministic evaluation approaches, anchored by recent benchmark evidence that curated skills can substantially improve agent success rates while self-generated skills may degrade them. We conclude with open challenges toward robust, verifiable, and certifiable skills for real-world autonomous agents.

Yanna Jiang, Delong Li, Haiyu Deng +4

University of Technology Sydney · CSIRO Data61

Agentic AI LLM Agent Skill Learning LLM Agent Security

Ask PDF

Abstract

Agent Skill Security Auditing

Explore similar work

HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?

SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills

SoK: Agentic Skills -- Beyond Tool Use in LLM Agents