SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills

Yinghan Hou; Zongyou Yang

cs.CRApr 8, 2026

SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills

Authors: Yinghan Hou, Zongyou Yang

Organizations: Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · Dyson School of Design Engineering, Imperial College London, London, United Kingdom

Abstract

Agent skills combine natural-language instructions with executable code while inheriting an agent's filesystem, credential, and network access. Attacks can span prose and files, whereas regex and code-only analyzers cover only one modality. SkillSieve applies three progressively deeper layers: recall-oriented regex, AST, and metadata triage; four parallel LLM security sub-tasks; and an independent three-model jury with debate on disagreement. We evaluate 49,592 real ClawHub skills, a 390-skill labeled benchmark, and 100 adversarial samples across five evasion techniques on a 440 USD ARM board. The full pipeline achieves F1 = 0.929 (precision 0.912, recall 0.945) at an average cost of $0.006 per skill. An optional XGBoost fast path reduces Layer-2/3 calls by 32% with a 1.7-point F1 decrease while preserving recall. On 52 Feishu/Lark packages, Layer 2 reclassifies 13 of 14 Layer-1 flags as safe after contextual analysis; we also deploy the system as a Feishu chat bot. Code, labels, and aggregate results are open-sourced.

Figures & tables

Figure 1: The SkillSieve three-layer triage architecture. Layer 1 filters

\sim

86% of skills via static analysis at zero cost. Layer 2 applies four parallel LLM sub-tasks to suspicious skills. Layer 3 convenes a multi-LLM jury for high-risk cases.

Method	P	R	F 1	F 2	FPR
ClawVet Shaikh (2026)	0.162	0.527	0.248	0.363	0.448
SkillScan Payne (2026)	0.152	1.000	0.264	0.473	0.916
SkillSieve L1	0.342	1.000	0.509	0.722	0.316
+ SSD	0.663	1.000	0.797	0.908	0.084
+ SSD + Jury	0.912	0.945	0.929	0.939	0.015

Table 1: End-to-end detection on the 390-skill labeled benchmark (55 malicious, 335 benign). Metrics are recomputed from the archived per-sample predictions after the final label audit. The L1+L2 row has TP=55, FP=28, TN=307, FN=0. Layer 3 applies a contested-as-malicious policy (ties route to human review while holding the skill as malicious).

Figure 2: Effect of Layer 1 threshold

\tau

on precision, recall, F 1 , and F 2 (L1 only). We choose

\tau=0.3

, the highest round-number threshold within the recall-1.0 plateau (the lowest malicious score in the benchmark is 0.35). At

\tau=0.3

, recall is 1.000, F

{}_{1}=0.509

, and F

{}_{2}=0.722

; raising

\tau

past the plateau edge improves F 1 modestly but drops recall below 1.0, permanently losing malicious skills that Layer 2 cannot recover.

Technique	L1 Score	Caught by	L1 Rule
Encoding	0.35	L1+L2	obfuscation
Cross-file	0.40	L1+L2	credential_theft
Conditional	0.70	L1	conditional_trigger
Homoglyph	0.80	L1+L2	prompt_injection
Time-delay	0.70	L1	time_delay

Table 2: Per-layer interception analysis on five bypass techniques. Each technique combines its evasion method with a credential theft payload.

Metric	CVet	SFort	VT	Ours
Avg local triage latency/skill	$\sim$ 1 s	$\sim$ 5 s	$\sim$ 3 s	$\sim$ 40 ms (Layer 1)
Estimated corpus-average API cost/skill	$0	$0	$\sim$ $0.01	$\sim$ $0.006 † (full cascade)
GPU required	No	No	No	Optional

Table 3: Efficiency comparison. CVet = ClawVet, SFort = SkillFortify, VT = VirusTotal Code Insight.

Metric	Value
Total skills scanned	49,592
Total scan time	1,863 s (31.0 min)
Avg latency / skill	37.6 ms
P95 latency / skill	126.6 ms
Skills flagged suspicious	6,871 (13.86%)
Errors (unparseable)	1,623 (3.27%)

Table 4: Layer 1 performance on Orange Pi AIpro (ARM64, 4-core, 24 GB RAM) scanning 49,592 real ClawHub skills.

Outcome	Count	Share
Unanimous Round 1 (no debate)	76	91.6%
Debate triggered (Round 2)	7	8.4%
Unanimous after debate	3
Majority vote	0
Contested (escalated to human)	4	4.8%

Table 5: Layer 3 jury dynamics across all 83 L3-eligible skills in the main benchmark. Jury: GLM-5.1, Qwen3-235B, DeepSeek-V3.1.

Variant	P	R	F 1	FPR	L2/L3 calls
Full pipeline	0.912	0.945	0.929	0.015	161
+ fast-path	0.881	0.945	0.912	0.021	109

Table 6: Full pipeline vs. cost-efficient fast-path variant (

\tau_{\text{high}}=0.98

, OOF-calibrated) on the 390-skill benchmark. “L2/L3 calls” counts skills that reach Layers 2 or 3; fast-accepted skills skip both.

AI Agent Security

Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.

See all 491 papers

Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem
Mar 17, 2026Florian Holzbauer, David Schmidt, Gabriel Gegenhuber +2AI Agent Security Agent Skill Security Auditing
Interdisciplinary Transformation University (IT:U), Linz, Austria · University of Vienna, CDL AsTra, Faculty of Computer Science, Vienna, Austria
Pretext: Defeating Malicious Skill Detection Frameworks for AI Agents
Sep 30, 2026Tobias Kaiser, Aritra DharAI Agent Security Agent Skill Security Auditing
Computing System Labs, Huawei Research Zurich
SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models
Oct 13, 2026Jerrold Huang, Junwei Quan, Saiyue Lyu +2Agent Skill Security Auditing AI Agent Security Benchmarks
University of Toronto · University of British Columbia · Rutgers University

Mar 17, 2026J/K move · Enter open · S save

Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem

Florian Holzbauer, David Schmidt, Gabriel Gegenhuber, Sebastian Schrittwieser+1

Interdisciplinary Transformation University (IT:U), Linz, Austria · University of Vienna, CDL AsTra, Faculty of Computer Science, Vienna, Austria

AI Agent Security Agent Skill Security Auditing Software Supply Chain Security

Open paper AskPDF arXiv

Agent skills extend local AI agents, such as Claude Code and OpenClaw, with additional functionality. Their growing popularity has led to dedicated marketplaces resembling mobile app stores, as well as automated scanners that assess whether skills are benign or malicious. However, scanner reports from individual marketplaces classify up to 46.8% of skills as malicious, raising concerns about false positives. We present the largest empirical security analysis of the AI agent skill ecosystem to date. We collect 238,180 unique skills from three major distribution platforms and GitHub, and analyze their contents, behavior, and repository context. Unlike existing scanner-based assessments, which evaluate skills largely in isolation, our repository-aware analysis checks whether a flagged skill is consistent with its surrounding GitHub project. This context substantially reduces the number of suspicious skills: only 0.52% remain suspicious after repository-aware analysis. Our results show that existing scanners can substantially overestimate maliciousness when repository context is ignored. At the same time, we identify previously undocumented real-world attack vectors, including the hijacking of skills hosted in abandoned GitHub repositories. Overall, our findings provide a more robust view of the agent-skill ecosystem's current risk surface and highlight the need for context-aware security evaluation.

Abstract

AI Agent Security

Explore similar work

Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem

Pretext: Defeating Malicious Skill Detection Frameworks for AI Agents

SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models