Scan Before You Install: Detecting and Mitigating Local and Composite Vulnerabilities in Skills

Parth A. Gandhi; Avital Aviv; Prasanna N. Wudali; Yuval Elovici; Asaf Shabtai

cs.CROct 27, 2026

Scan Before You Install: Detecting and Mitigating Local and Composite Vulnerabilities in Skills

Authors: Parth A. Gandhi, Avital Aviv, Prasanna N. Wudali, Yuval Elovici, Asaf Shabtai

Organizations: Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Beer-Sheva, Israel

Abstract

An agent skill packages natural-language instructions, scripts, and reference files that an LLM-based agent loads for a task. Skills may access the file system, shell, and network under user-authorized permissions; weaknesses can lead to unsafe execution. Existing scanners primarily target malicious skills, not unintended vulnerabilities in public repositories, and analyze skills in isolation, overlooking multi-skill interactions. To address these gaps, we developed a six-module scanner: Content Extraction, Local Analysis, Composite Analysis, Validation, Reporting, and Mitigation. Local Analysis detects unintended vulnerabilities in benign skills. On a 556-case leaderboard subset of SkillTrustBench, it achieved 95.90% recall with a 4.22% false positive rate; on a balanced sample of 1,000 MalSkillBench skills, its F1 was 0.947, outperforming Cisco's Skill Scanner, NVIDIA's SkillSpector, AgentSkillsScanner, and SkillSieve. In 1,587 skills from the 75 most-starred GitHub repositories, three analysts confirmed 53 of 59 findings (89.8% precision) across 23 repositories. We developed proof-of-concept (PoC) attacks for each vulnerability, disclosed 10 medium- and high-severity cases to maintainers, and tested mitigations on the remaining cases. Composite Analysis models dependencies among skills through shared resources, invocations, and instruction-mediated interactions. On SCR-Bench, its F1 score was 0.991 (98.7% precision, 99.6% recall), versus 0.816-0.858 for three SkillsGuard configurations released with CompoSkill. It identified two composite findings on GitHub, validated by PoC attacks. The Mitigation module recommends skill substitution, patches, and configuration changes. For 47 of 55 GitHub findings, mitigations prevented PoC attacks in all trials while preserving tested legitimate functionality.

Figures & tables

Figure 1: Anatomy of a skill: its directory structure, the contents of SKILL.md , and staged loading by an agent.

Figure 2: A vulnerable skill contains an unintentional weakness, whereas a malicious skill deliberately embeds harmful behavior.

Figure 3: Prompt injection induces an agent to invoke a backup skill followed by a debugging skill, exposing credentials through a shared cache even though each skill has a legitimate purpose.

Figure 4: The six modules of the scanner: Content Extraction, Local Analysis, Composite Analysis, Validation, Reporting, and Mitigation.

Check	Reported	Confirmed	Precision
Value-flow check	17	14	82.4%
Pattern checks	2	2	100.0%
Instruction review	40	37	92.5%
Capability-declaration check	0	0	–
Installation-source check	–	–	–
Total	59	53	89.8%

Table 1: Local Analysis reported and confirmed findings.

Threat category	Rep.	Conf.	L	M	H
			Confirmed severity
Embedded credentials	0	0	0	0	0
Path traversal	7	5	5	0	0
Server-side request forgery	3	3	3	0	0
Data exfiltration	7	6	5	0	1
Log injection	0	0	0	0	0

Table 2: Reported and confirmed Local Analysis findings by threat category and severity.

Figure 5: Performance comparison of the Local Analysis module and existing scanners (precision, recall,

F_{1}

, and false-positive rate). The left graph reports the results for our scanner on the full SkillTrustBench benchmark. The middle graph compares the performance of all evaluated scanners on the 556-case leaderboard subset of SkillTrustBench benchmark, and the right graph compares the performance of the scanners on a balanced random samples of 1,000-skill sample from MalSkillBench benchmark. For each metric, the best value in the middle and right graphs are outlined in black.

Method	TP	FN	FP	TN	Precision (%)	Recall (%)	$F_{1}$
Our scanner	665	3	9	648	98.7	99.6	0.991
SG review	663	5	214	443	75.6	99.3	0.858
SG regex	551	117	71	586	88.6	82.5	0.854
SG high	499	169	56	601	89.9	74.7	0.816
LLM-only	657	11	362	295	64.5	98.4	0.779

Table 3: Composite detection on the adapted SCR-Bench evaluation set.

Config.	Precision (%)	Recall (%)	FPR (%)	$F_{1}$
D	$85.3$	$68.2$	$27.7$	$0.758$
L	$97.0\pm 0.4$	$94.2\pm 0.6$	$6.9\pm 1.0$	$0.956\pm 0.002$
D + L	$87.1\pm 0.4$	$96.5\pm 0.5$	$33.5\pm 1.3$	$0.916\pm 0.001$
D + V	$98.7\pm 0.2$	$67.8\pm 0.4$	$2.2\pm 0.3$	$0.804\pm 0.002$
L + V	$98.0\pm 0.3$	$95.8\pm 0.1$	$4.6\pm 0.7$	$0.969\pm 0.001$
D + L + V	$98.0\pm 0.4$	$96.0\pm 0.1$	$4.6\pm 0.9$	$0.970\pm 0.001$

Table 4: Local Analysis ablation study on the 556-case SkillTrustBench leaderboard subset. D: deterministic detection; L: LLM detection; V: package-level validation. Metrics are reported as mean

\pm

sample standard deviation across five runs; exactly zero standard deviations are omitted.

Config.	Precision (%)	Recall (%)	FPR (%)	$F_{1}$
$G_{0}$ + IL	$100.0$	$54.7$	$0.0$	$0.707$
$G_{0}$ + ME	$94.6\pm 2.3$	$98.9\pm 0.5$	$7.0\pm 3.0$	$0.967\pm 0.010$
$G_{0}$ + IL + ME	$94.6\pm 2.3$	$99.2\pm 0.2$	$7.0\pm 3.0$	$0.968\pm 0.012$
$G_{0}$ + IL + V	$100.0$	$54.2\pm 0.2$	$0.0$	$0.703\pm 0.001$
$G_{0}$ + ME + V	$98.0\pm 0.8$	$95.2\pm 1.5$	$2.4\pm 1.0$	$0.966\pm 0.011$
$G_{0}$ + IL + ME + V	$98.0\pm 0.8$	$95.4\pm 1.6$	$2.4\pm 1.0$	$0.967\pm 0.012$

Table 5: Composite Analysis ablation study on the 485 SCR-CapFlow and SCR-AuthBlur sets.

Mitigation	Findings
Substitution alone	13
Substitution with configuration	1
Executable-code patch alone	4
Executable-code patch with configuration	0
Instruction patch alone	17
Instruction patch with configuration	20

Table 6: Mitigations selected to address the 55 reported findings.

Outcome	Findings
Effective in the tested conditions	47
Partially mitigated	6
Unresolved after testing	0
Effectiveness not evaluated	2
Total	55

Table 7: Outcomes for the final selected mitigations across 55 findings.

Skill	Gate on Sending	Local Analysis Result
slackapi/slack-messaging *	host (MCP tool approval)	none reported
slackapi/slack-api *	instruction (confirm writes)	none reported
idanbeck/slack-skill	instruction (confirm each send)	undeclared network use
openclaw/slack-personal	optional draft review	none reported
devxoul/agent-slack	none	excessive agency
devxoul/agent-slackbot	none	excessive agency

Table 8: Slack-sending substitutes considered by the Mitigation module, followed by the original sender (host: Codex). All support sending agent-written text. Placeholder credentials in documentation examples are not counted as Local Analysis module findings. Unvalidated value-flow candidates are distinguished from confirmed findings.

Agent Skill Security Auditing

Agent skill security auditing examines reusable agent instructions, code, and tools for malicious behavior, unsafe permissions, and supply-chain risks.

See all 69 papers

Mar 17, 2026

Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem

Agent skills extend local AI agents, such as Claude Code and OpenClaw, with additional functionality. Their growing popularity has led to dedicated marketplaces resembling mobile app stores, as well as automated scanners that assess whether skills are benign or malicious. However, scanner reports from individual marketplaces classify up to 46.8% of skills as malicious, raising concerns about false positives. We present the largest empirical security analysis of the AI agent skill ecosystem to date. We collect 238,180 unique skills from three major distribution platforms and GitHub, and analyze their contents, behavior, and repository context. Unlike existing scanner-based assessments, which evaluate skills largely in isolation, our repository-aware analysis checks whether a flagged skill is consistent with its surrounding GitHub project. This context substantially reduces the number of suspicious skills: only 0.52% remain suspicious after repository-aware analysis. Our results show that existing scanners can substantially overestimate maliciousness when repository context is ignored. At the same time, we identify previously undocumented real-world attack vectors, including the hijacking of skills hosted in abandoned GitHub repositories. Overall, our findings provide a more robust view of the agent-skill ecosystem's current risk surface and highlight the need for context-aware security evaluation.

Florian Holzbauer, David Schmidt, Gabriel Gegenhuber +2

Interdisciplinary Transformation University (IT:U), Linz, Austria · University of Vienna, CDL AsTra, Faculty of Computer Science, Vienna, Austria

AI Agent Security Agent Skill Security Auditing Software Supply Chain Security

Ask PDF

Jun 14, 2026

SkillVetBench: LLM-as-Judge for Multi-Dimensional Security Risk Evaluation in Open-Source LLM Agent Skills

Open-source LLM agent ecosystems are growing rapidly, yet the security of community-contributed skills - modular tool definitions that extend agent capabilities - remains largely unvetted. The gap we fill: existing scanners operate at the code layer and are structurally blind to instruction-layer and multi-agent risk - natural-language directives that hijack an agent, exfiltrate data through encoded side channels, or chain harm across pipelines - so what is needed is a semantic, multi-dimensional vetting system rather than another signature matcher. We present SKILLVETBENCH, a live public leaderboard on Hugging Face that uses an LLM-as-Judge to vet agent skills. What is new: SARS (Skill Agentic Risk Score), a five-dimensional agentic-risk metric with a principled weighted formula for instruction-following systems. What is integrated: full CVSS v4.0 vector decomposition and a ClawHub dual-view that places our LLM-generated review beside the official marketplace verdict. What is demonstrated: drawing on our companion benchmark paper [ 1], the LLM-as-Judge stage achieves zero false negatives across 78 confirmed-malicious skills and zero false positives across 22 benign controls, while the best static baseline (SKILLSIEVE) still misses 15%; for instruction-layer categories such as Prompt Injection and Memory Poisoning, conventional tools miss between 89% and 100% of threats (e.g., CODEBERT detects none of nine memory-poisoning skills). Detection rates vary from 35% to 95% across four LLM evaluators, motivating ensemble scoring in production deployments.

Ismail Hossain, Sai Puppala, Md Jahangir Alam +2

SUPREME Lab, University of Texas at El Paso, Texas, USA.

LLM-as-a-Judge LLM Agent Security AI Agent Security Benchmarks LLM Agent Verification Agent Skill Security Auditing

Ask PDF

Sep 30, 2026

Pretext: Defeating Malicious Skill Detection Frameworks for AI Agents

Skills extend an agent's capabilities by injecting instructions and information into the context, and are widely used by agents such as OpenClaw and Claude Code. Prior work shows third-party marketplaces host malicious skills that give attackers direct influence over the victim's agent. The emerging defense scans skills before installation, pairing deterministic static checks with an LLM-based semantic judge, as in NVIDIA's SkillSpector. We show that such defenses fall to an attacker who knows the detector. Our white-box LLM attacker, Pretext, iteratively crafts skills that evade detection while still delivering the payload and performing the benign task: moving the payload from code into natural language leaves static analysis inert, while framing it as the skill's legitimate purpose and splitting instructions across files keeps the LLM stage below its blocking threshold. Across three open-source models, Pretext achieves up to 97% and 77% against a frozen detector and a co-adaptive one, respectively, revealing major gaps in current skill scanners.

Tobias Kaiser, Aritra Dhar

Computing System Labs, Huawei Research Zurich

AI Agent Security Agent Skill Security Auditing

Ask PDF

Abstract

Agent Skill Security Auditing

Explore similar work

Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem

SkillVetBench: LLM-as-Judge for Multi-Dimensional Security Risk Evaluation in Open-Source LLM Agent Skills

Pretext: Defeating Malicious Skill Detection Frameworks for AI Agents