When Skills Tell Agents Not to Ask: A Large-Scale Study of Oversight Directives in Agent Skills

Timofey Sanko; Yuan Tian

cs.CROct 27, 2026

When Skills Tell Agents Not to Ask: A Large-Scale Study of Oversight Directives in Agent Skills

Authors: Timofey Sanko, Yuan Tian

Organizations: School of Computing, Queen’s University, Kingston, Ontario, Canada

Abstract

Agent skills, reusable instruction packages centred on a SKILL.md file, extend AI agents on demand, and a single line in a widely copied skill reaches every agent that loads it. Some lines are oversight-loosening directives: they tell the agent to proceed without approval, to bypass checks with --no-verify, or to keep working without checking in. Although such directives can remove a checkpoint from every agent that loads them, no prior work has examined them. We present the first large-scale study of oversight-loosening directives, analyzing 1.53 million English skills from GitSkills. We split skills into units (sentences, bullets, table cells, and code lines), develop a codebook that labels oversight directives by polarity, mechanism, and scope, label 2,097 units by hand, and train a detector to scale the analysis. Our results show that about one in seventeen skills (about 88,000 skills) contains a loosening directive. These directives most often loosen oversight through permission flags, and most of them (70.1%) apply unconditionally rather than only in specific contexts such as CI. Compared with skills that have the same amount of oversight language, skills that run shell code, install packages, or use sudo have 1.7-1.8 times the odds of containing a loosening directive, an association that holds across all our robustness checks, and they do not compensate with more tightening. Loosening spreads through a few upstream skills rather than through edits by their adopters: 2.4% of the skill families that carry loosening account for half of all copies of loosening-carrying skills, and 1.1% of distinct loosening units for half of all loosening occurrences. Open-source skill scanners catch at most 46% of loosening directives, with a precision below 0.4, while the LLM behind them, prompted with our codebook instead, catches 61% at a precision of 0.71.

Figures & tables

Figure 2. Study pipeline. Green: Codebook Development. Blue: Human Labels. Each labelled set is marked with the codebook version and coding it received: joint = coded by the first author together with an LLM, discussing ambiguous units; 2 $\times$ = coded independently by the first author and an LLM, with every disagreement adjudicated by the first author; reviewed = proposed by an LLM and reviewed by the first author. Pilot and calibration units were later relabelled with v2. In the RQ1 column, $\circ$ marks the census, which is used only for copy-weighted estimates.

Mechanism	Loosening example (paraphrased)	Share of L	Tightening example (paraphrased)	Share of T
PERM (permission flags)	Run npx -y … / pass --yes	33.9%	Never use --force	2.4%
CONF (confirmation)	Proceed without asking the user	27.4%	Wait for explicit approval before deploying	46.3%
PERSIST (stopping)	Don’t stop between tasks. Keep going until done	16.0%	Stop and ask when blocked	1.0%
SAFE (safeguards)	Commit with --no-verify	7.1%	Never skip the test suite	13.8%
AUTO (auto-action)	Auto-commit completed units	6.2%	Never auto-merge	4.1%
TRANS (transparency)	Do not announce each commit	4.5%	Never apply a change silently, report it	11.3%

Table 1. Oversight directive mechanisms with examples.

Round	n	Label	Polarity	Mechanism	Scope †
Round 1 (v1)	400	0.67	0.86	0.80	0.17
Round 2 (v2)	200	0.80	0.97	0.87	0.75
Gold set (v2)	400	0.82	0.92	0.84	0.85

Table 2. Inter-rater reliability (Cohen’s

\kappa

). Label: neither / loosening / tightening, over all units; polarity and mechanism: over units both coders labelled as directives; scope: over units both labelled as loosening.

Method	Acc.	$\kappa$	Loos. F1	Cost
RoBERTa (fine-tuned)	0.74	0.55	0.52	0
Jev 1.13	0.74	0.59	0.62	0.06
GPT-5.4-mini	0.73	0.54	0.56	0.36
GPT-5.4	0.78	0.62	0.65	0.7–1.1
RoBERTa + Jev (ours)	0.80	0.65	0.65	0.06

Table 3. Detectors on the gold units. LLMs are prompted with our codebook. Cost is US$ per 1,000 units.

Measure	Human	Detector
Oversight directive	49.7% [46.8, 52.6]	49.6%
Loosening	13.3% [11.7, 15.0]	10.3%
… of which unscoped	70.1% [63.9, 75.8]	n/a ∗
Tightening	36.4% [33.7, 39.2]	39.3%
Mention only	11.6% [10.1, 13.1]	n/a ∗

Table 4. Prevalence among distinct candidate units of the repository-balanced frame (370,285 units). Human: prevalence sample, 95% CIs. Detector: all units, no sampling error.

Capability Features	% L	Loosening OR	Tight. OR
Shell code block	24.8 / 11.1	1.82 [1.73, 1.92]	0.95 †
Package install	29.7 / 14.7	1.82 [1.69, 1.95]	0.59
sudo	38.8 / 16.7	1.74 [1.61, 1.89]	0.75
curl $\|$ sh	40.6 / 16.9	1.31 [1.18, 1.45]	0.77
Network calls	26.7 / 15.8	1.19 [1.13, 1.25]	0.88
Bundled executables	21.7 / 16.5	1.12 [1.00, 1.27]	1.01 †

Table 5. Odds ratios for

\geq

1 loosening (M2) and

\geq

1 tightening (M3) directive.

n

= 512,876 skills; 95% CIs clustered by repository; controls for log length and log number of candidate units. Detector reference (§ 3.5 ).

Figure 3. Loosening is concentrated in a few upstream skills. (a) Cumulative share of loosening-carrying skill occurrences held by the top- $k$ families, against the share of all skill occurrences held by the $k$ largest families. (b) Cumulative share of loosening-unit occurrences held by the top- $k$ distinct loosening units.

Method	Prec.	Recall [95% CI]	F1	%T
Keyword lexicon	0.20	0.82 [0.71, 0.90]	0.33	24%
Cisco offline	0.14	0.02 [0.00, 0.09]	0.03	29%
Cisco + LLM	0.39	0.15 [0.08, 0.26]	0.21	13%
SkillSpector offline	0.21	0.21 [0.13, 0.33]	0.21	29%
SkillSpector + LLM	0.27	0.46 [0.34, 0.58]	0.34	23%
Either + LLM	0.28	0.51 [0.39, 0.63]	0.36	23%

Table 6. Flagging loosening directives on the 400 gold units (61 loosening). Scanner flag: a medium or higher finding on the unit’s line or quoting it. LLM: GPT-5.4. %T: share of flags on tightening directives.

Agent Skill Security Auditing

Agent skill security auditing examines reusable agent instructions, code, and tools for malicious behavior, unsafe permissions, and supply-chain risks.

See all 69 papers

Mar 17, 2026

Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem

Agent skills extend local AI agents, such as Claude Code and OpenClaw, with additional functionality. Their growing popularity has led to dedicated marketplaces resembling mobile app stores, as well as automated scanners that assess whether skills are benign or malicious. However, scanner reports from individual marketplaces classify up to 46.8% of skills as malicious, raising concerns about false positives. We present the largest empirical security analysis of the AI agent skill ecosystem to date. We collect 238,180 unique skills from three major distribution platforms and GitHub, and analyze their contents, behavior, and repository context. Unlike existing scanner-based assessments, which evaluate skills largely in isolation, our repository-aware analysis checks whether a flagged skill is consistent with its surrounding GitHub project. This context substantially reduces the number of suspicious skills: only 0.52% remain suspicious after repository-aware analysis. Our results show that existing scanners can substantially overestimate maliciousness when repository context is ignored. At the same time, we identify previously undocumented real-world attack vectors, including the hijacking of skills hosted in abandoned GitHub repositories. Overall, our findings provide a more robust view of the agent-skill ecosystem's current risk surface and highlight the need for context-aware security evaluation.

Florian Holzbauer, David Schmidt, Gabriel Gegenhuber +2

Interdisciplinary Transformation University (IT:U), Linz, Austria · University of Vienna, CDL AsTra, Faculty of Computer Science, Vienna, Austria

AI Agent Security Agent Skill Security Auditing Software Supply Chain Security

Ask PDF

Aug 9, 2026

What Keeps Agent Skills from Being Reusable? Evidence from 138K SKILL.md Files

Under the current standard, Agent Skills are SKILL.md files that combine instructions with supporting resources, enabling Large Language Model (LLM) agents to reuse procedures beyond a single conversation. Yet many public skills appear to originate from a single task, repository, or conversation, even when they are shared as reusable components. We analyze this gap across 138,133 public SKILL.md files from 20,556 repositories using a two-tier defect taxonomy grounded in the official specification and best-practice guidance. We find that 91.8% of skills contain at least one detected defect, with stable estimates across lenient and strict thresholds (88.8-94.6%). The dominant failures are ordinary packaging problems rather than exotic attacks: weak routing metadata, bloated or non-actionable bodies, and poor resource organization. A deterministic routing stress test over 20,000 skills shows the functional impact: skills with valid routing metadata are retrieved more reliably from startup descriptions than skills with routing defects. Defect rates vary by platform and provenance: specification-aware skills contain fewer defects, while AI-marked skills show more safety and portability problems. Lightweight enforcement and repair experiments support a quality-assured generation workflow combining spec-aware prompting, lightweight linting, automated repair, and safety gating.

Chi Zhang, Yimin Liu, Xinze Chen +1

The Graduate Center, City University of New York · The Ohio State University · Hunter College, City University of New York

LLM Agent Skill Retrieval

Ask PDF

May 19, 2026

Toward User Comprehension Supports for LLM Agent Skill Specifications

Users often interpret and select agent skills through their SKILL markdown specifications. To protect users, existing audits mainly focus on malicious or unsafe skills. We study the complementary question of whether specifications help users form bounded expectations about what a skill consumes, produces, and covers. Across 878 cybersecurity skills, we used rule-based coding to measure textual cues for four comprehension anchors, namely operational basis, output contract, boundary disclosure, and example capability demonstration. Cues for operational basis were common, but only 19.0% of specifications exhibited cues for an example task, sample, or expected outcome, and only 2.3% exhibited cues for all four anchors. We further examined a small DNS/C2 telemetry subset (n

=

6) to illustrate why missing examples may matter. Examples appeared to make first local checks easier to construct, while no-example skills typically required helper code inspection to recover command arguments or output fields. We argue that agent-skill evaluation should treat specifications as user-facing capability disclosures, not merely as containers for executable instructions.

Zikai Alex Wen

University of Washington, Tacoma School of Engineering & Technology

LLM Agent Evaluation LLM Agent Skill Evaluation

Ask PDF

Abstract

Agent Skill Security Auditing

Explore similar work

Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem

What Keeps Agent Skills from Being Reusable? Evidence from 138K SKILL.md Files

Toward User Comprehension Supports for LLM Agent Skill Specifications