When Skills Tell Agents Not to Ask: A Large-Scale Study of Oversight Directives in Agent Skills

Timofey Sanko; Yuan Tian

cs.CROct 27, 2026

When Skills Tell Agents Not to Ask: A Large-Scale Study of Oversight Directives in Agent Skills

Authors: Timofey Sanko, Yuan Tian

Organizations: School of Computing, Queen’s University, Kingston, Ontario, Canada

Abstract

Agent skills, reusable instruction packages centred on a SKILL.md file, extend AI agents on demand, and a single line in a widely copied skill reaches every agent that loads it. Some lines are oversight-loosening directives: they tell the agent to proceed without approval, to bypass checks with --no-verify, or to keep working without checking in. Although such directives can remove a checkpoint from every agent that loads them, no prior work has examined them. We present the first large-scale study of oversight-loosening directives, analyzing 1.53 million English skills from GitSkills. We split skills into units (sentences, bullets, table cells, and code lines), develop a codebook that labels oversight directives by polarity, mechanism, and scope, label 2,097 units by hand, and train a detector to scale the analysis. Our results show that about one in seventeen skills (about 88,000 skills) contains a loosening directive. These directives most often loosen oversight through permission flags, and most of them (70.1%) apply unconditionally rather than only in specific contexts such as CI. Compared with skills that have the same amount of oversight language, skills that run shell code, install packages, or use sudo have 1.7-1.8 times the odds of containing a loosening directive, an association that holds across all our robustness checks, and they do not compensate with more tightening. Loosening spreads through a few upstream skills rather than through edits by their adopters: 2.4% of the skill families that carry loosening account for half of all copies of loosening-carrying skills, and 1.1% of distinct loosening units for half of all loosening occurrences. Open-source skill scanners catch at most 46% of loosening directives, with a precision below 0.4, while the LLM behind them, prompted with our codebook instead, catches 61% at a precision of 0.71.

Figures & tables

Figure 2. Study pipeline. Green: Codebook Development. Blue: Human Labels. Each labelled set is marked with the codebook version and coding it received: joint = coded by the first author together with an LLM, discussing ambiguous units; 2 $\times$ = coded independently by the first author and an LLM, with every disagreement adjudicated by the first author; reviewed = proposed by an LLM and reviewed by the first author. Pilot and calibration units were later relabelled with v2. In the RQ1 column, $\circ$ marks the census, which is used only for copy-weighted estimates.

Mechanism	Loosening example (paraphrased)	Share of L	Tightening example (paraphrased)	Share of T
PERM (permission flags)	Run npx -y … / pass --yes	33.9%	Never use --force	2.4%
CONF (confirmation)	Proceed without asking the user	27.4%	Wait for explicit approval before deploying	46.3%
PERSIST (stopping)	Don’t stop between tasks. Keep going until done	16.0%	Stop and ask when blocked	1.0%
SAFE (safeguards)	Commit with --no-verify	7.1%	Never skip the test suite	13.8%
AUTO (auto-action)	Auto-commit completed units	6.2%	Never auto-merge	4.1%
TRANS (transparency)	Do not announce each commit	4.5%	Never apply a change silently, report it	11.3%

Table 1. Oversight directive mechanisms with examples.

Round	n	Label	Polarity	Mechanism	Scope †
Round 1 (v1)	400	0.67	0.86	0.80	0.17
Round 2 (v2)	200	0.80	0.97	0.87	0.75
Gold set (v2)	400	0.82	0.92	0.84	0.85

Table 2. Inter-rater reliability (Cohen’s

\kappa

). Label: neither / loosening / tightening, over all units; polarity and mechanism: over units both coders labelled as directives; scope: over units both labelled as loosening.

Method	Acc.	$\kappa$	Loos. F1	Cost
RoBERTa (fine-tuned)	0.74	0.55	0.52	0
Jev 1.13	0.74	0.59	0.62	0.06
GPT-5.4-mini	0.73	0.54	0.56	0.36
GPT-5.4	0.78	0.62	0.65	0.7–1.1
RoBERTa + Jev (ours)	0.80	0.65	0.65	0.06

Table 3. Detectors on the gold units. LLMs are prompted with our codebook. Cost is US$ per 1,000 units.

Measure	Human	Detector
Oversight directive	49.7% [46.8, 52.6]	49.6%
Loosening	13.3% [11.7, 15.0]	10.3%
… of which unscoped	70.1% [63.9, 75.8]	n/a ∗
Tightening	36.4% [33.7, 39.2]	39.3%
Mention only	11.6% [10.1, 13.1]	n/a ∗

Table 4. Prevalence among distinct candidate units of the repository-balanced frame (370,285 units). Human: prevalence sample, 95% CIs. Detector: all units, no sampling error.

Capability Features	% L	Loosening OR	Tight. OR
Shell code block	24.8 / 11.1	1.82 [1.73, 1.92]	0.95 †
Package install	29.7 / 14.7	1.82 [1.69, 1.95]	0.59
sudo	38.8 / 16.7	1.74 [1.61, 1.89]	0.75
curl $\|$ sh	40.6 / 16.9	1.31 [1.18, 1.45]	0.77
Network calls	26.7 / 15.8	1.19 [1.13, 1.25]	0.88
Bundled executables	21.7 / 16.5	1.12 [1.00, 1.27]	1.01 †

Table 5. Odds ratios for

\geq

1 loosening (M2) and

\geq

1 tightening (M3) directive.

n

= 512,876 skills; 95% CIs clustered by repository; controls for log length and log number of candidate units. Detector reference (§ 3.5 ).

Figure 3. Loosening is concentrated in a few upstream skills. (a) Cumulative share of loosening-carrying skill occurrences held by the top- $k$ families, against the share of all skill occurrences held by the $k$ largest families. (b) Cumulative share of loosening-unit occurrences held by the top- $k$ distinct loosening units.

Method	Prec.	Recall [95% CI]	F1	%T
Keyword lexicon	0.20	0.82 [0.71, 0.90]	0.33	24%
Cisco offline	0.14	0.02 [0.00, 0.09]	0.03	29%
Cisco + LLM	0.39	0.15 [0.08, 0.26]	0.21	13%
SkillSpector offline	0.21	0.21 [0.13, 0.33]	0.21	29%
SkillSpector + LLM	0.27	0.46 [0.34, 0.58]	0.34	23%
Either + LLM	0.28	0.51 [0.39, 0.63]	0.36	23%

Table 6. Flagging loosening directives on the 400 gold units (61 loosening). Scanner flag: a medium or higher finding on the unit’s line or quoting it. LLM: GPT-5.4. %T: share of flags on tightening directives.

Agent Skill Security Auditing

Agent skill security auditing examines reusable agent instructions, code, and tools for malicious behavior, unsafe permissions, and supply-chain risks.

See all 69 papers

Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem
Mar 17, 2026Florian Holzbauer, David Schmidt, Gabriel Gegenhuber +2AI Agent Security Agent Skill Security Auditing
Interdisciplinary Transformation University (IT:U), Linz, Austria · University of Vienna, CDL AsTra, Faculty of Computer Science, Vienna, Austria
What Keeps Agent Skills from Being Reusable? Evidence from 138K SKILL.md Files
Aug 9, 2026Chi Zhang, Yimin Liu, Xinze Chen +1LLM Agent Skill Retrieval
The Graduate Center, City University of New York · The Ohio State University · Hunter College, City University of New York
Toward User Comprehension Supports for LLM Agent Skill Specifications
May 19, 2026Zikai Alex WenLLM Agent Evaluation LLM Agent Skill Evaluation
University of Washington, Tacoma School of Engineering & Technology

Mar 17, 2026J/K move · Enter open · S save

Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem

Florian Holzbauer, David Schmidt, Gabriel Gegenhuber, Sebastian Schrittwieser+1

Interdisciplinary Transformation University (IT:U), Linz, Austria · University of Vienna, CDL AsTra, Faculty of Computer Science, Vienna, Austria

AI Agent Security Agent Skill Security Auditing Software Supply Chain Security

Open paper AskPDF arXiv

Agent skills extend local AI agents, such as Claude Code and OpenClaw, with additional functionality. Their growing popularity has led to dedicated marketplaces resembling mobile app stores, as well as automated scanners that assess whether skills are benign or malicious. However, scanner reports from individual marketplaces classify up to 46.8% of skills as malicious, raising concerns about false positives. We present the largest empirical security analysis of the AI agent skill ecosystem to date. We collect 238,180 unique skills from three major distribution platforms and GitHub, and analyze their contents, behavior, and repository context. Unlike existing scanner-based assessments, which evaluate skills largely in isolation, our repository-aware analysis checks whether a flagged skill is consistent with its surrounding GitHub project. This context substantially reduces the number of suspicious skills: only 0.52% remain suspicious after repository-aware analysis. Our results show that existing scanners can substantially overestimate maliciousness when repository context is ignored. At the same time, we identify previously undocumented real-world attack vectors, including the hijacking of skills hosted in abandoned GitHub repositories. Overall, our findings provide a more robust view of the agent-skill ecosystem's current risk surface and highlight the need for context-aware security evaluation.

Abstract

Agent Skill Security Auditing

Explore similar work

Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem

What Keeps Agent Skills from Being Reusable? Evidence from 138K SKILL.md Files

Toward User Comprehension Supports for LLM Agent Skill Specifications