SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models

Jerrold Huang; Junwei Quan; Saiyue Lyu; Zihan Guo; Yingqiang Ge

cs.CROct 13, 2026

SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models

Authors: Jerrold Huang, Junwei Quan, Saiyue Lyu, Zihan Guo, Yingqiang Ge

Organizations: University of Toronto · University of British Columbia · Rutgers University

Abstract

Agent skills turn natural-language instructions into actions, creating a security challenge: legitimate automation and malicious behaviour can look deceptively similar. A practical screen must catch nearly every attack while keeping the review load manageable. We present SkillScreen, a compact decision model trained to score skill maliciousness directly from SKILL $.$ md in a single forward pass. On MaliciousSkillBench, SkillScreen flags 20-25% of benign skills at 99% recall on unseen attack families, substantially fewer than the word-level classifier and prompted LLM judges tested. Under the same training protocol, it also generalizes better than a fully fine-tuned encoder to unseen sources. Yet unfamiliar benign sources sharply increase false positives, showing that recognizing new attacks does not ensure reliable screening of new skill collections. Our results demonstrate the value of task-trained decision models for high-recall skill screening. Project page: https://jerroldh.github.io/SkillScreen-webpage/

Figures & tables

Figure 1: SkillScreen specializes a small language model for false-positive control at high recall. Training and inference use the same skill, hypothesis and model. Cross-entropy supervises NLI outputs with skill labels. Dashed teal arrows mark updates to LoRA adapters and the head. Validation data calibrate scores and select a threshold for 99% recall. Model, calibration and threshold are frozen before test scoring. Layers and score axis are schematic.

Off-the-shelf detectors
	MSD (Unseen attack structures)				SD (Unseen sources)
	Default threshold			99% recall	Default threshold			99% recall
Method	Recall $\uparrow$	FPR $\downarrow$	AUROC $\uparrow$	FPR $\downarrow$	Recall $\uparrow$	FPR $\downarrow$	AUROC $\uparrow$	FPR $\downarrow$
Untuned OpenJev-2B	4.2	0.0	0.678	95.5	0.6	0.0	0.568	99.8
TypeSafe Jev 1.13	33.8	0.2	0.841	100.0	11.3	0.0	0.652	100.0
SkillSpector	10.5	0.7	0.618	100.0	1.3	1.8	0.433	100.0

Table 1: Malicious skill detection on MSD and SD. Recall and FPR are percentages. Shaded columns give FPR at 99% test recall. SkillScreen and ModernBERT report five-seed means with subscript standard deviations, using converged SD encoder runs. Bold and underlining mark the best and second-best AUROC and high-recall FPR.

Figure 2: Source-level error patterns underlying the MSD results. At the default thresholds, panel (a) shows the malicious miss rate and panel (b) the benign FPR; darker cells indicate more errors. The adjacent $n$ gives the number of records in the corresponding class. SkillScreen values are averaged over five seeds. Blank grey cells mark sources with fewer than five records in that class. Sources are ordered by the miss rate of SkillScreen.

Variant	Recall	FPR	AUROC	FPR@99
SkillScreen (5 seeds)	95.6 $\pm$ 0.6	3.3 $\pm$ 1.3	0.992	22.6
Plain base (5 seeds)	96.9 $\pm$ 0.8	4.1 $\pm$ 1.6	0.992	21.1
Paraphrased hypothesis	95.1	1.3	0.993	21.0
Unrelated hypothesis (5 seeds)	95.7 $\pm$ 0.8	3.3 $\pm$ 1.2	0.990	28.9
Empty hypothesis	95.1	2.0	0.988	44.7
0.8B base (2 seeds)	95.4 $\pm$ 0.0	2.9 $\pm$ 0.9	0.991	26.0

Table 2: Ablations on MSD. Recall and FPR are measured at probability 0.5; the last column reports the benign FPR at 99% test recall from the threshold sweep of each run. All rates are seed means in percent. The plain base is Qwen3.5-2B with a randomly initialized head; the paraphrased and empty hypotheses use a single seed. Data, epoch budget and adapter settings are held fixed. Appendix K reports paired intervals relative to SkillScreen; at 99% recall, only the empty hypothesis is clearly worse (

+22.1

points,

[+2.0,+40.4]

Figure 3: With limited labels, the full recipe learns most: OpenJev with the task-naming hypothesis leads in AUROC at 10% of the labels and flags about half as many benign skills as either alternative at 25–50%. Panels show (a) AUROC, (b) recall and (c) FPR on MSD, with recall and FPR measured at probability 0.5. Each point at 10%, 25% and 50% averages nine runs (three stratified subset draws with three seeds each), and each full-data point averages five seeds. Shaded bands are 95% intervals of the mean from resampling the runs; hollow markers at 0% show each starting model before task training. All runs are trained for one epoch.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Protocol	Source	Train (benign / malicious)	Validation	Test
MSD	MalSkillBench	0 / 1,919	0 / 537	0 / 967
MSD	SkillTrustBench	1,134 / 2,348	171 / 109	338 / 324
MSD	SkillHarm	0 / 601	0 / 35	0 / 92
MSD	ATR Skill Security Benchmark	329 / 13	43 / 3	83 / 5
MSD	SkillFortifyBench	70 / 86	7 / 2	13 / 2
MSD	MaliciousAgentSkillsBench	0 / 58	0 / 34	0 / 61

Appendix

Table 3: Benign / malicious counts after the five missing-text exclusions; SD also omits the eight malicious MalSkillBench records that its official split excludes. SD test sources are absent from training. MSD shares sources and separates malicious structural families. Empty partitions are shown with an en dash. Sources: MalSkillBench ( Guo et al., 2026 ) , SkillTrustBench ( cuhk-zhuque, 2026 ) , SkillHarm ( Ning et al., 2026 ) , ATR Skill Security Benchmark ( Agent-Threat-Rule, 2026 ) , SkillFortifyBench ( Bhardwaj, 2026 ) , MaliciousAgentSkillsBench ( Liu et al., 2026 ) , SkillSafetyBench ( Jin et al., 2026 ) , AgentTrap ( Zhuang et al., 2026 ) , SkillLeakBench ( Chen et al., 2026 ) , Agent Skill Malware ( yoonholee, 2026 ) and SkillTrojan ( Feng et al., 2026 ) .

Figure 4: Threshold sweeps for MSD SkillScreen and context variants (a), SD SkillScreen (b), MSD zero-shot judges (c) and MSD Word-SVM (d). Blue and orange show recall and FPR in (a), (b) and (d). Judges use solid recall and dotted FPR. Thin SkillScreen lines show seeds and bold lines their mean. Probability thresholds use a logit scale. SVM thresholds use margins. Vertical lines mark default thresholds. Circles mark thresholds chosen on validation. Judge markers show 0.5, filled for recall and hollow for FPR.

Method	at 95.6% recall	at 99% recall
SkillScreen (5 seeds)	2.0–4.0	19.9–24.6
ModernBERT (5)	2.2–3.8	25.1–36.2
Word-SVM	13.6	45.6
GPT-5.5	34.7	91.1
GPT-5.5 + few-shot (8)	28.2	95.7
GPT-5.5 + medium reasoning	44.3	86.8

Appendix

Table 4: MSD FPR (%) at the highest observed test threshold reaching each recall target, read off the test score curves. SkillScreen and ModernBERT report seed ranges; Table 10 adds bootstrap intervals.

Method	AP
SkillScreen (5 seeds)	0.998
+ findings context	0.998
+ full context	0.998
ModernBERT-large (5 seeds)	0.997
Word-SVM	0.993
TypeSafe Jev 1.13	0.942

Appendix

Table 5: MSD average precision at 77% malicious prevalence. SkillScreen reports the five-seed mean with std below 0.001. Scanner AP uses 1,946 scored records. Other methods use all 1,947.

Input	Recall	FPR	AUROC	FPR 99
Skill text	95.6	3.3	0.992	22.6
+ findings	95.4	1.3	0.993	22.1
+ full context	95.1	2.5	0.992	20.6

Appendix

Table 6: Scanner-context variants on MSD, using the metrics of Table 1 . Skill text averages five seeds. Each context variant uses one seed.

Run	Seed	Val. FPR@99	Gate recall / FPR		FP	FN	AP	At 0.5 recall / FPR
MSD	42	23.7	98.93	21.70	97	16	0.9978	95.20	1.79
MSD	1	20.5	98.73	20.13	90	19	0.9976	96.47	5.15
MSD	2	21.4	99.00	21.25	95	15	0.9980	95.40	3.80
MSD	3	20.5	98.93	20.13	90	16	0.9977	95.07	3.13
MSD	4	27.7	99.13	23.94	107	13	0.9977	96.07	2.68
SD	42	8.3	98.93	55.60	303	9	0.9325	96.07	35.41

Appendix

Table 7: SkillScreen results by seed. Validation FPR is measured at 99% recall. Test columns give recall, FPR, FP and FN at the frozen threshold, AP from continuous scores, and recall/FPR at probability 0.5. The fifteen current-code runs selected the final checkpoint. The historical SD and SBR runs use earlier code.

Learning rate	Seeds	Val. FPR@99	Recall	FPR	AUROC	Gate recall / FPR	FPR@99	$\Delta$ vs SkillScreen @99
$2\times 10^{-5}$ (default)	5	27.7	96.0	4.1	0.990	98.9 / 30.2	30.4	$+7.8$ [ $-3.4$ , $+22.1$ ]
first set (CPU, float32)	4 of 5	31.8	96.1	4.8	0.990	99.1 / 34.7	33.9	$+11.3$ [ $-0.7$ , $+20.7$ ]
$10^{-5}$	3	36.0	96.1	10.0	0.985	99.0 / 39.7	37.2	$+14.6$ [ $+3.1$ , $+28.4$ ]
$5\times 10^{-5}$	3	34.1	96.0	3.0	0.991	99.3 / 36.5	29.4	$+6.8$ [ $-5.6$ , $+17.4$ ]
SkillScreen	5	22.8	95.6	3.3	0.992	98.9 / 21.4	22.6	–

Appendix

Table 8: ModernBERT-large learning rates on MSD, each trained for one epoch (seeds 42, 1 and 3 for the alternatives; the first default set excludes its failed seed 2). The main text reports the default learning rate. Rates are seed-mean percentages. Validation FPR@99 is measured at each run’s selected checkpoint. Recall and FPR use probability 0.5 on test. AUROC is computed from continuous test scores. The gate is frozen for 99% validation recall; FPR@99 is read off the test sweep with the paired difference against SkillScreen’s five seeds.

Split	Seed	Val. FPR@99	Gate recall / FPR		FP	FN	AP	At 0.5 recall / FPR
MSD	42	32.6	99.00	38.03	170	15	0.9970	95.47	2.24
MSD	1	25.4	98.80	25.06	112	18	0.9974	95.87	3.80
MSD	2	31.7	99.13	34.23	153	13	0.9972	95.93	3.80
MSD	3	21.4	98.47	25.28	113	23	0.9970	96.00	3.58
MSD	4	27.2	99.00	28.41	127	15	0.9973	96.67	7.16
MSD, CPU set	42	44.2	99.27	46.09	206	11	0.9971	95.60	4.47

Appendix

Table 9: ModernBERT-large with the default recipe by seed. The MSD CPU set is the first MSD set, trained in float32 on CPUs; every other run used bf16 on a GPU slice. Rates are percentages. Gate columns use thresholds frozen for 99% validation recall. The remaining columns give test AP and results at probability 0.5.

Figure 5: MSD recall–FPR trade-offs. Panel (a) shows the full range and (b) enlarges the high-recall region. SkillScreen and ModernBERT-large show five-seed means at each FPR, with min–max shading. Filled markers show default operating points. Open markers show test performance at thresholds frozen for 99% validation recall. Markers average seeds where applicable. The test set contains 1,500 malicious and 447 benign skills.

Figure 6: SD recall–FPR trade-offs, following the conventions of Figure 5 . Both panels retain the full FPR range. SkillScreen and ModernBERT-large summarize five seeds, using the converged runs for the encoder. GPT-5.5 is shown zero-shot and with eight SD training examples. The test set contains 839 malicious and 545 benign skills.

Figure 7: SD errors by source at default thresholds. Cells and class counts follow Figure 2 . SkillScreen averages five current-code seeds. Benign records occur in two sources.

Method	[email protected]%	FPR@99% [CI]	$\Delta$ vs SkillScreen @99% [CI]
SkillScreen (5)	3.2	22.6 [13.5, 37.0]	–
ModernBERT-large (5)	3.2	30.4 [21.2, 51.4]	$+7.8$ [ $-3.4$ , $+22.1$ ]
CPU set, 4 converged	2.8	33.9 [21.2, 46.9]	$+11.3$ [ $-0.7$ , $+20.7$ ]
CPU set, all 5	18.8	45.9 [35.8, 56.4]	$+23.3$ [ $+12.1$ , $+31.3$ ]
Word-SVM	13.6	45.6 [30.0, 66.2]	$+23.0$ [ $+6.8$ , $+39.8$ ]
Plain base (5)	2.6	21.1 [11.5, 39.5]	$-1.5$ [ $-11.9$ , $+8.0$ ]

Appendix

Table 10: Benign FPR (%) on the MSD test set at the highest threshold of each method that reaches the stated recall, with the threshold re-chosen in every bootstrap replicate; intervals are 95% percentile intervals of the seed mean, and differences are paired. On SD the five SkillScreen seeds need 27.4% [23.6, 32.4] at 95.6% recall and 50.5% [42.1, 59.9] at 99%, the five converged ModernBERT runs 56.0% and 74.7% [69.8, 79.7] (

+24.1

[

+14.4

+32.7

] at 99%; all six 78.2%,

+27.7

[

+18.1

+36.1

]), and the SVM 62.8% and 82.2%. On SBR SkillScreen needs 6.6% [2.4, 13.5] at 99% recall and the SVM 28.1%.

Split	Partition	Malicious	Families	Singletons	$\geq 5$
MSD	train	5,250	2,458	50%	129
MSD	test	1,500	1,364	90%	0
SBR	train	5,249	3,542	75%	61
SBR	test	1,501	1,249	88%	8
SD	train	5,989	3,992	72%	69
SD	test	839	304	71%	43

Appendix

Table 11: Malicious structural families per partition: number of records and families, share of families with one record, and families with at least five records. The largest MSD training family has 146 records; no MSD test family has more than two. On SD, 322 of 839 malicious test skills belong to a family seen in training, and the largest test family has 141 records.

Rewrite	Rule	Before	After	Drop per adapter
Paraphrase	0.5	96.6	96.0	0.5–1.0
Paraphrase	gate	99.8	99.6	0.0–0.5
Script move	0.5	96.6	67.5	24.5–35.0
Script move	gate	99.8	87.0	11.5–14.0

Appendix

Table 12: Recall (%) of the four MSD adapters on 200 malicious test skills before and after rewriting, averaged over adapters, with the range of per-adapter drops in points. Paired bootstrap intervals (2,000 replicates over records) exclude zero for every script-move drop.

Agent Skill Security Auditing

Agent skill security auditing examines reusable agent instructions, code, and tools for malicious behavior, unsafe permissions, and supply-chain risks.

See all 63 papers

SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills
Apr 8, 2026Yinghan Hou, Zongyou YangAI Agent Security Agent Skill Security Auditing
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · Dyson School of Design Engineering, Imperial College London, London, United Kingdom
MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection
Aug 20, 2026Yue Wang, Yi Liu, Gelei Deng +4AI Agent Security Benchmarks Agent Skill Security Auditing
State Key Laboratory for Novel Software Technology, Nanjing University · Griffith University · Nanyang Technological University +2
Pretext: Defeating Malicious Skill Detection Frameworks for AI Agents
Sep 30, 2026Tobias Kaiser, Aritra DharAI Agent Security Agent Skill Security Auditing
Computing System Labs, Huawei Research Zurich

Apr 8, 2026J/K move · Enter open · S save

SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills

Yinghan Hou, Zongyou Yang

Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · Dyson School of Design Engineering, Imperial College London, London, United Kingdom

AI Agent Security Agent Skill Security Auditing AI Agent Security Benchmarks

Open paper AskPDF arXiv

Agent skills combine natural-language instructions with executable code while inheriting an agent's filesystem, credential, and network access. Attacks can span prose and files, whereas regex and code-only analyzers cover only one modality. SkillSieve applies three progressively deeper layers: recall-oriented regex, AST, and metadata triage; four parallel LLM security sub-tasks; and an independent three-model jury with debate on disagreement. We evaluate 49,592 real ClawHub skills, a 390-skill labeled benchmark, and 100 adversarial samples across five evasion techniques on a 440 USD ARM board. The full pipeline achieves F1 = 0.929 (precision 0.912, recall 0.945) at an average cost of $0.006 per skill. An optional XGBoost fast path reduces Layer-2/3 calls by 32% with a 1.7-point F1 decrease while preserving recall. On 52 Feishu/Lark packages, Layer 2 reclassifies 13 of 14 Layer-1 flags as safe after contextual analysis; we also deploy the system as a Feishu chat bot. Code, labels, and aggregate results are open-sourced.

Abstract

Agent Skill Security Auditing

Explore similar work

SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills

MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

Pretext: Defeating Malicious Skill Detection Frameworks for AI Agents