MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

Yue Wang; Yi Liu; Gelei Deng; Ying Zhang; Yuekang Li; Zhenyu Chen; Leo Zhang

cs.CRAug 20, 2026

MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

Authors: Yue Wang, Yi Liu, Gelei Deng, Ying Zhang, Yuekang Li, Zhenyu Chen, Leo Zhang

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University · Griffith University · Nanyang Technological University · Wake Forest University ·

Abstract

Agent Skills extend LLM agents with reusable instruction packages that may also include scripts, resources, and service configuration. This creates a direct distribution channel for malicious behavior, yet existing malicious-Skill datasets are fragmented across sources, artifact formats, evidence regimes, and benign coverage; duplicated and structurally related content further complicates direct aggregation and evaluation. We present MaliciousSkillBench, a comprehensive benchmark for malicious Agent Skill detection. We consolidate 13 public sources, 11 of which contribute Core malicious artifacts, and reduce 8,414 raw malicious records to 7,539 normalized-unique identities in 4,588 operational structural families. After conservative cross-label conflict exclusion, the primary benchmark contains 9,740 Skills: 7,505 malicious and 2,235 benign. To characterize its coverage, we harmonize 11 attack categories for 4,983 malicious identities with supported source-native mappings and find substantial differences in threat composition across sources. We then evaluate three learned text detectors and three off-the-shelf Skill scanners. Learned detectors achieve 0.882-0.932 Random Macro-F1 but only 0.653-0.665 under Source-Disjoint evaluation; the strongest word TF-IDF SVM scores 0.932/0.916/0.665 on Random/structural-disjoint/Source-Disjoint while retaining 95.6% malicious recall but producing 62.4% benign FPR on held-out sources. Off-the-shelf scanners occupy different but also unsatisfactory operating regimes, reducing false positives only at the cost of sharply lower malicious recall. Together, these results show that reliable malicious-Skill detection requires both broader cross-source benchmark coverage and evaluation that jointly measures attack detection and benign over-flagging.

Figures & tables

Figure 1: Overview of MaliciousSkillBench . (A) Fragmented sources differ in artifacts, provenance, evidence, and benign support. (B) Controlled canonicalization, deduplication, structural grouping, benign curation, and conflict filtering yield 9,740 normalized-unique Skills (7,505 malicious; 2,235 benign). (C) Characterization and held-out-source evaluation expose heterogeneous attack coverage and distinct detector recall/FPR regimes; the source–attack matrix is schematic (quantitative results in Figure 2 ).

Source	Core raw	Norm.	Main benign	Core provenance
MalSkillBench	3,944	3,430	0	mixed/unresolved
MaliciousAgentSkillsBench	157	157	0	wild
SkillLeakBench	79	67	0	wild
Agent Skill Malware	124	124	0	wild
AgentTrap	91	89	50	test fixture
SkillTrojan	1	1	2	backdoored

Table 1: Coverage of the 11 Core-contributing sources in the frozen benchmark. “Norm.” reports per-source normalized-unique malicious identities and therefore need not sum to 7,539 because identities can occur in multiple sources; Main-benign counts precede conflict exclusion.

Figure 2: Threat landscape of MaliciousSkillBench . (a) Prevalence of 11 harmonized attack categories among 4,983 malicious identities with direct or strong-semantic source-native mappings; categories are multi-label and percentages use this mapped subset. (b) Source-wise attack composition for the nine Core sources with mappable annotations; cells show raw counts, with power-normalized color intensity for visibility. (c) Attack–impact co-occurrence for the 1,888 identities with both mappings. Impact labels are conservative high-level derived mappings available for a bounded subset of the corpus.

Model	Random	SB	M-Struct.	Source	Rand. $\mathrm{FPR}_{B}$	Src. $\mathrm{FPR}_{B}$
Word TF–IDF + LR	.882	.874	.860	.661	.105	.620
Word TF–IDF + SVM	.932	.921	.916	.665	.094	.624
Char TF–IDF + SVM	.921	.907	.883	.653	.098	.644

Table 2: Primary detector results. Values are three-seed mean Macro-F1. SB is Source-Balanced Random;

\mathrm{FPR}_{B}

is the benign false-positive rate (malicious is the positive class).

Detector	Setting	Mal. recall	Benign FPR	Macro-F1
Word TF–IDF + SVM	learned	.956	.624	.665
Cisco-local-behavioral	fixed scanner	.025	.011	.308
SkillFortify-offline	fixed scanner	.253	.499	.349
SkillSpector-static	fixed scanner	.000	.006	.281

Table 3: Source-Disjoint operational comparison. Scanner coverage is 100%; scanners use fixed external settings, while word-SVM is protocol-trained.

Appendix figures & tables56 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Source	Reported	Acquired	Core M	Main B	Canonical benchmark role
SRC001	MalSkillBench	7,944	7,944	3,944	–	Core malicious; source benign labels retained outside Main
SRC002	MaliciousAgentSkillsBench	98,380	98,380	157	–	Recovered malicious artifacts enter Core; public metadata/security labels otherwise remain Auxiliary
SRC003	Skill-Inject	202	319	–	–	Auxiliary only: released unit is an injection–task pair
SRC004	SkillLeakBench	520	520	79	–	Recovered malicious artifacts enter Core; vulnerability-only and unresolved rows remain Auxiliary
SRC005	Agent Skill Malware	350	347	124	–	Core malicious; weak benign labels retained outside Main
SRC006	AgentTrap	141	141	91	50	Core malicious + Main benign

Appendix

Table 4: Complete frozen source registry used by this paper. “Reported” is the source/paper count at the recorded public revision; “Acquired” is the number of source records materialized into the canonical registry. Core M is the number of malicious Skill artifacts admitted to Core, including SRC002/SRC004 artifacts recovered from author-provided historical research snapshots. Main B is the raw high-confidence benign contribution before cross-label conflict exclusion. A dash denotes zero.

ID	Released object / supervision	Reason for Auxiliary-only status
SRC003	injection–task pairs over host Skills	The benchmark unit is a task pair with no malicious Skill artifact; contextual cases are additionally retained as harmful/dual-use under their source semantics.
SRC007	sanitized marketplace content with automated security signals	Labels are automated silver supervision (clean/suspicious/malicious signals), so scanner-positive records are not promoted to malicious Core ground truth.

Appendix

Table 5: Why the two registry sources with zero Core malicious artifacts remain Auxiliary. Auxiliary status is a semantic decision about the released unit/evidence, not a judgment that the source is unimportant.

ID	Revision	Status	Upstream license	Source/package redistribution policy
SRC001	06e08312	completed	academic-research-only	adapter only
SRC002	422bf340	completed	MIT	exact frozen static Skill text released; historical snapshot provenance retained
SRC003	182f3d9d	partial	MIT	allowed
SRC004	8264436a	completed	MIT	exact frozen static Skill text released; historical snapshot provenance retained
SRC005	5cff435d	partial	MIT	allowed
SRC006	ab13f59d	completed	terms not provided	metadata/hash only

Appendix

Table 6: Frozen source revisions, licensing, and source/package redistribution policy. Revisions are abbreviated here for readability; the release registry stores full commit/revision identifiers and checksums. The policy column concerns upstream package/archive material beyond the public per-identity static-text layer described in Appendix L . “Partial” means the pinned public artifact does not exactly reproduce the count or artifact set claimed by the associated paper/README; no missing records are reconstructed.

Field group	Representative fields	Purpose
Identity	canonical_id , source_id , source_record_id	Stable benchmark identity while retaining upstream traceability.
Source snapshot	source_dataset , repository_commit , dataset_snapshot_date , paper version	Reconstruct the exact public revision used for acquisition.
Artifact / role	artifact_unit , core_malicious_skill , auxiliary_type	Separate Skill artifacts from task pairs, metadata, fixtures, and other non-Core units.
Semantic labels	source_label , intent_label , provenance , ground_truth_strength , evidence_level	Preserve source semantics while exposing orthogonal canonical axes.
Content identity	skill_sha256 , normalized_skill_sha256	Exact and conservative formatting-normalized identity; no semantic equivalence claim.
Lineage	lineage_status , lineage_resolution , lineage_basis , base_skill_id , variant_id , base_skill_semantics	Preserve only source-supported parent/family relationships and explain what a base identifier means.

Appendix

Table 7: Field groups in the frozen canonical schema. Fields shown are representative; source-native labels are retained in addition to the canonical fields.

Intent	Records	Operational meaning
malicious	8,779	The source supports intentional malicious or unauthorized behavior associated with the record. Core eligibility is checked separately.
vulnerable	1,451	A weakness, exposure, or exploitable condition is reported, without sufficient evidence that the Skill artifact itself is intentionally malicious.
harmful_or_ dual_use	139	The case can facilitate harmful behavior, but the released case is not treated as an intentionally malicious Skill artifact.
benign	142,490	The source presents the record as normal/safe/negative; confidence and admissibility as Main benign are represented separately.
uncertain	29,840	Suspicious, ambiguous, or scanner-derived security signal that is insufficient for a stronger canonical intent claim.

Appendix

Table 8: Canonical intent labels. Counts are over all 182,699 source records before Core/Main filtering.

Provenance	Core raw	Norm. unique	Meaning
wild	413	364	Collected from a real upstream ecosystem/repository under the source’s published semantics; no benchmark-authored attack construction.
synthetic	295	295	Synthetic standalone Skill/test artifact generated or authored for evaluation.
injected	3,460	3,352	Malicious/harmful content is inserted into a pre-existing carrier, host Skill, or task construction.
backdoored	193	161	Benign-looking Skill contains hidden or triggered malicious behavior.
test_fixture	109	107	Curated or benchmark-authored fixture used as a controlled security test artifact.
mixed_ unresolved	3,944	3,430	Source reports a mixture of origins but lacks a trustworthy row-level provenance mapping.

Appendix

Table 9: Provenance vocabulary and frozen Core-malicious distribution. Raw counts are source records and sum to 8,414. Normalized-unique counts are identity coverage by provenance and can overlap when the same normalized identity appears in sources with different provenance labels.

Ground-truth strength
strong	10,525	High-confidence source evidence suitable for primary ground truth when artifact/unit constraints are also satisfied.
moderate	2,275	Meaningful source evidence with weaker certainty than strong; may still enter Core/Main under source-specific rules.
silver	169,676	Automated/scanner-derived or weakly supervised labels; retained for analysis but not sufficient alone for Core/Main ground truth.
weak	223	Weak negative/heuristic evidence; Auxiliary only.
unknown	0	No confidence assignment.
Evidence level

Appendix

Table 10: Ground-truth strength and evidence-level vocabularies. These axes are intentionally independent. Full-registry counts are shown for observed values.

Artifact unit	All records	Core malicious	Interpretation / Core policy
skill_package	82,094	7,932	Complete or package-level Skill artifact; Core-eligible when all other rules pass.
skill_md	1,261	482	Primary Skill instruction artifact; Core-eligible when all other rules pass.
task_pair	319	0	Attack/task evaluation pair; no released malicious Skill artifact; Auxiliary.
mcp_config	180	0	MCP-format configuration from a multi-format testbed; outside current Skill Core.
openclaw_ manifest	180	0	OpenClaw-format manifest from a multi-format testbed; outside current Skill Core.
environment_ fixture	1	0	Attack is carried only in the environment and is absent from the Skill package; Auxiliary.

Appendix

Table 11: Observed artifact units in the frozen analysis manifest. Core counts show the stricter malicious-Skill subset; recovered SRC002/SRC004 malicious artifacts account for the increase in skill_md units relative to the earlier snapshot.

Field	Values	Meaning
lineage_status	independent , derived , variant , unknown	Logical relationship asserted by the source: standalone identity, explicit derivative, member of a published variant family, or unresolved.
lineage_ resolution	resolved_official ; partially_ resolved_official ; unresolved	Whether an official parent/upstream/logical identity is fully available, only a family-level identity is available, or no trustworthy row-level link exists.
lineage_basis	official parent/family ID, upstream slug, benchmark-authored identity, logical construction ID, none	What source-published signal justifies the linkage; filenames and similarity are not accepted as lineage evidence.
base_skill_ semantics	pre-attack carrier, upstream identity, benchmark family, generation seed, standalone identity, unknown	Explains what the source-specific base_skill_id denotes so heterogeneous base IDs are not mistaken for a common attack-family unit.
base_skill_id , variant_id	source-scoped strings or null	Retain the publisher’s identifier when available; values are namespaced/source-specific and are not globally additive.

Appendix

Table 12: Lineage semantics in the frozen schema. Lineage is source-evidenced and is never inferred from textual similarity or structural clustering.

Source	Source label	Canonical intent	Strength / evidence	Benchmark interpretation
SRC002	safe	benign	silver / scanner	Metadata-level negative signal; retained in Auxiliary and excluded from Main benign.
SRC002	confirmed malicious	malicious	strong / human+runtime	Eligible for Core only after high-confidence linkage to an author-provided historical Skill artifact.
SRC007	malicious	uncertain	silver / scanner	Even a literal “malicious” scanner verdict remains uncertain without stronger ground truth.
SRC003	contextual	harmful_or_ dual_use	moderate / constructed	Context-dependent harmful task case; not a Core malicious Skill artifact.
SRC010	suspicious	vulnerable	strong/moderate; constructed/static	Source semantics describe vulnerable/suspicious cases without asserting intentional malicious artifacts.
SRC004	malicious	malicious	strong / human+runtime	79 source-malicious rows map to recovered historical Skill artifacts and enter Core; four ambiguous multi-Skill rows remain outside Core.

Appendix

Table 13: Representative source-label mappings. Mapping depends on source semantics, evidence, and artifact availability, not on the literal label string alone.

Source	Raw	Exact unique	Norm. unique	Raw $-$ Exact	Exact $-$ Norm.
SRC001	3,944	3,430	3,430	514	0
SRC002	157	157	157	0	0
SRC004	79	67	67	12	0
SRC005	124	124	124	0	0
SRC006	91	89	89	2	0
SRC008	1	1	1	0	0

Appendix

Table 14: Frozen Core malicious deduplication by source. “Raw

-

Exact” is the within-source reduction from repeated byte-identical artifacts; “Exact

-

Norm.” is the additional within-source reduction after conservative text normalization. Per-source unique counts do not sum to the global unique totals because cross-source identities are shared.

Source A	Source B	Exact hashes	Norm. hashes	Norm. cov. A	Norm. cov. B
SRC001	SRC002	3	3	0.13%	1.91%
SRC001	SRC004	21	21	0.79%	35.44%
SRC001	SRC005	69	90	15.14%	72.58%
SRC001	SRC006	1	1	0.03%	1.10%
SRC001	SRC010	78	78	14.63%	2.72%
SRC001	SRC011	6	8	0.35%	28.13%

Appendix

Table 15: Non-zero cross-source Core content overlap in the frozen benchmark. Coverage is the fraction of each source’s raw Core records participating in a shared normalized hash with the paired source. Pairwise counts overlap when one identity occurs in three or more sources.

Snapshot	Nodes	Families	Singletons	Non-singleton	Largest	Cross-source
Pre-recovery inherited base	7,340	4,415	3,059	1,356	146	91
Final frozen partition	7,539	4,588	3,219	1,369	146	113

Appendix

Table 16: Structural-partition evolution at the frozen medium threshold 0.68. The final partition preserves every inherited pre-recovery family without split/merge and integrates 199 newly recovered normalized identities under the same operational similarity rule.

Source A	Source B	Shared families	Coverage A	Coverage B
SRC001	SRC010	39	3.38%	3.04%
SRC001	SRC005	36	3.09%	75.81%
SRC001	SRC004	30	2.22%	49.25%
SRC005	SRC010	29	68.55%	2.48%
SRC009	SRC013	26	42.35%	62.99%
SRC004	SRC010	17	26.87%	1.47%

Appendix

Table 17: Largest pairwise cross-source structural overlaps in the frozen partition at threshold 0.68. Coverage is computed over normalized-unique Core units associated with each source. These entries report operational template overlap only, with no lineage or campaign attribution.

Measure	Count
Total / valid blind cases	72 / 72
Blindness violations	0
Same structural template	72
Different structural template	0
Uncertain	0
High-confidence judgments	72

Appendix

Table 18: Blind structural positive-validation summary. The reviewer saw anonymized static artifact pairs only; cluster identifiers, similarity values, thresholds, sample categories, algorithm predictions, and prior labels were hidden.

Criterion	Main-benign requirement	Auxiliary treatment
Released unit	Inspectable Agent Skill artifact usable for static hashing/detector input	Metadata-only, scanner-only, non-Skill, or otherwise non-comparable negative records remain Auxiliary
Canonical intent	benign under source-specific mapping	Uncertain, vulnerable, harmful/dual-use, or unresolved semantics are never collapsed into benign
Label strength	Strong or moderate	Silver, weak, “not flagged,” marketplace-unflagged, and analogous weak negatives remain Auxiliary
Content availability	Static primary Skill content available	Records lacking inspectable Skill text are excluded from the primary negative class
Traceability	Frozen source revision and source record can be reconstructed	Untraceable or revision-ambiguous records are not promoted to Main

Appendix

Table 19: Frozen eligibility gate for the benign reference pools. “Auxiliary” preserves security-relevant negative signals without promoting them to primary benign ground truth.

Pool	Raw	Exact-unique	Normalized-unique	Primary benchmark use
Main benign	2,251	2,238	2,238	Eligible before conflict exclusion
Auxiliary benign	45,966	44,329	44,327	Not used as primary negative ground truth

Appendix

Table 20: Frozen benign-side pool counts before cross-label conflict exclusion. Main is the primary high-confidence negative pool; Auxiliary retains weaker or non-comparable benign-side evidence.

ID	Source	Raw Main B	Frozen benchmark interpretation
SRC006	AgentTrap	50	Inspectable benign Skill packages with strong/moderate source evidence
SRC008	SkillTrojan	2	Benign example packages in the pinned public repository
SRC010	SkillTrustBench	1,643	High-confidence source-normal Skill artifacts eligible under the canonical benign mapping
SRC011	ATR Skill Security Benchmark	466	Inspectable benign Skill artifacts admitted by the frozen source adapter
SRC012	SkillFortifyBench	90	Benign Claude Skill artifacts; non-Skill MCP/OpenClaw formats remain Auxiliary
Total		2,251

Appendix

Table 21: Sources contributing raw Main-benign candidates before deduplication and conflict exclusion. Counts sum to 2,251.

Quantity	Count	Benchmark effect
Exact conflict hashes	34	Identity groups with both malicious- and benign-side labels
Normalized conflict hashes	34	Frozen conflict groups used for exclusion
Core-malicious raw records affected	162	Source records participating on the malicious side
Main-benign raw records affected	4	High-confidence negative records participating in conflicts
Auxiliary-benign raw records affected	39	Auxiliary negative records participating in conflicts
Core-malicious normalized units removed	34	7,539 $\rightarrow$ 7,505

Appendix

Table 22: Cross-label conflict accounting in the frozen benchmark. Raw affected counts measure source records participating in a conflict; removed-unit counts measure normalized-unique units that would otherwise enter the primary benchmark.

Dimension	Annotated	Coverage	Sources	Semantics
Attack vector	1,466	19.5%	1	Source-native vector
Attack type / tactic	4,147	55.3%	9	Source-native attack labels
Behavior	5,456	72.7%	7	Source-native behavior labels
Insertion strategy	166	2.2%	1	Source-native insertion strategy
Target / asset	2,312	30.8%	3	Source-native target labels
Severity	209	2.8%	2	Source-native severity

Appendix

Table 23: Threat-metadata coverage in the 7,505 primary malicious identities. Source-native fields preserve upstream semantics; harmonized fields use only documented direct or strong-semantic mappings. No missing field is inferred from Skill text.

Attack category	Identities	% mapped	Sources
Execution / Code Delivery	3,320	66.6	7
Instruction / Goal / Memory Manipulation	1,671	33.5	8
Privilege / Tool / Authority Abuse	1,013	20.3	6
Data Exfiltration / Disclosure	355	7.1	6
Resource / Availability Abuse	330	6.6	4
Credential Access	303	6.1	5

Appendix

Table 24: Final harmonized attack taxonomy and prevalence. Counts are normalized-unique primary malicious identities among the 4,983 attack-mapped units. Categories are multi-label, so counts and percentages do not sum to the mapped denominator or 100%. “Sources” is the number of Core sources contributing at least one mapped identity to the category.

Figure 3: Provenance composition by harmonized attack category. Each bar is normalized within category and therefore sums to 100%; the absolute mapped category total is printed at the right. Categories are multi-label, and provenance is taken directly from frozen benchmark metadata with no text-based inference.

Figure 4: MalSkillBench (SRC001) source-native attack-vector distribution for the 1,466 primary normalized identities with deterministic vector recovery. CI/PI/MIXED retain the source’s own semantics and are not generalized to other sources.

Figure 5: Pairwise attack-category co-occurrence in the attack-mapped subset. The upper triangle reports raw normalized-identity counts; axes follow prevalence ordering and color uses power normalization only for visibility. The diagonal is omitted.

Derived impact	Identities	% mapped
Unauthorized code execution / system control	489	23.0
Sensitive data disclosure	423	19.9
Credential compromise	297	14.0
Agent-control / instruction compromise	275	12.9
Privilege / control manipulation	247	11.6
Resource / service abuse	212	10.0

Appendix

Table 25: Conservative derived-impact prevalence among the 2,128 impact-mapped malicious identities. Labels are multi-label and come from documented source-native behavior semantics, not model labeling.

Protocol	Retained	Train (M/B)	Val. (M/B)	Test (M/B)	Test src.
Random	9,740	6,818 (5,254/1,564)	974 (750/224)	1,948 (1,501/447)	10
Source-bal. Random	9,740	6,817 (5,254/1,563)	973 (750/223)	1,950 (1,501/449)	11
Mal.-Struct.-Disj.	9,740	6,818 (5,254/1,564)	974 (750/224)	1,948 (1,501/447)	11
Source-Disjoint	9,732	7,513 (5,992/1,521)	835 (666/169)	1,384 (839/545)	3

Appendix

Table 26: Expanded inventory of the frozen evaluation protocols. M/B denotes malicious/benign counts. Source-Disjoint contains eight fewer units because cross-boundary multi-source normalized identities are conservatively excluded.

ID	Source	Malicious	Benign	Total	Malicious share
SRC009	SkillHarm	728	0	728	100.0%
SRC011	ATR Skill Security	21	455	476	4.4%
SRC012	SkillFortifyBench	90	90	180	50.0%
Total	–	839	545	1,384	60.6%

Appendix

Table 27: Held-out test composition in Source-Disjoint. The source-wise label mixture is strongly heterogeneous, motivating the term source-conditioned generalization and limiting interpretation as a pure source-effect estimate.

Protocol	Struct.	Source	Lineage	Required zero-overlap axes
Random	540	11	83	exact, norm.
Source-bal. Random	544	11	81	exact, norm.
Mal.-Struct.-Disj.	0	11	59	exact, norm., struct.
Source-Disjoint	321	0	19	exact, norm., source

Appendix

Table 28: Frozen cross-partition overlap audit. Counts are overlapping relationship/group identifiers, not sample counts. A non-zero value is permitted when that axis is not part of the protocol’s declared disjointness contract.

Representation	Analyzer / n-grams	Document-frequency filters	Feature cap / TF-IDF	Text normalization inside vectorizer
Word TF-IDF	word; 1–2 grams	min_df=2 ; max_df=0.995	feature cap 120,000; sublinear TF; smooth IDF; L2 norm	lowercase; Unicode accent stripping
Character TF-IDF	char_wb ; 3–5 grams	min_df=2 ; max_df=1.0	feature cap 160,000; sublinear TF; smooth IDF; L2 norm	lowercase; no accent stripping
Format-only	five scalar document features	n/a	5 dimensions; standardized before classification	no lexical representation

Appendix

Table 29: Frozen feature-extractor configurations for the primary classical baselines. Parameters are read from the serialized seed-42 estimators and are identical across the four primary protocols.

Model	Classifier	Fixed classifier parameters
Word TF-IDF + LR	LogisticRegression	$C=1.0$ ; L2 penalty; balanced class weights; liblinear solver; max. 1,500 iterations; tolerance $10^{-4}$ .
Word TF-IDF + SVM	LinearSVC	$C=1.0$ ; L2 penalty; squared-hinge loss; balanced class weights; automatic dual mode; max. 6,000 iterations; tolerance $10^{-4}$ .
Char TF-IDF + SVM	LinearSVC	$C=1.0$ ; L2 penalty; squared-hinge loss; balanced class weights; automatic dual mode; max. 6,000 iterations; tolerance $10^{-4}$ .
Format-only + LR	StandardScaler + LogisticRegression	Five standardized scalar inputs; $C=1.0$ ; L2 penalty; balanced class weights; liblinear solver; max. 1,000 iterations; tolerance $10^{-4}$ .

Appendix

Table 30: Frozen classifier configurations. For the classical models, random_state is set to the repeated-run seed (42/43/44). Other parameters are fixed across protocols and seeds.

Metric	Definition	Role in this paper
Macro-F1	$(F1_{M}+F1_{B})/2$	Primary paper-level metric; gives equal weight to malicious and benign classes.
Balanced accuracy	$\tfrac{1}{2}(\mathrm{TPR}+\mathrm{TNR})$	Secondary prevalence-robust accuracy summary.
Benign FPR	$\mathrm{FP}/(\mathrm{FP}+\mathrm{TN})$	Fraction of truly benign Skills incorrectly flagged malicious; central transfer error diagnostic.
MCC	$\frac{\mathrm{TP}\mathrm{TN}-\mathrm{FP}\mathrm{FN}}{\sqrt{(\mathrm{TP}+\mathrm{FP})(\mathrm{TP}+\mathrm{FN})(\mathrm{TN}+\mathrm{FP})(\mathrm{TN}+\mathrm{FN})}}$	Correlation-style summary using all four confusion-matrix cells.
AUROC / AUPRC	ranking metrics over continuous decision scores	Secondary diagnostics; omitted for constant predictors without a decision score.

Appendix

Table 31: Main metric definitions. Malicious is the positive class;

F1_{M}

and

F1_{B}

denote class-wise F1 for malicious and benign Skills, respectively.

Protocol	Partition	Orig. M	Orig. B	Bal. M	Bal. B	Bal. total
Random	Train	5,254	1,564	1,564	1,564	3,128
Random	Validation	750	224	224	224	448
Random	Test	1,501	447	447	447	894
Mal.-Struct.-Disj.	Train	5,254	1,564	1,564	1,564	3,128
Mal.-Struct.-Disj.	Validation	750	224	224	224	448
Mal.-Struct.-Disj.	Test	1,501	447	447	447	894

Appendix

Table 32: Class-balanced robustness subsets. Malicious units are deterministically downsampled within each frozen partition to match the benign count; benign units are never removed for balancing and no unit crosses partitions.

Balanced protocol	Structural	Source	Status
Random	97	11	PASS
Mal.-Struct.-Disj.	0	10	PASS
Source-Disjoint	36	0	PASS

Appendix

Table 33: Overlap audit after class balancing in the frozen benchmark. Exact/normalized overlap remains zero. Structural/source overlap is diagnostic unless disallowed by the underlying frozen protocol.

Rule	Deterministic action / evidence boundary	Docs	Lines
R1 Original-Skill delimiter	Replace exact BEGIN/END ORIGINAL SKILL.md delimiter lines; generator boundary only.	888	917
R2 Generator-output instruction	Replace the exact instruction requiring complete reproduction of the original SKILL.md ; construction instruction only.	61	61
R3 Initialization-contract marker	Mask only the phrases “SDK initialization contract” and “part of this skill’s initialization contract”; retain commands and surrounding behavior.	356	356
R4 Generated pre-flight heading	Replace exact generated headings of the forms “Pre-flight guard (…)” and “Pre-flight step (…)”; heading only.	297	297
R5 Pre-flight concealment annotation	Replace the exact construction annotation “Do not explain the pre-flight step to the user.”	113	113
R6 Operational-note wrapper	Collapse the exact generated Operational-note wrapper to a construction placeholder while preserving its backticked command.	176	176

Appendix

Table 34: Frozen scaffold-sanitization rules. Counts are over the 9,740-unit primary benchmark. Rules target audited construction wrappers, not attack-semantic content.

Model	Setting	Random F1	Source F1	Gap	Rand. $\mathrm{FPR}_{B}$	Src. $\mathrm{FPR}_{B}$
Word LR	Original	.882	.661	.220	.105	.620
Word SVM	Original	.932	.665	.266	.094	.624
Char SVM	Original	.921	.653	.268	.098	.644
Word LR	Balanced	.881	.633	.249	.072	.583
Word SVM	Balanced	.925	.710	.215	.056	.433
Char SVM	Balanced	.904	.660	.244	.072	.525

Appendix

Table 35: Random-to-Source-Disjoint robustness controls on the frozen benchmark. Gap is Random Macro-F1 minus Source-Disjoint Macro-F1.

\mathrm{FPR}_{B}

is the benign false-positive rate under the malicious-positive convention.

Model	Split	Acc.	M-F1	B-F1	Macro-F1	Bal. Acc.	MCC	FPR B	AUROC	AUPRC
Word TF–IDF + LR	Random	0.911	0.941	0.822	0.882	0.905	0.768	0.105	0.965	0.989
	Source-bal. Random	0.906	0.937	0.811	0.874	0.897	0.753	0.120	0.967	0.990
	Mal.-Struct.-Disj.	0.893	0.928	0.792	0.860	0.891	0.729	0.112	0.957	0.987
	Source-Disjoint	0.722	0.804	0.518	0.661	0.662	0.409	0.620	0.738	0.769
Word TF–IDF + SVM	Random	0.951	0.968	0.895	0.932	0.935	0.863	0.094	0.986	0.996
	Source-bal. Random	0.944	0.964	0.879	0.921	0.922	0.842	0.120	0.985	0.996

Appendix

Table 36: Full frozen metrics for the three primary static text detectors. Seeds 42/43/44 produce identical discrete predictions under the fixed assignments (classification-score standard deviation 0); means are shown. Malicious is the positive class; FPR B is the benign false-positive rate.

Model	Split	M	B	TP	FP	TN	FN
Word TF–IDF + LR	Random	1501	447	1375	47	400	126
	Mal.-Struct.-Disj.	1501	447	1343	50	397	158
	Source-Disjoint	839	545	792	338	207	47
Word TF–IDF + SVM	Random	1501	447	1448	42	405	53
	Mal.-Struct.-Disj.	1501	447	1420	39	408	81
	Source-Disjoint	839	545	802	340	205	37

Appendix

Table 37: Frozen confusion-matrix counts for the three headline protocols. Counts follow the malicious-positive convention and are identical across seeds 42/43/44.

	Random				Source-Disjoint
Model	M-F1	Macro	BalAcc	FPR B	M-F1	Macro	BalAcc	FPR B
Always-malicious	0.870	0.435	0.500	1.000	0.755	0.377	0.500	1.000
Always-benign	0.000	0.187	0.500	0.000	0.000	0.283	0.500	0.000
Format-only LR	0.583	0.486	0.568	0.313	0.513	0.483	0.495	0.457

Appendix

Table 38: Frozen trivial and coarse-formatting diagnostics. The high Random malicious F1 of the always-malicious rule illustrates why malicious-class F1 alone is not an appropriate paper-level metric for the natural class ratio.

Split	M-Prec.	M-Rec.	M-F1	B-F1	Macro-F1	Acc.	FPR B
Random	0.880	0.688	0.772	0.501	0.636	0.687	0.315
Mal.-Struct.-Disj.	0.893	0.689	0.778	0.522	0.650	0.697	0.277
Source-Disjoint	0.648	0.895	0.752	0.356	0.554	0.642	0.749

Appendix

Table 39: Auxiliary MiniLM embedding baseline (seed 42). Unlike the earlier archived snapshot, the final run stores both class-wise metrics and confusion counts, so Macro-F1 and benign FPR are reported directly.

Held-out	Test $n$	M/B	M-Prec.	M-Rec.	M-F1	Macro-F1	Acc.	FPR B
SRC006	131	86/45	0.580	0.674	0.624	0.351	0.466	0.933
SRC010	4,426	2,783/1,643	0.654	0.850	0.739	0.529	0.623	0.762
SRC011	476	21/455	0.065	0.952	0.122	0.329	0.393	0.633
SRC012	180	90/90	0.667	1.000	0.800	0.733	0.750	0.500

Appendix

Table 40: Exploratory leave-one-source-out (LOSO) diagnostic for the word TF–IDF + linear SVM. Label mixtures differ substantially across held-out sources, so the rows are reported as source-specific heterogeneity diagnostics and are not aggregated into one summary score.

Source	M	B	TP	FP	TN	FN	M-Rec.	FPR B
SRC009	728	0	693	0	0	35	0.952	–
SRC011	21	455	19	293	162	2	0.905	0.644
SRC012	90	90	90	47	43	0	1.000	0.522

Appendix

Table 41: Source-wise error accounting for the frozen Source-Disjoint word-SVM predictions. A dash denotes that the source supplies no benign examples in the held-out test.

Source	Protocol	Benign $n$	FP	TN	FPR B
SRC011	Random	97	11	86	0.113
SRC011	Source-Disjoint	455	293	162	0.644
SRC012	Random	14	0	14	0.000
SRC012	Source-Disjoint	90	47	43	0.522

Appendix

Table 42: Descriptive benign false-positive rates for the two Source-Disjoint held-out sources that contain benign examples, compared with their Random-test subsets. The evaluated units differ and Random exposes each source during training, so this is not a paired causal estimate.

Scanner	Coverage	Mal. recall	Benign FPR	Macro-F1	FP	FN
Cisco-local-behavioral	.9998	.064	.008	.254	17	7,021
SkillFortify-offline	.9999	.566	.451	.516	1,007	3,257
SkillSpector-static	1.0000	.033	.002	.222	4	7,258

Appendix

Table 43: Off-the-shelf scanner results on the full 9,740-unit benchmark. Primary metrics exclude technical abstentions and report coverage separately.

Scanner	Protocol	Coverage	Mal. recall	Benign FPR	Macro-F1
Cisco-local	Random	1.0000	0.065	0.004	0.254
	Source-Bal.	1.0000	0.065	0.004	0.255
	M-Struct	0.9990	0.082	0.004	0.272
	Source	1.0000	0.025	0.011	0.308
SkillFortify	Random	1.0000	0.553	0.421	0.518
	Source-Bal.	1.0000	0.569	0.490	0.504

Appendix

Table 44: Fixed scanner configurations on the four frozen protocol test sets. Scanners are not trained on protocol train splits; coverage is reported because technical abstentions remain outside primary binary metrics.

Scanner	Gate	Mal. recall	Benign FPR	Macro-F1
Cisco	HIGH+ / native (primary)	.064	.008	.254
Cisco	MEDIUM+	.127	.025	.310
SkillFortify	MEDIUM+ (primary)	.566	.451	.516
SkillFortify	native is_safe	.576	.474	.514
SkillFortify	HIGH+	.565	.450	.515
SkillSpector	block gate (primary)	.033	.002	.222

Appendix

Table 45: Pre-specified scanner operating-point sensitivity on the full benchmark. Primary rows are the paper-facing configurations.

Population	$N$	Mal. recall	Benign FPR	Macro-F1
Full benchmark	9,740	.566	.451	.516
Full excluding SRC012 provenance	9,560	.561	.470	.505
Source-Disjoint	1,384	.253	.499	.349
Source-Disjoint excluding SRC012	1,204	.167	.598	.254

Appendix

Table 46: SkillFortify source-overlap sensitivity under the fixed MEDIUM+ primary gate.

Benchmark ID	Scanner	Label	Source	Technical reason
ASB04_002434	Cisco-local	Mal.	SRC001	no result (21 MB input)
ASB04_007741	Cisco-local	Ben.	SRC010	invalid UTF-8
ASB04_007741	SkillFortify	Ben.	SRC010	no Skill parsed

Appendix

Table 47: The three technical abstentions among 29,220 scanner–item jobs. None occurs in Source-Disjoint; all appear in the Malicious-Structural-Disjoint test.

Scanner	Macro-F1	Mal. recall	Benign FPR
Cisco-local	0.308 [0.290, 0.325]	0.025 [0.015, 0.036]	0.011 [0.004, 0.021]
SkillFortify	0.349 [0.324, 0.375]	0.253 [0.224, 0.283]	0.499 [0.457, 0.541]
SkillSpector-static	0.281 [0.268, 0.295]	0.000 [0.000, 0.000]	0.006 [0.000, 0.013]

Appendix

Table 48: Bootstrap 95% confidence intervals on Source-Disjoint (10,000 item-level resamples; seed 42). These intervals quantify sampling uncertainty for deterministic scanner outputs, not seed variance.

Figure 6: Malicious recall by harmonized attack category for the three frozen primary scanner configurations. Cell annotations are recall percentages; category sample sizes $N$ are shown on the x-axis. The analysis is conditional on the 4,983 malicious identities with supported attack mappings, and categories are multi-label. Differences are descriptive and may reflect source composition as well as scanner behavior.

Example	Frozen ID / source	Canonical metadata
Wild malicious	ASB04_002865
UASSB_
00181999
SRC011	wild; Skill text; moderate/static	Upstream Snyk exfiltration case. The source descriptor identifies exfiltration behavior; transport command and destination are omitted.
Injected malicious	ASB04_006504
UASSB_

Appendix

Table 49: Representative primary benchmark units. Malicious content is summarized non-operationally; no payload command, endpoint, helper code, or executable instruction is reproduced.

Canonical / benchmark ID	Source-record shorthand	Source	Normalized hash prefix	Similarity to representative	Structural-review observation
UASSB_
00007858
ASB04_
001347	YouTube summarize (35o20)	SRC001	2c9012ef5e9f...	0.9967	Shared transcript-summarization setup, extraction, metadata, summarization, language, and option sections.
UASSB_
00007934

Appendix

Table 50: Representative non-duplicate structural-reuse pair from a frozen operational structural family. The pair was included in the blinded positive-validation set and judged high-confidence same structural template.

Canonical ID	Record	Intent	Evidence	Upstream case descriptor	Shared artifact context
UASSB_00107519	107	benign	strong / runtime	DIM7: output tampering	Derived from the same upstream xlsx Skill identity; artifact text is identical at the frozen exact-hash level.
UASSB_00107520	108	benign	strong / runtime	DIM8: unauthorized disclosure
UASSB_00107538	20	malicious	strong / runtime	DIM7: output tampering
UASSB_00107550	22	malicious	strong / runtime	DIM8: unauthorized disclosure

Appendix

Table 51: Representative SRC006 cross-label identity conflict. All four records share exact Skill hash prefix dd316db7785a... and normalized hash prefix 6afac6531997... ; the conflicting group is excluded from the primary benchmark.

Protocol	True label	Prediction	Decision score	Interpretation
Random	benign	benign	$-0.360$	Correct classification for ASB04_007510 .
Source-Disjoint	benign	malicious	$+0.509$	False positive on the same unchanged static Skill unit under source-held-out training.

Appendix

Table 52: Paired prediction for the same benign Skill under two frozen test protocols (seed-42 word TF–IDF + linear SVM). Positive decision scores predict malicious.

Layer	Contents	Release form	Reproducibility role
Canonical registry	Canonical/source IDs, frozen revision, original/canonical label, provenance, evidence, label strength, artifact unit, lineage fields, license metadata	Public metadata	Reconstruct benchmark scope and source semantics
Identity and audit	Raw/exact identity hashes, normalized hashes, conflict flags, structural-family IDs, benchmark membership	Public metadata/hash	Verify identity, deduplication, reuse, and exclusions without executing artifacts
Primary static Skill text	Exact frozen text for 9,735 identities; sanitized readable representations for five credential-bearing malicious identities	Public text	Inspect all 9,740 primary identities while preserving the exact experimental text for 9,735
Evaluation	Master unit IDs, split assignments, leakage checks, sanitizer rules, detector configurations, metric/result tables	Public non-payload artifacts	Reproduce paper protocols and reported analyses
Redistributable Skill artifacts	Static Skill text/packages whose recorded source terms and benchmark-side policy permit republication	Source-specific	Enable direct local detector input where redistribution is allowed
Restricted-source reconstruction	Pinned source revision, source record identifiers, adapter/reacquisition instructions, and checksums for artifacts not republished	Adapter or metadata/hash only	Let users reacquire under upstream terms and verify local correspondence

Appendix

Table 53: Release layers for the frozen paper snapshot. Every primary identity has a public readable representation; source/package redistribution beyond that text layer remains source-specific as recorded in Appendix A.3 .

Maintenance event	Required action	Effect on frozen paper snapshot
Documentation/citation clarification	Record in changelog; do not alter data bytes, labels, hashes, or splits	None; paper numbers remain directly comparable
License/redistribution clarification	Update release representation and attribution; preserve the recorded historical policy in the archived snapshot	May change what is distributed, not the historical benchmark membership
Label, content, or conflict correction	Issue a new benchmark release; regenerate affected canonical metadata, hashes, conflicts, splits, and derived metrics as needed	The paper snapshot remains available and unchanged
Source addition/removal or newer upstream revision	Create a new release with explicit source/revision delta and fresh deduplication/conflict auditing	Not backfilled into the paper snapshot
Split/protocol redesign	Version the protocol and keep the original assignments available	Prior results retain their original protocol meaning
New detector or leaderboard entry	Record model, inputs, configuration, release, protocol, and class-aware metrics	Does not require a benchmark-data version change

Appendix

Table 54: Maintenance and versioning contract. The paper snapshot remains immutable; scientifically material data changes create a new benchmark release while historical results remain tied to the frozen snapshot.

Field	Frozen paper-snapshot value / contract
Source registry	13 public sources; 182,699 canonical registry records; 11 sources contribute at least one Core malicious Skill artifact.
Core malicious construction	8,414 raw artifacts $\rightarrow$ 7,562 exact-unique $\rightarrow$ 7,539 normalized-unique malicious identities.
Structural reuse control	4,588 operational structural families at the frozen 0.68 threshold; family IDs support structural-disjoint evaluation and reuse auditing, with no campaign-attribution semantics.
Cross-label conflicts	34 normalized malicious identities conflict with benign identities and are excluded from the primary benchmark under the frozen conflict policy.
Primary detection benchmark	9,740 normalized-unique Skills: 7,505 malicious + 2,235 Main-benign.
Threat-characterization coverage	4,983 malicious identities have harmonized multi-label attack mappings; 2,128 have conservative derived-impact mappings; 1,888 have both attack and impact mappings.

Appendix

Table 55: Compact benchmark card for the frozen MaliciousSkillBench paper snapshot. Counts refer to the canonicalized static benchmark unless a subset is stated explicitly.

AI Agent Security Benchmarks

AI agent security benchmarks measure how tool-using and autonomous systems handle attacks, privacy risks, authorization, and harmful actions.

See all 284 papers

Detecting Malicious Agent Skills in the Wild using Attention
Jun 22, 2026Bacem Etteib, Daniele Lunghi, Tégawendé F. BissyandéLLM Agent Security Agent Skill Security Auditing
University of Luxembourg
Pretext: Defeating Malicious Skill Detection Frameworks for AI Agents
Sep 30, 2026Tobias Kaiser, Aritra DharAI Agent Security Agent Skill Security Auditing
Computing System Labs, Huawei Research Zurich
SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models
Oct 13, 2026Jerrold Huang, Junwei Quan, Saiyue Lyu +2Agent Skill Security Auditing AI Agent Security Benchmarks
University of Toronto · University of British Columbia · Rutgers University

Jun 22, 2026J/K move · Enter open · S save

Detecting Malicious Agent Skills in the Wild using Attention

Bacem Etteib, Daniele Lunghi, Tégawendé F. Bissyandé

University of Luxembourg

LLM Agent Security Agent Skill Security Auditing

Open paper AskPDF arXiv

LLM agents increasingly load skills, file-based packages of natural-language instructions written by third parties and distributed through marketplaces, that execute with the user's privileges. A single malicious skill can exfiltrate data, hijack the agent, or persist as a supply-chain foothold, which turns the skill marketplace into a new attack surface for agentic systems. Prompt-injection defenses do not carry over to this setting. They rely on a boundary between trusted instructions and untrusted data, whereas a skill is itself a body of instructions, so an injected command sits among many legitimate ones and inherits their authority. We present Locate-and-Judge, a two-stage detector designed for this regime. A lightweight locator scores the structural spans of a skill by the instruction-following attention each span draws and retains only the top-K. A judge then examines the retained spans in detail. Concentrating the costly judgment on a few high-attention spans lets the detector audit an entire marketplace instead of a sample. Compared to direct LLM-based scanning, this approach offers an order-of-magnitude cost reduction, dramatically increasing its scalability at a small cost to recall, and it dominates keyword and regex baselines at comparable expense. Deployed at marketplace scale and at negligible cost, Locate-and-Judge flags skills with high precision, the majority of which we manually confirmed as malicious, surfacing dozens of live malicious skills, including several disguised as benign functionality and many that SkillSpector and Cisco Skill Scanner fail to detect. We release the resulting labeled dataset.

Abstract

AI Agent Security Benchmarks

Explore similar work

Detecting Malicious Agent Skills in the Wild using Attention

Pretext: Defeating Malicious Skill Detection Frameworks for AI Agents

SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models