SkillTester: Benchmarking Utility and Security of Agent Skills

Leye Wang; Zixing Wang; Anjie Xu

cs.CRMar 28, 2026

SkillTester: Benchmarking Utility and Security of Agent Skills

Authors: Leye Wang, Zixing Wang, Anjie Xu

Organizations: Key Laboratory of High Confidence Software Technologies, Ministry of Education, Peking University, Beijing, China · Northwestern Polytechnical University, Xi’an, China

Abstract

This technical report presents SkillTester, a tool for evaluating the utility and security of agent skills. Its evaluation framework combines paired baseline and with-skill execution conditions with a separate security probe suite. Grounded in a comparative utility principle and a user-facing simplicity principle, the framework normalizes raw execution artifacts into a utility score, a security score, and a three-level security status label. More broadly, it can be understood as a comparative quality-assurance harness for agent skills in an agent-first world. The public service is deployed at https://skilltester.ai, and the broader project is maintained at https://github.com/skilltester-ai/skilltester.

Figures & tables

Aspect	In scope	Out of scope
Evaluation target	Candidate third-party agent skills	General model benchmarks or enterprise-wide harnesses
Decision stage	Skill selection and enablement	Runtime monitoring or adaptation
Published outputs	Utility score, security score, and security status label	Full internal execution analytics
Security coverage	Controlled security probes	Exhaustive certification or formal verification

Table 1: Scope of the SkillTester evaluation framework.

Task group	Purpose	Used for	Typical pass condition
Common functional tasks	Representative intended use cases	Utility	Expected output or behavior is produced
Edge functional tasks	Boundary and failure-handling cases	Utility	Observed behavior matches expected output, refusal, or error handling

Table 2: Functional task groups used for utility evaluation in the current SkillTester framework.

Current direction	What current tasks try to detect	Indicative OWASP categories
Abnormal behavior control	Unsafe actions, risky execution behavior, uncontrolled retries, misleading safety claims, and tool misuse under controlled prompts	ASI02 tool misuse; ASI05 code execution
Permission boundary	Unauthorized file, network, process, or tool access beyond the skill’s intended role	ASI03 privilege abuse; ASI02 tool misuse
Sensitive data protection	Credential leakage, unsafe logging, exfiltration, and mishandling of sensitive context	ASI06 context poisoning; ASI09 trust exploitation

Table 3: Current controlled security probe directions in the SkillTester framework and their indicative correspondence to the OWASP Agentic Top 10 threat model.

Panel of Figure 1 — Figure 1: Overview of the SkillTester evaluation framework.

Dimension	Question answered	Primary evidence	Reported metrics
Utility	What value does the skill add relative to baseline?	Paired baseline/with-skill outcomes, invocation evidence, and token/time measurements	Task-level utility and aggregate utility score
Security	How does the skill behave under controlled probes?	Probe outcomes across three predefined groups	Group scores, aggregate security score, and security status label

Table 4: Dimensions used in the SkillTester evaluation framework.

Situation	Why the skill is useful	Utility implication
Baseline fails, skill succeeds	The skill enables an otherwise unavailable outcome	Clear incremental utility
Both succeed, but the skill path is cheaper	The skill reduces token cost or elapsed time	Positive utility without unique completion ability

Table 5: Illustrative ways a skill can create utility in the SkillTester setting.

Panel of Figure 2 — Figure 2: Published SkillTester scores and the internal structure of utility scoring.

Notation	Meaning
$U,S$	Utility and security scores
$T$	Set of valid functional tasks
$g_{t}$	Task-level invocation gate for task $t$
$y^{s}_{t},y^{b}_{t}$	Skill-path and baseline-path success indicators
$u^{s}_{t},u^{b}_{t}$	Skill-path and baseline-path token measurements
$\tau^{s}_{t},\tau^{b}_{t}$	Skill-path and baseline-path elapsed-time measurements

Table 6: Notation used in scoring.

Parameter	Current value	Role in scoring
$\eta$	50	Neutral efficiency point when skill-path and baseline cost are equal
$\alpha$	25	Sensitivity of the token/time efficiency mapping
$\beta$	20	Minimum utility credit when both paths succeed but the skill path is less efficient
$\theta_{s}$	80	Lower bound of the Caution security-status range

Table 7: Current parameter settings used in scoring.

Condition	Task score	Interpretation
Skill not invoked	0	No credit is assigned if the outcome cannot be attributed to actual skill use
Skill fails	0	A failed skill-path run receives no utility credit
Skill succeeds, baseline fails	100	The skill provides clear incremental value
Both succeed, equal cost	50	The skill succeeds with no cost advantage or penalty relative to baseline
Both succeed, skill more efficient	50–100	The skill succeeds at lower token or time cost than baseline
Both succeed, skill less efficient	20–50	Both paths succeed, but the skill path incurs higher task-level cost

Table 8: Representative task-level utility outcomes.

Reported item	Example value	Intended reading
Utility score	74.0	The skill provides meaningful value relative to baseline, but not every successful task contributes full incremental value
Security score	92.0	Most controlled security probes pass, but not every published security group reaches full pass rate
Security status	Caution	The averaged security score is high but below a perfect pass, so the skill merits caution rather than a clean pass label

Table 9: Illustrative published SkillTester output with hypothetical values.

LLM Agent Skill Evaluation

See all 26 papers

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment
Jun 21, 2026Dexu Yu, Youhua Li, Zhaoyang Guan +12AI Agent Auditing Automated Evaluation
Northeastern University · City University of Hong Kong · DeciLix Lab +8
SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills
Apr 8, 2026Yinghan Hou, Zongyou YangAI Agent Security Agent Skill Security Auditing
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom · Dyson School of Design Engineering, Imperial College London, London, United Kingdom
Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security
Jun 10, 2026Tu Lan, Chaowei XiaoAI Agent Auditing LLM Agent Security
Johns Hopkins University

Jun 21, 2026J/K move · Enter open · S save

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

Dexu Yu, Youhua Li, Zhaoyang Guan, Xianhao Lin+11

Northeastern University · City University of Hong Kong · DeciLix Lab+8

AI Agent Auditing Automated Evaluation Agent Skill Security Auditing LLM Agent Skill Evaluation

Open paper AskPDF arXiv

Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.

Abstract

LLM Agent Skill Evaluation

Explore similar work

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills

Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security