All against the machine: the Solo score for rating skill in variable environments

Agent skills provide reusable knowledge that helps agents solve specialized tasks. As their use expands, evaluating skill quality becomes increasingly important. Existing evaluations often measure skill quality based on improvements in agent performance on specific tasks. However, a reusable skill may apply across multiple scenarios. Evaluation on a single downstream task mainly reflects the compatibility between a skill and the evaluated task, provides only a partial view of skill quality, and does not identify which aspects need improvement. We find that general properties of the SKILL.md document play an important role in skill quality. To evaluate these properties, we propose SkillEval, an interpretable framework for document-level skill evaluation. SkillEval produces interpretable scores for each property using a fixed, inspectable scoring direction. It further measures and reduces the influence of unrelated document features, so each score more specifically captures its intended semantic property. We use SkillEval to evaluate skills in controlled quality tests, showing that it reliably distinguishes skills of different quality. Its scores closely reflect downstream task performance, providing an early indication of skill effectiveness. We further explore SkillEval for diagnosing document weaknesses and guiding targeted revisions that improve the targeted properties and increase downstream task pass rates.

Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.

This technical report presents SkillTester, a tool for evaluating the utility and security of agent skills. Its evaluation framework combines paired baseline and with-skill execution conditions with a separate security probe suite. Grounded in a comparative utility principle and a user-facing simplicity principle, the framework normalizes raw execution artifacts into a utility score, a security score, and a three-level security status label. More broadly, it can be understood as a comparative quality-assurance harness for agent skills in an agent-first world. The public service is deployed at https://skilltester.ai, and the broader project is maintained at https://github.com/skilltester-ai/skilltester.