Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills

Christopher Kevin; Narendran Raghavan; Jean-Francois Puget; Roshni Malani; Meghana Puvvadi; Moshe Abramovitch; Mohit Gupta; Rama Akkiraju

cs.AIAug 20, 2026

Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills

Authors: Christopher Kevin, Narendran Raghavan, Jean-Francois Puget, Roshni Malani, Meghana Puvvadi, Moshe Abramovitch, Mohit Gupta, Rama Akkiraju,

Organizations: NVIDIA

Abstract

Enterprise agent programs are moving from prototypes into production, where reusable skills, tools, and workflow packages must be reviewed with evidence rather than prose. Current gates often scan these artifacts for structure, style, and security, but they do not answer the deployment question: does the capability package help a live agent complete enterprise tasks under the same model, sandbox, and grading policy? We present ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts. ACES runs paired live trials with and without a target skill, normalizes trajectories into the Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics, and reports Skill Lift: the target skill's added value for a fixed task, harness, workspace, and scorer. The same protocol supports product-owned task suites that compare baseline, skill, bundle, team-skill, and plugin targets. On 145 real skills from internal enterprise repositories and public catalogs, scan-only gates surface useful authoring issues but measure complementary facets (structural versus LLM-judge Spearman $ρ= 0.14$ ). Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95% paired-case CI [0.1967, 0.2301]); mean outcome-only lift, the average of accuracy and goal accuracy, is 0.1799. Composite lift is positive in 72.8% of paired cases. The largest process-metric gains appear in skill execution, behavior check, and skill efficiency---signals about discovery, routing, workflow following, and tool use that document scans cannot observe. An open-source implementation of the methodology is available in NVIDIA SkillEvaluator.

Figures & tables

Dimension (weight)	Rule families
Correctness (0.35)	Frontmatter contract ( name , description , version , author , tags , tools ); name/directory match; Instructions section; run_script mention; script-file presence; header-table documentation; injection-suspicious characters.
Discoverability (0.25)	Description length (50–150 preferred; $>$ 200 flagged as overlong); trigger vocabulary ( use , when , for ); avoidance of vague wording; explicit negative/boundary phrasing; directory-name conventions; tone (first/second person); “WHEN to use” guidance.
Reliability (0.25)	Error-handling vocabulary; input-validation hints in scripts/* ; Prerequisites , Limitations , Troubleshooting sections; MCP-connection guidance when MCP is declared.
Efficiency (0.15)	Token budget; repeated lines; list-format instructions; nested-reference depth; corporate/hedge language; oversize-skill warnings.

Table 1. Static check families per dimension (

{\approx}50

rules total). Each family contains multiple rules that fire independently; each rule deducts from its dimension score.

Criterion	What the judge is asked
description clarity	Does the description communicate what the skill does in one pass, without ambiguity?
instruction clarity	Are the instructions specific, ordered, and executable by an agent reading them cold?
example quality	Do worked examples cover the prompt variations an agent will actually see?
documentation completeness	Are the scripts, inputs, and expected outcomes documented?
scope definition	Is the boundary between what this skill does and what it does not explicit?
professional tone	Is the writing direct, concise, and free of marketing or hedge language?

Table 2. LLM-as-Judge rubric criteria (9 base + 1 conditional). Each is scored 0–10 with free-form reasoning.

Figure 1. Doc-scan evidence motivates, but does not replace, live evaluation. (a) The

0

–

100

structural score aggregates roughly 50 deterministic rule checks across Correctness, Discoverability, Reliability, and Efficiency for

n{=}145

real skills. The default

70

-point gate is permissive: 94.5% of skills clear it, but only 48.9% reach the

80

-point line. (b) Structural scores and LLM-judge rubric scores disagree on the same corpus (Spearman

\rho=0.14

, Pearson

r=0.08

). Together, the panels show that document scanning surfaces real authoring issues, but neither scan-only axis observes discovery, tool use, workflow order, or task success. Two-panel figure. The left panel is a histogram of structural scores for 145 real skills from internal enterprise repositories and public catalogs, with vertical markers at the 70-point gate and the 80-point line. Most skills are between 75 and 85, and annotations note that 94.5 percent pass the gate while 48.9 percent reach 80. The right panel is a scatter plot of structural score versus LLM-judge score for the same skills, colored by judge, with a y-equals-x reference line and annotations showing weak Spearman and Pearson correlation.

Figure 2. Top-10 most frequently violated static rules across

n{=}145

skills, colored by structural dimension. Four of the five most common violations are frontmatter-contract rules that 91–99% of skills fail. Horizontal bar chart of the ten most frequently violated static rules, with percentage of skills on the x-axis and rule description on the y-axis. Bars are color-coded by evaluation dimension. The top four bars exceed 90 percent and are all frontmatter-related rules.

Approach	Art.	Val.	Multi	CI	HITL
SkillsBench ( Chen et al., 2026b )	$-$	✓	✓	$-$	$-$
SkillTester ( Wang et al., 2026 )	✓	$-$	$-$	$-$	$-$
In-the-Wild ( Liu and others, 2026 )	$-$	✓	$-$	$-$	$-$
Anthropic skill-creator ( Anthropic, 2025d )	✓	$-$	$-$	$-$	$-$
Terminal-Bench ( Merrill and Terminal-Bench Team, 2026 )	$-$	$-$	✓	$-$	$-$
This work (ACES)	✓	✓	✓	✓	✓

Table 3. Capability gap across prior skill-evaluation tooling.

\checkmark

= covered;

-

= not covered. Artifact = scores the skill artifact directly; Value = measures marginal value via lift; Multi = evaluates across multiple agent harnesses; CI = CI-native deployment; HITL = human-in-the-loop dataset refinement.

Panel of Figure 3 — Figure 3. ACES runtime evaluation flow. Evaluation assets feed the ACES adapter and task emitter, which creates paired with-skill and baseline runs while holding configured support skills fixed. Sandboxed agent runs emit or are normalized into ATIF. The same ATIF trajectory feeds the default metrics, optional BYOG metrics, stakeholder dimensions, and the paired Skill Lift report used for review. Runtime evaluation flow. Evaluation assets feed the ACES adapter and task emitter. The adapter creates paired with-skill and baseline conditions that run across sandboxed agent harnesses. Agent outputs are normalized into ATIF trajectories. ATIF feeds six default metrics and optional BYOG metrics, which map to five stakeholder dimensions and a Skill Lift report.

Panel of Figure 4 — Figure 4. Evaluation-dataset creation and refinement. Skill authors create or update evals/evals.json while building the skill: they can write it directly, bootstrap it from skill artifacts, refine it with author intent or saved ATIF trajectories, and then use it for paired Skill Lift evaluation. Compact flow diagram. Skill artifacts feed dataset creation, which produces evals/evals.json. Developers review prompts, expected behavior, truth, and edge cases. Live evaluation produces a paired Skill Lift report and can feed a trajectory-grounded refinement loop.

Dimension	Source metric(s)
Security	security .
Correctness	accuracy .
Discoverability	skill_execution .
Effectiveness	goal_accuracy and behavior_check .
Efficiency	skill_efficiency .

Table 4. Five stakeholder dimensions and their source metrics in the default ACES metric set.

Panel of Figure 5 — Figure 5. The ACES adapter and task emitter generate ephemeral per-task environments from a skill directory and its evaluation assets. The same adapter produces both with-skill tasks and baseline tasks with the target skill withheld, stages configured supporting skills, injects the verifier, and schedules multiple agents in parallel against the same task set. Results are collected per agent for Skill Lift computation. Architecture diagram. A skill directory and evaluation assets flow into an ACES adapter and task-emitter box. The adapter translates assets into paired tasks, stages skills, injects the verifier, and emits per-task Dockerfiles, verifiers, and instruction files. Three agent boxes run in parallel below the adapter, each receiving the generated tasks; their outputs flow into a results box with per-agent summary, lift, and trajectory files.

Panel of Figure 6 — Figure 6. Isolation versus group testing for the public git-skill . Isolation (left) places only the target skill in the workspace, so the agent has no choice but to use it; the resulting lift measures the skill’s content contribution. Group (right) places the same target alongside four configured supporting or decoy skills ( api-debugger , log-triage , config-validator , release-planner ); the agent must select git-skill from the configured workspace before executing. The difference between the two lifts is the routing premium , a direct measure of how well the skill’s name and description let the agent discriminate it from neighbors. Side-by-side diagram. On the left, an Isolation workspace contains only git-skill; the user issues a request, the agent has only one skill to use, and the resulting Lift measures content alone. On the right, a Group workspace contains git-skill plus four configured supporting or decoy skills named api-debugger, log-triage, config-validator, and release-planner; the agent selects git-skill from the configured workspace; the resulting Lift measures content plus routing. A dashed annotation below states that the difference between the two lifts equals the routing premium.

Panel of Figure 7 — Figure 7. Evaluation-native skill development. Evaluation assets stay with the skill, and repository automation turns each skill change into review evidence before merge. A compact five-step flow: skill change, evals assets, scan and security, optional live run, and review evidence.

Category	# Skills	Mean Structural	Median Structural
System Access	49	78.7	79.0
Deployment	41	78.8	78.8
Platform	29	79.9	80.0
Data Infra	21	79.8	80.0
Troubleshooting	3	82.1	83.0
Dev-Tooling	1	80.5	80.5

Table 5. Corpus characterization by category (

n{=}145

Evidence item	Count
Scanned skills	145
Unique live-eval skill variants	89
Scaling-study variants excluded	25
Production skills in main inventory	64
Production skills with scored paired cases	58
Primary agent harnesses	4

Table 6. Evidence and filtering inventory for the production analysis.

Figure 8. Paired mean Skill Lift for the six default ACES metrics plus overall score, with 95% confidence intervals over 947 paired task cases. Parentheses show the fraction of paired cases with positive lift for that metric. Horizontal bar chart showing positive paired Skill Lift for overall score and the six metrics. Skill execution, behavior checking, and skill efficiency have the largest metric lifts.

Figure 9. Compact Skill Lift heatmap over the 64 production skills in the trial-file inventory and four primary harnesses. Rows are anonymized skill identifiers; color encodes paired overall Skill Lift. The 947 scored paired cases come from 58 of these skills. Compact heatmap with anonymized skills as rows and agent harnesses as columns. Most cells are positive, but the color intensity varies by skill and harness.

Figure 10. How Skill Lift appears in a live evaluation report. (a) For each agent harness, ACES runs isolated with-skill and baseline jobs from the same evaluation assets, records rewards, and reports the paired difference as Skill Lift. (b) A compact harness–model view keeps the valid agent/model cells visible. (c) A same-harness Codex model sweep shows that model changes alter both absolute rewards and marginal lift; stronger baselines can reduce measured skill value even when the with-skill score remains high. Three-panel figure. The first panel is a horizontal bar chart showing with-skill and baseline rewards for each harness, with the difference labeled as Skill Lift. The second panel is a horizontal paired-bar chart for valid harness-model cells, including Claude Code with Opus 4.6 and Codex with three GPT models. The third panel is a line chart for the Codex harness across three models, showing with-skill and baseline rewards and the vertical gap between them as Skill Lift.

Case	Trace or log observation	Paper signal
Positive routing and workflow	A public technical workflow skill was activated through the expected tool path, made 1/1 productive tool calls, and satisfied 5/5 expected behaviors.	Shows a concrete mechanism behind high lift: correct routing, tool use, and author-specified workflow compliance.
Partial success	An authentication workflow diagnosed missing OAuth tokens, but did not present the actual authorization URL and did not rerun verification after the missing prerequisite was identified.	Shows why trajectory and behavior checks add nuance beyond final answer text: the agent made progress but skipped a required verification step.
Actionable author feedback	A multi-step artifact workflow trace produced suggestions for explicit polling guidance, retry logic, and clearer remediation steps after an operation remained pending.	Shows that live evaluation can become author feedback, not just a pass/fail score.
Security separation	The security backfill found secret-like strings, destructive command patterns, unauthorized-path access, and missing-trajectory cases across the live artifacts.	Motivates keeping security as a separate metric from behavior_check , so safety failures are visible even when task behavior otherwise looks acceptable.

Table 7. Anonymized qualitative evidence from saved trajectories and run logs.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Contract
Security	Evidence: deterministic scan of trace commands, paths, observations, and outputs; repository scanners remain separate. Scale: 1 absent unsafe patterns, reduced by findings. Weight/view: $1/6$ ; Security. Caveat: pattern false positives/negatives; missing trajectories cannot be fully scanned.
Skill execution	Evidence: expected activation, script/tool use, and workflow order. Scale: fraction of checks completed. Weight/view: $1/6$ ; Discoverability and Effectiveness. Caveat: equivalent workflows absent from the dataset may be under-credited.
Skill efficiency	Evidence: skills read, relevant versus irrelevant skill/tool calls, and productive calls. Scale: routing/tool-efficiency fraction. Weight/view: $1/6$ ; Efficiency and Discoverability. Caveat: exploration can look inefficient; visible-skill count changes routing pressure.
Accuracy	Evidence: LLM-as-Judge assessment of answer and trajectory against task and ground truth. Scale: $[0,1]$ . Weight/view: $1/6$ ; Correctness. Caveat: judge-model and wording sensitivity.
Goal accuracy	Evidence: reference-based outcome match. Scale: RAGAS/LLM goal score in $[0,1]$ . Weight/view: $1/6$ ; Correctness and Effectiveness. Caveat: artificially low when tasks require unavailable network or product endpoints.
Behavior check	Evidence: LLM yes/no judgments of ordered expected_behavior assertions against the ATIF trajectory. Scale: fraction satisfied. Weight/view: $1/6$ ; Effectiveness, plus Security for safety assertions. Caveat: phrasing-sensitive; can reward process despite weak final success.

Appendix

Table 8. Default ACES metric definitions. All metrics are normalized to

[0,1]

before aggregation; higher is better.

Dimension	SkillsBench	Skills in the Wild	ACES
Primary unit	Shared tasks with curated skills	Queries against a large real-world corpus	Repository skill or product capability package under review
Paired skill/no-skill	Yes	Yes	Yes
Retrieval pressure	Bundle-size experiments	Yes	Configurable sibling/decoy skills
CI gate for arbitrary production-skill changes	No	No	Yes
Cross-harness trajectory contract	No	No	Yes, via ATIF

Appendix

Table 9. Scope comparison with two representative skill-evaluation studies.

Case	Static, live, and trace evidence
Public OpenClaw sanitization	Static: 11/11; quality 89/100; security scan skipped; code check missed renderer. Live ( $n=1$ , local): renderer retained a synthetic nvapi- canary in an intermediate artifact; the agent repaired the final file. Artifact: canary absence was 0 with skill, 1 at baseline. Action: add NVIDIA-token redaction and fail-closed tests.
Enterprise authentication workflow	Static: 84 / 88; both pass. Live: positive lift, but incomplete workflow. Trace: diagnosed missing OAuth tokens but omitted the authorization URL and did not rerun verification, yielding concrete authoring actions.
Enterprise artifact workflow	Static: 82.8 / 75; both pass. Live: accuracy $-0.0667$ , goal accuracy $-0.1667$ , behavior check $-0.3333$ in one harness. Trace: exposed missing polling/retry guidance and a less complete workflow.

Appendix

Table 10. Static review and live evidence answer different questions. Tier 1/Tier 2 document scores use 0–100; live deltas are matched with-skill minus baseline.

Enterprise AI Agents

See all 15 papers

Jun 21, 2026

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.

Dexu Yu, Youhua Li, Zhaoyang Guan +12

Northeastern University · City University of Hong Kong · DeciLix Lab +8

AI Agent Auditing Automated Evaluation Agent Skill Security Auditing LLM Agent Skill Evaluation

Ask PDF

Jun 16, 2026

A Framework for Evaluating Agentic Skills at Scale

Agent skills -- structured, reusable knowledge artifacts that augment LLM agent capabilities -- have been rapidly adopted in industry, yet their cross-domain impact and use across commercial and open-source models remain under-studied, and no reusable methodology exists for evaluating an individual skill. In this work, we present an evaluation framework that lets a skill author construct realistic tasks to rigorously assess the aspects of a skill that matter most to them, and that estimates skill utility by solving those tasks. Further, we apply our evaluation approach at scale to 500 real-world skills, generating 1,000 tasks derived from the skills' content, along with instruction-following and goal-completion scoring rubrics. Using these metrics, we evaluate how 19 agent-model configurations, both proprietary and open-source, perform on the tasks. Our results show that models vary widely in how closely they adhere to the instructions encoded in skills, leading to substantial differences in their performance gains. Furthermore, we show that access to a skill significantly changes model behavior compared to the no-skill setup, providing an essential mechanism for encoding opinionated workflows into LLM agents. We release our evaluation dataset to support future work on agent skills.

Maksim Shaposhnikov, Nicolas Fortuin, Simon Stipcich +3

Tessl London, United Kingdom

LLM Agent Skill Evaluation LLM Agent Evaluation Instruction Following Agent Benchmarks

Ask PDF

Mar 28, 2026

SkillTester: Benchmarking Utility and Security of Agent Skills

This technical report presents SkillTester, a tool for evaluating the utility and security of agent skills. Its evaluation framework combines paired baseline and with-skill execution conditions with a separate security probe suite. Grounded in a comparative utility principle and a user-facing simplicity principle, the framework normalizes raw execution artifacts into a utility score, a security score, and a three-level security status label. More broadly, it can be understood as a comparative quality-assurance harness for agent skills in an agent-first world. The public service is deployed at https://skilltester.ai, and the broader project is maintained at https://github.com/skilltester-ai/skilltester.

Leye Wang, Zixing Wang, Anjie Xu

Key Laboratory of High Confidence Software Technologies, Ministry of Education, Peking University, Beijing, China · Northwestern Polytechnical University, Xi’an, China

LLM Agent Skill Evaluation AI Agent Security Benchmarks Agent Skill Security Auditing

Ask PDF

Abstract

Enterprise AI Agents

Explore similar work

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

A Framework for Evaluating Agentic Skills at Scale

SkillTester: Benchmarking Utility and Security of Agent Skills