cs.AIAug 20, 2026
SaveEvaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills
Organizations: NVIDIA
Abstract
Enterprise agent programs are moving from prototypes into production, where reusable skills, tools, and workflow packages must be reviewed with evidence rather than prose. Current gates often scan these artifacts for structure, style, and security, but they do not answer the deployment question: does the capability package help a live agent complete enterprise tasks under the same model, sandbox, and grading policy? We present ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts. ACES runs paired live trials with and without a target skill, normalizes trajectories into the Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics, and reports Skill Lift: the target skill's added value for a fixed task, harness, workspace, and scorer. The same protocol supports product-owned task suites that compare baseline, skill, bundle, team-skill, and plugin targets. On 145 real skills from internal enterprise repositories and public catalogs, scan-only gates surface useful authoring issues but measure complementary facets (structural versus LLM-judge Spearman ). Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95% paired-case CI [0.1967, 0.2301]); mean outcome-only lift, the average of accuracy and goal accuracy, is 0.1799. Composite lift is positive in 72.8% of paired cases. The largest process-metric gains appear in skill execution, behavior check, and skill efficiency---signals about discovery, routing, workflow following, and tool use that document scans cannot observe. An open-source implementation of the methodology is available in NVIDIA SkillEvaluator.
Figures & tables
| Dimension (weight) | Rule families |
|---|---|
| Correctness (0.35) | Frontmatter contract ( name , description , version , author , tags , tools ); name/directory match; Instructions section; run_script mention; script-file presence; header-table documentation; injection-suspicious characters. |
| Discoverability (0.25) | Description length (50–150 preferred; 200 flagged as overlong); trigger vocabulary ( use , when , for ); avoidance of vague wording; explicit negative/boundary phrasing; directory-name conventions; tone (first/second person); “WHEN to use” guidance. |
| Reliability (0.25) | Error-handling vocabulary; input-validation hints in scripts/* ; Prerequisites , Limitations , Troubleshooting sections; MCP-connection guidance when MCP is declared. |
| Efficiency (0.15) | Token budget; repeated lines; list-format instructions; nested-reference depth; corporate/hedge language; oversize-skill warnings. |
Table 1. Static check families per dimension ( rules total). Each family contains multiple rules that fire independently; each rule deducts from its dimension score.
| Criterion | What the judge is asked |
|---|---|
| description clarity | Does the description communicate what the skill does in one pass, without ambiguity? |
| instruction clarity | Are the instructions specific, ordered, and executable by an agent reading them cold? |
| example quality | Do worked examples cover the prompt variations an agent will actually see? |
| documentation completeness | Are the scripts, inputs, and expected outcomes documented? |
| scope definition | Is the boundary between what this skill does and what it does not explicit? |
| professional tone | Is the writing direct, concise, and free of marketing or hedge language? |
Table 2. LLM-as-Judge rubric criteria (9 base + 1 conditional). Each is scored 0–10 with free-form reasoning.
Figure 1. Doc-scan evidence motivates, but does not replace, live evaluation. (a) The – structural score aggregates roughly 50 deterministic rule checks across Correctness, Discoverability, Reliability, and Efficiency for real skills. The default -point gate is permissive: 94.5% of skills clear it, but only 48.9% reach the -point line. (b) Structural scores and LLM-judge rubric scores disagree on the same corpus (Spearman , Pearson ). Together, the panels show that document scanning surfaces real authoring issues, but neither scan-only axis observes discovery, tool use, workflow order, or task success. Two-panel figure. The left panel is a histogram of structural scores for 145 real skills from internal enterprise repositories and public catalogs, with vertical markers at the 70-point gate and the 80-point line. Most skills are between 75 and 85, and annotations note that 94.5 percent pass the gate while 48.9 percent reach 80. The right panel is a scatter plot of structural score versus LLM-judge score for the same skills, colored by judge, with a y-equals-x reference line and annotations showing weak Spearman and Pearson correlation.
Figure 2. Top-10 most frequently violated static rules across skills, colored by structural dimension. Four of the five most common violations are frontmatter-contract rules that 91–99% of skills fail. Horizontal bar chart of the ten most frequently violated static rules, with percentage of skills on the x-axis and rule description on the y-axis. Bars are color-coded by evaluation dimension. The top four bars exceed 90 percent and are all frontmatter-related rules.
| Approach | Art. | Val. | Multi | CI | HITL |
|---|---|---|---|---|---|
| SkillsBench ( Chen et al., 2026b ) | ✓ | ✓ | |||
| SkillTester ( Wang et al., 2026 ) | ✓ | ||||
| In-the-Wild ( Liu and others, 2026 ) | ✓ | ||||
| Anthropic skill-creator ( Anthropic, 2025d ) | ✓ | ||||
| Terminal-Bench ( Merrill and Terminal-Bench Team, 2026 ) | ✓ | ||||
| This work (ACES) | ✓ | ✓ | ✓ | ✓ | ✓ |
Table 3. Capability gap across prior skill-evaluation tooling. = covered; = not covered. Artifact = scores the skill artifact directly; Value = measures marginal value via lift; Multi = evaluates across multiple agent harnesses; CI = CI-native deployment; HITL = human-in-the-loop dataset refinement.
Figure 3. ACES runtime evaluation flow. Evaluation assets feed the ACES adapter and task emitter, which creates paired with-skill and baseline runs while holding configured support skills fixed. Sandboxed agent runs emit or are normalized into ATIF. The same ATIF trajectory feeds the default metrics, optional BYOG metrics, stakeholder dimensions, and the paired Skill Lift report used for review. Runtime evaluation flow. Evaluation assets feed the ACES adapter and task emitter. The adapter creates paired with-skill and baseline conditions that run across sandboxed agent harnesses. Agent outputs are normalized into ATIF trajectories. ATIF feeds six default metrics and optional BYOG metrics, which map to five stakeholder dimensions and a Skill Lift report.
Figure 4. Evaluation-dataset creation and refinement. Skill authors create or update evals/evals.json while building the skill: they can write it directly, bootstrap it from skill artifacts, refine it with author intent or saved ATIF trajectories, and then use it for paired Skill Lift evaluation. Compact flow diagram. Skill artifacts feed dataset creation, which produces evals/evals.json. Developers review prompts, expected behavior, truth, and edge cases. Live evaluation produces a paired Skill Lift report and can feed a trajectory-grounded refinement loop.
| Dimension | Source metric(s) |
|---|---|
| Security | security . |
| Correctness | accuracy . |
| Discoverability | skill_execution . |
| Effectiveness | goal_accuracy and behavior_check . |
| Efficiency | skill_efficiency . |
Table 4. Five stakeholder dimensions and their source metrics in the default ACES metric set.
Figure 5. The ACES adapter and task emitter generate ephemeral per-task environments from a skill directory and its evaluation assets. The same adapter produces both with-skill tasks and baseline tasks with the target skill withheld, stages configured supporting skills, injects the verifier, and schedules multiple agents in parallel against the same task set. Results are collected per agent for Skill Lift computation. Architecture diagram. A skill directory and evaluation assets flow into an ACES adapter and task-emitter box. The adapter translates assets into paired tasks, stages skills, injects the verifier, and emits per-task Dockerfiles, verifiers, and instruction files. Three agent boxes run in parallel below the adapter, each receiving the generated tasks; their outputs flow into a results box with per-agent summary, lift, and trajectory files.
Figure 6. Isolation versus group testing for the public git-skill . Isolation (left) places only the target skill in the workspace, so the agent has no choice but to use it; the resulting lift measures the skill’s content contribution. Group (right) places the same target alongside four configured supporting or decoy skills ( api-debugger , log-triage , config-validator , release-planner ); the agent must select git-skill from the configured workspace before executing. The difference between the two lifts is the routing premium , a direct measure of how well the skill’s name and description let the agent discriminate it from neighbors. Side-by-side diagram. On the left, an Isolation workspace contains only git-skill; the user issues a request, the agent has only one skill to use, and the resulting Lift measures content alone. On the right, a Group workspace contains git-skill plus four configured supporting or decoy skills named api-debugger, log-triage, config-validator, and release-planner; the agent selects git-skill from the configured workspace; the resulting Lift measures content plus routing. A dashed annotation below states that the difference between the two lifts equals the routing premium.
Figure 7. Evaluation-native skill development. Evaluation assets stay with the skill, and repository automation turns each skill change into review evidence before merge. A compact five-step flow: skill change, evals assets, scan and security, optional live run, and review evidence.
| Category | # Skills | Mean Structural | Median Structural |
|---|---|---|---|
| System Access | 49 | 78.7 | 79.0 |
| Deployment | 41 | 78.8 | 78.8 |
| Platform | 29 | 79.9 | 80.0 |
| Data Infra | 21 | 79.8 | 80.0 |
| Troubleshooting | 3 | 82.1 | 83.0 |
| Dev-Tooling | 1 | 80.5 | 80.5 |
Table 5. Corpus characterization by category ( ).
| Evidence item | Count |
|---|---|
| Scanned skills | 145 |
| Unique live-eval skill variants | 89 |
| Scaling-study variants excluded | 25 |
| Production skills in main inventory | 64 |
| Production skills with scored paired cases | 58 |
| Primary agent harnesses | 4 |
Table 6. Evidence and filtering inventory for the production analysis.
Figure 8. Paired mean Skill Lift for the six default ACES metrics plus overall score, with 95% confidence intervals over 947 paired task cases. Parentheses show the fraction of paired cases with positive lift for that metric. Horizontal bar chart showing positive paired Skill Lift for overall score and the six metrics. Skill execution, behavior checking, and skill efficiency have the largest metric lifts.
Figure 9. Compact Skill Lift heatmap over the 64 production skills in the trial-file inventory and four primary harnesses. Rows are anonymized skill identifiers; color encodes paired overall Skill Lift. The 947 scored paired cases come from 58 of these skills. Compact heatmap with anonymized skills as rows and agent harnesses as columns. Most cells are positive, but the color intensity varies by skill and harness.
Figure 10. How Skill Lift appears in a live evaluation report. (a) For each agent harness, ACES runs isolated with-skill and baseline jobs from the same evaluation assets, records rewards, and reports the paired difference as Skill Lift. (b) A compact harness–model view keeps the valid agent/model cells visible. (c) A same-harness Codex model sweep shows that model changes alter both absolute rewards and marginal lift; stronger baselines can reduce measured skill value even when the with-skill score remains high. Three-panel figure. The first panel is a horizontal bar chart showing with-skill and baseline rewards for each harness, with the difference labeled as Skill Lift. The second panel is a horizontal paired-bar chart for valid harness-model cells, including Claude Code with Opus 4.6 and Codex with three GPT models. The third panel is a line chart for the Codex harness across three models, showing with-skill and baseline rewards and the vertical gap between them as Skill Lift.
| Case | Trace or log observation | Paper signal |
|---|---|---|
| Positive routing and workflow | A public technical workflow skill was activated through the expected tool path, made 1/1 productive tool calls, and satisfied 5/5 expected behaviors. | Shows a concrete mechanism behind high lift: correct routing, tool use, and author-specified workflow compliance. |
| Partial success | An authentication workflow diagnosed missing OAuth tokens, but did not present the actual authorization URL and did not rerun verification after the missing prerequisite was identified. | Shows why trajectory and behavior checks add nuance beyond final answer text: the agent made progress but skipped a required verification step. |
| Actionable author feedback | A multi-step artifact workflow trace produced suggestions for explicit polling guidance, retry logic, and clearer remediation steps after an operation remained pending. | Shows that live evaluation can become author feedback, not just a pass/fail score. |
| Security separation | The security backfill found secret-like strings, destructive command patterns, unauthorized-path access, and missing-trajectory cases across the live artifacts. | Motivates keeping security as a separate metric from behavior_check , so safety failures are visible even when task behavior otherwise looks acceptable. |
Table 7. Anonymized qualitative evidence from saved trajectories and run logs.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | Contract |
|---|---|
| Security | Evidence: deterministic scan of trace commands, paths, observations, and outputs; repository scanners remain separate. Scale: 1 absent unsafe patterns, reduced by findings. Weight/view: ; Security. Caveat: pattern false positives/negatives; missing trajectories cannot be fully scanned. |
| Skill execution | Evidence: expected activation, script/tool use, and workflow order. Scale: fraction of checks completed. Weight/view: ; Discoverability and Effectiveness. Caveat: equivalent workflows absent from the dataset may be under-credited. |
| Skill efficiency | Evidence: skills read, relevant versus irrelevant skill/tool calls, and productive calls. Scale: routing/tool-efficiency fraction. Weight/view: ; Efficiency and Discoverability. Caveat: exploration can look inefficient; visible-skill count changes routing pressure. |
| Accuracy | Evidence: LLM-as-Judge assessment of answer and trajectory against task and ground truth. Scale: . Weight/view: ; Correctness. Caveat: judge-model and wording sensitivity. |
| Goal accuracy | Evidence: reference-based outcome match. Scale: RAGAS/LLM goal score in . Weight/view: ; Correctness and Effectiveness. Caveat: artificially low when tasks require unavailable network or product endpoints. |
| Behavior check | Evidence: LLM yes/no judgments of ordered expected_behavior assertions against the ATIF trajectory. Scale: fraction satisfied. Weight/view: ; Effectiveness, plus Security for safety assertions. Caveat: phrasing-sensitive; can reward process despite weak final success. |
Table 8. Default ACES metric definitions. All metrics are normalized to before aggregation; higher is better.
| Dimension | SkillsBench | Skills in the Wild | ACES |
|---|---|---|---|
| Primary unit | Shared tasks with curated skills | Queries against a large real-world corpus | Repository skill or product capability package under review |
| Paired skill/no-skill | Yes | Yes | Yes |
| Retrieval pressure | Bundle-size experiments | Yes | Configurable sibling/decoy skills |
| CI gate for arbitrary production-skill changes | No | No | Yes |
| Cross-harness trajectory contract | No | No | Yes, via ATIF |
Table 9. Scope comparison with two representative skill-evaluation studies.
| Case | Static, live, and trace evidence |
|---|---|
| Public OpenClaw sanitization | Static: 11/11; quality 89/100; security scan skipped; code check missed renderer. Live ( , local): renderer retained a synthetic nvapi- canary in an intermediate artifact; the agent repaired the final file. Artifact: canary absence was 0 with skill, 1 at baseline. Action: add NVIDIA-token redaction and fail-closed tests. |
| Enterprise authentication workflow | Static: 84 / 88; both pass. Live: positive lift, but incomplete workflow. Trace: diagnosed missing OAuth tokens but omitted the authorization URL and did not rerun verification, yielding concrete authoring actions. |
| Enterprise artifact workflow | Static: 82.8 / 75; both pass. Live: accuracy , goal accuracy , behavior check in one harness. Trace: exposed missing polling/retry guidance and a less complete workflow. |
Table 10. Static review and live evidence answer different questions. Tier 1/Tier 2 document scores use 0–100; live deltas are matched with-skill minus baseline.