Agent skills package instructions, scripts, and resources for specialized tasks and are increasingly distributed through public Web repositories. However, existing skill benchmarks primarily focus on downstream task execution, insufficiently assessing specification quality, behavioral robustness, semantic alignment, and responsibility boundaries. Their reliance on task-specific tools, environments, and repeated executions limits scalability and pre-deployment assessment. To address these challenges, we introduce SkillCT, a benchmark that broadens skill assessment from downstream utility to a multidimensional characterization of skill quality. SkillCT evaluates skills across nine complementary dimensions spanning three aspects: Static Assessment, which captures intrinsic properties of the skill specification; Dynamic Assessment, which measures whether the induced behavior remains stable across backbone models and request variations; and Human-Assisted Assessment, which examines behavior-chain consistency and task granularity. These dimensions provide a broader view of skill quality than execution success. Applying SkillCT to a public skill corpus establishes a reference benchmark for analyzing quality weaknesses and relationships across dimensions. Building on this benchmark, we propose an execution-free quality evaluation baseline that predicts the multidimensional quality profiles of unseen skills by retrieving and leveraging similar reference skills, without executing the skills under evaluation. This reduces token consumption and evaluation time, while preserving consistency with the full assessment. Furthermore, SkillCT-guided optimization improves downstream task success and execution efficiency, demonstrating its value for skill refinement. These findings highlight the effectiveness of SkillCT in supporting scalable skill assessment, optimization, and repository maintenance.
Figures & tables
Figure 1. Overview of SkillCT. a , Overall workflow for assessing public agent skills and producing multidimensional quality profiles. b , SkillCT assessment framework and execution-free baseline. Three assessment modules establish the reference benchmark, which supports quality estimation for new skill profiles through reference retrieval. c , Nine skill quality dimensions covered by SkillCT. d , Distribution of the 4,098-skill reference benchmark and the independent 3,050-skill evaluation set across 14 categories.
Figure 2. Execution-free assessment across different agent skill categories.
Metric
Score Distribution
Sample Proportion (%)
Assessment
Assessment Dimension
Mean
Std.
P25
P50
P75
P95
≥0.8
<0.5
≈0
≈1
Overall Score
0.801
0.046
0.776
0.807
0.834
0.864
56.4
0.0
0.0
0.0
Static Assessment
Static Overall
0.740
0.067
0.695
0.751
0.790
0.832
19.1
0.1
0.0
0.0
Structural Completeness
0.780
0.137
0.675
0.800
0.900
0.975
53.7
2.1
0.0
0.0
Conciseness
0.874
0.047
0.863
0.885
0.901
0.923
93.4
0.0
0.0
0.0
Description-Content Consistency
0.565
0.106
0.495
0.568
0.634
0.734
1.7
26.2
0.0
0.0
Table 1. Distributional statistics of SkillCT scores across the evaluated agent skills.
Source
Static Assessment
Dynamic Assessment
Human-Assisted Assessment
Structural Completeness
Conci- seness
Description- Content Consistency
Cross-Model Robustness
Intent- Perturbation Stability
Capability- Boundary Handling
Effi- ciency
Behavior-Chain Semantic Consistency
Task Abstraction Granularity
Human
0.814
0.751
0.919
0.848
0.826
0.554
0.885
0.874
0.713
Execution-Free
0.806
0.859
0.567
0.761
0.819
0.964
0.908
0.920
0.661
Full
0.806
0.859
0.567
0.730
0.841
0.916
0.881
0.922
0.653
Table 2. Comparison of Human Assessment, Execution-Free SkillCT Assessment, and Full SkillCT Assessment.
Figure 3. Spearman correlations across dimensions.
Setting
Static Assessment
Dynamic Assessment
Human-Assisted Assessment
Overall
Structural Complete- ness
Conci- seness
Description- Content Consistency
Cross-Model Robust- ness
Intent- Perturbation Stability
Capability- Boundary Handling
Effi- ciency
Behavior-Chain Semantic Consistency
Task Abstraction Granularity
Original
0.813
0.857
0.564
0.760
0.816
0.961
0.909
0.919
0.658
0.806
SkillOpt
0.904
0.864
0.730
0.799
0.827
0.981
0.923
0.929
0.703
0.851
Model (Original → SkillOpt)
Task Success Rate (%) ( ↑ )
Avg. Execution Steps ( ↓ )
Avg. Input Tokens ( ↓ )
Avg. Output Tokens ( ↓ )
GLM-4.5-Air
91.9 → 96.9
7.47 → 5.79
1388 → 1457
828 → 706
DeepSeek-V4-Flash
87.3 → 95.1
8.23 → 6.72
1391 → 1447
905 → 813
Table 3. SkillCT assessment and task execution results before and after skill optimization.
Model
Cross- Model Robustness
Intent- Perturbation Stability
Capability- Boundary Handling
Efficiency
Behavior-Chain Semantic Consistency
Task Abstraction Granularity
Overall
Seed-1.6-Flash
0.769
0.804
0.924
0.908
0.882
0.705
0.801
GLM-4.5-Air
0.702
0.723
0.900
0.836
0.914
0.734
0.781
Qwen3.5-Flash
0.724
0.767
0.933
0.869
0.952
0.762
0.803
DeepSeek-V4-Flash
0.685
0.712
0.884
0.820
0.914
0.720
0.773
GPT-5
0.724
0.754
0.909
0.846
0.957
0.746
0.795
Dimensions with identical scores across models
Table 4. Cross-LLM Assessment Consistency in SkillCT.
Assessment Dimension
MAE ↓
RMSE ↓
Bias
Pearson r↑
Spearman ρ↑
Cross-Model Robustness
0.037
0.057
0.004
0.245
0.249
Intent-Perturbation Stability
0.051
0.071
-0.004
0.339
0.352
Capability-Boundary Handling
0.022
0.039
-0.003
0.670
0.721
Efficiency
0.056
0.075
0.003
0.291
0.325
Behavior-Chain Semantic Consistency
0.088
0.134
0.014
0.565
0.482
Task Abstraction Granularity
0.130
0.174
0.005
0.517
0.442
Table 5. Holdout Validation of Execution-Free SkillCT.
Method
MAE ↓
Pearson r↑
Spearman ρ↑
Category Mean
0.078
0.165
0.161
Semantic Retrieval
0.069
0.372
0.367
TF-IDF Retrieval (Ours)
0.064
0.438
0.429
Table 6. Comparison of execution-free estimation methods on the holdout set across six estimated dimensions.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Static Assessment
Dynamic Assessment
Human-Assisted Assessment
Overall
Category
N
Structural Complete- ness
Conci- seness
Description- Content Consistency
Cross- Model Robustness
Intent- Perturbation Stability
Capability- Boundary Handling
Effi- ciency
Behavior-Chain Semantic Consistency
Task Abstraction Granularity
Finance
275
0.755
0.882
0.553
0.743
0.772
0.912
0.885
0.872
0.732
0.790
Communication
275
0.775
0.875
0.560
0.701
0.728
0.857
0.848
0.891
0.716
0.772
Knowledge
329
0.793
0.886
0.567
0.793
0.805
0.969
0.929
0.912
0.734
0.821
Productivity
275
0.745
0.883
0.560
0.716
0.749
0.879
0.862
0.873
0.733
0.778
Creative
268
0.813
0.898
0.577
0.773
0.801
0.931
0.893
0.911
0.769
0.819
Appendix
Table 7. Assessment results across different Agent Skill categories in the SkillCT reference benchmark.
Category
Static Assessment
Dynamic Assessment
Human-Assisted Assessment
Structural Completeness
Conciseness
Description–Content Consistency
Cross-Model Robustness
Intent-Perturbation Stability
Capability-Boundary Handling
Efficiency
Behavior-Chain Semantic Consistency
Task Abstraction Granularity
Finance
0.790/0.892/0.888
0.868/0.874/0.885
0.541/0.723/0.703
0.757/0.796/0.807
0.827/0.838/0.826
0.974/0.990/0.959
0.919/0.932/0.903
0.921/0.931/0.948
0.659/0.704/0.671
Communication
0.789/0.891/0.886
0.851/0.858/0.869
0.545/0.725/0.712
0.765/0.804/0.803
0.820/0.831/0.768
0.975/0.991/0.983
0.910/0.924/0.882
0.905/0.916/0.917
0.632/0.679/0.653
Knowledge
0.784/0.889/0.885
0.857/0.864/0.863
0.564/0.730/0.720
0.750/0.789/0.800
0.790/0.802/0.821
0.951/0.973/0.953
0.905/0.919/0.918
0.922/0.932/0.948
0.660/0.705/0.686
Productivity
0.751/0.873/0.861
0.863/0.869/0.880
0.528/0.719/0.704
0.737/0.777/0.773
0.793/0.805/0.780
0.927/0.956/0.931
0.879/0.895/0.883
0.913/0.923/0.906
0.637/0.683/0.662
Creative
0.789/0.891/0.884
0.864/0.870/0.881
0.583/0.736/0.730
0.770/0.808/0.804
0.823/0.834/0.886
0.972/0.988/0.981
0.934/0.946/0.970
0.932/0.941/0.968
0.681/0.725/0.744
Appendix
Table 8. Category-wise SkillCT assessment scores before and after Skill optimization (50 Skills per category). Each cell reports Original / GLM-4.5-Air / DeepSeek-V4-Flash.
Model Pair
Overall Score
Dimension ( ρ )
ρ
MAE
IPS
BCSC
TAG
Seed–GLM
0.978
0.021
0.873
0.763
0.887
Seed–Qwen
0.991
0.003
0.974
0.915
0.970
Seed–DeepSeek
0.990
0.028
0.976
0.974
0.998
Seed–GPT-5
0.991
0.007
0.955
0.904
0.977
GLM–Qwen
0.991
0.022
0.807
0.833
0.908
Appendix
Table 9. Category-level assessment agreement across LLMs on the SkillCT reference benchmark.
Criterion
Condition
Weight
Frontmatter
YAML frontmatter is present
2.0
Name
A Skill name is provided
1.5
Description
A description longer than 10 characters is provided
2.5
Description length
Description length is between 10 and 500 characters
1.0
Instructional body
The instructional body contains more than 50 characters
2.0
Section structure
At least two second-level sections are present
1.5
Appendix
Table 10. Structural Completeness criteria and weights.
Request Type
Response
Score
Unrelated
Identified as unsupported with an explanation
1.0
Identified as unsupported without an explanation
0.8
Treated as supported
0.0
Adversarial
Identified as unsupported with an explanation
1.0
Identified as unsupported without an explanation
0.8
Treated as supported
0.0
Appendix
Table 11. Scoring rules for Capability-Boundary Handling.
Quantity
Operating Interval
Upper Threshold
Weight
Prompt tokens
500–2000
8000
0.30
Plan steps
3–7
20
0.25
Response latency
500–3000 ms
15000 ms
0.20
Tools or commands
1–5
15
0.25
Appendix
Table 12. Normalization settings for Efficiency.
Score
Criterion
1.00
All actions directly support the declared objective, with no irrelevant or redundant operations.
0.75
Most actions support the objective, with only one or two peripheral actions.
0.50
Only part of the behavior chain supports the objective, with clear irrelevant or redundant operations.
0.25
Most actions are weakly related or unrelated to the declared objective, resulting in a poorly aligned behavior chain.
0.00
The behavior chain is almost entirely unrelated to the declared objective.
Appendix
Table 13. Scoring rubric for Behavior-Chain Semantic Consistency.
Score
Criterion
1.00
The Skill represents a complete and meaningful unit of work with a clear and self-contained responsibility.
0.75
The task scope is generally appropriate but is slightly narrower or broader than the intended unit of work.
0.50
The responsibility is noticeably too narrow or too broad.
0.25
The task scope is severely misaligned and would substantially benefit from being split or combined with other functionality.
0.00
The Skill either reduces to an isolated operation with little additional procedure or attempts to cover an entire domain without a coherent task scope.
Appendix
Table 14. Scoring rubric for Task Abstraction Granularity.
Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.
Dexu Yu, Youhua Li, Zhaoyang Guan +12
Northeastern University · City University of Hong Kong · DeciLix Lab +8
Agent skills provide reusable knowledge that helps agents solve specialized tasks. As their use expands, evaluating skill quality becomes increasingly important. Existing evaluations often measure skill quality based on improvements in agent performance on specific tasks. However, a reusable skill may apply across multiple scenarios. Evaluation on a single downstream task mainly reflects the compatibility between a skill and the evaluated task, provides only a partial view of skill quality, and does not identify which aspects need improvement. We find that general properties of the SKILL.md document play an important role in skill quality. To evaluate these properties, we propose SkillEval, an interpretable framework for document-level skill evaluation. SkillEval produces interpretable scores for each property using a fixed, inspectable scoring direction. It further measures and reduces the influence of unrelated document features, so each score more specifically captures its intended semantic property. We use SkillEval to evaluate skills in controlled quality tests, showing that it reliably distinguishes skills of different quality. Its scores closely reflect downstream task performance, providing an early indication of skill effectiveness. We further explore SkillEval for diagnosing document weaknesses and guiding targeted revisions that improve the targeted properties and increase downstream task pass rates.
Jiahui Han, Qinuo Li, Ziheng Peng +6
Xi’an Jiaotong University · Shanghai AI Laboratory · Harbin Institute of Technology +2
This technical report presents SkillTester, a tool for evaluating the utility and security of agent skills. Its evaluation framework combines paired baseline and with-skill execution conditions with a separate security probe suite. Grounded in a comparative utility principle and a user-facing simplicity principle, the framework normalizes raw execution artifacts into a utility score, a security score, and a three-level security status label. More broadly, it can be understood as a comparative quality-assurance harness for agent skills in an agent-first world. The public service is deployed at https://skilltester.ai, and the broader project is maintained at https://github.com/skilltester-ai/skilltester.
Leye Wang, Zixing Wang, Anjie Xu
Key Laboratory of High Confidence Software Technologies, Ministry of Education, Peking University, Beijing, China · Northwestern Polytechnical University, Xi’an, China