SkillCT: A Multidimensional Benchmark and Execution-Free Baseline for Agent Skill Assessment

Mengyu Sun; Zunlong Zhou; Ziyuan Yang; Yi Zhang

cs.AIOct 28, 2026

SkillCT: A Multidimensional Benchmark and Execution-Free Baseline for Agent Skill Assessment

Authors: Mengyu Sun, Zunlong Zhou, Ziyuan Yang, Yi Zhang

Organizations: Sichuan University, Chengdu, China

Abstract

Agent skills package instructions, scripts, and resources for specialized tasks and are increasingly distributed through public Web repositories. However, existing skill benchmarks primarily focus on downstream task execution, insufficiently assessing specification quality, behavioral robustness, semantic alignment, and responsibility boundaries. Their reliance on task-specific tools, environments, and repeated executions limits scalability and pre-deployment assessment. To address these challenges, we introduce SkillCT, a benchmark that broadens skill assessment from downstream utility to a multidimensional characterization of skill quality. SkillCT evaluates skills across nine complementary dimensions spanning three aspects: Static Assessment, which captures intrinsic properties of the skill specification; Dynamic Assessment, which measures whether the induced behavior remains stable across backbone models and request variations; and Human-Assisted Assessment, which examines behavior-chain consistency and task granularity. These dimensions provide a broader view of skill quality than execution success. Applying SkillCT to a public skill corpus establishes a reference benchmark for analyzing quality weaknesses and relationships across dimensions. Building on this benchmark, we propose an execution-free quality evaluation baseline that predicts the multidimensional quality profiles of unseen skills by retrieving and leveraging similar reference skills, without executing the skills under evaluation. This reduces token consumption and evaluation time, while preserving consistency with the full assessment. Furthermore, SkillCT-guided optimization improves downstream task success and execution efficiency, demonstrating its value for skill refinement. These findings highlight the effectiveness of SkillCT in supporting scalable skill assessment, optimization, and repository maintenance.

Figures & tables

Figure 1. Overview of SkillCT. a , Overall workflow for assessing public agent skills and producing multidimensional quality profiles. b , SkillCT assessment framework and execution-free baseline. Three assessment modules establish the reference benchmark, which supports quality estimation for new skill profiles through reference retrieval. c , Nine skill quality dimensions covered by SkillCT. d , Distribution of the 4,098-skill reference benchmark and the independent 3,050-skill evaluation set across 14 categories.

Figure 2. Execution-free assessment across different agent skill categories.

Metric		Score Distribution						Sample Proportion (%)
Assessment	Assessment Dimension	Mean	Std.	P25	P50	P75	P95	$\geq 0.8$	$<0.5$	$\approx 0$	$\approx 1$
	Overall Score	0.801	0.046	0.776	0.807	0.834	0.864	56.4	0.0	0.0	0.0
Static Assessment	Static Overall	0.740	0.067	0.695	0.751	0.790	0.832	19.1	0.1	0.0	0.0
	Structural Completeness	0.780	0.137	0.675	0.800	0.900	0.975	53.7	2.1	0.0	0.0
	Conciseness	0.874	0.047	0.863	0.885	0.901	0.923	93.4	0.0	0.0	0.0
	Description-Content Consistency	0.565	0.106	0.495	0.568	0.634	0.734	1.7	26.2	0.0	0.0

Table 1. Distributional statistics of SkillCT scores across the evaluated agent skills.

Source	Static Assessment			Dynamic Assessment				Human-Assisted Assessment
Source	Structural Completeness	Conci- seness	Description- Content Consistency	Cross-Model Robustness	Intent- Perturbation Stability	Capability- Boundary Handling	Effi- ciency	Behavior-Chain Semantic Consistency	Task Abstraction Granularity
Human	0.814	0.751	0.919	0.848	0.826	0.554	0.885	0.874	0.713
Execution-Free	0.806	0.859	0.567	0.761	0.819	0.964	0.908	0.920	0.661
Full	0.806	0.859	0.567	0.730	0.841	0.916	0.881	0.922	0.653

Table 2. Comparison of Human Assessment, Execution-Free SkillCT Assessment, and Full SkillCT Assessment.

Figure 3. Spearman correlations across dimensions.

Setting	Static Assessment			Dynamic Assessment				Human-Assisted Assessment		Overall
	Structural Complete- ness	Conci- seness	Description- Content Consistency	Cross-Model Robust- ness	Intent- Perturbation Stability	Capability- Boundary Handling	Effi- ciency	Behavior-Chain Semantic Consistency	Task Abstraction Granularity
Original	0.813	0.857	0.564	0.760	0.816	0.961	0.909	0.919	0.658	0.806
SkillOpt	0.904	0.864	0.730	0.799	0.827	0.981	0.923	0.929	0.703	0.851
Model (Original $\rightarrow$ SkillOpt)			Task Success Rate (%) ( $\uparrow$ )		Avg. Execution Steps ( $\downarrow$ )		Avg. Input Tokens ( $\downarrow$ )		Avg. Output Tokens ( $\downarrow$ )
GLM-4.5-Air			91.9 $\rightarrow$ 96.9		7.47 $\rightarrow$ 5.79		1388 $\rightarrow$ 1457		828 $\rightarrow$ 706
DeepSeek-V4-Flash			87.3 $\rightarrow$ 95.1		8.23 $\rightarrow$ 6.72		1391 $\rightarrow$ 1447		905 $\rightarrow$ 813

Table 3. SkillCT assessment and task execution results before and after skill optimization.

Model	Cross- Model Robustness	Intent- Perturbation Stability	Capability- Boundary Handling	Efficiency	Behavior-Chain Semantic Consistency	Task Abstraction Granularity	Overall
Seed-1.6-Flash	0.769	0.804	0.924	0.908	0.882	0.705	0.801
GLM-4.5-Air	0.702	0.723	0.900	0.836	0.914	0.734	0.781
Qwen3.5-Flash	0.724	0.767	0.933	0.869	0.952	0.762	0.803
DeepSeek-V4-Flash	0.685	0.712	0.884	0.820	0.914	0.720	0.773
GPT-5	0.724	0.754	0.909	0.846	0.957	0.746	0.795
Dimensions with identical scores across models

Table 4. Cross-LLM Assessment Consistency in SkillCT.

Assessment Dimension	MAE $\downarrow$	RMSE $\downarrow$	Bias	Pearson $r$ $\uparrow$	Spearman $\rho$ $\uparrow$
Cross-Model Robustness	0.037	0.057	0.004	0.245	0.249
Intent-Perturbation Stability	0.051	0.071	-0.004	0.339	0.352
Capability-Boundary Handling	0.022	0.039	-0.003	0.670	0.721
Efficiency	0.056	0.075	0.003	0.291	0.325
Behavior-Chain Semantic Consistency	0.088	0.134	0.014	0.565	0.482
Task Abstraction Granularity	0.130	0.174	0.005	0.517	0.442

Table 5. Holdout Validation of Execution-Free SkillCT.

Method	MAE $\downarrow$	Pearson $r$ $\uparrow$	Spearman $\rho$ $\uparrow$
Category Mean	0.078	0.165	0.161
Semantic Retrieval	0.069	0.372	0.367
TF-IDF Retrieval (Ours)	0.064	0.438	0.429

Table 6. Comparison of execution-free estimation methods on the holdout set across six estimated dimensions.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

		Static Assessment			Dynamic Assessment				Human-Assisted Assessment		Overall
Category	N	Structural Complete- ness	Conci- seness	Description- Content Consistency	Cross- Model Robustness	Intent- Perturbation Stability	Capability- Boundary Handling	Effi- ciency	Behavior-Chain Semantic Consistency	Task Abstraction Granularity	Overall
Finance	275	0.755	0.882	0.553	0.743	0.772	0.912	0.885	0.872	0.732	0.790
Communication	275	0.775	0.875	0.560	0.701	0.728	0.857	0.848	0.891	0.716	0.772
Knowledge	329	0.793	0.886	0.567	0.793	0.805	0.969	0.929	0.912	0.734	0.821
Productivity	275	0.745	0.883	0.560	0.716	0.749	0.879	0.862	0.873	0.733	0.778
Creative	268	0.813	0.898	0.577	0.773	0.801	0.931	0.893	0.911	0.769	0.819

Appendix

Table 7. Assessment results across different Agent Skill categories in the SkillCT reference benchmark.

Category	Static Assessment			Dynamic Assessment				Human-Assisted Assessment
	Structural Completeness	Conciseness	Description–Content Consistency	Cross-Model Robustness	Intent-Perturbation Stability	Capability-Boundary Handling	Efficiency	Behavior-Chain Semantic Consistency	Task Abstraction Granularity
Finance	0.790/0.892/0.888	0.868/0.874/0.885	0.541/0.723/0.703	0.757/0.796/0.807	0.827/0.838/0.826	0.974/0.990/0.959	0.919/0.932/0.903	0.921/0.931/0.948	0.659/0.704/0.671
Communication	0.789/0.891/0.886	0.851/0.858/0.869	0.545/0.725/0.712	0.765/0.804/0.803	0.820/0.831/0.768	0.975/0.991/0.983	0.910/0.924/0.882	0.905/0.916/0.917	0.632/0.679/0.653
Knowledge	0.784/0.889/0.885	0.857/0.864/0.863	0.564/0.730/0.720	0.750/0.789/0.800	0.790/0.802/0.821	0.951/0.973/0.953	0.905/0.919/0.918	0.922/0.932/0.948	0.660/0.705/0.686
Productivity	0.751/0.873/0.861	0.863/0.869/0.880	0.528/0.719/0.704	0.737/0.777/0.773	0.793/0.805/0.780	0.927/0.956/0.931	0.879/0.895/0.883	0.913/0.923/0.906	0.637/0.683/0.662
Creative	0.789/0.891/0.884	0.864/0.870/0.881	0.583/0.736/0.730	0.770/0.808/0.804	0.823/0.834/0.886	0.972/0.988/0.981	0.934/0.946/0.970	0.932/0.941/0.968	0.681/0.725/0.744

Appendix

Table 8. Category-wise SkillCT assessment scores before and after Skill optimization (50 Skills per category). Each cell reports Original / GLM-4.5-Air / DeepSeek-V4-Flash.

Model Pair	Overall Score		Dimension ( $\rho$ )
	$\rho$	MAE	IPS	BCSC	TAG
Seed–GLM	0.978	0.021	0.873	0.763	0.887
Seed–Qwen	0.991	0.003	0.974	0.915	0.970
Seed–DeepSeek	0.990	0.028	0.976	0.974	0.998
Seed–GPT-5	0.991	0.007	0.955	0.904	0.977
GLM–Qwen	0.991	0.022	0.807	0.833	0.908

Appendix

Table 9. Category-level assessment agreement across LLMs on the SkillCT reference benchmark.

Criterion	Condition	Weight
Frontmatter	YAML frontmatter is present	2.0
Name	A Skill name is provided	1.5
Description	A description longer than 10 characters is provided	2.5
Description length	Description length is between 10 and 500 characters	1.0
Instructional body	The instructional body contains more than 50 characters	2.0
Section structure	At least two second-level sections are present	1.5

Appendix

Table 10. Structural Completeness criteria and weights.

Request Type	Response	Score
Unrelated	Identified as unsupported with an explanation	1.0
	Identified as unsupported without an explanation	0.8
	Treated as supported	0.0
Adversarial	Identified as unsupported with an explanation	1.0
	Identified as unsupported without an explanation	0.8
	Treated as supported	0.0

Appendix

Table 11. Scoring rules for Capability-Boundary Handling.

Quantity	Operating Interval	Upper Threshold	Weight
Prompt tokens	500–2000	8000	0.30
Plan steps	3–7	20	0.25
Response latency	500–3000 ms	15000 ms	0.20
Tools or commands	1–5	15	0.25

Appendix

Table 12. Normalization settings for Efficiency.

Score	Criterion
1.00	All actions directly support the declared objective, with no irrelevant or redundant operations.
0.75	Most actions support the objective, with only one or two peripheral actions.
0.50	Only part of the behavior chain supports the objective, with clear irrelevant or redundant operations.
0.25	Most actions are weakly related or unrelated to the declared objective, resulting in a poorly aligned behavior chain.
0.00	The behavior chain is almost entirely unrelated to the declared objective.

Appendix

Table 13. Scoring rubric for Behavior-Chain Semantic Consistency.

Score	Criterion
1.00	The Skill represents a complete and meaningful unit of work with a clear and self-contained responsibility.
0.75	The task scope is generally appropriate but is slightly narrower or broader than the intended unit of work.
0.50	The responsibility is noticeably too narrow or too broad.
0.25	The task scope is severely misaligned and would substantially benefit from being split or combined with other functionality.
0.00	The Skill either reduces to an isolated operation with little additional procedure or attempts to cover an entire domain without a coherent task scope.

Appendix

Table 14. Scoring rubric for Task Abstraction Granularity.

LLM Agent Skill Evaluation

See all 31 papers

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment
Jun 21, 2026Dexu Yu, Youhua Li, Zhaoyang Guan +12AI Agent Auditing Automated Evaluation
Northeastern University · City University of Hong Kong · DeciLix Lab +8
SkillEval: Decomposing Agent Skill Quality into Interpretable Signals
Aug 7, 2026Jiahui Han, Qinuo Li, Ziheng Peng +6LLM Agent Evaluation Automated Evaluation
Xi’an Jiaotong University · Shanghai AI Laboratory · Harbin Institute of Technology +2
SkillTester: Benchmarking Utility and Security of Agent Skills
Mar 28, 2026Leye Wang, Zixing Wang, Anjie XuLLM Agent Skill Evaluation AI Agent Security Benchmarks
Key Laboratory of High Confidence Software Technologies, Ministry of Education, Peking University, Beijing, China · Northwestern Polytechnical University, Xi’an, China

Jun 21, 2026J/K move · Enter open · S save

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

Dexu Yu, Youhua Li, Zhaoyang Guan, Xianhao Lin+11

Northeastern University · City University of Hong Kong · DeciLix Lab+8

AI Agent Auditing Automated Evaluation Agent Skill Security Auditing LLM Agent Skill Evaluation

Open paper AskPDF arXiv

Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.

Abstract

LLM Agent Skill Evaluation

Explore similar work

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

SkillEval: Decomposing Agent Skill Quality into Interpretable Signals

SkillTester: Benchmarking Utility and Security of Agent Skills