SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications

Zhengxi Li; Junyi Zhou; Qiyuan Zhang; Chuhan Wu; Fuyuan Lyu; Le Tian; Xiao Zhou; Chen Ma

cs.AIOct 10, 2026

SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications

Authors: Zhengxi Li, Junyi Zhou, Qiyuan Zhang, Chuhan Wu, Fuyuan Lyu, Le Tian, Xiao Zhou, Chen Ma,

Organizations: MBZUAI · City University of Hong Kong · WeChat AI, Tencent · McGill – Mila & Quebec AI Institute

Abstract

Evaluating an agent's ability to strictly satisfy diverse and procedurally complex requirements is essential to determining its practical reliability. Conducting such an evaluation demands agentic tasks that reconcile broad domain coverage with deep procedural complexity. Existing paradigms face a fundamental trade-off: expert-curated benchmarks achieve procedural depth but suffer from domain diversity bottlenecks, whereas crowdsourced benchmarks span diverse scenarios but lack execution rigor. To resolve this tension, we present SkillForgeBench, which leverages agent skills as unifying execution blueprints across three co-designed stages: (1) Resource Preparation uses domain-balanced skills to guide artifact retrieval, establishing broad workflow coverage; (2) Skill-driven Task Synthesis employs skills as procedural anchors to partition artifact roles, synthesize naturalistic task instructions, and package sandboxed scenarios for deep procedural execution; and (3) Holistic Evaluation Design translates skill-embedded requirements into fine-grained completion and process rubrics to enable strict requirement-level scoring. Comprising 842 sandboxed tasks across five major professional domains, SkillForgeBench reveals a steep capability hierarchy among eight mainstream agents, led by Opus 4.8 ( $40.9\%$ strict completion). Requiring all constrains satisfaction uncovers severe capability bottlenecks across all model tiers that remain masked under individual rubric pass rates ( $70.0\%$ to $85.7\%$ vs. $12.6\%$ to $40.9\%$ ). Crucially, under the external guidance from skills, top-tier agents obtain larger gains than low-tier agents (i.e., $Δ19.2\%$ for Opus 4.8 vs. $Δ5.8\%$ for Haiku 4.5). Finally, agents satisfy process rubrics far less frequently than completion rubrics (under $3\%$ for Haiku 4.5), proving that completion-only checks overestimate real-world agent reliability.

Figures & tables

Benchmark	Task Source	Scale	Real [-1pt]Demand	Authentic [-1pt]Env.	Fine [-1pt]Rubrics	Process [-1pt]Eval.	Skill [-1pt]Use
WebArena ( Zhou et al., 2024 )	Authored web intents	812	$\bullet$	$\bullet$	✗	✗	✗
AppWorld ( Trivedi et al., 2024 )	Authored app workflows	750	$\bullet$	$\bullet$	✗	✗	✗
Terminal-Bench 2.0 ( Merrill et al., 2026 )	Expert workflows	89	$\bullet$	✓	✗	✗	✗
ClawBench ( Zhang et al., 2026 )	Everyday assistant tasks	153	$\bullet$	✓	$\bullet$	✓	✗
LiveClawBench ( Long et al., 2026 )	Real usage cases	134	$\bullet$	$\bullet$	✓	✗	✗
RealClawBench ( Lv et al., 2026 )	Production sessions	281	✓	✓	✗	✗	✗

Table 1: Comparison of SkillForgeBench with representative agent benchmarks. Real Demand and Authentic Env. measure task and environment realism; Fine Rubrics and Process Eval. characterize evaluation granularity; and Skill Use indicates whether skills are explicitly evaluated. A green check, orange dot, and gray cross denote full, partial, and no support.

Figure 1: Overview of the SkillForgeBench construction pipeline, demonstrated using a citation verification skill as a running example: (a) Skills guide retrieval of real artifacts and user queries; (b) These inputs support task formulation, instruction composition, and environment packaging; and (c) Contrastive rollouts with and without skill access inform completion and process rubrics, followed by human task validation.

Model	$S_{\text{comp}}$	$S_{\text{proc}}$	$S_{\text{comp}}$	$S_{\text{comp}}$	$S_{\text{proc}}$	$S_{\text{comp}}$	# Assistant		# Tool-call
	Loose			Strict			Efficiency
	w/ skill	w/ skill	w/o skill	w/ skill	w/ skill	w/o skill	w/ skill	w/o skill	w/ skill	w/o skill
Opus 4.8	85.7	87.9	79.8	40.9	27.7	21.7	28.6	24.6	41.0	38.5
GLM-5.2	84.1	86.7	76.9	36.8	28.7	20.0	42.6	39.6	58.0	54.8
Sonnet 4.6	81.5	76.0	75.3	28.5	10.2	15.0	15.5	11.7	38.2	29.6
HY3	80.9	84.2	73.9	27.3	21.6	14.0	34.7	31.8	44.3	41.0

Table 2: Main leaderboard of model performance and execution efficiency. Task-level mean scores (%) are reported under loose and strict scoring for Completion (

S_{\text{comp}}

), evaluated with and without skill, and Process (

S_{\text{proc}}

), evaluated with skill. The efficiency columns report the mean total number of interaction steps (# Assistant) and tool calls (# Tool-call) for runs with and without skill. Bold indicates the highest score and underlining indicates the second-highest score in each performance column.

Figure 2: Completion gains from skill. Bars compare Completion scores with and without skill under strict and loose scoring.

\Delta=S_{\mathrm{comp}}^{\mathrm{WS}}-S_{\mathrm{comp}}^{\mathrm{NS}}

, shown to the right of each bar, denotes the Completion-score gain from skill access.

Audit Target	# Units	Expert 1	Expert 2	GPT-5.6-sol	Human Agr. (%)
Task Quality	100	4.56	4.71	4.28	98.0
Completion Rubrics	771	4.47	4.56	4.43	96.1
Process Rubrics	840	4.44	4.49	4.13	95.4
Overall	1,711	4.46	4.53	4.27	95.9

Table 3: Independent audit of SkillForgeBench task and rubric quality. Values are mean ratings on a 1–5 scale. Human agreement is the percentage of paired ratings that differ by at most one point. Task quality covers 100 tasks; rubric quality covers 771 completion and 840 process criteria. The overall row aggregates all 5,133 dimension-level ratings.

Figure 3: Model performance versus release date under loose scoring. Each panel plots $S_{\mathrm{comp}}^{\mathrm{WS}}$ , $S_{\mathrm{comp}}^{\mathrm{NS}}$ , and $S_{\mathrm{proc}}^{\mathrm{WS}}$ against official release date, with dotted lines denoting linear least-squares fits. All three scores are positively associated with release date ( $r>0.68$ ), with fitted slopes of 1.32, 1.14, and 2.17 pp/month, respectively. Corresponding strict scoring results are reported in Figure 7 .

Figure 4: Loose rubric-category scores for models at different leaderboard positions. Opus 4.8, HY3, and Gemini 3.6 Flash are selected as representatives of different positions in the main leaderboard. For the Completion categories, WS and NS denote evaluation with and without the skill, respectively; the Process categories are evaluated only with the skill. The three models achieve relatively similar Inspectable Deliverable scores but separate more clearly in the remaining categories, particularly the three Process categories.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Gains from skill access versus baseline completion performance. We define the skill gain,

\Delta

, as the difference between the completion scores obtained with and without skill access:

\Delta=S_{\mathrm{comp}}^{\mathrm{WS}}-S_{\mathrm{comp}}^{\mathrm{NS}}

, where WS and NS denote the with-skill and without-skill conditions, respectively. The scatter plots relate the loose-score gain to the loose

S_{\mathrm{comp}}^{\mathrm{NS}}

baseline and the strict-score gain to both loose and strict

S_{\mathrm{comp}}^{\mathrm{NS}}

baselines. Each point represents one model, dotted lines show least-squares fits, and annotations report Pearson’s

r

. Loose-score gains are comparatively uniform (

5.2\%

–

7.4\%

;

r=0.42

), whereas strict-score gains increase strongly with baseline performance (

r=0.94

and

r=0.97

Figure 6: Model performance versus trajectory length. Each panel plots model performance against mean trajectory length, measured in interaction steps. Rows correspond to loose and strict scoring, while columns report completion w/ skill ( $S_{\mathrm{comp}}^{\mathrm{WS}}$ ), completion w/o skill ( $S_{\mathrm{comp}}^{\mathrm{NS}}$ ), and process with the skill ( $S_{\mathrm{proc}}^{\mathrm{WS}}$ ). The horizontal axis is reversed, such that shorter trajectories appear farther to the right. Dashed lines indicate the cross-model mean score and mean trajectory length; consequently, the shaded upper-right region identifies models that achieve above-average performance with fewer-than-average steps. Across metrics and scoring schemes, longer trajectories do not consistently correspond to higher performance.

Figure 7: Model performance versus release date under strict scoring. Each panel plots model performance against official release date for completion with the skill (Completion WS), completion without the skill (Completion NS), and skill following with the skill (Following WS). Points represent individual models, dotted lines show linear least-squares fits, and the annotated slopes are measured in percentage points (pp) per month. Release date is positively correlated with all three metrics, with Pearson’s $r$ values of $0.536$ , $0.502$ , and $0.574$ , respectively. As under loose scoring, Following WS exhibits the steepest fitted trend ( $2.08$ pp/month), compared with Completion WS ( $1.82$ pp/month) and Completion NS ( $0.96$ pp/month).

ID	Forum name	Description
1	Reddit	Topic-based communities called subreddits.
2	Quora	Questions, answers, and opinion discussions.
3	Stack Exchange	Expert Q&A network, including Stack Overflow.
4	Stack Overflow	Programming questions and answers.
5	Tripadvisor Forums	Travel discussions.
6	XDA Developers	Android, mobile devices, ROMs, and device modification.

Appendix

Table 4: Forums and their primary discussion topics.

Figure 8: Domain and sub-field composition of the 842 SkillForgeBench tasks. The inner ring shows the five domains and the outer ring their 14 sub-fields, each sized by its number of tasks; since each task comes from a distinct skill, the same numbers hold for skills.

Domain / sub-field	Skills	%	Topics	Most frequent topics (skills)
Technology	257	30.5	27
Software engineering	190	22.6	10	Code implementation & refactoring (88); architecture & design decisions (31); debugging & root-cause analysis (19)
Infrastructure & DevOps	28	3.3	6	Cloud platforms & IaC (8); observability & alerting (7); CI/CD & release management (6)
Security & privacy	27	3.2	6	Security operations & incident response (10); threat modeling & security architecture (7); privacy & compliance auditing (4)
Data & machine learning	12	1.4	5	Data processing & pipelines (4); querying & SQL (3); machine & deep learning (3)
Business	211	25.1	17

Appendix

Table 5: Domain distribution of the skills behind the 842 SkillForgeBench tasks. Each task is synthesized from a distinct skill, so skill counts equal task counts. The % column gives the share of the 842 skills in each row. Topics is the number of distinct fine-grained topics a sub-field covers (83 in total). The last column lists the three most frequent topics with their skill counts; ties are broken by taxonomy order.

Model	$S_{\text{comp}}$						$S_{\text{follow}}$			$S_{\text{comp}}$						$S_{\text{follow}}$
	Loose									Strict
	Durable Artifact		Output Discipline		Domain Operationalization		Phase Adherence	Empirical Verification Loop	Scope Discipline	Durable Artifact		Output Discipline		Domain Operationalization		Phase Adherence	Empirical Verification Loop	Scope Discipline
	w/ skill	wo/ skill	w/ skill	wo/ skill	w/ skill	wo/ skill	w/ skill	w/ skill	w/ skill	w/ skill	wo/ skill	w/ skill	wo/ skill	w/ skill	wo/ skill	w/ skill	w/ skill	w/ skill
Opus 4.8	89.2	87.5	88.9	81.5	87.1	82.2	94.1	87.3	90.1	72.2	66.0	69.4	52.6	57.7	45.5	75.1	50.1	69.8
GLM-5.2	88.3	85.9	85.5	76.8	85.6	80.3	96.3	90.1	80.7	69.3	62.0	64.6	46.6	55.7	44.7	86.2	63.2	48.7
HY3	85.3	82.6	83.0	75.2	83.2	76.5	93.3	83.1	86.9	63.5	57.0	58.9	42.3	50.1	34.4	74.2	41.5	62.8

Appendix

Table 6: Rubric-category scores under loose and strict scoring. Category-level averages are reported over the common set of 842 evaluation tasks. The completion metric (

S_{\mathrm{comp}}

) comprises Durable Artifact, Output Discipline, and Domain Operationalization, evaluated both with and without the skill. The following metric (

S_{\mathrm{follow}}

) comprises Phase Adherence, Empirical Verification Loop, and Scope Discipline, evaluated with the skill. Under loose scoring, each task–category instance receives the weighted mean of its applicable criteria. Under strict scoring, it receives full credit only when every applicable criterion attains its maximum score, and zero otherwise. All reported values are percentages. Bold marks the highest score, and underlining marks the second-highest score in each column.

Figure 9: Strict rubric-category scores for models at different leaderboard positions. Opus 4.8, HY3, and Gemini 3.6 Flash are selected as representatives of different positions in the main leaderboard. For the Completion categories, WS and NS denote evaluation with and without the skill, respectively; the Process categories are evaluated only with the skill. Under strict scoring, credit is awarded only when all applicable requirements within a category are fully satisfied. Gemini 3.6 Flash remains comparable to HY3 in Inspectable Deliverable but trails both HY3 and Opus 4.8 in Output Structure, Terminology Use, and all three Process categories. Full numerical results are reported in Table 6 .

Measure	Mean	Population SD	Median
Skill files per task	11.32	28.90	6
Distinct file types per task	9.46	11.17	4
Logical task size (MiB)	13.67	27.32	3.37
Files per task	172.89	434.80	10
Instruction length (characters)	1,333.05	311.37	1,306.5

Appendix

Table 7: Descriptive statistics for the 842-task cohort.

Agent Evaluation

Agent evaluation measures how reliably AI systems act, use tools, and complete tasks across interactive environments.

See all 144 papers

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment
Jun 21, 2026Dexu Yu, Youhua Li, Zhaoyang Guan +12AI Agent Auditing Automated Evaluation
Northeastern University · City University of Hong Kong · DeciLix Lab +8
Skill Coverage: A Test Adequacy Metric for Agent Skills
Jun 9, 2026Boyin Tan, Xiaowei Huang, Youcheng SunLLM Agent Evaluation AI Agent Benchmarks
1Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates · University of Liverpool Liverpool, United Kingdom
SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents
May 18, 2026Yifan Zhou, Zhentao Zhang, Ziming Cheng +8LLM Agent Evaluation LLM Agent Skill Learning
1SJTU · 2XJTU · 3NUS +6

Jun 21, 2026J/K move · Enter open · S save

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

Dexu Yu, Youhua Li, Zhaoyang Guan, Xianhao Lin+11

Northeastern University · City University of Hong Kong · DeciLix Lab+8

AI Agent Auditing Automated Evaluation Agent Skill Security Auditing LLM Agent Skill Evaluation

Open paper AskPDF arXiv

Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.

Abstract

Agent Evaluation

Explore similar work

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

Skill Coverage: A Test Adequacy Metric for Agent Skills

SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents