Evaluating an agent's ability to strictly satisfy diverse and procedurally complex requirements is essential to determining its practical reliability. Conducting such an evaluation demands agentic tasks that reconcile broad domain coverage with deep procedural complexity. Existing paradigms face a fundamental trade-off: expert-curated benchmarks achieve procedural depth but suffer from domain diversity bottlenecks, whereas crowdsourced benchmarks span diverse scenarios but lack execution rigor. To resolve this tension, we present SkillForgeBench, which leverages agent skills as unifying execution blueprints across three co-designed stages: (1) Resource Preparation uses domain-balanced skills to guide artifact retrieval, establishing broad workflow coverage; (2) Skill-driven Task Synthesis employs skills as procedural anchors to partition artifact roles, synthesize naturalistic task instructions, and package sandboxed scenarios for deep procedural execution; and (3) Holistic Evaluation Design translates skill-embedded requirements into fine-grained completion and process rubrics to enable strict requirement-level scoring. Comprising 842 sandboxed tasks across five major professional domains, SkillForgeBench reveals a steep capability hierarchy among eight mainstream agents, led by Opus 4.8 (40.9% strict completion). Requiring all constrains satisfaction uncovers severe capability bottlenecks across all model tiers that remain masked under individual rubric pass rates (70.0% to 85.7% vs. 12.6% to 40.9%). Crucially, under the external guidance from skills, top-tier agents obtain larger gains than low-tier agents (i.e., Δ19.2% for Opus 4.8 vs. Δ5.8% for Haiku 4.5). Finally, agents satisfy process rubrics far less frequently than completion rubrics (under 3% for Haiku 4.5), proving that completion-only checks overestimate real-world agent reliability.
Figures & tables
Benchmark
Task Source
Scale
Real [-1pt]Demand
Authentic [-1pt]Env.
Fine [-1pt]Rubrics
Process [-1pt]Eval.
Skill [-1pt]Use
WebArena ( Zhou et al., 2024 )
Authored web intents
812
∙
∙
✗
✗
✗
AppWorld ( Trivedi et al., 2024 )
Authored app workflows
750
∙
∙
✗
✗
✗
Terminal-Bench 2.0 ( Merrill et al., 2026 )
Expert workflows
89
∙
✓
✗
✗
✗
ClawBench ( Zhang et al., 2026 )
Everyday assistant tasks
153
∙
✓
∙
✓
✗
LiveClawBench ( Long et al., 2026 )
Real usage cases
134
∙
∙
✓
✗
✗
RealClawBench ( Lv et al., 2026 )
Production sessions
281
✓
✓
✗
✗
✗
Table 1: Comparison of SkillForgeBench with representative agent benchmarks. Real Demand and Authentic Env. measure task and environment realism; Fine Rubrics and Process Eval. characterize evaluation granularity; and Skill Use indicates whether skills are explicitly evaluated. A green check, orange dot, and gray cross denote full, partial, and no support.
Figure 1: Overview of the SkillForgeBench construction pipeline, demonstrated using a citation verification skill as a running example: (a) Skills guide retrieval of real artifacts and user queries; (b) These inputs support task formulation, instruction composition, and environment packaging; and (c) Contrastive rollouts with and without skill access inform completion and process rubrics, followed by human task validation.
Loose
Strict
Efficiency
Model
Scomp
Sproc
Scomp
Scomp
Sproc
Scomp
# Assistant
# Tool-call
w/ skill
w/ skill
w/o skill
w/ skill
w/ skill
w/o skill
w/ skill
w/o skill
w/ skill
w/o skill
Opus 4.8
85.7
87.9
79.8
40.9
27.7
21.7
28.6
24.6
41.0
38.5
GLM-5.2
84.1
86.7
76.9
36.8
28.7
20.0
42.6
39.6
58.0
54.8
Sonnet 4.6
81.5
76.0
75.3
28.5
10.2
15.0
15.5
11.7
38.2
29.6
HY3
80.9
84.2
73.9
27.3
21.6
14.0
34.7
31.8
44.3
41.0
Table 2: Main leaderboard of model performance and execution efficiency. Task-level mean scores (%) are reported under loose and strict scoring for Completion ( Scomp ), evaluated with and without skill, and Process ( Sproc ), evaluated with skill. The efficiency columns report the mean total number of interaction steps (# Assistant) and tool calls (# Tool-call) for runs with and without skill. Bold indicates the highest score and underlining indicates the second-highest score in each performance column.
Figure 2: Completion gains from skill. Bars compare Completion scores with and without skill under strict and loose scoring. Δ=ScompWS−ScompNS , shown to the right of each bar, denotes the Completion-score gain from skill access.
Audit Target
# Units
Expert 1
Expert 2
GPT-5.6-sol
Human Agr. (%)
Task Quality
100
4.56
4.71
4.28
98.0
Completion Rubrics
771
4.47
4.56
4.43
96.1
Process Rubrics
840
4.44
4.49
4.13
95.4
Overall
1,711
4.46
4.53
4.27
95.9
Table 3: Independent audit of SkillForgeBench task and rubric quality. Values are mean ratings on a 1–5 scale. Human agreement is the percentage of paired ratings that differ by at most one point. Task quality covers 100 tasks; rubric quality covers 771 completion and 840 process criteria. The overall row aggregates all 5,133 dimension-level ratings.
Figure 3: Model performance versus release date under loose scoring. Each panel plots ScompWS , ScompNS , and SprocWS against official release date, with dotted lines denoting linear least-squares fits. All three scores are positively associated with release date ( r>0.68 ), with fitted slopes of 1.32, 1.14, and 2.17 pp/month, respectively. Corresponding strict scoring results are reported in Figure 7 .
Figure 4: Loose rubric-category scores for models at different leaderboard positions. Opus 4.8, HY3, and Gemini 3.6 Flash are selected as representatives of different positions in the main leaderboard. For the Completion categories, WS and NS denote evaluation with and without the skill, respectively; the Process categories are evaluated only with the skill. The three models achieve relatively similar Inspectable Deliverable scores but separate more clearly in the remaining categories, particularly the three Process categories.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Gains from skill access versus baseline completion performance. We define the skill gain, Δ , as the difference between the completion scores obtained with and without skill access: Δ=ScompWS−ScompNS , where WS and NS denote the with-skill and without-skill conditions, respectively. The scatter plots relate the loose-score gain to the loose ScompNS baseline and the strict-score gain to both loose and strict ScompNS baselines. Each point represents one model, dotted lines show least-squares fits, and annotations report Pearson’s r . Loose-score gains are comparatively uniform ( 5.2% – 7.4% ; r=0.42 ), whereas strict-score gains increase strongly with baseline performance ( r=0.94 and r=0.97 ).
Figure 6: Model performance versus trajectory length. Each panel plots model performance against mean trajectory length, measured in interaction steps. Rows correspond to loose and strict scoring, while columns report completion w/ skill ( ScompWS ), completion w/o skill ( ScompNS ), and process with the skill ( SprocWS ). The horizontal axis is reversed, such that shorter trajectories appear farther to the right. Dashed lines indicate the cross-model mean score and mean trajectory length; consequently, the shaded upper-right region identifies models that achieve above-average performance with fewer-than-average steps. Across metrics and scoring schemes, longer trajectories do not consistently correspond to higher performance.
Figure 7: Model performance versus release date under strict scoring. Each panel plots model performance against official release date for completion with the skill (Completion WS), completion without the skill (Completion NS), and skill following with the skill (Following WS). Points represent individual models, dotted lines show linear least-squares fits, and the annotated slopes are measured in percentage points (pp) per month. Release date is positively correlated with all three metrics, with Pearson’s r values of 0.536 , 0.502 , and 0.574 , respectively. As under loose scoring, Following WS exhibits the steepest fitted trend ( 2.08 pp/month), compared with Completion WS ( 1.82 pp/month) and Completion NS ( 0.96 pp/month).
ID
Forum name
Description
1
Reddit
Topic-based communities called subreddits.
2
Quora
Questions, answers, and opinion discussions.
3
Stack Exchange
Expert Q&A network, including Stack Overflow.
4
Stack Overflow
Programming questions and answers.
5
Tripadvisor Forums
Travel discussions.
6
XDA Developers
Android, mobile devices, ROMs, and device modification.
Appendix
Table 4: Forums and their primary discussion topics.
Figure 8: Domain and sub-field composition of the 842 SkillForgeBench tasks. The inner ring shows the five domains and the outer ring their 14 sub-fields, each sized by its number of tasks; since each task comes from a distinct skill, the same numbers hold for skills.
Data processing & pipelines (4); querying & SQL (3); machine & deep learning (3)
Business
211
25.1
17
Appendix
Table 5: Domain distribution of the skills behind the 842 SkillForgeBench tasks. Each task is synthesized from a distinct skill, so skill counts equal task counts. The % column gives the share of the 842 skills in each row. Topics is the number of distinct fine-grained topics a sub-field covers (83 in total). The last column lists the three most frequent topics with their skill counts; ties are broken by taxonomy order.
Loose
Strict
Model
Scomp
Sfollow
Scomp
Sfollow
Durable Artifact
Output Discipline
Domain Operationalization
Phase Adherence
Empirical Verification Loop
Scope Discipline
Durable Artifact
Output Discipline
Domain Operationalization
Phase Adherence
Empirical Verification Loop
Scope Discipline
w/ skill
wo/ skill
w/ skill
wo/ skill
w/ skill
wo/ skill
w/ skill
w/ skill
w/ skill
w/ skill
wo/ skill
w/ skill
wo/ skill
w/ skill
wo/ skill
w/ skill
w/ skill
w/ skill
Opus 4.8
89.2
87.5
88.9
81.5
87.1
82.2
94.1
87.3
90.1
72.2
66.0
69.4
52.6
57.7
45.5
75.1
50.1
69.8
GLM-5.2
88.3
85.9
85.5
76.8
85.6
80.3
96.3
90.1
80.7
69.3
62.0
64.6
46.6
55.7
44.7
86.2
63.2
48.7
HY3
85.3
82.6
83.0
75.2
83.2
76.5
93.3
83.1
86.9
63.5
57.0
58.9
42.3
50.1
34.4
74.2
41.5
62.8
Appendix
Table 6: Rubric-category scores under loose and strict scoring. Category-level averages are reported over the common set of 842 evaluation tasks. The completion metric ( Scomp ) comprises Durable Artifact, Output Discipline, and Domain Operationalization, evaluated both with and without the skill. The following metric ( Sfollow ) comprises Phase Adherence, Empirical Verification Loop, and Scope Discipline, evaluated with the skill. Under loose scoring, each task–category instance receives the weighted mean of its applicable criteria. Under strict scoring, it receives full credit only when every applicable criterion attains its maximum score, and zero otherwise. All reported values are percentages. Bold marks the highest score, and underlining marks the second-highest score in each column.
Figure 9: Strict rubric-category scores for models at different leaderboard positions. Opus 4.8, HY3, and Gemini 3.6 Flash are selected as representatives of different positions in the main leaderboard. For the Completion categories, WS and NS denote evaluation with and without the skill, respectively; the Process categories are evaluated only with the skill. Under strict scoring, credit is awarded only when all applicable requirements within a category are fully satisfied. Gemini 3.6 Flash remains comparable to HY3 in Inspectable Deliverable but trails both HY3 and Opus 4.8 in Output Structure, Terminology Use, and all three Process categories. Full numerical results are reported in Table 6 .
Measure
Mean
Population SD
Median
Skill files per task
11.32
28.90
6
Distinct file types per task
9.46
11.17
4
Logical task size (MiB)
13.67
27.32
3.37
Files per task
172.89
434.80
10
Instruction length (characters)
1,333.05
311.37
1,306.5
Appendix
Table 7: Descriptive statistics for the 842-task cohort.
Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.
Dexu Yu, Youhua Li, Zhaoyang Guan +12
Northeastern University · City University of Hong Kong · DeciLix Lab +8
Agent skills encode reusable procedural knowledge for large language model (LLM) agents, and existing benchmarks show that such skills can improve task-level performance. However, a task outcome does not reveal which parts of a reusable skill were exercised, nor whether the agent followed the relevant skill instructions when those parts were exercised. This gap makes it unclear whether a skill has been adequately tested, or whether observed task failures provide actionable evidence for improving agent skill effectiveness. To fill this gap, we introduce skill coverage, a trajectory-based test-adequacy metric for reusable agent skills. Our framework extracts skill behavior constraints from each skill, translating natural-language skill instructions into semi-structured constraints that specify the expected agent behavior under particular conditions. It then determines whether each constraint is covered by an agent trajectory and, for covered constraints, assigns a Pass or Fail verdict according to the agent behavior. We apply this framework to SkillsBench. The results show that agent trajectories on the benchmark leaderboard cover only 38.66 to 45.51% of the extracted skill behavior constraints on average. We then use Fail verdicts to strengthen the corresponding skill content only by emphasizing the original instructions that the agent failed to follow, and run the same tasks with the strengthened skills. This emphasis yields an average 16.0% recovery rate of the failed tasks across the five agent-model rows. These results show that skill coverage is both a test-adequacy metric and a fine-grained signal for observing skill-use behavior. In failed tasks, failed constraint labels provide actionable evidence for improving agent skill effectiveness. A project website accompanies the paper.
Boyin Tan, Xiaowei Huang, Youcheng Sun
1Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates · University of Liverpool Liverpool, United Kingdom
As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether they can generate correct, reusable, and executable skills from repositories and documents. Existing benchmarks primarily evaluate the efficacy of given skills or the ability of agents to solve downstream tasks from raw context, but they do not isolate skill generation itself as the object of study. We introduce SkillGenBench, a benchmark for evaluating skill generation pipelines under a unified and controlled protocol. In SkillGenBench, a generator receives raw corpora and produces standardized skill artifacts, which are then executed under fixed harnesses and assessed with unified evaluation procedures. The benchmark covers two generation regimes: task-conditioned generation, where a task-specific skill is synthesized after the task is revealed, and task-agnostic generation, where a reusable skill library must be distilled before downstream tasks are known. It also spans two complementary procedural sources: repository-grounded instances, where procedures are distributed across code, configuration, and scripts, and document-grounded instances, where procedures and constraints must be distilled from long-form text. We provide standardized task specifications, pinned environments, and evaluation protocols centered on deterministic execution-based checks, supplemented by auxiliary signals for diagnosis. Experiments across a range of skill-generation methods and backbones show substantial performance variation, highlight the difficulty of reusable skill distillation, and reveal distinct failure modes in skill generation from software repositories versus long-form documents. SkillGenBench establishes a reproducible testbed for studying skill generation as an independent research problem in agent systems.