Skill Availability and Presentation Granularity in Large-Language-Model Agents: A Controlled SkillsBench Study

Agent Skills augment large language model (LLM) agents with procedural knowledge at inference time, but current benchmarks rarely distinguish what a Skill says from how it is organized. We study this distinction through Progressive Disclosure, where a concise root file points agents to supporting resources on demand, and compare it with a normalized flat baseline. We present SkillJuror, a framework for evaluating Skill writing paradigms through semantically controlled variants, matched multi-trial evaluations, and trajectory evidence while holding task knowledge fixed. In an 82-task SkillsBench study, Progressive Disclosure changes runtime behavior before aggregate outcomes: distinct Skill resources touched per trajectory rise from 1.18 to 3.85, and effective uptake events rise from 1.33 to 3.92. It also yields 17 additional verifier-passing trials out of 410 matched trials (+4.1%) over the normalized flat baseline. The benefit is task-dependent. Progressive Disclosure helps when supporting resources guide implementation, checking, or repair, but is weaker when success hinges on exact output conventions, numerical thresholds, or long artifact-generation pipelines. These results show that Skill organization is not mere presentation: it can change how agents search and apply procedural knowledge, while outcome gains depend on whether the exposed resources are actionable for the task. Code is available at https://github.com/zhiyuchen-ai/skill-juror.

Large language model (LLM) agents increasingly rely on skills, structured documents that specify when to act, which procedure to follow, and which tools are allowed. Existing evaluations mostly judge the quality of a skill or its contribution to task success, leaving unexamined whether an agent can recognize a relevant skill and apply it on its own. We introduce Skill-Use, a benchmark that evaluates skill use under progressive disclosure, where an agent sees only a skill's name and short description and must retrieve the full procedure before following it. Skill-Use separates three facets of skill use. Trigger measures whether the agent invokes the relevant skill, Compliance measures how faithfully it follows the prescribed procedure, and Boundary measures whether it avoids forbidden operations. A Skill-Use (SU) score combines the three and credits execution only after the skill is triggered. Skill-Use pairs 79 real skills with 177 executable tasks across nine domains, each grounded in real files, run in an isolated Docker sandbox, and scored by a trajectory-based rubric. Evaluating eight LLMs under two agent harnesses, we find that reliable skill use remains out of reach, as the strongest configuration reaches an SU of only 0.613. Triggering and procedural compliance fail as independent bottlenecks, and both scores and model rankings shift with the harness, so skill use behaves as a capability conditioned on the harness rather than a fixed property of the model.

Agent skills encode reusable procedural knowledge for large language model (LLM) agents, and existing benchmarks show that such skills can improve task-level performance. However, a task outcome does not reveal which parts of a reusable skill were exercised, nor whether the agent followed the relevant skill instructions when those parts were exercised. This gap makes it unclear whether a skill has been adequately tested, or whether observed task failures provide actionable evidence for improving agent skill effectiveness. To fill this gap, we introduce skill coverage, a trajectory-based test-adequacy metric for reusable agent skills. Our framework extracts skill behavior constraints from each skill, translating natural-language skill instructions into semi-structured constraints that specify the expected agent behavior under particular conditions. It then determines whether each constraint is covered by an agent trajectory and, for covered constraints, assigns a Pass or Fail verdict according to the agent behavior. We apply this framework to SkillsBench. The results show that agent trajectories on the benchmark leaderboard cover only 38.66 to 45.51% of the extracted skill behavior constraints on average. We then use Fail verdicts to strengthen the corresponding skill content only by emphasizing the original instructions that the agent failed to follow, and run the same tasks with the strengthened skills. This emphasis yields an average 16.0% recovery rate of the failed tasks across the five agent-model rows. These results show that skill coverage is both a test-adequacy metric and a fine-grained signal for observing skill-use behavior. In failed tasks, failed constraint labels provide actionable evidence for improving agent skill effectiveness. A project website accompanies the paper.