cs.AINov 17, 2026

SkillBuffBench: Measuring What Agent Skills Contribute across Harnesses, Models, Skill Access Modes and Memory

Authors: Gengyue Dong, Ye Wei, Zheng Zhu, Yidong Han, Jiale Miao, Xiaojun Jia, Guanghui Li

Organizations: Department of Engineering Science, University of Oxford, Oxford, UK · MRC Weatherall Institute of Molecular Medicine, University of Oxford, Oxford, UK · Department of Materials, Imperial College London, London, UK · Department of Mathematics, University College London, London, UK · 

Abstract

Agent Skills, file-based packages of procedural knowledge that a coding agent can discover and load on demand, are now shipped by every major agent harness, yet the evidence for how much they help comes almost entirely from settings in which the right Skill is hand-picked for the task and the agent starts from a blank slate. Neither holds in deployment, where an agent chooses from whatever library happens to be installed and frequently carries memory from earlier work. We present SkillBuffBench, a benchmark and paired evaluation protocol that treats Skill access as a three-level factor (no Skill, a curated Skill, and automatic invocation from an installed library), crosses it with memory (off and on), with four agent harnesses and with multiple models (the state-of-the-art proprietary models from Anthropic, Google and OpenAI, and the state-of-the-art open-weight models from Chinese providers), every model being run under every harness, and evaluates the result on research tasks contributed by domain experts together with the Skills they use, currently in materials science and biology, with the suite extended on a rolling basis. Every run is marked by a deterministic verifier; uncertainty is reported with a paired, task-level bootstrap; and cost, agent rounds and wall-clock time are reported alongside the score rather than folded into it. This version of the paper describes the design of the benchmark, its evaluation protocol and its current tasks and Skills. The evaluation is in progress: results will be added once the runs are complete, and the task suite, results and analysis will continue to be updated on a rolling basis.

Figures & tables

Explore similar work

CardsList
  1. SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications

    Oct 10, 2026Zhengxi Li, Junyi Zhou, Qiyuan Zhang +6Agent EvaluationLLM Agent Evaluation

  2. Demystifying Agent Skills: Why They Work-Until They Don't

    Aug 14, 2026Zhiyuan Jiang, Fangrui Huang, Hanwen Xing +6LLM Agent Skill EvaluationLLM Agents

  3. SkillTester: Benchmarking Utility and Security of Agent Skills

    Mar 28, 2026Leye Wang, Zixing Wang, Anjie XuLLM Agent Skill EvaluationAI Agent Security Benchmarks