SkillBuffBench: Measuring What Agent Skills Contribute across Harnesses, Models, Skill Access Modes and Memory
Authors: Gengyue Dong, Ye Wei, Zheng Zhu, Yidong Han, Jiale Miao, Xiaojun Jia, Guanghui Li
Organizations: Department of Engineering Science, University of Oxford, Oxford, UK · MRC Weatherall Institute of Molecular Medicine, University of Oxford, Oxford, UK · Department of Materials, Imperial College London, London, UK · Department of Mathematics, University College London, London, UK · Nanyang Technological University, Singapore · BraneMatrix AI, Shanghai, China
Agent Skills, file-based packages of procedural knowledge that a coding agent can discover and load on demand, are now shipped by every major agent harness, yet the evidence for how much they help comes almost entirely from settings in which the right Skill is hand-picked for the task and the agent starts from a blank slate. Neither holds in deployment, where an agent chooses from whatever library happens to be installed and frequently carries memory from earlier work. We present SkillBuffBench, a benchmark and paired evaluation protocol that treats Skill access as a three-level factor (no Skill, a curated Skill, and automatic invocation from an installed library), crosses it with memory (off and on), with four agent harnesses and with multiple models (the state-of-the-art proprietary models from Anthropic, Google and OpenAI, and the state-of-the-art open-weight models from Chinese providers), every model being run under every harness, and evaluates the result on research tasks contributed by domain experts together with the Skills they use, currently in materials science and biology, with the suite extended on a rolling basis. Every run is marked by a deterministic verifier; uncertainty is reported with a paired, task-level bootstrap; and cost, agent rounds and wall-clock time are reported alongside the score rather than folded into it. This version of the paper describes the design of the benchmark, its evaluation protocol and its current tasks and Skills. The evaluation is in progress: results will be added once the runs are complete, and the task suite, results and analysis will continue to be updated on a rolling basis.
Figures & tables
Figure 1: SkillBuffBench at a glance. Expert-contributed research tasks, currently 4 in materials science and biology, are run under a paired factorial design that crosses four harnesses and multiple models, each model run under every harness, with three Skill conditions and two memory modes. A deterministic verifier marks every run; the effect of Skills and of memory is reported with paired bootstrap confidence intervals, next to reliability, efficiency and failure types.
Figure 2: Evaluation protocol. (a) Each task is run under three Skill conditions that differ only in Skill access. The macro-average scores P∅ , PS and PA give the curated gain ΔS and the automatic gain ΔA against the same baseline. (b) Every run starts from a reset environment, and the verifier sees only what the run leaves behind.
Factor
Levels
Values
Harness
4
Ginkley, Codex, Claude Code, OpenCode
Model
n (now 11)
one pool shared by all harnesses, updated as new state-of-the-art models are released (Section 2.2 )
Skill condition
3
∅ no Skill; S curated Skill; A automatic invocation from an installed library
Memory mode
2
M0 memory off, isolated; M1 memory on
Table 1: Experimental factors. “Levels” is the number of values a factor takes; n is the size of the model pool. Each cell is evaluated on all 4 tasks with three repeats.
Group
Provider
Models
Proprietary
Anthropic
Claude Fable 5.2; Claude Opus 5.5
Google
Gemini 4 Argon; Gemini 3.8 Flash
OpenAI
GPT-6 Astra; GPT-6.1 Sol
Open-weight
DeepSeek
DeepSeek-V4-Pro
Alibaba
Qwen3.8
Moonshot AI
Kimi K3.1 Pro
Table 2: The model pool at the time of writing ( n=11 ). Two models from each international provider and one open-weight model from each Chinese provider.
Inputs
Independent variables
harness; model; Skill condition ( ∅ , S , A ); memory mode ( M0 , M1 )
Given to the agent
task prompt; input assets (documents, datasets, starter files); the task’s declared toolset; a fresh, resettable sandbox workspace; under S or A , the Skills or Skill library; under M1 , the memory store
Held fixed
model version; sampling parameters; output-token limit; budgets on agent rounds, wall-clock time and tokens; tool versions; task order; verifier
Hidden from the agent
reference solution; golden values; verifier code
Outputs recorded for every run
Outcome
verifier score in [0,1] ; pass or fail against the task’s threshold; result of each individual check
Table 3: Inputs and outputs of SkillBuffBench.
Failure type
Meaning
Task comprehension
misread the goal, missed a key requirement, or solved a different problem
Tool use
approach roughly right, but a tool call, command argument, script or software operation failed
Data processing
error in reading input, unit conversion, field matching, or handling of missing or anomalous values
Reasoning / planning
task and tools understood, but wrong ordering of steps, experimental design or intermediate decision
Output format
result close to correct, but a required file, column name, schema or document structure is missing
Verification
no self-check, or too few checks run to confirm the artifact is correct
Table 4: Failure taxonomy. Diagnostic only; it does not enter the score.
Task
Given to the agent
Asked for
How it is verified
Materials science (2)
Oxygen tracer diffusion from isotope-exchange SIMS
raw 16 O and 18 O line-scan or depth-profile data with acquisition metadata; the fitting software
the tracer diffusion coefficient D∗ and surface exchange coefficient k∗ , with a review of the surface position, the fit residuals and the data quality, and a statement of how far the result can be trusted
point-by-point residuals of the fit and their RMS; fitted parameters and their 95% confidence intervals against reference values on data with known ground truth; whether the data determine D∗ and k∗ separately; checks of parsing and unit handling; diagnostic findings against annotated cases, where reporting insufficient evidence can be the correct answer
Phase identification and unit-cell refinement from powder XRD
a powder diffraction pattern and the elements present in the sample; access to a crystal-structure database [ 14 ] and to GSAS-II [ 32 ]
the crystalline phase and its refined unit-cell parameters from a Le Bail fit [ 18 ] , with fit statistics and a refinement figure
fit quality by Rwp (below 10%) and χ2 (the closer to one the better); identified phase against the reference phase; refined cell parameters against reference values within tolerance; required artifacts present
Biology (2)
Regulatory target prioritisation
a table of candidate enhancer–gene pairs in one cell context with published predictive scores, from a public CRISPR perturbation study [ 13 ] ; experimental outcomes withheld
a ranking of candidate regulatory links for each gene, an evidence summary that states what is missing, and a proposed follow-up experiment
the frozen ranking is scored against the withheld outcomes by average precision, reported with its coverage; predictions must cover exactly the supplied records; the agent runs without access to the outcome files
Spheroid structure and cell distribution from 3D microscopy
a three-dimensional fluorescence microscopy image stack of a cell spheroid, with its axes, its voxel spacing in physical units and a record of what each channel shows
a statement of which analyses the image supports; the envelope of the spheroid as a three-dimensional mask and surface, with volume and surface area; the positions of the nuclear centres; the density of nuclei as a function of distance from the centre and of depth below the surface; the distances between neighbouring nuclei; and images for reviewing each result
envelope against a manually drawn reference (overlap, surface distance, volume error); nuclear centres matched one to one with annotated centres; required artifacts present; no claim the image cannot support
Table 5: Task suite in its present state: 4 tasks in 2 domains.
Evaluating an agent's ability to strictly satisfy diverse and procedurally complex requirements is essential to determining its practical reliability. Conducting such an evaluation demands agentic tasks that reconcile broad domain coverage with deep procedural complexity. Existing paradigms face a fundamental trade-off: expert-curated benchmarks achieve procedural depth but suffer from domain diversity bottlenecks, whereas crowdsourced benchmarks span diverse scenarios but lack execution rigor. To resolve this tension, we present SkillForgeBench, which leverages agent skills as unifying execution blueprints across three co-designed stages: (1) Resource Preparation uses domain-balanced skills to guide artifact retrieval, establishing broad workflow coverage; (2) Skill-driven Task Synthesis employs skills as procedural anchors to partition artifact roles, synthesize naturalistic task instructions, and package sandboxed scenarios for deep procedural execution; and (3) Holistic Evaluation Design translates skill-embedded requirements into fine-grained completion and process rubrics to enable strict requirement-level scoring. Comprising 842 sandboxed tasks across five major professional domains, SkillForgeBench reveals a steep capability hierarchy among eight mainstream agents, led by Opus 4.8 (40.9% strict completion). Requiring all constrains satisfaction uncovers severe capability bottlenecks across all model tiers that remain masked under individual rubric pass rates (70.0% to 85.7% vs. 12.6% to 40.9%). Crucially, under the external guidance from skills, top-tier agents obtain larger gains than low-tier agents (i.e., Δ19.2% for Opus 4.8 vs. Δ5.8% for Haiku 4.5). Finally, agents satisfy process rubrics far less frequently than completion rubrics (under 3% for Haiku 4.5), proving that completion-only checks overestimate real-world agent reliability.
Zhengxi Li, Junyi Zhou, Qiyuan Zhang +6
MBZUAI · City University of Hong Kong · WeChat AI, Tencent +1
Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: When do skills help, why do they work, and where do they fail? Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7% of skill cases, versus 4.5% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6% to 3.3%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.
Zhiyuan Jiang, Fangrui Huang, Hanwen Xing +6
Princeton University · Stanford University · University of Southern California +2
This technical report presents SkillTester, a tool for evaluating the utility and security of agent skills. Its evaluation framework combines paired baseline and with-skill execution conditions with a separate security probe suite. Grounded in a comparative utility principle and a user-facing simplicity principle, the framework normalizes raw execution artifacts into a utility score, a security score, and a three-level security status label. More broadly, it can be understood as a comparative quality-assurance harness for agent skills in an agent-first world. The public service is deployed at https://skilltester.ai, and the broader project is maintained at https://github.com/skilltester-ai/skilltester.
Leye Wang, Zixing Wang, Anjie Xu
Key Laboratory of High Confidence Software Technologies, Ministry of Education, Peking University, Beijing, China · Northwestern Polytechnical University, Xi’an, China