SkillBuffBench: Measuring What Agent Skills Contribute across Harnesses, Models, Skill Access Modes and Memory
Authors: Gengyue Dong, Ye Wei, Zheng Zhu, Yidong Han, Jiale Miao, Xiaojun Jia, Guanghui Li
Organizations: Department of Engineering Science, University of Oxford, Oxford, UK · MRC Weatherall Institute of Molecular Medicine, University of Oxford, Oxford, UK · Department of Materials, Imperial College London, London, UK · Department of Mathematics, University College London, London, UK · Nanyang Technological University, Singapore · BraneMatrix AI, Shanghai, China
Agent Skills, file-based packages of procedural knowledge that a coding agent can discover and load on demand, are now shipped by every major agent harness, yet the evidence for how much they help comes almost entirely from settings in which the right Skill is hand-picked for the task and the agent starts from a blank slate. Neither holds in deployment, where an agent chooses from whatever library happens to be installed and frequently carries memory from earlier work. We present SkillBuffBench, a benchmark and paired evaluation protocol that treats Skill access as a three-level factor (no Skill, a curated Skill, and automatic invocation from an installed library), crosses it with memory (off and on), with four agent harnesses and with multiple models (the state-of-the-art proprietary models from Anthropic, Google and OpenAI, and the state-of-the-art open-weight models from Chinese providers), every model being run under every harness, and evaluates the result on research tasks contributed by domain experts together with the Skills they use, currently in materials science and biology, with the suite extended on a rolling basis. Every run is marked by a deterministic verifier; uncertainty is reported with a paired, task-level bootstrap; and cost, agent rounds and wall-clock time are reported alongside the score rather than folded into it. This version of the paper describes the design of the benchmark, its evaluation protocol and its current tasks and Skills. The evaluation is in progress: results will be added once the runs are complete, and the task suite, results and analysis will continue to be updated on a rolling basis.
Figures & tables
Figure 1: SkillBuffBench at a glance. Expert-contributed research tasks, currently 4 in materials science and biology, are run under a paired factorial design that crosses four harnesses and multiple models, each model run under every harness, with three Skill conditions and two memory modes. A deterministic verifier marks every run; the effect of Skills and of memory is reported with paired bootstrap confidence intervals, next to reliability, efficiency and failure types.
Figure 2: Evaluation protocol. (a) Each task is run under three Skill conditions that differ only in Skill access. The macro-average scores P∅ , PS and PA give the curated gain ΔS and the automatic gain ΔA against the same baseline. (b) Every run starts from a reset environment, and the verifier sees only what the run leaves behind.
Factor
Levels
Values
Harness
4
Ginkley, Codex, Claude Code, OpenCode
Model
n (now 11)
one pool shared by all harnesses, updated as new state-of-the-art models are released (Section 2.2 )
Skill condition
3
∅ no Skill; S curated Skill; A automatic invocation from an installed library
Memory mode
2
M0 memory off, isolated; M1 memory on
Table 1: Experimental factors. “Levels” is the number of values a factor takes; n is the size of the model pool. Each cell is evaluated on all 4 tasks with three repeats.
Group
Provider
Models
Proprietary
Anthropic
Claude Fable 5.2; Claude Opus 5.5
Google
Gemini 4 Argon; Gemini 3.8 Flash
OpenAI
GPT-6 Astra; GPT-6.1 Sol
Open-weight
DeepSeek
DeepSeek-V4-Pro
Alibaba
Qwen3.8
Moonshot AI
Kimi K3.1 Pro
Table 2: The model pool at the time of writing ( n=11 ). Two models from each international provider and one open-weight model from each Chinese provider.
Inputs
Independent variables
harness; model; Skill condition ( ∅ , S , A ); memory mode ( M0 , M1 )
Given to the agent
task prompt; input assets (documents, datasets, starter files); the task’s declared toolset; a fresh, resettable sandbox workspace; under S or A , the Skills or Skill library; under M1 , the memory store
Held fixed
model version; sampling parameters; output-token limit; budgets on agent rounds, wall-clock time and tokens; tool versions; task order; verifier
Hidden from the agent
reference solution; golden values; verifier code
Outputs recorded for every run
Outcome
verifier score in [0,1] ; pass or fail against the task’s threshold; result of each individual check
Table 3: Inputs and outputs of SkillBuffBench.
Failure type
Meaning
Task comprehension
misread the goal, missed a key requirement, or solved a different problem
Tool use
approach roughly right, but a tool call, command argument, script or software operation failed
Data processing
error in reading input, unit conversion, field matching, or handling of missing or anomalous values
Reasoning / planning
task and tools understood, but wrong ordering of steps, experimental design or intermediate decision
Output format
result close to correct, but a required file, column name, schema or document structure is missing
Verification
no self-check, or too few checks run to confirm the artifact is correct
Table 4: Failure taxonomy. Diagnostic only; it does not enter the score.
Task
Given to the agent
Asked for
How it is verified
Materials science (2)
Oxygen tracer diffusion from isotope-exchange SIMS
raw 16 O and 18 O line-scan or depth-profile data with acquisition metadata; the fitting software
the tracer diffusion coefficient D∗ and surface exchange coefficient k∗ , with a review of the surface position, the fit residuals and the data quality, and a statement of how far the result can be trusted
point-by-point residuals of the fit and their RMS; fitted parameters and their 95% confidence intervals against reference values on data with known ground truth; whether the data determine D∗ and k∗ separately; checks of parsing and unit handling; diagnostic findings against annotated cases, where reporting insufficient evidence can be the correct answer
Phase identification and unit-cell refinement from powder XRD
a powder diffraction pattern and the elements present in the sample; access to a crystal-structure database [ 14 ] and to GSAS-II [ 32 ]
the crystalline phase and its refined unit-cell parameters from a Le Bail fit [ 18 ] , with fit statistics and a refinement figure
fit quality by Rwp (below 10%) and χ2 (the closer to one the better); identified phase against the reference phase; refined cell parameters against reference values within tolerance; required artifacts present
Biology (2)
Regulatory target prioritisation
a table of candidate enhancer–gene pairs in one cell context with published predictive scores, from a public CRISPR perturbation study [ 13 ] ; experimental outcomes withheld
a ranking of candidate regulatory links for each gene, an evidence summary that states what is missing, and a proposed follow-up experiment
the frozen ranking is scored against the withheld outcomes by average precision, reported with its coverage; predictions must cover exactly the supplied records; the agent runs without access to the outcome files
Spheroid structure and cell distribution from 3D microscopy
a three-dimensional fluorescence microscopy image stack of a cell spheroid, with its axes, its voxel spacing in physical units and a record of what each channel shows
a statement of which analyses the image supports; the envelope of the spheroid as a three-dimensional mask and surface, with volume and surface area; the positions of the nuclear centres; the density of nuclei as a function of distance from the centre and of depth below the surface; the distances between neighbouring nuclei; and images for reviewing each result
envelope against a manually drawn reference (overlap, surface distance, volume error); nuclear centres matched one to one with annotated centres; required artifacts present; no claim the image cannot support
Table 5: Task suite in its present state: 4 tasks in 2 domains.
Key Laboratory of High Confidence Software Technologies, Ministry of Education, Peking University, Beijing, China · Northwestern Polytechnical University, Xi’an, China