Signal or Noise? A Benchmark Study of Agent Skills in Web Development

Ziyue Yang; Fan Ding

cs.CLAug 24, 2026

Signal or Noise? A Benchmark Study of Agent Skills in Web Development

Authors: Ziyue Yang, Fan Ding

Organizations: Baidu, NLP Beijing, China

Abstract

Agent Skills are reusable procedural modules that are increasingly injected into coding-agent sessions to encode framework conventions, anti-patterns, and reusable tools. However, because each injected Skill expands the prompt of every query, an effective Skill benchmark must determine not only whether an agent can solve a task, but whether the Skill should have been injected at all. We introduce WebDev-Skills-Bench and use it for a controlled empirical study of 31 public WebDev Skills on 50 Web-Bench projects and 1,000 ordered tasks. The benchmark compares four matched conditions, including a length-matched irrelevant control and leave-one-out component ablations. To isolate Skill effects from prompt-length artifacts, we place only SKILL.md in the prompt while mounting auxiliary files into the agent workspace. Across four models, target Skill injection reduces mean Pass@2 by 1.3% to 4.2%, lowers task completion depth, and increases token cost by 72% to 394%, with gains in only 17% to 36% of Skill-project pairs. Length-matched controls reveal two failure modes: some models are length-distracted, where an equally long irrelevant Skill reproduces most of the loss, while others are content-misled, where prompt length is neutral but Skill content still lowers Pass@2 by 1.1% to 1.4%. Further analysis shows that losses concentrate on easy early tasks, Skill rankings transfer weakly across models, and anti-pattern rules outperform example-heavy content within helpful Skills. These findings recast a matched Skill as a hypothesis about a particular Skill-project-model triple rather than a portable asset, reframing injection as a per-deployment routing decision and making length-matched controls and per-model audits a minimum standard for Agent-Skill evaluation.

Figures & tables

Model	$N$	$\Delta$ P@1 (pp)	$\Delta$ P@2 (pp)	$\Delta$ TCD	Win/Tie/Loss (%)	$\rho$ (tokens)
Claude Sonnet 4	3	$-2.8$ [ $-4.6,-1.1$ ]	$-4.2$ [ $-6.9,-1.9$ ]	$-0.85$	$30/9/61$	$+\phantom{0}72$ %
GPT-5.1	3	$-1.6$ [ $-3.3,-0.1$ ]	$-1.3$ [ $-3.4,+0.5$ ]	$-0.26$	$35/20/45$	$+\phantom{0}74$ %
Qwen3 Coder 30B	3	$-3.2$ [ $-4.3,-2.1$ ]	$-2.3$ [ $-3.4,-1.3$ ]	$-0.47$	$17/35/48$	$+\phantom{0}91$ %
DeepSeek V4 Flash	3	$-4.1$ [ $-5.8,-2.2$ ]	$-2.0$ [ $-3.8,-0.2$ ]	$-0.40$	$36/16/48$	$+394$ %

Table 1: Model-level C1

-

C0 effects on the 117 core pairs (

N{=}3

per cell). Brackets give 95% paired-bootstrap CIs.

\rho

is relative total-token overhead.

Model	easy	moderate	challenging
Sonnet 4	$\mathbf{-5.3}$ [ $-10.0,-0.7$ ]	$+3.3$ [ $-2.7,+8.9$ ]	$-2.6$ [ $-11.5,+6.4$ ]
GPT-5.1	$\mathbf{-4.0}$ [ $-7.8,-0.4$ ]	$-7.5$ [ $-23.3,+7.5$ ]	$+6.6$ [ $-12.2,+22.7$ ]
Qwen3 Coder 30B	$\mathbf{-10.7}$ [ $-16.7,-5.4$ ]	—	—
DeepSeek V4 Flash	$\mathbf{-7.3}$ [ $-14.6,-0.5$ ]	$+11.7$ [ $+0.4,+24.3$ ]	$+0.8$ [ $-10.0,+10.4$ ]

Table 2:

\Delta

Pass@2 (pp) by Web-Bench task difficulty (

N{=}3

); brackets are 95% paired-bootstrap CIs. Bold marks a CI that excludes zero. A dash marks fewer than five pair-cells.

Model	$\Delta$ Total	$\Delta$ Length	$\Delta$ Content	Survival
Sonnet 4	$-4.2$	$-3.3$	$-0.9$	$60\%$
Qwen	$-2.3$	$-3.5$	$+1.2$	$95\%$
GPT-5.1	$-1.3$	$-0.2$	$-1.1$	$71\%$
DeepSeek	$-2.0$	$-0.6$	$-1.4$	$64\%$

Table 3: Length/content decomposition (mean

\Delta

Pass@2 in pp,

N{=}3

). Survival is the share of C1 wins that remain positive after the length control.

	DeepSeek	GPT-5.1	Qwen
GPT-5.1	$-0.00$
Qwen	$+0.09$	$+0.12$
Sonnet 4	$-0.07$	$-0.06$	$-0.08$

Table 4: Cross-model Pearson correlation of per-pair

\Delta

Pass@2 (

C1{-}C0

) over the 117 core pairs (

N{=}3

). All coefficients are near zero (

|r|\leq 0.12

; Spearman is similar,

|\rho_{s}|\leq 0.16

): a Skill’s measured utility on one model barely predicts its utility on another.

Slice	All ( $N{=}20$ )	$-$ Sonnet ( $N{=}15$ )
Whole Skill ( $C1{-}C0$ )	$+5.1$ [ $1.0,9.3$ ]	$+6.1$ [ $1.6,10.9$ ]
$R_{p}$ (positive rules)	$+0.0$ [ $-5.1,4.9$ ]	$+4.6$ [ $1.1,8.4$ ]
$R_{n}$ (anti-patterns)	$+3.1$ [ $-3.3,9.8$ ]	$+6.1$ [ $0.3,12.9$ ]
$X$ (examples)	$-0.7$ [ $-5.6,3.6$ ]	$+4.2$ [ $2.0,6.8$ ]

Table 5: C3 slice contributions to Pass@2 (pp),

C1-(\text{variant})

; cell-level means with paired-bootstrap CIs. “All” pools 20 cells; “

-

Sonnet” the 15 non-Sonnet cells. Positive = slice helps. Wilcoxon signed-rank and McNemar significance tests are reported in Appendix D .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	C0 P@2	C1 P@2	$\Delta$ P@2 (pp)
Sonnet 4	$33\%$	$67\%$	$+33$
GPT-5.1	$22\%$	$20\%$	$-2$
DeepSeek	$42\%$	$20\%$	$-22$
Qwen	$22\%$	$\phantom{0}0\%$	$-22$

Appendix

Table 6: Cross-model effects on lowdb

\times

database-optimizer (S14), an extreme, illustrative single pair (

N{=}3

). The same Skill content produces opposite-signed effects across the panel.

Figure 3: Per-slice contribution to Pass@2, pooled over the 20 model $\times$ project cells (bar = mean, whisker = paired-bootstrap CI; each gray dot is one cell). Bars are green when positive and red when negative, matching the shading of Table 5 . Only anti-patterns ( $R_{n}$ ) sit reliably above zero; positive rules and examples are null on average with wide per-cell spread.

Figure 4: Contribution of example code ( $X$ ) to Pass@2 by model ( $C1-(-X)$ , cell mean with paired-bootstrap CI). Examples help DeepSeek, Qwen, and weakly GPT-5.1 but hurt Sonnet, so the pooled effect is near zero.

LLM Agent Skill Evaluation

See all 26 papers

SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
Mar 16, 2026Tingxu Han, Yi Zhang, Wei Song +4Software Engineering Agents Software Engineering Benchmarks
Nanjing University · Mohamed bin Zayed University of Artificial Intelligence · South China University of Technology +1
Skill Coverage: A Test Adequacy Metric for Agent Skills
Jun 9, 2026Boyin Tan, Xiaowei Huang, Youcheng SunLLM Agent Evaluation AI Agent Benchmarks
1Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates · University of Liverpool Liverpool, United Kingdom
A Framework for Evaluating Agentic Skills at Scale
Jun 16, 2026Maksim Shaposhnikov, Nicolas Fortuin, Simon Stipcich +3LLM Agent Skill Evaluation LLM Agent Evaluation
Tessl London, United Kingdom

Mar 16, 2026J/K move · Enter open · S save

SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang+3

Nanjing University · Mohamed bin Zayed University of Artificial Intelligence · South China University of Technology+1

Software Engineering Agents Software Engineering Benchmarks LLM Agent Evaluation

Open paper AskPDF arXiv

Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear. We present SWE-Skills-Bench, the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering (SWE). It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria, yielding approximately 565 task instances across six SWE subdomains. We introduce a deterministic verification framework that maps each task's acceptance criteria to execution-based tests, enabling controlled paired evaluation with and without the skill. Our results show that skill injection benefits are far more limited than rapid adoption suggests: 39 of 49 skills yield zero pass-rate improvement, and the average gain is only +1.2%. Token overhead varies from modest savings to a 451% increase while pass rates remain unchanged. Only seven specialized skills produce meaningful gains (up to +30%), while three degrade performance (up to -10%) due to version-mismatched guidance conflicting with project context. These findings suggest that agent skills are a narrow intervention whose utility depends strongly on domain fit, abstraction level, and contextual compatibility. SWE-Skills-Bench provides a testbed for evaluating the design, selection, and deployment of skills in software engineering agents. SWE-Skills-Bench is available at https://github.com/GeniusHTX/SWE-Skills-Bench.

Abstract

LLM Agent Skill Evaluation

Explore similar work

SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

Skill Coverage: A Test Adequacy Metric for Agent Skills

A Framework for Evaluating Agentic Skills at Scale