SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
Authors: Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, Lijie Hu
Organizations: Nanjing University · Mohamed bin Zayed University of Artificial Intelligence · South China University of Technology · The University of New South Wales
Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear. We present SWE-Skills-Bench, the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering (SWE). It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria, yielding approximately 565 task instances across six SWE subdomains. We introduce a deterministic verification framework that maps each task's acceptance criteria to execution-based tests, enabling controlled paired evaluation with and without the skill. Our results show that skill injection benefits are far more limited than rapid adoption suggests: 39 of 49 skills yield zero pass-rate improvement, and the average gain is only +1.2%. Token overhead varies from modest savings to a 451% increase while pass rates remain unchanged. Only seven specialized skills produce meaningful gains (up to +30%), while three degrade performance (up to -10%) due to version-mismatched guidance conflicting with project context. These findings suggest that agent skills are a narrow intervention whose utility depends strongly on domain fit, abstraction level, and contextual compatibility. SWE-Skills-Bench provides a testbed for evaluating the design, selection, and deployment of skills in software engineering agents. SWE-Skills-Bench is available at https://github.com/GeniusHTX/SWE-Skills-Bench.
Figures & tables
Figure 1: Illustration of how agent skills are used in a software engineering workflow. Given a natural-language requirement, the LLM-based agent selects the most relevant skill from its skill library, including skills such as writing code, running tests, debugging, creating pull requests, and deploying, and injects it into the context window. The agent then executes a series of SWE actions to produce the final software artifacts (such as code) that fulfill the requirement.
Benchmark
Size
Skill Cond.
Real Projects
Det. Verifier
SWE-Focused
SWE-Bench Verified Jimenez et al. (2024)
500
None
Yes
Yes
Yes
TerminalBench Merrill and others (2026)
200
None
Yes
Yes
Yes
HumanEval Chen and others (2021)
164
None
No
Partial
No
SkillsBench Li et al. (2026)
84
Yes
Yes
Yes
Partial
SWE-Skills-Bench
565
Yes
Yes
Yes
Yes
Table 1: Comparison of SWE-Skills-Bench with existing benchmarks. “Skill Cond.” indicates whether the benchmark includes agent skills. “Det. Verifier” indicates whether deterministic (non-LLM) verification is included. “SWE-Focused” indicates whether the benchmark is specifically designed for software engineering tasks.
Figure 2: The distribution of the curated skills and generated tasks.
Figure 3: Overview of the SWE-Skills-Bench construction pipeline. We begin with 84,192 public skills and narrow them down through three filtering stages: category selection, semantic filtering, and feasibility screening. This process yields 49 SWE skills (Stage 1). Next, for each skill, we identify a matching GitHub project and generate 565 task instances of the form (R,E,P,S) (Stage 2). For each criterion in the requirements document P , we build deterministic verifiers using pytest unit tests (Stage 3). Finally, we run a paired evaluation that compares agent performance with and without the SKILL.md file, allowing us to measure the effectiveness of the skill (Stage 4).
Figure 4: The pipeline of task instance generation.
Skills
# Tasks
Pass+
Pass−
ΔP
C+
C−
ρ
CE
add-uint-support
12
100.0%
100.0%
0.0%
880K
414K
+112.6%
—
analytics-events
10
100.0%
100.0%
0.0%
321K
157K
+104.6%
—
analyze-ci
11
100.0%
100.0%
0.0%
66K
74K
-10.6%
—
dbt-transformation-patterns
10
100.0%
100.0%
0.0%
422K
208K
+103.2%
—
gitops-workflow
13
100.0%
100.0%
0.0%
130K
57K
+127.1%
—
grafana-dashboards
12
100.0%
100.0%
0.0%
150K
116K
+29.3%
—
Table 2: Evaluation results across all 49 skills. Pass+ and Pass− denote pass rates with and without skill injection, respectively. ΔP is the skill utility delta, C+ and C− are average token costs, ρ is the token overhead ratio, and CE is cost efficiency. Best viewed in color.
Figure 5: Context interference in the linkerd-patterns skill ( ΔP=−9.1% ). The task requires a Server CRD and a ServerAuthorization CRD enforcing mTLS identity verification for a gRPC service. Left: Template 5 from the injected skill, which near-matches the task but encodes different concrete values: API version v1beta1 with proxyProtocol: HTTP/1 , and multiple authorization modes (meshTLS, unauthenticated, and CIDR-based). Center: Without the skill, the agent reasons from first principles and produces a correct solution using v1beta3 , gRPC , and standard meshTLS.serviceAccounts . Right: With the skill, the agent’s output degrades through three stages, each traceable to a specific region of the template (matched by circled numbers): 1 Surface anchoring, the agent copies v1beta1 and HTTP/1 verbatim; 2 Hallucination, while reconciling the template’s mixed authorization modes, the agent fabricates a nonexistent rules/metricsServers field; 3 Concept bleed, the template’s NetworkPolicy example causes the agent to append an unrequested resource, conflating Linkerd-level and Kubernetes-level authorization.
Figure 6: The prompt used for requirement-driven verification generation.
Figure 7: The prompt used for task instance requirement generation.
Figure 8: An example of the generated requirement in SWE-Skills-Bench.
Agent skills are increasingly used to configure coding agents for software engineering (SE) tasks, yet current practice treats them as static, hand-crafted assets, or evolved on pass rate alone. This is insufficient: a skill can improve task success while substantially raising token cost, or introducing misleading guidance. We argue that SE agent skill bundles can be treated as multi-objective search objects and present SkillMOO, a framework that evolves skill bundles through LLM-proposed edits and NSGA-II Pareto selection on pass rate and inference cost. Evaluated across all 16 SkillsBench SE tasks, SkillMOO achieves the top pass rate rank on 11 of 12 non-zero-pass tasks while achieving cost reductions of up to 31.7% over static bundles, with pass rate gains up to 21 percentage points. Analysis of 38 skill edits shows that pruning and substitution dominate successful operations, offering actionable principles for skill bundle design. Thereby, the current practice of deploying skills without cost-aware validation leaves better skill configurations unexplored, motivating a new class of cost-aware, search-based skill engineering.
Jingzhi Gong, Ruizhen Gu, Zhiwei Fei +7
King’s College London, London, United Kingdom · Queen’s University Belfast, Belfast, United Kingdom · Nanjing University, Nanjing, China +3
Agent Skills are reusable procedural modules that are increasingly injected into coding-agent sessions to encode framework conventions, anti-patterns, and reusable tools. However, because each injected Skill expands the prompt of every query, an effective Skill benchmark must determine not only whether an agent can solve a task, but whether the Skill should have been injected at all. We introduce WebDev-Skills-Bench and use it for a controlled empirical study of 31 public WebDev Skills on 50 Web-Bench projects and 1,000 ordered tasks. The benchmark compares four matched conditions, including a length-matched irrelevant control and leave-one-out component ablations. To isolate Skill effects from prompt-length artifacts, we place only SKILL.md in the prompt while mounting auxiliary files into the agent workspace. Across four models, target Skill injection reduces mean Pass@2 by 1.3% to 4.2%, lowers task completion depth, and increases token cost by 72% to 394%, with gains in only 17% to 36% of Skill-project pairs. Length-matched controls reveal two failure modes: some models are length-distracted, where an equally long irrelevant Skill reproduces most of the loss, while others are content-misled, where prompt length is neutral but Skill content still lowers Pass@2 by 1.1% to 1.4%. Further analysis shows that losses concentrate on easy early tasks, Skill rankings transfer weakly across models, and anti-pattern rules outperform example-heavy content within helpful Skills. These findings recast a matched Skill as a hypothesis about a particular Skill-project-model triple rather than a portable asset, reframing injection as a per-deployment routing decision and making length-matched controls and per-model audits a minimum standard for Agent-Skill evaluation.
SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present SWE-Bench Pro Verified, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines anti-hacking safeguards that eliminate major leakage channels without disrupting normal agent functionality, with task refinement that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously evaluated, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.
Pujun Zheng, Zixin Shang, Shufan Jiang +5
East China Normal University · Shanghai Artificial Intelligence Laboratory · Fudan University