SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?

Shiqi Chen; Jingze Gai; Ruochen Zhou; Jinghan Zhang; Tongyao Zhu; Junlong Li; Kangrui Wang; Zihan Wang

cs.CLFeb 28, 2026

SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?

Authors: Shiqi Chen, Jingze Gai, Ruochen Zhou, Jinghan Zhang, Tongyao Zhu, Junlong Li, Kangrui Wang, Zihan Wang,

Organizations: University of Oxford · City University of Hong Kong · Hong Kong University of Science and Technology · National University of Singapore ·

Abstract

Real-world tool-using agents operate over long-horizon workflows with recurring structure and diverse demands, where effective behavior requires not only invoking atomic tools but also abstracting, and reusing higher-level tool compositions. However, existing benchmarks mainly measure instance-level success under static tool sets, offering limited insight into agents' ability to acquire such reusable skills. We address this gap by introducing SkillCraft, a benchmark explicitly stress-test agent ability to form and reuse higher-level tool compositions, where we call Skills. SkillCraft features realistic, highly compositional tool-use scenarios with difficulty scaled along both quantitative and structural dimensions, designed to elicit skill abstraction and cross-task reuse. We further propose a lightweight evaluation protocol that enables agents to auto-compose atomic tools into executable Skills, cache and reuse them inside and across tasks, thereby improving efficiency while accumulating a persistent library of reusable skills. Evaluating state-of-the-art agents on SkillCraft, we observe substantial efficiency gains, with token usage reduced by up to 80% by skill saving and reuse. Moreover, success rate strongly correlates with tool composition ability at test time, underscoring compositional skill acquisition as a core capability.

Figures & tables

Figure 1: Skill Mode demo. Demonstrating how skills are automatically discovered, cached locally, and subsequently reused.

Figure 2: SkillCraft Protocol Pipeline Overview. The pipeline consists of three stages: (1) Test-Time Tool-Chain Evolution: The agent solves tasks from the Task Library by exploring and chaining atomic tools, forming executable tool sequences. (2) Iterative Skill Composition: Successful sequences are abstracted into candidate skills, executed and verified in a coding environment; failed executions trigger re-exploration, while validated skills are stored. (3) Skill Library and Reuse: A growing repository of verified, reusable skills that can be retrieved in later tasks to replace low-level tool exploration, enabling test-time skill accumulation and efficient composition.

Figure 3: Three-stage task construction pipeline for SkillCraft . In Stage 1 , we explore existing benchmarks through systematic experimentation to identify effective task design principles . In Stage 2 , we construct seed tasks from three sources: (i) selected tasks from Stage 1 with unified interfaces , (ii) newly handcrafted web API-based tasks , and (iii) local file and data processing tasks . In Stage 3 , we systematically scale the seed tasks via quantitative scaling (increasing subtask count) and complexity scaling (increasing tool calls per subtask), producing a task repository with graduated difficulty levels .

Figure 4: Task distribution in SkillCraft. The chart shows 21 task families across 6 application domains. The table summarizes difficulty levels: Entity Num = number of target items (subtasks) per task; Complexity = tool calls required per entity.

Stage	Description	Source	#Tasks
Stage 1: Exploratory Phase	Explore existing benchmarks to identify task design principles	Existing benchmarks like Toolathlon, WebArena, TextArena, M3Eval, etc.	60+
Stage 2: Seed Task Creation	(i) Select & adapt quality tasks from Stage 1	Existing benchmarks	5
	(ii) Handcraft web API-based tasks	GitLab, OpenMeteo, etc.	12
	(iii) Handcraft local processing tasks	Custom datasets & files	4
	Total Seed Tasks		21
Stage 3: Systematic Scaling	Scale entity number (N: 3→4→5)	From Seed tasks	126

Table 1: Task statistics across the three-stage construction pipeline. In Stage 1 , we explore 60+ tasks from existing benchmarks to identify effective task design principles. In Stage 2 , we construct 21 seed tasks from three sources: adapted benchmark tasks, handcrafted web API-based tasks, and local processing tasks. In Stage 3 , we systematically scale seed tasks by increasing entity number and subtask complexity , producing 126 tasks across 6 difficulty levels.

Model	Skill Stats		Avg Tokens			Avg Cost ($)			Avg Turns			Avg Tool Calls			Success Rate (Overall)		Success Rate (Hard)
Model	Exec	Reuse	Base	Skill	Diff	Base	Skill	Diff	Base	Skill	Diff	Base	Skill	Diff	Base	Skill	Base	Skill
Open-Source Models
Kimi-K2-Thinking	70%	3.4 $\times$	0.51M	0.30M	-42%	0.21	0.13	-39%	6.7	8.3	+24%	16.8	11.9	-29%	55/126 (44%)	56/126 (44%)	8/21 (38%)	7/21 (33%)
DeepSeek-V3.2-EXP	71%	4.8 $\times$	1.04M	0.53M	-49%	0.21	0.10	-51%	18.8	15.4	-18%	19.2	14.9	-23%	76/126 (60%)	87/126 (69%)	9/21 (42%)	15/21 (71%)
DeepSeek-R1	62%	3.4 $\times$	0.58M	0.41M	-30%	0.24	0.18	-24%	9.0	9.9	+10%	13.4	11.7	-12%	89/126 (71%)	101/126 (80%)	11/21 (52%)	15/21 (71%)
GLM-4.7	91%	3.7 $\times$	0.78M	0.48M	-39%	0.20	0.12	-41%	13.5	13.0	-4%	16.9	13.3	-21%	91/126 (72%)	108/126 (86%)	12/21 (57%)	15/21 (71%)

Table 2: Results (base vs skill mode) across models on 126 tasks. Success Rate (Overall) : task completion rate (score

\geq

90) for Baseline (no skills) and Skill (with skills) modes, plus Success Rate (Hard) for the hard subset only. Skill Stats : Exec = skill execution success rate; Reuse = average times each skill is invoked. Efficiency metrics (Tokens, Cost, Turns, Tools): per-task averages computed over tasks where both modes succeeded; each shows Base , Skill , and Diff values. Diff : percentage change (Skill

-

Baseline) / Baseline; negative values indicate improvement, positive values indicate degradation.

Figure 5: Cross-metric correlation heatmap. Metrics are grouped into four categories: Success, Skill, Eff_Base, and Eff_Save. Key findings: (1) Skill execution rate correlates with task success (r=0.65); (2) Stronger models achieve greater efficiency gains from skills (r=0.53).

Model	Mode	Success		Skill		Tokens		Cost ($)		Turns		Tools
Model	Mode	N	%	Ex	Re	Val	$\Delta$	Val	$\Delta$	Val	$\Delta$	Val	$\Delta$
DeepSeek-V3.2	Base	76	60	–	–	1.04M	–	0.21	–	18.8	–	19.2	–
	Skill	87	69	71	4.8	0.53M	-49	0.10	-51	15.4	-18	14.9	-23
	Hier	92	73	75	3.0	0.68M	-36	0.10	-66	16.5	-13	15.4	-20
Claude-4.5	Base	119	94	–	–	1.36M	–	1.08	–	15.3	–	14.3	–
	Skill	121	96	81	3.4	0.40M	-71	0.28	-74	10.2	-33	9.2	-36

Table 3: Three-mode comparison across models. Base : No skill library. Skill : With skill library. Hier : Hierarchical mode with skill nesting. N : Number of successful tasks out of 126 total. Ex : Execution success rate (%). Re : Reuse factor (

\times

\Delta

: Relative change vs. Base (%).

Figure 6: (a) Hierarchical skill composition in Iteration mode. A task organized as a depth-3 skill hierarchy, where atomic tools are encapsulated by low-level skills , composed into medium-level skills with additional processing, and orchestrated by a high-level skill . Efficiency gains compound across levels. (b) Error propagation in hierarchical skills. A null value returned by a low-level skill triggers a TypeError in the medium-level skill, which cascades into complete failure of the high-level skill. The tree structure amplifies the impact of edge-case bugs.

Model	Setting	Success Rate		Skill	Avg Tokens			Avg Cost ($)
Model	Setting	Base	Skill	Exec	Base	Skill	Diff	Base	Skill	Diff
Claude	E $\rightarrow$ H	20/21 (95%)	21/21 (100%)	100%	1.92	1.56	-19%	1.41	1.07	-24%
	H $\rightarrow$ E	60/63 (95%)	60/63 (95%)	97%	1.06	0.69	-35%	0.81	0.44	-45%
	H $\rightarrow$ H	20/21 (95%)	20/21 (95%)	98%	1.96	0.47	-76%	1.46	0.43	-71%
Gemini	E $\rightarrow$ H	16/21 (76%)	19/21 (90%)	99%	1.33	0.78	-41%	1.26	0.76	-39%
	H $\rightarrow$ E	55/63 (87%)	60/63 (95%)	100%	0.55	0.36	-35%	0.46	0.30	-35%

Table 4: Cross-task skill generalization results. E

\rightarrow

H : Skills learned from easy tasks (e1–e3) transferred to hard tasks (h1). H

\rightarrow

E : Skills from hard tasks applied to easy tasks. H

\rightarrow

H : Skills from hard tasks reapplied to the same hard tasks. Base : Baseline without skill transfer. Skill : With cross-task skill transfer. Skill Exec : Skill execution success rate. Efficiency metrics computed over tasks where both modes succeeded. Claude is Claude-4.5-Sonnet and Gemini is Gemini-3-Pro. Avg Tokens are in millions.

Panel of Figure 7 — Figure 7: Cross-model skill reuse heatmaps. Each cell $(i,j)$ shows the result when model $j$ executes skills created by model $i$ . Bold borders highlight key findings. Token saving uses a diverging colormap: blue = increased cost, red = reduced cost.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Pseudo-code for the four primitive tools that enable Skill Mode.

Figure 9: Skill Mode improves efficiency through two mechanisms. First, it reduces argument passing overhead by enabling direct tool chaining (Tool A $\rightarrow$ Tool B $\rightarrow$ Tool C) rather than shuttling intermediate outputs through the agent (Tool A $\rightarrow$ Agent $\rightarrow$ Tool B $\rightarrow$ Agent $\rightarrow$ Tool C). Second, it amortizes planning cost by allowing agents to reuse previously discovered tool sequences, eliminating the need to reason about recurring multi-step patterns from scratch.

Task Family	Domain	Tools	Source
Cat Facts Collector	Reference	5	https://catfact.ninja
Cocktail Menu Generator	Food	5	https://thecocktaildb.com
Countries Encyclopedia	Reference	5	https://restcountries.com
D&D Campaign Builder	Gaming	6	https://dnd5eapi.co
D&D Monster Compendium	Gaming	6	https://dnd5eapi.co
Dog Breeds Encyclopedia	Reference	5	https://dog.ceo/api

Appendix

Table 5: Complete list of API sources used in SkillCraft . The benchmark comprises 21 task families across 6 domains (Entertainment, Reference, Education, Developer, Science, Food). Tools : number of distinct API-wrapping functions per task. Each task family includes 6 difficulty-scaled variants (Easy: 3 subtasks, Medium: 5 subtasks, Hard: 7 subtasks), totaling 126 tasks. All APIs except Local DNA Analysis are publicly available REST endpoints.

Model	Diff.	Success Rate		Skill Stats		Avg Tokens			Avg Cost ($)			Avg Turns			Avg Tool Calls
Model	Diff.	Base	Skill	Exec	Reuse	Base	Skill	Diff	Base	Skill	Diff	Base	Skill	Diff	Base	Skill	Diff
Kimi-K2-Thinking	Easy	30/63 (48%)	32/63 (51%)	81%	2.6 $\times$	427K	293K	-31%	0.17	0.12	-29%	9.0	13.7	+53%	12.2	10.6	-13%
	Medium	17/42 (40%)	17/42 (40%)	76%	2.9 $\times$	576K	335K	-42%	0.23	0.14	-39%	8.1	15.1	+87%	19.5	12.5	-36%
	Hard	8/21 (38%)	7/21 (33%)	66%	4.8 $\times$	622K	285K	-54%	0.25	0.13	-50%	10.4	16.4	+58%	27.8	18.4	-34%
DeepSeek-V3.2-EXP	Easy	42/63 (66%)	47/63 (74%)	94%	2.5 $\times$	943K	512K	-46%	0.27	0.22	-18%	28.2	25.7	-9%	15.4	13.2	-14%
	Medium	25/42 (59%)	25/42 (59%)	89%	3.2 $\times$	1.34M	556K	-59%	0.22	0.08	-63%	36.2	30.4	-16%	23.4	17.2	-26%

Appendix

Table 6: Results breakdown by difficulty level (Easy: e1–e3, Medium: m1–m2, Hard: h1). Success Rate : task completion rate (score

\geq

90) for Baseline and Skill modes. Skill Stats : Exec = skill execution success rate; Reuse = average times each skill is invoked. Efficiency metrics : per-task averages computed over tasks where both modes succeeded. Diff : percentage change; negative = improvement, positive = degradation.

Model	Mode	Success Rate		Skill Stats		Avg Tokens		Avg Cost ($)		Avg Turns		Avg Tool Calls
Model	Mode	Succ	Rate	Exec	Reuse	Val	Diff	Val	Diff	Val	Diff	Val	Diff
Claude-4.5-Sonnet	Base	47/48	98%	–	–	1.72M	–	1.73	–	15.7	–	14.7	–
	Skill	43/48	90%	99%	3.7 $\times$	0.34M	-80%	0.22	-87%	10.5	-33%	9.5	-36%
	Direct Exec	46/48	96%	68%	3.1 $\times$	0.16M	-90%	0.17	-89%	5.8	-64%	4.8	-68%
GPT-5.2	Base	45/48	94%	–	–	1.18M	–	1.52	–	24.5	–	23.1	–
	Skill	43/48	90%	97%	3.5 $\times$	0.26M	-78%	0.39	-74%	8.9	-64%	7.9	-66%

Appendix

Table 7: Three-mode comparison (Base, Skill, Direct Exec) on 48-task subset. Base : No skill library. Skill : With skill library from previous runs. Direct Exec : Skills are directly executed without agent intervention. Efficiency metrics are computed over tasks where both Base and the respective mode succeeded.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 286 papers

Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?
Aug 5, 2026Jinyi Han, Yuanjian Xu, Ying Liao +6LLM Agent Evaluation LLM Agent Skill Retrieval
1East China Normal University · 2Hong Kong University of Science and Technology · 3Fudan University +1
Demystifying Agent Skills: Why They Work-Until They Don't
Aug 14, 2026Zhiyuan Jiang, Fangrui Huang, Hanwen Xing +6LLM Agent Skill Evaluation LLM Agents
Princeton University · Stanford University · University of Southern California +2
Generative Skill Composition for LLM Agents
Jun 30, 2026Xinyu Zhao, Zhen Tan, Vaishnav Tadiparthi +5AI Coding Agents LLM Agent Skill Learning
University of North Carolina at Chapel Hill · Arizona State University · Honda Research Institute USA

Aug 5, 2026J/K move · Enter open · S save

Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?

Jinyi Han, Yuanjian Xu, Ying Liao, Xinyi Wang+5

1East China Normal University · 2Hong Kong University of Science and Technology · 3Fudan University+1

LLM Agent Evaluation LLM Agent Skill Retrieval LLM Agents LLM Agent Harnesses LLM Agent Skill Evaluation

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly rely on skills, structured documents that specify when to act, which procedure to follow, and which tools are allowed. Existing evaluations mostly judge the quality of a skill or its contribution to task success, leaving unexamined whether an agent can recognize a relevant skill and apply it on its own. We introduce Skill-Use, a benchmark that evaluates skill use under progressive disclosure, where an agent sees only a skill's name and short description and must retrieve the full procedure before following it. Skill-Use separates three facets of skill use. Trigger measures whether the agent invokes the relevant skill, Compliance measures how faithfully it follows the prescribed procedure, and Boundary measures whether it avoids forbidden operations. A Skill-Use (SU) score combines the three and credits execution only after the skill is triggered. Skill-Use pairs 79 real skills with 177 executable tasks across nine domains, each grounded in real files, run in an isolated Docker sandbox, and scored by a trajectory-based rubric. Evaluating eight LLMs under two agent harnesses, we find that reliable skill use remains out of reach, as the strongest configuration reaches an SU of only 0.613. Triggering and procedural compliance fail as independent bottlenecks, and both scores and model rankings shift with the harness, so skill use behaves as a capability conditioned on the harness rather than a fixed property of the model.

Abstract

Tool-Using Agents

Explore similar work

Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?

Demystifying Agent Skills: Why They Work-Until They Don't

Generative Skill Composition for LLM Agents