SkillAlchemy: Open-World Agent Skill Creation

Hengjun Wang; Shuyue Wei; Boyi Liu; Jun Yang; Yongxin Tong

cs.AIAug 24, 2026

SkillAlchemy: Open-World Agent Skill Creation

Authors: Hengjun Wang, Shuyue Wei, Boyi Liu, Jun Yang, Yongxin Tong

Organizations: State Key Laboratory of Complex & Critical Software Environment, Beihang University, Beijing, China · Joint SDU–NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University, Jinan, China · School of Automation, Northwestern Polytechnical University, Xi’an, China

Abstract

Agent skills are reusable procedural artifacts that extend language agents with specialized workflows, tool conventions, and domain behaviors at inference time. However, creating reliable skills still depends largely on human authorship, model priors, or execution traces. These sources are often unavailable for unfamiliar tasks, suggesting the need to create skills from open-world materials. In this paper, we study open-world skill creation: given an underspecified skill brief and a source-access specification, a creator must discover behavior-relevant requirements omitted by the brief and determine how broadly each source-derived procedure is justified. We propose SkillAlchemy, an admission-centered framework for source-grounded skill creation. SkillAlchemy identifies implicit requirements through contrastive evidence, admits candidate procedures based on evidence-supported scope, and compiles the admitted content into a grammar-guided skill package. Extensive experiments across 87 SkillsBench v1.1 tasks demonstrate that SkillAlchemy improves pass rate over no-skill execution by 19.9 percentage points and the strongest automated baseline by 8.6 percentage points, while achieving performance comparable to human-curated skills.

Figures & tables

Figure 1: Pilot Study : Open-world source access improves skill creation but does not close the gap to human-curated skills.

Figure 2: Example illustrating why open-world skill creation is non-trivial : ① task briefs underspecify implicit requirements, and ② directly adopting open-world findings without scope justification may lead to over-specific practices being mistaken for reusable instructions.

Figure 3: Overview of SkillAlchemy : $\Phi_{1}$ converts an underspecified brief into focused questions and structured findings, $\Phi_{2}$ induces candidate procedures and admits them via the evidence-supported scope, and $\Phi_{3}$ compiles the admitted content into an agent skill package.

Agent	Model	Skill Setting	Overall ( $n=87$ )	$\Delta$ (pp)	Soft. ( $n=16$ )	Office ( $n=14$ )	Sci. ( $n=14$ )	Media ( $n=5$ )	Cyber ( $n=7$ )	Fin. ( $n=9$ )	Ind. ( $n=14$ )	Math ( $n=8$ )
Claude Code	DeepSeek-V4-Pro	No Skill	23.4	–	22.5	32.9	18.6	20.0	14.3	26.7	25.7	20.0
Claude Code	DeepSeek-V4-Pro	Anthropic Skill-Creator	31.7	+8.3	35.0	35.7	32.9	44.0	22.9	13.3	34.3	32.5
Claude Code	DeepSeek-V4-Pro	OpenAI Skill-Creator	33.3	+9.9	33.8	41.4	44.3	32.0	22.9	17.8	28.6	35.0
Claude Code	DeepSeek-V4-Pro	Human-Curated Skill	51.3	+27.9	43.8	47.1	72.9	80.0	42.9	40.0	44.3	50.0
Claude Code	DeepSeek-V4-Pro	OpenSkill	42.3	+18.9	35.0	38.6	61.4	68.0	37.1	28.9	37.1	42.5
Claude Code	DeepSeek-V4-Pro	MUSE-Autoskill	43.2	+19.8	37.5	38.6	61.4	64.0	40.0	31.1	37.1	45.0

Table 1: Main results on SkillsBench.

n

is the task number and

\Delta=p_{\mathrm{skill}}-p_{\mathrm{no\text{-}skill}}

reports the difference from no-skill in percentage points.

Figure 4: Task-level comparison with Human-Curated Skills. Each tile represents one task and is grouped by the sign of its avg@5 difference after averaging equally over the four agent–model configurations. Color intensity encodes the absolute difference, and the rightmost column reports the category mean in percentage points.

Skill Setting	Passes / Trials	avg@5	95% CI
No Skill	624 / 1740	35.9	[33.6, 38.1]
Anthropic Skill-Creator	706 / 1740	40.6	[38.3, 42.9]
OpenAI Skill-Creator	734 / 1740	42.2	[39.9, 44.5]
Human-Curated Skill	946 / 1740	54.4	[52.0, 56.7]
OpenSkill	800 / 1740	46.0	[43.6, 48.3]
MUSE-Autoskill	821 / 1740	47.2	[44.8, 49.5]

Table 2: Combined results on four agent-model configurations. We combine all runs from the configurations, giving 1,740 binary outcomes per condition (over 87 tasks

\times

5 runs

\times

4 configurations). The 95% CI denotes the 95% Wilson Confidence Intervals.

Skill Setting	Tok./Call (K)	Time/Task (min)	Tok./Run (K)	Time/Run (min)
	Creation		Execution
No Skill	N/A	N/A	661	6.44
Anthropic Skill-Creator	59.4	6.47	612	5.79
OpenAI Skill-Creator	58.7	6.88	583	5.61
Human-Curated Skill	N/A	N/A	716	6.69
OpenSkill	65.9	36.37	694	6.78

Table 3: Skill creation and downstream execution resource use. Creation token is averaged per LLM call, the execution-token usage is per evaluation run and time is end-to-end per task or run.

Figure 5: Component ablation across three SkillsBench domains. Bars report avg@5 for the full method and variants without implicit requirement discovery ( Req. ), structured findings ( Find. ), procedure admission ( Adm. ), or the skill grammar ( Gram. ). Labels report within-domain decrease in pp relative to the full method.

Skill Setting	Payload promotion $\downarrow$			Downstream passes $\uparrow$
Skill Setting	Irr.	Conf.	Adv.	Clean	Irr.	Conf.	Adv.
Anthropic Skill-Creator	1/4	3/4	1/4	14/20	11/20	6/20	13/20
OpenAI Skill-Creator	1/4	2/4	1/4	13/20	12/20	8/20	12/20
OpenSkill	1/4	3/4	1/4	16/20	15/20	11/20	15/20
MUSE-Autoskill	2/4	1/4	1/4	15/20	14/20	10/20	15/20
SkillAlchemy	0/4	0/4	0/4	18/20	18/20	17/20	18/20

Table 4: Robustness test under three-type source perturbations . Promotion reports created skills where injected payload be a runtime-facing instruction. Passes report successful executions of all

20

runs.

Setting	Representative Instruction	Orig.	Filt.	Sem.	$\boldsymbol{\Delta}_{\mathrm{sum}}$
No Skill	—	5/5	4/5	3/5	$-3$
Human-Curated	Redact sensitive data (e.g., student IDs) and insert an allowed mask.	5/5	4/5	1/5	$-5$
Ant. Skill-Creator	Redact rather than cover text; replace content and validate.	4/5	3/5	3/5	$-2$
OpenAI Skill-Creator	Select redaction, execute the edit plan, and validate.	4/5	3/5	2/5	$-3$
OpenSkill	Classify the edit, redact the target, and verify the result.	4/5	3/5	3/5	$-2$
MUSE-Autoskill	Remove text-layer content and verify deletion.	5/5	2/5	3/5	$-5$

Table 5: Artifact scope audit and frozen-skill reuse. Representative instructions summarize the operational scope of matched PDF-redaction artifacts, while scores report unchanged reuse on the threejs-to-obj . Each skill is created for the original task, frozen, and evaluated on two variants.

\Delta_{\mathrm{sum}}=(\mathrm{Filt.}-\mathrm{Orig.})+(\mathrm{Sem.}-\mathrm{Orig.})

reports cumulative change across the variants.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Evaluation-only asset	Role in evaluation and isolation
Held-Out Task inputs	Task inputs reserved for downstream evaluation. Creators receive only the visible task description and context. Kept separate by the benchmark runtime and our protocol.
Oracle/Reference Assets	Reference solutions, gold outputs, or expected artifacts used to establish task success. Isolated by the benchmark runtime and our protocol.
Verifier Implementations	Scoring Programs whose task-specific criteria map a submission to a pass/fail signal. Isolated by the benchmark runtime and our protocol.
Human-Curated Skill	Human-Authored Packages used only in the curated evaluation condition, not as creation sources.

Appendix

Table A.1: Evaluation-only assets excluded during skill creation. The benchmark runtime isolates evaluation-time state from the downstream agent, while our protocol applies the stated exclusions to all skill-creation methods.

Figure A.1: Skill package anatomy across four agent–model configurations. Columns (a)–(d) correspond to Claude Code with DeepSeek-V4-Pro-Preview, Claude Code with Opus 4.8, Codex with DeepSeek-V4-Pro-Preview, and Codex with GPT-5.5. The upper row reports main-file length distributions, and the lower row reports the proportions of SKILL.md , scripts, references, and other files.

Component	Choices and enforced property
Activation metadata	Choices: keyword, context, explicit, hybrid, or always-on activation. Purpose: state what the skill does and when it should be invoked, avoiding missed or unrelated activation.
Workflow form	Choices: ordered steps, a decision tree, template filling, or operation cards. Purpose: match the structure to procedure dependencies and expose branch conditions.
Executable procedure	Form: condition, action, observable output, and a supported failure route. Purpose: replace vague advice with instructions an agent can execute and inspect.
Applicability boundary	Choices: source, version, capability, scope, legal, or confidence limits. Purpose: preserve scope and prevent unsupported generalization across tasks and contexts.
Verification form	Choices: a postcondition, checklist, or artifact check. Purpose: express an admitted verification component as a directly observable output.
Output contract	Choices: an analysis report, executable code, conversation, checklist, or task-dependent mixture. Purpose: make the output usable by its downstream consumer.

Appendix

Table A.2: The operational skill grammar used for package organization. The grammar constrains organization and execution form; it does not supply task knowledge or override the evidence-based admission process.

Panel of Figure 14 — Figure A.2: Case-level comparison across all seven conditions. Results use Codex with GPT-5.5. Result counts successes over five matched runs, and Criteria gives verifier-criterion coverage. Success requires all criteria.

Method	Search design, evidence, and outcome
OpenAI Skill-Creator	Framing: task-relevant spreadsheet functions. Evidence: Microsoft function documentation and openpyxl formula-writing guidance. Outcome: calculation-engine compatibility and cached-value validation remain uncovered; 0/5 runs pass.
OpenSkill	Framing: technologies for general spreadsheet construction. Evidence: workbook properties, recalculation engines, Excel functions, and IMF interfaces. Outcome: the risks are retrieved but not connected to a delivered-file requirement; 0/5 runs pass.
SkillAlchemy	Framing: omitted execution risks identified through implicit requirement discovery and boundary probes. Evidence: formula compatibility, recalculation, cached values, array alignment, reference stability, economic semantics, and workbook preservation. Outcome: the evidence becomes explicit final-workbook checks; 5/5 runs pass.

Appendix

Table A.3: Search strategy, retrieved evidence, and downstream result on weighted-gdp-calc . Retrieved content is summarized from the skill-creation search records.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 711 papers

SkillCenter: A Large-Scale Source-Grounded Skill Library for Autonomous AI Agents
Jul 8, 2026Tianming Sha, Yue Zhao, Lichao Sun +1LLM Grounding LLM Agent Skill Learning
1Stony Brook University · University of Southern California · 3Lehigh University +1
SkillGen: Verified Inference-Time Agent Skill Synthesis
May 9, 2026Yuchen Ma, Yue Huang, Han Bao +5LLM Agent Self-Improvement LLM Agent Skill Learning
Munich Center for Machine Learning, LMU Munich · University of Notre Dame · Microsoft Research
SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents
May 18, 2026Yifan Zhou, Zhentao Zhang, Ziming Cheng +8LLM Agent Evaluation LLM Agent Skill Learning
1SJTU · 2XJTU · 3NUS +6

Jul 8, 2026J/K move · Enter open · S save

SkillCenter: A Large-Scale Source-Grounded Skill Library for Autonomous AI Agents

Tianming Sha, Yue Zhao, Lichao Sun, Yushun Dong

1Stony Brook University · University of Southern California · 3Lehigh University+1

LLM Grounding LLM Agent Skill Learning LLM Agent Skill Retrieval LLM Agents

Open paper AskPDF arXiv

Autonomous AI agents can execute complex tasks with limited human review, yet they often lack the grounded operational knowledge to make their outputs not just executable but correct, secure, and maintainable. We introduce SkillCenter, to our knowledge the largest open skill library for agents by total count: 216,938 structured skills across 24 domain bundles. A SkillGate-filtered pipeline contributes 114,565 source-grounded skills from peer-reviewed journals, ArXiv, and over 24,000 technical sources, integrated with 102,373 community skills from GitHub and the ClawHub marketplace. We present the end-to-end framework that builds the pipeline subset: multi-source acquisition, an LLM-based quality gate (SkillGate), template-driven generation, iterative source-grounding, and quality-controlled publishing. Source grounding is a traceability guarantee: each retained claim maps to an exact quotation in its source. All skills ship as offline-searchable SQLite FTS5 bundles.

Abstract

LLM Agents

Explore similar work

SkillCenter: A Large-Scale Source-Grounded Skill Library for Autonomous AI Agents

SkillGen: Verified Inference-Time Agent Skill Synthesis

SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents