Organizations: State Key Laboratory of Complex & Critical Software Environment, Beihang University, Beijing, China · Joint SDU–NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University, Jinan, China · School of Automation, Northwestern Polytechnical University, Xi’an, China
Agent skills are reusable procedural artifacts that extend language agents with specialized workflows, tool conventions, and domain behaviors at inference time. However, creating reliable skills still depends largely on human authorship, model priors, or execution traces. These sources are often unavailable for unfamiliar tasks, suggesting the need to create skills from open-world materials. In this paper, we study open-world skill creation: given an underspecified skill brief and a source-access specification, a creator must discover behavior-relevant requirements omitted by the brief and determine how broadly each source-derived procedure is justified. We propose SkillAlchemy, an admission-centered framework for source-grounded skill creation. SkillAlchemy identifies implicit requirements through contrastive evidence, admits candidate procedures based on evidence-supported scope, and compiles the admitted content into a grammar-guided skill package. Extensive experiments across 87 SkillsBench v1.1 tasks demonstrate that SkillAlchemy improves pass rate over no-skill execution by 19.9 percentage points and the strongest automated baseline by 8.6 percentage points, while achieving performance comparable to human-curated skills.
Figures & tables
Figure 1: Pilot Study : Open-world source access improves skill creation but does not close the gap to human-curated skills.
Figure 2: Example illustrating why open-world skill creation is non-trivial : ① task briefs underspecify implicit requirements, and ② directly adopting open-world findings without scope justification may lead to over-specific practices being mistaken for reusable instructions.
Figure 3: Overview of SkillAlchemy : Φ1 converts an underspecified brief into focused questions and structured findings, Φ2 induces candidate procedures and admits them via the evidence-supported scope, and Φ3 compiles the admitted content into an agent skill package.
Agent
Model
Skill Setting
Overall ( n=87 )
Δ (pp)
Soft. ( n=16 )
Office ( n=14 )
Sci. ( n=14 )
Media ( n=5 )
Cyber ( n=7 )
Fin. ( n=9 )
Ind. ( n=14 )
Math ( n=8 )
Claude Code
DeepSeek-V4-Pro
No Skill
23.4
–
22.5
32.9
18.6
20.0
14.3
26.7
25.7
20.0
Claude Code
DeepSeek-V4-Pro
Anthropic Skill-Creator
31.7
+8.3
35.0
35.7
32.9
44.0
22.9
13.3
34.3
32.5
Claude Code
DeepSeek-V4-Pro
OpenAI Skill-Creator
33.3
+9.9
33.8
41.4
44.3
32.0
22.9
17.8
28.6
35.0
Claude Code
DeepSeek-V4-Pro
Human-Curated Skill
51.3
+27.9
43.8
47.1
72.9
80.0
42.9
40.0
44.3
50.0
Claude Code
DeepSeek-V4-Pro
OpenSkill
42.3
+18.9
35.0
38.6
61.4
68.0
37.1
28.9
37.1
42.5
Claude Code
DeepSeek-V4-Pro
MUSE-Autoskill
43.2
+19.8
37.5
38.6
61.4
64.0
40.0
31.1
37.1
45.0
Table 1: Main results on SkillsBench. n is the task number and Δ=pskill−pno-skill reports the difference from no-skill in percentage points.
Figure 4: Task-level comparison with Human-Curated Skills. Each tile represents one task and is grouped by the sign of its avg@5 difference after averaging equally over the four agent–model configurations. Color intensity encodes the absolute difference, and the rightmost column reports the category mean in percentage points.
Skill Setting
Passes / Trials
avg@5
95% CI
No Skill
624 / 1740
35.9
[33.6, 38.1]
Anthropic Skill-Creator
706 / 1740
40.6
[38.3, 42.9]
OpenAI Skill-Creator
734 / 1740
42.2
[39.9, 44.5]
Human-Curated Skill
946 / 1740
54.4
[52.0, 56.7]
OpenSkill
800 / 1740
46.0
[43.6, 48.3]
MUSE-Autoskill
821 / 1740
47.2
[44.8, 49.5]
Table 2: Combined results on four agent-model configurations. We combine all runs from the configurations, giving 1,740 binary outcomes per condition (over 87 tasks × 5 runs × 4 configurations). The 95% CI denotes the 95% Wilson Confidence Intervals.
Creation
Execution
Skill Setting
Tok./Call (K)
Time/Task (min)
Tok./Run (K)
Time/Run (min)
No Skill
N/A
N/A
661
6.44
Anthropic Skill-Creator
59.4
6.47
612
5.79
OpenAI Skill-Creator
58.7
6.88
583
5.61
Human-Curated Skill
N/A
N/A
716
6.69
OpenSkill
65.9
36.37
694
6.78
Table 3: Skill creation and downstream execution resource use. Creation token is averaged per LLM call, the execution-token usage is per evaluation run and time is end-to-end per task or run.
Figure 5: Component ablation across three SkillsBench domains. Bars report avg@5 for the full method and variants without implicit requirement discovery ( Req. ), structured findings ( Find. ), procedure admission ( Adm. ), or the skill grammar ( Gram. ). Labels report within-domain decrease in pp relative to the full method.
Skill Setting
Payload promotion ↓
Downstream passes ↑
Irr.
Conf.
Adv.
Clean
Irr.
Conf.
Adv.
Anthropic Skill-Creator
1/4
3/4
1/4
14/20
11/20
6/20
13/20
OpenAI Skill-Creator
1/4
2/4
1/4
13/20
12/20
8/20
12/20
OpenSkill
1/4
3/4
1/4
16/20
15/20
11/20
15/20
MUSE-Autoskill
2/4
1/4
1/4
15/20
14/20
10/20
15/20
SkillAlchemy
0/4
0/4
0/4
18/20
18/20
17/20
18/20
Table 4: Robustness test under three-type source perturbations . Promotion reports created skills where injected payload be a runtime-facing instruction. Passes report successful executions of all 20 runs.
Setting
Representative Instruction
Orig.
Filt.
Sem.
Δsum
No Skill
—
5/5
4/5
3/5
−3
Human-Curated
Redact sensitive data (e.g., student IDs) and insert an allowed mask.
5/5
4/5
1/5
−5
Ant. Skill-Creator
Redact rather than cover text; replace content and validate.
4/5
3/5
3/5
−2
OpenAI Skill-Creator
Select redaction, execute the edit plan, and validate.
4/5
3/5
2/5
−3
OpenSkill
Classify the edit, redact the target, and verify the result.
4/5
3/5
3/5
−2
MUSE-Autoskill
Remove text-layer content and verify deletion.
5/5
2/5
3/5
−5
Table 5: Artifact scope audit and frozen-skill reuse. Representative instructions summarize the operational scope of matched PDF-redaction artifacts, while scores report unchanged reuse on the threejs-to-obj . Each skill is created for the original task, frozen, and evaluated on two variants. Δsum=(Filt.−Orig.)+(Sem.−Orig.) reports cumulative change across the variants.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Evaluation-only asset
Role in evaluation and isolation
Held-Out Task inputs
Task inputs reserved for downstream evaluation. Creators receive only the visible task description and context. Kept separate by the benchmark runtime and our protocol.
Oracle/Reference Assets
Reference solutions, gold outputs, or expected artifacts used to establish task success. Isolated by the benchmark runtime and our protocol.
Verifier Implementations
Scoring Programs whose task-specific criteria map a submission to a pass/fail signal. Isolated by the benchmark runtime and our protocol.
Human-Curated Skill
Human-Authored Packages used only in the curated evaluation condition, not as creation sources.
Appendix
Table A.1: Evaluation-only assets excluded during skill creation. The benchmark runtime isolates evaluation-time state from the downstream agent, while our protocol applies the stated exclusions to all skill-creation methods.
Figure A.1: Skill package anatomy across four agent–model configurations. Columns (a)–(d) correspond to Claude Code with DeepSeek-V4-Pro-Preview, Claude Code with Opus 4.8, Codex with DeepSeek-V4-Pro-Preview, and Codex with GPT-5.5. The upper row reports main-file length distributions, and the lower row reports the proportions of SKILL.md , scripts, references, and other files.
Component
Choices and enforced property
Activation metadata
Choices: keyword, context, explicit, hybrid, or always-on activation. Purpose: state what the skill does and when it should be invoked, avoiding missed or unrelated activation.
Workflow form
Choices: ordered steps, a decision tree, template filling, or operation cards. Purpose: match the structure to procedure dependencies and expose branch conditions.
Executable procedure
Form: condition, action, observable output, and a supported failure route. Purpose: replace vague advice with instructions an agent can execute and inspect.
Applicability boundary
Choices: source, version, capability, scope, legal, or confidence limits. Purpose: preserve scope and prevent unsupported generalization across tasks and contexts.
Verification form
Choices: a postcondition, checklist, or artifact check. Purpose: express an admitted verification component as a directly observable output.
Output contract
Choices: an analysis report, executable code, conversation, checklist, or task-dependent mixture. Purpose: make the output usable by its downstream consumer.
Appendix
Table A.2: The operational skill grammar used for package organization. The grammar constrains organization and execution form; it does not supply task knowledge or override the evidence-based admission process.
Figure A.2: Case-level comparison across all seven conditions. Results use Codex with GPT-5.5. Result counts successes over five matched runs, and Criteria gives verifier-criterion coverage. Success requires all criteria.
Method
Search design, evidence, and outcome
OpenAI Skill-Creator
Framing: task-relevant spreadsheet functions. Evidence: Microsoft function documentation and openpyxl formula-writing guidance. Outcome: calculation-engine compatibility and cached-value validation remain uncovered; 0/5 runs pass.
OpenSkill
Framing: technologies for general spreadsheet construction. Evidence: workbook properties, recalculation engines, Excel functions, and IMF interfaces. Outcome: the risks are retrieved but not connected to a delivered-file requirement; 0/5 runs pass.
SkillAlchemy
Framing: omitted execution risks identified through implicit requirement discovery and boundary probes. Evidence: formula compatibility, recalculation, cached values, array alignment, reference stability, economic semantics, and workbook preservation. Outcome: the evidence becomes explicit final-workbook checks; 5/5 runs pass.
Appendix
Table A.3: Search strategy, retrieved evidence, and downstream result on weighted-gdp-calc . Retrieved content is summarized from the skill-creation search records.
Autonomous AI agents can execute complex tasks with limited human review, yet they often lack the grounded operational knowledge to make their outputs not just executable but correct, secure, and maintainable. We introduce SkillCenter, to our knowledge the largest open skill library for agents by total count: 216,938 structured skills across 24 domain bundles. A SkillGate-filtered pipeline contributes 114,565 source-grounded skills from peer-reviewed journals, ArXiv, and over 24,000 technical sources, integrated with 102,373 community skills from GitHub and the ClawHub marketplace. We present the end-to-end framework that builds the pipeline subset: multi-source acquisition, an LLM-based quality gate (SkillGate), template-driven generation, iterative source-grounding, and quality-controlled publishing. Source grounding is a traceability guarantee: each retained claim maps to an exact quotation in its source. All skills ship as offline-searchable SQLite FTS5 bundles.
Tianming Sha, Yue Zhao, Lichao Sun +1
1Stony Brook University · University of Southern California · 3Lehigh University +1
Skills are a promising way to improve LLM agent capabilities without retraining, while keeping the added procedure reusable and controllable. However, high-quality skills are still largely written by hand. We introduce SkillGen, a multi-agent framework that synthesizes a single auditable skill from trajectories generated by a base agent. The output is a human-readable artifact that can be inspected before use. Rather than merely summarizing trajectories, SkillGen leverages contrastive induction over both successful and failed trajectories to identify reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures. SkillGen then generates candidate skills and iteratively refines the skill. A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance. Specifically, we compare outcomes on the same instances with and without the skill, so that we account for both repairs (cases where the skill fixes a baseline failure) and regressions (cases where the skill breaks a baseline success). Across a broad range of agents and datasets, SkillGen consistently improves held-out performance, outperforms existing skill-generation baselines, and produces skills that transfer across models.
Yuchen Ma, Yue Huang, Han Bao +5
Munich Center for Machine Learning, LMU Munich · University of Notre Dame · Microsoft Research
As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether they can generate correct, reusable, and executable skills from repositories and documents. Existing benchmarks primarily evaluate the efficacy of given skills or the ability of agents to solve downstream tasks from raw context, but they do not isolate skill generation itself as the object of study. We introduce SkillGenBench, a benchmark for evaluating skill generation pipelines under a unified and controlled protocol. In SkillGenBench, a generator receives raw corpora and produces standardized skill artifacts, which are then executed under fixed harnesses and assessed with unified evaluation procedures. The benchmark covers two generation regimes: task-conditioned generation, where a task-specific skill is synthesized after the task is revealed, and task-agnostic generation, where a reusable skill library must be distilled before downstream tasks are known. It also spans two complementary procedural sources: repository-grounded instances, where procedures are distributed across code, configuration, and scripts, and document-grounded instances, where procedures and constraints must be distilled from long-form text. We provide standardized task specifications, pinned environments, and evaluation protocols centered on deterministic execution-based checks, supplemented by auxiliary signals for diagnosis. Experiments across a range of skill-generation methods and backbones show substantial performance variation, highlight the difficulty of reusable skill distillation, and reveal distinct failure modes in skill generation from software repositories versus long-form documents. SkillGenBench establishes a reproducible testbed for studying skill generation as an independent research problem in agent systems.