Prompt2Skill: Unsupervised Skill Optimization From Natural Language Instructions

Bo Ni; Li Li; Ryan A. Rossi; Franck Dernoncourt; Tyler Derr

cs.CLSep 29, 2026

Prompt2Skill: Unsupervised Skill Optimization From Natural Language Instructions

Authors: Bo Ni, Li Li, Ryan A. Rossi, Franck Dernoncourt, Tyler Derr

Organizations: Vanderbilt University · University of Southern California · Adobe Systems

Abstract

Skills are external artifacts that Large Language Models (LLMs) consume at inference time to improve their performance on specialized domains by incorporating relevant procedural and domain knowledge. Expert-authored skills are expensive to produce, and the resulting artifacts are not optimized for the specific model that consumes them, whose failure modes can vary with version, scale and training. In addition, emerging tasks may fall outside the scope of existing skill libraries, creating a need to develop new skills before curated training data become available. Recent works have explored automated skill optimization through reflection, but they require a curated, in-distribution training set, which users might not always have. To address these limitations, we present Prompt2Skill, a framework that builds skills from natural-language task description alone. From the prompt, the system derives a task specification, discovers or synthesizes datasets, and refines the skill in a closed loop of reflective editing. Across four domains spanning question answering, reading comprehension, spreadsheet manipulation, and mathematical reasoning, Prompt2Skill consistently outperforms the direct prompting baseline, achieving an average improvement of 10.8 across open-source and frontier models.

Figures & tables

Figure 1: Prompt2Skill vs. Direct Prompting.

Figure 2: Overview of Prompt2Skill . The Data Agent turns the task description into a task specification, a seed skill, and a pool of validated items; the Skill Agent refines the seed in a closed loop of rollouts, reflective edits, and statistical acceptance on fresh batches. The final skill is chosen on a frozen validation set that is consulted exactly once.

		Method	SearchQA	SQuAD	AIME	Spreadsheet	Avg. $\Delta$
Open-source	Qwen3 [-1pt] 8B	Direct	0.431 $\pm$ 0.002	0.738 $\pm$ 0.001	0.208 $\pm$ 0.014	0.044 $\pm$ 0.005	0.0%
		Off-the-shelf	0.496 $\pm$ 0.003	0.657 $\pm$ 0.004	0.128 $\pm$ 0.013	0.085 $\pm$ 0.007	+14.7%
		LLM-generated	0.557 $\pm$ 0.003	0.611 $\pm$ 0.001	0.158 $\pm$ 0.017	0.082 $\pm$ 0.011	+18.6%
		P2S	0.640 $\pm$ 0.019	0.768 $\pm$ 0.097	0.175 $\pm$ 0.000	0.079 $\pm$ 0.011	+29.1%
	Qwen3 [-1pt] 32B	Direct	0.613 $\pm$ 0.002	0.694 $\pm$ 0.002	0.211 $\pm$ 0.017	0.152 $\pm$ 0.023	0.0%
		Off-the-shelf	0.537 $\pm$ 0.005	0.630 $\pm$ 0.003	0.136 $\pm$ 0.013	0.180 $\pm$ 0.012	-9.7%

Table 1: Main results comparing direct prompting, existing general-purpose skills ( Off-the-shelf ), skills generated directly by an LLM ( LLM-generated ), and Prompt2Skill (P2S). Llama-3.2-1B was not able to complete any SpreadsheetBench task.

Method	SearchQA EM	SQuAD EM	SQuAD F1
SkillOpt	45.79	81.30	90.50
P2S	48.71	82.50	91.11

Table 2: Matched-budget results.

Panel of Figure 3 — Figure 3: Ablations on SearchQA and SQuAD. Scores are percentages; higher is better. Each ablation condition uses one run. Zero-example Prompt2Skill scores in (b) are three-run means from the main table; the matched zero-example runs failed during data acquisition.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: AutoML Performance. P2S stands for Prompt2Skill , while P2M stands for Prompt2Model ( Viswanathan et al., 2023 ) .

Task	Recorded sources	Workbook matches
SearchQA	openaccess-ai-collective/jeopardy	—
SQuAD	allenai/sciq ; hotpotqa/hotpot_qa	—
AIME	nvidia/OpenMathInstruct-2 ; SuperSecureHuman/competition_math_hf_dataset	—
SpreadsheetBench	Generated workbook tasks	0

Appendix

Table 3: Recorded proxy sources and retrospective overlap findings. Dataset identifiers refer to Hugging Face repositories. Text flags are exact, containment, or near-text matches under the checks described below. Workbook matches use two SHA-256 checks; a dash denotes not applicable. Zero detected matches apply only to the audited artifacts.

Explore similar work

SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement
Jun 9, 2026Srishti Gautam, Arjun Radhakrishna, Sumit GulwaniSkills Instruction
Microsoft
Self-Supervised Skill Optimization
Jul 30, 2026Siran Peng, Cuiyu Yang, Tianyu Fu +9Unsupervised Skill Discovery
1MAIS, CASIA · 3Mininglamp Technology · 2SAI, UCAS +2
SkillGym: Internalizing Human Skills into LLMs for Real-World Problem Solving
Sep 23, 2026Zhilong Ge, Yuting Shao, Yutao Yang +6Reusable Agent Skills Skills
School of Computer Science and Technology, East China Normal University, Shanghai · Shanghai AI Laboratory

Jun 9, 2026, cs.MAJ/K move · Enter open · S save

SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement

Srishti Gautam, Arjun Radhakrishna, Sumit Gulwani

Microsoft

Skills Instruction

Open paper AskPDF arXiv

Skill documents, structured natural-language instructions that guide Large Language Model (LLM) agents, are critical to modern agent frameworks, yet LLMs struggle to write skills that actually work. On SkillsBench, human-authored skills improve pass rates by 16.2 percentage points, while LLM-authored skills provide no measurable gain. We introduce SkillAxe, a fully unsupervised framework that enables LLMs to iteratively diagnose and refine their own skills. SkillAxe decomposes skill quality into four interpretable dimensions (quality impact, trigger precision, instruction compliance with fault attribution, and solution-path coverage), producing structured improvement briefs that require no ground-truth labels, test suites, or environment rewards. On SkillsBench, SkillAxe improves pass rates by 28% relative over unimproved LLM skills and closes 47--67% of the gap to human-authored skills. We validate the approach as a continuous improvement engine in the wild on SpreadsheetBench, where a SkillAxe-built skill library learns from past agent trajectories and raises pass rate from 16.0% to 52.0% using only 22 skills.

Abstract

Explore similar work

SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement

Self-Supervised Skill Optimization

SkillGym: Internalizing Human Skills into LLMs for Real-World Problem Solving