SkillGym: Training Skill-Use Agents with Automatic Verifiable Environment Generation

Renxi Wang; Mingshan Hee; Fajri Koto; Timothy Baldwin; Haonan Li

cs.AISep 29, 2026

SkillGym: Training Skill-Use Agents with Automatic Verifiable Environment Generation

Authors: Renxi Wang, Mingshan Hee, Fajri Koto, Timothy Baldwin, Haonan Li

Organizations: Mohamed bin Zayed University of Artificial Intelligence

Abstract

Skills equip LLM agents with professional knowledge and guidance to complete long-horizon and complex tasks. Although skills have been widely adopted in recent agent paradigms and harnesses, how to synthesize reliable training data and how to train agents for skill use remain underexplored. In this work, we propose SkillGym, an automatic pipeline to build verifiable environments, collect trajectories, and train skill-use agents. SkillGym first crawls a large volume of skills from the internet, then keeps those whose workflows can run reproducibly offline. A builder-reviewer pipeline is used to construct difficulty-controlled tasks, spanning four task types, each with a reference solution and an executable verifier. With this pipeline, we build 6.8k environments and collect 19k verified successful trajectories for supervised finetuning. Finetuning on these trajectories improves LLMs of different families and sizes, from 2B to 122B parameters across four skill-use benchmarks; Our Qwen3.5-9B SFT model outperforms the 397B untrained model on two of them. Further analysis shows that training teaches agents to invoke skills, raising the rate of reading the relevant skill from 28% to 96%, and that the gains hold across reasoning structures, extending to task types that form a minority of the training data and to skills held out from training

Figures & tables

Figure 1: Overview of SkillGym. Public skill packages are curated and transformed into executable tasks through two stages of builder-reviewer interaction: raw environment building and task authoring. Execution validation and quality review support task refinement, followed by multi-harness trajectory collection and supervised finetuning.

Figure 2: (a) The parts shared by every task profile; the task-type contract differs across the four reasoning structures. (b) Tasks per domain (6,772 tasks from 3,494 skills). (c) Token length of successful (19.1k) and unsuccessful (15.6k) teacher trajectories; dashed lines mark the medians.

Base and SkillGym SFT comparisons
Model	Size	SkillGym	SkillEval	SkillsBench	Skill-Use-Bench
Qwen3.5	397B/17B	$55.8$	$73.8\pm 0.8$	$33.2\pm 3.0$	$23.0$
GLM-5.2 ⋄	753B	$66.5$	$79.6\pm 0.8$	$57.8\pm 4.8$	$76.3$
DeepSeek-V4-Flash ⋄	284B/13B	$61.5$	$78.6\pm 0.2$	$58.7\pm 1.0$	$56.1$
Kimi-K3 ⋄	2.8T/104B	$68.8$	$81.0\pm 0.9$	$53.5\pm 4.8$	$76.3$
MiniCPM5	2B	$33.2$	$63.0\pm 0.6$	$\mathbf{10.8\pm 2.7}$	$24.1$

Table 1: Main evaluation results, higher is better. SkillEval and SkillsBench show mean

\pm

standard deviation over three runs; the other benchmarks use one run. Bold marks the higher score within each pair where both results are available. ⋄ Teacher models whose trajectories form our training data. More reference models are in Appendix E . § Reported by Tan et al. (2026) from their paper.

Figure 3: (a) Qwen3.5-9B success rate before and after SkillGym SFT, overall (400 tasks) and per skill split (200 each), with skills available. (b) Overall success rate without and with skills.

Figure 4: Mean component scores on Skill-Use-Bench for Qwen3.5-9B.

Figure 5: SkillsBench gains by what the skill provides and what the task asks. Lines show 95% task-bootstrap intervals; the dashed line is the overall gain; the hollow marker has fewer than 15 tasks.

	Ours	SkillsBench	SkillEval		Skill-Use-Bench
Training data			Pass	Strict	SU	Compl.
Qwen3.5-9B (no SFT)	41.3	14.8	65.7	17.3	14.8	51.4
SkillGym (full)	59.5	22.4	74.7	33.3	49.6	46.9
Validated only	40.5	10.8	41.2	18.3	45.1	43.1
Review-approved	45.2	12.0	47.9	24.7	47.4	52.4
Procedural only	42.5	17.4	56.0	24.7	52.2	56.2

Table 2: Data ablations on Qwen3.5-9B (64k context, 2 epochs). Each pair is trained on a token-matched subset of SkillGym trajectories (110M tokens for review, 132M for structure).

Task type	w/o	w/	Gain
Procedural	31.0	47.0	+16.0
Constraint sat.	79.8	89.9	+10.1
Abductive	65.7	69.7	+4.0
Partial order	65.7	68.7	+3.0
All	60.5	68.8	+8.3

Table 3: Kimi-K3 on the SkillGym test set without and with skills.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Level	Field	Values (%)	Selection
(i) Screening ( $n{=}51{,}131$ )	Label	real skill 88.9, unclear 5.5, meta/doc 3.3, template 1.0, test/demo 0.7, placeholder 0.5	real skill
	Language	English 92.4, Chinese 3.0, Japanese 1.8, Korean 0.9, mixed 0.9, other 1.0	English
	Repository ‡	archived flag, bulk-publisher flag, license, stars	not archived or bulk
	Size ‡	number of files in the skill folder	$\leq$ 300
(ii) Identity ( $n{=}27{,}569$ )	Domain	21 domains; software engineering 53.0, agents/meta 8.8, productivity 3.9, infrastructure 3.9, writing 3.8, media 3.7, …	–
	Task type †	generate 51.7, validate 48.4, analyze 46.8, integrate 30.7, plan 26.7, orchestrate 25.2, transform 18.4, …	–

Appendix

Table 4: Skill annotation fields, their values (share of annotated skills, %), and the rule used to select skills for task creation. Fields marked † allow several values per skill, so shares can exceed 100%. Fields marked ‡ are rule-based, computed from repository metadata and the file listing; all others are LLM labels. “–” means the field is recorded but not used for selection.

Model	Size	SkillGym	SkillEval	SkillsBench	Skill-Use-Bench
Qwen3.5	397B/17B	$55.8$	$73.8\pm 0.8$	$33.2\pm 3.0$	$23.0$
Qwen3.8	27B	$69.0$	$81.6\pm 0.8$	$65.0\pm 1.5$	$68.0$
GLM-5.2 ⋄	753B	$66.5$	$79.6\pm 0.8$	$57.8\pm 4.8$	$76.3$
GLM-5.3	753B	$68.8$	$80.8\pm 0.7$	$56.9\pm 2.0$	$78.2$
DeepSeek-V4-Pro	1.6T/49B	$61.5$	$78.8\pm 0.1$	$57.0\pm 2.6$	$57.7$
DeepSeek-V4-Flash ⋄	284B/13B	$61.5$	$78.6\pm 0.2$	$58.7\pm 1.0$	$56.1$

Appendix

Table 5: All reference models, with skills available, evaluated under the same protocol as Table 1 . ∗ SkillGym score over the tasks graded so far. ⋄ Teacher models.

Stage	skills.sh	Registry	Total
Crawled (manifest)	9,724	174,706	184,430
Fetched and deduplicated	8,593	42,538	51,131
Screened as a real skill	8,104	37,333	45,437
Annotated (not archived or bulk-published)	6,776	20,793	27,569
Selected (rules in Table 4 )	2,838	9,268	12,106
After cross-source deduplication	2,838	9,059	11,897

Appendix

Table 6: Skill curation funnel by source. Registry skills are fetched from their GitHub repositories; skills that could not be fetched and exact duplicates are removed before screening. The last row removes skills that appear in both sources.

Domain	Selected skills	Skills with tasks	Tasks
Marketing	406	374	798
Software engineering	7,726	707	791
Productivity	321	285	618
Cybersecurity	341	289	602
ML / AI tooling	286	263	598
Infrastructure	317	283	594

Appendix

Table 7: Domain distribution of the selected skill pool and of the constructed tasks. Task construction samples skills with per-domain quotas, which cap software engineering and exclude agents/meta and writing skills, so the task distribution is more balanced than the pool.

Profile	Passed	Unresolved	Total
Procedural execution	1,293	869	2,162
Abductive diagnosis	2,400	430	2,830
Constraint satisfaction	1,498	282	1,780
Total	5,191	1,581	6,772
Size	Median	Mean	P10–P90
Instruction length (words)	336	365.4	215–556

Appendix

Table 8: Constructed tasks. Top: counts by reasoning profile and review outcome ( passed : validity gate and reviewer approval; unresolved : valid, but the reviewer’s strictness concerns were not fully resolved within the round limit). Partial-order tasks were built in separate construction runs and are not included in these counts. Bottom: task size over the 6,692 locally available packages (median, mean, and 10th–90th percentile).

		Successful	Unsuccessful
Trajectories		19,070	15,649
Harness	MiniSwe-Agent	6,433	4,360
	AgentFly	5,936	6,909
	Terminus-2	6,701	4,380
Teacher	Kimi-K3	10,603	7,649
	DeepSeek-V4-Flash	6,521	5,507

Appendix

Table 9: Collected trajectories. Successful trajectories pass the task verifier; unsuccessful ones are graded and fail. Length statistics give the median (mean) per trajectory; Terminus-2 issues commands inside its JSON reply rather than as tool calls.

Figure 6: Share of tasks annotated with each reasoning structure (left; a task can have several) and each skill role (right). Columns: a stratified SkillGym training sample ( $n{=}300$ ), the SkillGym test set ( $n{=}400$ ), SkillEval ( $n{=}100$ ), and SkillsBench ( $n{=}87$ ).

Figure 7: Change in success from Qwen3.5-9B base to SkillGym SFT for tasks with each structure, with skills available. Dots show mean paired differences and lines 95% task-bootstrap intervals; coral marks intervals that exclude zero, and hollow markers denote fewer than 15 tasks. Dashed lines show each set’s overall gain. Row labels give each structure’s share of the training sample. SkillEval uses strict success averaged over three runs; SkillsBench uses reward averaged over three runs, excluding trials lost to an environment permission failure.

Term	Coefficient	95% CI
Sequential procedure	$+4.6$	$[-4.2,+12.5]$
Dependency ordering	$+0.4$	$[-6.5,+7.2]$
Diagnosis and repair	$-2.3$	$[-9.8,+5.6]$
Interacting constraints	$+1.0$	$[-8.7,+11.2]$
Rule application	$+6.3$	$[-2.0,+15.0]$
SkillEval	$-1.2$	$[-11.7,+9.2]$

Appendix

Table 10: Regression of the paired SFT

-

base difference (points) on task properties (

n{=}585

tasks). Intervals are 95% bootstrap intervals over tasks (1,000 resamples). Set effects are relative to SkillGym; difficulty ranges from 0 to 1.

Stage	Priced as	Cached in	New in	Output	Cost (USD)
Task construction, measured	V4-Pro	58.4	0.54	1.00	3.6k
Task construction, all attempts	V4-Pro	extrapolated			up to 6.3k
Quality review, assumed	V4-Pro	0.28	0.39	0.84	1.9k
Trajectory collection	V4-Flash	6.44	0.48	0.08	0.14k
Total					5.7k–8.4k

Appendix

Table 11: Estimated API cost of building SkillGym at DeepSeek off-peak prices per million tokens, with

0.022,

0.66, and

1.98 for cached input, new input, and output on DeepSeek-V4-Pro, and

0.003,

0.15, and

0.60 on DeepSeek-V4-Flash. Token counts are in billions.

Explore similar work

SkillGym: Internalizing Human Skills into LLMs for Real-World Problem Solving
Sep 23, 2026Zhilong Ge, Yuting Shao, Yutao Yang +6Reusable Agent Skills Skills
School of Computer Science and Technology, East China Normal University, Shanghai · Shanghai AI Laboratory
SkillGen: Verified Inference-Time Agent Skill Synthesis
May 9, 2026Yuchen Ma, Yue Huang, Han Bao +5Reusable Agent Skills Skills
Munich Center for Machine Learning, LMU Munich · University of Notre Dame · Microsoft Research
SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation
Aug 3, 2026Zelin Tan, Yiqun Zhang, Hao Li +11Synthetic Task Language-Model Agents
† Work done during an internship at Shanghai Artificial Intelligence Laboratory. · Shanghai Artificial Intelligence Laboratory

Sep 23, 2026, cs.CLJ/K move · Enter open · S save

SkillGym: Internalizing Human Skills into LLMs for Real-World Problem Solving

Zhilong Ge, Yuting Shao, Yutao Yang, Yuxuan Cai+5

School of Computer Science and Technology, East China Normal University, Shanghai · Shanghai AI Laboratory

Reusable Agent Skills Skills Large Language Model Agents Skillgym Real-World

Open paper AskPDF arXiv

Human-written agent skills encode rich workflows for real-world problem solving, but are typically used as external inference-time instructions rather than internalized as reusable model capabilities. We introduce \texttt{SkillGym}, a framework that transforms these skills into executable, verifiable training environments for large language model agents. Its skill-to-task pipeline instantiates concrete tasks, verifies outcomes with code-based checkers, and assesses empirical skill dependence through contrastive executions. We construct and release 2,756 environments across 12 categories and collect 8,364 successful trajectories from multiple models and harnesses, averaging 49 tool calls and over 60k logged text tokens. These resources support supervised fine-tuning on verified workflows and reinforcement learning with outcome-based rewards. Under Claude Code, supervised fine-tuning improves Qwen3.5-35B-A3B by 199 Elo on GDPval-AA v2, 19.10 percentage points on Terminal-Bench 2.1, and 28.13 and 12.38 points on SkillsBench v1.1 with and without skills, respectively. Our 35B \texttt{SkillGym-Agent} reaches 51.47% on skill-assisted SkillsBench, exceeding reported scores for Claude Sonnet 4.6, GPT-5.4 Mini, and DeepSeek V4 Pro. Without skills, it also surpasses skill-assisted bases under Codex and Claude Code, suggesting reusable procedural competence.

Abstract

Explore similar work

SkillGym: Internalizing Human Skills into LLMs for Real-World Problem Solving

SkillGen: Verified Inference-Time Agent Skill Synthesis

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation