SPT: Skills as Pre-Training Data for Agentic Language Models

Yufei Sun; Yudong Li; Yiming Cheng

cs.CLAug 27, 2026

SPT: Skills as Pre-Training Data for Agentic Language Models

Authors: Yufei Sun, Yudong Li, Yiming Cheng

Organizations: Beijing University of Posts and Telecommunications · Tsinghua University

Abstract

Agentic (tool-using) language models are mainly trained on tool-call traces and agent trajectories during post-training. These data provide direct behavioral supervision, but producing them requires task environments, execution, and verification, making broad tool and task coverage expensive. Publicly available skills offer another source of training data: they encode reusable tool semantics and workflows but are typically used only as inference-time context. We introduce Skill Pre-Training (SPT), a mid-training method that applies causal language modeling to SkillCorpus, a collection of public multi-file skill packages, optionally mixed with general data. To preserve relations among files within each package, we also introduce Reference Insert, a reference-aware assembly strategy that places supporting files near their mentions in the primary instruction. Experiments across multiple model scales and post-training recipes show that SPT consistently improves agentic performance over mid-training on general or trajectory data, while largely preserving general performance. Data mixture experiments show additional benefits from combining skill data with general annealing corpora. These results indicate that skill packages are a valuable data source for pre-training agentic language models.

Figures & tables

Figure 1: Public skill data growth and the SPT mid-training pipeline. (a) Snapshots from the npm registry show that the number of skills available on the platform grew

9.6\times

from February to June 2026. (b) SPT mixes an

\alpha

fraction of SkillCorpus with

(1-\alpha)

general data during mid-training, followed by post-training.

Figure 2: Task and file composition of SkillCorpus : (a) task categories across its 38,040 packages; (b) roles of 218,277 files.

Figure 3: Overview of the SPT data pipeline. (1) A skill package combines a primary instruction file with supporting resources; (2) Reference Insert places each referenced file near its first mention and appends the remaining files; (3) serialized packages form 4,096-token training blocks while retaining file and package boundaries; and (4) alternative file organizations are used to ablate reference-aware serialization.

MeCo-1.6B-DCLM-160B
Training Data		Agentic Benchmarks $\uparrow$					General Benchmarks $\uparrow$
Mid-train	Post-train	API	Meta	APT	Eyes	Avg.	ARC	BQ	HS	PIQA	WG	MMLU	Avg.
No mid-training	xLAM-FC	9.82	12.50	10.22	22.87	13.85	43.94	69.70	67.40	73.10	64.48	36.43	59.18
Dolmino		15.76	16.16	17.47	23.43	18.20	44.37	69.50	66.30	72.90	63.85	36.51	58.91
AgentBank		18.39	17.13	19.47	27.49	20.62	43.09	69.10	66.80	73.90	64.64	36.38	58.99
SkillCorpus		29.13	21.30	23.02	28.14	25.40	43.69	70.90	65.70	73.40	64.09	36.20	59.00

Table 1: Main results for the mid-training–post-training pipeline. API, Meta, APT, Eyes, ARC, BQ, HS, and WG abbreviate API-Bank, MetaTool, APTBench, ToolEyes, ARC-Challenge, BoolQ, HellaSwag, and WinoGrande. The Avg. columns report the unweighted mean within each benchmark group. Best and second-best scores are shown in bold and underlined, respectively.

Figure 4: Agentic and general scores across SkillCorpus mixture ratios. Circles denote mixtures with Dolmino, and squares denote mixtures with SmolLM; dashed curves show the fitted trends across both general-data sources.

Strategy	API-Bank	MetaTool	APT	ToolEyes	Gen.
DeepSeek-Coder	24.40	15.22	16.14	25.01	57.30
Random File	17.76	10.55	14.72	21.34	57.34
Metadata	11.46	15.03	13.64	25.86	57.35
Skill-FIM	15.43	17.06	13.37	26.06	57.33
Reference Insert	26.47	17.39	17.97	26.39	58.20

Table 2: Effect of multi-file skill assembly on agentic and general performance. Gen. denotes the average score of six general benchmarks.

Mid-training	API-Bank	MetaTool	APT	ToolEyes	Gen.
None	5.44	1.16	4.54	8.64	40.10
Dolmino	7.69	4.75	7.66	8.90	40.96
SkillCorpus	15.93	10.88	9.43	11.52	41.49

Table 3: Scores under identical Tulu 3 SFT and GRPO post-training for three mid-training conditions.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Instances	Matched	13-gram Rate	Max Inst.
Agentic benchmarks
API-Bank	1,602	0	0.0000%	0.00%
MetaTool	6,214	13	0.0022%	1.76%
APTBench	3,561	66	0.0179%	4.65%
ToolEyes	1,121	40	0.0269%	4.34%
OLMES benchmarks

Appendix

Table 8: Exact-overlap diagnostic on the cleaned candidate pool before package-level decontamination. The Matched column counts benchmark instances sharing at least one normalized 13-gram with the candidate pool. The 13-gram Rate is the fraction of unique benchmark 13-grams matched by the candidate pool; Max Inst. is the largest matched 13-gram fraction for any single benchmark instance. Benchmark instances remain in the evaluation sets.

Parameter	Mid-training	Tulu 3 SFT	xLAM-FC SFT
GPUs	4 $\times$ A100 80GB	4 $\times$ A100 80GB	4 $\times$ A100 80GB
Backbone / initialization	MeCo-1.6B-DCLM-160B	Mid-training checkpoint; the no-mid-training condition uses the backbone directly	Mid-training checkpoint; the no-mid-training condition uses the backbone directly
Training objective	Causal language modeling on fixed token blocks	Assistant-only supervised chat loss	Assistant-only function-call loss
Training data	Mid-training corpus (experiment-specific)	Tulu 3	xLAM-FC
Total budget	1 epoch	3 epochs	3 epochs
Sequence length	4,096	2,048	2,048

Appendix

Table 9: Training hyperparameters for the 1.6B experiments with mid-training followed by post-training. Both Tulu 3 and xLAM-FC SFT settings are used for the main mid-training corpus comparison; the Tulu 3 settings are additionally used for the mixture-ratio and data-organization experiments.

Parameter	Mid-training	Tulu 3 SFT	xLAM-FC SFT
GPUs	4 $\times$ A100 80GB	4 $\times$ A100 80GB	4 $\times$ A100 80GB
Backbone / initialization	OLMo-3-1025-7B stage1-step999000	Mid-training checkpoint; the no-mid-training condition uses the backbone directly	Mid-training checkpoint; the no-mid-training condition uses the backbone directly
Training objective	Causal language modeling on fixed token blocks	Assistant-only supervised chat loss	Assistant-only function-call loss
Training data	Mid-training corpus (experiment-specific)	Tulu 3	xLAM-FC
Total budget	1 epoch	3 epochs	3 epochs
Sequence length	4,096	2,048	2,048

Appendix

Table 10: Training hyperparameters for the 7B OLMo experiments with mid-training followed by post-training.

Parameter	Mid-training	Tulu 3 SFT	xLAM-FC SFT
GPUs	4 $\times$ A100 80GB	4 $\times$ A100 80GB	4 $\times$ A100 80GB
Backbone / initialization	Instella-3B-Stage1	Mid-training checkpoint; the no-mid-training condition uses the backbone directly	Mid-training checkpoint; the no-mid-training condition uses the backbone directly
Training objective	Causal language modeling on fixed token blocks	Assistant-only supervised chat loss	Assistant-only function-call loss
Training data	Mid-training corpus (experiment-specific)	Tulu 3	xLAM-FC
Total budget	1 epoch	3 epochs	3 epochs
Sequence length	4,096	2,048	2,048

Appendix

Table 11: Training hyperparameters for the Instella-3B experiments with mid-training followed by post-training.

Ratio	API-Bank	MetaTool	APTBench	ToolEyes
SkillCorpus + Dolmino
0/100	11.34	15.85	13.58	21.47
1/99	15.05	12.46	13.91	29.84
5/95	17.41	12.42	13.81	45.29
10/90	18.34	16.25	24.02	40.84
20/80	31.77	22.50	23.88	41.88

Appendix

Table 12: Agentic benchmark results for different mixture ratios of SkillCorpus with Dolmino and SmolLM.

Ratio	ARC	BQ	HS	PIQA	WG	MMLU
SkillCorpus + Dolmino
0/100	44.26	67.90	68.10	75.00	63.19	37.32
1/99	44.53	69.20	67.20	73.80	64.98	36.90
5/95	43.94	68.70	67.20	73.00	63.85	36.32
10/90	44.20	68.70	67.00	73.30	63.38	36.24
20/80	43.94	68.60	66.70	73.10	63.85	36.42

Appendix

Table 13: Results on general benchmarks for different skill-to-general-data mixture ratios.

Method	API-Bank	MetaTool	APTBench	ToolEyes
DeepSeek-Coder	24.40	15.22	16.14	25.01
Random File	17.76	10.55	14.72	21.34
Metadata	11.46	15.03	13.64	25.86
Skill-FIM	15.43	17.06	13.37	26.06
Reference Insert	26.47	17.39	17.97	26.39

Appendix

Table 14: Agentic benchmark results for different serialization strategies.

Method	ARC	BQ	HS	PIQA	WG	MMLU
DeepSeek-Coder	43.34	62.54	67.00	73.90	62.35	34.65
Random File	44.03	61.50	66.20	73.50	63.46	35.36
Metadata	43.00	62.00	66.30	73.50	64.01	35.30
Skill-FIM	43.60	63.40	66.10	73.60	62.98	34.30
Reference Insert	43.94	65.30	66.20	73.80	63.69	36.24

Appendix

Table 15: General benchmark results for different serialization strategies.

Parameter	Mid-training	Tulu 3 SFT
GPUs	4 $\times$ A100 80GB	4 $\times$ A100 80GB
Backbone / initialization	SmolLM2-360M	Mid-training checkpoint; the no-mid-training condition uses the backbone directly
Training objective	Causal language modeling on fixed token blocks	Assistant-only supervised chat loss
Training data	Mid-training corpus (experiment-specific)	Tulu 3
Total budget	1 epoch	3 epochs
Sequence length	4,096	2,048

Appendix

Table 16: Mid-training and SFT hyperparameters for the SmolLM2-360M RLVR experiments.

Parameter	RLVR (GRPO)
GPUs	4 $\times$ A100 80GB
Backbone / initialization	SFT checkpoint (per group)
Training objective	GRPO with verifiable rewards
Training data	RLVR-GSM-MATH-IF-Mixed-Constraints
Total budget	1 epoch
Prompt/completion lengths	1,024 / 256

Appendix

Table 17: RLVR (GRPO) hyperparameters for the SmolLM2-360M experiments. All evaluated configurations (No mid-training, Dolmino, SkillCorpus ) share identical RLVR settings.

Mid-training	API-Bank	MetaTool	APTBench	ToolEyes
No mid-training	5.44	1.16	4.54	8.64
Dolmino	7.69	4.75	7.66	8.90
SkillCorpus	15.93	10.88	9.43	11.52

Appendix

Table 18: Full agentic benchmark results after RLVR post-training. All models use the SmolLM2-360M backbone.

Mid-training	ARC	BQ	HS	PIQA	WG	MMLU
No mid-training	23.21	62.80	24.10	49.20	49.70	31.57
Dolmino	24.66	63.50	24.60	50.20	50.43	32.36
SkillCorpus	24.83	63.80	26.00	51.20	50.67	32.43

Appendix

Table 19: Full general benchmark results after RLVR post-training.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 586 papers

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation
Aug 3, 2026Zelin Tan, Yiqun Zhang, Hao Li +11LLM Agent Skill Learning LLM Agent Trajectory Synthesis
Shanghai Artificial Intelligence Laboratory
MidTool: Mid-training Data Synthesis for Agentic Tool Use
Aug 20, 2026Fengqing Jiang, Yite Wang, Boyi Liu +5LLM Tool Use LLM Agent Training
University of Washington · Snowflake · University of North Carolina at Chapel Hill
OpenThoughts-Agent: Data Recipes for Agentic Models
Jun 23, 2026Negin Raoof, Richard Zhuang, Marianna Nezhurina +47Training Data Curation LLM Agent Training

Aug 3, 2026J/K move · Enter open · S save

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Zelin Tan, Yiqun Zhang, Hao Li, Zhiyao Cui+10

Shanghai Artificial Intelligence Laboratory

LLM Agent Skill Learning LLM Agent Trajectory Synthesis Synthetic Data Generation for Language Models Agent Benchmarks

Open paper AskPDF arXiv

Agent skills have become an important mechanism for equipping language-model agents with reusable procedural knowledge. However, providing skills alone does not guarantee that current models can effectively identify, apply, and coordinate them. To improve skill-use capabilities, we introduce SKT, a verified data synthesis pipeline that constructs skill-grounded tasks and executable trajectories from large collections of agent skills. SKT selects suitable single-skill and multi-skill configurations, synthesizes tasks through rule-based and agent-based verification with feedback-guided repair, and retains only successful trajectories that substantially use every required skill. Using 2,000 public skills, SKT produces 4,000 task packages and 27,164 verified trajectories. Based on the same pipeline and a disjoint test pool, we further construct SkillEval, a held-out executable benchmark for evaluating skill use. Experiments across diverse models, benchmarks, and agent harnesses show that supervised fine-tuning on SKT-generated trajectories consistently improves skill-use performance. Verification ablations, cross-harness evaluation, and scaling experiments further demonstrate that these gains depend on high-quality supervision, extend beyond a single agent interface, and increase with broader skill coverage. Together, these results establish verified data synthesis as an effective and scalable approach for skill-use training.

Abstract

LLM Agent Training

Explore similar work

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

MidTool: Mid-training Data Synthesis for Agentic Tool Use

OpenThoughts-Agent: Data Recipes for Agentic Models