SkillForge: Evolving Verifiable Skills for Reinforcement Learning Agents

Shidong Yang; Ziyu Ma; Tongwen Huang; Xucong Wang; Renda Li; Yiming Hu; Yong Wang; Xiangxiang Chu

cs.CLAug 25, 2026

SkillForge: Evolving Verifiable Skills for Reinforcement Learning Agents

Authors: Shidong Yang, Ziyu Ma, Tongwen Huang, Xucong Wang, Renda Li, Yiming Hu, Yong Wang, Xiangxiang Chu

Organizations: AMAP, Alibaba Group

Abstract

Large language model (LLM) agents are trained with reinforcement learning (RL) for complex decision-making tasks. However, most RL-trained agents remain episodic and cannot accumulate reusable knowledge across episodes. Recent skill-based approaches, such as SkillRL, attempt to address this issue by extracting skills from raw trajectories, but treat the skill bank as an append-only repository without verifying whether stored skills remain effective. In this paper, we propose SkillForge, a framework for continuous skill evolution that enables skills to be verified and refined through environment interaction. By making skill usage explicit during agent interaction, RL can directly optimize both environment actions and skill invocation decisions. SkillForge further introduces evidence-based skill verification and multi-pathway skill induction, allowing the skill bank to continuously grow while maintaining its quality. Extensive experiments on ALFWorld, WebShop, and AppWorld show that SkillForge consistently outperforms SkillRL, demonstrating the effectiveness of continuously verified skills in training stronger LLM agents.

Figures & tables

Figure 1: (a) Existing methods distill trajectories into skills and append them to the bank, but skills are injected in bulk without explicit usage tracking or quality control. (b) SkillForge introduces explicit skill calling with per-skill effectiveness tracking to update existing skills, and multi-pathway induction to continuously add new skills to the evolving bank.

Figure 2: Overview of the SkillForge framework. Skills are retrieved from the skill bank and injected as a compact catalog into the agent’s prompt (§ 3.1 ). During rollout, the agent explicitly invokes skills via structured tags, and the policy is optimized with GRPO (§ 3.2 ). Rollout trajectories then drive multi-pathway skill induction and evidence-based verification to continuously evolve the skill bank (§ 3.3 ).

Method	ALFWorld							WebShop		AppWorld
Method	Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Succ.	TGC	SGC
Closed-source LLMs
GPT-4o	75.3	60.8	31.2	56.7	21.6	49.8	48.0	31.8	23.7	48.8	32.1
Gemini-2.5-Pro	92.8	63.3	62.1	69.0	26.6	58.7	60.3	42.5	35.9	–	–
Qwen2.5-7B-Instruct
Qwen2.5	33.4	21.6	19.3	6.90	2.80	3.20	14.8	26.4	7.80	0.59	0.00

Table 1: Performance on ALFWorld, WebShop, and AppWorld. For ALFWorld, we report the average success rate (%) for each subtask as well as the overall result. For WebShop, we report both the average score and the average success rate (%). For AppWorld, we report Task Goal Completion (TGC) and Scenario Goal Completion (SGC). ∗ denotes the results replicated from ( Feng et al., 2025 ) .

Figure 3: Evolution of the skill bank during training on ALFWorld and AppWorld. The y-axis shows the number of active skills, grouped into general and task-specific categories. New skills are continuously induced via trajectory analysis, while potentially underperforming ones are reviewed and revised when needed through reflexion, resulting in steady but controlled growth.

Skill Calling
Method	ALFWorld	AppWorld
SkillForge	87.9	44.6
w/o Explicit Calling	77.9	33.3
w/o Skill Bank	79.3	34.5
Skill Induction
w/o Multi-Pathway	82.1	36.9

Table 2: Ablation study on Qwen3-4B. We report ALFWorld success rate (%) and AppWorld TGC (%).

Figure 4: t-SNE visualization of skill intent embeddings at different training steps on ALFWorld (top) and AppWorld (bottom). Gray points denote the initial skill bank, while colored points represent skills induced during training. As training progresses (Step 20, 40, 60, and 80), newly generated skills gradually expand into previously unexplored regions of the embedding space, indicating the progressive diversification of the skill bank through trajectory-driven induction and reflexion-based refinement.

Skill Bank Source	ALFWorld	AppWorld
No skill	26.4	26.8
Qwen3-4B @ init (step 0)	27.9	27.4
Qwen3-4B @ step 20	28.6	28.6
Qwen3-4B @ step 40	28.6	31.5
Qwen3-4B @ step 80	32.9	31.5
Qwen3-30B-A3B self-evolved	34.3	30.4

Table 3: Skill transferability across model scales. Skill banks produced by Qwen3-4B-Instruct at different training stages are directly applied to Qwen3-30B-A3B-Instruct without further training.

Figure 5: Training curves on ALFWorld and AppWorld. SkillForge achieves faster convergence and higher asymptotic performance than the variant without explicit calling.

Figure 6: Evolution timeline of a representative skill ( Authenticate Before Acting ) across training stages on AppWorld. The figure shows how the skill evolves from Step 0 to Step 40 and Step 80 through trajectory-driven induction and reflexion. The initial version captures a basic authentication strategy. As training progresses, the skill is refined to include explicit credential verification and stronger applicability conditions. Green text marks newly added content, while red text indicates revised or removed parts of the skill definition.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Learning rate	$1\mathrm{e}{-6}$
Group size ( $n$ )	8
Training batch size	32
Optimizer	AdamW
Clip ratio low	0.20
Clip ratio high	0.28

Appendix

Table 4: Hyperparameters for RL training.

Aspect	SkillRL / SkillForge
Pipeline	SFT + RL / RL only
Skill use	Prompt injection / Explicit call
Signal	Trajectory-level / Skill-level
Update	Library expansion / Per-skill verify
Optimization	Skills as context / Action + skill

Appendix

Table 5: Conceptual comparison between SkillRL and SkillForge.

Benchmark	SkillForge	SkillRL
AppWorld	16.5h	18.7h
ALFWorld	5.9h	6.7h
WebShop	7.0h	7.7h

Appendix

Table 6: End-to-end training time comparison.

Setting	Score
WebShop
GRPO (no teacher)	79.4
Self-teacher (4B)	82.0
Qwen3-Max teacher	84.0
ALFWorld
GRPO (no teacher)	79.3

Appendix

Table 7: Teacher sensitivity analysis with Qwen3-4B-Instruct as the fixed policy model.

Metric	Value
AppWorld
GRPO score / time	34.5 / 16.1h
SkillForge score / time	44.6 / 16.5h
Skill retrieve & calling ratio	2.48%
Skill update ratio	1.70%
Total skill-related ratio	4.18%

Appendix

Table 8: Efficiency breakdown of SkillForge-specific overhead across three benchmarks.

Figure 7: Case study on ALFWorld. The agent invokes Open Then Heat upon navigating to the microwave with the tomato in hand. The skill output provides the procedural sequence (open

\rightarrow

heat), which the agent follows to complete the task successfully.

Figure 8: Case study on AppWorld. The agent invokes Enumerate User-Owned Entities First to scope playlist retrieval to user-owned data via show_playlist_library with pagination, avoiding the failure mode of substituting global search results for user-owned items.

Figure 9: Case study on WebShop. When no search result satisfies all constraints, the agent invokes Fallback to Best Match . Guided by the skill, it clicks into the nearest candidate’s product page, where hidden variant options (flavor: “sesame seeds”; size: “4.25 ounce (pack of 6)”) are progressively disclosed, enabling a successful purchase at $34.44.

LLM Agent Skill Learning

Large language model agent skill learning studies how agents acquire, represent, reuse, and refine procedures for multi-step tasks.

See all 402 papers

SkillForge: Co-Evolving Skills and Agents via Dynamic Skill Lifecycles
Oct 7, 2026Yuyao Ge, Yiwei Wang, Yuchen He +5LLM Agent Skill Learning LLM Agents
Institute of Computing Technology, Chinese Academy of Sciences · University of California, Merced · Tsinghua University
Skill-R1: Agent Skill Evolution via Reinforcement Learning
May 10, 2026Yash Vishe, Rohan Surana, Xunyi Jiang +8Agent Skill Learning RL for LLM Agents
UC San Diego · Adobe Research
SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
Feb 9, 2026Peng Xia, Jianwen Chen, Hanyang Wang +10LLM Agent Skill Learning RL for LLM Agents
UNC-Chapel Hill · University of Chicago · University of California San Diego +3

Oct 7, 2026J/K move · Enter open · S save

SkillForge: Co-Evolving Skills and Agents via Dynamic Skill Lifecycles

Yuyao Ge, Yiwei Wang, Yuchen He, Baolong Bi+4

Institute of Computing Technology, Chinese Academy of Sciences · University of California, Merced · Tsinghua University

LLM Agent Skill Learning LLM Agents RL for LLM Agents Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Memory-augmented reinforcement learning strengthens LLM agents' ability to solve complex long-horizon tasks. Skills are one such form of memory, pairing instructions with an applicability condition over task types. However, retaining every skill indiscriminately as the policy improves lets obsolete or harmful entries accumulate and mislead the agent. We propose SkillForge, an agentic RL method that compiles and evolves the skill library through a fitness-driven skill lifecycle of trial, active, stable, and retired states, so that the skills and the model co-evolve throughout training. A pre-RL evaluation phase first uses the base model's own rollouts to pre-retire low-fitness skills, yielding a filtered library that then seeds supervised fine-tuning. Reinforcement learning takes over from this checkpoint, and at each iteration selective retirement, stabilization, and LLM-guided mutation continue to forge the skill library alongside policy optimization. Across multiple interactive agent benchmarks, SkillForge achieves the highest aggregate success rate, delivering up to 7.8% relative improvement over the strongest baseline while keeping the skill library compact throughout training. We introduce SkillFurnace, a dataset of 5k+ annotated records bundling retirement-filtered SFT trajectories, evolved skill libraries with fitness annotations, and retirement events with human-annotated failure categories to support research on skill quality and lifecycle management.

Abstract

LLM Agent Skill Learning

Explore similar work

SkillForge: Co-Evolving Skills and Agents via Dynamic Skill Lifecycles

Skill-R1: Agent Skill Evolution via Reinforcement Learning

SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning