Trackable Skill Library Lifecycle Management for Agentic Reinforcement Learning

Xi Zhu; Yuanlin Duan; Changyi Yang; Kai Mei; Riyang Bao; Shichao Zhu; Dimitris N. Metaxas

cs.LGOct 18, 2026

Trackable Skill Library Lifecycle Management for Agentic Reinforcement Learning

Authors: Xi Zhu, Yuanlin Duan, Changyi Yang, Kai Mei, Riyang Bao, Shichao Zhu, Dimitris N. Metaxas

Organizations: Rutgers University · Carnegie Mellon University · TikTok Inc.

Abstract

Skill-augmented agentic RL has emerged as an effective approach to improving LLM agents on long-horizon tasks. Despite this progress, individual skills may follow distinct trajectories as the policy evolves, with some becoming redundant after the model internalizes their capabilities, others remaining valuable as external scaffolding, and still others turning harmful. However, existing skill library systems fail to identify each skill's true contribution due to free-riding among co-retrieved skills, difficulty confounding across tasks, and the absence of skill-specific counterfactual evidence, which prevents reliable skill lifecycle management. We propose TrackRL, an agentic RL framework where skill tracking drives skill library full lifecycle management. Specifically, within each GRPO group, we turn grouped sampling into online randomized controlled experiments. Independent random injection and within-group contrasts on the same task isolate each skill's true contribution, and reliability-weighted aggregation accumulates evidence across groups. As evidence accumulates, we continuously track skill utility, which measures the skill's causal contribution as external scaffolding, and model mastery, which reflects how far the corresponding capability has been internalized. Along this line, we formulate skill lifecycle management as an evidence-gated state-transition process that governs each skill from generation and admission through protected incubation to retention, retirement, or removal, while a soft capacity bound controls library growth. Experiments show that TrackRL outperforms recent skill-augmented RL baselines in both library-assisted and skill-free evaluation, with further analyses revealing how skill value shifts during policy learning and how evidence-driven state transitions shape skill library evolution. Our code is available at https://github.com/xizhu1022/TrackRL.

Figures & tables

Figure 1: Skill lifecycle. Tracked skill utility, model mastery, and evidence drive the lifecycle of each skill. Retained skills serve as external scaffolding, whereas retired skills have been internalized by the policy.

Figure 2: Overview of TrackRL. We maintain a skill library of task-level and step-level skills, where we randomize their injection within GRPO groups to create per-skill treatment and control arms. From these comparisons, we track skill utility, model mastery, and evidence to guide skill lifecycle management. The updated library then supports subsequent policy optimization, closing the loop of policy–library co-evolution.

Method	Seen, per task class						Seen	Seen	Unseen	Unseen
Method	Pick	Clean	Cool	Look	Heat	Pick2	skill- assisted	skill- free	skill- assisted	skill- free
Closed-source LLMs
Gemini-2.5-Flash	88.6	29.6	28.0	23.1	56.2	50.0	-	50.0	-	49.2
GPT-5.6-Luna	74.3	29.6	44.0	53.8	43.8	45.8	-	50.0	-	50.0
RL from the base model (Qwen3-4B-Instruct-2507)
Origin	40.5	11.5	8.0	21.4	26.7	0.0	-	19.3	-	27.6

Table 1: Results on ALFWorld. SR (%) in the skill-assisted and skill-free modes on the same checkpoint, with per-class results on the seen split. Origin is the base model before RL, Luna and Flash denote the reflection model. "-" marks a method without a skill library. In each setting the best summary result is shown in bold .

Closed-source LLMs
Method	Skill-assisted				Skill-free
Method	TGC-N	SGC-N	TGC-C	SGC-C	TGC-N	SGC-N	TGC-C	SGC-C
Gemini-2.5-Flash	-	-	-	-	41.1	26.8	28.3	15.1
GPT-5.6-Luna	-	-	-	-	83.9	76.8	76.7	58.3
RL from the base model (Qwen3-4B-Instruct-2507)
Origin	-	-	-	-	5.9	0.0	2.4	0.7

Table 2: Results on AppWorld. TGC and SGC (%) on the normal (N) and challenge (C) splits in the skill-assisted and skill-free modes on the same checkpoint. Luna and Flash denote the reflection model. In each setting the best summary result is shown in bold .

Figure 3: Ablation on ALFWorld. Success rate (%) of TrackRL and its three variants in the four evaluation settings, namely the seen and unseen splits evaluated in the skill-assisted and skill-free modes. The variant w/o both has no library, so its two modes coincide.

Figure 4: Skill library growth on ALFWorld. (a, c) Library size of TrackRL, D2Skill, and SkillRL under cold start and warm start. (b, d) Composition of the TrackRL library by skill state. Significantly beneficial skills have utility exceeding zero by two standard errors.

Figure 5: Task-level skills on the utility–mastery plane. Each marker represents an active skill positioned by its current utility

u_{s}

and mastery

m_{s}

, with color indicating its eventual lifecycle outcome and size reflecting its accumulated evidence. Corner numbers report the skill count in each quadrant, while arrows trace the earlier positions of skills later retired or removed.

Figure 6: TrackRL lifecycle events on ALFWorld. Bars show admissions upward and exits downward over five-step intervals, while lines show active and protected skill counts.

Figure 7: Three skill lifecycles from ALFWorld. The three skills end in removal, retirement, and retention, respectively. Shaded bands indicate utility confidence intervals.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Introduced in
$\pi_{\theta},\ \pi_{\mathrm{old}},\ \pi_{\mathrm{ref}}$	current, behavior, and reference policy	Eq. ( 3 )
$M,\ B$	tasks per step and episodes per group	Section 2.1
$g,\ e$	a group of episodes and a single episode	Section 2.1
$x,\ \kappa(s)$	retrieval query and the retrieval key of skill $s$	Eq. ( 1 )
$y_{e},\ \phi_{e},\ R_{e},\ A_{e}$	outcome, invalid-action fraction, reward, group-normalized advantage	Eq. ( 3 )
$z_{s,e}$	injection assignment of skill $s$ in episode $e$	Eq. ( 4 )

Appendix

Table 3: Mathematical Notation. The upper block is the sampling and optimization setting, the middle block is the state TrackRL tracks for a skill, and the lower block is the parameters that govern retrieval and the lifecycle.

Parameter	Symbol	Value
Policy model	$\pi_{\theta}$	Qwen3-4B-Instruct-2507
Tasks per training step	$M$	16
Episodes per group	$B$	8
Episodes per training step	$M\cdot B$	128
Training steps	—	160
Learning rate	—	$1\times 10^{-6}$

Appendix

Table 4: Reinforcement learning hyperparameters, shared by all trained methods.

Parameter	Symbol	Value
Injection probability, randomized episodes	$p$	$1/2$
Skill-free episodes reserved per group	—	1 task-level, 1 step-level
Recency discount	$\gamma$	0.98
Evidence threshold for any exit	$E_{\min}$	10
Confidence level of both one-sided tests	$\lambda$	2
Absolute mastery floor for retirement	$m_{\min}$	0.5

Appendix

Table 5: TrackRL parameters. The upper block is introduced by skill tracking and lifecycle management. The lower block is the retrieval configuration shared with the skill-augmented baselines.

Method	Mode	Pick	Clean	Cool	Look	Heat	Pick2	Overall
D2Skill (Luna)	skill-assisted	46.7	63.0	47.8	45.5	46.7	72.7	54.7
	skill-free	55.9	40.0	70.8	54.5	35.7	45.0	52.3
TrackRL (Luna)	skill-assisted	61.3	65.2	82.6	61.5	92.9	75.0	71.9
	skill-free	63.3	66.7	52.2	91.7	75.0	52.2	64.1
TrackRL (Luna) + SFT	skill-assisted	100.0	96.2	96.0	100.0	100.0	95.2	97.7
	skill-free	87.5	88.0	100.0	100.0	100.0	100.0	95.3

Appendix

Table 6: Results on the unseen split of ALFWorld by task class. SR (%) by task class, with the overall values of Table 1 .

Figure 8: Task-level skills by family (ALFWorld, base model, Luna reflection). One panel per family, showing the mean mastery of the family’s active task-level skills on the left axis, the success rate of the policy without the library on the seen validation games of that family on the same axis as a dashed line and as a three-point moving average since each family has about twenty games per validation, the family utility pooled over the family’s skills on the right axis, and one vertical tick per retirement of a skill of that family. Panel titles give the number of active task-level skills of the family at the last step and the number retired.

LLM Agent Skill Learning

Large language model agent skill learning studies how agents acquire, represent, reuse, and refine procedures for multi-step tasks.

See all 402 papers

SkillForge: Co-Evolving Skills and Agents via Dynamic Skill Lifecycles
Oct 7, 2026Yuyao Ge, Yiwei Wang, Yuchen He +5LLM Agent Skill Learning LLM Agents
Institute of Computing Technology, Chinese Academy of Sciences · University of California, Merced · Tsinghua University
Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning
May 11, 2026Junhao Shen, Teng Zhang, Xiaoyan Zhao +1Agentic RL LLM Agent Skill Learning
Database Group, The Chinese University of Hong Kong · Lanzhou University
ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL
Jun 1, 2026Zelin He, Haotian Lin, Boran Han +6Reinforcement Learning LLM Agent Skill Learning
The Pennsylvania State University · † Work done during an internship at Amazon · Amazon IntelliHub +1

Oct 7, 2026J/K move · Enter open · S save

SkillForge: Co-Evolving Skills and Agents via Dynamic Skill Lifecycles

Yuyao Ge, Yiwei Wang, Yuchen He, Baolong Bi+4

Institute of Computing Technology, Chinese Academy of Sciences · University of California, Merced · Tsinghua University

LLM Agent Skill Learning LLM Agents RL for LLM Agents Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Memory-augmented reinforcement learning strengthens LLM agents' ability to solve complex long-horizon tasks. Skills are one such form of memory, pairing instructions with an applicability condition over task types. However, retaining every skill indiscriminately as the policy improves lets obsolete or harmful entries accumulate and mislead the agent. We propose SkillForge, an agentic RL method that compiles and evolves the skill library through a fitness-driven skill lifecycle of trial, active, stable, and retired states, so that the skills and the model co-evolve throughout training. A pre-RL evaluation phase first uses the base model's own rollouts to pre-retire low-fitness skills, yielding a filtered library that then seeds supervised fine-tuning. Reinforcement learning takes over from this checkpoint, and at each iteration selective retirement, stabilization, and LLM-guided mutation continue to forge the skill library alongside policy optimization. Across multiple interactive agent benchmarks, SkillForge achieves the highest aggregate success rate, delivering up to 7.8% relative improvement over the strongest baseline while keeping the skill library compact throughout training. We introduce SkillFurnace, a dataset of 5k+ annotated records bundling retirement-filtered SFT trajectories, evolved skill libraries with fitness annotations, and retirement events with human-annotated failure categories to support research on skill quality and lifecycle management.

Abstract

LLM Agent Skill Learning

Explore similar work

SkillForge: Co-Evolving Skills and Agents via Dynamic Skill Lifecycles

Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL