Skill-augmented agentic RL has emerged as an effective approach to improving LLM agents on long-horizon tasks. Despite this progress, individual skills may follow distinct trajectories as the policy evolves, with some becoming redundant after the model internalizes their capabilities, others remaining valuable as external scaffolding, and still others turning harmful. However, existing skill library systems fail to identify each skill's true contribution due to free-riding among co-retrieved skills, difficulty confounding across tasks, and the absence of skill-specific counterfactual evidence, which prevents reliable skill lifecycle management. We propose TrackRL, an agentic RL framework where skill tracking drives skill library full lifecycle management. Specifically, within each GRPO group, we turn grouped sampling into online randomized controlled experiments. Independent random injection and within-group contrasts on the same task isolate each skill's true contribution, and reliability-weighted aggregation accumulates evidence across groups. As evidence accumulates, we continuously track skill utility, which measures the skill's causal contribution as external scaffolding, and model mastery, which reflects how far the corresponding capability has been internalized. Along this line, we formulate skill lifecycle management as an evidence-gated state-transition process that governs each skill from generation and admission through protected incubation to retention, retirement, or removal, while a soft capacity bound controls library growth. Experiments show that TrackRL outperforms recent skill-augmented RL baselines in both library-assisted and skill-free evaluation, with further analyses revealing how skill value shifts during policy learning and how evidence-driven state transitions shape skill library evolution. Our code is available at https://github.com/xizhu1022/TrackRL.
Figures & tables
Figure 1: Skill lifecycle. Tracked skill utility, model mastery, and evidence drive the lifecycle of each skill. Retained skills serve as external scaffolding, whereas retired skills have been internalized by the policy.
Figure 2: Overview of TrackRL. We maintain a skill library of task-level and step-level skills, where we randomize their injection within GRPO groups to create per-skill treatment and control arms. From these comparisons, we track skill utility, model mastery, and evidence to guide skill lifecycle management. The updated library then supports subsequent policy optimization, closing the loop of policy–library co-evolution.
Method
Seen, per task class
Seen
Seen
Unseen
Unseen
Pick
Clean
Cool
Look
Heat
Pick2
skill- assisted
skill- free
skill- assisted
skill- free
Closed-source LLMs
Gemini-2.5-Flash
88.6
29.6
28.0
23.1
56.2
50.0
-
50.0
-
49.2
GPT-5.6-Luna
74.3
29.6
44.0
53.8
43.8
45.8
-
50.0
-
50.0
RL from the base model (Qwen3-4B-Instruct-2507)
Origin
40.5
11.5
8.0
21.4
26.7
0.0
-
19.3
-
27.6
Table 1: Results on ALFWorld. SR (%) in the skill-assisted and skill-free modes on the same checkpoint, with per-class results on the seen split. Origin is the base model before RL, Luna and Flash denote the reflection model. "-" marks a method without a skill library. In each setting the best summary result is shown in bold .
Method
Skill-assisted
Skill-free
TGC-N
SGC-N
TGC-C
SGC-C
TGC-N
SGC-N
TGC-C
SGC-C
Closed-source LLMs
Gemini-2.5-Flash
-
-
-
-
41.1
26.8
28.3
15.1
GPT-5.6-Luna
-
-
-
-
83.9
76.8
76.7
58.3
RL from the base model (Qwen3-4B-Instruct-2507)
Origin
-
-
-
-
5.9
0.0
2.4
0.7
Table 2: Results on AppWorld. TGC and SGC (%) on the normal (N) and challenge (C) splits in the skill-assisted and skill-free modes on the same checkpoint. Luna and Flash denote the reflection model. In each setting the best summary result is shown in bold .
Figure 3: Ablation on ALFWorld. Success rate (%) of TrackRL and its three variants in the four evaluation settings, namely the seen and unseen splits evaluated in the skill-assisted and skill-free modes. The variant w/o both has no library, so its two modes coincide.
Figure 4: Skill library growth on ALFWorld. (a, c) Library size of TrackRL, D2Skill, and SkillRL under cold start and warm start. (b, d) Composition of the TrackRL library by skill state. Significantly beneficial skills have utility exceeding zero by two standard errors.
Figure 5: Task-level skills on the utility–mastery plane. Each marker represents an active skill positioned by its current utility us and mastery ms , with color indicating its eventual lifecycle outcome and size reflecting its accumulated evidence. Corner numbers report the skill count in each quadrant, while arrows trace the earlier positions of skills later retired or removed.
Figure 6: TrackRL lifecycle events on ALFWorld. Bars show admissions upward and exits downward over five-step intervals, while lines show active and protected skill counts.
Figure 7: Three skill lifecycles from ALFWorld. The three skills end in removal, retirement, and retention, respectively. Shaded bands indicate utility confidence intervals.
Table 3: Mathematical Notation. The upper block is the sampling and optimization setting, the middle block is the state TrackRL tracks for a skill, and the lower block is the parameters that govern retrieval and the lifecycle.
Parameter
Symbol
Value
Policy model
πθ
Qwen3-4B-Instruct-2507
Tasks per training step
M
16
Episodes per group
B
8
Episodes per training step
M⋅B
128
Training steps
—
160
Learning rate
—
1×10−6
Appendix
Table 4: Reinforcement learning hyperparameters, shared by all trained methods.
Parameter
Symbol
Value
Injection probability, randomized episodes
p
1/2
Skill-free episodes reserved per group
—
1 task-level, 1 step-level
Recency discount
γ
0.98
Evidence threshold for any exit
Emin
10
Confidence level of both one-sided tests
λ
2
Absolute mastery floor for retirement
mmin
0.5
Appendix
Table 5: TrackRL parameters. The upper block is introduced by skill tracking and lifecycle management. The lower block is the retrieval configuration shared with the skill-augmented baselines.
Method
Mode
Pick
Clean
Cool
Look
Heat
Pick2
Overall
D2Skill (Luna)
skill-assisted
46.7
63.0
47.8
45.5
46.7
72.7
54.7
skill-free
55.9
40.0
70.8
54.5
35.7
45.0
52.3
TrackRL (Luna)
skill-assisted
61.3
65.2
82.6
61.5
92.9
75.0
71.9
skill-free
63.3
66.7
52.2
91.7
75.0
52.2
64.1
TrackRL (Luna) + SFT
skill-assisted
100.0
96.2
96.0
100.0
100.0
95.2
97.7
skill-free
87.5
88.0
100.0
100.0
100.0
100.0
95.3
Appendix
Table 6: Results on the unseen split of ALFWorld by task class. SR (%) by task class, with the overall values of Table 1 .
Figure 8: Task-level skills by family (ALFWorld, base model, Luna reflection). One panel per family, showing the mean mastery of the family’s active task-level skills on the left axis, the success rate of the policy without the library on the seen validation games of that family on the same axis as a dashed line and as a three-point moving average since each family has about twenty games per validation, the family utility pooled over the family’s skills on the right axis, and one vertical tick per retirement of a skill of that family. Panel titles give the number of active task-level skills of the family at the last step and the number retired.
Memory-augmented reinforcement learning strengthens LLM agents' ability to solve complex long-horizon tasks. Skills are one such form of memory, pairing instructions with an applicability condition over task types. However, retaining every skill indiscriminately as the policy improves lets obsolete or harmful entries accumulate and mislead the agent. We propose SkillForge, an agentic RL method that compiles and evolves the skill library through a fitness-driven skill lifecycle of trial, active, stable, and retired states, so that the skills and the model co-evolve throughout training. A pre-RL evaluation phase first uses the base model's own rollouts to pre-retire low-fitness skills, yielding a filtered library that then seeds supervised fine-tuning. Reinforcement learning takes over from this checkpoint, and at each iteration selective retirement, stabilization, and LLM-guided mutation continue to forge the skill library alongside policy optimization. Across multiple interactive agent benchmarks, SkillForge achieves the highest aggregate success rate, delivering up to 7.8% relative improvement over the strongest baseline while keeping the skill library compact throughout training. We introduce SkillFurnace, a dataset of 5k+ annotated records bundling retirement-filtered SFT trajectories, evolved skill libraries with fitness annotations, and retirement events with human-annotated failure categories to support research on skill quality and lifecycle management.
Yuyao Ge, Yiwei Wang, Yuchen He +5
Institute of Computing Technology, Chinese Academy of Sciences · University of California, Merced · Tsinghua University
Large language model agents increasingly rely on external skills to solve complex tasks, where skills act as modular units that extend their capabilities beyond what parametric memory alone supports. Existing skill-based RL methods assume external skills either accumulate as persistent guidance or internalized into the policy, eventually leading to zero-skill inference. We argue this assumption is overly restrictive, since with limited parametric capacity and uneven marginal contribution across skills, the optimal active skill set is non-monotonic, task- and stage-dependent. In this work, we propose SLIM, a framework of dynamic Skill LIfecycle Management for agentic reinforcement learning (RL), which dynamically updates the active external skill set as the policy evolves.Specifically, SLIM estimates each active skill's marginal external contribution through leave-one-skill-out validation, then applies three lifecycle operations: retaining high-value skills, retiring skills whose contribution becomes negligible after sufficient exposure, and expanding the skill bank when persistent failures reveal missing capability coverage. Experiments show that SLIM outperforms the best baselines by an average of 7.1% points across ALFWorld and SearchQA. Results further indicate that policy learning and external skill retention are not mutually exclusive: some skills are absorbed into the policy, while others continue to provide external value, supporting SLIM as a more general paradigm for skill-based agentic RL.
Junhao Shen, Teng Zhang, Xiaoyan Zhao +1
Database Group, The Chinese University of Hong Kong · Lanzhou University
Agentic reinforcement learning (RL) enables LLM agents to improve continuously from environment rewards, yet the resulting policies do not systematically accumulate reusable strategies that generalize across tasks. Modular skills can provide such reusable strategies, yet existing skill-augmented RL methods decouple skill creation from policy optimization, risking adopting skills that conflict with the evolving policy. Inspired by Anthropic's Skill Creator, we introduce ReSkill, an RL-in-the-loop skill creation framework that reconciles skill evolution with policy learning. ReSkill exploits the group-wise structure of GRPO to naturally embed three mechanisms with only marginal additional overhead: (1) an assertion-driven skill creator that diagnoses failures from past experience and proposes conditional, trigger-based skill revisions; (2) within-group rollout sampling that enables controlled comparison of skill versions, capturing which version best supports the policy's ongoing learning; and (3) Thompson Sampling with adaptive discounting to balance exploration and exploitation in skill version selection as the policy evolves. Across several domains, ReSkill consistently outperforms existing memory and skill-based RL methods, with the largest gains on unseen tasks. Analysis of the skill lifecycle shows skills being automatically created, tested, refined, and pruned as the policy improves, demonstrating reconciled skill-policy co-evolution.
Zelin He, Haotian Lin, Boran Han +6
The Pennsylvania State University · † Work done during an internship at Amazon · Amazon IntelliHub +1