Dynamic Dual-Granularity Skill Bank for Agentic RL

Songjun Tu; Chengdong Xu; Qichao Zhang; Yaocheng Zhang; Xiangyuan Lan; Linjing Li; Dong Li; Dongbin Zhao

cs.AIMar 30, 2026

Dynamic Dual-Granularity Skill Bank for Agentic RL

Authors: Songjun Tu, Chengdong Xu, Qichao Zhang, Yaocheng Zhang, Xiangyuan Lan, Linjing Li, Dong Li, Dongbin Zhao

Organizations: Institute of Automation, Chinese Academy of Sciences · Pengcheng Laboratory · Sun Yat-Sen University · MemoraX AI

Abstract

Agentic RL can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill, a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld, WebShop, and Search-Augmented QA tasks show that D2Skill substantially improves performance over skill-free baselines across models of different scales. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of D2Skill. (a) The dynamic dual-granularity skill bank with retrieval, reflection-driven generation, and management. (b) Overall results on ALFWorld and WebShop .

Figure 2: Step skills correct local search failures. Without step skills, the agent can repeat the same failed search after a negative observation. A retrieved step skill redirects it to another plausible location.

Panel of Figure 3 — Figure 3: Training and skill-bank dynamics motivating D2Skill. (a) ALFWorld training curves for the D2Skill skill group, the paired baseline group, and GRPO . (b) Skill-bank dynamics with and without management, measured by average skill utility and retrieval statistics.

Figure 4: Overall framework of D2Skill. D2Skill couples RL with a dynamic dual-granularity skill bank. For each task, training rollouts are divided into a baseline group and a skill group, whose performance gap yields hindsight signals for policy optimization and skill utility estimation. When performance is poor, reflection on representative failed trajectories produces task skills for high-level guidance and step skills for local error correction. Skills are stored with retrieval keys, reused during interaction, and periodically pruned by utility-based bank management.

Method	ALFWorld							WebShop
Method	Pick	Clean	Cool	Look	Heat	Pick2	All	Score	Success
Closed-source LLMs
Gemini-3-Flash	96.4	57.1	96.2	85.7	72.2	95.3	85.2	14.1	16.5
O3	64.3	19.1	23.1	64.3	33.3	61.9	43.8	5.8	4.7
Base Model: Qwen2.5-7B-Instruct
Origin	17.9	4.8	3.8	64.3	0.0	5.3	12.5	16.6	3.9

Table 1: Performance on ALFWorld and WebShop . For ALFWorld , we report the average success rate (%) for each subtask and the overall success rate. For WebShop , we report the average score and average success rate (%). Evaluation details are given in Appendix C.1 . ∗ denotes results replicated from ( Feng et al., 2025 ) and ( Xia et al., 2026 ) . The best and second-best results are highlighted in red and blue , respectively.

Method	Single-Hop QA			Multi-Hop QA			Avg.
Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Avg.
Base Model: Qwen2.5-7B-Instruct
Origin	11.6	35.6	1.20	16.4	22.2	4.80	15.3
Search-R1	39.3	61.0	39.7	37.0	40.1	14.6	38.6
ZeroSearch	43.6	61.8	51.5	34.6	35.2	18.4	40.9
StepSearch	37.7	54.7	38.6	37.0	40.9	34.6	40.6

Table 2: Performance on Search-Augmented QA . We report accuracy (%) on single-hop and multi-hop datasets.

Method	Train		Val
Method	Skill	Baseline	Val
D2Skill	70.9	65.8	72.7
w/o task skills	59.1	53.7	62.7
w/o step skills	57.8	55.8	60.2
w/o skill management	60.0	57.4	57.8
w/o baseline group	63.9	-	68.8

Table 3: Ablation Study on ALFWorld .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

x_{s}=(\tau_{g},\mathcal{H}_{s}^{L},o_{s},\mathcal{A}_{s}^{\mathrm{adm}})

Appendix

Table 10

Category	Symbol / field	Value
Benchmark / prompt context
ALFWorld max steps		50
WebShop max steps		15
History length $L$	env.history_length	2
Environment seed	env.seed	0
Invalid-action penalty coef.	invalid_action_penalty_coef	1.0

Appendix

Table 4: Hyperparameters and training protocol for main experiments.

LLM Agent Skill Learning

Large language model agent skill learning studies how agents acquire, represent, reuse, and refine procedures for multi-step tasks.

See all 398 papers

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL
Jun 1, 2026Zelin He, Haotian Lin, Boran Han +6Reinforcement Learning LLM Agent Skill Learning
The Pennsylvania State University · † Work done during an internship at Amazon · Amazon IntelliHub +1
Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning
May 11, 2026Junhao Shen, Teng Zhang, Xiaoyan Zhao +1Agentic RL LLM Agent Skill Learning
Database Group, The Chinese University of Hong Kong · Lanzhou University
SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
Feb 9, 2026Peng Xia, Jianwen Chen, Hanyang Wang +10LLM Agent Skill Learning RL for LLM Agents
UNC-Chapel Hill · University of Chicago · University of California San Diego +3

Jun 1, 2026J/K move · Enter open · S save

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL

Zelin He, Haotian Lin, Boran Han, Wei Zhu+5

The Pennsylvania State University · † Work done during an internship at Amazon · Amazon IntelliHub+1

Reinforcement Learning LLM Agent Skill Learning RL for LLM Agents

Open paper AskPDF arXiv

Agentic reinforcement learning (RL) enables LLM agents to improve continuously from environment rewards, yet the resulting policies do not systematically accumulate reusable strategies that generalize across tasks. Modular skills can provide such reusable strategies, yet existing skill-augmented RL methods decouple skill creation from policy optimization, risking adopting skills that conflict with the evolving policy. Inspired by Anthropic's Skill Creator, we introduce ReSkill, an RL-in-the-loop skill creation framework that reconciles skill evolution with policy learning. ReSkill exploits the group-wise structure of GRPO to naturally embed three mechanisms with only marginal additional overhead: (1) an assertion-driven skill creator that diagnoses failures from past experience and proposes conditional, trigger-based skill revisions; (2) within-group rollout sampling that enables controlled comparison of skill versions, capturing which version best supports the policy's ongoing learning; and (3) Thompson Sampling with adaptive discounting to balance exploration and exploitation in skill version selection as the policy evolves. Across several domains, ReSkill consistently outperforms existing memory and skill-based RL methods, with the largest gains on unseen tasks. Analysis of the skill lifecycle shows skills being automatically created, tested, refined, and pruned as the policy improves, demonstrating reconciled skill-policy co-evolution.

Abstract

LLM Agent Skill Learning

Explore similar work

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL

Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning

SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning