SkillCycle: Co-Evolving Agent Policies and Skill Banks

Ling Li; Qiuyu Shen; Zheng Jiang; Qinwei Ma; Yuxuan Liu; Zhidong Deng

cs.CVOct 7, 2026

SkillCycle: Co-Evolving Agent Policies and Skill Banks

Authors: Ling Li, Qiuyu Shen, Zheng Jiang, Qinwei Ma, Yuxuan Liu, Zhidong Deng

Organizations: Tsinghua University · Northwestern Polytechnical University

Abstract

Internalizing external skills changes a language agent's capabilities and, with them, the value of its remaining guidance: rules can become redundant, misleading, or insufficient for newly encountered decisions. This creates a coupled problem of learning from skills and adapting the skills that supervise further learning. We introduce SkillCycle, a framework for co-evolving agent policies and skill banks through a feedback loop between skill internalization and rule revision. Our central contribution is to give distillation feedback a second role: token-level contextual differences help locate rules for inspection, while interaction outcomes guide edits to their content and applicability. SkillCycle alternates between two phases: policy learning with a fixed skill bank and router, and rule revision with a frozen policy. Candidate edits undergo rule-level and whole-bank environment comparisons before they guide the next learning cycle. On WebShop, SkillCycle with a 3B model achieves a success rate of 74.74% and a score of 88.37 without inference-time skill inputs, representing relative improvements of 0.73% and 3.96% over the state-of-the-art (SOTA) model, respectively. In Cycle 3 ablations on ALFWorld and WebShop, SkillCycle's no-skill success rates improve by 10.18% and 18.11% relative to a static skill bank, and by 2.41% and 2.50% relative to a single bank update, respectively. These results show that continually revising skill guidance as the agent's capabilities change helps transform external skills into policy capabilities that require no skill inputs at inference. We will release code, configurations, skill banks, and evaluation protocols.

Figures & tables

Figure 1: Comparison between skill augmentation and skill internalization. Existing approaches (a) augment the policy with an external skill bank or (b) internalize skills into the policy to reduce reliance on external guidance. (c) Our proposed SkillCycle couples skill internalization with evidence-driven skill-bank revision, co-evolving the policy and the bank for skill-free inference. Bars and arrows illustrate conceptual trends, not measured trajectories.

Figure 2: SkillCycle architecture. (a) State-aware routing selects skill guidance from the bank. (b) With the bank and router frozen, a shared-weight, skill-conditioned Teacher rescores the no-skill Student’s responses; signed OPD signals and environment feedback guide policy learning. (c) OPD diagnostics and trajectory outcomes inform offline Keep , Rewrite , Drop , and Add proposals. (d) Paired rollouts under a frozen policy verify candidate revisions; accepted updates guide the next learning cycle. WebShop additionally uses verified selective Teacher demonstrations. Here, $h_{t}$ denotes the task instruction and observable interaction history. Bars and trajectories are schematic.

			ALFWorld							WebShop
Method	C	S/T	Pick	Look	Clean	Heat	Cool	Pick2	Avg.	Score	SR
(a) Qwen3-1.7B-Instruct
Vanilla ( Yang et al., 2025 )	–		25.00	22.20	3.10	0.00	21.40	4.20	12.50	46.50	4.70
Skill-Prompt ∗ ( Yang et al., 2026b )	–		10.30	50.00	16.10	0.00	0.00	5.00	9.40	23.00	2.30
OPSD ( Zhao et al., 2026 )	–		26.30	33.30	9.10	0.00	4.50	5.30	14.10	47.40	9.30
GRPO ( Shao et al., 2024 )	–		71.10	41.70	36.40	40.00	31.80	31.60	46.10	67.30	38.30

Table 1: Comparison with RL and skill-distillation methods. C: training cycle; S: no-skill Student; T: skill-conditioned Teacher. Baseline scores follow Yang et al. (2026b) ; evaluation panels are specified in Section 3.1 . Bold marks maxima within each reference block or local role; boxes mark final results.

\Delta

: last-minus-first change (points). All metrics are on a 0–100 scale; – denotes missing results.

Figure 3: Trajectory comparison on an ALFWorld cleaning task: the no-skill model fails at the 30-step limit (top), while the same model with SkillCycle’s routed skills succeeds in 8 steps (bottom). Snapshots show observations, actions, outcomes, and interaction progress; the bottom row also shows the selected skills.

	ALFWorld Macro SR		WebShop SR
Condition	Student	Teacher	Student	Teacher
GiGPO ( Feng et al., 2025 )	55.03 [50.03, 60.03]	61.29 [56.29, 66.29]	63.54 [58.62, 68.20]	64.84 [59.94, 69.45]
Static	48.61 [43.61, 53.61]	51.82 [46.82, 56.82]	63.28 [58.35, 67.95]	66.41 [61.54, 70.95]
Refresh once	52.30 [47.30, 57.30]	58.81 [53.81, 63.81]	72.92 [68.26, 77.12]	76.82 [72.35, 80.77]
SkillCycle	53.56 [49.16, 63.81]	58.28 [53.24, 63.78]	74.74 [70.16, 78.83]	76.82 [72.35, 80.77]
$\Delta$ vs. Static	+4.95	+6.46	+11.46	+10.42

Table 2: Bank-update ablations at Cycle 3. ALFWorld uses Qwen3-1.7B; WebShop uses Qwen2.5-3B. Student is evaluated without skills; Teacher uses routed skills. Cells show SR (%) with reported intervals below;

\Delta

denotes percentage-point differences. Bold marks column maxima. Interval reporting and full task/cycle results appear in Appendix K .

(a) ALFWorld: frozen Qwen3-1.7B
Revision	Pick	Look	Clean	Heat	Cool	Pick2	Macro SR	$\boldsymbol{\Delta}$
1	32.14	43.81	60.47	70.95	61.03	28.16	49.43	–
2	37.84	44.71	61.94	72.38	57.77	34.06	51.45	+2.02
3	40.12	44.93	63.25	72.10	57.99	36.28	52.45	+3.02

Table 3: Skill-bank revision with a frozen policy. Revision indexes bank updates with unchanged model weights. Rates are percentages;

\Delta

is the change from the first record (pp). Initial guidance settings and comparison scope are discussed in Appendix K .

Figure 4: Learning dynamics and skill-conditioned performance. (a,b) No-skill Student SR; dashed lines mark the common Cycle 3 endpoint, shading later SkillCycle-only training. (c,d) Cycle 3 Student (circles) and Teacher (diamonds) SR, paired by condition. Axes use separate ranges; Appendix K lists all condition-wise numerical results and their reported intervals.

Operation	Pairs	Gains	Losses	Net	Decision
Add simple-search recovery	48	1	1	0	Reject
Rewrite clean readiness	48	0	1	$-1$	Reject
Rewrite cool-search frontier	48	3	6	$-3$	Reject
Rewrite heat visible acquisition	48	0	1	$-1$	Reject
Rewrite heat retention	48	0	4	$-4$	Reject
Rewrite invalid-action contract	240	4	5	$-1$	Reject

Table 4: ALFWorld-3B Cycle 3 operation verification at fixed weights. Matched-pair counts compare each operation’s inclusion with its removal. Net is Gains minus Losses; accepted edits are highlighted.

Figure 5: Skill routing across four API models. Bars show three-seed SR; columns give gains over No Skill and Static (pp). Axes start at zero with different ranges. Full results: Table 7 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$x,\mathcal{D}$	Task instruction and task distribution.
$s_{t},o_{t}$	Latent environment state and exposed observation.
$h_{t},z_{t}$	Observable interaction history and constructed model context.
$C,\mathcal{C}$	Number of training segments and the context constructor.
$y_{t},y_{t,j}$	Generated response and its token at position $j$ .
$a_{t},\psi_{x}$	Executed action and response-to-action interface.

Appendix

Table 10

Setting	WebShop 1.7B	WebShop 3B	ALFWorld 3B
$(\eta,\lambda_{o},\lambda_{d})$	$(0,.001,.1)$	$(0,0,.1)$	$(.1,0,0)$
Demo tokens / reduction	Action / $\mathfrak{T}$	Full response / $\mathfrak{R}$	Disabled
Formal actor updates	8	8	8
Learning rate	$10^{-6}$	$10^{-6}$	$10^{-6}$
Reference coefficient $\beta$	$.01$	$.01$	$.01$
Configuration	Auxiliary + demo	Demo	Advantage injection

Appendix

Table 6: Cycle 3 optimization configurations.

1	for $c=1,\ldots,C$ do
2	$\theta_{c,0}\leftarrow\theta_{c-1}$ ; $E_{c}\leftarrow\varnothing$ Phase 1: fix $H_{c-1}$
3	for $u=0,\ldots,N_{c}-1$ do
4	$x_{b}\sim\mathcal{D}_{\mathrm{train}}$ , $\tau_{b}\sim p_{\theta_{c,u},\varnothing}(\cdot\mid x_{b})$ ; $\mathcal{T}_{c,u}\leftarrow\{\tau_{b}\}_{b}$
5	$k_{t}\leftarrow R_{c-1}(h_{t};B_{c-1})$ for each decision $t\in\mathcal{T}_{c,u}$
6	$g_{t,j}\leftarrow\operatorname{sg}[\log\pi^{T}_{\theta_{c,u}}(y_{t,j}\mid h_{t},k_{t},y_{t,<j})-\log\pi^{S}_{\theta_{c,u}}(y_{t,j}\mid h_{t},y_{t,<j})]$
7	$A_{c,u}\leftarrow\operatorname{Adv}_{\omega}(\mathcal{T}_{c,u})$ ; $Z_{c,u}\leftarrow\operatorname{Aux}_{\omega}(\mathcal{T}_{c,u})$

Appendix

Algorithm 1 Detailed SkillCycle training and verification

DeepSeek-V4-Pro
	ALFWorld								WebShop
Condition	Pick	Look	Clean	Heat	Cool	Pick2	Overall	Macro	Score	SR
No Skill	86.11	50.00	69.57	58.33	21.74	57.69	60.94	57.24	7.54	5.73
Static General Skill	88.89	62.50	69.57	36.11	63.77	56.41	67.71	62.87	11.86	8.33
Skill + Router	86.11	75.00	69.57	50.00	47.83	57.69	66.41	64.37	15.59	11.20
GLM-5.1

Appendix

Table 7: Skill and routing comparisons on API models. Three evaluation seeds; all metrics are on a 0–100 scale (

\uparrow

). ALFWorld includes overall and macro SR. Blue rows mark skill-conditioned inference, with stronger shading for Skill + Router; darker cells mark its within-model maxima. Bold denotes maxima across conditions, including ties.

		ALFWorld ( $\uparrow$ )							WebShop ( $\uparrow$ )
Method	Cycle	Pick	Look	Clean	Heat	Cool	Pick2	Avg.	Score	SR
Vanilla ( Yang et al., 2025 )	–	25.00	22.20	3.10	0.00	21.40	4.20	12.50	46.50	4.70
Skill-Prompt ∗ ( Yang et al., 2026b )	–	10.30	50.00	16.10	0.00	0.00	5.00	9.40	23.00	2.30
OPSD ( Zhao et al., 2026 )	–	26.30	33.30	9.10	0.00	4.50	5.30	14.10	47.40	9.30
GRPO ( Shao et al., 2024 )	–	71.10	41.70	36.40	40.00	31.80	31.60	46.10	67.30	38.30
Skill-GRPO ( Yang et al., 2026b )	–	27.60	54.50	22.70	27.30	0.00	19.20	21.10	73.40	46.10

Appendix

Table 8: Benchmark results on Qwen3-1.7B-Instruct. Blue rows denote SkillCycle ; darker cells mark the best value across cycles for each role. Bold marks reference-block maxima.

		ALFWorld ( $\uparrow$ )							WebShop ( $\uparrow$ )
Method	Cycle	Pick	Look	Clean	Heat	Cool	Pick2	Avg.	Score	SR
Vanilla ( Yang et al., 2024 )	–	44.40	11.10	6.20	15.40	28.60	12.50	21.90	6.70	0.80
Skill-Prompt ∗ ( Yang et al., 2026b )	–	51.70	66.70	48.40	0.00	4.30	10.00	28.90	0.20	0.80
OPSD ( Zhao et al., 2026 )	–	48.80	41.70	16.70	0.00	15.80	16.70	28.10	11.30	3.10
GRPO ( Shao et al., 2024 )	–	91.20	62.50	96.20	61.90	65.00	47.40	75.00	79.80	63.30
Skill-GRPO ( Yang et al., 2026b )	–	88.90	71.40	58.80	70.60	40.70	29.20	60.20	77.30	60.90

Appendix

Table 9: Benchmark results on Qwen2.5-3B-Instruct. Highlighting follows Table 8 . Dashes denote missing entries in this benchmark record; additional WebShop cycles appear in Table 13 .

		ALFWorld ( $\uparrow$ )							WebShop ( $\uparrow$ )
Method	Cycle	Pick	Look	Clean	Heat	Cool	Pick2	Avg.	Score	SR
Vanilla ( Yang et al., 2024 )	–	36.10	22.20	3.10	0.00	0.00	0.00	12.50	5.90	1.60
Skill-Prompt ∗ ( Yang et al., 2026b )	–	51.70	50.00	32.30	5.30	4.30	0.00	23.40	1.70	0.80
OPSD ( Zhao et al., 2026 )	–	50.00	60.00	22.70	21.40	17.60	9.50	32.80	4.50	2.30
GRPO ( Shao et al., 2024 )	–	91.20	87.50	96.20	81.00	65.00	57.90	81.20	80.90	72.60
Skill-GRPO ( Yang et al., 2026b )	–	88.50	66.70	65.20	61.10	57.70	73.10	69.50	80.40	71.90

Appendix

Table 10: Benchmark results on Qwen2.5-7B-Instruct. Blue rows denote SkillCycle at Cycle 0. Bold values mark the best within the OPID reference block. API results: Table 7 .

Condition	Cycle	Pick	Look	Clean	Heat	Cool	Pick2	Overall
GiGPO	1	63.02 [58.07, 67.97]	53.12 [42.97, 63.02]	57.03 [52.08, 61.98]	64.06 [59.11, 69.01]	33.07 [21.09, 45.05]	27.08 [15.10, 39.06]	49.56 [44.56, 54.56]
	2	65.89 [60.94, 71.09]	54.95 [45.05, 65.10]	59.90 [54.95, 65.10]	66.93 [61.98, 71.88]	36.98 [25.00, 48.96]	30.99 [19.01, 42.97]	52.61 [47.61, 57.61]
	3	67.97 [63.02, 72.92]	57.03 [46.88, 66.93]	61.98 [57.03, 66.93]	69.01 [64.06, 73.96]	40.10 [28.12, 52.08]	34.11 [21.88, 46.09]	55.03 [50.03, 60.03]
Static	1	61.46 [56.51, 66.41]	50.00 [40.10, 59.90]	53.91 [48.96, 59.11]	62.50 [57.55, 67.45]	29.43 [17.45, 41.41]	23.96 [11.98, 35.94]	46.88 [41.88, 51.88]
	2	62.50 [57.55, 67.45]	50.78 [40.89, 60.68]	54.69 [49.74, 59.90]	63.28 [58.07, 68.23]	30.47 [18.49, 42.45]	25.00 [13.02, 36.98]	47.79 [42.79, 52.79]
	3	63.02 [58.07, 67.97]	51.56 [41.41, 61.46]	55.47 [50.52, 60.42]	64.06 [59.11, 69.01]	31.51 [19.53, 43.49]	26.04 [14.06, 38.02]	48.61 [43.61, 53.61]

Appendix

Table 11: ALFWorld ablations: Student, Qwen3-1.7B. Cells show SR (%) over reported intervals; Overall is the six-family macro mean. Highlighting and Cycle 3 differences: Appendix K .

Condition	Cycle	Pick	Look	Clean	Heat	Cool	Pick2	Overall
GiGPO	1	67.97 [63.02, 72.92]	58.07 [47.92, 67.97]	61.98 [57.03, 66.93]	67.97 [63.02, 72.92]	40.10 [28.12, 52.08]	34.11 [21.88, 46.09]	55.03 [50.03, 60.03]
	2	71.88 [66.93, 77.08]	60.94 [51.04, 71.09]	65.89 [60.94, 71.09]	71.09 [65.89, 76.04]	45.05 [33.07, 57.03]	38.02 [26.04, 50.00]	58.81 [53.81, 63.81]
	3	73.96 [69.01, 78.91]	63.02 [53.12, 72.92]	67.97 [63.02, 72.92]	72.92 [67.97, 78.12]	47.92 [35.94, 59.90]	41.93 [29.95, 53.91]	61.29 [56.29, 66.29]
Static	1	65.10 [59.90, 70.05]	53.12 [42.97, 63.02]	57.03 [52.08, 61.98]	65.10 [59.90, 70.05]	34.11 [21.88, 46.09]	28.12 [15.89, 40.10]	50.43 [45.43, 55.43]
	2	65.10 [59.90, 70.05]	53.91 [44.01, 64.06]	58.07 [53.12, 63.02]	65.89 [60.94, 71.09]	34.90 [22.92, 46.88]	28.91 [16.93, 40.89]	51.13 [46.13, 56.13]
	3	65.62 [60.42, 70.57]	54.43 [44.53, 64.58]	58.59 [53.39, 63.54]	66.41 [61.46, 71.61]	35.94 [23.96, 47.92]	29.95 [17.97, 41.93]	51.82 [46.82, 56.82]

Appendix

Table 12: ALFWorld ablations: Teacher, Qwen3-1.7B. Cells show SR (%) over reported intervals; Overall is the six-family macro mean. Highlighting and Cycle 3 differences: Appendix K .

Condition	Cycle	Student SR (%)	Teacher SR (%)
GiGPO	1	54.17 [49.17, 59.08]	66.67 [61.81, 71.20]
	2	63.02 [58.09, 67.70]	64.58 [59.68, 69.20]
	3	63.54 [58.62, 68.20]	64.84 [59.94, 69.45]
Static	1	60.42 [55.45, 65.18]	65.89 [61.01, 70.45]
	2	66.67 [61.81, 71.20]	66.67 [61.81, 71.20]
	3	63.28 [58.35, 67.95]	66.41 [61.54, 70.95]

Appendix

Table 13: WebShop ablations, Qwen2.5-3B. SR (%) and source-reported intervals. Student uses no skills; Teacher uses routed skills. Highlighting and Cycle 3 differences follow Appendix K .

Figure 6: Pick2 rule retirement: input prompt, OPD evidence, and paired outcomes.

Figure 21

Task	Seed	No Skill	Router + Skill
A: clean soapbar, place in cabinet	1	F / 30	S / 10
	2	F / 30	S / 11
	3	F / 30	S / 11
B: cool cup, place in microwave	1	F / 30	S / 8
	2	F / 30	S / 8
	3	S / 13	S / 15

Appendix

Table 14: Outcomes and action counts in two routing cases (F: failure; S: success).

Explore similar work

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning
May 7, 2026Yaorui Shi, Yuxin Chen, Zhengxi Lu +6Skills Agentic Reinforcement Learning
University of Science and Technology of China · 2Meituan · 3National University of Singapore +2
Skill-R1: Agent Skill Evolution via Reinforcement Learning
May 10, 2026Yash Vishe, Rohan Surana, Xunyi Jiang +8Self-Evolving Skill Libraries Unsupervised Skill Discovery
UC San Diego · Adobe Research
SkillEvolver: Skill Learning as a Meta-Skill
May 11, 2026Genrui Zhang, Erle Zhu, Jinfeng Zhou +2Skills
2Beijing Jiaotong University · 1Tsinghua University

May 7, 2026, cs.AIJ/K move · Enter open · S save

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao+5

University of Science and Technology of China · 2Meituan · 3National University of Singapore+2

Skills Agentic Reinforcement Learning Language-Model Agents Co-Evolution Evolutionary Optimization Methods

Open paper AskPDF arXiv

A persistent skill library allows language model agents to reuse successful strategies across tasks. Maintaining such a library requires three coupled capabilities. The agent selects a relevant skill, utilizes it during execution, and distills new skills from experience. Existing methods optimize these capabilities in isolation or with separate reward sources, resulting in partial and conflicting evolution. We propose Skill1, a framework that trains a single policy to co-evolve skill selection, utilization, and distillation toward a shared task-outcome objective. The policy generates a query to search the skill library, re-ranks candidates to select one, solves the task conditioned on it, and distills a new skill from the trajectory. All learning derives from a single task-outcome signal. Its low-frequency trend credits selection and its high-frequency variation credits distillation. Experiments on ALFWorld and WebShop show that Skill1 outperforms prior skill-based and reinforcement learning baselines. Training dynamics confirm the co-evolution of the three capabilities, and ablations show that removing any credit signal degrades the evolution.

Abstract

Explore similar work

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

Skill-R1: Agent Skill Evolution via Reinforcement Learning

SkillEvolver: Skill Learning as a Meta-Skill