SkillScaffold: Scaffolding Agentic Reinforcement Learning with Policy-Adaptive Skill Credit Assignment and Selective Injection

Wenlin Zhong; Hao Wang; Kun Yuan; Fan Yang; Jinluan Yang; Dian Jin; Xu Tan; Yiquan Wu

cs.LGOct 13, 2026

SkillScaffold: Scaffolding Agentic Reinforcement Learning with Policy-Adaptive Skill Credit Assignment and Selective Injection

Authors: Wenlin Zhong, Hao Wang, Kun Yuan, Fan Yang, Jinluan Yang, Dian Jin, Xu Tan, Yiquan Wu,

Organizations: Zhejiang University · Kuaishou Technology · University of the Chinese Academy of Sciences · Zhejiang University of Science and Technology

Abstract

Agentic reinforcement learning (RL) enables large language models (LLMs) to learn from multi-turn interactions with verifiable outcomes. Textual skills can improve exploration by distilling prior interactions into reusable guidance. However, existing skill-based RL approaches often either assume skill injection is beneficial or rely on indirect feedback or rollout-intensive evaluation to assess its utility. Our pilot studies show that rollout-based utility estimation for skill credit assignment is costly and unstable under limited sampling, while skill benefits vary across candidates and depend on the current policy's competence on each query. To address these challenges, we introduce SkillScaffold, a policy-adaptive training scaffold that combines low-cost skill-utility estimation with selective skill injection. By reusing outcome-verified training trajectories, SkillScaffold generates candidate skills and estimates their utility through outcome-aware likelihood comparisons, without candidate-specific environment rollouts. The resulting utility guides whether and which skill to inject, retaining skill-free execution when no candidate is estimated to help. Joint training enables the Generator to produce skills tailored to the evolving Executor. Scaffolded RL encourages skill internalization for skill-free inference. Experiments on ALFWorld, WebShop, ScienceWorld, and $τ^2$ -Bench demonstrate gains over strong agentic-RL and skill-based RL baselines across both short- and long-horizon tasks. Ablations and further analyses support the importance of utility-based selection and abstention. Our code is available at https://github.com/Vaidurya00/SkillScaffold.

Figures & tables

Figure 1: Overview and empirical summary of SkillScaffold . Low-cost, policy-relative skill-utility estimates guide selective skill injection during training, while inference remains skill-free. Across four agentic benchmarks, SkillScaffold consistently outperforms GRPO and SkillRL while injecting skills for only a small fraction of training queries.

Figure 2: Rollout-based utility estimation is costly and unstable. (a) Sign disagreement between independent estimates. (b) More rollout samples distinguish more candidate groups.

Figure 3: Skill utility is query- and policy-dependent. (a) Candidate effects vary within a query. (b) Skill benefit decreases as skill-free policy competence improves.

Figure 4: Overview of SkillScaffold . The five stages show skill-free scouting, outcome-anchored generation, utility-based selection, adaptive injection, and joint policy optimization.

	WebShop										ALFWorld
	ID					OOD				Avg.	Task Type						Avg.
Method	App.	Ele.	Foo.	Oth.	ID	Acc.	Bea.	Hom.	OOD		Pic.	Hea.	Coo.	Cle.	Loo.	Pk2.
ReAct	18.8	16.9	18.1	17.0	17.6	14.1	16.0	8.6	12.6	16.1	59.4	32.1	41.1	29.8	41.0	5.2	35.6
Prompt-based Agentic / Memory-based Methods
Reflexion	22.7	19.5	20.8	21.1	20.9	16.7	24.5	13.0	17.5	19.8	59.9	21.7	42.9	41.5	38.9	25.7	39.3
Mem0	18.3	15.7	14.5	17.0	16.4	13.3	15.3	8.6	12.1	15.1	37.0	25.0	41.7	41.9	30.6	29.4	35.0

Table 1: Performance on WebShop and ALFWorld with Qwen3-8B. We report success rates (%) across task categories and their averages.

\dagger

denotes evaluation with skills, while

*

denotes evaluation without skills. Best and second-best results are marked accordingly.

	ScienceWorld			$\tau^{2}$ -Bench
Method	ID	OOD	Avg.	Airline	Retail	Telecom	Avg.
ReAct	19.3	11.1	16.0	17.8	35.5	7.6	20.3
SFT	–	–	–	41.0	40.1	56.6	45.9
GRPO	92.6	43.6	72.9	45.8	57.2	77.0	60.0
SkillRL	88.2	47.7	71.9	46.0	57.9	75.7	59.9
SkillScaffold	96.2 $\uparrow_{3.6}$	48.8 $\uparrow_{1.1}$	77.1 $\uparrow_{4.2}$	46.0	60.8 $\uparrow_{2.9}$	83.3 $\uparrow_{6.3}$	63.4 $\uparrow_{3.4}$

Table 2: Performance on ScienceWorld and

\tau^{2}

-Bench with Qwen3-8B. We report success rates (%) on the ScienceWorld ID/OOD splits and the three

\tau^{2}

-Bench domains. Teal arrows indicate absolute gains in percentage points over the strongest competing result. Best and second-best results are marked accordingly.

Variant	ALF.	Web.
SkillScaffold (Full)	94.8	45.1
Generator Learning
w/o Generator learning	89.6	39.5
No Gen. loss + random selection	81.3	40.3
Utility Selection
Random selection	79.9	39.2

Table 3: Ablation results on ALFWorld and WebShop.

Figure 6: ALFWorld exploration dynamics. Scout/Post-Scout success (top) and skill-injection rates (bottom), grouped by all-wrong, mixed, and all-correct Scout outcomes.

Benchmark	$N$	Recovery (pp)
WebShop	19	$+19.7\ [0.0,\ 39.5]$
ALFWorld	54	$+29.6\ [20.4,\ 38.9]$
ScienceWorld	50	$+65.0\ [56.0,\ 73.5]$
$\tau^{2}$ -Bench	98	$+16.6\ [10.7,\ 22.5]$

Table 4: Skill-free evidence of internalization. Later-minus-early skill-free success on fixed early skill-assisted training cohorts, with 95% bootstrap intervals.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Reinforcement learning
Parameter	ALFWorld	WebShop	ScienceWorld	$\tau^{2}$ -Bench
Model family	Qwen3-8B	Qwen3-8B	Qwen3-8B	Qwen3-8B
Training steps	100	200	160	200
Optimizer	Adam	Adam	Adam	Adam
Learning rate (constant)	$10^{-6}$	$10^{-6}$	$10^{-6}$	$10^{-6}$
Adam $(\beta_{1},\beta_{2})$	$(0.9,0.98)$	$(0.9,0.98)$	$(0.9,0.98)$	$(0.9,0.999)$

Appendix

Table 5: Training configurations for the reported SkillScaffold models.

	Airline			Retail			Telecom			Average
Method	P@1	P@4	P@8	P@1	P@4	P@8	P@1	P@4	P@8	P@1	P@4	P@8
ReAct	17.8	34.0	46.0	35.5	61.7	70.2	7.6	23.1	32.5	20.3	39.6	49.5
SFT	41.0	62.1	74.0	40.1	71.3	84.2	56.6	91.8	97.4	45.9	75.1	85.2
GRPO	45.8	63.8	70.0	57.2	85.4	92.1	77.0	97.2	99.1	60.0	82.1	87.1
SkillRL	46.0	64.2	70.0	57.9	83.6	90.4	75.7	95.3	99.1	59.9	81.0	86.5
SkillScaffold	46.0	70.3 $\uparrow_{6.1}$	78.0 $\uparrow_{4.0}$	60.8 $\uparrow_{2.9}$	85.6 $\uparrow_{0.2}$	91.2	83.3 $\uparrow_{6.3}$	98.5 $\uparrow_{1.3}$	100.0 $\uparrow_{0.9}$	63.4 $\uparrow_{3.4}$	84.8 $\uparrow_{2.7}$	89.7 $\uparrow_{2.6}$

Appendix

Table 6: Full

\tau^{2}

-Bench Pass@

k

results (%) with Qwen3-8B. Average columns are macro averages over Airline, Retail, and Telecom. Teal arrows indicate absolute gains in percentage points over the strongest competing result.

Figure 7: Training success-rate curves across benchmarks. GRPO, SkillRL, and SkillScaffold on (a) WebShop, (b) ALFWorld, (c) $\tau^{2}$ -Bench Airline, (d) Retail, (e) Telecom, and (f) the $\tau^{2}$ -Bench macro average. Dark curves show smoothed trends and faint curves show raw measurements.

Panel of Figure 8 — Figure 8: Training-time skill injection and selection rates. Panels (a)–(b) report skill injection on WebShop and ScienceWorld; panel (c) reports positive-utility selection on $\tau^{2}$ -Bench.

Figure 9: Selective anchor-guided exploration on WebShop. Using a verified successful Scout as evidence, the anchor-based variant selectively guides mixed groups and maintains strong Post-Scout performance as the Scout policy approaches the all-correct regime.

Figure 10: Corrective group-guided exploration on WebShop. Outcome-labeled Scout groups support corrective skill generation and produce persistent nonzero Post-Scout success for initially all-wrong queries. Skill activation concentrates on all-wrong and mixed groups and naturally diminishes once all Scouts succeed.

Figure 11: Competence-adaptive exploration on ScienceWorld. Group-based scaffolding creates successful trajectories for initially all-wrong groups and improves mixed-group performance during later training. Positive-utility candidates become less frequent as the Scout policy improves, reflecting competence-adaptive skill use.

Figure 12: Competence-adaptive exploration on $\tau^{2}$ -Bench. Post-Scout rollouts recover successful trajectories from initially all-wrong groups and improve performance across much of the mixed regime. Positive-utility candidates are frequent for all-wrong groups, selective for mixed groups, and unnecessary once all Scouts succeed.

Figure 13: From skill-guided exploration to skill-free execution. An ALFWorld example in which a selected skill introduces the missing clean-before-place strategy, which is later reproduced without skill input.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,739 papers

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL
Jun 1, 2026Zelin He, Haotian Lin, Boran Han +6Reinforcement Learning LLM Agent Skill Learning
The Pennsylvania State University · † Work done during an internship at Amazon · Amazon IntelliHub +1
Skill-R1: Agent Skill Evolution via Reinforcement Learning
May 10, 2026Yash Vishe, Rohan Surana, Xunyi Jiang +8Agent Skill Learning RL for LLM Agents
UC San Diego · Adobe Research
Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning
May 11, 2026Junhao Shen, Teng Zhang, Xiaoyan Zhao +1Agentic RL LLM Agent Skill Learning
Database Group, The Chinese University of Hong Kong · Lanzhou University

Jun 1, 2026J/K move · Enter open · S save

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL

Zelin He, Haotian Lin, Boran Han, Wei Zhu+5

The Pennsylvania State University · † Work done during an internship at Amazon · Amazon IntelliHub+1

Reinforcement Learning LLM Agent Skill Learning RL for LLM Agents

Open paper AskPDF arXiv

Agentic reinforcement learning (RL) enables LLM agents to improve continuously from environment rewards, yet the resulting policies do not systematically accumulate reusable strategies that generalize across tasks. Modular skills can provide such reusable strategies, yet existing skill-augmented RL methods decouple skill creation from policy optimization, risking adopting skills that conflict with the evolving policy. Inspired by Anthropic's Skill Creator, we introduce ReSkill, an RL-in-the-loop skill creation framework that reconciles skill evolution with policy learning. ReSkill exploits the group-wise structure of GRPO to naturally embed three mechanisms with only marginal additional overhead: (1) an assertion-driven skill creator that diagnoses failures from past experience and proposes conditional, trigger-based skill revisions; (2) within-group rollout sampling that enables controlled comparison of skill versions, capturing which version best supports the policy's ongoing learning; and (3) Thompson Sampling with adaptive discounting to balance exploration and exploitation in skill version selection as the policy evolves. Across several domains, ReSkill consistently outperforms existing memory and skill-based RL methods, with the largest gains on unseen tasks. Analysis of the skill lifecycle shows skills being automatically created, tested, refined, and pruned as the policy improves, demonstrating reconciled skill-policy co-evolution.

Abstract

Reinforcement Learning

Explore similar work

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL

Skill-R1: Agent Skill Evolution via Reinforcement Learning

Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning