Write, Execute, Refine: From Skill Follower to Skill Optimizer via Reinforcement Learning from Execution Feedback

Kang Peng; Zhiwei Zhang; Yichen Zhang; Zezhong Wang; Yiming Du; Geng Tu; Baojun Wang; Bin Liang

cs.CLAug 18, 2026

Write, Execute, Refine: From Skill Follower to Skill Optimizer via Reinforcement Learning from Execution Feedback

Authors: Kang Peng, Zhiwei Zhang, Yichen Zhang, Zezhong Wang, Yiming Du, Geng Tu, Baojun Wang, Bin Liang,

Organizations: Harbin Institute of Technology, Shenzhen, China · The Chinese University of Hong Kong · MoE Key Laboratory of High Confidence Software Technologies · Harbin Institute of Technology, Harbin, China ·

Abstract

Expert-written natural language skills can improve tool-using agents, yet agent-authored skills perform 8--11 points worse than using no skill. While inference-time loops can repair individual skills per task, they leave the underlying generator's refinement capability unimproved. We study how to organize execution experience from intermediate skills into training states for an optimizer. We introduce WER(Write, Execute, and Refine), a multi-phase framework that trains a Skill Optimizer outside a frozen executor. The optimizer proposes skills, a frozen agent executes each repeatedly, and a programmatic verifier scores the outcomes. These scores serve as group-relative rewards and filter for informative, mixed-outcome instances. Matched successful and failed trajectories from these records form the next phase's refinement states, so the optimizer learns from the consequences of its earlier outputs. On BFCL v4 multi-turn and tau2-bench, WER improves average Pass@1 over the no-skill baseline by 7.80 and 6.06 points, respectively. Under an identical refinement workflow, it outperforms the same backbone without optimizer training by 10.02 and 12.50 points. Notably, our trained 4B optimizer achieves 76.63% on BFCL v4, outperforming all evaluated off-the-shelf general-purpose LLMs in the same role on average. Our code is available at https://github.com/littlepkk/wer-training.

Figures & tables

Figure 1: The Skill Optimizer remains outside the environment and observes execution. It writes a skill for a frozen agent and revises the skill from the resulting trajectory.

Figure 2: Training loop. A refinement state is assembled from the task context, interaction history, and current skill, from which the Skill Optimizer samples $K$ candidate skills (§ 3.1 ). A frozen skill-conditioned agent executes each candidate in the sandbox, producing trajectories and verifier outcomes (§ 3.2 ). The outcomes provide group-relative advantages for updating $\pi_{\theta}$ (§ 3.3 ), while the skills and trajectories enter the experience buffer to assemble next-phase states (§ 3.4 ). As illustrated on the left, retained skills seed the candidates of the next round, forming a revision tree across phases. For clarity, the figure shows binary verifier rewards; the actual reward also includes the format and length terms in Eq. 4 .

Method	BFCL v4 Multi-Turn					$\tau^{2}$ -bench
Method	File System	Trading	Travel	Vehicle	Avg. $\uparrow$	Airline	Retail	Telecom	Avg. $\uparrow$
No Skill	68.42	86.84	62.16	57.89	68.83	45.00	64.84	30.77	46.87
GPT-5.1 Seed Skill	63.16	78.94	62.16	64.86	67.28	47.50	63.73	31.86	47.70
Qwen3-4B as Skill Optimizer	57.89	84.21	59.46	64.86	66.61	40.00	60.43	20.87	40.43
Skill-R1 Vishe et al. (2026)	71.05	84.21	64.86	67.56	71.92	47.50	56.04	20.87	41.47
Trace2Skill Ni et al. (2026)	63.16	87.55	67.57	75.68	73.49	52.50	58.34	29.67	46.84

Table 1: Main results on BFCL v4 and

\tau^{2}

-bench (Pass@1, %). Best results are in bold .

Phase	File Sys.	Trading	Travel	Vehicle	Avg. $\uparrow$
1	60.53	76.32	62.16	78.38	69.35
2	73.68	81.58	59.64	70.27	71.29
3	71.05	86.84	70.27	78.38	76.63

Table 2: Phase-wise BFCL v4 performance (Pass@1, %). Best results are in bold .

Figure 3: BFCL v4 performance trends across refinement depths.

Depth	File Sys.	Trading	Travel	Vehicle	Avg. $\uparrow$
0	63.16	78.94	62.16	64.86	67.28
1	60.53	86.84	67.57	67.57	70.67
2	71.05	86.84	70.27	78.38	76.63
3	68.42	89.47	67.57	75.68	75.33

Table 3: BFCL v4 performance across refinement depths (Pass@1, %). Best results are in bold .

Figure 4: Skill-optimizer backbones on BFCL v4 (Pass@1, %). The WER-trained Qwen3-4B optimizer has the highest average despite its smaller size. Per-domain results are reported in Table 4 (Appendix B.4 ).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Optimizer	File Sys.	Trading	Travel	Vehicle	Avg. $\uparrow$
DeepSeek-V4-Flash	$68.42$	$84.21$	$67.56$	$70.27$	$72.61$
GPT-5.5	$69.74$	89.48	$63.52$	78.38	$75.28$
Gemini 3.5 Flash	$60.53$	89.48	$64.86$	$72.97$	$71.96$
Claude Sonnet 4.6	$68.42$	$84.21$	$54.05$	$72.97$	$69.91$
WER (Qwen3-4B)	71.05	$86.84$	70.27	78.38	76.63

Appendix

Table 4: BFCL v4 performance with different skill optimizers (Pass@1, %). Best results are in bold .

Merging Model	File Sys.	Trading	Travel	Vehicle	Avg. $\uparrow$
DeepSeek-V4-Flash	$71.05$	86.84	$67.57$	$70.27$	$73.93$
GPT-4o	$63.16$	$84.21$	72.97	$75.68$	$74.00$
Qwen3-4B	73.68	$78.95$	$67.57$	78.38	$74.64$
GPT-5.5	$71.05$	86.84	$70.27$	78.38	76.63

Appendix

Table 5: BFCL v4 performance with different skill-merging models (Pass@1, %). All settings use the WER-trained Qwen3-4B optimizer. The GPT-5.5 row reproduces the WER results in Table 1 . Best results are in bold .

Figure 5: Token consumption of SkillOpt and WER across the four BFCL v4 domains. Optimization tokens are shown on the left axis and execution tokens on the right axis. Both axes report millions of tokens.

Figure 6: Three-stage skill refinement on a representative BFCL v4 multi-turn task.

Figure 7: Prompt used by the skill optimizer to revise a skill from successful and failed execution trajectories.

Figure 8: Prompt used to consolidate a set of related skills into a single reusable skill.

LLM Agent Skill Learning

Large language model agent skill learning studies how agents acquire, represent, reuse, and refine procedures for multi-step tasks.

See all 398 papers

SkillOpt: Executive Strategy for Self-Evolving Agent Skills
May 22, 2026Yifan Yang, Ziyang Gong, Weiquan Huang +12Large Language Model-Guided Optimization LLM Agent Self-Improvement
Microsoft · Shanghai Jiao Tong University · Tongji University +1
Skill-R1: Agent Skill Evolution via Reinforcement Learning
May 10, 2026Yash Vishe, Rohan Surana, Xunyi Jiang +8Agent Skill Learning RL for LLM Agents
UC San Diego · Adobe Research
SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement
Jun 9, 2026Srishti Gautam, Arjun Radhakrishna, Sumit GulwaniLLM Self-Refinement LLM Agent Self-Improvement
Microsoft

May 22, 2026J/K move · Enter open · S save

SkillOpt: Executive Strategy for Self-Evolving Agent Skills

Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang+11

Microsoft · Shanghai Jiao Tong University · Tongji University+1

Large Language Model-Guided Optimization LLM Agent Self-Improvement Agent Skill Learning

Open paper AskPDF arXiv

Agent skills today are hand-crafted, generated one-shot, or evolved through loosely controlled self-revision, none of which behaves like a deep-learning optimizer for the skill, and none of which reliably improves over its starting point under feedback. We argue the skill should instead be trained as the external state of a frozen agent, with the same discipline that makes weight-space optimization reproducible. SkillOpt is, to our knowledge, the first systematic controllable text-space optimizer for agent skills: a separate optimizer model turns scored rollouts into bounded add/delete/replace edits on a single skill document, and an edit is accepted only when it strictly improves a held-out validation score. A textual learning-rate budget, rejected-edit buffer, and epoch-wise slow/meta update make skill training stable while adding zero inference-time model calls at deployment. Across six benchmarks, seven target models, and three execution harnesses (direct chat, Codex, Claude Code), SkillOpt is best or tied on all 52 evaluated (model, benchmark, harness) cells and beats every per-cell competitor among human, one-shot LLM, Trace2Skill, TextGrad, GEPA, and EvoSkill skills. On GPT-5.5 it lifts the average no-skill accuracy by +23.5 points in direct chat, by +24.8 inside the Codex agentic loop, and by +19.1 inside Claude Code. Transfer experiments further show that optimized skill artifacts retain value when moved across model scales, between Codex and Claude Code execution environments, and to a nearby math benchmark without further optimization. Code: https://aka.ms/skillopt

Abstract

LLM Agent Skill Learning

Explore similar work

SkillOpt: Executive Strategy for Self-Evolving Agent Skills

Skill-R1: Agent Skill Evolution via Reinforcement Learning

SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement