RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution

Fan Zhang; Hao Wu; Jinyang Wu; Hongyu Qu; Zheng Lian; Xiao Luo; Yefeng Zheng; Bo Zhang

cs.CLOct 11, 2026

RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution

Authors: Fan Zhang, Hao Wu, Jinyang Wu, Hongyu Qu, Zheng Lian, Xiao Luo, Yefeng Zheng, Bo Zhang,

Organizations: CUHK · THU · NUS · Tongji University ·

Abstract

Recent focus in agentic reinforcement learning (RL) seeks to improve credit assignment in long-horizon tasks. One promising direction combines reinforcement learning with verifiable rewards (RLVR) and on-policy self-distillation (OPSD), integrating trajectory-level advantages with dense token-level supervision. This line of work often conditions a teacher model on privileged context, such as agentic skills, to score tokens generated by the student policy. Despite their progress, they largely overlook the on-policy evolution of agentic skills and the reliability of teacher supervision. Towards this end, we introduce Recursive Self-Improving Policy Optimization (RSIPO), a self-evolving agentic RL framework that enables agentic skills and the policy model to co-evolve throughout the training process. Specifically, RSIPO enables the policy itself to extract skills from completed trajectories and explicitly optimizes its skill-generation capability through trajectory replay. It further employs empirical Bayes shrinkage to dynamically calibrate the student's trust in teacher scores, accounting for the fact that skill-enhanced teacher guidance can sometimes be less reliable. Experiments with Qwen-family models on embodied interaction and search-based agentic tasks demonstrate that RSIPO outperforms all the compared baseline approaches. With Qwen2.5-7B as the base model, RSIPO achieves an average success rate of 98.6% on ALFWorld, yielding absolute improvements of 2.5% and 17.4% over the second-best baseline and standard GRPO, respectively.

Figures & tables

Figure 1: Comparison of three paradigms of agentic RL. (a) RLVR-based methods assign equal trajectory-level advantages to all response tokens. (b) OPSD-based methods provide token-level supervision from a self-teacher. (c) Hybrid methods combine the two branches, typically using agentic skills as the teacher’s privileged context.

Figure 2: Preliminary analysis with Qwen2.5-7B-Instruct on ALFWorld. (a) Success rate of the skill generator rollouts on SFT data and of the policy rollouts during RL training. (b) Share of training tasks on which the skill-enhanced teacher succeeds more or less often than the student, grouped by the student’s successes in four separate rollouts on the task. (c) The same shares over RL training, averaged over blocks of 15 steps.

Figure 3: Overview of the proposed RSIPO framework. Replay gains measured by the execution check reward on-policy skill generation and calibrate the student’s trust in the skill-enhanced teacher.

Qwen2.5-7B-Instruct
Method	ALFWorld							Search-Based Tasks
Method	Pick	Look	Clean	Heat	Cool	Pick2	Avg	NQ	Triv	Pop	Hotp	2Wk	MuS	Bam	Avg
Vanilla	36.1	22.2	3.1	0.0	0.0	0.0	12.5	25.2	50.8	29.5	29.0	29.0	10.4	63.7	33.9
Skill-Prompt*	51.7	50.0	32.3	5.3	4.3	0.0	23.4	30.9	52.1	32.7	32.7	27.9	12.7	66.1	36.4
OPSD	50.0	60.0	22.7	21.4	17.6	9.5	32.8	8.8	8.6	17.5	2.5	4.2	0.5	1.2	6.2
GRPO	91.2	87.5	96.2	81.0	65.0	57.9	81.2	45.1	63.7	44.0	43.6	43.2	16.8	37.6	42.0

Table 1: Results on ALFWorld and search-based agentic tasks. We report success rates (%) on ALFWorld and exact-match accuracy (%) on the seven search-based datasets. Methods marked with * indicate they use skills at test time. Within each base model, the best and second-best results are highlighted.

Figure 4: (a) Training dynamics of RSIPO with Qwen2.5-3B-Instruct, where thick lines are 10-step moving averages. (b) Ablation on ALFWorld with Qwen2.5-3B-Instruct, where dots mark the components of each variant.

Figure 5: On-policy skill evolution of Qwen2.5-7B-Instruct on ALFWorld. Left: skill content and replay gain

\Delta_{x}

. Right: condensed skills for tasks that place two credit cards, where step 0 is the SFT policy.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	ALFWorld	Search
RL Training
Training Steps	150	150
Tasks per Step	16	128
Rollouts per Task $G$	8	8
Learning Rate	$10^{-6}$	$10^{-6}$
Warmup Ratio	0	0.1

Appendix

Table 2: Hyperparameters on ALFWorld and search-based tasks. † 4096 for Qwen2.5-7B.

Qwen2.5-3B-Instruct
Method	ALFWorld							Search-Based Tasks
Method	Pick	Look	Clean	Heat	Cool	Pick2	Avg	NQ	Triv	Pop	Hotp	2Wk	MuS	Bam	Avg
Vanilla	44.4	11.1	6.2	15.4	28.6	12.5	21.9	24.6	48.1	31.0	26.3	25.3	7.2	59.7	31.7
Skill-Prompt*	51.7	66.7	48.4	0.0	4.3	10.0	28.9	23.7	46.2	30.6	24.4	22.1	7.5	12.5	23.9
OPSD	48.8	41.7	16.7	0.0	15.8	16.7	28.1	0.1	0.1	0.1	0.0	0.0	0.0	0.0	0.0
GRPO	91.2	62.5	96.2	61.9	65.0	47.4	75.0	39.3	60.6	41.1	37.4	34.6	15.4	26.4	36.4

Appendix

Table 3: Results on ALFWorld and search-based agentic tasks with Qwen2.5-3B-Instruct. We report success rates (%) on ALFWorld and exact-match accuracy (%) on the seven search datasets. Methods marked with * use skills at test time. The best and second-best results are highlighted.

Figure 6: Training dynamics of RSIPO with (a) Qwen2.5-7B and (b) Qwen3-1.7B on both benchmarks. Thin lines show per-step values and thick lines show 10-step moving averages.

Figure 7: Task-execution prompts on (a) ALFWorld and (b) search-based tasks. The dashed block is inserted only for the skill-enhanced teacher, and the history is omitted at the first turn.

Figure 8: Skill-generation prompt. (a) The analysis template, whose instruction for the episode skill depends on the outcome of the trajectory. (b) Additions on search-based tasks.

Figure 9: A helpful skill written by Qwen2.5-7B for a pick-two task on ALFWorld.

Figure 10: A helpful skill written by Qwen2.5-7B for a 2WikiMultihopQA question on search-based tasks.

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 232 papers

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL
Jun 1, 2026Zelin He, Haotian Lin, Boran Han +6Reinforcement Learning LLM Agent Skill Learning
The Pennsylvania State University · † Work done during an internship at Amazon · Amazon IntelliHub +1
UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation
Jun 28, 2026Songjun Tu, Chengdong Xu, Qichao Zhang +6Credit Assignment in RL LLM Agent Skill Learning
Institute of Automation, Chinese Academy of Sciences · Pengcheng Laboratory · Memorax AI
Group-Reflective Self-Distillation for Agentic Reinforcement Learning
Jul 30, 2026Binbin Zheng, Zijun Xie, Guanqun Zhao +4Agentic RL Credit Assignment in RL
University of Science and Technology of China · 2Baidu Inc. · 3Peking University +3

Jun 1, 2026J/K move · Enter open · S save

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL

Zelin He, Haotian Lin, Boran Han, Wei Zhu+5

The Pennsylvania State University · † Work done during an internship at Amazon · Amazon IntelliHub+1

Reinforcement Learning LLM Agent Skill Learning RL for LLM Agents

Open paper AskPDF arXiv

Agentic reinforcement learning (RL) enables LLM agents to improve continuously from environment rewards, yet the resulting policies do not systematically accumulate reusable strategies that generalize across tasks. Modular skills can provide such reusable strategies, yet existing skill-augmented RL methods decouple skill creation from policy optimization, risking adopting skills that conflict with the evolving policy. Inspired by Anthropic's Skill Creator, we introduce ReSkill, an RL-in-the-loop skill creation framework that reconciles skill evolution with policy learning. ReSkill exploits the group-wise structure of GRPO to naturally embed three mechanisms with only marginal additional overhead: (1) an assertion-driven skill creator that diagnoses failures from past experience and proposes conditional, trigger-based skill revisions; (2) within-group rollout sampling that enables controlled comparison of skill versions, capturing which version best supports the policy's ongoing learning; and (3) Thompson Sampling with adaptive discounting to balance exploration and exploitation in skill version selection as the policy evolves. Across several domains, ReSkill consistently outperforms existing memory and skill-based RL methods, with the largest gains on unseen tasks. Analysis of the skill lifecycle shows skills being automatically created, tested, refined, and pruned as the policy improves, demonstrating reconciled skill-policy co-evolution.

Abstract

Agentic RL

Explore similar work

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL

UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation

Group-Reflective Self-Distillation for Agentic Reinforcement Learning