Recent focus in agentic reinforcement learning (RL) seeks to improve credit assignment in long-horizon tasks. One promising direction combines reinforcement learning with verifiable rewards (RLVR) and on-policy self-distillation (OPSD), integrating trajectory-level advantages with dense token-level supervision. This line of work often conditions a teacher model on privileged context, such as agentic skills, to score tokens generated by the student policy. Despite their progress, they largely overlook the on-policy evolution of agentic skills and the reliability of teacher supervision. Towards this end, we introduce Recursive Self-Improving Policy Optimization (RSIPO), a self-evolving agentic RL framework that enables agentic skills and the policy model to co-evolve throughout the training process. Specifically, RSIPO enables the policy itself to extract skills from completed trajectories and explicitly optimizes its skill-generation capability through trajectory replay. It further employs empirical Bayes shrinkage to dynamically calibrate the student's trust in teacher scores, accounting for the fact that skill-enhanced teacher guidance can sometimes be less reliable. Experiments with Qwen-family models on embodied interaction and search-based agentic tasks demonstrate that RSIPO outperforms all the compared baseline approaches. With Qwen2.5-7B as the base model, RSIPO achieves an average success rate of 98.6% on ALFWorld, yielding absolute improvements of 2.5% and 17.4% over the second-best baseline and standard GRPO, respectively.
Figures & tables
Figure 1: Comparison of three paradigms of agentic RL. (a) RLVR-based methods assign equal trajectory-level advantages to all response tokens. (b) OPSD-based methods provide token-level supervision from a self-teacher. (c) Hybrid methods combine the two branches, typically using agentic skills as the teacher’s privileged context.
Figure 2: Preliminary analysis with Qwen2.5-7B-Instruct on ALFWorld. (a) Success rate of the skill generator rollouts on SFT data and of the policy rollouts during RL training. (b) Share of training tasks on which the skill-enhanced teacher succeeds more or less often than the student, grouped by the student’s successes in four separate rollouts on the task. (c) The same shares over RL training, averaged over blocks of 15 steps.
Figure 3: Overview of the proposed RSIPO framework. Replay gains measured by the execution check reward on-policy skill generation and calibrate the student’s trust in the skill-enhanced teacher.
Method
ALFWorld
Search-Based Tasks
Pick
Look
Clean
Heat
Cool
Pick2
Avg
NQ
Triv
Pop
Hotp
2Wk
MuS
Bam
Avg
Qwen2.5-7B-Instruct
Vanilla
36.1
22.2
3.1
0.0
0.0
0.0
12.5
25.2
50.8
29.5
29.0
29.0
10.4
63.7
33.9
Skill-Prompt*
51.7
50.0
32.3
5.3
4.3
0.0
23.4
30.9
52.1
32.7
32.7
27.9
12.7
66.1
36.4
OPSD
50.0
60.0
22.7
21.4
17.6
9.5
32.8
8.8
8.6
17.5
2.5
4.2
0.5
1.2
6.2
GRPO
91.2
87.5
96.2
81.0
65.0
57.9
81.2
45.1
63.7
44.0
43.6
43.2
16.8
37.6
42.0
Table 1: Results on ALFWorld and search-based agentic tasks. We report success rates (%) on ALFWorld and exact-match accuracy (%) on the seven search-based datasets. Methods marked with * indicate they use skills at test time. Within each base model, the best and second-best results are highlighted.
Figure 4: (a) Training dynamics of RSIPO with Qwen2.5-3B-Instruct, where thick lines are 10-step moving averages. (b) Ablation on ALFWorld with Qwen2.5-3B-Instruct, where dots mark the components of each variant.
Figure 5: On-policy skill evolution of Qwen2.5-7B-Instruct on ALFWorld. Left: skill content and replay gain Δx . Right: condensed skills for tasks that place two credit cards, where step 0 is the SFT policy.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
ALFWorld
Search
RL Training
Training Steps
150
150
Tasks per Step
16
128
Rollouts per Task G
8
8
Learning Rate
10−6
10−6
Warmup Ratio
0
0.1
Appendix
Table 2: Hyperparameters on ALFWorld and search-based tasks. † 4096 for Qwen2.5-7B.
Method
ALFWorld
Search-Based Tasks
Pick
Look
Clean
Heat
Cool
Pick2
Avg
NQ
Triv
Pop
Hotp
2Wk
MuS
Bam
Avg
Qwen2.5-3B-Instruct
Vanilla
44.4
11.1
6.2
15.4
28.6
12.5
21.9
24.6
48.1
31.0
26.3
25.3
7.2
59.7
31.7
Skill-Prompt*
51.7
66.7
48.4
0.0
4.3
10.0
28.9
23.7
46.2
30.6
24.4
22.1
7.5
12.5
23.9
OPSD
48.8
41.7
16.7
0.0
15.8
16.7
28.1
0.1
0.1
0.1
0.0
0.0
0.0
0.0
0.0
GRPO
91.2
62.5
96.2
61.9
65.0
47.4
75.0
39.3
60.6
41.1
37.4
34.6
15.4
26.4
36.4
Appendix
Table 3: Results on ALFWorld and search-based agentic tasks with Qwen2.5-3B-Instruct. We report success rates (%) on ALFWorld and exact-match accuracy (%) on the seven search datasets. Methods marked with * use skills at test time. The best and second-best results are highlighted.
Figure 6: Training dynamics of RSIPO with (a) Qwen2.5-7B and (b) Qwen3-1.7B on both benchmarks. Thin lines show per-step values and thick lines show 10-step moving averages.
Figure 7: Task-execution prompts on (a) ALFWorld and (b) search-based tasks. The dashed block is inserted only for the skill-enhanced teacher, and the history is omitted at the first turn.
Figure 8: Skill-generation prompt. (a) The analysis template, whose instruction for the episode skill depends on the outcome of the trajectory. (b) Additions on search-based tasks.
Figure 9: A helpful skill written by Qwen2.5-7B for a pick-two task on ALFWorld.
Figure 10: A helpful skill written by Qwen2.5-7B for a 2WikiMultihopQA question on search-based tasks.
Agentic reinforcement learning (RL) enables LLM agents to improve continuously from environment rewards, yet the resulting policies do not systematically accumulate reusable strategies that generalize across tasks. Modular skills can provide such reusable strategies, yet existing skill-augmented RL methods decouple skill creation from policy optimization, risking adopting skills that conflict with the evolving policy. Inspired by Anthropic's Skill Creator, we introduce ReSkill, an RL-in-the-loop skill creation framework that reconciles skill evolution with policy learning. ReSkill exploits the group-wise structure of GRPO to naturally embed three mechanisms with only marginal additional overhead: (1) an assertion-driven skill creator that diagnoses failures from past experience and proposes conditional, trigger-based skill revisions; (2) within-group rollout sampling that enables controlled comparison of skill versions, capturing which version best supports the policy's ongoing learning; and (3) Thompson Sampling with adaptive discounting to balance exploration and exploitation in skill version selection as the policy evolves. Across several domains, ReSkill consistently outperforms existing memory and skill-based RL methods, with the largest gains on unseen tasks. Analysis of the skill lifecycle shows skills being automatically created, tested, refined, and pruned as the policy improves, demonstrating reconciled skill-policy co-evolution.
Zelin He, Haotian Lin, Boran Han +6
The Pennsylvania State University · † Work done during an internship at Amazon · Amazon IntelliHub +1
Skill memories can improve agentic reinforcement learning by reusing past experience as textual guidance, but retrieved skills are not oracular: they may help in one state while misleading the same policy in another. This makes the common privileged-teacher assumption fragile, namely that a skill-conditioned prompt can be treated as a fixed teacher for the no-skill prompt. We introduce UCOB, a framework for learning to utilize and evolve agentic skills via credit-aware on-policy bidirectional self-distillation. UCOB treats skill-conditioned and no-skill prompts as two on-policy context views of the same model, compares their return-to-go within the same task and anchor state, and uses the higher-return view as the local teacher. This local credit signal internalizes useful skill-conditioned behavior, corrects misleading skill usage, and guides task/state skill memory updates, utility-aware retrieval, and reflection self-training. Experiments on agentic tasks, including ALFWorld, WebShop, and Search-QA, show that UCOB outperforms skill-free RL, skill-memory baselines, and self-distillation methods across model scales, with up to 23.5 and 18.0 point gains over SOTA baselines on ALFWorld and WebShop. Ablations and analyses further validate its core mechanisms and efficiency.
Songjun Tu, Chengdong Xu, Qichao Zhang +6
Institute of Automation, Chinese Academy of Sciences · Pengcheng Laboratory · Memorax AI
Reinforcement learning with verifiable rewards (RLVR) is effective for training large language model agents. However, terminal rewards provide only coarse trajectory-level supervision, leaving successful behaviors, recurring mistakes, and incidental choices entangled in the same outcome signal. Existing agentic self-distillation methods enrich sparse supervision with natural-language skills, but skills retrieved externally or extracted from a single trajectory by stronger models may mismatch current experience, exceed the policy's capability, or remain path-specific. We propose Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts. For each prompt, the policy reflects on each verified trajectory in an on-policy group, and a stop-gradient snapshot contrasts the resulting reflections from successful and failed rollouts to construct group-level privileged guidance. Conditioned on this guidance, a self-teacher refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction. Experiments across multiple agentic environments and model scales demonstrate that GRSD consistently outperforms competitive baselines and generalizes more effectively to unseen tasks.
Binbin Zheng, Zijun Xie, Guanqun Zhao +4
University of Science and Technology of China · 2Baidu Inc. · 3Peking University +3