AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization
Authors: Huizu Lin, Chengkai Huang, Tianqi Gao, Tao Huang, Daijiao Liu, Tongxin Li, Xiaoyan Sun, Lina Yao
Organizations: University of New South Wales Sydney, Australia · University of Science and Technology of China, Hefei Comprehensive National Science Center Hefei, China
Skills play different roles as an agent's policy evolves: they should first provide learnable knowledge, then support capability formation, and finally be invoked only when they improve individual decisions. Existing methods rarely model this lifecycle. They either keep skills outside the model, fully internalize them, or select among internalization and utilization objectives through noisy task-level success rates. Such designs fragment training and assign uniform importance to actions within the same trajectory, even though skill guidance may help some decisions while distracting others. To solve these problems, we introduce AUSO (Action-level Unified Skill Optimization), which unifies skill learning and skill use through a progressive, action-aware optimization process. At the beginning of training, AUSO jointly learns from teacher guidance and environmental outcomes, enabling the policy to acquire foundational skills without losing task-oriented feedback. It subsequently emphasizes outcome-based policy optimization to consolidate autonomous problem-solving ability. As the policy matures, AUSO evaluates each sampled action under both skill-conditioned and skill-free contexts. The resulting action-level information signal is coupled with the trajectory outcome advantage, allowing beneficial skill-sensitive actions to receive stronger updates and harmful ones to be suppressed. Therefore, skills gradually transition from an external source of supervision into decision knowledge whose utilization is adapted to its action-level benefit, while reinforcement learning remains the shared backbone across all stages. Experiments on ALFWorld, WebShop, and SearchQA show that AUSO consistently improves agent performance and out-of-distribution generalization over competitive baselines.
Figures & tables
Figure 1. Empirical evidence of ambiguous medium/easy skill-routing boundaries in Skill0.5. Samples across the success threshold can differ by as little as 1/8 , revealing the ambiguity of threshold-based routing.
Figure 2. Overview of Our AUSO (Action-Level Unified Skill Optimization from Internalization to Utilization): (1) General Skills Internalization: Internalize transferable general skills through action-level teacher–student guidance. (2) Skills Exploration: Explore and optimize diverse skills via GRPO. (3) Specific Skill Utilization: Evaluate the benefit of task-specific skills for each action to guide action-level skill utilization.
Method
WebShop
ALFWorld
ID
OOD
ID
OOD
App.
Elec.
Foot.
Other
Avg.
Access.
Beauty
Home
Avg.
Pick
Cool
Clean
Avg.
Look
Heat
Pick2
Avg.
Prompt-based Methods
Zero-shot
4.4
4.6
3.4
1.0
3.5
2.5
3.7
5.5
3.9
28.6
12.0
18.5
20.7
38.5
12.5
12.5
18.9
Few-shot
14.2
15.1
13.5
24.0
16.5
18.8
29.6
5.5
16.9
62.9
44.0
63.0
57.5
46.2
31.2
8.3
24.5
ReAct
12.4
11.1
4.5
12.0
10.4
11.2
24.1
2.7
11.6
71.4
28.0
33.3
47.1
46.2
18.8
12.5
22.6
Table 1. Performance comparison on WebShop and ALFWorld under in-distribution (ID) and out-of-distribution (OOD) settings.
Figure 3. Single-hop and multi-hop validation success rates and training success rates of our method and Skill0.5 with task-specific skills on Search QA benchmarks.
Method
Single-hop
Multi-hop
Avg.
NQ
Triv
Pop
Hotp
2Wk
MuS
Bam
Zero-Shot
10.4
32.4
22.3
15.8
15.4
7.2
19.2
17.5
Few-Shot †
12.3
36.8
24.5
17.7
18.2
6.5
24.8
20.1
Zero-Shot ⋆
6.9
30.4
12.0
10.5
9.1
5.5
24.0
14.0
Few-Shot ⋆†
10.5
31.9
18.7
14.2
14.4
6.9
24.8
17.3
GRPO
45.1
63.7
44.0
43.6
43.2
16.8
37.6
41.9
Table 2. Performance comparison on Search-QA benchmark.
Variant
ALFWorld
WebShop
ID Avg.
OOD Avg.
ID Avg.
OOD Avg.
w/o Int.-Action
85.1
54.7
44.7
47.3
w/o Ut.-Action
89.7
49.1
47.5
45.9
w/o Int.-Decrease
83.9
54.7
43.9
44.4
w/o Ut.-Increase
86.2
39.6
47.3
43.5
w/o Gate
86.2
47.2
46.7
43.0
Table 3. Ablation studies of the average success rate on ALFWorld and WebShop to validate the effectiveness of all our core modules; the best performance of each metric is bolded, and the second - best result is underlined.
Variant
ALFWorld
ID Avg.
OOD Avg.
Ratio (3:5:2)
85.1
62.3
Ratio (3.3:3.3:3.3)
87.4
41.5
AUSO (2:5:3)
94.3
67.9
Table 4. Ablation Study for Period time for Internalize/Explore/Utilize in ALFWorld.
Figure 4. The ID and OOD success rate for different hyperparameter K in the ALFWorld benchmark.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5. ID and OOD validation success rates and training success rates of our method and Skill0.5 with task-specific skills on Alfworld and Webshop benchmarks.
Figure 6. Training curves of AUSO vs. GRPO on ALFWorld: AUSO achieves higher success rate and reward with lower policy entropy and shorter episode length, indicating more confident and sample-efficient learning.
Domain
Method
Query / Task
#Steps
Correct
Final output / action
SearchQA (multi-hop)
AUSO (Ours)
Frank Lamson-Scribner was adopted by a family near which town in Kennebec County?
2
✓
Manchester
Skill0.5
4
×
Manchester, Maine
ALFWorld (OOD)
AUSO (Ours)
put a spraybottle in garbagecan
4
✓
put spraybottle 2 in/on garbagecan 1
Skill0.5
put a cool lettuce in countertop
25
×
go to diningtable 1
Appendix
Table 5. Representative cases comparing AUSO and Skill0.5.
Figure 7. Success rate of each action on the ALFWorld benchmark for AUSO (ID for training part and OOD for validation)
Figure 8. Success rate of each action on the Webshop benchmark for AUSO (ID for training part and OOD for validation)
Figure 9. Success Rate of each Action in validation set in SearchQA benchmark for our AUSO method
A persistent skill library allows language model agents to reuse successful strategies across tasks. Maintaining such a library requires three coupled capabilities. The agent selects a relevant skill, utilizes it during execution, and distills new skills from experience. Existing methods optimize these capabilities in isolation or with separate reward sources, resulting in partial and conflicting evolution. We propose Skill1, a framework that trains a single policy to co-evolve skill selection, utilization, and distillation toward a shared task-outcome objective. The policy generates a query to search the skill library, re-ranks candidates to select one, solves the task conditioned on it, and distills a new skill from the trajectory. All learning derives from a single task-outcome signal. Its low-frequency trend credits selection and its high-frequency variation credits distillation. Experiments on ALFWorld and WebShop show that Skill1 outperforms prior skill-based and reinforcement learning baselines. Training dynamics confirm the co-evolution of the three capabilities, and ablations show that removing any credit signal degrades the evolution.
Yaorui Shi, Yuxin Chen, Zhengxi Lu +6
University of Science and Technology of China · 2Meituan · 3National University of Singapore +2
Equipping large language models with explicit skills has emerged as a promising paradigm for enabling autonomous agents to solve complex tasks. Agent skills can be inherently divided into general skills for broad cognitive transfer and task-specific skills for dynamic execution. However, existing skill-based reinforcement learning (RL) methods typically force a rigid choice between full externalization, which incurs prohibitive context overhead, and full internalization, which risks overfitting and knowledge conflicts. To address this dilemma, we propose Skill0.5, a novel agentic RL framework that explicitly differentiates skill treatments by combining general skill internalization with task-specific skill utilization. Driven by a dynamic, difficulty-aware router, Skill0.5 streams tasks into distinct mastery tiers to apply tailored optimization strategies: it internalizes general skills via privileged distillation to build a cognitive foundation for hard tasks, while using diagnostic probing on easy tasks to penalize shortcuts and enforce specific skill utilization. Experiments on ALFWorld and WebShop demonstrate that Skill0.5 outperforms both memory-based and skill-based RL baselines, yielding performance improvements across both in-distribution and out-of-distribution scenarios.
Jiapeng Zhu, Jianxiang Yu, Yibo Zhao +5
1East China Normal University, 2Meituan Longcat Team · 1East China Normal University · 2Meituan Longcat Team
Large language model agents can improve across tasks by retaining reusable skills distilled from prior interactions. Recent work jointly optimizes task execution and skill extraction, enabling the policy and skillbank to co-evolve. However, as the actor continues learning, rewarding skill proposals through their reuse in subsequent training steps may conflate skill benefits with actor improvement, while directly testing each proposed skill requires costly additional actor rollouts. In this paper, we introduce UniSkill, which uses a shared policy to interact with the environment and propose skillbank edits (Add, Update, or No Edit) from the resulting trajectories. Specifically, the actor learns from environment rewards, while contrastive action feedback guides skill proposal learning. This feedback provides an actor-alignment signal by measuring how replacing the retrieved skill with a proposed skill changes the current actor's action log-likelihood gap between previously collected successful and failed trajectories from the same task, thereby avoiding new rollouts for each proposal. Since proposal-level feedback may suppress an otherwise appropriate edit operation when the proposed skill content scores poorly, we further apply skill-edit support regularization to preserve exploration. Empirically, UniSkill achieves strong performance, reaching 98.4% success on ALFWorld and 84.7% on WebShop while maintaining stable joint training. Further ALFWorld experiments show that UniSkill remains effective when the shared policy uses a smaller backbone. Our implementation is available at https://github.com/LimOkii/UniSKill.
Yifei Lu, Cheng Liu, Dianzhi Yu +4
Qiantang Credit, Hangzhou, China · Ant Group, Hangzhou, China · The Chinese University of Hong Kong