AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization
Authors: Huizu Lin, Chengkai Huang, Tianqi Gao, Tao Huang, Daijiao Liu, Tongxin Li, Xiaoyan Sun, Lina Yao
Organizations: University of New South Wales Sydney, Australia · University of Science and Technology of China, Hefei Comprehensive National Science Center Hefei, China
Skills play different roles as an agent's policy evolves: they should first provide learnable knowledge, then support capability formation, and finally be invoked only when they improve individual decisions. Existing methods rarely model this lifecycle. They either keep skills outside the model, fully internalize them, or select among internalization and utilization objectives through noisy task-level success rates. Such designs fragment training and assign uniform importance to actions within the same trajectory, even though skill guidance may help some decisions while distracting others. To solve these problems, we introduce AUSO (Action-level Unified Skill Optimization), which unifies skill learning and skill use through a progressive, action-aware optimization process. At the beginning of training, AUSO jointly learns from teacher guidance and environmental outcomes, enabling the policy to acquire foundational skills without losing task-oriented feedback. It subsequently emphasizes outcome-based policy optimization to consolidate autonomous problem-solving ability. As the policy matures, AUSO evaluates each sampled action under both skill-conditioned and skill-free contexts. The resulting action-level information signal is coupled with the trajectory outcome advantage, allowing beneficial skill-sensitive actions to receive stronger updates and harmful ones to be suppressed. Therefore, skills gradually transition from an external source of supervision into decision knowledge whose utilization is adapted to its action-level benefit, while reinforcement learning remains the shared backbone across all stages. Experiments on ALFWorld, WebShop, and SearchQA show that AUSO consistently improves agent performance and out-of-distribution generalization over competitive baselines.
Figures & tables
Figure 1. Empirical evidence of ambiguous medium/easy skill-routing boundaries in Skill0.5. Samples across the success threshold can differ by as little as 1/8 , revealing the ambiguity of threshold-based routing.
Figure 2. Overview of Our AUSO (Action-Level Unified Skill Optimization from Internalization to Utilization): (1) General Skills Internalization: Internalize transferable general skills through action-level teacher–student guidance. (2) Skills Exploration: Explore and optimize diverse skills via GRPO. (3) Specific Skill Utilization: Evaluate the benefit of task-specific skills for each action to guide action-level skill utilization.
Method
WebShop
ALFWorld
ID
OOD
ID
OOD
App.
Elec.
Foot.
Other
Avg.
Access.
Beauty
Home
Avg.
Pick
Cool
Clean
Avg.
Look
Heat
Pick2
Avg.
Prompt-based Methods
Zero-shot
4.4
4.6
3.4
1.0
3.5
2.5
3.7
5.5
3.9
28.6
12.0
18.5
20.7
38.5
12.5
12.5
18.9
Few-shot
14.2
15.1
13.5
24.0
16.5
18.8
29.6
5.5
16.9
62.9
44.0
63.0
57.5
46.2
31.2
8.3
24.5
ReAct
12.4
11.1
4.5
12.0
10.4
11.2
24.1
2.7
11.6
71.4
28.0
33.3
47.1
46.2
18.8
12.5
22.6
Table 1. Performance comparison on WebShop and ALFWorld under in-distribution (ID) and out-of-distribution (OOD) settings.
Figure 3. Single-hop and multi-hop validation success rates and training success rates of our method and Skill0.5 with task-specific skills on Search QA benchmarks.
Method
Single-hop
Multi-hop
Avg.
NQ
Triv
Pop
Hotp
2Wk
MuS
Bam
Zero-Shot
10.4
32.4
22.3
15.8
15.4
7.2
19.2
17.5
Few-Shot †
12.3
36.8
24.5
17.7
18.2
6.5
24.8
20.1
Zero-Shot ⋆
6.9
30.4
12.0
10.5
9.1
5.5
24.0
14.0
Few-Shot ⋆†
10.5
31.9
18.7
14.2
14.4
6.9
24.8
17.3
GRPO
45.1
63.7
44.0
43.6
43.2
16.8
37.6
41.9
Table 2. Performance comparison on Search-QA benchmark.
Variant
ALFWorld
WebShop
ID Avg.
OOD Avg.
ID Avg.
OOD Avg.
w/o Int.-Action
85.1
54.7
44.7
47.3
w/o Ut.-Action
89.7
49.1
47.5
45.9
w/o Int.-Decrease
83.9
54.7
43.9
44.4
w/o Ut.-Increase
86.2
39.6
47.3
43.5
w/o Gate
86.2
47.2
46.7
43.0
Table 3. Ablation studies of the average success rate on ALFWorld and WebShop to validate the effectiveness of all our core modules; the best performance of each metric is bolded, and the second - best result is underlined.
Variant
ALFWorld
ID Avg.
OOD Avg.
Ratio (3:5:2)
85.1
62.3
Ratio (3.3:3.3:3.3)
87.4
41.5
AUSO (2:5:3)
94.3
67.9
Table 4. Ablation Study for Period time for Internalize/Explore/Utilize in ALFWorld.
Figure 4. The ID and OOD success rate for different hyperparameter K in the ALFWorld benchmark.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5. ID and OOD validation success rates and training success rates of our method and Skill0.5 with task-specific skills on Alfworld and Webshop benchmarks.
Figure 6. Training curves of AUSO vs. GRPO on ALFWorld: AUSO achieves higher success rate and reward with lower policy entropy and shorter episode length, indicating more confident and sample-efficient learning.
Domain
Method
Query / Task
#Steps
Correct
Final output / action
SearchQA (multi-hop)
AUSO (Ours)
Frank Lamson-Scribner was adopted by a family near which town in Kennebec County?
2
✓
Manchester
Skill0.5
4
×
Manchester, Maine
ALFWorld (OOD)
AUSO (Ours)
put a spraybottle in garbagecan
4
✓
put spraybottle 2 in/on garbagecan 1
Skill0.5
put a cool lettuce in countertop
25
×
go to diningtable 1
Appendix
Table 5. Representative cases comparing AUSO and Skill0.5.
Figure 7. Success rate of each action on the ALFWorld benchmark for AUSO (ID for training part and OOD for validation)
Figure 8. Success rate of each action on the Webshop benchmark for AUSO (ID for training part and OOD for validation)
Figure 9. Success Rate of each Action in validation set in SearchQA benchmark for our AUSO method