Organizations: Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China · University of Chinese Academy of Sciences, Beijing, 100049, China
While skill-centric approaches leverage foundation models to enhance generalization in compositional tasks, they often rely on fixed skill libraries, limiting adaptability to new tasks without manual intervention. To address this, we propose Uni-Skill, a Unified Skill-centric framework that supports skill-aware planning and facilitates automatic skill evolution. Unlike prior methods that restrict planning to predefined skills, Uni-Skill requests for new skill implementations when existing ones are insufficient, ensuring adaptable planning with self-augmented skill library. To support automatic implementation of diverse skills requested by the planning module, we construct SkillFolder, a VerbNet-inspired repository derived from large-scale unstructured robotic videos. SkillFolder introduces a hierarchical skill taxonomy that captures diverse skill descriptions at multiple levels of abstraction. By populating this taxonomy with large-scale, automatically annotated demonstrations, Uni-Skill shifts the paradigm of skill acquisition from inefficient manual annotation to efficient offline structural retrieval. Retrieved examples provide semantic supervision over behavior patterns and fine-grained references for spatial trajectories, enabling few-shot skill inference without deployment-time demonstrations. Comprehensive experiments in both simulation and real-world settings verify the state-of-the-art performance of Uni-Skill over existing VLM-based skill-centric approaches, highlighting its advanced reasoning capabilities and strong zero-shot generalization across a wide range of novel tasks.
Figures & tables
Fig. 1 : Uni-Skill facilitates the generalization of the planning process beyond predefined skills with self-augmented skill descriptions. By organizing extensive unstructured robotic videos with the hierarchical skill repository, SkillFolder, we enable efficient retrieval of skill demonstrations and automatic implementation of newly defined skills at deployment.
Fig. 2 : The automatic skill annotation pipeline. Skill functions derived from procedural descriptions are aligned with the video segments and iteratively involved in the construction of SkillFolder.
Fig. 3 : (a) Skill examples are retrieved from SkillFolder hierarchically at deployment. (b) Semantic constraints and action flows serve as in-context examples for waypoints generation, which are further lifted to 6-DoF poses with rotational patterns.
Models
setting
Push Buttons
Stack Blocks
Close Jar
Stack Cups
Sweep Dirt
Slide Block
Screw Bulb
Put in Board
Avg.
CaP (GPT-3.5) [ 5 ]
ICL
0.76 ±0.04
0.12 ±0.11
0.00 ±0.00
0.00 ±0.00
0.43 ±0.14
0.04 ±0.00
0.03 ±0.02
0.04 ±0.04
0.18 ±0.01
CaP (GPT-4o) [ 36 ]
ICL
0.84 ±0.00
0.31 ±0.06
0.03 ±0.02
0.01 ±0.02
0.64 ±0.00
0.21 ±0.02
0.13 ±0.02
0.08 ±0.04
0.28 ±0.02
Uni-Skill (ours)
ZSL
0.77 ±0.02
0.39 ±0.05
0.48 ±0.04
0.05 ±0.02
0.64 ±0.00
0.67 ±0.06
0.31 ±0.02
0.08 ±0.04
0.42 ±0.01
TABLE I : Success rate on 8 RLBench tasks covered by the basic skills.
Models
Close Micro
Close Fridge
Seat Down
Close Laptop
Close Drawer
Press Switch
Water Plants
Open Door
Unplug Charger
Lift Number
Avg.
CaP [ 5 ]
0.00 ±0.00
0.00 ±0.00
0.09 ±0.05
0.04 ±0.04
0.00 ±0.00
0.00 ±0.00
0.00 ±0.00
0.00 ±0.00
0.00 ±0.00
0.00 ±0.00
0.01 ± 0.01
MOKA [ 13 ]
0.05 ±0.02
0.17 ±0.06
0.23 ±0.08
0.07 ±0.05
0.19 ±0.05
0.33 ±0.06
0.00 ±0.00
0.00 ±0.00
0.00 ±0.00
0.00 ±0.00
0.10 ±0.02
Uni-Skill (ours)
0.49 ±0.06
0.57 ±0.05
0.68 ±0.04
0.33 ±0.02
0.56 ±0.00
0.40 ±0.04
0.05 ±0.02
0.33 ±0.02
0.39 ±0.02
0.31 ±0.02
0.41 ±0.01
TABLE II : Success rate on 10 RLBench tasks out of the basic skill set.
Fig. 4 : Example scene arrangement for real-world tasks.
Task
CaP [ 36 ]
MOKA [ 13 ]
Uni-Skill (ours)
Pick Place
0.80
0.60
0.90
Stack Blocks
0.20
0.10
0.50
Clean Table
0.80
1.00
1.00
Fold Cloth
0.00
0.20
0.70
Shake Bell
0.50
0.60
0.60
Close Door
0.00
0.10
0.80
TABLE III : The zero-shot success rate of Uni-Skill and baselines across 8 real-world manipulation tasks.
Fig. 5 : Ablation Results on RLBench.
Data Source
VerbNet class
Skill count
Success Rate
DROID [ 35 ]
106
1659
0.41 ±0.01
sth2sth [ 41 ]
122
1432
0.29 ±0.02
TABLE IV : Comparison on different raw data sources.
Fig. 6 : Failure modes of Uni-Skill out of the basic skill set.
Cross-task generalization is a core challenge in open-world robotic manipulation, and the key lies in extracting transferable manipulation knowledge from seen tasks. Recent in-context learning approaches leverage seen task demonstrations to generate actions for unseen tasks without parameter updates. However, existing methods provide only low-level continuous action sequences as context, failing to capture composable skill knowledge and causing models to degenerate into superficial trajectory imitation. We propose Decompose and Recompose, a skill reasoning framework using atomic skill-action pairs as intermediate representations. Our approach decomposes seen demonstrations into interpretable skill--action alignments, enabling the model to recompose these skills for unseen tasks through compositional reasoning. Specifically, we construct a task-adaptive dynamic demonstration library via visual-semantic retrieval combined with skill sequences from a planning agent, complemented by a coverage-aware static library to fill missing skill patterns. Together, these yield skill-comprehensive demonstrations that explicitly elicit compositional reasoning for skill composition and execution ordering. Experiments on the AGNOSTOS benchmark and real-world environments validate our method's zero-shot cross-task generalization capability.
Xitie Zhang, Aming Wu, Yahong Han
School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China
Embodied visuomotor models, including Diffusion Policy (DP) and Vision-Language-Action (VLA) models, have demonstrated promising performance on robotic manipulation benchmarks. However, their potential remains fundamentally constrained by the scarcity of large-scale embodied trajectory datasets, leading to insufficient compositional generalization in out-of-distribution (OOD) scenarios with limited capability to capture reusable skill structures. To address this limitation, we propose Skill-Based Memory (SkillMemo) framework that implicitly decomposes long-horizon demonstrations into latent atomic skills and integrates skill-level features into a dynamic episodic memory bank for solving compositional tasks. Specifically, we first introduce an expert-guided trajectory segmentation module built upon a Mixture-of-Experts (MoE) architecture, which implicitly partitions trajectories into distinct skill primitives represented by learned gating coefficients. We further design a skill-level episodic memory architecture that stores compact skill representations as retrievable key-value pairs. During inference, the memory bank retrieves the most relevant skill primitives which are subsequently fused with the model's current gating distribution, providing a robust contextual prior to refine action predictions. Extensive experiments on the simulation benchmark and real-world manipulation tasks demonstrate that SkillMemo consistently enhances both DP and VLA backbones, achieving state-of-the-art performance and outperforming π0.5, while exhibiting strong compositional generalization to unseen task configurations.
Changyuan Wang, Chubin Zhang, Zhenyu Wu +8
Shenzhen International Graduate School, Tsinghua University · Department of Automation, Tsinghua University · Nanyang Technological University +1
Vision-language-action and world-action models have demonstrated impressive capabilities in robotics, yet generalization to unseen tasks remains challenging. More recently, general-purpose multimodal agents have shown great potential for zero-shot robotic task solving. However, they often incur high execution costs by reasoning and exploring the physical world from scratch. To reduce these costs, we introduce RoboSkill, a framework that connects skill acquisition and reuse through an Explore, Execute, Evolve loop. Within this loop, the agent explores to gather task-relevant information, executes tasks while adapting to feedback, and evolves its skill library based on execution records. It then reuses these skills to guide exploration and execution in the next cycle, closing the loop. To improve loop efficiency, we complement vision with tactile feedback to reduce uncertainty during physical interaction. We further augment textual guidance with reusable code to reduce reasoning overhead during skill reuse. On LIBERO-10, RoboSkill improves first-episode success rates by 12.5--25.0 percentage points and reduces average runtime by 7.6--72.4% across four agents. On real robots, it improves success rates by 8.3 percentage points and reduces average runtime for successful trials by at least 14.4%.
Sicheng Xie, Yitong Chen, Haidong Cao +3
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · NeoteAI.