Authors: Xudong Wang, Zebin Han, Zhiyu Liu, Gan Li, Jiahua Dong, Baichen Liu, Lianqing Liu, Zhi Han
Organizations: State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · North University of China · Southeast University · Mohamed bin Zayed University of Artificial Intelligence
Traditional language-conditioned manipulation agent sequential adaptation to new manipulation skills leads to catastrophic forgetting of old skills, limiting dynamic scene practical deployment. In this paper, we propose SkillsCrafter, a novel robotic manipulation framework designed to continually learn multiple skills while reducing catastrophic forgetting of old skills. Specifically, we propose a Manipulation Skills Adaptation to retain the old skills knowledge while inheriting the shared knowledge between new and old skills to facilitate learning of new skills. Meanwhile, we perform the singular value decomposition on the diverse skill instructions to obtain common skill semantic subspace projection matrices, thereby recording the essential semantic space of skills. To achieve forget-less and generalization manipulation, we propose a Skills Specialization Aggregation to compute inter-skills similarity in skill semantic subspaces, achieving aggregation of the previously learned skill knowledge for any new or unknown skill. Extensive experiments demonstrate the effectiveness and superiority of our proposed SkillsCrafter.
Figures & tables
Figure 1: The proposed Lifelong Robotic Manipulation task. (a) SkillsCrafter is able to evolve and learn new skills based on learned skills. It maintains a skill knowledge base that stores learned knowledge to allow for efficient learning of new skills using shared knowledge of old skills, and for performing any unknown skill in the open world using previously learned knowledge. (b) Skills catastrophic forgetting under lifelong learning settings. (c) Our SkillsCrafter has a better anti-forgetting performance.
Figure 2: Illustration of the three sets of observation. (a) Subspaces Consistency: The semantic space can be used to associate with the parameter subspace. (b) Different Layers across Various Skills: The importance of different layers is not the same for different skills. (c) Knowledge Decoupled: A tends to learn shared knowledge, while B to learn specific knowledge, naturally.
Figure 3: Illustration of the proposed SkillsCrafter pipeline. It includes (a) a Manipulation Skills Adaptation to achieve retaining the old skills knowledge while inheriting the shared knowledge between new and old skills to facilitate learning of new skills; (b) a Skills Specialization Aggregation to compute inter-skills similarity in skill-specific subspaces to achieve adaptive aggregation of skills knowledge; (c) a Skills Specified Inference to loads aggregated knowledge to achieve any skill manipulation inference.
Figure 4: Illustration of the experiment robotic skill tasks setting. We establish a total of 12 robotic simulator skills and 6 real-environment robotic skills for incremental learning.
Methods
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
Avg.
Seq-FT (sequence fine-tune all)
0
0
0
0
0
0
0
0
0
0
0
0
12
4
4
80
8
8
6.4
LwF-LoRA ( Li and Hoiem 2017 )
0
4
0
4
0
4
0
0
0
0
0
0
8
12
0
84
4
0
6.7
EWC-LoRA ( Xiang et al. 2023 )
0
0
0
0
4
0
0
0
0
0
0
0
16
8
4
80
8
4
6.9
Dense MoLE ( Chen et al. 2024 )
4
24
4
12
8
8
8
0
4
0
0
8
24
20
28
80
8
8
13.8
Sparse MoLE ( Dou et al. 2024 )
16
32
8
8
12
4
8
4
4
8
0
4
28
16
24
76
4
4
14.4
MoLA ( Gao et al. 2024 )
12
20
12
8
16
8
12
4
0
4
4
12
44
24
24
80
0
8
16.2
Table 1: Test results (Skill-Wise Average Success Rate, ASR ↑ , %) of comparison experiment with LLCRM settings.
Methods
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
Avg.
Seq-FT
100
100
100
100
100
100
100
100
100
100
100
100
88
96
95
5
85
88
92.0
LwF-LoRA
100
95
100
95
100
93
100
100
100
100
100
100
92
87
100
0
92
100
91.9
EWC-LoRA
100
100
100
100
93
100
100
100
100
100
100
100
83
91
95
0
85
94
91.2
Dense MoLE
93
71
91
86
86
86
85
100
86
100
100
90
75
78
68
0
85
88
81.5
Sparse MoLE
73
62
82
90
79
93
85
80
86
50
100
95
71
83
73
0
92
94
77.0
MoLA
80
76
73
90
71
86
77
80
100
75
75
85
54
74
73
0
100
88
75.4
Table 2: Test results (Skill-Wise Forgetting Rate, FR ↓ , %) of comparison experiment with LLCRM settings.
Figure 5: Illustration for some visualization examples.
Methods
GGM
SOT
INA
ASR %
FR %
Baseline
✗
✗
✗
5.8
92.8
SkillsCrafter w/o GGM
✗
✓
✓
46.9
26.5
SkillsCrafter w/o INA
✓
✓
✗
48.2
23.9
SkillsCrafter w/o SOT
✓
✗
✓
49.1
22.9
SkillsCrafter
✓
✓
✓
52.0
16.0
Table 3: Ablation study for skills shared/special knowledge.
Methods
IM
VM
AVG
TOP
ASR (%) ↑
FR (%)
SkillsCrafter w/ TOP
✗
✗
✗
✓
50.4
16.5
SkillsCrafter w/ Avg
✗
✗
✓
✗
44.4
26.3
SkillsCrafter w/ VM
✗
✓
✗
✗
48.9
20.2
SkillsCrafter
✓
✗
✗
✗
52.0
16.0
Table 4: Ablation study on skill knowledge aggregation.
Notations
Definitions
I
user language instruction
O
vision observation
V(O)
CLIP vision-encoder
S
multi-skills sequence set
{S1,S2,...,St−1}
old skills
St
current skill
Table 5: Some important notations and definitions.
Hyperparameters Name
Value
image size
128 × 128 × 3
per device train batch size B
16
learning rate
0.0001
num train epochs
4
computational bits
16
eval max steps
150
Table 6: Hyper-parameters used in SkillsCrafter.
No.
Skill in Simulator
Instruction Examples
1
open-drawer
“open the top drawer”, “open the middle drawer”, “open the bottom drawer”
2
slide-block
“slide the block to target”
3
put-money
“put the money away in the safe on the top shelf”, “put the money away in the safe on the bottom shelf”
4
sweep-dustpan
“sweep dirt to dustpan”
5
push-buttons
“push the maroon button”, “push the maroon button, then push the blue button”, “push the purple button”
6
turn-tap
“turn right tap”, “turn left tap”
Table 7: Details of Simulator & Real-world Manipulation Skills
Methods
GGM
SOT
INA
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
Avg.
Baseline
✗
✗
✗
0
0
0
0
0
0
0
0
0
0
0
0
12
4
4
80
4
0
5.8
SkillsCrafter w/o GGM
✗
✓
✓
48
76
32
48
48
48
40
4
12
8
12
68
92
76
72
76
40
44
46.9
SkillsCrafter w/o INA
✓
✓
✗
52
72
28
52
52
48
44
12
12
12
8
72
84
80
80
84
32
44
48.2
SkillsCrafter w/o SOT
✓
✗
✓
56
80
40
60
52
44
48
8
16
8
8
68
84
80
72
76
36
48
49.1
SkillsCrafter
✓
✓
✓
56
80
36
64
48
52
44
12
20
12
12
72
92
84
80
80
40
52
52.0
Table 8: Ablation study for skills shared/special knowledge with ASR %.
Methods
GGM
SOT
INA
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
Avg.
Baseline
✗
✗
✗
100
100
100
100
100
100
100
100
100
100
100
100
88
96
95
0
92
100
92.8
SkillsCrafter w/o GGM
✗
✓
✓
20
10
27
43
14
14
23
80
57
50
25
15
4
17
18
0
23
35
26.5
SkillsCrafter w/o INA
✓
✓
✗
13
14
36
38
7
14
15
40
57
25
50
0
13
13
9
0
38
35
23.9
SkillsCrafter w/o SOT
✓
✗
✓
7
5
9
29
7
21
8
60
43
50
50
15
13
13
18
5
31
29
22.9
SkillsCrafter
✓
✓
✓
7
5
18
24
14
7
15
40
29
25
25
10
4
9
9
0
23
24
16.0
Table 9: Ablation study for skills shared/special knowledge with FR %.
Methods
IM
VM
Avg
TOP
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
Avg.
SkillsCrafter w/ TOP
✗
✗
✗
✓
60
80
44
76
56
52
48
16
24
12
16
72
92
88
80
80
4
8
50.4
SkillsCrafter w/ Avg
✗
✗
✓
✗
60
80
40
64
44
48
44
12
16
8
16
68
88
40
84
72
8
8
44.4
SkillsCrafter w VM
✗
✓
✗
✗
52
76
32
60
44
52
48
16
24
12
8
76
84
88
84
76
36
12
48.9
SkillsCrafter
✓
✗
✗
✗
56
80
36
64
48
52
44
12
20
12
12
72
92
84
80
80
40
52
52.0
Table 10: Ablation study for skills Knowledge aggregation with ASR %.
Methods
IM
VM
Avg
TOP
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
Avg.
SkillsCrafter w/ TOP
✗
✗
✗
✓
0
5
0
10
0
7
8
20
14
25
0
10
4
4
9
0
92
88
16.5
SkillsCrafter w/ Avg
✗
✗
✓
✗
0
5
9
24
21
14
15
40
43
50
0
15
8
57
5
-5
85
88
26.3
SkillsCrafter w VM
✗
✓
✗
✗
13
10
27
29
21
7
8
20
14
25
50
5
13
4
5
0
31
82
20.2
SkillsCrafter
✓
✗
✗
✗
7
5
18
24
14
7
15
40
29
25
25
10
4
9
9
0
23
24
16.0
Table 11: Ablation study for skills Knowledge aggregation with FR %.
Figure 6: A photograph of the proposed UR5 robotic system.
Figure 7: Illustration of the proposed UR5 robotic system.
Scaling imitation learning to diverse multi-task robot manipulation remains challenging due to suboptimal demonstrations, behavioral multi-modality, and destructive interference across tasks. While skill-based methods offer a promising direction by decomposing behaviors into reusable abstractions, existing approaches often learn skills that are either biased toward linguistic structure or lack semantic alignment across tasks, limiting generalization. In this work, we propose AtomSkill, a novel framework that learns a semantically aligned Atomic Skill Space from demonstrations and enables robust long-horizon execution through keypose imagination. Our method introduces: (1) semantic contrastive skill alignment, which partitions demonstrations into variable-length atomic skills and employs a contrastive objective to jointly enforce semantic consistency and temporal coherence, yielding a compact and reusable skill library; and (2) action decoding with keypose imagining, where the policy predicts both a skill's terminal keypose and immediate actions, thereby supporting progress-aware skill transitions. During inference, an atomic skill diffusion sampler generates plausible skill sequences, while predicted keyposes autonomously trigger smooth skill chaining. Extensive experiments in simulation and real-world settings show that AtomSkill consistently outperforms state-of-the-art imitation learning and skill-based baselines. Project page: https://atom-skill.github.io.
Yihang Zhu, Weiqing Wang, Shijie Wu +2
ShanghaiTech University, Shanghai, China · InstAdapt
Transferring robotic skills from simulation to reality requires task knowledge that remains usable across differences in perception, dynamics, and embodiment. We introduce Skill2Real, an agentic policy framework that learns executable skills through a shared application programming interface (API). A Proposer-Verifier-Governor (PVG) loop uses privileged simulation evidence to diagnose outcomes and validate updates, while keeping learned skills grounded in public observations and API semantics. The Cerebellum first acquires local manipulation skills; the Brain then learns task-level composition with the Cerebellum frozen. Both memories transfer to the real robot without task-policy fine-tuning or skill-memory updates. As GPT-5.6 Sol learns skills on LIBERO-90, evaluating each frozen checkpoint with GPT-6 Astra raises LIBERO-Pro Long success from 2.0% to 56.3%, without training on Pro Long. Independent Robosuite training reaches 85.1% and 89.4% mean success with Sol and Opus 5 across seven tasks, respectively. Frozen Sol-trained LIBERO-90 skills achieve 78.75% mean completion across four real-world manipulation tasks with Astra. Removing the Verifier or Governor during LIBERO-90 training lowers final Pro Long success by 17.3 and 13.3 percentage points, respectively. These results support learning and transferring a hierarchy of executable skills through a common robot interface.
Xincheng He, Siyu Ma, Chang Yu +5
University of California, Los Angeles · Shanghai Jiao Tong University · University of Utah +1
Similar to the natural capabilities of humans to sequentially learn new tasks, robots with Vision-Language-Action (VLA) models should possess lifelong learning ability to learn a new task when deployed in open-world environments. However, most recently proposed lifelong learning models aim to effectively learn the current task (plasticity) or maintain high accuracy on previous tasks (stability), while the plasticity-stability trade-off remains largely unsolved in robotic manipulation models. To address this fundamental challenge, we propose a cache-efficient lifelong Vision-Language-Action learning framework for robotic manipulation (i.e., LifelongVLA), which alleviates the plasticity-stability trade-off with a dual-timescale adaptation mechanism while achieving low-cost robotic deployment with a cache-efficient replay strategy. More concretely, we propose a dual-timescale LoRA gating module to decompose VLA adaptation into two lightweight pathways: a short-term adapter for plasticity and a long-term adapter for stable consolidation. These pathways are integrated via a task-aware gate, enabling explicit control of the plasticity-stability trade-off. In the skill replay phase, a cache-efficient stochastic replay strategy is proposed to preserve more balanced retention signals without full-trajectory storage. Finally, experiments show that LifelongVLA outperforms existing baselines, demonstrating efficient skill expansion, robust retention of learned manipulation behaviors, and reduced reliance on retraining for real-world deployment on an xArm robot.
Yao He, Gan Sun, Wenqi Liang +2
South China University of Technology · University of Trento