Lifelong Language-Conditioned Robotic Manipulation Learning

Xudong Wang; Zebin Han; Zhiyu Liu; Gan Li; Jiahua Dong; Baichen Liu; Lianqing Liu; Zhi Han

cs.ROMar 5, 2026

Lifelong Language-Conditioned Robotic Manipulation Learning

Authors: Xudong Wang, Zebin Han, Zhiyu Liu, Gan Li, Jiahua Dong, Baichen Liu, Lianqing Liu, Zhi Han

Organizations: State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · North University of China · Southeast University ·

Abstract

Traditional language-conditioned manipulation agent sequential adaptation to new manipulation skills leads to catastrophic forgetting of old skills, limiting dynamic scene practical deployment. In this paper, we propose SkillsCrafter, a novel robotic manipulation framework designed to continually learn multiple skills while reducing catastrophic forgetting of old skills. Specifically, we propose a Manipulation Skills Adaptation to retain the old skills knowledge while inheriting the shared knowledge between new and old skills to facilitate learning of new skills. Meanwhile, we perform the singular value decomposition on the diverse skill instructions to obtain common skill semantic subspace projection matrices, thereby recording the essential semantic space of skills. To achieve forget-less and generalization manipulation, we propose a Skills Specialization Aggregation to compute inter-skills similarity in skill semantic subspaces, achieving aggregation of the previously learned skill knowledge for any new or unknown skill. Extensive experiments demonstrate the effectiveness and superiority of our proposed SkillsCrafter.

Figures & tables

Figure 1: The proposed Lifelong Robotic Manipulation task. (a) SkillsCrafter is able to evolve and learn new skills based on learned skills. It maintains a skill knowledge base that stores learned knowledge to allow for efficient learning of new skills using shared knowledge of old skills, and for performing any unknown skill in the open world using previously learned knowledge. (b) Skills catastrophic forgetting under lifelong learning settings. (c) Our SkillsCrafter has a better anti-forgetting performance.

Figure 2: Illustration of the three sets of observation. (a) Subspaces Consistency: The semantic space can be used to associate with the parameter subspace. (b) Different Layers across Various Skills: The importance of different layers is not the same for different skills. (c) Knowledge Decoupled: A tends to learn shared knowledge, while B to learn specific knowledge, naturally.

Figure 3: Illustration of the proposed SkillsCrafter pipeline. It includes (a) a Manipulation Skills Adaptation to achieve retaining the old skills knowledge while inheriting the shared knowledge between new and old skills to facilitate learning of new skills; (b) a Skills Specialization Aggregation to compute inter-skills similarity in skill-specific subspaces to achieve adaptive aggregation of skills knowledge; (c) a Skills Specified Inference to loads aggregated knowledge to achieve any skill manipulation inference.

Figure 4: Illustration of the experiment robotic skill tasks setting. We establish a total of 12 robotic simulator skills and 6 real-environment robotic skills for incremental learning.

Methods	S1	S2	S3	S4	S5	S6	S7	S8	S9	S10	S11	S12	S13	S14	S15	S16	S17	S18	Avg.
Seq-FT (sequence fine-tune all)	0	0	0	0	0	0	0	0	0	0	0	0	12	4	4	80	8	8	6.4
LwF-LoRA ( Li and Hoiem 2017 )	0	4	0	4	0	4	0	0	0	0	0	0	8	12	0	84	4	0	6.7
EWC-LoRA ( Xiang et al. 2023 )	0	0	0	0	4	0	0	0	0	0	0	0	16	8	4	80	8	4	6.9
Dense MoLE ( Chen et al. 2024 )	4	24	4	12	8	8	8	0	4	0	0	8	24	20	28	80	8	8	13.8
Sparse MoLE ( Dou et al. 2024 )	16	32	8	8	12	4	8	4	4	8	0	4	28	16	24	76	4	4	14.4
MoLA ( Gao et al. 2024 )	12	20	12	8	16	8	12	4	0	4	4	12	44	24	24	80	0	8	16.2

Table 1: Test results (Skill-Wise Average Success Rate, ASR

\uparrow

, %) of comparison experiment with LLCRM settings.

Methods	S1	S2	S3	S4	S5	S6	S7	S8	S9	S10	S11	S12	S13	S14	S15	S16	S17	S18	Avg.
Seq-FT	100	100	100	100	100	100	100	100	100	100	100	100	88	96	95	5	85	88	92.0
LwF-LoRA	100	95	100	95	100	93	100	100	100	100	100	100	92	87	100	0	92	100	91.9
EWC-LoRA	100	100	100	100	93	100	100	100	100	100	100	100	83	91	95	0	85	94	91.2
Dense MoLE	93	71	91	86	86	86	85	100	86	100	100	90	75	78	68	0	85	88	81.5
Sparse MoLE	73	62	82	90	79	93	85	80	86	50	100	95	71	83	73	0	92	94	77.0
MoLA	80	76	73	90	71	86	77	80	100	75	75	85	54	74	73	0	100	88	75.4

Table 2: Test results (Skill-Wise Forgetting Rate, FR

\downarrow

, %) of comparison experiment with LLCRM settings.

Figure 5: Illustration for some visualization examples.

Methods	GGM	SOT	INA	ASR %	FR %
Baseline	✗	✗	✗	5.8	92.8
SkillsCrafter w/o GGM	✗	✓	✓	46.9	26.5
SkillsCrafter w/o INA	✓	✓	✗	48.2	23.9
SkillsCrafter w/o SOT	✓	✗	✓	49.1	22.9
SkillsCrafter	✓	✓	✓	52.0	16.0

Table 3: Ablation study for skills shared/special knowledge.

Methods	IM	VM	AVG	TOP	ASR (%) $\uparrow$	FR (%)
SkillsCrafter w/ TOP	✗	✗	✗	✓	50.4	16.5
SkillsCrafter w/ Avg	✗	✗	✓	✗	44.4	26.3
SkillsCrafter w/ VM	✗	✓	✗	✗	48.9	20.2
SkillsCrafter	✓	✗	✗	✗	52.0	16.0

Table 4: Ablation study on skill knowledge aggregation.

Notations	Definitions
$\mathcal{I}$	user language instruction
$\mathcal{O}$	vision observation
$\mathcal{V}(\mathcal{O})$	CLIP vision-encoder
$\mathcal{S}$	multi-skills sequence set
$\{S_{1},S_{2},...,S_{t-1}\}$	old skills
$S_{t}$	current skill

Table 5: Some important notations and definitions.

Hyperparameters Name	Value
image size	128 × 128 × 3
per device train batch size $B$	16
learning rate	0.0001
num train epochs	4
computational bits	16
eval max steps	150

Table 6: Hyper-parameters used in SkillsCrafter.

No.	Skill in Simulator	Instruction Examples
1	open-drawer	“open the top drawer”, “open the middle drawer”, “open the bottom drawer”
2	slide-block	“slide the block to target”
3	put-money	“put the money away in the safe on the top shelf”, “put the money away in the safe on the bottom shelf”
4	sweep-dustpan	“sweep dirt to dustpan”
5	push-buttons	“push the maroon button”, “push the maroon button, then push the blue button”, “push the purple button”
6	turn-tap	“turn right tap”, “turn left tap”

Table 7: Details of Simulator & Real-world Manipulation Skills

Methods	GGM	SOT	INA	S1	S2	S3	S4	S5	S6	S7	S8	S9	S10	S11	S12	S13	S14	S15	S16	S17	S18	Avg.
Baseline	✗	✗	✗	0	0	0	0	0	0	0	0	0	0	0	0	12	4	4	80	4	0	5.8
SkillsCrafter w/o GGM	✗	✓	✓	48	76	32	48	48	48	40	4	12	8	12	68	92	76	72	76	40	44	46.9
SkillsCrafter w/o INA	✓	✓	✗	52	72	28	52	52	48	44	12	12	12	8	72	84	80	80	84	32	44	48.2
SkillsCrafter w/o SOT	✓	✗	✓	56	80	40	60	52	44	48	8	16	8	8	68	84	80	72	76	36	48	49.1
SkillsCrafter	✓	✓	✓	56	80	36	64	48	52	44	12	20	12	12	72	92	84	80	80	40	52	52.0

Table 8: Ablation study for skills shared/special knowledge with ASR %.

Methods	GGM	SOT	INA	S1	S2	S3	S4	S5	S6	S7	S8	S9	S10	S11	S12	S13	S14	S15	S16	S17	S18	Avg.
Baseline	✗	✗	✗	100	100	100	100	100	100	100	100	100	100	100	100	88	96	95	0	92	100	92.8
SkillsCrafter w/o GGM	✗	✓	✓	20	10	27	43	14	14	23	80	57	50	25	15	4	17	18	0	23	35	26.5
SkillsCrafter w/o INA	✓	✓	✗	13	14	36	38	7	14	15	40	57	25	50	0	13	13	9	0	38	35	23.9
SkillsCrafter w/o SOT	✓	✗	✓	7	5	9	29	7	21	8	60	43	50	50	15	13	13	18	5	31	29	22.9
SkillsCrafter	✓	✓	✓	7	5	18	24	14	7	15	40	29	25	25	10	4	9	9	0	23	24	16.0

Table 9: Ablation study for skills shared/special knowledge with FR %.

Methods	IM	VM	Avg	TOP	S1	S2	S3	S4	S5	S6	S7	S8	S9	S10	S11	S12	S13	S14	S15	S16	S17	S18	Avg.
SkillsCrafter w/ TOP	✗	✗	✗	✓	60	80	44	76	56	52	48	16	24	12	16	72	92	88	80	80	4	8	50.4
SkillsCrafter w/ Avg	✗	✗	✓	✗	60	80	40	64	44	48	44	12	16	8	16	68	88	40	84	72	8	8	44.4
SkillsCrafter w VM	✗	✓	✗	✗	52	76	32	60	44	52	48	16	24	12	8	76	84	88	84	76	36	12	48.9
SkillsCrafter	✓	✗	✗	✗	56	80	36	64	48	52	44	12	20	12	12	72	92	84	80	80	40	52	52.0

Table 10: Ablation study for skills Knowledge aggregation with ASR %.

Methods	IM	VM	Avg	TOP	S1	S2	S3	S4	S5	S6	S7	S8	S9	S10	S11	S12	S13	S14	S15	S16	S17	S18	Avg.
SkillsCrafter w/ TOP	✗	✗	✗	✓	0	5	0	10	0	7	8	20	14	25	0	10	4	4	9	0	92	88	16.5
SkillsCrafter w/ Avg	✗	✗	✓	✗	0	5	9	24	21	14	15	40	43	50	0	15	8	57	5	-5	85	88	26.3
SkillsCrafter w VM	✗	✓	✗	✗	13	10	27	29	21	7	8	20	14	25	50	5	13	4	5	0	31	82	20.2
SkillsCrafter	✓	✗	✗	✗	7	5	18	24	14	7	15	40	29	25	25	10	4	9	9	0	23	24	16.0

Table 11: Ablation study for skills Knowledge aggregation with FR %.

Figure 6: A photograph of the proposed UR5 robotic system.

Figure 7: Illustration of the proposed UR5 robotic system.

Language-Conditioned Robot Manipulation

Language-conditioned robot manipulation connects natural-language instructions to robot perception, planning, and physical actions.

See all 239 papers

Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation
Dec 20, 2025Yihang Zhu, Weiqing Wang, Shijie Wu +2Long-Horizon Robotic Manipulation Robot Skill Learning
ShanghaiTech University, Shanghai, China · InstAdapt
Skill2Real: Agentic Skill Learning for Zero-Shot Sim-to-Real Robot Manipulation
Oct 2, 2026Xincheng He, Siyu Ma, Chang Yu +5Robot Skill Learning Sim-to-Real Transfer
University of California, Los Angeles · Shanghai Jiao Tong University · University of Utah +1
Towards Human-like Physical Intelligence: LifelongVision-Language-Action Learning for Robotic Manipulation
Jul 16, 2026Yao He, Gan Sun, Wenqi Liang +2Continual Robot Learning Robot Skill Learning
South China University of Technology · University of Trento

Dec 20, 2025J/K move · Enter open · S save

Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation

Yihang Zhu, Weiqing Wang, Shijie Wu, Ye Shi+1

ShanghaiTech University, Shanghai, China · InstAdapt

Long-Horizon Robotic Manipulation Robot Skill Learning Generalization in Robotic Manipulation Multi-Task Robotic Manipulation Robotic Manipulation Robot Imitation Learning+1

Open paper AskPDF arXiv

Scaling imitation learning to diverse multi-task robot manipulation remains challenging due to suboptimal demonstrations, behavioral multi-modality, and destructive interference across tasks. While skill-based methods offer a promising direction by decomposing behaviors into reusable abstractions, existing approaches often learn skills that are either biased toward linguistic structure or lack semantic alignment across tasks, limiting generalization. In this work, we propose AtomSkill, a novel framework that learns a semantically aligned Atomic Skill Space from demonstrations and enables robust long-horizon execution through keypose imagination. Our method introduces: (1) semantic contrastive skill alignment, which partitions demonstrations into variable-length atomic skills and employs a contrastive objective to jointly enforce semantic consistency and temporal coherence, yielding a compact and reusable skill library; and (2) action decoding with keypose imagining, where the policy predicts both a skill's terminal keypose and immediate actions, thereby supporting progress-aware skill transitions. During inference, an atomic skill diffusion sampler generates plausible skill sequences, while predicted keyposes autonomously trigger smooth skill chaining. Extensive experiments in simulation and real-world settings show that AtomSkill consistently outperforms state-of-the-art imitation learning and skill-based baselines. Project page: https://atom-skill.github.io.

Abstract

Language-Conditioned Robot Manipulation

Explore similar work

Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation

Skill2Real: Agentic Skill Learning for Zero-Shot Sim-to-Real Robot Manipulation

Towards Human-like Physical Intelligence: LifelongVision-Language-Action Learning for Robotic Manipulation