OrthoSkillVLA: Continual Skill Learning via Gradient-Informed Skill Subspace Adaptation

Jiaqi Wang; Zhou Fang; Qiongfeng Shi; Yi Zhou

cs.ROAug 20, 2026

OrthoSkillVLA: Continual Skill Learning via Gradient-Informed Skill Subspace Adaptation

Authors: Jiaqi Wang, Zhou Fang, Qiongfeng Shi, Yi Zhou

Organizations: School of Computer Science and Engineering, Southeast University, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China · School of Electronic Science and Engineering, Southeast University, China

Abstract

Pretrained Vision-Language-Action models provide a strong foundation for robot learning, but sequentially adapting them to diverse skills can perturb the representations and velocity mappings used by previous skills, leading to catastrophic forgetting. Architecture-based approaches improve retention by isolating skills but lead to increased inference footprint. Recent subspace-constrained methods restrict parameter updates in an orthogonal subspace to minimize interference but impose a unified constraint on the entire model. We analyze the distinct roles of internal VLA components and identify two VLA-specific challenges. First, the VLM maintains broad semantic representations, making it vulnerable to capacity exhaustion, whereas the ActionHead refines semantics into localized velocity patterns that are highly sensitive to perturbations. Second, the final velocity decoder serves as a readout layer. Freezing it forms an output-stage expressivity bottleneck, while updating it risks overwriting previous velocity mappings. To this end, we propose OrthoSkillVLA, a parameter-efficient framework for continual skill learning in pretrained VLA models without demonstration replay. Given the representation heterogeneity, we impose separate subspace constraints on the VLM and ActionHead, preserving reusable semantic capacity while protecting localized velocity patterns. For the output layer, we introduce a lightweight feature-aware MoE decoder, where each skill is allocated a compact expert and a training-free router selects the expert according to feature-space affinity. Extensive simulated and real-world evaluations, together with ablations, demonstrate that OrthoSkillVLA better preserves prior skills while acquiring new ones.

Figures & tables

Figure 1: Overview of OrthoSkillVLA. For the VLM and ActionHead, OrthoSkillVLA constrains new skill learning to gradient-informed orthogonal subspaces with module-specific budgets, preserving reusable semantic capacity while protecting localized velocity patterns. For the final velocity decoder, it allocates a compact expert for each skill, and selects experts through a training-free projection router based on feature-space affinity.

Figure 2: Simulation results under skill-incremental continual learning. (a) Average success rate on seen skills. (b) Skill-wise success matrices across learning phases.

Method	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	Final SR (%) $\uparrow$
SeqLoRA [ 20 ]	$0.92\pm 0.04$	$0.89\pm 0.05$	$0.58\pm 0.03$	$32.44\pm 3.31$
IncLoRA [ 20 ]	$0.90\pm 0.03$	$0.83\pm 0.06$	$0.58\pm 0.01$	$34.11\pm 1.70$
EWC [ 7 ]	$0.73\pm 0.02$	$0.68\pm 0.04$	$0.46\pm 0.04$	$25.17\pm 4.38$
OLoRA [ 22 ]	$0.88\pm 0.10$	$0.85\pm 0.10$	$0.54\pm 0.07$	$30.50\pm 4.34$
KeepLoRA [ 14 ]	$0.90\pm 0.05$	$0.46\pm 0.16$	$0.72\pm 0.02$	$56.61\pm 7.22$
OrthoSkillVLA	0.94 $\pm$ 0.03	0.13 $\pm$ 0.06	0.88 $\pm$ 0.01	83.50 $\pm$ 1.42

Table 1: Comparison of continual learning metrics across different methods in the simulated skill-incremental setting. The best results are highlighted in bold .

	KeepLoRA [ 14 ]				OrthoSkillVLA
Training Phase	Flip	Pick	Push	Press	Flip	Pick	Push	Press
Phase 1: Learn Flip	16	—	—	—	19	—	—	—
Phase 2: Learn Pick	14	18	—	—	18	20	—	—
Phase 3: Learn Push	11	13	15	—	15	17	19	—
Phase 4: Learn Press	11	12	15	19	16	16	17	20

Table 2: Real-world success rates during continual skill learning phases for 20 trials. We compare the KeepLoRA [ 14 ] baseline and OrthoSkillVLA.

Threshold	Decoder	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	Final SR (%) $\uparrow$
Unified	Frozen	$0.90\pm 0.05$	$0.46\pm 0.16$	$0.72\pm 0.02$	$56.61\pm 7.22$
Separated	Frozen	$0.91\pm 0.06$	$0.42\pm 0.05$	$0.75\pm 0.04$	$60.44\pm 3.10$
Unified	MoE	$0.92\pm 0.03$	$0.33\pm 0.15$	$0.79\pm 0.03$	$67.94\pm 4.40$
Separated	MoE	$0.94\pm 0.03$	$0.13\pm 0.06$	$0.88\pm 0.01$	$83.50\pm 1.42$

Table 3: Ablation on module-aware budgeting and MoE velocity decoding.

Figure 3: Evolution of SOR under varying unified energy thresholds.

Table 7

Figure 4: Evaluation of the training-free routing mechanism on routing accuracy (a) and success rate comparison with oracle routing (b) in our simulated benchmark.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Skill	Task instruction
OpenClose	close the bottom drawer of the cabinet
	open the bottom drawer of the cabinet
	close the top drawer of the cabinet
	open the microwave
PickPlace	put the black bowl at the back on the plate
	pick up the alphabet soup and put it in the basket

Appendix

Table 6: Task instructions of the three skills. See Figures 6 – 8 for execution snapshots.

Order ID	Skill orderings
Ordering 1	OpenClose $\rightarrow$ PickPlace $\rightarrow$ Turn
Ordering 2	OpenClose $\rightarrow$ Turn $\rightarrow$ PickPlace
Ordering 3	PickPlace $\rightarrow$ OpenClose $\rightarrow$ Turn

Appendix

Table 7: Skill orders used in the simulated continual learning experiments.

Figure 5: Qualitative comparison of model performance under the three skill orderings used in the simulated continual learning experiments.

Setting	Item	Value
Common	Optimizer	AdamW
	Peak learning rate	$5\times 10^{-5}$
	Warmup steps	1000
	Scheduler	cosine annealing
	Batch size	16
	Precision	bfloat16

Appendix

Table 8: Training configuration and baseline-specific settings used in the simulated experiments.

Rank	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	Final SR(%) $\uparrow$
32	0.93 $\pm$ 0.01	0.17 $\pm$ 0.03	0.86 $\pm$ 0.02	79.50 $\pm$ 3.06
64	0.94 $\pm$ 0.03	0.13 $\pm$ 0.06	0.88 $\pm$ 0.01	83.50 $\pm$ 1.42
96	0.95 $\pm$ 0.01	0.29 $\pm$ 0.05	0.84 $\pm$ 0.02	74.06 $\pm$ 3.86

Appendix

Table 9: Ablation on the impact of the LoRA rank.

Figure 6: Visualization of the tasks in skill OpenClose .

Figure 7: Visualization of the tasks in skill PickPlace .

Figure 8: Visualization of the tasks in skill Turn .

Figure 9: Visualization of the four real-world skills, with each row showing keyframes from an autonomous rollout of Flip , Pick , Push , and Press .

Continual Learning for VLA Models

Continual learning for vision-language-action (VLA) models studies how robots acquire new skills without losing previously learned behaviors.

See all 23 papers

Inline Memory Meets Reusable Skills: Memory-centric Framework for Vision-Language-Action Model
Sep 30, 2026Zaijing Li, Rui Shao, Bing Hu +3Memory-Augmented VLMs VLA Model Adaptation
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory
NestedVLA: Learning to Consolidate and Generate Skills for Vision-Language-Action Models
Oct 16, 2026Guodong Du, Wanyu LinVision-Language-Action Models Continual Learning for VLA Models
Department of Computing, The Hong Kong Polytechnic University
Continually Evolving Skill Knowledge in Vision Language Action Model
Nov 22, 2025Yuxuan Wu, Guangming Wang, Zhiheng Yang +4Continual Learning for VLA Models Continual Robot Learning
Shanghai Jiao Tong University · Shanghai Innovation Institute · University of Cambridge +4

Sep 30, 2026J/K move · Enter open · S save

Inline Memory Meets Reusable Skills: Memory-centric Framework for Vision-Language-Action Model

Zaijing Li, Rui Shao, Bing Hu, Haoyu Zhang+2

Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory

Memory-Augmented VLMs VLA Model Adaptation Robot Skill Learning Continual Learning for VLA Models Data-Efficient Robot Learning Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have shown strong promise for general-purpose robotic manipulation, yet adapting them to new tasks and domains remains inefficient: existing methods often rely on parameter tuning, incurring substantial costs and risking catastrophic forgetting of previously learned tasks. To address this, we propose Optimus-R, a memory-centric VLA framework that formulates robotic adaptation as explicit query-skill memory tuning. Optimus-R introduces: (i) An Inline Memory Interface for skill extraction. It inserts learnable memory tokens into the VLA prefix stream, allowing the backbone to derive control-aware query and skill representations within the native action-conditioning pathway. (ii) A Query-Skill Memory Bank for skill learning. It externalizes skills into query prototypes for deciding what to retrieve and skill values for specifying how to act, supporting skill reuse and expansion with limited parameter updates. (iii) A lightweight Bridge-and-Adapt mechanism for skill updating. It aligns target-domain queries and skills with the existing memory space through a lightweight adapter and residual memory updates. Experiments on in-domain adaptation, cross-domain adaptation, and lifelong learning show that Optimus-R enables data-efficient skill learning while mitigating catastrophic forgetting.

Abstract

Continual Learning for VLA Models

Explore similar work

Inline Memory Meets Reusable Skills: Memory-centric Framework for Vision-Language-Action Model

NestedVLA: Learning to Consolidate and Generate Skills for Vision-Language-Action Models

Continually Evolving Skill Knowledge in Vision Language Action Model