OrthoSkillVLA: Continual Skill Learning via Gradient-Informed Skill Subspace Adaptation
Authors: Jiaqi Wang, Zhou Fang, Qiongfeng Shi, Yi Zhou
Organizations: School of Computer Science and Engineering, Southeast University, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China · School of Electronic Science and Engineering, Southeast University, China
Pretrained Vision-Language-Action models provide a strong foundation for robot learning, but sequentially adapting them to diverse skills can perturb the representations and velocity mappings used by previous skills, leading to catastrophic forgetting. Architecture-based approaches improve retention by isolating skills but lead to increased inference footprint. Recent subspace-constrained methods restrict parameter updates in an orthogonal subspace to minimize interference but impose a unified constraint on the entire model. We analyze the distinct roles of internal VLA components and identify two VLA-specific challenges. First, the VLM maintains broad semantic representations, making it vulnerable to capacity exhaustion, whereas the ActionHead refines semantics into localized velocity patterns that are highly sensitive to perturbations. Second, the final velocity decoder serves as a readout layer. Freezing it forms an output-stage expressivity bottleneck, while updating it risks overwriting previous velocity mappings. To this end, we propose OrthoSkillVLA, a parameter-efficient framework for continual skill learning in pretrained VLA models without demonstration replay. Given the representation heterogeneity, we impose separate subspace constraints on the VLM and ActionHead, preserving reusable semantic capacity while protecting localized velocity patterns. For the output layer, we introduce a lightweight feature-aware MoE decoder, where each skill is allocated a compact expert and a training-free router selects the expert according to feature-space affinity. Extensive simulated and real-world evaluations, together with ablations, demonstrate that OrthoSkillVLA better preserves prior skills while acquiring new ones.
Figures & tables
Figure 1: Overview of OrthoSkillVLA. For the VLM and ActionHead, OrthoSkillVLA constrains new skill learning to gradient-informed orthogonal subspaces with module-specific budgets, preserving reusable semantic capacity while protecting localized velocity patterns. For the final velocity decoder, it allocates a compact expert for each skill, and selects experts through a training-free projection router based on feature-space affinity.
Figure 2: Simulation results under skill-incremental continual learning. (a) Average success rate on seen skills. (b) Skill-wise success matrices across learning phases.
Method
FWT ↑
NBT ↓
AUC ↑
Final SR (%) ↑
SeqLoRA [ 20 ]
0.92±0.04
0.89±0.05
0.58±0.03
32.44±3.31
IncLoRA [ 20 ]
0.90±0.03
0.83±0.06
0.58±0.01
34.11±1.70
EWC [ 7 ]
0.73±0.02
0.68±0.04
0.46±0.04
25.17±4.38
OLoRA [ 22 ]
0.88±0.10
0.85±0.10
0.54±0.07
30.50±4.34
KeepLoRA [ 14 ]
0.90±0.05
0.46±0.16
0.72±0.02
56.61±7.22
OrthoSkillVLA
0.94 ± 0.03
0.13 ± 0.06
0.88 ± 0.01
83.50 ± 1.42
Table 1: Comparison of continual learning metrics across different methods in the simulated skill-incremental setting. The best results are highlighted in bold .
KeepLoRA [ 14 ]
OrthoSkillVLA
Training Phase
Flip
Pick
Push
Press
Flip
Pick
Push
Press
Phase 1: Learn Flip
16
—
—
—
19
—
—
—
Phase 2: Learn Pick
14
18
—
—
18
20
—
—
Phase 3: Learn Push
11
13
15
—
15
17
19
—
Phase 4: Learn Press
11
12
15
19
16
16
17
20
Table 2: Real-world success rates during continual skill learning phases for 20 trials. We compare the KeepLoRA [ 14 ] baseline and OrthoSkillVLA.
Threshold
Decoder
FWT ↑
NBT ↓
AUC ↑
Final SR (%) ↑
Unified
Frozen
0.90±0.05
0.46±0.16
0.72±0.02
56.61±7.22
Separated
Frozen
0.91±0.06
0.42±0.05
0.75±0.04
60.44±3.10
Unified
MoE
0.92±0.03
0.33±0.15
0.79±0.03
67.94±4.40
Separated
MoE
0.94±0.03
0.13±0.06
0.88±0.01
83.50±1.42
Table 3: Ablation on module-aware budgeting and MoE velocity decoding.
Figure 3: Evolution of SOR under varying unified energy thresholds.
Table 7
Figure 4: Evaluation of the training-free routing mechanism on routing accuracy (a) and success rate comparison with oracle routing (b) in our simulated benchmark.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Skill
Task instruction
OpenClose
close the bottom drawer of the cabinet
open the bottom drawer of the cabinet
close the top drawer of the cabinet
open the microwave
PickPlace
put the black bowl at the back on the plate
pick up the alphabet soup and put it in the basket
Appendix
Table 6: Task instructions of the three skills. See Figures 6 – 8 for execution snapshots.
Order ID
Skill orderings
Ordering 1
OpenClose → PickPlace → Turn
Ordering 2
OpenClose → Turn → PickPlace
Ordering 3
PickPlace → OpenClose → Turn
Appendix
Table 7: Skill orders used in the simulated continual learning experiments.
Figure 5: Qualitative comparison of model performance under the three skill orderings used in the simulated continual learning experiments.
Setting
Item
Value
Common
Optimizer
AdamW
Peak learning rate
5×10−5
Warmup steps
1000
Scheduler
cosine annealing
Batch size
16
Precision
bfloat16
Appendix
Table 8: Training configuration and baseline-specific settings used in the simulated experiments.
Rank
FWT ↑
NBT ↓
AUC ↑
Final SR(%) ↑
32
0.93 ± 0.01
0.17 ± 0.03
0.86 ± 0.02
79.50 ± 3.06
64
0.94 ± 0.03
0.13 ± 0.06
0.88 ± 0.01
83.50 ± 1.42
96
0.95 ± 0.01
0.29 ± 0.05
0.84 ± 0.02
74.06 ± 3.86
Appendix
Table 9: Ablation on the impact of the LoRA rank.
Figure 6: Visualization of the tasks in skill OpenClose .
Figure 7: Visualization of the tasks in skill PickPlace .
Figure 8: Visualization of the tasks in skill Turn .
Figure 9: Visualization of the four real-world skills, with each row showing keyframes from an autonomous rollout of Flip , Pick , Push , and Press .
Vision-Language-Action (VLA) models have shown strong promise for general-purpose robotic manipulation, yet adapting them to new tasks and domains remains inefficient: existing methods often rely on parameter tuning, incurring substantial costs and risking catastrophic forgetting of previously learned tasks. To address this, we propose Optimus-R, a memory-centric VLA framework that formulates robotic adaptation as explicit query-skill memory tuning. Optimus-R introduces: (i) An Inline Memory Interface for skill extraction. It inserts learnable memory tokens into the VLA prefix stream, allowing the backbone to derive control-aware query and skill representations within the native action-conditioning pathway. (ii) A Query-Skill Memory Bank for skill learning. It externalizes skills into query prototypes for deciding what to retrieve and skill values for specifying how to act, supporting skill reuse and expansion with limited parameter updates. (iii) A lightweight Bridge-and-Adapt mechanism for skill updating. It aligns target-domain queries and skills with the existing memory space through a lightweight adapter and residual memory updates. Experiments on in-domain adaptation, cross-domain adaptation, and lifelong learning show that Optimus-R enables data-efficient skill learning while mitigating catastrophic forgetting.
Zaijing Li, Rui Shao, Bing Hu +3
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory
Vision-language-action (VLA) models can learn a wide range of robotic skills, but combining separately trained skills remains challenging. The original training data may be unavailable for joint retraining, while directly merging model weights can cause interference. We propose NestedVLA, which consolidates independently fine-tuned VLA policies without access to their original training data. Instead of merging all updates into one checkpoint, a hypernetwork embedded in the VLA generates action-expert weight updates for a selected task. Each task's compressed vision-language model update is stored separately, and both updates are added to a frozen base VLA to instantiate the corresponding policy. We evaluate NestedVLA by consolidating three policies fine-tuned with reinforcement learning on MetaWorld, ManiSkill, and CALVIN. It achieves 66.7% average success, improving on the strongest model-merging baseline by 41.1 percentage points. It performs within 1 percentage point of per-task fine-tuning on ManiSkill and substantially narrows the gap on CALVIN. In this three-policy setting, total storage is 1.5x that of a single full checkpoint. Code is available at https://github.com/WanyuGroup/NeurIPS2026-NestedVLA.
Guodong Du, Wanyu Lin
Department of Computing, The Hong Kong Polytechnic University
Vision-language-action (VLA) models show promising knowledge accumulation ability through pretraining, yet continual learning remains challenging. Existing continual imitation learning (CIL) methods often require additional parameters or modules, limiting scalability for large VLA models. We propose Stellar VLA, a knowledge-driven CIL framework without parameter growth, featuring two variants: T-Stellar for task-centric modeling and TS-Stellar for hierarchical modeling of reusable skills across tasks. Stellar VLA jointly optimizes task representations and a self-evolving knowledge space, with knowledge-posterior-guided expert routing using knowledge relations and Top-K semantic embeddings for task specialization. Experiments on LIBERO demonstrate strong performance against VLA and CIL baselines with only 1% replay. Dual-arm real-world experiments validate knowledge transfer across embodiments and scenes. TS-Stellar excels in hierarchical manipulation, while visualizations demonstrate knowledge retention and task discovery. Project Website: https://stellarvla.github.io/
Yuxuan Wu, Guangming Wang, Zhiheng Yang +4
Shanghai Jiao Tong University · Shanghai Innovation Institute · University of Cambridge +4