Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models is a promising direction toward self-improving embodied agents that can adapt in openended, evolving environments. However, conventional wisdom from continual learning suggests that naive Sequential Fine-Tuning (Seq. FT) leads to catastrophic forgetting, necessitating complex CRL strategies. In this work, we take a step back and conduct a systematic study of CRL for large pretrained VLAs across diverse lifelong RL benchmarks. We find that, contrary to established belief, simple Seq. FT with low-rank adaptation (LoRA) is remarkably strong: it achieves high plasticity, exhibits little to no forgetting, and retains strong zero-shot generalization, frequently outperforming more sophisticated CRL methods. Through detailed analysis, we show that this robustness arises from a synergy between the large pretrained model, parameter-efficient adaptation, and on-policy RL. Together, these components reshape the stability-plasticity trade-off, making continual adaptation both stable and scalable. Our results position Sequential Fine-Tuning as a powerful method for continual RL with VLAs and provide new insights into lifelong learning in the large model era. Code is available at https://github.com/UT-Austin-RobIn/continual-vla-rl
Figures & tables
Figure 1: VLAs as Natural Continual Learners . We show that the synergy between pre-trained VLA, on-policy RL, and LoRA is enough to overcome catastrophic forgetting while maintaining plasticity, enabling simple Sequential Fine-Tuning to achieve surprisingly good performance.
Figure 2: Our evaluation spans diverse tasks and benchmarks. Here we show one task from each benchmark. For visualization and description of all the tasks, see Appendix H .
Domain / Method
Metrics (%)
AVG ↑
Δ AVG ↑
NBT ↓
FWT ↑
ZS ↑
Δ ZS ↑
libero-spatial
Sequential Fine-Tuning
81.2 ± 0.4
+24.3
0.3 ± 0.5
3.9 ± 1.5
57.1 ± 1.1
+5.6
Elastic Weight Consolidation
66.1 ± 0.9
+9.3
0.7 ± 1.7
1.5 ± 0.3
52.6 ± 0.9
+1.1
Expert Replay
80.2 ± 0.5
+23.3
0.6 ± 1.1
-2.3 ± 0.1
49.2 ± 1.0
-2.3
Dark Experience Replay
73.4 ± 1.3
+16.6
4.7 ± 1.3
0.7 ± 0.9
55.2 ± 0.7
+3.7
Table 1: Comparison of performance across CRL algorithms. Each number represent success rate of tasks (%). In addition to the metrics discussed in Sec. 3.3 , we report Δ between the initial checkpoint and the final checkpoint to indicate performance change during training. We bold the highest-performing method for each metric, not including the multitask oracle.
Figure 3: Each line tracks a single training task’s success rate, normalized to 100 % at the point it was first learned. Subsequent x-values show how that task’s performance changes as additional tasks are learned. Sequential Fine-Tuning shows little forgetting throughout the entire training.
Figure 4: Averaged across three benchmarks, Seq, FT obtains strong performance in both performance (AVG) and generalization (ZS).
Domain / Method
Metrics (%)
AVG ↑
Δ AVG ↑
NBT ↓
FWT ↑
ZS ↑
Δ ZS ↑
Camera Perturbation
Seq. FT
75.5 ± 0.2
+18.9
-0.5 ± 0.5
3.7 ± 1.1
46.7 ± 0.2
-0.6
Multitask (Oracle)
75.2 ± 0.1
+18.6
–
–
43.8 ± 0.5
-3.6
Lighting Perturbation
Seq. FT
82.4 ± 0.5
+26.7
0.2 ± 0.4
5.7 ± 0.1
54.9 ± 1.0
+1.9
Table 2: Examining the consistency of Seq. FT performance across different perturbations. We bold the metrics for which Seq. FT outperforms the multitask oracle.
Figure 5: Final training success rates: by simply prolonging the Seq. FT training steps, we can obtain on-par performance with multitask oracle.
Figure 6: VLA, on-policy RL, and LoRA are all crucial to stability. Removing any one of them results in catastrophic forgetting, shown by the curve.
Ablation
Metrics
AVG ↑
Δ AVG ↑
NBT ↓
FWT ↑
ZS ↑
Δ ZS ↑
Seq. FT (Original)
81.2±0.4
+24.3
0.3±0.5
0.3±0.5
57.1±1.1
+5.6
Supervised fine-tuning instead of RL
29.9 ± 2.3
-27.0
78.7 ± 1.9
-53.8 ± 0.0
1.1 ± 0.9
-50.4
Smaller Policy
13.1±0.9
-53.7
11.4 ± 3.7
-63.4±0.5
0.0 ± 0.0
-56.2
Without LoRA
7.3 ± 5.2
-49.6
40.9 ± 11.8
-50.4 ± 1.3
0.0 ± 0.0
-51.5
Table 3: Ablation studies on the libero-spatial benchmark
Method
Effective Rank (mean)
Effective Rank (std)
Nuclear Norm
NBT ↓
LoRA Rank 32 (default)
27.5
5.7
0.48
0.3
Full Finetuning
324.7
465.0
4.31
40.9
LoRA Rank 512
303.4
89.3
0.42
0.6
Table 4: Examining the properties of the delta weight with and without LoRA
Figure 7: Performance on tasks after sequentially training on 30 tasks. Seq. FT remains robust under extended length of the training task sequence.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
libero-object
libero-spatial
libero-long-horizon
RoboCasa
maniskill
Base Model
OpenVLA-OFT
OpenVLA-OFT
OpenVLA-OFT
Pi-0
OpenVLA
# of SFT Demos
10
10
432
240
140
Initial Training Success
55.6
56.9
83.0
18.9
51.6
# of Training Tasks
5
5
5
4
4
Episodes per Task
10240
10240
5120
3840
10240
Episode Length
512
512
512
480
80
Appendix
Table 5: Experiment setup across benchmarks.
Parameter
Supervised fine-tuning instead of RL
Smaller Policy
Without LoRA
Base Model
7B OpenVLA-OFT
12M CNN with MLP Head
7B OpenVLA-OFT
# of Training Tasks
5
5
5
Pre-training Demos
10
30
10
Initial Training Success
56.9
66.8
56.9
Batch Size
256
8192
8192
RL Episodes per Task
-
10240
10240
Appendix
Table 6: Ablation setup for libero-spatial benchmark.
Continual learning is a long-standing challenge in robot policy learning, where a policy must acquire new skills over time without catastrophically forgetting previously learned ones. While prior work has extensively studied continual learning in relatively small behavior cloning (BC) policy models trained from scratch, its behavior in modern large-scale pretrained Vision-Language-Action (VLA) models remains underexplored. In this work, we found that pretrained VLAs are remarkably resistant to forgetting compared with smaller policy models trained from scratch. Simple Experience Replay (ER) works surprisingly well on VLAs, sometimes achieving zero forgetting even with a small replay data size. Our analysis reveals that pretraining plays a critical role in downstream continual learning performance: large pretrained models mitigate forgetting with a small replay buffer size while maintaining strong forward learning capabilities. Furthermore, we found that VLAs can retain relevant knowledge from prior tasks despite performance degradation during learning new tasks. This knowledge retention enables rapid recovery of seemingly forgotten skills through finetuning. Together, these insights imply that large-scale pretraining fundamentally changes the dynamics of continual learning, enabling models to continually acquire new skills over time with simple replay. Code and more information can be found at https://continual-vlas.github.io/forget-me-not/
Huihan Liu, Changyeon Kim, Bo Liu +2
The University of Texas at Austin · KAIST · Microsoft Superintelligence.
When deployed in open-ended robotic environments, Vision--Language--Action (VLA) models need to continually acquire new skills, yet suffer from severe catastrophic forgetting. We observe that this degradation is related to the deterioration of cross-modal information structure, where dependencies among visual observations, language instructions, and actions progressively diffuse during continual adaptation. But existing continual learning methods fail to preserve such cross-modal information dependencies. Thus, we propose Info-VLA, an information-preserving continual learning framework that maintains cross-modal information structure through two complementary constraints. Replay Anchor Contrastive Learning constructs stable alignment anchors from a frozen teacher model, preserving cross-modal alignment in the representation space. Cross-Modal Mutual Information Maximization further preserves dependency structure between visual and language representations through mutual information constraints. By jointly preserving historical alignment and cross-modal dependency information, Info-VLA balances stability and plasticity during continual learning. Furthermore, experiments on the LIBERO show that Info-VLA significantly outperforms existing methods in both task retention and adaptation.
Libang Zhao, Qixin Zeng, Hongyin Zhang +1
Westlake University, China · University of Southampton, UK
Vision-language-action (VLA) models show promising knowledge accumulation ability through pretraining, yet continual learning remains challenging. Existing continual imitation learning (CIL) methods often require additional parameters or modules, limiting scalability for large VLA models. We propose Stellar VLA, a knowledge-driven CIL framework without parameter growth, featuring two variants: T-Stellar for task-centric modeling and TS-Stellar for hierarchical modeling of reusable skills across tasks. Stellar VLA jointly optimizes task representations and a self-evolving knowledge space, with knowledge-posterior-guided expert routing using knowledge relations and Top-K semantic embeddings for task specialization. Experiments on LIBERO demonstrate strong performance against VLA and CIL baselines with only 1% replay. Dual-arm real-world experiments validate knowledge transfer across embodiments and scenes. TS-Stellar excels in hierarchical manipulation, while visualizations demonstrate knowledge retention and task discovery. Project Website: https://stellarvla.github.io/
Yuxuan Wu, Guangming Wang, Zhiheng Yang +4
Shanghai Jiao Tong University · Shanghai Innovation Institute · University of Cambridge +4