cs.LGMar 12, 2026

Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning

Authors: Jiaheng Hu, Jay Shim, Chen Tang, Yoonchang Sung, Bo Liu, Peter Stone, Roberto Martin-Martin

Organizations: UT Austin · UCLA · NTU · Sony AI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models is a promising direction toward self-improving embodied agents that can adapt in openended, evolving environments. However, conventional wisdom from continual learning suggests that naive Sequential Fine-Tuning (Seq. FT) leads to catastrophic forgetting, necessitating complex CRL strategies. In this work, we take a step back and conduct a systematic study of CRL for large pretrained VLAs across diverse lifelong RL benchmarks. We find that, contrary to established belief, simple Seq. FT with low-rank adaptation (LoRA) is remarkably strong: it achieves high plasticity, exhibits little to no forgetting, and retains strong zero-shot generalization, frequently outperforming more sophisticated CRL methods. Through detailed analysis, we show that this robustness arises from a synergy between the large pretrained model, parameter-efficient adaptation, and on-policy RL. Together, these components reshape the stability-plasticity trade-off, making continual adaptation both stable and scalable. Our results position Sequential Fine-Tuning as a powerful method for continual RL with VLAs and provide new insights into lifelong learning in the large model era. Code is available at https://github.com/UT-Austin-RobIn/continual-vla-rl

Figures & tables

Figure 1: VLAs as Natural Continual Learners . We show that the synergy between pre-trained VLA, on-policy RL, and LoRA is enough to overcome catastrophic forgetting while maintaining plasticity, enabling simple Sequential Fine-Tuning to achieve surprisingly good performance.
Figure 2: Our evaluation spans diverse tasks and benchmarks. Here we show one task from each benchmark. For visualization and description of all the tasks, see Appendix H .
Domain / MethodMetrics (%)
 AVG ↑\uparrowΔ\Delta AVG ↑\uparrowNBT ↓\downarrowFWT ↑\uparrowZS ↑\uparrowΔ\Delta ZS ↑\uparrow
libero-spatial
Sequential Fine-Tuning81.2 ±\pm 0.4+24.30.3 ±\pm 0.53.9 ±\pm 1.557.1 ±\pm 1.1+5.6
Elastic Weight Consolidation66.1 ±\pm 0.9+9.30.7 ±\pm 1.71.5 ±\pm 0.352.6 ±\pm 0.9+1.1
Expert Replay80.2 ±\pm 0.5+23.30.6 ±\pm 1.1-2.3 ±\pm 0.149.2 ±\pm 1.0-2.3
Dark Experience Replay73.4 ±\pm 1.3+16.64.7 ±\pm 1.30.7 ±\pm 0.955.2 ±\pm 0.7+3.7
Table 1: Comparison of performance across CRL algorithms. Each number represent success rate of tasks (%). In addition to the metrics discussed in Sec. 3.3 , we report Δ\Delta between the initial checkpoint and the final checkpoint to indicate performance change during training. We bold the highest-performing method for each metric, not including the multitask oracle.
Figure 3: Each line tracks a single training task’s success rate, normalized to 100 %\% at the point it was first learned. Subsequent x-values show how that task’s performance changes as additional tasks are learned. Sequential Fine-Tuning shows little forgetting throughout the entire training.
Figure 4: Averaged across three benchmarks, Seq, FT obtains strong performance in both performance (AVG) and generalization (ZS).
Domain / MethodMetrics (%)
 AVG ↑\uparrowΔ\Delta AVG ↑\uparrowNBT ↓\downarrowFWT ↑\uparrowZS ↑\uparrowΔ\Delta ZS ↑\uparrow
Camera Perturbation
Seq. FT75.5 ±\pm 0.2+18.9-0.5 ±\pm 0.53.7 ±\pm 1.146.7 ±\pm 0.2-0.6
Multitask (Oracle)75.2 ±\pm 0.1+18.6––43.8 ±\pm 0.5-3.6
Lighting Perturbation
Seq. FT82.4 ±\pm 0.5+26.70.2 ±\pm 0.45.7 ±\pm 0.154.9 ±\pm 1.0+1.9
Table 2: Examining the consistency of Seq. FT performance across different perturbations. We bold the metrics for which Seq. FT outperforms the multitask oracle.
Figure 5: Final training success rates: by simply prolonging the Seq. FT training steps, we can obtain on-par performance with multitask oracle.
Figure 6: VLA, on-policy RL, and LoRA are all crucial to stability. Removing any one of them results in catastrophic forgetting, shown by the curve.
AblationMetrics
 AVG ↑\uparrowΔ\Delta AVG ↑\uparrowNBT ↓\downarrowFWT ↑\uparrowZS ↑\uparrowΔ\Delta ZS ↑\uparrow
Seq. FT (Original)81.2±0.4+24.30.3±0.50.3±0.557.1±1.1+5.6
Supervised fine-tuning instead of RL29.9 ±\pm 2.3-27.078.7 ±\pm 1.9-53.8 ±\pm 0.01.1 ±\pm 0.9-50.4
Smaller Policy13.1±0.9-53.711.4 ±\pm 3.7-63.4±0.50.0 ±\pm 0.0-56.2
Without LoRA7.3 ±\pm 5.2-49.640.9 ±\pm 11.8-50.4 ±\pm 1.30.0 ±\pm 0.0-51.5
Table 3: Ablation studies on the libero-spatial benchmark
MethodEffective Rank (mean)Effective Rank (std)Nuclear NormNBT ↓\downarrow
LoRA Rank 32 (default)27.55.70.480.3
Full Finetuning324.7465.04.3140.9
LoRA Rank 512303.489.30.420.6
Table 4: Examining the properties of the delta weight with and without LoRA
Figure 7: Performance on tasks after sequentially training on 30 tasks. Seq. FT remains robust under extended length of the training task sequence.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameterlibero-objectlibero-spatiallibero-long-horizonRoboCasamaniskill
Base ModelOpenVLA-OFTOpenVLA-OFTOpenVLA-OFTPi-0OpenVLA
# of SFT Demos1010432240140
Initial Training Success55.656.983.018.951.6
# of Training Tasks55544
Episodes per Task10240102405120384010240
Episode Length51251251248080
Appendix
Table 5: Experiment setup across benchmarks.
ParameterSupervised fine-tuning instead of RLSmaller PolicyWithout LoRA
Base Model7B OpenVLA-OFT12M CNN with MLP Head7B OpenVLA-OFT
# of Training Tasks555
Pre-training Demos103010
Initial Training Success56.966.856.9
Batch Size25681928192
RL Episodes per Task-1024010240
Appendix
Table 6: Ablation setup for libero-spatial benchmark.
AlgorithmNameValue
GRPOOptimizerAdamW
Learning rate2×10−52\times 10^{-5}
AdamW β1\beta_{1}0.9
AdamW β2\beta_{2}0.999
Adamw ϵ\epsilon10−510^{-5}
Gradient clip norm1.0
Appendix
Table 7: Hyperparameters for RL post-training.
AlgorithmNameValue
EWCregularization coefficient λ\lambda1×1061\times 10^{6}
fisher estimation samples65536
ERReplay loss weight λreplay\lambda_{\text{replay}}0.03
Replay # trajectories10
Replay global batch size8192
DERReplay loss weight λreplay\lambda_{\text{replay}}0.03
Appendix
Table 8: Algorithm-specific hyperparameters.
Figure 8: Changing camera angles.
Figure 9: Changing Lighting conditions.
Figure 10: Changing Robot initial position.
More in this topic

Continual Learning for VLA Models

Continual learning for vision-language-action (VLA) models studies how robots acquire new skills without losing previously learned behaviors.

See all 23 papers

Explore similar work

CardsList

  1. Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning

    Mar 4, 2026Huihan Liu, Changyeon Kim, Bo Liu +2Continual Robot LearningContinual Learning for VLA Models

    The University of Texas at Austin · KAIST · Microsoft Superintelligence.

  2. Information-Theoretic Constraints for Continual Vision-Language-Action Alignment

    Mar 6, 2026Libang Zhao, Qixin Zeng, Hongyin Zhang +1Continual Learning for VLA ModelsVision-Language Alignment

    Westlake University, China · University of Southampton, UK

  3. Continually Evolving Skill Knowledge in Vision Language Action Model

    Nov 22, 2025Yuxuan Wu, Guangming Wang, Zhiheng Yang +4Continual Learning for VLA ModelsContinual Robot Learning

    Shanghai Jiao Tong University · Shanghai Innovation Institute · University of Cambridge +4