SAMBAR: Selective Anchoring via Method of Multipliers for Balanced Knowledge Acquisition and Retention in Vision-Language-Action Models

Aayushi Shrivastava; Xunlan Zhou; Hongrui Zhao; Ziyu Chen; Negar Mehr

cs.LGSep 26, 2026

SAMBAR: Selective Anchoring via Method of Multipliers for Balanced Knowledge Acquisition and Retention in Vision-Language-Action Models

Authors: Aayushi Shrivastava, Xunlan Zhou, Hongrui Zhao, Ziyu Chen, Negar Mehr

Organizations: Department of Mechanical Engineering, University of California Berkeley, Berkeley, CA 94709, USA

Abstract

Vision-Language-Action (VLA) models leverage large-scale pretraining to ultimately achieve generalist manipulation. Deployed VLA policies must support continual learning to acquire new tasks over time. Teaching a VLA a new task generally requires finetuning it on demonstrations of that task. However, naively finetuning on downstream tasks causes the policy to forget earlier tasks and degrades generalist capabilities. This failure is known as catastrophic forgetting. Most continual learning methods counter it by replaying data from earlier tasks. However, the old task demonstrations are not always readily available. In this paper, we introduce SAMBAR, a continual learning algorithm that prevents catastrophic forgetting during VLA finetuning without requiring access to the demonstrations of any previously learned task. We propose to cast continual learning as a constrained optimization problem and solve it with the method of multipliers. In our approach, the method of multipliers drives the policy to learn the new task without the model parameters drifting far away from their previous values. In contrast to a standard regularization penalty, the method of multipliers raises the penalty as the constraint violation accumulates by using a dual variable. We also selectively anchor the parameters critical to previous tasks to preserve past knowledge, leaving other parameters free for new task acquisition. The combination of dual variable and selective anchoring, therefore, balances knowledge acquisition with knowledge retention. We evaluate our method, SAMBAR, on the LIBERO simulation benchmark and on hardware. When sequentially finetuning on a VLA, every replay-free baseline we compare against completely forgets the first task it learned, whereas SAMBAR retains every task it has learned.

Figures & tables

Fig. 1: Overview of SAMBAR for replay-free continual VLA learning. A policy learns tasks $T_{1},T_{2},T_{3}$ sequentially using only the current task’s demonstrations. Orange , blue , and green indicate parameter importance for ${\color[rgb]{0.8945,0.4219,0.043}T_{1}}$ , ${\color[rgb]{0.3281,0.5586,0.8359}T_{2}}$ , and ${\color[rgb]{0.4688,0.5781,0.2344}T_{3}}$ , respectively, with darker shades indicating greater importance. Standard supervised finetuning (SFT) can overwrite parameters important to earlier tasks, causing forgetting. SAMBAR combines an importance-weighted parameter anchor with the method of multipliers to limit changes to parameters important to prior tasks while allowing new-task adaptation.

Fig. 2: The three finetuning LIBERO tasks we learn in sequence. Starting from a policy pretrained on 32 LIBERO tasks, we finetune on pot-on-stove , then bowl-in-drawer , then bottle-on-rack , in the order shown left to right. Each panel shows the initial scene the policy is given. The first two are long-horizon tasks, while the third is a goal task.

Method	Pretrain SR $\uparrow$	Finetune SR $\uparrow$	Weighted Avg SR $\uparrow$
Finetune task: pot-on-stove
Pretrained (no finetuning)	93.2 $\pm$ 1.2	4.0 $\pm$ 11.1	-
SFT	89.0 $\pm$ 0.9	100.0 $\pm$ 0.0	89.1
RETAIN [ 31 ]	89.7 $\pm$ 1.2	100.0 $\pm$ 0.0	89.8
LoRA [ 19 ]	88.1 $\pm$ 0.6	100.0 $\pm$ 0.0	88.2
EWC [ 5 ]	89.5 $\pm$ 1.4	100.0 $\pm$ 0.0	89.6

TABLE I: Learning one new task from a broadly pretrained policy. We finetune a pi0.5 policy pretrained on 117 LIBERO tasks on finetune tasks, and evaluate it on the 117 pretraining tasks ( Pretrain SR ) and on the finetune task ( Finetune SR ). All entries are success rates in percent, averaged over 5 random seeds with 5 episodes per seed, reported as mean

\pm

95% confidence interval; higher is better (

\uparrow

) everywhere. Weighted Avg SR is the task-count-weighted mean over all 118 tasks. Forgetting is mild in this regime, and SAMBAR learns each new task without significantly forgetting the 117 pretrained tasks.

Method	Pretrain SR $\uparrow$	Finetune SR $\uparrow$	Weighted Avg SR $\uparrow$
Finetune task: pot-on-stove
Pretrained (no finetuning)	93.9 $\pm$ 1.4	6.0 $\pm$ 16.7	-
Co-Training	93.8 $\pm$ 1.3	94.0 $\pm$ 6.8	93.8
SFT	56.1 $\pm$ 2.6	96.0 $\pm$ 6.8	57.3
RETAIN [ 31 ]	66.5 $\pm$ 2.1	94.0 $\pm$ 11.1	67.3
LoRA [ 19 ]	75.4 $\pm$ 1.8	100.0 $\pm$ 0.0	76.2

TABLE II: Learning one new task from a narrowly pretrained policy. We finetune a pi0.5 policy pretrained on 32 LIBERO tasks (8 from each suite) on two LIBERO-Long finetune tasks, and evaluate it on the 32 pretraining tasks ( Pretrain SR ) and on the finetune task ( Finetune SR ). All entries are success rates in percent, averaged over 5 random seeds with 10 episodes per seed, reported as mean

\pm

95% confidence interval; higher is better (

\uparrow

) everywhere. Weighted Avg SR is the task-count-weighted mean over all 33 tasks. Forgetting is severe in this regime: every baseline forgets the pretraining tasks, whereas SAMBAR remembers the pretraining tasks.

		Success on each newly learned task
Method	Pretrain SR $\uparrow$	pot-on-stove $\uparrow$	bowl-in-drawer $\uparrow$	bottle-on-rack $\uparrow$	Weighted Avg SR $\uparrow$
Co-Training	95.4 $\pm$ 1.0	100.0 $\pm$ 0.0	98.0 $\pm$ 5.6	88.0 $\pm$ 13.6	95.4
SFT	12.9 $\pm$ 1.9	0.0 $\pm$ 0.0	60.0 $\pm$ 21.5	84.0 $\pm$ 11.1	15.9
RETAIN [ 31 ]	17.4 $\pm$ 0.7	0.0 $\pm$ 0.0	68.0 $\pm$ 16.2	96.0 $\pm$ 6.8	20.6
LoRA [ 19 ]	45.4 $\pm$ 1.3	0.0 $\pm$ 0.0	18.0 $\pm$ 13.6	92.0 $\pm$ 10.4	44.6
EWC [ 5 ]	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	100.0 $\pm$ 0.0	2.9

TABLE III: Sequential Continual Learning. Starting from the policy pretrained on 32 tasks, we finetune on pot-on-stove , then bowl-in-drawer , then bottle-on-rack . We evaluate the policy after learning all three tasks. All entries are success rates in percent, averaged over 5 random seeds, 10 episodes/seed and reported as mean

\pm

95% confidence interval; higher is better (

\uparrow

) for every column. Weighted Avg SR is the task-count-weighted mean over all 35 tasks. After three tasks every baseline has completely forgotten pot-on-stove task, while SAMBAR still performs it

84.0\%

of the time along with high success rate on 32 pretrained tasks.

	Pretraining tasks
Method	bird-in-bin $\uparrow$	two-birds-in-bin $\uparrow$	eraser-in-mug $\uparrow$	eraser-in-bowl $\uparrow$	Pretrain SR $\uparrow$	Finetune SR $\uparrow$	Weighted Avg SR $\uparrow$
Finetune task: cake-on-cake-tray
SFT	4/5	1/5	5/5	2/5	60.0	5/5	68.0
RETAIN ( $\alpha=0.5$ )	5/5	4/5	3/5	5/5	85.0	5/5	88.0
EWC	5/5	4/5	5/5	2/5	80.0	4/5	80.0
SAMBAR (ours)	5/5	5/5	5/5	5/5	100.0	5/5	100.0

TABLE IV: Learning one new task on hardware. We pretrain pi0.5 on four tasks and then finetune it on each of the two finetuning tasks. We evaluate every task with 5 rollouts and report the individual tasks as the number of successful rollouts out of 5. Pretrain SR is the percentage of the 20 pretraining rollouts that succeed, Weighted Avg SR is the percentage of all 25 rollouts that succeed. Higher is better (

\uparrow

) for every column. SAMBAR is the only method that keeps all four pretraining tasks intact while learning the new one.

	Pretraining tasks					Finetune tasks
Method	bird-in- bin $\uparrow$	two-birds- in-bin $\uparrow$	eraser-in- mug $\uparrow$	eraser-in- bowl $\uparrow$	Pretrain SR $\uparrow$	cake-on- cake-tray $\uparrow$	remove-lid-and- place-bird $\uparrow$	Weighted Avg SR $\uparrow$
SFT	5/5	5/5	5/5	3/5	90.0	4/5	3/5	83.3
RETAIN ( $\alpha=0.5$ )	5/5	5/5	4/5	4/5	90.0	5/5	0/5	76.7
EWC	5/5	5/5	5/5	5/5	100.0	4/5	2/5	86.7
SAMBAR (ours)	5/5	5/5	5/5	5/5	100.0	4/5	4/5	93.3

TABLE V: Sequential Continual Learning on hardware. Starting from the policy pretrained on the four tasks, we finetune on the cake-on-cake-tray task and then on the remove-lid-and-place-bird task, and evaluate all six tasks after learning the remove-lid-and-place-bird task. Entries follow the same convention as Table IV , with Weighted Avg SR now the percentage of all 30 rollouts that succeed.

Fig. 3: The hardware experiment tasks. We pretrain pi0.5 on the four tasks on the left and finetune on the two on the right. Each panel shows the initial scene the policy is given and the arrows shows the task policy needs to perform.

Continual Learning for VLA Models

Continual learning for vision-language-action (VLA) models studies how robots acquire new skills without losing previously learned behaviors.

See all 23 papers

Information-Theoretic Constraints for Continual Vision-Language-Action Alignment
Mar 6, 2026Libang Zhao, Qixin Zeng, Hongyin Zhang +1Continual Learning for VLA Models Vision-Language Alignment
Westlake University, China · University of Southampton, UK
Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning
Mar 12, 2026Jiaheng Hu, Jay Shim, Chen Tang +4Continual Learning for VLA Models VLA Model Adaptation
UT Austin · UCLA · NTU +1
Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning
Mar 4, 2026Huihan Liu, Changyeon Kim, Bo Liu +2Continual Robot Learning Continual Learning for VLA Models
The University of Texas at Austin · KAIST · Microsoft Superintelligence.

Mar 6, 2026J/K move · Enter open · S save

Information-Theoretic Constraints for Continual Vision-Language-Action Alignment

Libang Zhao, Qixin Zeng, Hongyin Zhang, Donglin Wang

Westlake University, China · University of Southampton, UK

Continual Learning for VLA Models Vision-Language Alignment

Open paper AskPDF arXiv

When deployed in open-ended robotic environments, Vision--Language--Action (VLA) models need to continually acquire new skills, yet suffer from severe catastrophic forgetting. We observe that this degradation is related to the deterioration of cross-modal information structure, where dependencies among visual observations, language instructions, and actions progressively diffuse during continual adaptation. But existing continual learning methods fail to preserve such cross-modal information dependencies. Thus, we propose Info-VLA, an information-preserving continual learning framework that maintains cross-modal information structure through two complementary constraints. Replay Anchor Contrastive Learning constructs stable alignment anchors from a frozen teacher model, preserving cross-modal alignment in the representation space. Cross-Modal Mutual Information Maximization further preserves dependency structure between visual and language representations through mutual information constraints. By jointly preserving historical alignment and cross-modal dependency information, Info-VLA balances stability and plasticity during continual learning. Furthermore, experiments on the LIBERO show that Info-VLA significantly outperforms existing methods in both task retention and adaptation.

Abstract

Continual Learning for VLA Models

Explore similar work

Information-Theoretic Constraints for Continual Vision-Language-Action Alignment

Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning

Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning