SARAM: Sparse Adapter Allocation with Reuse and Merging for Lifelong Robot Learning

SeongJun Jeong; Chungwoo Lee; Gi-Cheon Kang; Byoung-Tak Zhang

cs.ROOct 17, 2026

SARAM: Sparse Adapter Allocation with Reuse and Merging for Lifelong Robot Learning

Authors: SeongJun Jeong, Chungwoo Lee, Gi-Cheon Kang, Byoung-Tak Zhang

Organizations: Seoul National University · Ajou University

Abstract

Lifelong robot learning requires acquiring new tasks sequentially without forgetting earlier ones. With large pre-trained vision-language-action (VLA) backbones, adapter-based methods effectively mitigate this challenge. However, their task-specific adapters and selection modules accumulate with the task stream, creating a scalability challenge in on-disk storage and deployment cost under fixed onboard resource budgets. We propose SARAM, a lifelong learning framework that limits storage growth while keeping GPU memory and per-step inference cost independent of the task stream length. For each new task, SARAM reuses frozen adapters from previous tasks and introduces new adapters only at sparsely selected layers, with a duplication penalty encouraging them to capture information not already represented by the adapter library. At deployment, a compact signature derived from backbone activation statistics identifies the task once per episode, and the corresponding task update is merged into the backbone, leaving only the backbone in the control loop. Across four LIBERO suites, SARAM matches or exceeds adapter-based baselines while adding up to 4.5x less on-disk state per task. GPU memory usage and control frequency remain comparable to those of the backbone as the task stream grows. Experiments on a real UR5 arm with a different backbone further demonstrate retention throughout a sequential task stream.

Figures & tables

Fig. 1: Two growth axes of adapter-based lifelong robot learning. Prior methods accumulate adapters and selection modules on disk and keep them available on the GPU during control. SARAM adds new adapters sparsely and deploys a single merged backbone, adding up to $4.5\times$ less on-disk state per task in our experiments.

Fig. 2: Overview of SARAM . Top: each task reuses the frozen library (Phase 1), allocates new adapters to a sparse subset of layers (Phase 2), trains them under a duplication penalty (Phase 3), and stores coefficients, adapters, and a task signature. Bottom: at deployment, one adapter-free pass selects the task and its update is merged, so only the backbone runs.

LIBERO-Long

LIBERO-Goal

LIBERO-Spatial

LIBERO-Object

Method

FWT

\uparrow

SR final

\uparrow

NBT

\downarrow

AUC

\uparrow

FWT

\uparrow

SR final

\uparrow

NBT

\downarrow

AUC

\uparrow

FWT

\uparrow

SR final

\uparrow

NBT

\downarrow

AUC

\uparrow

FWT

\uparrow

SR final

\uparrow

NBT

\downarrow

AUC

\uparrow

SeqFFT

70.1

\pm

2.0

8.8

\pm

0.2

74.7

\pm

0.8

20.9

\pm

0.0

90.7

\pm

1.0

9.4

\pm

0.5

87.6

\pm

1.4

25.6

\pm

0.5

87.0

\pm

0.8

9.2

\pm

0.9

82.7

\pm

0.7

26.2

\pm

0.5

93.2

\pm

1.7

21.8

\pm

1.1

78.4

\pm

0.3

36.2

\pm

0.6

SeqLoRA

57.0

\pm

0.8

4.1

\pm

0.9

66.5

\pm

1.9

15.7

\pm

0.4

82.4

\pm

0.3

6.8

\pm

1.8

83.7

\pm

0.3

23.4

\pm

0.1

75.6

\pm

0.7

7.8

\pm

0.5

77.9

\pm

2.0

22.4

\pm

0.5

82.9

\pm

0.2

15.0

\pm

1.2

75.9

\pm

1.5

31.1

\pm

1.1

PackNet

12.4

\pm

2.8

0.0

\pm

0.0

26.3

\pm

4.0

1.5

\pm

0.4

25.9

\pm

1.8

0.0

\pm

0.0

40.8

\pm

2.9

3.8

\pm

0.2

23.5

\pm

1.7

0.0

\pm

0.0

35.7

\pm

2.8

\pm

0.1

15.2

\pm

1.4

0.0

\pm

0.0

26.3

\pm

4.9

1.7

\pm

0.2

EWC

60.3

\pm

0.4

5.7

\pm

0.8

70.3

\pm

1.5

16.8

\pm

0.9

83.6

\pm

1.2

5.4

\pm

1.2

85.3

\pm

0.7

22.9

\pm

1.0

83.8

\pm

0.9

9.2

\pm

1.4

83.1

\pm

0.6

25.3

\pm

0.2

90.7

\pm

0.7

26.5

\pm

4.0

74.1

\pm

1.4

38.7

\pm

0.6

64.5

\pm

3.4

61.5

\pm

3.8

18.2

\pm

1.6

61.6

\pm

3.5

89.1

\pm

0.3

83.2

\pm

1.5

10.3

\pm

2.1

85.8

\pm

1.6

85.8

\pm

0.3

81.2

\pm

1.8

11.6

\pm

2.0

81.8

\pm

1.7

92.0

\pm

0.9

90.1

\pm

2.5

6.5

\pm

1.7

91.8

\pm

1.4

TABLE I: Results on the four LIBERO suites (3 seeds, mean

\pm

sd ). SARAM attains the highest

\mathrm{SR}_{\mathrm{final}}

and AUC on all suites with competitive FWT.

Fig. 3: Deployment cost versus tasks learned. GPU memory and control frequency stay at the backbone’s level for SARAM , and its stored parameters and on-disk state grow more slowly than adapter-based baselines.

	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	$\Delta$
SARAM (full)	73.0 $\pm$ 1.2	7.5 $\pm$ 0.9	80.9 $\pm$ 1.7
w/o coefficient pre-fit	68.7 $\pm$ 1.7	6.8 $\pm$ 0.9	79.0 $\pm$ 0.5	$-1.9$
random allocation per task	68.3 $\pm$ 3.7	9.4 $\pm$ 4.1	74.1 $\pm$ 7.1	$-6.7$
fixed random allocation	71.1 $\pm$ 3.5	7.1 $\pm$ 2.9	78.9 $\pm$ 3.8	$-2.0$
fixed first-task allocation	72.5 $\pm$ 2.3	10.8 $\pm$ 3.4	77.4 $\pm$ 3.8	$-3.5$
w/o library reuse	67.3 $\pm$ 0.7	8.5 $\pm$ 3.2	76.1 $\pm$ 2.0	$-4.7$

TABLE II: Component ablations on LIBERO-Long .

\Delta

is AUC change from full.

$\xi$	$\lambda$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	Params (M)
2	0	67.9 $\pm$ 10.5	7.0 $\pm$ 2.8	76.8 $\pm$ 9.1	6.8 $\pm$ 2.0
2	0.3	73.0 $\pm$ 1.2	7.5 $\pm$ 0.9	80.9 $\pm$ 1.7	6.9 $\pm$ 2.1
2	0.6	68.1 $\pm$ 8.1	10.0 $\pm$ 2.7	75.3 $\pm$ 4.1	6.8 $\pm$ 1.9
3	0.3	68.5 $\pm$ 5.6	7.4 $\pm$ 2.1	75.2 $\pm$ 4.4	2.4 $\pm$ 0.4
4	0.3	59.8 $\pm$ 5.0	11.6 $\pm$ 3.5	64.3 $\pm$ 4.2	1.0 $\pm$ 0.2

TABLE III: Effect of the hyperparameters

\xi

and

\lambda

on LIBERO-Long . Params denotes the cumulative number of stored parameters after ten tasks. Larger

\xi

trades performance for storage; the duplication penalty helps at moderate strength.

Fig. 4: Layers allocated on LIBERO-Long. Colored bars mark layers that keep an adapter for each task (rows) and seed (blocks).

Fig. 5: Real-robot task stream. Seven UR5 manipulation tasks learned sequentially.

Fig. 6: Task performance across the UR5 stream.

Method	$\mathrm{SR}_{\mathrm{learn}}$ $\uparrow$	$\mathrm{SR}_{\mathrm{final}}$ $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
Per-task LoRA (oracle ref.)	97.1	–	–	–
SeqLoRA	91.4	14.3	77.1	35.8
ER	84.3	41.4	40.2	53.6
SARAM (ours)	94.3	94.3	1.0	93.4

TABLE IV: Real-robot results on the UR5 stream .

\mathrm{SR}_{\mathrm{learn}}

is used as the NBT reference and the initial AUC point because no intermediate training probes are collected.

	Simulation (LIBERO)				Real
Instructions	Long	Goal	Spatial	Object	UR5
Original	100	100	100	100	100
Mild paraphrase	100	100	100	100	100
Strong paraphrase	100	90	90	100	85.3

TABLE V: Task-selection accuracy (%) under instruction paraphrases.

Continual Robot Learning

Continual robot learning develops robots that acquire new skills and adapt to changing conditions while retaining previously learned behavior.

See all 88 papers

SAMBAR: Selective Anchoring via Method of Multipliers for Balanced Knowledge Acquisition and Retention in Vision-Language-Action Models
Sep 26, 2026Aayushi Shrivastava, Xunlan Zhou, Hongrui Zhao +2Continual Learning for VLA Models Vision-Language-Action Models
Department of Mechanical Engineering, University of California Berkeley, Berkeley, CA 94709, USA
Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Continual Learning for LLMs Continual LoRA
College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1
Inline Memory Meets Reusable Skills: Memory-centric Framework for Vision-Language-Action Model
Sep 30, 2026Zaijing Li, Rui Shao, Bing Hu +3Memory-Augmented VLMs VLA Model Adaptation
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory

Sep 26, 2026J/K move · Enter open · S save

SAMBAR: Selective Anchoring via Method of Multipliers for Balanced Knowledge Acquisition and Retention in Vision-Language-Action Models

Aayushi Shrivastava, Xunlan Zhou, Hongrui Zhao, Ziyu Chen+1

Department of Mechanical Engineering, University of California Berkeley, Berkeley, CA 94709, USA

Continual Learning for VLA Models Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models leverage large-scale pretraining to ultimately achieve generalist manipulation. Deployed VLA policies must support continual learning to acquire new tasks over time. Teaching a VLA a new task generally requires finetuning it on demonstrations of that task. However, naively finetuning on downstream tasks causes the policy to forget earlier tasks and degrades generalist capabilities. This failure is known as catastrophic forgetting. Most continual learning methods counter it by replaying data from earlier tasks. However, the old task demonstrations are not always readily available. In this paper, we introduce SAMBAR, a continual learning algorithm that prevents catastrophic forgetting during VLA finetuning without requiring access to the demonstrations of any previously learned task. We propose to cast continual learning as a constrained optimization problem and solve it with the method of multipliers. In our approach, the method of multipliers drives the policy to learn the new task without the model parameters drifting far away from their previous values. In contrast to a standard regularization penalty, the method of multipliers raises the penalty as the constraint violation accumulates by using a dual variable. We also selectively anchor the parameters critical to previous tasks to preserve past knowledge, leaving other parameters free for new task acquisition. The combination of dual variable and selective anchoring, therefore, balances knowledge acquisition with knowledge retention. We evaluate our method, SAMBAR, on the LIBERO simulation benchmark and on hardware. When sequentially finetuning on a VLA, every replay-free baseline we compare against completely forgets the first task it learned, whereas SAMBAR retains every task it has learned.

Abstract

Continual Robot Learning

Explore similar work

SAMBAR: Selective Anchoring via Method of Multipliers for Balanced Knowledge Acquisition and Retention in Vision-Language-Action Models

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Inline Memory Meets Reusable Skills: Memory-centric Framework for Vision-Language-Action Model