Lifelong Imitation Learning with Multimodal Latent Replay and Incremental Adjustment

Fanqi Yu; Matteo Tiezzi; Tommaso Apicella; Cigdem Beyan; Vittorio Murino

cs.CVMar 11, 2026

Lifelong Imitation Learning with Multimodal Latent Replay and Incremental Adjustment

Authors: Fanqi Yu, Matteo Tiezzi, Tommaso Apicella, Cigdem Beyan, Vittorio Murino

Organizations: AI for Good (AIGO), Istituto Italiano di Tecnologia, Genoa, Italy · DITEN, University of Genoa, Genoa, Italy · PAVIS, Istituto Italiano di Tecnologia, Genoa, Italy · Department of Computer Science, University of Verona, Verona, Italy

Abstract

We introduce a lifelong imitation learning framework that enables continual policy refinement across sequential tasks under realistic memory and data constraints. Our approach departs from conventional experience replay by operating entirely in a multimodal latent space, where compact representations of visual, linguistic, and robot's state information are stored and reused to support future learning. To further stabilize adaptation, we introduce an incremental feature adjustment mechanism that regularizes the evolution of task embeddings through an angular margin constraint, preserving inter-task distinctiveness. Our method establishes a new state of the art in the LIBERO benchmarks, achieving 10-17 point gains in AUC and up to 65% less forgetting compared to previous leading methods. Ablation studies confirm the effectiveness of each component, showing consistent gains over alternative strategies. The code is available at: https://github.com/yfqi/lifelong_mlr_ifa.

Figures & tables

Figure 1: Illustration of Incremental Feature Adjustment (IFA). The figure displays a 2D projection of the global latent representations $\mathbf{g}$ during policy rollout for two related tasks, $T_{j}$ (previously learned) and $T_{k}$ (newly learned). The stars ( $\mathbf{\color[rgb]{1,0,0}\star}$ for $T_{j}$ and $\mathbf{\color[rgb]{0,1,0}\star}$ for $T_{k}$ ) represent the stable language reference embeddings $\mathbf{h}^{(r)}$ , while the circles ( $\mathbf{\color[rgb]{1,0,0}\bullet}$ for $T_{j}$ and $\mathbf{\color[rgb]{0,1,0}\bullet}$ for $T_{k}$ ) are the global embeddings $\mathbf{g}$ . (Left) Without IFA, the new task’s embeddings ( $\mathbf{\color[rgb]{0,1,0}\bullet}$ ) exhibit representation drift by clustering close to the old task’s embeddings ( $\mathbf{\color[rgb]{1,0,0}\bullet}$ ). (Right) With IFA, the loss $\mathcal{L}_{\text{IFA}}$ enforces a constraint on the distances: the distance to the own reference ( $D_{\text{own}}$ ) plus a margin $\delta$ must be less than or equal to the distance to the other task’s reference ( $D_{\text{other}}$ ). This mechanism forces the $\mathbf{g}(T_{k})$ ( $\mathbf{\color[rgb]{0,1,0}\bullet}$ ) cluster away from $\mathbf{h}^{(r)}(T_{j})$ ( $\mathbf{\color[rgb]{1,0,0}\star}$ ) and closer to $\mathbf{h}^{(r)}(T_{k})$ ( $\mathbf{\color[rgb]{0,1,0}\star}$ ) , achieving inter-task disentanglement.

Figure 2: Our method is a general multimodal architecture composed of modality-specific encoders (language, vision, and state), a modulation network, a temporal decoder, and a policy head. During the pretraining phase, all architecture modules are trained. In the lifelong learning phase, only the temporal decoder and policy head are updated using both the new task data and the samples stochastically stored in the replay buffer. The buffer stores the multimodal features, output of the modulation layer. The model is jointly supervised by both the Behavior Cloning and the Incremental Feature Adjustment loss, processing the current task and previously stored tasks.

Method	LIBERO-OBJECT			LIBERO-GOAL			LIBERO-50
Method	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
Sequential [ 14 ]	62.0 $\pm$ 1.0	63.0 $\pm$ 2.0	30.0 $\pm$ 1.0	55.0 $\pm$ 1.0	70.0 $\pm$ 1.0	23.0 $\pm$ 1.0	32.0 $\pm$ 1.0	90.0 $\pm$ 2.0	14.0 $\pm$ 2.0
ER [ 14 , 2 ]	56.0 $\pm$ 1.0	24.0 $\pm$ 1.0	49.0 $\pm$ 1.0	53.0 $\pm$ 1.0	36.0 $\pm$ 1.0	47.0 $\pm$ 2.0	35.0 $\pm$ 3.0	49.0 $\pm$ 1.0	36.0 $\pm$ 3.0
BUDS [ 33 , 26 ]	52.0 $\pm$ 2.0	21.0 $\pm$ 1.0	47.0 $\pm$ 1.0	50.0 $\pm$ 1.0	39.0 $\pm$ 1.0	42.0 $\pm$ 1.0	29 $\pm$ 3.0	50.0 $\pm$ 4.0	33.0 $\pm$ 3.0
LOTUS [ 26 ]	74.0 $\pm$ 3.0	11.0 $\pm$ 1.0	65.0 $\pm$ 3.0	61.0 $\pm$ 3.0	30.0 $\pm$ 1.0	56.0 $\pm$ 1.0	39 $\pm$ 2.0	43.0 $\pm$ 1.0	45.0 $\pm$ 2.0
ISCIL [ 10 ]	71.7 $\pm$ 1.9	11.9 $\pm$ 5.1	66.3 $\pm$ 3.7	70.4 $\pm$ 3.6	19.4 $\pm$ 4.1	60.5 $\pm$ 2.5	47.8 $\pm$ 1.3	15.0 $\pm$ 4.8	37.7 $\pm$ 2.1

Table 1: Comparison of results on the LIBERO benchmarks (mean

\pm

standard deviation). “NA” indicates not available. Best results are shown in bold.

Modality	LIBERO-OBJECT			LIBERO-GOAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
AV	83.8 $\pm$ 2.2	11.1 $\pm$ 10.4	78.9 $\pm$ 2.6	77.7 $\pm$ 2.6	7.8 $\pm$ 2.7	73.9 $\pm$ 2.6
Lan	83.3 $\pm$ 0.7	13.5 $\pm$ 3.1	77.3 $\pm$ 1.9	77.1 $\pm$ 1.9	12.3 $\pm$ 0.8	71.8 $\pm$ 2.0
EIH	84.4 $\pm$ 0.9	16.1 $\pm$ 3.1	77.1 $\pm$ 1.1	75.0 $\pm$ 3.3	5.6 $\pm$ 3.9	72.7 $\pm$ 4.2
Act	84.6 $\pm$ 2.6	20.7 $\pm$ 3.6	75.1 $\pm$ 1.4	73.8 $\pm$ 2.5	2.5 $\pm$ 4.2	73.2 $\pm$ 1.1
Sta	82.5 $\pm$ 3.3	11.2 $\pm$ 10.6	77.5 $\pm$ 2.6	78.3 $\pm$ 5.1	8.2 $\pm$ 2.4	74.9 $\pm$ 5.9

Table 2: Influence of different modality similarity on the proposed method’s performance. “AV”, “Lan”, “EIH”, “Act”, and “Sta” denote Agent-view, Language, Eye-in-hand, Action, and State modalities, respectively.

	Buffer	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
LIBERO-OBJECT	33.3%	83.3 $\pm$ 2.6	12.3 $\pm$ 2.4	77.6 $\pm$ 3.0
	50%	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5
	66.6%	83.3 $\pm$ 1.4	16.4 $\pm$ 9.0	75.8 $\pm$ 5.3
LIBERO-GOAL	33.3%	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8
	50%	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8
	66.6%	82.0 $\pm$ 0.9	15.4 $\pm$ 16.7	75.6 $\pm$ 8.7

Table 3: Ablation on task pair selection proportion used when calculating IFA.

	Reference	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
LIBERO-OBJECT	Mean global	79.6 $\pm$ 7.21	8.3 $\pm$ 12.2	75.7 $\pm$ 1.1
LIBERO-OBJECT	Lan+agent view	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5
LIBERO-GOAL	Mean global	77.1 $\pm$ 3.1	15.7 $\pm$ 6.1	70.5 $\pm$ 6.0
LIBERO-GOAL	Lan+agent view	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8

Table 4: Effectiveness of choosing different references. “Lan” stands for language. See the text for a description of the “mean global” reference.

	Probability	Storage	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
LIBERO-OBJECT	0.50	188MB	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5
	0.20	75.3MB	80.5 $\pm$ 7.5	12.5 $\pm$ 7.1	77.7 $\pm$ 10.0
	0.10	37.6MB	79.9 $\pm$ 12.6	13.4 $\pm$ 6.1	76.6 $\pm$ 14.1
LIBERO-GOAL	0.50	121.2MB	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8
	0.20	60.6MB	78.3 $\pm$ 1.9	10.6 $\pm$ 8.0	77.4 $\pm$ 4.6
	0.10	30.3MB	78.7 $\pm$ 1.3	25.2 $\pm$ 10.4	66.2 $\pm$ 6.8

Table 5: Impact of varying the probability of storing features in the buffer, affecting buffer size.

Figure 3: Comparison between cosine distance and angle-based IFA loss calculation.

Panel of Figure 4 — Figure 4: UMAP visualization of the global latent representation $g(T_{k})$ obtained when not enforcing the IFA loss (left) or when enforcing it (right). Each color represents the global latent representations of one task.

	LIBERO-OBJECT			LIBERO-GOAL
	T6	T7	T8	T6	T7	T8
w/o	0.1265	0.1374	0.3590	0.2191	0.2319	0.3652
w	0.1197	0.1218	0.3482	0.1977	0.2345	0.3877

Table 6: Average cosine similarity between task 8 embeddings and references from other tasks after lifelong training. “w/o” and “w” denote training without and with IFA, respectively (see main text for details).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$\alpha$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
LIBERO-OBJECT	0.10	81.8 $\pm$ 3.2	12.4 $\pm$ 3.6	77.3 $\pm$ 1.4
	0.30	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5
	0.50	82.1 $\pm$ 0.7	18.3 $\pm$ 10.3	73.2 $\pm$ 4.1
	0.70	81.7 $\pm$ 0.7	8.4 $\pm$ 3.3	77.3 $\pm$ 2.4
LIBERO-GOAL	0.10	80.8 $\pm$ 4.0	12.0 $\pm$ 11.3	75.4 $\pm$ 8.8
	0.30	81.3 $\pm$ 3.3	15.2 $\pm$ 7.3	74.7 $\pm$ 6.7

Appendix

Table 7: Ablation on different angular scaling factor

\alpha

in the adaptive IFA.

	LIBERO-OBJECT			LIBERO-GOAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
R=8	57.5 $\pm$ 0.2	7.9 $\pm$ 1.0	52.8 $\pm$ 5.9	59.3 $\pm$ 0.9	11.5 $\pm$ 3.3	54.1 $\pm$ 0.7
R=16	48.9 $\pm$ 7.1	-5.6 $\pm$ 0.4	50.4 $\pm$ 6.9	56.3 $\pm$ 5.3	4.4 $\pm$ 18.9	53.6 $\pm$ 2.4
R=32	53.8 $\pm$ 0.2	-0.2 $\pm$ 6.7	54.2 $\pm$ 2.3	63.8 $\pm$ 7.1	7.5 $\pm$ 13.0	60.0 $\pm$ 1.5
FFT	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8

Appendix

Table 8: Comparison between LoRA adapters with different ranks (R) and full fine-tuning (FFT).

	LIBERO-OBJECT			LIBERO-GOAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
w/o FiLM	43.8 $\pm$ 3.5	8.8 $\pm$ 7.7	41.6 $\pm$ 0.2	57.6 $\pm$ 17.7	0.83 $\pm$ 7.5	56.3 $\pm$ 2.1
w FiLM	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8

Appendix

Table 9: Effect of using FiLM layers during lifelong learning.

	LIBERO-OBJECT				LIBERO-GOAL
Task Reference	Task 6	Task 7	Task 8	Task 9	Task 6	Task 7	Task 8	Task 9
Mean global	0.565	0.555	0.549	0.454	0.485	0.354	0.501	0.437
Global	0.811	0.802	0.797	0.703	0.735	0.583	0.752	0.684
Language	0.668	0.701	0.656	0.677	0.591	0.610	0.629	0.680
AgentView	0.947	0.951	0.973	0.994	0.976	0.957	0.980	1.051
Eye-in-hand	1.008	0.974	1.005	1.005	1.018	1.037	0.987	1.042

Appendix

Table 10: Task-reference comparison on LIBERO-OBJECT and LIBERO-GOAL . See text for the explanation of “Mean Global” and “Global”. “Language”, “AgentView”, “Eye-in-hand”, and “State” use the corresponding modality-specific latent feature as the task reference. The smallest cosine distance is highlighted in bold , and the second smallest is underlined .

Figure 5: UMAP visualization of the global latent representations $g(T_{k})$ obtained when not enforcing the IFA loss, each color represents the global latent representations of one task.

Panel of Figure 6 — Figure 6: Illustration of the cosine function and its inverse.

Method	$P$	Forward Time (ms)	S-FLOPs	T-FLOPs
BASE	-	75.5 $\pm$ 2.43	$3.63\times 10^{11}$	$4.36\times 10^{17}$
BASE + RR	0.5	75.8 $\pm$ 2.80	$3.63\times 10^{11}$	$12.96\times 10^{17}$
BASE + MLR	0.5	75.5 $\pm$ 2.43	$3.63\times 10^{11}$	$7.62\times 10^{17}$
BASE + MLR + IFA	0.5	75.8 $\pm$ 2.80	$3.63\times 10^{11}$	$7.62\times 10^{17}$
BASE + MLR + IFA	0.2	75.8 $\pm$ 2.43	$3.63\times 10^{11}$	$5.66\times 10^{17}$
BASE + MLR + IFA	0.1	75.8 $\pm$ 2.43	$3.63\times 10^{11}$	$5.01\times 10^{17}$

Appendix

Table 11: Runtime and FLOPs comparison. See text for the explanation of Forward Time, S-FLOPs, and T-FLOPs. All experiments are conducted on LIBERO-GOAL .

P

means the probability of storing features in the buffer, which is ablated in Tab. 5 . RR stands for raw replay, which stores past samples and replays them during training. Note that BASE stands for running our method without MLR or IFA. It is included solely to quantify the computational cost of the base architecture.

Explore similar work

Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retention
Sep 30, 2026Siddeshwar Raghavan, Ziqin Yuan, Fengqing Zhu +1Robot Imitation Learning
Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA. · Department of Computer and Information Technology,Purdue University, West Lafayette, IN, USA. · Computer Science and Intelligent Systems Engineering, Indiana University Bloomington, IN, USA.
SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning
Mar 9, 2026Kaushik Roy, Giovanni D'urso, Nicholas Lawrance +2Imitation Learning Robot Imitation Learning
CSIRO Robotics, DATA61, CSIRO, Australia. · Queensland University of Technology (QUT), Brisbane, Australia.
Reinforcement Learning-Guided Retrieval with Soft Fusion for Robust Multimodal Imitation Learning under Missing Modalities
Jun 13, 2026Hassan Ismkhan, Hamid BouchahciaMissing-Modality Learning Multimodal Robustness
Bournemouth University Bournemouth, UK

Sep 30, 2026J/K move · Enter open · S save

Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retention

Siddeshwar Raghavan, Ziqin Yuan, Fengqing Zhu, Byung-Cheol Min

Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA. · Department of Computer and Information Technology,Purdue University, West Lafayette, IN, USA. · Computer Science and Intelligent Systems Engineering, Indiana University Bloomington, IN, USA.

Robot Imitation Learning

Open paper AskPDF arXiv

Continual imitation learning evaluates whether a robot can learn new knowledge without forgetting previously learned skills. However, retaining task performance does not ensure the behavior remains grounded in language because policies may rely on scene cues, object associations, or memorized task structure. We introduce a benchmark protocol to study how language-guided behavior changes as robotic policies learn successive tasks. We construct meaning-preserving and meaning-changing instruction variants for the Goal, Spatial, Object, and Long suites of LIBERO. Policy experiments focus on LIBERO-Goal, evaluating Original and Paraphrase instructions after each continual-learning stage. We compare representative continual imitation learning methods under their original assumptions while separating task competence from language sensitivity. The proposed diagnostics complement standard learning and forgetting metrics by measuring semantic robustness, goal adaptation, and language sensitivity. Results show that strong continual-learning performance does not always translate to reliable language grounding, and our diagnostics help determine whether retained skills remain correctly guided by their instructions. Additional materials are available at https://sites.google.com/view/stillgrounded

Abstract

Explore similar work

Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retention

SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning

Reinforcement Learning-Guided Retrieval with Soft Fusion for Robust Multimodal Imitation Learning under Missing Modalities