Lifelong Imitation Learning with Multimodal Latent Replay and Incremental Adjustment

Fanqi Yu; Matteo Tiezzi; Tommaso Apicella; Cigdem Beyan; Vittorio Murino

cs.CVMar 11, 2026

Lifelong Imitation Learning with Multimodal Latent Replay and Incremental Adjustment

Authors: Fanqi Yu, Matteo Tiezzi, Tommaso Apicella, Cigdem Beyan, Vittorio Murino

Organizations: AI for Good (AIGO), Istituto Italiano di Tecnologia, Genoa, Italy · DITEN, University of Genoa, Genoa, Italy · PAVIS, Istituto Italiano di Tecnologia, Genoa, Italy · Department of Computer Science, University of Verona, Verona, Italy

Abstract

We introduce a lifelong imitation learning framework that enables continual policy refinement across sequential tasks under realistic memory and data constraints. Our approach departs from conventional experience replay by operating entirely in a multimodal latent space, where compact representations of visual, linguistic, and robot's state information are stored and reused to support future learning. To further stabilize adaptation, we introduce an incremental feature adjustment mechanism that regularizes the evolution of task embeddings through an angular margin constraint, preserving inter-task distinctiveness. Our method establishes a new state of the art in the LIBERO benchmarks, achieving 10-17 point gains in AUC and up to 65% less forgetting compared to previous leading methods. Ablation studies confirm the effectiveness of each component, showing consistent gains over alternative strategies. The code is available at: https://github.com/yfqi/lifelong_mlr_ifa.

Figures & tables

Figure 1: Illustration of Incremental Feature Adjustment (IFA). The figure displays a 2D projection of the global latent representations $\mathbf{g}$ during policy rollout for two related tasks, $T_{j}$ (previously learned) and $T_{k}$ (newly learned). The stars ( $\mathbf{\color[rgb]{1,0,0}\star}$ for $T_{j}$ and $\mathbf{\color[rgb]{0,1,0}\star}$ for $T_{k}$ ) represent the stable language reference embeddings $\mathbf{h}^{(r)}$ , while the circles ( $\mathbf{\color[rgb]{1,0,0}\bullet}$ for $T_{j}$ and $\mathbf{\color[rgb]{0,1,0}\bullet}$ for $T_{k}$ ) are the global embeddings $\mathbf{g}$ . (Left) Without IFA, the new task’s embeddings ( $\mathbf{\color[rgb]{0,1,0}\bullet}$ ) exhibit representation drift by clustering close to the old task’s embeddings ( $\mathbf{\color[rgb]{1,0,0}\bullet}$ ). (Right) With IFA, the loss $\mathcal{L}_{\text{IFA}}$ enforces a constraint on the distances: the distance to the own reference ( $D_{\text{own}}$ ) plus a margin $\delta$ must be less than or equal to the distance to the other task’s reference ( $D_{\text{other}}$ ). This mechanism forces the $\mathbf{g}(T_{k})$ ( $\mathbf{\color[rgb]{0,1,0}\bullet}$ ) cluster away from $\mathbf{h}^{(r)}(T_{j})$ ( $\mathbf{\color[rgb]{1,0,0}\star}$ ) and closer to $\mathbf{h}^{(r)}(T_{k})$ ( $\mathbf{\color[rgb]{0,1,0}\star}$ ) , achieving inter-task disentanglement.

Figure 2: Our method is a general multimodal architecture composed of modality-specific encoders (language, vision, and state), a modulation network, a temporal decoder, and a policy head. During the pretraining phase, all architecture modules are trained. In the lifelong learning phase, only the temporal decoder and policy head are updated using both the new task data and the samples stochastically stored in the replay buffer. The buffer stores the multimodal features, output of the modulation layer. The model is jointly supervised by both the Behavior Cloning and the Incremental Feature Adjustment loss, processing the current task and previously stored tasks.

Method	LIBERO-OBJECT			LIBERO-GOAL			LIBERO-50
Method	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
Sequential [ 14 ]	62.0 $\pm$ 1.0	63.0 $\pm$ 2.0	30.0 $\pm$ 1.0	55.0 $\pm$ 1.0	70.0 $\pm$ 1.0	23.0 $\pm$ 1.0	32.0 $\pm$ 1.0	90.0 $\pm$ 2.0	14.0 $\pm$ 2.0
ER [ 14 , 2 ]	56.0 $\pm$ 1.0	24.0 $\pm$ 1.0	49.0 $\pm$ 1.0	53.0 $\pm$ 1.0	36.0 $\pm$ 1.0	47.0 $\pm$ 2.0	35.0 $\pm$ 3.0	49.0 $\pm$ 1.0	36.0 $\pm$ 3.0
BUDS [ 33 , 26 ]	52.0 $\pm$ 2.0	21.0 $\pm$ 1.0	47.0 $\pm$ 1.0	50.0 $\pm$ 1.0	39.0 $\pm$ 1.0	42.0 $\pm$ 1.0	29 $\pm$ 3.0	50.0 $\pm$ 4.0	33.0 $\pm$ 3.0
LOTUS [ 26 ]	74.0 $\pm$ 3.0	11.0 $\pm$ 1.0	65.0 $\pm$ 3.0	61.0 $\pm$ 3.0	30.0 $\pm$ 1.0	56.0 $\pm$ 1.0	39 $\pm$ 2.0	43.0 $\pm$ 1.0	45.0 $\pm$ 2.0
ISCIL [ 10 ]	71.7 $\pm$ 1.9	11.9 $\pm$ 5.1	66.3 $\pm$ 3.7	70.4 $\pm$ 3.6	19.4 $\pm$ 4.1	60.5 $\pm$ 2.5	47.8 $\pm$ 1.3	15.0 $\pm$ 4.8	37.7 $\pm$ 2.1

Table 1: Comparison of results on the LIBERO benchmarks (mean

\pm

standard deviation). “NA” indicates not available. Best results are shown in bold.

Modality	LIBERO-OBJECT			LIBERO-GOAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
AV	83.8 $\pm$ 2.2	11.1 $\pm$ 10.4	78.9 $\pm$ 2.6	77.7 $\pm$ 2.6	7.8 $\pm$ 2.7	73.9 $\pm$ 2.6
Lan	83.3 $\pm$ 0.7	13.5 $\pm$ 3.1	77.3 $\pm$ 1.9	77.1 $\pm$ 1.9	12.3 $\pm$ 0.8	71.8 $\pm$ 2.0
EIH	84.4 $\pm$ 0.9	16.1 $\pm$ 3.1	77.1 $\pm$ 1.1	75.0 $\pm$ 3.3	5.6 $\pm$ 3.9	72.7 $\pm$ 4.2
Act	84.6 $\pm$ 2.6	20.7 $\pm$ 3.6	75.1 $\pm$ 1.4	73.8 $\pm$ 2.5	2.5 $\pm$ 4.2	73.2 $\pm$ 1.1
Sta	82.5 $\pm$ 3.3	11.2 $\pm$ 10.6	77.5 $\pm$ 2.6	78.3 $\pm$ 5.1	8.2 $\pm$ 2.4	74.9 $\pm$ 5.9

Table 2: Influence of different modality similarity on the proposed method’s performance. “AV”, “Lan”, “EIH”, “Act”, and “Sta” denote Agent-view, Language, Eye-in-hand, Action, and State modalities, respectively.

	Buffer	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
LIBERO-OBJECT	33.3%	83.3 $\pm$ 2.6	12.3 $\pm$ 2.4	77.6 $\pm$ 3.0
	50%	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5
	66.6%	83.3 $\pm$ 1.4	16.4 $\pm$ 9.0	75.8 $\pm$ 5.3
LIBERO-GOAL	33.3%	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8
	50%	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8
	66.6%	82.0 $\pm$ 0.9	15.4 $\pm$ 16.7	75.6 $\pm$ 8.7

Table 3: Ablation on task pair selection proportion used when calculating IFA.

	Reference	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
LIBERO-OBJECT	Mean global	79.6 $\pm$ 7.21	8.3 $\pm$ 12.2	75.7 $\pm$ 1.1
LIBERO-OBJECT	Lan+agent view	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5
LIBERO-GOAL	Mean global	77.1 $\pm$ 3.1	15.7 $\pm$ 6.1	70.5 $\pm$ 6.0
LIBERO-GOAL	Lan+agent view	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8

Table 4: Effectiveness of choosing different references. “Lan” stands for language. See the text for a description of the “mean global” reference.

	Probability	Storage	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
LIBERO-OBJECT	0.50	188MB	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5
	0.20	75.3MB	80.5 $\pm$ 7.5	12.5 $\pm$ 7.1	77.7 $\pm$ 10.0
	0.10	37.6MB	79.9 $\pm$ 12.6	13.4 $\pm$ 6.1	76.6 $\pm$ 14.1
LIBERO-GOAL	0.50	121.2MB	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8
	0.20	60.6MB	78.3 $\pm$ 1.9	10.6 $\pm$ 8.0	77.4 $\pm$ 4.6
	0.10	30.3MB	78.7 $\pm$ 1.3	25.2 $\pm$ 10.4	66.2 $\pm$ 6.8

Table 5: Impact of varying the probability of storing features in the buffer, affecting buffer size.

Figure 3: Comparison between cosine distance and angle-based IFA loss calculation.

Panel of Figure 4 — Figure 4: UMAP visualization of the global latent representation $g(T_{k})$ obtained when not enforcing the IFA loss (left) or when enforcing it (right). Each color represents the global latent representations of one task.

	LIBERO-OBJECT			LIBERO-GOAL
	T6	T7	T8	T6	T7	T8
w/o	0.1265	0.1374	0.3590	0.2191	0.2319	0.3652
w	0.1197	0.1218	0.3482	0.1977	0.2345	0.3877

Table 6: Average cosine similarity between task 8 embeddings and references from other tasks after lifelong training. “w/o” and “w” denote training without and with IFA, respectively (see main text for details).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$\alpha$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
LIBERO-OBJECT	0.10	81.8 $\pm$ 3.2	12.4 $\pm$ 3.6	77.3 $\pm$ 1.4
	0.30	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5
	0.50	82.1 $\pm$ 0.7	18.3 $\pm$ 10.3	73.2 $\pm$ 4.1
	0.70	81.7 $\pm$ 0.7	8.4 $\pm$ 3.3	77.3 $\pm$ 2.4
LIBERO-GOAL	0.10	80.8 $\pm$ 4.0	12.0 $\pm$ 11.3	75.4 $\pm$ 8.8
	0.30	81.3 $\pm$ 3.3	15.2 $\pm$ 7.3	74.7 $\pm$ 6.7

Appendix

Table 7: Ablation on different angular scaling factor

\alpha

in the adaptive IFA.

	LIBERO-OBJECT			LIBERO-GOAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
R=8	57.5 $\pm$ 0.2	7.9 $\pm$ 1.0	52.8 $\pm$ 5.9	59.3 $\pm$ 0.9	11.5 $\pm$ 3.3	54.1 $\pm$ 0.7
R=16	48.9 $\pm$ 7.1	-5.6 $\pm$ 0.4	50.4 $\pm$ 6.9	56.3 $\pm$ 5.3	4.4 $\pm$ 18.9	53.6 $\pm$ 2.4
R=32	53.8 $\pm$ 0.2	-0.2 $\pm$ 6.7	54.2 $\pm$ 2.3	63.8 $\pm$ 7.1	7.5 $\pm$ 13.0	60.0 $\pm$ 1.5
FFT	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8

Appendix

Table 8: Comparison between LoRA adapters with different ranks (R) and full fine-tuning (FFT).

	LIBERO-OBJECT			LIBERO-GOAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
w/o FiLM	43.8 $\pm$ 3.5	8.8 $\pm$ 7.7	41.6 $\pm$ 0.2	57.6 $\pm$ 17.7	0.83 $\pm$ 7.5	56.3 $\pm$ 2.1
w FiLM	84.6 $\pm$ 1.9	11.4 $\pm$ 5.6	79.4 $\pm$ 1.5	80.0 $\pm$ 2.5	6.9 $\pm$ 0.9	77.2 $\pm$ 1.8

Appendix

Table 9: Effect of using FiLM layers during lifelong learning.

	LIBERO-OBJECT				LIBERO-GOAL
Task Reference	Task 6	Task 7	Task 8	Task 9	Task 6	Task 7	Task 8	Task 9
Mean global	0.565	0.555	0.549	0.454	0.485	0.354	0.501	0.437
Global	0.811	0.802	0.797	0.703	0.735	0.583	0.752	0.684
Language	0.668	0.701	0.656	0.677	0.591	0.610	0.629	0.680
AgentView	0.947	0.951	0.973	0.994	0.976	0.957	0.980	1.051
Eye-in-hand	1.008	0.974	1.005	1.005	1.018	1.037	0.987	1.042

Appendix

Table 10: Task-reference comparison on LIBERO-OBJECT and LIBERO-GOAL . See text for the explanation of “Mean Global” and “Global”. “Language”, “AgentView”, “Eye-in-hand”, and “State” use the corresponding modality-specific latent feature as the task reference. The smallest cosine distance is highlighted in bold , and the second smallest is underlined .

Figure 5: UMAP visualization of the global latent representations $g(T_{k})$ obtained when not enforcing the IFA loss, each color represents the global latent representations of one task.

Panel of Figure 6 — Figure 6: Illustration of the cosine function and its inverse.

Method	$P$	Forward Time (ms)	S-FLOPs	T-FLOPs
BASE	-	75.5 $\pm$ 2.43	$3.63\times 10^{11}$	$4.36\times 10^{17}$
BASE + RR	0.5	75.8 $\pm$ 2.80	$3.63\times 10^{11}$	$12.96\times 10^{17}$
BASE + MLR	0.5	75.5 $\pm$ 2.43	$3.63\times 10^{11}$	$7.62\times 10^{17}$
BASE + MLR + IFA	0.5	75.8 $\pm$ 2.80	$3.63\times 10^{11}$	$7.62\times 10^{17}$
BASE + MLR + IFA	0.2	75.8 $\pm$ 2.43	$3.63\times 10^{11}$	$5.66\times 10^{17}$
BASE + MLR + IFA	0.1	75.8 $\pm$ 2.43	$3.63\times 10^{11}$	$5.01\times 10^{17}$

Appendix

Table 11: Runtime and FLOPs comparison. See text for the explanation of Forward Time, S-FLOPs, and T-FLOPs. All experiments are conducted on LIBERO-GOAL .

P

means the probability of storing features in the buffer, which is ablated in Tab. 5 . RR stands for raw replay, which stores past samples and replays them during training. Note that BASE stands for running our method without MLR or IFA. It is included solely to quantify the computational cost of the base architecture.

Explore similar work

Sep 30, 2026

Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retention

Continual imitation learning evaluates whether a robot can learn new knowledge without forgetting previously learned skills. However, retaining task performance does not ensure the behavior remains grounded in language because policies may rely on scene cues, object associations, or memorized task structure. We introduce a benchmark protocol to study how language-guided behavior changes as robotic policies learn successive tasks. We construct meaning-preserving and meaning-changing instruction variants for the Goal, Spatial, Object, and Long suites of LIBERO. Policy experiments focus on LIBERO-Goal, evaluating Original and Paraphrase instructions after each continual-learning stage. We compare representative continual imitation learning methods under their original assumptions while separating task competence from language sensitivity. The proposed diagnostics complement standard learning and forgetting metrics by measuring semantic robustness, goal adaptation, and language sensitivity. Results show that strong continual-learning performance does not always translate to reliable language grounding, and our diagnostics help determine whether retained skills remain correctly guided by their instructions. Additional materials are available at https://sites.google.com/view/stillgrounded

Siddeshwar Raghavan, Ziqin Yuan, Fengqing Zhu +1

Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA. · Department of Computer and Information Technology,Purdue University, West Lafayette, IN, USA. · Computer Science and Intelligent Systems Engineering, Indiana University Bloomington, IN, USA.

Robot Imitation Learning

Ask PDF

Mar 9, 2026

SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning

A key challenge in lifelong imitation learning (LIL) is enabling agents to acquire new skills from expert demonstrations while retaining prior knowledge. This requires preserving the low-dimensional manifolds and geometric structures that underlie task representations across sequential learning. Existing distillation methods, which rely on L2-norm feature matching in raw feature space, are sensitive to noise and high-dimensional variability, often failing to preserve intrinsic task manifolds. To address this, we introduce SPREAD, a geometry-preserving framework that employs singular value decomposition (SVD) to align policy representations across tasks within low-rank subspaces. This alignment maintains the underlying geometry of multimodal features, facilitating stable transfer, robustness, and generalization. Additionally, we propose a confidence-guided distillation strategy that applies a Kullback-Leibler divergence loss restricted to the top-M most confident action samples, emphasizing reliable modes and improving optimization stability. Experiments on the LIBERO, lifelong imitation learning benchmark, show that SPREAD substantially improves knowledge transfer, mitigates catastrophic forgetting, and achieves state-of-the-art performance.

Kaushik Roy, Giovanni D'urso, Nicholas Lawrance +2

CSIRO Robotics, DATA61, CSIRO, Australia. · Queensland University of Technology (QUT), Brisbane, Australia.

Imitation Learning Robot Imitation Learning Knowledge Distillation Feature-Based Knowledge Distillation Selective Knowledge Distillation

Ask PDF

Jun 13, 2026

Reinforcement Learning-Guided Retrieval with Soft Fusion for Robust Multimodal Imitation Learning under Missing Modalities

Robotic systems perceive the world through multiple input modalities -- including visual camera streams and natural language instructions -- and must select appropriate actions based on these signals. However, assuming the permanent availability of all input devices is unrealistic, as sensors may fail, become occluded, or drop out entirely during deployment. Robust handling of such missing-modality scenarios is therefore essential for real-world robot operation. This paper introduces RL4IL, a reinforcement learning guided method for imitation learning that selects the most suitable action for a given observation by identifying the most relevant expert demonstrations from a training library. A reinforcement learning policy, trained via Proximal Policy Optimisation over Breadth-First Search candidate sets, ranks candidate demonstrations and a soft cross-attention fusion head aggregates their action signals to produce the final prediction. When a modality is missing at inference time, a dedicated per-modality RL retrieval policy identifies donor demonstrations from the training library, and a soft imputation head reconstructs the missing embedding via cross-attention over the top-ranked donors -- without requiring any retraining of the system. Experiments on three LIBERO benchmark suites demonstrate that RL4IL substantially outperforms state-of-the-art imitation learning methods under sensor dropout conditions, while requiring no policy network training. The code can be found at https://github.com/h-ismkhan/Reinforcement-Learning-via-kNN-for-Robotic-Learning-with-Missing-Camera

Hassan Ismkhan, Hamid Bouchahcia

Bournemouth University Bournemouth, UK

Missing-Modality Learning Multimodal Robustness Imitation Learning Interactive Imitation Learning Robot Imitation Learning

Ask PDF