HIDRA: Hierarchical Dual-Routing Attention for Replay-Free Lifelong Imitation Learning

Fanqi Yu; Matteo Tiezzi; Cigdem Beyan; Tommaso Apicella; Vittorio Murino

cs.ROOct 26, 2026

HIDRA: Hierarchical Dual-Routing Attention for Replay-Free Lifelong Imitation Learning

Authors: Fanqi Yu, Matteo Tiezzi, Cigdem Beyan, Tommaso Apicella, Vittorio Murino

Organizations: AI for Good (AIGO) Research Unit, Istituto Italiano di Tecnologia, Genoa, Italy · DITEN, University of Genoa, Genoa, Italy · Pattern Analysis and Computer Vision (PAVIS), Istituto Italiano di Tecnologia, Genoa, Italy · Department of Computer Science, University of Verona, Verona, Italy

Abstract

Robotic agents operating in real-world environments must adapt continuously from a stream of multimodal demonstrations, often under constraints that preclude storing or revisiting past data. In this replay-free, single-pass lifelong imitation learning setting, sequential updates can progressively distort the representation space, making expert-based approaches particularly sensitive to routing interference, causing expert selection to degrade over time. We propose HIDRA, a structured routing framework that mitigates this issue through a hierarchical dual-attention mechanism, which decouples instruction-level expert retrieval from context-dependent refinement. To further stabilize routing under representation drift, HIDRA introduces a key-level geometric regularization that enforces both alignment within tasks and separation across tasks. The resulting approach enables reliable expert selection while supporting an expandable set of experts for continual adaptation. We evaluate HIDRA on several LIBERO benchmark suites, including more challenging variants with heterogeneous task sequences and paraphrased language instructions. Our approach consistently outperforms both replay-free and replay-based baselines, improving AUC and forward transfer while maintaining low forgetting, with stronger gains under high interference and language variation. The code is available at: https://github.com/yfqi/HIDRA.

Figures & tables

Figure 1: Left: Existing methods (e.g., Lei et al. (2025) ) perform expert routing using a single projection over entangled representations, which is unstable without replay. Right: hidra decouples routing into instruction retrieval and context refinement with regularization, ensuring reliable expert selection in replay-free LIL.

Figure 2: Overview of hidra . A frozen pretrained multimodal policy produces an instruction query $q_{t}^{\mathrm{ins}}$ and a context query $q_{t}^{\mathrm{ctx}}$ . These drive a two-stage hierarchical dual-routing attention mechanism. Stage I performs instruction-level retrieval via $q_{t}^{\mathrm{ins}}$ to select an anchor expert $j^{*}$ and an instruction candidate set $\mathcal{S}_{j^{*}}$ over task-level keys $\{k_{j}^{\mathrm{ins}}\}$ . Stage II then uses $q_{t}^{\mathrm{ctx}}$ to perform context-level refinement within $\mathcal{S}_{j^{*}}$ using context keys $\{k_{j}^{\mathrm{ctx}}\}$ , producing fused routing scores and selecting the active experts. Key-space regularization (Reg.) promotes intra-task compactness and inter-task separation, enabling stable and interference-resistant expert selection as new task-specific experts are introduced during replay-free lifelong adaptation.

Method	Replay	LIBERO-OBJECT			LIBERO-GOAL			LIBERO-SPATIAL			LIBERO-10
	Free	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
ER Liu et al. (2023a) ; Chaudhry et al. (2019)	✗	46.0 $\pm$ 3.6	53.3 $\pm$ 3.2	19.8 $\pm$ 2.9	36.0 $\pm$ 2.6	17.3 $\pm$ 3.4	23.6 $\pm$ 3.3	15.0 $\pm$ 2.8	-1.8 $\pm$ 4.5	16.2 $\pm$ 3.7	7.0 $\pm$ 2.9	4.7 $\pm$ 4.0	3.4 $\pm$ 3.5
MLR-IFA Yu et al. (2026)	✗	34.0 $\pm$ 2.0	-6.2 $\pm$ 2.4	38.7 $\pm$ 2.2	43.0 $\pm$ 2.4	17.3 $\pm$ 1.6	38.2 $\pm$ 2.1	27.0 $\pm$ 1.5	10.1 $\pm$ 1.9	19.9 $\pm$ 2.0	21.1 $\pm$ 2.7	-4.9 $\pm$ 3.0	24.6 $\pm$ 2.5
DMPEL (50%) Lei et al. (2025)	✗	77.0 $\pm$ 1.2	-4.4 $\pm$ 0.6	78.0 $\pm$ 1.6	62.0 $\pm$ 1.8	4.2 $\pm$ 1.5	61.0 $\pm$ 1.7	46.0 $\pm$ 2.9	2.7 $\pm$ 2.6	43.8 $\pm$ 2.0	62.0 $\pm$ 1.5	7.9 $\pm$ 3.0	54.5 $\pm$ 1.5
DMPEL (100%) Lei et al. (2025)	✗	82.0 $\pm$ 1.0	-0.7 $\pm$ 0.8	82.2 $\pm$ 1.4	64.0 $\pm$ 1.6	-0.4 $\pm$ 1.4	64.3 $\pm$ 1.8	44.0 $\pm$ 2.6	-4.3 $\pm$ 1.2	47.0 $\pm$ 2.3	56.5 $\pm$ 2.6	4.7 $\pm$ 0.3	53.8 $\pm$ 2.6
L2M Schmied et al. (2023)	✓	1.0 $\pm$ 0.5	0.9 $\pm$ 0	0.7 $\pm$ 0.2	4.0 $\pm$ 0.6	2.8 $\pm$ 0.5	2.3 $\pm$ 0	8.0 $\pm$ 0.6	3.1 $\pm$ 0.4	5.6 $\pm$ 0.4	7.0 $\pm$ 0.8	4.5 $\pm$ 0.3	3.7 $\pm$ 1.1

Table 1: Comparison on LIBERO benchmarks (mean

\pm

std). Best shown in bold.

	LIBERO-MIX @ Inc1			LIBERO-MIX @ Inc4			LIBERO-INS
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
MLR-IFA Yu et al. (2026)	28.5 $\pm$ 3.4	4.5 $\pm$ 2.0	25.0 $\pm$ 2.1	21.5 $\pm$ 3.0	-5.2 $\pm$ 2.2	24.7 $\pm$ 2.8	4.0 $\pm$ 1.8	-4.9 $\pm$ 6.2	7.9 $\pm$ 2.6
DMPEL (100%) Lei et al. (2025)	53.0 $\pm$ 1.0	-3.9 $\pm$ 2.8	55.2 $\pm$ 2.5	61.0 $\pm$ 1.7	2.4 $\pm$ 0.9	59.9 $\pm$ 1.4	23.0 $\pm$ 3.6	-6.1 $\pm$ 8.0	27.1 $\pm$ 4.3
ISCIL Lee et al. (2024)	21.1 $\pm$ 2.9	0.4 $\pm$ 1.6	20.6 $\pm$ 2.8	14.8 $\pm$ 3.5	-0.3 $\pm$ 1.3	14.7 $\pm$ 0.4	11.0 $\pm$ 2.7	1.7 $\pm$ 2.1	10.0 $\pm$ 3.2
hidra (Ours)	71.3 $\pm$ 2.1	2.8 $\pm$ 1.7	68.0 $\pm$ 1.0	66.2 $\pm$ 2.5	-2.0 $\pm$ 3.2	67.0 $\pm$ 1.2	25.5 $\pm$ 1.9	-3.0 $\pm$ 2.4	27.6 $\pm$ 3.8

Table 2: Comparison of results on the LIBERO-MIX and LIBERO-INS benchmarks (mean

\pm

std). Best shown in bold.

	LIBERO-OBJECT			LIBERO-SPATIAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
Simple Routing (no attention)	71.0 $\pm$ 3.7	62.0 $\pm$ 2.9	26.1 $\pm$ 2.5	43.5 $\pm$ 2.5	47.1 $\pm$ 3.8	9.2 $\pm$ 2.0
$\text{SingleAttn}(a^{\text{ctx}}_{j,t})$	72.0 $\pm$ 1.9	2.5 $\pm$ 2.1	70.0 $\pm$ 2.2	41.0 $\pm$ 3.4	-0.3 $\pm$ 2.6	41.1 $\pm$ 3.1
$\text{SingleAttn}(a^{\text{ins}}_{j,t})$	83.0 $\pm$ 2.4	-0.5 $\pm$ 1.2	83.2 $\pm$ 2.0	53.0 $\pm$ 1.6	3.0 $\pm$ 0.7	51.7 $\pm$ 1.6
$\text{DualAttn}(a^{\text{ins}}_{j,t},a^{\text{ctx}}_{j,t})$	85.5 $\pm$ 3.1	-1.0 $\pm$ 2.2	85.8 $\pm$ 2.6	54.5 $\pm$ 1.7	1.7 $\pm$ 0.7	54.2 $\pm$ 1.9
$+\ \mathcal{L}_{\text{intra}}$	87.5 $\pm$ 2.2	-1.9 $\pm$ 1.3	87.4 $\pm$ 2.6	55.5 $\pm$ 2.9	2.1 $\pm$ 2.7	54.6 $\pm$ 3.3

Table 3: Ablation of the routing and regularization components of the proposed method.

Panel of Figure 3 — Figure 3: AUC comparison of prototype estimation (left), inter-task regularization (center), and routing strategies (right) on LIBERO-OBJECT and LIBERO-SPATIAL, respectively. Gray bars denote baseline variants, and blue bars denote our method. See the text for abbreviation definitions.

Panel of Figure 4 — Figure 4: Expert activation heatmaps aggregated at the task-group level under different settings. Each matrix shows the average attention from query task groups (rows) to expert groups (columns). Diagonal entries indicate within-group routing, while off-diagonal entries reflect cross-group activation.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

$\lambda_{\text{intra}}$	$\lambda_{\text{inter}}$	LIBERO-OBJECT			LIBERO-SPATIAL
$\lambda_{\text{intra}}$	$\lambda_{\text{inter}}$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
1.0	0.01	93.5 $\pm$ 1.2	6.4 $\pm$ 1.7	89.0 $\pm$ 0.5	56.0 $\pm$ 4.0	0.4 $\pm$ 4.5	56.3 $\pm$ 1.4
1.0	0.1	84.0 $\pm$ 2.1	-2.4 $\pm$ 1.1	85.6 $\pm$ 0.9	51.0 $\pm$ 1.8	-4.8 $\pm$ 2.3	54.4 $\pm$ 1.7
0.01	1.0	87.0 $\pm$ 0.7	-1.3 $\pm$ 1.2	87.4 $\pm$ 0.6	52.0 $\pm$ 1.3	-0.4 $\pm$ 2.5	52.4 $\pm$ 2.7
0.1	1.0	86.5 $\pm$ 2.1	-0.1 $\pm$ 1.2	86.6 $\pm$ 1.8	54.5 $\pm$ 1.6	3.0 $\pm$ 3.2	52.8 $\pm$ 2.1
1.0	1.0	93.5 $\pm$ 1.5	2.2 $\pm$ 1.9	91.5 $\pm$ 0.1	58.0 $\pm$ 1.5	-1.4 $\pm$ 0.8	58.7 $\pm$ 1.0

Appendix

Table 4: The impact of the weighting coefficients

\lambda_{\text{intra}}

and

\lambda_{\text{inter}}

# Experts	LIBERO-OBJECT			LIBERO-SPATIAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
2	89.0 $\pm$ 1.6	0.1 $\pm$ 1.6	88.6 $\pm$ 2.2	53.0 $\pm$ 1.9	-2.6 $\pm$ 1.4	54.7 $\pm$ 2.0
3	93.0 $\pm$ 1.5	2.2 $\pm$ 1.9	91.5 $\pm$ 0.1	58.0 $\pm$ 1.5	-1.4 $\pm$ 0.8	58.7 $\pm$ 1.0
4	87.0 $\pm$ 1.5	2.8 $\pm$ 1.4	84.9 $\pm$ 1.7	56.5 $\pm$ 1.7	-0.2 $\pm$ 1.2	56.3 $\pm$ 1.8

Appendix

Table 5: Influence of the number of selected experts on performance.

MLP Dim.	LIBERO-OBJECT			LIBERO-SPATIAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
4	86.5 $\pm$ 2.4	1.5 $\pm$ 2.4	85.2 $\pm$ 2.0	47.0 $\pm$ 2.6	-5.4 $\pm$ 2.1	50.9 $\pm$ 1.6
8	93.0 $\pm$ 1.5	2.2 $\pm$ 1.9	91.5 $\pm$ 0.1	58.0 $\pm$ 1.5	-1.4 $\pm$ 0.8	58.7 $\pm$ 1.0
16	83.0 $\pm$ 1.4	1.2 $\pm$ 2.0	82.6 $\pm$ 1.6	56.0 $\pm$ 1.2	5.0 $\pm$ 1.3	53.0 $\pm$ 1.4
32	83.0 $\pm$ 1.2	-3.2 $\pm$ 1.5	85.6 $\pm$ 1.2	58.0 $\pm$ 1.6	3.0 $\pm$ 3.0	56.8 $\pm$ 1.8

Appendix

Table 6: Influence of the hidden dimension of the MLP used to compute routing scores. The MLP has a fixed depth of two layers.

	LIBERO-OBJECT			LIBERO-SPATIAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
10	54.0 $\pm$ 2.2	5.9 $\pm$ 1.7	49.4 $\pm$ 0.8	16.5 $\pm$ 1.9	-2.7 $\pm$ 3.0	18.6 $\pm$ 3.5
20	70.0 $\pm$ 4.6	-2.1 $\pm$ 3.6	71.2 $\pm$ 2.4	40.0 $\pm$ 1.6	-0.8 $\pm$ 1.2	40.4 $\pm$ 1.6
50	93.0 $\pm$ 1.5	2.2 $\pm$ 1.9	91.5 $\pm$ 0.1	58.0 $\pm$ 1.5	-1.4 $\pm$ 0.8	58.7 $\pm$ 1.0

Appendix

Table 7: Effect of the number of demonstrations per task.

	LIBERO-OBJECT			LIBERO-SPATIAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
Batch Mean	88.0 $\pm$ 1.9	2.5 $\pm$ 1.4	84.5 $\pm$ 0.4	54.0 $\pm$ 2.0	1.3 $\pm$ 2.5	51.3 $\pm$ 1.2
EMA	81.5 $\pm$ 1.8	-4.3 $\pm$ 1.5	85.1 $\pm$ 2.2	49.0 $\pm$ 2.6	-6.3 $\pm$ 2.4	52.8 $\pm$ 1.9
Oracle Mean	87.0 $\pm$ 1.6	0.3 $\pm$ 1.8	86.1 $\pm$ 2.0	49.5 $\pm$ 2.5	-4.0 $\pm$ 2.5	54.0 $\pm$ 1.3
Oracle EMA	82.0 $\pm$ 3.6	-2.3 $\pm$ 0.8	85.0 $\pm$ 2.2	53.0 $\pm$ 1.6	2.2 $\pm$ 2.5	51.3 $\pm$ 2.1
Running Mean (Ours)	93.0 $\pm$ 1.5	2.2 $\pm$ 1.9	91.5 $\pm$ 0.1	58.0 $\pm$ 1.5	-1.4 $\pm$ 0.8	58.7 $\pm$ 1.0

Appendix

Table 8: Comparison of prototype estimation strategies for intra-task regularization.

	LIBERO-OBJECT			LIBERO-SPATIAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
Hinge loss	82.0 $\pm$ 2.6	-2.6 $\pm$ 1.4	85.5 $\pm$ 2.4	56.5 $\pm$ 2.2	5.0 $\pm$ 2.9	53.5 $\pm$ 1.9
InfoNCE loss	85.0 $\pm$ 1.7	-3.6 $\pm$ 0.9	87.5 $\pm$ 2.1	55.0 $\pm$ 1.8	-2.6 $\pm$ 2.5	56.7 $\pm$ 2.5
Orthogonality loss (Ours)	93.0 $\pm$ 1.5	2.2 $\pm$ 1.9	91.5 $\pm$ 0.1	58.0 $\pm$ 1.5	-1.4 $\pm$ 0.8	58.7 $\pm$ 1.0

Appendix

Table 9: Comparison of inter-task regularization strategies.

	LIBERO-OBJECT			LIBERO-SPATIAL
	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
Mixed Score Ranking	91.5 $\pm$ 1.0	-3.6 $\pm$ 0.8	88.3 $\pm$ 0.5	52.5 $\pm$ 3.6	-2.0 $\pm$ 2.9	55.0 $\pm$ 3.0
Mixed Score Sum	87.5 $\pm$ 2.2	3.1 $\pm$ 1.8	85.6 $\pm$ 2.0	50.0 $\pm$ 3.4	-1.2 $\pm$ 2.8	51.5 $\pm$ 4.0
Context-only (No Mask)	89.5 $\pm$ 1.2	19.1 $\pm$ 1.5	74.0 $\pm$ 1.2	55.0 $\pm$ 2.4	23.1 $\pm$ 3.8	36.9 $\pm$ 3.4
hidra (Ours)	93.0 $\pm$ 1.5	2.2 $\pm$ 1.9	91.5 $\pm$ 0.1	58.0 $\pm$ 1.5	-1.4 $\pm$ 0.8	58.7 $\pm$ 1.0

Appendix

Table 10: Comparison of different routing strategies.

	SG-Count	SG-Attn	CG-Count	CG-Attn	Ent-Count	Ent-Attn
$\text{DualAttn}(a^{\text{ins}}_{j,t},a^{\text{ctx}}_{j,t})$	$0.5829\pm 0.1606$	$0.6830\pm 0.1192$	$0.4171\pm 0.1606$	$0.3170\pm 0.1192$	$1.6799\pm 0.0866$	$1.4780\pm 0.0803$
$+\ \mathcal{L}_{\text{intra}}$	$0.6253\pm 0.1400$	$0.7185\pm 0.0987$	$0.3747\pm 0.1400$	$0.2815\pm 0.0987$	$2.0324\pm 0.0546$	$1.7496\pm 0.0444$
$+\ \mathcal{L}_{\text{intra}}+\mathcal{L}_{\text{inter}}$ (Ours)	$0.6466\pm 0.1454$	$0.7339\pm 0.1040$	$0.3534\pm 0.1454$	$0.2661\pm 0.1040$	$2.0376\pm 0.0942$	$1.7577\pm 0.0787$
DMPEL (100%)	$0.3784\pm 0.0879$	$0.4254\pm 0.1208$	$0.6216\pm 0.0879$	$0.5746\pm 0.1208$	$2.3407\pm 0.0908$	$2.2335\pm 0.0818$

Appendix

Table 11: Routing statistics on the last four tasks of LIBERO-MIX . SG and CG denote same-group and cross-group proportions of selected experts and attention mass, respectively. Entropy measures the dispersion of routing decisions. Higher SG values indicate more semantically consistent routing. See text for more details.

	Forward Time (ms)	FLOPs (per forward)	Train FLOPs
Base (DualAttn)	125.2 $\pm$ 5.0	$5.08\times 10^{11}$	$1.19\times 10^{16}$
+ $\mathcal{L}_{\mathrm{intra}}$	124.8 $\pm$ 3.8	$5.08\times 10^{11}$	$1.19\times 10^{16}$
+ $\mathcal{L}_{\mathrm{intra}}+\mathcal{L}_{\mathrm{inter}}$ (Ours)	125.8 $\pm$ 5.8	$5.08\times 10^{11}$	$1.19\times 10^{16}$
DMPEL (100%)	125.5 $\pm$ 6.0	$5.08\times 10^{11}$	$1.19\times 10^{16}$

Appendix

Table 12: Computational cost on LIBERO-OBJECT . All variants exhibit nearly identical forward latency and FLOPs, indicating that the proposed regularization and routing mechanisms introduce negligible computational overhead.

Component	Storage (KB)	Ratio (%)
Adapter-based experts	2647	95.4
Router parameters	125	4.5
Auxiliary components	3	0.1
Total (Ours)	2773	100
DMPEL (100%)	48000	100

Appendix

Table 13: Per-task memory overhead of our method. Storage is reported in KB and includes all task-specific components. Ratios indicate the contribution of each component to the total overhead. DMPEL is shown for comparison.

LIBERO-OBJECT
Method	3-pass			5-pass
Method	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$	FWT $\uparrow$	NBT $\downarrow$	AUC $\uparrow$
MLR-IFA [ 7 ]	85.7 $\pm$ 1.6	2.5 $\pm$ 0.3	84.3 $\pm$ 0.7	80.0 $\pm$ 1.3	-3.1 $\pm$ 2.7	83.6 $\pm$ 1.4
ISCIL [ 6 ]	85.5 $\pm$ 1.4	-2.5 $\pm$ 2.5	87.3 $\pm$ 1.8	81.5 $\pm$ 1.8	-7.6 $\pm$ 1.9	86.9 $\pm$ 1.2
DMPEL (100%) [ 26 ]	81.0 $\pm$ 1.5	-2.3 $\pm$ 2.2	83.1 $\pm$ 1.9	88.5 $\pm$ 0.9	1.7 $\pm$ 1.5	90.4 $\pm$ 0.8
hidra (Ours)	87.5 $\pm$ 1.0	-1.8 $\pm$ 1.2	88.7 $\pm$ 0.7	90.5 $\pm$ 0.6	-3.6 $\pm$ 1.2	92.8 $\pm$ 0.5

Appendix

Table 14: Performance under multi-pass training (mean

\pm

std). Best in bold .

Figure 5: UMAP visualization of context query representations for all tasks in LIBERO-MIX.

Task	Original instruction during adaptation	Paraphrased instruction during testing
1	put both the alphabet soup and the tomato sauce in the basket	place the alphabet soup and the tomato sauce into the basket
2	put both the cream cheese box and the butter in the basket	move the cream cheese box and the butter into the basket
3	turn on the stove and put the moka pot on it	switch on the stove and place the moka pot on it
4	put the black bowl in the bottom drawer of the cabinet and close it	put the black bowl into the lower drawer of the cabinet, then close the drawer
5	put the white mug on the left plate and put the yellow and white mug on the right plate	place the white mug on the left plate and the yellow and white mug on the right plate
6	pick up the book and place it in the back compartment of the caddy	pick up the book and put it into the rear compartment of the caddy

Appendix

Table 15: Original and paraphrased instructions used for the LIBERO-INS.

Task	Instruction
1	pick up the black bowl between the plate and the ramekin and place it on the plate
2	pick up the alphabet soup and place it in the basket
3	open the middle drawer of the cabinet
4	put both the alphabet soup and the tomato sauce in the basket
5	pick up the black bowl next to the ramekin and place it on the plate
6	pick up the cream cheese and place it in the basket

Appendix

Table 16: Tasks used in LIBERO-MIX.

HIDRA	OBJECT	GOAL	SPATIAL	LIBERO-10
vs MLR-IFA	$<0.001^{*}$	$<0.001^{*}$	$<0.001^{*}$	$<0.001^{*}$
vs ISCIL	$<0.001^{*}$	$<0.001^{*}$	$<0.001^{*}$	$<0.001^{*}$
vs DMPEL (100%)	$<0.001^{*}$	$0.18$	$<0.001^{*}$	$0.09$
vs DMPEL (50%)	$<0.001^{*}$	$0.07$	$<0.001^{*}$	$0.06$

Appendix

Table 17: Approximate statistical significance (Welch’s t-test) for AUC between HIDRA and representative baselines. ∗ indicates

p<0.05

Robot Imitation Learning

Robot imitation learning trains robots to perform tasks by reproducing and adapting behaviors shown in demonstrations.

See all 691 papers

Mar 11, 2026

Lifelong Imitation Learning with Multimodal Latent Replay and Incremental Adjustment

We introduce a lifelong imitation learning framework that enables continual policy refinement across sequential tasks under realistic memory and data constraints. Our approach departs from conventional experience replay by operating entirely in a multimodal latent space, where compact representations of visual, linguistic, and robot's state information are stored and reused to support future learning. To further stabilize adaptation, we introduce an incremental feature adjustment mechanism that regularizes the evolution of task embeddings through an angular margin constraint, preserving inter-task distinctiveness. Our method establishes a new state of the art in the LIBERO benchmarks, achieving 10-17 point gains in AUC and up to 65% less forgetting compared to previous leading methods. Ablation studies confirm the effectiveness of each component, showing consistent gains over alternative strategies. The code is available at: https://github.com/yfqi/lifelong_mlr_ifa.

Fanqi Yu, Matteo Tiezzi, Tommaso Apicella +2

AI for Good (AIGO), Istituto Italiano di Tecnologia, Genoa, Italy · DITEN, University of Genoa, Genoa, Italy · PAVIS, Istituto Italiano di Tecnologia, Genoa, Italy +1

Imitation Learning Robot Imitation Learning Replay-Based Continual Learning

Ask PDF

Mar 9, 2026

SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning

A key challenge in lifelong imitation learning (LIL) is enabling agents to acquire new skills from expert demonstrations while retaining prior knowledge. This requires preserving the low-dimensional manifolds and geometric structures that underlie task representations across sequential learning. Existing distillation methods, which rely on L2-norm feature matching in raw feature space, are sensitive to noise and high-dimensional variability, often failing to preserve intrinsic task manifolds. To address this, we introduce SPREAD, a geometry-preserving framework that employs singular value decomposition (SVD) to align policy representations across tasks within low-rank subspaces. This alignment maintains the underlying geometry of multimodal features, facilitating stable transfer, robustness, and generalization. Additionally, we propose a confidence-guided distillation strategy that applies a Kullback-Leibler divergence loss restricted to the top-M most confident action samples, emphasizing reliable modes and improving optimization stability. Experiments on the LIBERO, lifelong imitation learning benchmark, show that SPREAD substantially improves knowledge transfer, mitigates catastrophic forgetting, and achieves state-of-the-art performance.

Kaushik Roy, Giovanni D'urso, Nicholas Lawrance +2

CSIRO Robotics, DATA61, CSIRO, Australia. · Queensland University of Technology (QUT), Brisbane, Australia.

Imitation Learning Robot Imitation Learning Knowledge Distillation Feature-Based Knowledge Distillation Selective Knowledge Distillation

Ask PDF

Mar 9, 2026

LAR-MoE: Latent-Aligned Routing for Mixture of Experts in Robotic Imitation Learning

Imitation learning enables robots to acquire manipulation skills from demonstrations, yet deploying a policy across tasks with heterogeneous dynamics remains challenging, as models tend to average over distinct behavioral modes present in the demonstrations. Mixture-of-Experts (MoE) architectures address this by activating specialized subnetworks, but requires meaningful skill decompositions for expert routing. We introduce Latent-Aligned Routing for Mixture of Experts (LAR-MoE), a two-stage framework that decouples unsupervised skill discovery from policy learning. In pre-training, we learn a joint latent representation between observations and future actions through student-teacher co-training. In a post-training stage, the expert routing is regularized to follow the structure of the learned latent space, preventing expert collapse while maintaining parameter efficiency. We evaluate LAR-MoE in simulation and on hardware. On the LIBERO benchmark, our method achieves a 95.2% average success rate with 150M parameters. On a surgical bowel grasping and retraction task, LAR-MoE matches a supervised MoE baseline without requiring any phase annotations, and transfers zero-shot to ex vivo porcine tissue. Our findings suggest that latent-aligned routing provides a principled alternative to supervised skill decomposition, enabling structured expert specialization from unlabeled demonstrations.

Ariel Rodriguez, Chenpan Li, Lorenzo Mazza +5

Department of Translational Surgical Oncology, NCT/UCC Dresden, a partnership between DKFZ, MF UKD, TUD, HZDR Dresden, Germany. · Cluster of Excellence-CeTI, TUD, Germany. · Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Germany.

Robot Imitation Learning Agent Skill Learning Robotic Manipulation Mixture-of-Experts Inference

Ask PDF

Abstract

Robot Imitation Learning

Explore similar work

Lifelong Imitation Learning with Multimodal Latent Replay and Incremental Adjustment

SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning

LAR-MoE: Latent-Aligned Routing for Mixture of Experts in Robotic Imitation Learning