Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retention

Siddeshwar Raghavan; Ziqin Yuan; Fengqing Zhu; Byung-Cheol Min

cs.ROSep 30, 2026

Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retention

Authors: Siddeshwar Raghavan, Ziqin Yuan, Fengqing Zhu, Byung-Cheol Min

Organizations: Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA. · Department of Computer and Information Technology,Purdue University, West Lafayette, IN, USA. · Computer Science and Intelligent Systems Engineering, Indiana University Bloomington, IN, USA.

Abstract

Continual imitation learning evaluates whether a robot can learn new knowledge without forgetting previously learned skills. However, retaining task performance does not ensure the behavior remains grounded in language because policies may rely on scene cues, object associations, or memorized task structure. We introduce a benchmark protocol to study how language-guided behavior changes as robotic policies learn successive tasks. We construct meaning-preserving and meaning-changing instruction variants for the Goal, Spatial, Object, and Long suites of LIBERO. Policy experiments focus on LIBERO-Goal, evaluating Original and Paraphrase instructions after each continual-learning stage. We compare representative continual imitation learning methods under their original assumptions while separating task competence from language sensitivity. The proposed diagnostics complement standard learning and forgetting metrics by measuring semantic robustness, goal adaptation, and language sensitivity. Results show that strong continual-learning performance does not always translate to reliable language grounding, and our diagnostics help determine whether retained skills remain correctly guided by their instructions. Additional materials are available at https://sites.google.com/view/stillgrounded

Figures & tables

Fig. 1: Task retention does not guarantee language grounding. Original instructions and paraphrases should lead to the same behavior, while semantic changes may require a different response. The final panel shows one possible response to an incompatible request. Our protocol evaluates these behaviors across continual learning checkpoints rather than relying only on original task success.

Fig. 2: Overview of the benchmark protocol: Original instructions and meaning-preserving paraphrases are evaluated across continual-learning stages, while one-slot minimal contrasts and scene-incompatible collisions assess final-stage goal switching and behavioral sensitivity.

Method	Original AUC $\uparrow$	Paraphrase AUC $\uparrow$	$\Delta_{\mathrm{AUC}}^{\mathrm{para}}$	Original Final SR $\uparrow$	Original NBT $\downarrow$
L2M	7.06 $\pm$ 0.02	5.44 $\pm$ 0.02	1.62 $\pm$ 0.04	5.01 $\pm$ 0.01	8.44 $\pm$ 0.05
SeqFT	24.63 $\pm$ 0.02	19.18 $\pm$ 0.03	5.45 $\pm$ 0.05	8.00 $\pm$ 0.02	76.49 $\pm$ 1.04
ER	$58.54\pm 0.18$	$51.82\pm 0.27$	$6.72\pm 0.11$	$59.23\pm 0.24$	$17.14\pm 0.09$
EWC	$32.77\pm 0.13$	$27.46\pm 0.21$	$5.31\pm 0.07$	$27.41\pm 0.29$	$46.18\pm 0.16$
PackNet	$53.34\pm 0.25$	$46.93\pm 0.14$	$6.41\pm 0.19$	$53.27\pm 0.08$	$15.36\pm 0.31$
DMPEL	$78.33\pm 0.12$	$70.28\pm 0.18$	$8.05\pm 0.22$	$81.12\pm 0.17$	$0.00\pm 0.00$

TABLE I: Semantic invariance on LIBERO-Goal . Metrics include Original AUC (

\uparrow

), Paraphrase AUC (

\uparrow

), signed paraphrase gap

\Delta_{\mathrm{AUC}}^{\mathrm{para}}

, Original Final SR (

\uparrow

), and Original NBT (

\downarrow

). We establish the standard deviation across three seeds.

Method	Target SR $\uparrow$	Average GSA $\uparrow$	Original Persistence $\downarrow$
L2M	4.42	2.81	4.61
SeqFT	7.35	4.41	8.82
ER	19.42	11.73	37.84
EWC	10.68	6.57	20.81
PackNet	15.76	8.24	42.63
DMPEL	30.91	21.68	44.72

TABLE II: Final-stage executable minimal-contrast results on LIBERO-Goal . Higher Target SR and GSA are better, while lower Original Persistence is better.

Fig. 3: Goal-Switch Accuracy (GSA) by edited semantic slot on LIBERO-Goal . Higher values indicate better adaptation to the modified instruction. No executable attribute contrasts were generated.

Fig. 4: Final-stage behavior under scene-incompatible instructions on LIBERO-Goal Left: the proportions of rollouts achieving the original goal, another audited canonical goal, or no recognized goal; vertical markers indicate the Action-Initiation Rate (AIR). Right: Language Margin versus Action Divergence, characterizing action-level sensitivity to the instruction perturbation.

Fig. 5: Distribution of the instruction perturbations used to evaluate language grounding across the four LIBERO suites. The figure summarizes the original tasks, paraphrases, generated minimal contrasts, scene-validated minimal contrasts, and scene-incompatible collision instructions, together with their semantic categories.

Fig. 6: Paraphrase sensitivity at the final stage and throughout continual learning. Final Gap measures the difference between original and paraphrase success at the final stage, whereas AUC Gap measures the corresponding difference across all stages. Lower values indicate greater semantic invariance.

Diagnostic Pair	Spearman $\rho$
AD vs. OGP	0.26
AIR vs. CGAR	0.80
LM vs. OGP	0.21

TABLE III: Association between action-level and outcome-level collision diagnostics across method-task pairs. Weak associations indicate that the measurements capture complementary policy behaviors.

Explore similar work

Language-Critique Imitation Learning from Suboptimal Demonstrations
Jul 1, 2026Chih-Han Yang, Dai-Jie Wu, Yun-Ping Huang +3Temporally Coherent Imitation Learning Strong Imitation Learning
Graduate Institute of Communication Engineering, National Taiwan University (NTU) · University of Utah · National Yang Ming Chiao Tung University +1
AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents
Jun 1, 2026Yiheng Shu, Bernal Jiménez Gutiérrez, Saisri Padmaja Jonnalagedda +3Multilingual Agents Replay-Based Continual Learning
The Ohio State University · Johns Hopkins University · Intuit AI Research
RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents
Sep 22, 2026Chang Guo, Yukun Xie, Bohan Tan +6Embodied Agents Robocasa
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Research Lab, Anyverse Dynamics

Jul 1, 2026, cs.LGJ/K move · Enter open · S save

Language-Critique Imitation Learning from Suboptimal Demonstrations

Chih-Han Yang, Dai-Jie Wu, Yun-Ping Huang, Ping-Chun Hsieh+2

Graduate Institute of Communication Engineering, National Taiwan University (NTU) · University of Utah · National Yang Ming Chiao Tung University+1

Temporally Coherent Imitation Learning Strong Imitation Learning Imitation Critique Feedback

Open paper AskPDF arXiv

Prior work on imitation learning from suboptimal demonstrations typically relies on compressed supervision signals such as confidence estimates, discriminator scores, or importance weights. These scalar signals are inherently limited, as they cannot explicitly express intermediate reasoning about task progress, failure modes, or corrective actions. We propose a language-critique framework for imitation learning from suboptimal demonstrations that instead leverages natural language as a structured supervision signal, avoiding the collapse of expressive feedback into scalars. Our method first constructs language labels from demonstrations that explicitly describe current progress, identify suboptimal behaviors, and provide fine-grained corrective guidance. We then introduce a language-critique loss that directly trains policies using these structured signals without reducing them to scalars, and instantiate it for both behavior cloning and diffusion policies, yielding LC-BC and LC-DP. We further provide a theoretical result showing that the proposed objective upper-bounds the expert performance gap under standard assumptions. Empirically, we evaluate on diverse continuous control tasks spanning navigation, manipulation, and gameplay, where our methods consistently outperform strong imitation learning and offline reinforcement learning baselines. These results demonstrate that language can serve as a powerful and structured form of supervision for learning robust policies from suboptimal data.

Abstract

Explore similar work

Language-Critique Imitation Learning from Suboptimal Demonstrations

AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents

RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents