Failure Identification in Imitation Learning Via Statistical and Semantic Filtering

Quentin Rolland; Fabrice Mayran de Chamisso; Jean-Baptiste Mouret

cs.ROApr 15, 2026

Failure Identification in Imitation Learning Via Statistical and Semantic Filtering

Authors: Quentin Rolland, Fabrice Mayran de Chamisso, Jean-Baptiste Mouret

Organizations: Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France · Inria, CNRS, Université de Lorraine, LORIA, F-54000 Nancy, France · Bleu Robotics, Paris, France

Abstract

Imitation learning (IL) policies in robotics deliver strong performance in controlled settings but remain brittle in real-world deployments: rare events such as hardware faults, defective parts, unexpected human actions, or any state that lies outside the training distribution can lead to failed executions. Vision-based Anomaly Detection (AD) methods emerged as an appropriate solution to detect these anomalous failure states but do not distinguish failures from benign deviations. We introduce FIDeL (Failure Identification in Demonstration Learning), a policy-independent failure detection module. Leveraging recent AD methods, FIDeL builds a compact representation of nominal demonstrations and aligns incoming observations via optimal transport matching to produce anomaly scores and heatmaps. Spatio-temporal thresholds are derived with an extension of conformal prediction, and a Vision-Language Model (VLM) performs semantic filtering to discriminate benign anomalies from genuine failures. We also introduce BotFails, a multimodal dataset of real-world tasks for failure detection in robotics. FIDeL consistently outperforms state-of-the-art baselines, yielding +5.30% percent AUROC in anomaly detection and +17.38% percent failure-detection accuracy on BotFails compared to existing methods.

Figures & tables

Fig. 1 : We introduce FIDeL , a framework for detecting failures in imitation learning (IL) policies. - Offline - Expert demonstrations are first encoded and stored in a memory buffer $\mathcal{M}$ . 0. Conformal Prediction Calibration — A decision threshold is computed from $\mathcal{M}$ using Conformal Prediction to determine when a score should be considered anomalous. - Online - 1. Anomaly Detection — During policy execution, anomaly scores and heatmaps are computed from incoming observations. 2. Thresholding — If the score exceeds the calibrated threshold, an anomaly is flagged. 3. Semantic Filtering — Since not all anomalies are failure-related, a VLM filter discards benign deviations and flags only failures that threaten task success.

Fig. 2 : BotFails dataset illustration - all images are illustrations of BotFails tasks executed by the expert - Domestic - 1. clear away the dishes, 2. make coffee, 3. set the table, 4. pour coffee, 5. sort groceries, 6. sort fruits and vegetables - Industrial - 7. sort screws, 8. measure voltage, 9. press buttons, 10. solder.

Fig. 3 : Score illustration - Real-

\pi

soldering - episode 15 over 20 episodes of the evaluation set, score obtained with Representation and CP time. The Anomaly Detection score corresponds to the output of the AD module

D_{A}(\phi(\mathcal{Y}_{N}),\phi_{t^{*}})

\mu^{t}_{p}

is obtained when computing

D_{A}(\phi(\mathcal{Y}_{N}),\phi(\mathcal{E}_{A}))

and allows to compute the Threshold values , see section III-E . When the Anomaly Detection score is below the Threshold , no anomaly is detected ( CP-Normal ) and when it is above, an anomaly is flagged ( CP-Anomalous ). We observe that transitioning from anomaly detection \tiny{1}⃝ to failure detection \tiny{2}⃝ through the use of the VLM reduces the number of false positives (highlighted in the yellow boxes). However, this additional filtering step also introduces a small number of false negatives.

Fig. 4 : Heatmaps illustration - obtained using Representation AD and temporal/spatial CP - Real- $\pi$ : \tiny{1}⃝ expert, \tiny{2}⃝ benign anomaly: screws present in the work plan, \tiny{3}⃝ failure: dropping the iron - BotFails : \tiny{4}⃝ expert, \tiny{5}⃝ benign anomaly: someone walking in the background (top left corner), \tiny{6}⃝ failure: spilling the cup.

	Representation (ours)		logpZ0 [ 12 ]		lopO		AE		STAC [ 11 ]
	AUROC $\uparrow$	F1@Opt $\uparrow$	AUROC $\uparrow$	F1@Opt $\uparrow$	AUROC $\uparrow$	F1@Opt $\uparrow$	AUROC $\uparrow$	F1@Opt $\uparrow$	AUROC $\uparrow$	F1@Opt $\uparrow$
BotFails	$70.50$	$62.90$	$65.20$	$59.87$	$58.30$	$53.48$	$49.90$	$52.12$	N/A	N/A
BotFails	$\pm 0$	$\pm 0$	$\pm 0.15$	$\pm 0.06$	$\pm 0.70$	$\pm 0.02$	$\pm 1.08$	$\pm 0.08$	N/A	N/A
Real- $\pi$ soldering	$86.72$	$78.29$	$82.62$	$78.03$	$76.16$	$70.99$	$81.07$	$76.33$	$53.55$	$63.93$
Real- $\pi$ soldering	$\pm 0$	$\pm 0$	$\pm 0.08$	$\pm 0.02$	$\pm 0.42$	$\pm 0.03$	$\pm 0.88$	$\pm 0.06$	$\pm 0$	$\pm 0$

TABLE I : Anomaly Detection evaluation before thresholding – mean AUROC and mean F1-score at optimal threshold ( in % ) for anomaly detection only evaluated on the BotFails dataset and Real-

\pi

soldering task , across various AD methods. Best results are highlighted in green and bold. Each task score reflects performance over multiple rollouts (several dozens per task).

Fig. 5 : Anomaly Detection evaluation with thresholding - mean True Positive Rate (TPR), True Negative Rate (TNR), balanced accuracy (Acc), weighted accuracy (W.Acc), ( in % ), for various Anomaly Detection types and various thresholding types–temporal Conformal Prediction (CP time), temporal and spatial Conformal Prediction (CP t&s), Multiple Gaussian–evaluated on real world task from the BotFails dataset and soldering task operated autonomously with ACT [ 3 ] . Best Acc and W.Acc in hatched red.

Fig. 6 : End-to-end system evaluation (including semantic filtering) - accuracy (Acc) and weighted accuracy (W.Acc) for various robotics failure detection systems. The labeling is different for this evaluation. While AD and Threshold module are evaluated using anomaly labels , the end-to-end system is evaluated using failures labels . Which means that all failure-related anomaly labels remain 1, while non-failure anomalies are relabeled from 1 to 0. FiDeL = Representation + CP t&s + Semantic filtering

Explore similar work

Jul 29, 2026

Failure Detection for Surgical Robot Imitation Policies via Flow-Matching World Modeling

Imitation learning has shown increasing promise for autonomous robotic surgery, yet safe deployment remains challenging due to the safety-critical nature of surgical tasks and the complexity and variability of surgical environments. Failure detection is therefore an essential safeguard, but its development remains difficult due to the challenges of scarce failure data, highly variable manipulation dynamics, and the need to balance missed detections against disruptive false alarms. To address these challenges, we introduce FoMo-FD (Flow-Matching World Model for Failure Detection), a failure detection method that learns nominal short-horizon visual dynamics with an action-conditioned flow-matching world model. FoMo-FD scores the inverse-transport nonconformity of observed endpoint latents, enabling window-level detection of visual-action inconsistencies without requiring failure demonstrations. Detection thresholds are obtained by conformal calibration on successful executions, yielding task-specific alarms without assuming future failure types. We evaluate FoMo-FD on four surgically relevant manipulation tasks with twenty failure modes across simulation and real-world experiments using the da Vinci Research Kit (dVRK). Results show that FoMo-FD outperforms observation-level anomaly baselines and a prediction-error variant of the same world model, with the wrist-camera view achieving the strongest performance, including a 96.6% failure detection rate (FDR) at a 1.3% false alarm rate (FAR).

Zhefeng Huang, Yilin Cai, Ankit Patel +3

Department of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA 30332 USA

Robot Failure Detection World Models Robot Imitation Learning Surgical Robotics Anomaly Detection

Ask PDF

May 8, 2026

How to Utilize Failure Demo Data?: Effective Data Selection for Imitation Learning Using Distribution Differences in Attention Mechanism

Imitation learning for robotic tasks has relied primarily on policies trained only on successful demonstrations, although failures are unavoidable during human data collection. Many existing approaches for exploiting failure data require additional data processing or iterative policy updates through autonomous rollouts, making it difficult to directly and stably utilize failure data accumulated during data collection. In this work, we propose a method that learns latent representations of success-failure discrepancies and incorporates them into the attention mechanism. During inference, an appropriate latent mode is selected from the initial observation to improve action stability. Furthermore, we introduce a post-training metric that quantifies the attention discrepancy between each failure sample and successful demonstrations to select failure data. Simulation results show that the proposed method improves task success rates when trained with failure data and that the proposed metric identifies failure samples that are beneficial for learning when combined with successful demonstrations. These results suggest that the proposed method can support more efficient use of collected demonstrations in robotic data collection pipelines.

Kana Miyamoto, Kanata Suzuki, Tetsuya Ogata

Faculty of Science and Engineering, Waseda University, Tokyo, Japan · Physical AI Laboratory, Fujitsu Limited, Kanagawa, Japan

Learning from Demonstration Robot Imitation Learning

Ask PDF

Apr 17, 2026

Rewind-IL: Online Failure Detection and State Respawning for Imitation Learning

Imitation learning has enabled robots to acquire complex visuomotor manipulation skills from demonstrations, but deployment failures remain a major obstacle, especially for long-horizon action-chunked policies. Once execution drifts off the demonstration manifold, these policies often continue producing locally plausible actions without recovering from the failure. Existing runtime monitors either require failure data, over-trigger under benign feature drift, or stop at failure detection without providing a recovery mechanism. We present Rewind-IL, a training-free online safeguard framework for generative action-chunked imitation policies. Rewind-IL combines a zero-shot failure detector based on Temporal Inter-chunk Discrepancy Estimate (TIDE), calibrated with split conformal prediction, with a state-respawning mechanism that returns the robot to a semantically verified safe intermediate state. Offline, a vision-language model identifies recovery checkpoints in demonstrations, and the frozen policy encoder is used to construct a compact checkpoint feature database. Online, Rewind-IL monitors self-consistency in overlapping action chunks, tracks similarity to the checkpoint library, and, upon failure, rewinds execution to the latest verified safe state before restarting inference from a clean policy state. Experiments on real-world and simulated long-horizon manipulation tasks, including transfer to flow-matching action-chunked policies, demonstrate that policy-internal consistency coupled with semantically grounded respawning offers a practical route to improved reliability in imitation learning. Supplemental materials are available at https://sjay05.github.io/rewind-il

Gehan Zheng, Sanjay Seenivasan, Matthew Johnson-Roberson +1

College of Connected Computing, Vanderbilt University, USA. · School of Computer Science, University of Waterloo, Canada. · School of Computer Science, The University of Sydney, Australia. +1

Robot Failure Detection Robot Failure Recovery Robotic Manipulation Robot Imitation Learning Anomaly Detection

Ask PDF