Failure Identification in Imitation Learning Via Statistical and Semantic Filtering

Quentin Rolland; Fabrice Mayran de Chamisso; Jean-Baptiste Mouret

cs.ROApr 15, 2026

Failure Identification in Imitation Learning Via Statistical and Semantic Filtering

Authors: Quentin Rolland, Fabrice Mayran de Chamisso, Jean-Baptiste Mouret

Organizations: Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France · Inria, CNRS, Université de Lorraine, LORIA, F-54000 Nancy, France · Bleu Robotics, Paris, France

Abstract

Imitation learning (IL) policies in robotics deliver strong performance in controlled settings but remain brittle in real-world deployments: rare events such as hardware faults, defective parts, unexpected human actions, or any state that lies outside the training distribution can lead to failed executions. Vision-based Anomaly Detection (AD) methods emerged as an appropriate solution to detect these anomalous failure states but do not distinguish failures from benign deviations. We introduce FIDeL (Failure Identification in Demonstration Learning), a policy-independent failure detection module. Leveraging recent AD methods, FIDeL builds a compact representation of nominal demonstrations and aligns incoming observations via optimal transport matching to produce anomaly scores and heatmaps. Spatio-temporal thresholds are derived with an extension of conformal prediction, and a Vision-Language Model (VLM) performs semantic filtering to discriminate benign anomalies from genuine failures. We also introduce BotFails, a multimodal dataset of real-world tasks for failure detection in robotics. FIDeL consistently outperforms state-of-the-art baselines, yielding +5.30% percent AUROC in anomaly detection and +17.38% percent failure-detection accuracy on BotFails compared to existing methods.

Figures & tables

Fig. 1 : We introduce FIDeL , a framework for detecting failures in imitation learning (IL) policies. - Offline - Expert demonstrations are first encoded and stored in a memory buffer $\mathcal{M}$ . 0. Conformal Prediction Calibration — A decision threshold is computed from $\mathcal{M}$ using Conformal Prediction to determine when a score should be considered anomalous. - Online - 1. Anomaly Detection — During policy execution, anomaly scores and heatmaps are computed from incoming observations. 2. Thresholding — If the score exceeds the calibrated threshold, an anomaly is flagged. 3. Semantic Filtering — Since not all anomalies are failure-related, a VLM filter discards benign deviations and flags only failures that threaten task success.

Fig. 2 : BotFails dataset illustration - all images are illustrations of BotFails tasks executed by the expert - Domestic - 1. clear away the dishes, 2. make coffee, 3. set the table, 4. pour coffee, 5. sort groceries, 6. sort fruits and vegetables - Industrial - 7. sort screws, 8. measure voltage, 9. press buttons, 10. solder.

Fig. 3 : Score illustration - Real-

\pi

soldering - episode 15 over 20 episodes of the evaluation set, score obtained with Representation and CP time. The Anomaly Detection score corresponds to the output of the AD module

D_{A}(\phi(\mathcal{Y}_{N}),\phi_{t^{*}})

\mu^{t}_{p}

is obtained when computing

D_{A}(\phi(\mathcal{Y}_{N}),\phi(\mathcal{E}_{A}))

and allows to compute the Threshold values , see section III-E . When the Anomaly Detection score is below the Threshold , no anomaly is detected ( CP-Normal ) and when it is above, an anomaly is flagged ( CP-Anomalous ). We observe that transitioning from anomaly detection \tiny{1}⃝ to failure detection \tiny{2}⃝ through the use of the VLM reduces the number of false positives (highlighted in the yellow boxes). However, this additional filtering step also introduces a small number of false negatives.

Fig. 4 : Heatmaps illustration - obtained using Representation AD and temporal/spatial CP - Real- $\pi$ : \tiny{1}⃝ expert, \tiny{2}⃝ benign anomaly: screws present in the work plan, \tiny{3}⃝ failure: dropping the iron - BotFails : \tiny{4}⃝ expert, \tiny{5}⃝ benign anomaly: someone walking in the background (top left corner), \tiny{6}⃝ failure: spilling the cup.

	Representation (ours)		logpZ0 [ 12 ]		lopO		AE		STAC [ 11 ]
	AUROC $\uparrow$	F1@Opt $\uparrow$	AUROC $\uparrow$	F1@Opt $\uparrow$	AUROC $\uparrow$	F1@Opt $\uparrow$	AUROC $\uparrow$	F1@Opt $\uparrow$	AUROC $\uparrow$	F1@Opt $\uparrow$
BotFails	$70.50$	$62.90$	$65.20$	$59.87$	$58.30$	$53.48$	$49.90$	$52.12$	N/A	N/A
BotFails	$\pm 0$	$\pm 0$	$\pm 0.15$	$\pm 0.06$	$\pm 0.70$	$\pm 0.02$	$\pm 1.08$	$\pm 0.08$	N/A	N/A
Real- $\pi$ soldering	$86.72$	$78.29$	$82.62$	$78.03$	$76.16$	$70.99$	$81.07$	$76.33$	$53.55$	$63.93$
Real- $\pi$ soldering	$\pm 0$	$\pm 0$	$\pm 0.08$	$\pm 0.02$	$\pm 0.42$	$\pm 0.03$	$\pm 0.88$	$\pm 0.06$	$\pm 0$	$\pm 0$

TABLE I : Anomaly Detection evaluation before thresholding – mean AUROC and mean F1-score at optimal threshold ( in % ) for anomaly detection only evaluated on the BotFails dataset and Real-

\pi

soldering task , across various AD methods. Best results are highlighted in green and bold. Each task score reflects performance over multiple rollouts (several dozens per task).

Fig. 5 : Anomaly Detection evaluation with thresholding - mean True Positive Rate (TPR), True Negative Rate (TNR), balanced accuracy (Acc), weighted accuracy (W.Acc), ( in % ), for various Anomaly Detection types and various thresholding types–temporal Conformal Prediction (CP time), temporal and spatial Conformal Prediction (CP t&s), Multiple Gaussian–evaluated on real world task from the BotFails dataset and soldering task operated autonomously with ACT [ 3 ] . Best Acc and W.Acc in hatched red.

Fig. 6 : End-to-end system evaluation (including semantic filtering) - accuracy (Acc) and weighted accuracy (W.Acc) for various robotics failure detection systems. The labeling is different for this evaluation. While AD and Threshold module are evaluated using anomaly labels , the end-to-end system is evaluated using failures labels . Which means that all failure-related anomaly labels remain 1, while non-failure anomalies are relabeled from 1 to 0. FiDeL = Representation + CP t&s + Semantic filtering

Explore similar work

Failure Detection for Surgical Robot Imitation Policies via Flow-Matching World Modeling
Jul 29, 2026Zhefeng Huang, Yilin Cai, Ankit Patel +3Robot Failure Detection World Models
Department of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA 30332 USA
How to Utilize Failure Demo Data?: Effective Data Selection for Imitation Learning Using Distribution Differences in Attention Mechanism
May 8, 2026Kana Miyamoto, Kanata Suzuki, Tetsuya OgataLearning from Demonstration Robot Imitation Learning
Faculty of Science and Engineering, Waseda University, Tokyo, Japan · Physical AI Laboratory, Fujitsu Limited, Kanagawa, Japan
Rewind-IL: Online Failure Detection and State Respawning for Imitation Learning
Apr 17, 2026Gehan Zheng, Sanjay Seenivasan, Matthew Johnson-Roberson +1Robot Failure Detection Robot Failure Recovery
College of Connected Computing, Vanderbilt University, USA. · School of Computer Science, University of Waterloo, Canada. · School of Computer Science, The University of Sydney, Australia. +1

Jul 29, 2026J/K move · Enter open · S save

Failure Detection for Surgical Robot Imitation Policies via Flow-Matching World Modeling

Zhefeng Huang, Yilin Cai, Ankit Patel, Mohammad Hajiha+2

Department of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA 30332 USA

Robot Failure Detection World Models Robot Imitation Learning Surgical Robotics Anomaly Detection

Open paper AskPDF arXiv

Imitation learning has shown increasing promise for autonomous robotic surgery, yet safe deployment remains challenging due to the safety-critical nature of surgical tasks and the complexity and variability of surgical environments. Failure detection is therefore an essential safeguard, but its development remains difficult due to the challenges of scarce failure data, highly variable manipulation dynamics, and the need to balance missed detections against disruptive false alarms. To address these challenges, we introduce FoMo-FD (Flow-Matching World Model for Failure Detection), a failure detection method that learns nominal short-horizon visual dynamics with an action-conditioned flow-matching world model. FoMo-FD scores the inverse-transport nonconformity of observed endpoint latents, enabling window-level detection of visual-action inconsistencies without requiring failure demonstrations. Detection thresholds are obtained by conformal calibration on successful executions, yielding task-specific alarms without assuming future failure types. We evaluate FoMo-FD on four surgically relevant manipulation tasks with twenty failure modes across simulation and real-world experiments using the da Vinci Research Kit (dVRK). Results show that FoMo-FD outperforms observation-level anomaly baselines and a prediction-error variant of the same world model, with the wrist-camera view achieving the strongest performance, including a 96.6% failure detection rate (FDR) at a 1.3% false alarm rate (FAR).

Abstract

Explore similar work

Failure Detection for Surgical Robot Imitation Policies via Flow-Matching World Modeling

How to Utilize Failure Demo Data?: Effective Data Selection for Imitation Learning Using Distribution Differences in Attention Mechanism

Rewind-IL: Online Failure Detection and State Respawning for Imitation Learning