Robot-WATCHDOG : failure detection through object-centric graph representation

Quentin Rolland; Fabrice Mayran de Chamisso; Jean-Baptiste Mouret

cs.ROOct 9, 2026

Robot-WATCHDOG : failure detection through object-centric graph representation

Authors: Quentin Rolland, Fabrice Mayran de Chamisso, Jean-Baptiste Mouret

Organizations: Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France · Inria, CNRS, Universit´e de Lorraine, LORIA, F-54000 Nancy, France · Bleu Robotics, Paris, France

Abstract

Reliable real-time failure detection is critical for deploying learned robotic policies, yet defining failures explicitly is infeasible in open-world settings. This motivates failure detection from nominal demonstrations alone. Existing methods, however, struggle to balance robustness to benign variations with the ability to capture semantic and interaction-level errors. We introduce Robot-WATCHDOG, a self-supervised framework that models manipulation scenes as object-centric spatio-temporal graphs. Building on this representation, we investigate two complementary failure scores: Trajectory Projection and Correlation (TPC), a non-parametric measure of geometric deviation and relational consistency, and GnnT, a Graph Attention Network with a Transformer that predicts future object states. We evaluate both on a public benchmark and on a new dataset of semantically complex manipulation tasks, across six tasks and ten random seeds. Both scores outperform image-based failure detectors, particularly on semantic and interaction-level failures. TPC provides the strongest performance in AUPR and MCC on both dataset averages, with low variance across seeds. Qualitative results are made available at https://cea-list.github.io/Robot-WATCHDOG/.

Figures & tables

Figure 1: Detecting failures through object-centric relational dynamics. Robot-WATCHDOG represents manipulation scenes as spatio-temporal graphs enabling robust detection of relational and temporal errors beyond raw visual or trajectory-based methods.

Figure 2: Overview of the Robot-WATCHDOG pipeline. Stage 1: object-centric graph extraction from RGB observations and robot proprioception. Stage 2: failure detection (illustrated with GnnT ), which takes the graph as input and outputs a scalar failure score $S_{\text{total}}\in\mathbb{R}$ at each timestep. Stage 3: conformal prediction, where $S_{\text{total}}$ is compared to a threshold calibrated offline on demonstration data, yielding a binary decision (failure / no failure).

Figure 3: Threshold-independent discriminative performance. Average Area Under the Precision-Recall Curve (AUPR) and maximum Matthews Correlation Coefficient (MCC) across the BotFails and Watchdog datasets. Error bars denote the standard deviation across 10 seeds. Best results framed in purple with a star.

Figure 4: Qualitative failure detection on the Table-setting task . (Left) Nominal execution (Ep. 1): The global score remains safely below the CP threshold. (Right) Relational failure (Ep. 18): The cup is spatially misplaced. Due to natural spatial variability, absolute kinematics cannot detect this relational error. However, our object-centric graph captures the relational positional inconsistency, triggering a global score spike above the threshold that aligns with the misplaced cup’s elevated local anomaly score (red trajectory points).

Figure 5: End-to-end failure detection performance (MCC) after conformal calibration (modulation $s_{1}$ ), averaged over the tasks of each dataset, over 10 seeds and over $\alpha\in[0.80,0.95]$ . Error bars: standard deviation across seeds.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

	Waste-sorting				Table-setting
Representation	TPR $\uparrow$	TNR $\uparrow$	AUPR $\uparrow$	MCC $\uparrow$	TPR $\uparrow$	TNR $\uparrow$	AUPR $\uparrow$	MCC $\uparrow$
Vision features	$29.7\,\pm\,0.2$	$84.3\,\pm\,1.7$	$29.7\,\pm\,2.4$	$16.7\,\pm\,1.8$	$30.6\,\pm\,0.6$	$0.0\,\pm\,0.0$	$23.3\,\pm\,1.2$	$0.0\,\pm\,0.0$
Trajectories	$58.7\,\pm\,1.2$	$\mathbf{92.9}\,\pm\,0.8$	$\mathbf{54.7}\,\pm\,2.5$	$\mathbf{56.5}\,\pm\,1.9$	$54.5\,\pm\,1.6$	$90.9\,\pm\,1.2$	$54.5\,\pm\,1.7$	$49.3\,\pm\,1.7$
Graph (Ours)	$\mathbf{81.8}\,\pm\,5.3$	$79.9\,\pm\,3.5$	$49.7\,\pm\,4.0$	$53.1\,\pm\,1.9$	$\mathbf{83.6}\,\pm\,6.9$	$\mathbf{92.4}\,\pm\,3.6$	$\mathbf{80.0}\,\pm\,7.3$	$\mathbf{73.1}\,\pm\,6.6$

Appendix

Table 1: Quantitative evaluation of different input representations on failure detection performance. We compare our proposed object-centric graph representation against a global vision feature baseline and a graph-free trajectory-only model across the WATCHDOG dataset. Metrics reported include True Positive Rate (TPR), True Negative Rate (TNR), Area Under the Precision-Recall Curve (AUPR), and Matthews Correlation Coefficient (MCC). All values are reported as percentages, with higher values (

\uparrow

) indicating better performance. The best results for each metric are highlighted in bold.

	Waste-sorting				Table-setting
Method	TPR $\uparrow$	TNR $\uparrow$	AUPR $\uparrow$	MCC $\uparrow$	TPR $\uparrow$	TNR $\uparrow$	AUPR $\uparrow$	MCC $\uparrow$
GnnT	$\mathbf{81.8}\,\pm\,5.3$	$79.9\,\pm\,3.5$	$\mathbf{49.7}\,\pm\,4.0$	$\mathbf{53.1}\,\pm\,1.9$	$\mathbf{83.6}\,\pm\,6.9$	$\mathbf{92.4}\,\pm\,3.6$	$\mathbf{80.0}\,\pm\,7.3$	$\mathbf{73.1}\,\pm\,6.6$
w/o Edge model	$73.7\,\pm\,5.6$	$\mathbf{80.9}\,\pm\,8.3$	$48.5\,\pm\,5.2$	$49.0\,\pm\,6.3$	$81.8\,\pm\,6.9$	$86.8\,\pm\,1.0$	$59.3\,\pm\,9.3$	$61.9\,\pm\,6.3$
Orthogonal projection only	$\mathbf{89.2}\,\pm\,0$	$87.3\,\pm\,0$	$\mathbf{67.7}\,\pm\,0$	$\mathbf{61.0}\,\pm\,0$	$83.7\,\pm\,0$	$95.1\,\pm\,0$	$92.5\,\pm\,0$	$78.2\,\pm\,0$
Correlation only	$82.1\,\pm\,0$	$86.1\,\pm\,0$	$65.1\,\pm\,0$	$53.9\,\pm\,0$	$\mathbf{94.9}\,\pm\,0$	$91.2\,\pm\,0$	$90.8\,\pm\,0$	$81.3\,\pm\,0$
Both (TPC)	$47.7\,\pm\,1.9$	$\mathbf{96.8}\,\pm\,1.2$	$65.8\,\pm\,1.5$	$55.1\,\pm\,2.7$	$92.3\,\pm\,5.4$	$\mathbf{97.1}\,\pm\,2.1$	$\mathbf{96.3}\,\pm\,2.3$	$\mathbf{88.8}\,\pm\,5.7$

Appendix

Table 2: Ablation study on relational modeling for both GnnT and TPC. Best results are highlighted in bold within each block. For GnnT, removing edge-based interactions degrades performance, most strongly on Table-setting , confirming that structured object relationships are important for robust failure detection. For TPC, geometric (projection) and relational (correlation) components capture complementary aspects of anomalies: projection alone is sufficient for trajectory-level deviations in Waste-sorting , while correlation improves the detection of interaction-level errors in Table-setting . Combining both signals yields the best AUPR, TNR and MCC on Table-setting , whereas on Waste-sorting it leads to a conservative operating point (highest TNR, lower TPR), highlighting the task-dependent role of relational modeling.

	Waste-sorting				Table-setting
Method	TPR $\uparrow$	TNR $\uparrow$	AUPR $\uparrow$	MCC $\uparrow$	TPR $\uparrow$	TNR $\uparrow$	AUPR $\uparrow$	MCC $\uparrow$
GnnT	$81.8\,\pm\,5.3$	$79.9\,\pm\,3.5$	$49.7\,\pm\,4.0$	$\mathbf{53.1}\,\pm\,1.9$	$83.6\,\pm\,6.9$	$92.4\,\pm\,3.6$	$\mathbf{80.0}\,\pm\,7.3$	$\mathbf{73.1}\,\pm\,6.6$
Transformer only	$\mathbf{82.3}\,\pm\,6.0$	$78.8\,\pm\,3.9$	$50.8\,\pm\,3.7$	$52.2\,\pm\,3.0$	$87.0\,\pm\,7.0$	$87.2\,\pm\,2.8$	$66.2\,\pm\,8.1$	$66.6\,\pm\,8.3$
GNN only	$70.8\,\pm\,5.6$	$\mathbf{82.9}\,\pm\,5.3$	$\mathbf{51.8}\,\pm\,1.4$	$48.6\,\pm\,4.0$	$83.3\,\pm\,3.0$	$\mathbf{92.8}\,\pm\,2.8$	$78.1\,\pm\,6.0$	$71.2\,\pm\,4.2$
MLP	$75.3\,\pm\,12.8$	$70.6\,\pm\,13.1$	$40.5\,\pm\,2.9$	$39.3\,\pm\,3.6$	$\mathbf{88.7}\,\pm\,6.6$	$46.2\,\pm\,7.3$	$31.4\,\pm\,3.7$	$28.8\,\pm\,5.1$

Appendix

Table 3: Quantitative results of the spatio-temporal architecture ablation. Metrics are reported as mean

\pm

standard deviation over multiple random seeds. The full GnnT architecture achieves the best Matthews Correlation Coefficient (MCC) on both tasks and the best Area Under the Precision-Recall curve (AUPR) on Table-setting , providing the best balance between detection sensitivity and false positive rejection. On Waste-sorting , differences between the non-MLP variants are small.

	Waste-sorting				Table-setting
Method	TPR $\uparrow$	TNR $\uparrow$	AUPR $\uparrow$	MCC $\uparrow$	TPR $\uparrow$	TNR $\uparrow$	AUPR $\uparrow$	MCC $\uparrow$
GnnT (prediction based)	$\mathbf{81.8}\,\pm\,5.3$	$79.9\,\pm\,3.5$	$\mathbf{49.7}\,\pm\,4.0$	$\mathbf{53.1}\,\pm\,1.9$	$83.6\,\pm\,6.9$	$\mathbf{92.4}\,\pm\,3.6$	$\mathbf{80.0}\,\pm\,7.3$	$\mathbf{73.1}\,\pm\,6.6$
Density based	$70.2\,\pm\,6.7$	$\mathbf{80.3}\,\pm\,4.0$	$48.6\,\pm\,6.7$	$45.1\,\pm\,3.7$	$\mathbf{87.2}\,\pm\,3.5$	$90.4\,\pm\,6.9$	$78.2\,\pm\,11.7$	$72.9\,\pm\,9.2$

Appendix

Table 4: Quantitative comparison of failure detection formulations: predictive modeling (GnnT) versus latent density estimation (Normalizing Flows). Mean and standard deviation are reported across 5 different seeds. The predictive approach achieves a substantially higher MCC on Waste-sorting and matches or slightly exceeds the density-based baseline on Table-setting , while exhibiting lower variance on the threshold-independent (AUPR) and balanced (MCC) metrics across both tasks.

Figure 6: Trajectory extraction pipeline. (Left) The initial frame of an episode where task-relevant objects and the robot end-effector are identified using a Vision-Language Model (Stage 1). (Right) The final frame displaying the continuous 2D spatial trajectories extracted by tracking the objects’ mask centroids throughout the episode via SAM2 (Stage 2). The top row illustrates the structured Table-setting task, while the bottom row demonstrates the dynamic Waste sorting task on a moving conveyor belt.

Task	$N_{\text{obj}}$	Object list
domotic_groceriesSorting	8	4 sweets, 4 vegetables
domotic_vegetablesAndFruitsSorting	6	3 fruits, 3 vegetables
domotic_dishTidyUp	8	1 whisk, 3 plates, 3 glasses, 1 cup
domotic_setTheTable	4	1 glass, 1 plate, 1 fork, 1 knife
aloha_setTheTable	4	1 cup, 1 plate, 1 fork, 1 spoon
sort_trash_conveyor	2	1 plastic waste, 1 metal can

Appendix

Table 5: Object lists supplied to Qwen3-VL per task.

Method	Correct (%)	Missing (%)	Identity Error (%)	Position Error (%)
Qwen (No Stdz)	92.4	5.3	1.8	0.6
Qwen (With Stdz)	94.4	3.2	1.8	0.6

Appendix

Table 6: VLM Initialization Robustness (evaluated on 341 episodes). ” Missing ” indicates the model failed to detect all of the objects related to the task in the scene. ” Identity ” means some of the label attributed to the objects are incorrect. Position indicates a highly inaccurate bounding box.

Robot-Watchdog Datasets
Task Type	Perfect Tracking (%)	Recovered (%)	Missing (%)	Fusion (%)
Aloha SetTheTable	97.9	0.5	0.5	1.0
Trash Conveyor	97.7	0.2	0.9	1.1
Average (Robot-Watchdog)	97.8	0.3	0.8	1.1
Botfails Datasets
Domotic Sorting (Fruits & Veg)	98.3	0.0	1.7	0.0

Appendix

Table 7: SAM2 Tracking Robustness initialized with Ground Truth bounding boxes.

Category	Parameter	Value
Sequence	Observation window ( $m$ )	$20$
Sequence	Prediction horizon ( $L$ )	$15$
Architecture	Hidden dimension ( $d_{h}$ )	$128$
	Spatial Encoder layers (GATv2)	$2$
	Spatial Encoder attention heads	$8$
	Temporal Encoder layers (Transformer)	$4$

Appendix

Table 8: Hyperparameters for the GnnT architecture and training routine.

Component	Execution Time	Frequency / Nature
VLM Initialization (Qwen)	7.03 s	Once per task
SAM2 Tracking	22.72 ms	Per frame
GnnT Inference	19.87 ms	Per frame
TPC Inference	7.00 ms	Per frame
End-to-End (SAM2 + GnnT)	42.59 ms	23.5 FPS
End-to-End (SAM2 + TPC)	29.72 ms	33.6 FPS

Appendix

Table 9: Latency Profiling of the Robot-Watchdog Pipeline components (NVIDIA RTX 3090).

Figure 7: Overview of the Robot-WATCHDOG evaluation dataset. The dataset comprises a structured Table-setting task (left) and a dynamic Waste sorting task (right). The top row illustrates in-distribution, nominal expert demonstrations. The subsequent rows depict out-of-distribution scenarios, categorized into critical task failures (red borders) and benign visual variations (orange borders). Specific evaluation scenarios include: (1) Spatial misplacement, where objects are arranged with an incorrect offset; (2) Object inversion, where the target locations of items are swapped; (3) Grasping failure, resulting in a dropped object; (4) Benign visual variation featuring an intrusive distractor object. This highlights our method’s robustness to visual clutter, which typically triggers false positives in baseline anomaly detectors; (5) identity sorting error, where an item is placed into the incorrect bin; and (6) Missed object, where the robot fails to make a decision or initiate a grasp.

Task	Split	Data	Robot	# Episodes	# Frames	FPS	Cameras	Action dim
Table-setting	Expert	Video RGB/Proprio/text	ALOHA	100	67,341	15	4 views	9
Table-setting	Test	Video RGB/Proprio/text	ALOHA	21	13,471	15	4 views	9
Waste sorting	Expert	Video RGB/Proprio/text	SO-100	100	59,562	30	Top view	6
Waste sorting	Test	Video RGB/Proprio/text	SO-100	51	30,378	30	Top view	6

Appendix

Table 10: Statistics of our dataset.

Method	Input Space	Core Mechanism	Spatial Rel.	Temporal Align.	Scoring Strategy
AP-Fréchet	Trajectories (2D)	Trajectory Matching	Independent	Arc-length Resampling	Min. Fréchet Distance
Soft-DTW	Trajectories (2D)	Trajectory Matching	Independent	Soft-DTW	Min. Alignment Cost
AP-RMS	Trajectories (2D)	Trajectory Matching	Independent	Arc-length Resampling	Min. Prefix RMS Error
FIDeL [ 32 ]	Patch Features (RGB)	Optimal Transport	Implicit (Patches)	Min. Cost over $T$	OT Cost (+ VLM Filter)
logpZO [ 41 ]	Observations	Density Estimation	Implicit (Network)	ODE Forward Pass	Latent Space $\ell_{2}$ -norm
lopO [ 41 ]	Observations	Density Estimation	Implicit (Network)	ODE Integration	Exact Log-likelihood

Appendix

Table 11: Overview of the anomaly detection baselines compared in this work, including our proposed TPC (few-shot) and GnnT (predictive graph) approaches.

	BotFails				Watchdog		Dataset average
Method	Table-setting	Dish storing	Veg. sorting	Groc. sorting	Table-setting	Waste sorting	BotFails	Watchdog
AUPR $\uparrow$
Chance (prevalence)	0.314	0.296	0.318	0.190	0.218	0.224	0.280	0.221
AP-RMS	$0.272{\scriptstyle\,\pm\,0.013}$	$0.234{\scriptstyle\,\pm\,0.011}$	$\underline{0.711}{\scriptstyle\,\pm\,0.065}$	$0.549{\scriptstyle\,\pm\,0.106}$	$0.468{\scriptstyle\,\pm\,0.287}$	$0.149{\scriptstyle\,\pm\,0.001}$	$0.442{\scriptstyle\,\pm\,0.029}$	$0.309{\scriptstyle\,\pm\,0.143}$
AP-Fréchet	$0.282{\scriptstyle\,\pm\,0.032}$	$0.226{\scriptstyle\,\pm\,0.011}$	$0.591{\scriptstyle\,\pm\,0.155}$	$0.608{\scriptstyle\,\pm\,0.262}$	$0.527{\scriptstyle\,\pm\,0.373}$	$0.166{\scriptstyle\,\pm\,0.001}$	$0.427{\scriptstyle\,\pm\,0.066}$	$0.346{\scriptstyle\,\pm\,0.187}$
Soft-DTW	$0.247{\scriptstyle\,\pm\,0.012}$	$0.202{\scriptstyle\,\pm\,0.004}$	$0.309{\scriptstyle\,\pm\,0.007}$	$0.166{\scriptstyle\,\pm\,0.041}$	$0.142{\scriptstyle\,\pm\,0.011}$	$0.139{\scriptstyle\,\pm\,0.000}$	$0.231{\scriptstyle\,\pm\,0.010}$	$0.140{\scriptstyle\,\pm\,0.005}$

Appendix

Table 12: Threshold-independent results per task: AUPR, maximum MCC and AUROC (mean

\pm

std over 10 seeds). Dataset averages are computed per seed over the tasks of the dataset, then averaged over seeds (as in Fig. 3 ). The chance level of AUPR is the fraction of failure steps. Best in bold, second underlined. † FIDeL: 9 seeds on Table-setting (Watchdog).

Calibrated MCC $\uparrow$
	BotFails				Watchdog		Dataset average
Method	Table-setting	Dish storing	Veg. sorting	Groc. sorting	Table-setting	Waste sorting	BotFails	Watchdog
AP-RMS	$0.375{\scriptstyle\,\pm\,0.126}$	$0.134{\scriptstyle\,\pm\,0.058}$	$0.636{\scriptstyle\,\pm\,0.088}$	$0.271{\scriptstyle\,\pm\,0.107}$	$0.706{\scriptstyle\,\pm\,0.043}$	$0.295{\scriptstyle\,\pm\,0.034}$	$0.354{\scriptstyle\,\pm\,0.050}$	$0.501{\scriptstyle\,\pm\,0.027}$
AP-Fréchet	$\underline{0.446}{\scriptstyle\,\pm\,0.040}$	$0.359{\scriptstyle\,\pm\,0.140}$	$\underline{0.656}{\scriptstyle\,\pm\,0.192}$	$0.348{\scriptstyle\,\pm\,0.189}$	$\underline{0.708}{\scriptstyle\,\pm\,0.169}$	$\mathbf{0.553}{\scriptstyle\,\pm\,0.022}$	$\underline{0.452}{\scriptstyle\,\pm\,0.078}$	$\underline{0.630}{\scriptstyle\,\pm\,0.090}$
Soft-DTW	$0.285{\scriptstyle\,\pm\,0.071}$	$0.037{\scriptstyle\,\pm\,0.059}$	$0.560{\scriptstyle\,\pm\,0.152}$	$0.085{\scriptstyle\,\pm\,0.189}$	$0.067{\scriptstyle\,\pm\,0.145}$	$0.249{\scriptstyle\,\pm\,0.049}$	$0.242{\scriptstyle\,\pm\,0.053}$	$0.158{\scriptstyle\,\pm\,0.069}$
FIDeL	$0.072{\scriptstyle\,\pm\,0.076}$	$0.363{\scriptstyle\,\pm\,0.025}$	$0.204{\scriptstyle\,\pm\,0.036}$	$0.241{\scriptstyle\,\pm\,0.010}$	$0.253^{\dagger}{\scriptstyle\,\pm\,0.018}$	$0.113{\scriptstyle\,\pm\,0.016}$	$0.220{\scriptstyle\,\pm\,0.014}$	$0.177^{\dagger}{\scriptstyle\,\pm\,0.019}$

Appendix

Table 13: End-to-end detection after conformal calibration (modulation

s_{1}

, averaged over

\alpha\in[0.80,0.95]

): calibrated MCC, TPR and TNR per task (mean

\pm

std over 10 seeds). Best MCC in bold, second underlined. TPR and TNR are not highlighted: a high value of one alone is not informative (a detector that flags everything has TPR = 1).

Robot Failure Detection

Robot failure detection identifies when autonomous systems are failing, predicts emerging faults, and supports diagnosis or recovery.

See all 44 papers

RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation
Sep 16, 2026Loris Schneider, Edgar Welte, Rania RayyesRobot Failure Detection Fault Detection
Institute for Material Handling and Logistics (IFL), Karlsruhe Institute of Technology, 76131, Germany.
Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents
Jun 22, 2026Haoran Zhang, Yifu Lu, Boyang Wang +5Fair Conformal Prediction Long-Horizon Robotic Manipulation
University of Michigan · Princeton University · University of Virginia
Fail-RAG : A Retrieval Augmented Generation Informed Framework for Robot Failure Identification
Jun 17, 2026Ameya Salvi, Jie HuRetrieval-Augmented Generation Robot Failure Detection
Hitachi America, Ltd., Michigan, U.S.A.

Sep 16, 2026J/K move · Enter open · S save

RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation

Loris Schneider, Edgar Welte, Rania Rayyes

Institute for Material Handling and Logistics (IFL), Karlsruhe Institute of Technology, 76131, Germany.

Robot Failure Detection Fault Detection Robotic Manipulation OOD Detection Robotic Anomaly Detection

Open paper AskPDF arXiv

Detecting failures during execution is essential for reliable robotic manipulation. Vision-language models (VLMs) can assess task outcomes semantically but add runtime computation, whereas out-of-distribution (OOD) detectors may respond to harmless scene variations rather than failure-relevant deviations. We introduce RAFAIL, a framework for detecting execution failures during robotic manipulation. RAFAIL identifies failures by detecting anomalies in task-relevant relationships between entities, such as a gripper and an object or an object and its target. By focusing OOD detection on relevant parts of the observation, RAFAIL reduces sensitivity to task-irrelevant scene variation. Offline, a VLM annotates successful demonstrations with task progress and relationship importance, which are used to learn point-cloud-based relationship representations without relying on policy-internal features. At runtime, relationship-specific OOD detectors evaluate these representations while relationship importance and task progress are predicted without VLM inference. RAFAIL requires no failure data and achieves 73.4% balanced accuracy across three real-world robotic manipulation tasks, outperforming the strongest evaluated OOD- and uncertainty-based baselines.

Abstract

Robot Failure Detection

Explore similar work

RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation

Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents

Fail-RAG : A Retrieval Augmented Generation Informed Framework for Robot Failure Identification