Foundational World Models Accurately Detect Bimanual Manipulator Failures

Isaac R. Ward; Michelle Ho; Houjun Liu; Aaron Feldman; Joseph Vincent; Liam Kruse; Sean Cheong; Duncan Eddy

cs.ROMar 7, 2026

Foundational World Models Accurately Detect Bimanual Manipulator Failures

Authors: Isaac R. Ward, Michelle Ho, Houjun Liu, Aaron Feldman, Joseph Vincent, Liam Kruse, Sean Cheong, Duncan Eddy, +2 more

Organizations: Stanford University, Stanford, California, 94305, USA. · Watney Robotics, San Francisco, California, 94110, USA.

Abstract

Deploying visuomotor robots at scale is challenging due to the potential for anomalous failures to degrade performance, cause damage, or endanger human life. Bimanual manipulators are no exception; these robots have vast state spaces comprised of high-dimensional images and proprioceptive signals. Explicitly defining failure modes within such state spaces is infeasible. In this work, we overcome these challenges by training a probabilistic, history informed, world model within the compressed latent space of a pretrained vision foundation model (NVIDIA's Cosmos Tokenizer). The model outputs uncertainty estimates alongside its predictions that serve as non-conformity scores within a conformal prediction framework. We use these scores to develop a runtime monitor, correlating periods of high uncertainty with anomalous failures. To test these methods, we use the simulated Push-T environment and the Bimanual Cable Manipulation dataset, the latter of which we introduce in this work. This new dataset features trajectories with multiple synchronized camera views, proprioceptive signals, and annotated failures from a challenging data center maintenance task. We benchmark our methods against baselines from the anomaly detection and out-of-distribution detection literature, and show that our approach considerably outperforms statistical techniques. Furthermore, we show that our approach requires approximately one twentieth of the trainable parameters as the next-best learning-based approach, yet outperforms it by 3.8% in terms of failure detection rate, paving the way toward safely deploying manipulator robots in real-world environments where reliability is non-negotiable.

Figures & tables

Fig. 1 : A schematic of the training process for the proposed method. A trajectory $\tau=(s_{1},a_{1},s_{2},a_{2},\dots)$ is sampled from the dataset and sliced into training samples consisting of a fixed history window $\mathbf{h}_{t}=(s_{t-H+1},a_{t-H+1},\dots,s_{t},a_{t})$ and future state $s_{t+1}$ . The multi-step, multi-view images captured at the robot’s grippers ( $s_{t}^{(1)}$ and $s_{t}^{(2)}$ ) are encoded using a foundation video encoder (Cosmos) and—alongside learned projections of the historical proprioceptive states $s_{t}^{\theta}$ and actions $a_{t}$ —are combined into latent feature maps. These are processed sequentially by a transformer to predict a tensor of distributions, the standard deviations of which quantify the uncertainty of the future prediction. This uncertainty is lower for nominal inputs (like those observed during training), and higher for anomalous inputs associated with failures. In other words: $\texttt{WM}(h_{t})\approx f(h_{t})$ only when $h_{t}\in\mathcal{M}_{\textnormal{nom}}$ . Sampling from this latent distribution gives a latent feature map $Z\sim\mathcal{N}(\mu,\sigma^{2})$ , which can be decoded and refined into predictions for the next immediate state ( $\hat{s}_{t+1}^{(1)}$ , $\hat{s}_{t+1}^{(2)}$ , $\hat{s}_{t}^{\theta}$ ).

Panel of Figure 2 — (a) Each trajectory in the Push-T dataset consists of a $25$ fps, $128\times 128\times 3$ single-view RGB video showing a blue dot (the agent) attempting to push a T-shaped object into a green T-shaped slot. The state $s_{t}^{\theta}$ is the $(x,y)$ position of the end effector, and the action $a_{t}$ (not visualized) specifies the next change in position of the end effector.

Fig. 3 : Histograms of the WM uncertainty scores for nominal (blue), anomalous recolored, and anomalous dynamically altered inputs, with overlaid visualizations of what these anomalous inputs look like. In each example the $90\%$ conformal prediction threshold is rendered in red. To test our method’s ability to handle visual and proprioceptive anomalies, we alter the color to orange (top left) and green (bottom left) as well as altering the friction to half the nominal friction coefficient (top right) and zero (bottom right).

Fig. 4 : The WR1 Data Center Mobile Manipulator robot.

	Avg. Classification Accuracy ( $32$ folds, %)
	Nominal	Failure	Weighted total
Method	( $N_{\checkmark}$ $=$ 7)	( $N_{\times}$ $=$ 9)	( $N$ $=$ 16)
WM uncertainty (ours)	$87.9\pm 17.0$	$\mathbf{95.1\pm 5.5}$	$\mathbf{92.0\pm 6.4}$
WM pred. error (ours)	$\mathbf{88.3\pm 17.8}$	$87.5\pm 12.3$	$87.9\pm 6.4$
logpZO	$86.8\pm 20.4$	$91.3\pm 6.7$	$89.3\pm 6.8$
AE recon. error	$80.6\pm 20.5$	$45.8\pm 18.6$	$61.0\pm 4.2$

TABLE I : Classification performance of different methods on the Bimanual Cable Manipulation dataset, using an

85\%

conformal prediction threshold.

Fig. 5 : We find that our WM -based approaches outperform competing methods on the Bimanual Cable Manipulation dataset, presumably because of how they uniquely incorporate historical context. See Table I for a class-based breakdown.

Fig. 6 : The time taken to generate a scalar non-conformity score value for a single step, for each method.

Fig. 7 : An example failure detection on an unseen failure case, visualizing captures from the left and right gripper cameras. (1) Nominal behavior—the robot is not gripping or attempting to manipulate a cable. (2) The uncertainty rises as the gripper approaches a cable and begins a sequence of grab attempts. Uncertainty tends to increase when a cable head is in view. (3) The first of three aborted grab attempts, such periods are usually associated with increased uncertainty as grab attempts represent a small portion of the nominal training set. Note that these do not constitute failures. (4) Uncertainty increases rapidly as the robot focuses instead on unplugging a cable that is already plugged in (observe the force being applied onto the cable head by the compliant grippers). (5) An actual failure occurs (dropped cable) as the robot fumbles the cable head during the unplug maneuver. (6) With the dropped cable fully out of frame, the robot is again not gripping or attempting to manipulate a cable as in (1). Uncertainty again reduces to nominal levels. Note that the world model is trained on proprioceptive state data, and is action conditioned, so some part of the predicted uncertainty likely stems from proprioceptive features not visualized in this figure.

Explore similar work

RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation
Sep 16, 2026Loris Schneider, Edgar Welte, Rania RayyesRobot Failure Detection Fault Detection
Institute for Material Handling and Logistics (IFL), Karlsruhe Institute of Technology, 76131, Germany.
FailBench: How Reliable are VLMs at Judging Robot Task Success?
Sep 3, 2026Zaruhi Navasardyan, Tatul Danielyan, Hrant DavtyanVLM Evaluation Robot Failure Detection
Metric AI Lab
Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents
Jun 22, 2026Haoran Zhang, Yifu Lu, Boyang Wang +5Long-Horizon Robotic Manipulation Robot Failure Detection
University of Michigan · Princeton University · University of Virginia

Sep 16, 2026J/K move · Enter open · S save

RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation

Loris Schneider, Edgar Welte, Rania Rayyes

Institute for Material Handling and Logistics (IFL), Karlsruhe Institute of Technology, 76131, Germany.

Robot Failure Detection Fault Detection Robotic Manipulation OOD Detection

Open paper AskPDF arXiv

Detecting failures during execution is essential for reliable robotic manipulation. Vision-language models (VLMs) can assess task outcomes semantically but add runtime computation, whereas out-of-distribution (OOD) detectors may respond to harmless scene variations rather than failure-relevant deviations. We introduce RAFAIL, a framework for detecting execution failures during robotic manipulation. RAFAIL identifies failures by detecting anomalies in task-relevant relationships between entities, such as a gripper and an object or an object and its target. By focusing OOD detection on relevant parts of the observation, RAFAIL reduces sensitivity to task-irrelevant scene variation. Offline, a VLM annotates successful demonstrations with task progress and relationship importance, which are used to learn point-cloud-based relationship representations without relying on policy-internal features. At runtime, relationship-specific OOD detectors evaluate these representations while relationship importance and task progress are predicted without VLM inference. RAFAIL requires no failure data and achieves 73.4% balanced accuracy across three real-world robotic manipulation tasks, outperforming the strongest evaluated OOD- and uncertainty-based baselines.

Abstract

Explore similar work

RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation

FailBench: How Reliable are VLMs at Judging Robot Task Success?

Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents