Adaptive World-Models for Anomaly Recognition (AWARE): A Framework for Robot Dynamics Prediction and Introspection

Luke Beddow; George Mavroghenis; Rodrigo Chacon Quesada; Kamil Dreczkowski; Cong Sun; Jiankai Wang; Oscar Kwong-Fai Pang; Antoine Cully

cs.ROOct 23, 2026

Adaptive World-Models for Anomaly Recognition (AWARE): A Framework for Robot Dynamics Prediction and Introspection

Authors: Luke Beddow, George Mavroghenis, Rodrigo Chacon Quesada, Kamil Dreczkowski, Cong Sun, Jiankai Wang, Oscar Kwong-Fai Pang, Antoine Cully

Organizations: BOLD, Imperial College London

Abstract

Unrecognized anomalies in critical infrastructure can rapidly cascade into catastrophic physical or operational failures. Current anomaly detection approaches often rely on direct access to the monitored system by utilizing onboard sensors, such as motor encoders, to detect anomalous states. However, legacy hardware often lacks these sensors, or strict security protocols may prohibit access altogether. We therefore present AWARE, a world model framework that is capable of anomaly detection using only external camera monitoring via system dynamics estimation and state prediction. Evaluated across 240 real-world robotic crane trajectories, including payload collisions and motor degradations, AWARE demonstrates a 26% improvement in anomaly classification over our strongest baseline under noisy, camera-only state estimation. Furthermore, the inferred dynamics parameter can be decoded to identify the source of faults, localizing the degrading motor with 88% accuracy.

Figures & tables

Figure 1: System and model overview, showing human-operated robotic crane used for anomaly detection experiments. AWARE operates as follows: 1) camera-based state estimation; 2) estimation of system hidden dynamics parameters; 3) prediction of future trajectory; 4) anomaly recognition via examination of both the estimated dynamics parameters and the trajectory prediction error.

Figure 2: AWARE framework. Our model consists of two key components. First, a Latent Dynamics Estimator (green) reconstructs a latent encoding of system dynamics ( $\hat{z}_{t}$ ) using a history of state-action pairs. Second, a World Model (blue) predicts the next state, given this encoding, the same history, and the current action. This integrated framework enables introspection (yellow) and anomaly detection (red) by comparing dynamics estimations and state prediction errors ( $e_{t}$ ) against their expected distributions under normal operation.

Figure 3: Deployment Method (1) Crane platform : The system is modeled by three actuated joints: slew ( $q_{1}$ ), luff ( $q_{2}$ ), and hoist ( $q_{5}$ ). The under-actuated dynamics are captured by a double pendulum model: two angles ( $q_{3}$ , $q_{4}$ ) defining the movement at the hoist connection point (cable swing), and two angles ( $q_{6}$ , $q_{7}$ ) defining the relative orientation of the payload with respect to the cable. (2) Camera-Based State Estimation : Operator actions are estimated by two hand-tracking cameras. Crane states $q_{1}$ and $q_{2}$ are derived from ArUco marker poses using a ZED X camera. The remaining states ( $q_{3}$ – $q_{7}$ ) are estimated by a boom-mounted ZED 2i camera utilizing a Convolutional Neural Network (CNN). (3) Operator Interface : A multi-color indicator lamp provides real-time visual feedback, signaling nominal operation (green light) or system alerts (red light).

Method	Motion Capture Quality State Estimation (Low Noise)		Camera Only State Estimation (High Noise)
Method	Avg. Precision $\pm$ std	F1 Score $\pm$ std	Avg. Precision $\pm$ std	F1 Score $\pm$ std
Randomly Generated Signal	0.17 $\pm$ 0.03	0.26 $\pm$ 0.03	0.17 $\pm$ 0.03	0.26 $\pm$ 0.03
Isolation Forest [ 26 ]	0.29 $\pm$ 0.00	0.36 $\pm$ 0.01	0.16 $\pm$ 0.00	0.25 $\pm$ 0.01
Statistical Anomaly Detection	0.40 $\pm$ 0.01	0.40 $\pm$ 0.01	0.22 $\pm$ 0.01	0.31 $\pm$ 0.02
AESOP [ 36 ]	0.47 $\pm$ 0.00	0.56 $\pm$ 0.00	0.24 $\pm$ 0.00	0.41 $\pm$ 0.00
AnyAnomaly ∗ [ 2 ]	0.37 $\pm$ 0.07	0.56 $\pm$ 0.06	0.37 $\pm$ 0.07	0.56 $\pm$ 0.06

Table 1: Anomaly recognition results over

240

real robotic trajectories, averaged over

10

seeds per trained model. Metrics detail classification performance for identifying anomalies at the correct times within trajectories, under two different passive state estimation schemes. Bold entries are statistically significant improvement over the best non-AWARE model. ∗ AnyAnomaly is only evaluated on the nominal and payload collision set, see Appendix C.3 for more detail.

Panel of Figure 4 — Figure 4: Contribution of Latent and Predictor to Anomaly Recognition across different Impairments types using the Transformer AWARE model under (a) Motion Capture Quality Estimation (low noise) and (b) Camera Only Estimation (high noise).

Panel of Figure 5 — Figure 5: Estimated Motor Strength using the Transformer AWARE model under (a) Low Noise State Estimation Data and (b) High Noise State Estimation Data. The solid curves and shaded bands represent the mean and standard deviation over multiple environments respectively. Decoding Accuracy is the rate of correctly detecting the impaired motor within a $\pm 12.5\%$ tolerance.

Figure 6: Average Precision (AP) of Transformer AWARE across three anomaly detection modes as increasing levels of noise, sampled from a Student’s

t

-distribution, are injected into the Motion Capture Quality dataset. Noise is scaled relative to the standard deviation (

\sigma

) of each joint. The noise level of the Camera Only Quality data relative to the Motion Capture Quality data is estimated by inverse interpolation of the AP score, with the cross indicating the interpolated

\sigma

corresponding to the Camera Only Quality data.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Config in MuJoco	Range
Motor Strength Scale Factor	ctrlrange	(0.5, 1.5)
Motor Rotor Inertia Scale Factor	armature	(0.1, 2.0)
Payload Mass / $kg$		(0.1, 2.0)

Appendix

Table 2: Parameters for Domain Randomization

Model	Anomaly Decision Input	Motion Capture Quality State Estimation (Low Noise)		Camera Only State Estimation (High Noise)
Model	Anomaly Decision Input	Avg. Precision $\pm$ std	F1 Score $\pm$ std	Avg. Precision $\pm$ std	F1 Score $\pm$ std
Randomly Generated Signal	Raw sensor data	0.17 $\pm$ 0.03	0.26 $\pm$ 0.03	0.17 $\pm$ 0.03	0.26 $\pm$ 0.03
Isolation Forest [ 26 ]		0.29 $\pm$ 0.00	0.36 $\pm$ 0.01	0.16 $\pm$ 0.00	0.25 $\pm$ 0.01
Statistical Anomaly Detection		0.40 $\pm$ 0.01	0.40 $\pm$ 0.01	0.22 $\pm$ 0.01	0.31 $\pm$ 0.02
AESOP [ 36 ]		0.47 $\pm$ 0.00	0.56 $\pm$ 0.00	0.24 $\pm$ 0.00	0.41 $\pm$ 0.00
AnyAnomaly ∗ [ 2 ]		0.37 $\pm$ 0.07	0.56 $\pm$ 0.00	0.37 $\pm$ 0.07	0.56 $\pm$ 0.00

Appendix

Table 3: Anomaly recognition results organized by anomaly decision input. Bold entries are statistically significant improvement over the best non-AWARE model. ∗ AnyAnomaly is only evaluated on the nominal and payload collision set.

	Baseline	Motion Capture Quality (Low Noise)		Camera Only (High Noise)
	Baseline	AP ( $p$ )	F1 ( $p$ )	AP ( $p$ )	F1 ( $p$ )
Transformer AWARE	RSSM AWARE (double)	0.00195	0.00195	0.02734	0.19336
	RWM (double)	0.00195	0.01172	0.00195	0.00195
	Transformer AWARE (latent)	0.00195	0.00195	0.00195	0.00195
	RSSM AWARE (latent)	0.00195	0.00195	0.00195	0.00195
	RWM (latent)	0.00195	0.00195	0.00195	0.00195

Appendix

Table 4: Per-metric raw

p

-values from two-sided Wilcoxon signed-rank tests over 10 paired seeds. Significance is assessed within each baseline using a Bonferroni-corrected threshold over four metrics (two datasets

\times

AP/F1), i.e.,

\alpha=0.05/4=0.0125

Figure 7: World model prediction error over a 100 step horizon. We condition the model on either Single Camera Quality Data or Motion Capture Quality Data, computing the error relative the Motion Capture data in both cases. The error is computed as a fraction of the joint’s total range, averaged over all joint dimensions then as a running cumulative error averaged over all experiments.

Figure 8: Training curves for RSSM AWARE. We ablate by reintroducing components from Regularized Online Adaptation [ 13 ] to AWARE, including: regularization of the target encoder that the latent dynamics estimator learns to follow; removing the latent decoder and reintroducing the latent bottleneck. We plot both average precision and F1 over 1.5 million training steps on the Single Camera Quality dataset.

Encode dynamics param to $\hat{z}_{t}$	Dynamics latents fixed to median $\hat{z}_{0}$	Error normalized by variance $\sigma_{t}$	Motion Capture Quality		Single Camera Quality
			State Estimation (Low Noise)		State Estimation (High Noise)
			AP	F1	AP	F1
	✓	✓	0.71 $\pm$ 0.02	0.65 $\pm$ 0.02	0.50 $\pm$ 0.01	0.54 $\pm$ 0.01
✓	✓		0.88 $\pm$ 0.01	0.83 $\pm$ 0.01	0.55 $\pm$ 0.01	0.58 $\pm$ 0.01
✓		✓	0.81 $\pm$ 0.01	0.75 $\pm$ 0.01	0.50 $\pm$ 0.01	0.54 $\pm$ 0.01
✓	✓	✓	0.90 $\pm$ 0.01	0.85 $\pm$ 0.01	0.59 $\pm$ 0.01	0.61 $\pm$ 0.01

Appendix

Table 5: RSSM AWARE results using only the world model

E_{pred}

to detect anomalies. We ablate over the enhancements we make to improve predictor performance, including learning a latent representation,

\hat{z}

, of the hidden dynamics parameter,

\gamma

; conditioning the world model on the fixed median nominal latent,

\hat{z}_{0}

; and normalizing the computed error by the world model’s variance.

Distance Measure	Operator	Motion Capture Quality State Estimation (Low Noise)		Camera Only State Estimation (High Noise)
Distance Measure	Operator	Avg. Precision $\pm$ std	F1 Score $\pm$ std	Avg. Precision $\pm$ std	F1 Score $\pm$ std
L1	OR	0.86 $\pm$ 0.02	0.80 $\pm$ 0.02	0.69 $\pm$ 0.03	0.63 $\pm$ 0.04
	AND	0.78 $\pm$ 0.04	0.72 $\pm$ 0.03	0.66 $\pm$ 0.01	0.59 $\pm$ 0.04
	SUM	0.78 $\pm$ 0.09	0.72 $\pm$ 0.07	0.50 $\pm$ 0.04	0.55 $\pm$ 0.04
L2	OR	0.83 $\pm$ 0.03	0.78 $\pm$ 0.02	0.67 $\pm$ 0.04	0.60 $\pm$ 0.04
	AND	0.75 $\pm$ 0.05	0.71 $\pm$ 0.03	0.65 $\pm$ 0.02	0.59 $\pm$ 0.04

Appendix

Table 6: Anomaly detection results averaged across AWARE methods using different distance measures to the nominal reference distribution and operators to combine the prediction and latent signal

E_{pred}

and

E_{lat}

. ’L1’ and ’L2’ use the Manhattan and Euclidean distance to the reference distribution mean. ’L2 standardized’ normalizes the L2 distance by the diagonal covariance.

Panel of Figure 9 — Figure 9: Examples of Anomaly Recognition by a pivoting measurement signal, where a successful detection is classified as either measurement signal exceeds its corresponding threshold. Nominal std gives the standard deviation of both measurement signals during nominal operations across all environments.

Panel of Figure 10 — Figure 10: Examples of predictor and latent-estimator measurement signals with and without anomalies. (a) Motion Capture Quality Estimation (low noise); (b) Camera Only Estimation (high noise). From left to right, the four columns correspond to True Positive, True Negative, False Positive, and False Negative cases (based on whether an anomaly was present and whether the model raised an alarm). Solid blue and green curves show the predictor and latent-estimator signals, respectively, while dashed blue and green lines denote their anomaly thresholds. The red dashed vertical line marks the time at which the anomaly is injected.

Robotic Anomaly Detection

See all 18 papers

World Model Failure Classification and Anomaly Detection for Autonomous Inspection
Feb 18, 2026Michelle Ho, Muhammad Fadhil Ginting, Isaac R. Ward +4Robot Failure Detection Robotic Anomaly Detection
Stanford University, Stanford, CA 94305 USA · Field AI, Mission Viejo, CA 92691 USA
RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulation
Mar 11, 2026Shijie Zhou, Bin Zhu, Jiarui Yang +3Robotic Anomaly Detection Anomaly Detection
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University
FARM: Reading Failure Signals from the Internal Predictive States of a Frozen Robotic World Model
Sep 10, 2026Haoran Pei, Mingrui Luo, Senbao Wang +4Robot Failure Detection Robotics
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Harbin Institute of Technology, Harbin, China

Feb 18, 2026J/K move · Enter open · S save

World Model Failure Classification and Anomaly Detection for Autonomous Inspection

Michelle Ho, Muhammad Fadhil Ginting, Isaac R. Ward, Andrzej Reinke+3

Stanford University, Stanford, CA 94305 USA · Field AI, Mission Viejo, CA 92691 USA

Robot Failure Detection Robotic Anomaly Detection Robotic Inspection Anomaly Detection OOD Detection

Open paper AskPDF arXiv

Autonomous inspection robots for monitoring industrial sites can reduce costs and risks associated with human-led inspection. However, accurate readings can be challenging due to occlusions, limited viewpoints, or unexpected environmental conditions. We propose a hybrid framework that combines supervised failure classification with anomaly detection, enabling classification of inspection tasks as a success, known failure, or anomaly (i.e., out-of-distribution) case. Our approach uses a world model backbone with compressed video inputs. This policy-agnostic, distribution-free framework determines classifications based on two decision functions set by conformal prediction (CP) thresholds before a human observer does. We evaluate the framework on gauge inspection feeds collected from office and industrial sites and demonstrate real-time deployment on a Boston Dynamics Spot. Experiments show over 90% accuracy in distinguishing between successes, failures, and OOD cases, with classifications occurring earlier than a human observer. These results highlight the potential for robust, anticipatory failure detection in autonomous inspection tasks or as a feedback signal for model training to assess and improve the quality of training data. Project website: https://autoinspection-classification.github.io

Abstract

Robotic Anomaly Detection

Explore similar work

World Model Failure Classification and Anomaly Detection for Autonomous Inspection

RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulation

FARM: Reading Failure Signals from the Internal Predictive States of a Frozen Robotic World Model