How Well Do Latent World Models Understand Partially Observable Safety Constraints?

Matthew Kim; Kensuke Nakamura; Andrea Bajcsy

cs.ROOct 7, 2025

How Well Do Latent World Models Understand Partially Observable Safety Constraints?

Authors: Matthew Kim, Kensuke Nakamura, Andrea Bajcsy

Organizations: UC San Diego · Carnegie Mellon University

Abstract

Latent world models are a promising approach for learning state representations and dynamics directly from high-dimensional observations, enabling robot control in hard-to-model settings. However, control performance ultimately depends on the latent representation encoding the required information for the task. In this work, we study latent-space safe control problems and show how partial observability can induce control failures when safety-relevant information is not preserved in the latent state. Specifically, we identify two world model failure modes: estimation gaps, where current observations do not reveal safety-critical quantities (e.g., temperature in a cooking task), and prediction gaps, where failures are observable once they occur but cannot be reliably anticipated from available observations. We introduce two diagnostics for these gaps: a mutual-information-based measure of safety observability and a rollout-based measure of future safety predictability. Finally, we present mitigation strategies for each failure mode: privileged multimodal supervision for estimation gaps and conformal risk calibration for prediction gaps. Across two hardware case studies -- using unimodal RGB world models and multimodal RGB+Tactile and RGB+Thermal variants -- we show that these mitigation strategies improve the safety of a Franka Research 3 manipulator on challenging cooking tasks under partial observability, albeit with increased conservativeness. More broadly, our work raises the question of when world model state representations are sufficient for reliable robot control

Figures & tables

Figure 1: How well do latent world models (WMs) understand safety constraints? We find that WMs can suffer from estimation gaps (left) where the latent state fails to encode safety-relevant states that are hard to observe (e.g., temperature from RGB images) and prediction gaps (right) where failures are identifiable from the latent state once they happen (e.g., a spill) but cannot be reliably predicted with current observations because of unobservables (e.g., if the bottle is filled).

	Wax Melting			Rice Pouring
Metric	RGB	IR	MM	RGB	Tactile	MM
$\frac{I(Y;X)}{H(Y)}$	0.221	0.801	0.853	0.962	0.221	0.968
Acc.	0.870	0.969	0.979	0.994	0.838	0.995
B. Acc.	0.626	0.983	0.988	0.988	0.687	0.988

Table 1: Safety-Relevant Signal in Observations. Mutual information estimates and classification accuracies.

Table 2: Safety Outcome Prediction. Outcomes of 16-step open-loop trajectories. The right table reports results only on samples where RGB and MM disagree on safety outcomes; rows follow the same method ordering as the left table.

Figure 2: Real vs. Imagined Observations & Safety Labels. Open-loop predictions of a held-out subtrajectory reveal that $\textbf{WM}_{\textbf{RGB}}$ misjudges whether lifting the wax plate will cause failure. In contrast, observing temperature via IR images allows the $\textbf{WM}_{\textbf{MM}}$ to correctly classify that this action leads to a safe outcome.

	World Models & Respective Safety Filters
Metric	$\textbf{WM}_{\textbf{RGB}}$	$\textbf{WM}_{\textbf{MM}}$	$\textbf{WM}_{\textbf{RGB-MM}}$
Lift Rate [95% CI]	$0.15\,[0.04,\,0.39]$	$1.0\,[0.80,\,1.0]$	$1.0\,[0.80,\,1.0]$
Pixel Intensity at Intervention	$0.587\pm 0.303$	$0.695\pm 0.073$	$0.522\pm 0.208$

Table 3: Hardware: Closed-Loop Safety Filter Performance. Lift rates over 20 trials with 95% Wilson-CC confidence intervals [ 32 ] .

\textbf{WM}_{\textbf{RGB}}

fails to lift reliably.

\textbf{WM}_{\textbf{MM}}

intervenes before failure, while

\textbf{WM}_{\textbf{RGB-MM}}

intervenes more variably, likely because it cannot directly observe safety-relevant quantities.

Figure 3: (Left) Ground-truth pixel intensities when each safety filter intervenes. The $\textbf{WM}_{\textbf{MM}}$ policy that takes in IR images concentrates just prior to the threshold used for failure labeling, whereas $\textbf{WM}_{\textbf{RGB}}$ and $\textbf{WM}_{\textbf{RGB-MM}}$ use only RGB as input are spread more conservatively, indicating they are unable to discern when to filter. (Right) Distribution of predicted failure probabilities during conformal calibration. The $\textbf{WM}_{\textbf{RGB}}$ frequently fails to predict spills leading to conservative thresholds.

Bottle	Model	Pour Motion	Fallback
		[95% CI]	[95% CI]
Filled	RGB	0.14 [0.06, 0.27]	0.86 [0.73, 0.94]
Filled	MM	0.14 [0.06, 0.27]	0.86 [0.73, 0.94]
Empty	RGB	0.18 [0.09, 0.32]	0.82 [0.68, 0.91]
Empty	MM	0.60 [0.45, 0.73]	0.40 [0.27, 0.55]

Table 4: Results: MPS for 50 trials. Rates are reported with 95% Wilson-CC confidence intervals [ 32 ] . After conformal calibration to control false acceptance, the multimodal MPS controller retains greater control authority in the empty-bottle setting while maintaining fallback behavior in higher-risk settings.

Figure 4: Experiment Testbeds. In simulation (left) and Wax Melting (right), we collect paired RGB and infrared (IR) observations. In both settings, the safety-relevant state variable is heat, which is directly observable from IR but only partially observable from RGB.

Figure 5: Safety Controllers without & with Observability. In the Thermal Unicycle example, the robot only appears burnt in RGB after it leaves the hot plate. Left: The RGB-only safety policy is myopic: it stays in the hot region, avoiding seeing failure. Right: When the WM is trained with RGB+IR observations, the resulting safety controller accelerates out of the hot region to prevent overheating.

Modality	$\mathbf{I(Y;X)/H(Y)}\uparrow$	Acc. $\uparrow$	B. Acc. $\uparrow$
RGB	0.182	0.864	0.738
IR	0.909	0.982	0.966
RGB+IR	0.903	0.981	0.976

Table 5: Simulation Observation Signal Metrics. Observation-level safety signal metrics for the simulated Thermal Unicycle task.

Eval.	Method	TS $\uparrow$	FS $\downarrow$	FU $\downarrow$	TU $\uparrow$	$\mathbf{F_{1}}\uparrow$
Latent State	RGB	0.623	0.013	0.104	0.272	0.914
Latent State	RGB+IR	0.701	0.002	0.026	0.283	0.981
Open-loop Pred.	RGB	0.643	0.012	0.089	0.260	0.927
Open-loop Pred.	RGB+IR	0.693	0.012	0.035	0.264	0.967

Table 6: Simulation Safety Classification Metrics. Latent-state and 16-step open-loop prediction metrics for the simulated Thermal Unicycle task.

Hyperparameter	Thermal Unicycle	Wax Melting
RGB Image Dimension	[128, 128, 3]	[224, 224, 3]
IR Image Dimension	[128, 128, 1]	[224, 224, 1]
Action Dimension	2	7
Stochastic Latent	Gaussian	Gaussian
Latent Dim (Deterministic)	512	512
Latent Dim (Stochastic)	32	32

Table 7: Dreamer RSSM hyperparameters for the Thermal Unicycle and Wax Melting experiments.

Hyperparameter	Value
Actor Architecture	[512, 512, 512]
Critic Architecture	[512, 512, 512]
Normalization	LayerNorm
Activation	ReLU
Discount Factor $\gamma$	0.9999
Learning Rate (Critic)	1e-3

Table 8: DDPG hyperparameters for latent HJ safety filter synthesis.

Table 14

Explore similar work

When World Models Lie: Adaptive Safety Analysis Under Wrong Imaginations
Sep 28, 2026John Cao, Somil BansalReachability Analysis World Models for Robotics
Department of Aeronautics and Astronautics, Stanford University
Modeling Latent Disturbances for Robust Decision-Making in World Models
Oct 6, 2026Junwon Seo, Andrea BajcsyRobot Policy Learning World Model-Based Planning
The authors are with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA.
Pixels to Proofs: Probabilistically-Safe Latent World Model Control via Parallel Conformal Robust MPC
Jun 14, 2026Devesh Nath, Anutam Srinivasan, Haoran Yin +3Constrained Motion Planning World Model-Based Planning
Georgia Institute of Technology

Sep 28, 2026J/K move · Enter open · S save

When World Models Lie: Adaptive Safety Analysis Under Wrong Imaginations

John Cao, Somil Bansal

Department of Aeronautics and Astronautics, Stanford University

Reachability Analysis World Models for Robotics Robust Control Safety Filtering

Open paper AskPDF arXiv

World models offer a powerful substrate for safety reasoning in high-dimensional robotic systems, but they are also fallible: their predictions can be biased, miscalibrated, or confidently wrong. This creates a central challenge for latent-space safety filters, which often learn Hamilton-Jacobi safety value functions on the dynamics of a world model. If the world model is incorrect, the resulting value function can inherit its errors and produce overconfident safety estimates. Existing latent safety filters often rely on auxiliary signals such as ensemble disagreement or value-target consistency residuals for adaptation, but these signals can remain small even when the world model's predictions deviate from observations. We propose an adaptive latent safety filter that calibrates safety reasoning using directly observed world-model error. Our method uses Adaptive Conformal Inference to construct online uncertainty sets from discrepancies between predicted and observation-inferred latent states, then evaluates safety pessimistically by minimizing the learned value function over these sets. This allows the filter to remain minimally conservative when the world model is accurate, while becoming more cautious when observations reveal model mismatch. We provide a finite-time coverage guarantee for the adaptive uncertainty radius. Through simulation and hardware experiments, we show that our method significantly reduces failures relative to state-of-the-art latent safety filters while preserving task completion.

Abstract

Explore similar work

When World Models Lie: Adaptive Safety Analysis Under Wrong Imaginations

Modeling Latent Disturbances for Robust Decision-Making in World Models

Pixels to Proofs: Probabilistically-Safe Latent World Model Control via Parallel Conformal Robust MPC