Modeling Latent Disturbances for Robust Decision-Making in World Models

Junwon Seo; Andrea Bajcsy

cs.ROOct 6, 2026

Modeling Latent Disturbances for Robust Decision-Making in World Models

Authors: Junwon Seo, Andrea Bajcsy

Organizations: The authors are with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA.

Abstract

In this paper, we study robust decision-making in the latent space of world models (WMs). Robust optimization is a mathematical framework where, given explicitly specified dynamics and physically meaningful disturbances, a robot can select actions that remain effective even under worst-case disturbances. However, applying this principle to the learned latent space of WMs introduces a fundamental challenge: because WMs have fully learned state spaces and dynamics inferred from high-dimensional observations, it is unclear how to define latent-space disturbances that faithfully represent uncertainty in the underlying system. Our key idea is to model a latent-space disturbance as a perturbation to the learned latent dynamics that induces pessimistic but plausible transitions. Specifically, we construct a set of plausible latent dynamics by combining a dynamics-aware similarity metric that captures plausible transitions with out-of-distribution detection that excludes implausible latent states. We calibrate this uncertainty set over latent dynamics using conformal prediction, ensuring that WM imaginations induced by the latent disturbance remain plausible without becoming overly pessimistic. We then jointly optimize robust robot actions and the worst-case latent disturbances through game-theoretic optimization. We leverage this latent-space robust optimization to robustify policy steering, considering two paradigms: latent safety filtering and sample-and-verify steering of a generative control policy. Our controlled simulation experiments show that our latent disturbance enables robust decision-making directly in WM latent spaces, and hardware experiments with a Franka manipulator show that modeling latent disturbances enables robust policy steering, reducing failures by 70% in safety filtering and 54% in sampling-based policy steering. Project website: https://junwon.me/LatentDisturbance/.

Figures & tables

Fig. 2: Naughty 3D Dubins’ Car. (a) Environment with the vehicle and a failure set at the center. (b) For an action sequence that turns right, an adverse disturbance can instead drive the vehicle into failure, making the decision non-robust. (c) Driving straight remains safe even in the worst case. However, an implausible imagination can make this robust action appear unsafe.

	Naughty				Positional Disturbances
Method	B.Acc. $\uparrow$	FPR $\downarrow$	FNR $\downarrow$	Fail $\downarrow$	B.Acc. $\uparrow$	FPR $\downarrow$	FNR $\downarrow$	Fail $\downarrow$
Nominal	0.932	0.006	0.130	0.114	0.831	0.000	0.337	0.218
WoN	0.959	0.057	0.026	0.061	0.911	0.162	0.017	0.034
$\mathrm{CVaR}_{0.05}$	0.944	0.017	0.095	0.062	0.943	0.028	0.087	0.084
$\mathrm{CVaR}_{0.1}$	0.935	0.012	0.118	0.061	0.936	0.018	0.109	0.101
$\mathrm{CVaR}_{0.2}$	0.927	0.010	0.135	0.062	0.927	0.011	0.135	0.134

TABLE I: 3D Dubins’ Car World Model: Quantitative Results.

Fig. 3: Qualitative Results: 3D Dubins’ Car. Solid lines represent the ground-truth unsafe-set boundaries, and red regions denote the unsafe sets computed by each method. LUCID closely approximates the ground-truth unsafe sets of robust optimization under two different types of disturbances. In contrast, Nominal is overly optimistic, while the sampling-based baselines such as WoN and CVaR inaccurately approximate unsafe sets.

	Uncertainty Set			Naughty			Positional Disturbances
Method	Dyn.	Conf.	OOD	B.Acc. $\uparrow$	FPR $\downarrow$	FNR $\downarrow$	B.Acc. $\uparrow$	FPR $\downarrow$	FNR $\downarrow$
LUCID	✓	✓	✓	0.958	0.037	0.046	0.956	0.023	0.066
w/o OOD	✓	✓	✗	0.809	0.381	0.001	0.621	0.759	0.000
$0.5\epsilon_{\mathrm{KL}}$	✓	✗	✓	0.927	0.011	0.136	0.864	0.001	0.271
$2.0\epsilon_{\mathrm{KL}}$	✓	✗	✓	0.926	0.126	0.022	0.630	0.733	0.007
Euclidean	✗	✓	✓	0.605	0.790	0.000	0.875	0.001	0.250

TABLE II: 3D Dubins’ Car: Uncertainty Set Ablation. Dyn., Conf., and OOD represent dynamics-aware similarity, conformal calibration, and in-distribution constraints, respectively.

Method	B.Acc. $\uparrow$	F1 $\uparrow$	FPR $\downarrow$	FNR $\downarrow$
Nominal	0.924	0.748	0.122	0.031
WoN	0.966	0.898	0.037	0.032
$\mathrm{CVaR}_{0.2}$	0.942	0.860	0.046	0.069
LUCID	0.966	0.947	0.009	0.059
w/o OOD	0.841	0.810	0.000	0.318
$0.5\epsilon_{\mathrm{KL}}$	0.956	0.861	0.055	0.034

TABLE III: Ablation: Optimistic Latent Disturbance.

Fig. 4: Qualitative: Naughty Dubins’ Car with Optimistic Disturbances. Solid lines show the ground-truth unsafe-set boundaries of the disturbance-free system, and dashed lines show the failure set. Without the in-distribution constraint, the disturbance exploits implausible latent states, trivially imagining trajectories that avoid failure from any non-failure state.

$\pi^{\text{task}}$	Safety Filter	Success $\uparrow$	Failure $\downarrow$	Safety Gain $\uparrow$	Cond. Failure $\downarrow$
DreamerV3	No Filter	0.44	0.54	–	–
	Nominal	0.45	0.54	-0.12 $\pm$ 0.21	0.82 $\pm$ 0.38
	WoN	0.46	0.50	0.01 $\pm$ 0.15	0.50 $\pm$ 0.50
	CVaR ${}_{\text{0.1}}$	0.39	0.48	-0.08 $\pm$ 0.16	0.48 $\pm$ 0.50
	UnConf.	0.00	0.43	0.08 $\pm$ 0.04	0.43 $\pm$ 0.50
	LUCID	0.73	0.19	0.22 $\pm$ 0.21	0.20 $\pm$ 0.40

TABLE IV: Quantitative Results: Vision-Based Block Pouring.

Fig. 5: Safety Value Function: Nominal vs. LUCID . Safeguarding the same $\pi^{\text{task}}$ from the same initial states, the robust safety filter proactively intervenes based on a plausibly pessimistic imagination ( $\hat{o}^{d}_{28}$ ) in which $\pi^{\text{task}}$ can lead to failure, while the nominal imagination ( $\hat{o}_{28}$ ) predicts a non-failure outcome. The resulting robust action prevents failure. In contrast, the nominal latent safety filter overestimates safety and intervenes too late based on an optimistic imagination ( $\hat{o}_{50}$ ), leading to failure under an adverse realization.

Safety Filter	Success $\uparrow$	Failure $\downarrow$	Safety Gain $\uparrow$	Cond. Failure $\downarrow$	Robust Rate $\uparrow$
No Filter	0.33	0.47	–	–	0.10
Nominal	0.31	0.40	0.17 $\pm$ 0.2	0.46 $\pm$ 0.5	0.11
WoN	0.35	0.31	-0.04 $\pm$ 0.2	0.32 $\pm$ 0.4	0.07
CVaR 0.1	0.30	0.31	-0.11 $\pm$ 0.2	0.32 $\pm$ 0.4	0.10
UnConf.	0.05	0.06	0.10 $\pm$ 0.03	0.06 $\pm$ 0.23	0.90
LUCID	0.34	0.11	0.21 $\pm$ 0.15	0.11 $\pm$ 0.31	0.77

TABLE V: Results: Trajectory Replay Across

20

Different Physics.

Fig. 6: Block Pouring: World Model Imaginations. (a) Initial state: the orange block is tilted while $\pi^{\text{task}}$ remains still. (b) The nominal imagination predicts a non-failure outcome. (c) With the conformalized uncertainty set, the latent disturbance imagines a plausible adverse transition in which the green block slides and falls. (d) Without the conformalized uncertainty set, the latent disturbance induces an implausible transition in which the green block falls in an infeasible manner, resulting in an overly pessimistic imagination.

$\pi^{\text{task}}$	Safety Filter	Success $\uparrow$	Failure $\downarrow$	Safety Gain $\uparrow$	Cond. Failure $\downarrow$
DreamerV3	No Filter	0.43	0.56	–	–
	Nominal	0.60	0.39	0.21 $\pm$ 0.24	0.40 $\pm$ 0.48
	WoN	0.34	0.54	0.10 $\pm$ 0.20	0.54 $\pm$ 0.49
	CVaR 0.1	0.00	0.34	-0.05 $\pm$ 0.05	0.34 $\pm$ 0.47
	UnConf.	0.00	0.00	0.29 $\pm$ 0.08	0.00 $\pm$ 0.06
	LUCID	0.81	0.18	0.27 $\pm$ 0.22	0.19 $\pm$ 0.39

TABLE VI: Ablation: Deterministic Block-Pouring with Fixed Physics

Fig. 7: Qualitative Results: Preventing Sunny-Side-Down Egg. LUCID preemptively identifies teleoperator actions that may cause the egg to fall or flip and intervenes with robust actions that remain safe under worst-case dynamics. In contrast, the Nominal safety filter relies on overly optimistic safety values and non-robust actions, leading to sunny-side-down outcomes.

Fig. 8: Filtering the Same $\pi^{\text{task}}$ Across Different Surfaces. ( Left ) Failure rates under each spatula surface condition. ( Right ) Robust rate, measuring the fraction of trajectories that remain safe across all surface conditions. LUCID consistently minimizes failures and achieves a higher robust rate, whereas the Nominal safety filter prevents failures only under certain surface conditions.

Fig. 9: Quantitative Results: Success Rates. ( Left ) Safety filtering over $20$ rollouts for Diffusion Policy and $\pi_{0.5}$ . ( Right ) Steering $\pi_{0.5}$ with sample-and-verify action selection, using world-model imaginations to evaluate candidate action chunks. In both settings, LUCID robustly improves success rates.

Fig. 10: Qualitative: Policy Steering with World Model. (a) Initial observation and sampled candidate actions. (b) For each candidate, Nominal imagines outcomes using $f_{z}$ , LUCID computes worst-case $f_{z}^{\pi_{d}}$ within the calibrated uncertainty set $\mathcal{F}$ , while Unconformalized $f_{z}^{\hat{\pi_{d}}}$ can exploit out-of-distribution latent states. (c) Imagined outcomes for candidate actions. LUCID selects an action that remains successful under the plausible worst-case, whereas the other schemes lose discriminability among candidate actions.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Gaussian	Categorical
Input	$[h,\mu,\sigma]$	$h$
Input dimension	$512+32+32$	$512$
Hidden layers	$4$ layers, $512$ units each
Normalization / activation	LayerNorm / ReLU
Output residual	$(\Delta\mu,\Delta\sigma)$	$\Delta\phi$
Output dimension	$32+32$	$32\times 32$

Appendix

TABLE VII: Residual parameterization of

\pi_{d}

in ( 11 ).

Hyperparameter	Value
Image Dimension	$128\times 128\times 3$
Action Dimension	$1$ (continuous)
Stochastic Latent	Gaussian
Deterministic Dimension	$512$
Stochastic Dimension	$32$
Activation Function	SiLU

Appendix

TABLE VIII: 3D Dubins’ Car Latent World Model Settings.

Parameter	Value
Actor Learning Rate	$10^{-3}$
Critic Learning Rate	$10^{-4}$
Disturbance Learning Rate	$10^{-4}$
Discount Factor	$0.85\rightarrow 0.9999$
Training Iterations	$100{,}000$
Replay Buffer Size	$100{,}000$

Appendix

TABLE IX: 3D Dubins’ Car Latent Safety Filter Settings.

Parameter	Randomization Range
Cube Position $x$	$[-0.02,\,0.02]$ m
Cube Position $y$	$[-0.02,\,0.02]$ m
Mass	$[0.8,\,1.2]$ kg
Static Friction	$[0.2,\,0.8]$
Dynamic Friction	$[0.1,\,0.5]$
Restitution	$[0.05,\,0.3]$

Appendix

TABLE X: Physical Randomization of Block Pouring in IsaacLab.

Hyperparameter	Value
Image Dimension	$2\times 128\times 128\times 3$
Proprioception Dimension	$7$ (eef position/quaternion)
Action Dimension	$7$ (continuous)
Stochastic Latent	Categorical
Deterministic Dimension	$512$
Stochastic Dimension	$32\times 32$

Appendix

TABLE XI: Block Pouring World Model Training Settings.

Parameter	Value
Actor Learning Rate	$10^{-3}$
Critic Learning Rate	$10^{-3}$
Disturbance Learning Rate	$10^{-3}$
Discount Factor	$0.95$
Training Iterations	$400{,}000$
Replay Buffer Size	$1{,}000{,}000$

Appendix

TABLE XII: Simulated manipulation safety-learning settings.

	RSSM	Transformer RSSM
Visual encoder	Trainable CNN	Frozen DINOv3
Temporal model	GRU	Causal transformer
History	Recurrent	$8$ -step attention window
Recon. Target	Pixels	Pixels and DINO features

Appendix

TABLE XIII: Transformer-Based Latent World Model

Hyperparameter	Value
Image Dimension	$2\times 192\times 256\times 3$
Proprioception Dimension	$8$
Action Dimension	$7$ (continuous)
Stochastic Latent	Gaussian
Deterministic Dimension	$512$
Stochastic Dimension	$32$

Appendix

TABLE XIV: Real World: World Model Hyperparameters.

Parameter	Value
Actor Learning Rate	$5\times 10^{-4}$
Critic Learning Rate	$10^{-3}$
Disturbance Learning Rate	$10^{-3}$
Target Smoothing	$0.005$
Discount Factor	$0.85\rightarrow 0.9999$
Training Iterations	$400{,}000$

Appendix

TABLE XV: Real World Latent Safety Filter Setups.

Explore similar work

When World Models Lie: Adaptive Safety Analysis Under Wrong Imaginations
Sep 28, 2026John Cao, Somil BansalSafety Filters Latent World Models
Department of Aeronautics and Astronautics, Stanford University
Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Control
Jul 16, 2026Jihoon Hong, Julian Skifstad, Qiyue Dai +2World Models Steering
Georgia Institute of Technology
LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Jun 14, 2026Jialei Chen, Kai Wang, Kang Chen +9Latent World Models Action Prediction
Jilin University · Zhongguancun Academy · Nankai University +4

Sep 28, 2026, cs.ROJ/K move · Enter open · S save

When World Models Lie: Adaptive Safety Analysis Under Wrong Imaginations

John Cao, Somil Bansal

Department of Aeronautics and Astronautics, Stanford University

Safety Filters Latent World Models World Models Hamilton-Jacobi Reachability Overconfidence Imagination

Open paper AskPDF arXiv

World models offer a powerful substrate for safety reasoning in high-dimensional robotic systems, but they are also fallible: their predictions can be biased, miscalibrated, or confidently wrong. This creates a central challenge for latent-space safety filters, which often learn Hamilton-Jacobi safety value functions on the dynamics of a world model. If the world model is incorrect, the resulting value function can inherit its errors and produce overconfident safety estimates. Existing latent safety filters often rely on auxiliary signals such as ensemble disagreement or value-target consistency residuals for adaptation, but these signals can remain small even when the world model's predictions deviate from observations. We propose an adaptive latent safety filter that calibrates safety reasoning using directly observed world-model error. Our method uses Adaptive Conformal Inference to construct online uncertainty sets from discrepancies between predicted and observation-inferred latent states, then evaluates safety pessimistically by minimizing the learned value function over these sets. This allows the filter to remain minimally conservative when the world model is accurate, while becoming more cautious when observations reveal model mismatch. We provide a finite-time coverage guarantee for the adaptive uncertainty radius. Through simulation and hardware experiments, we show that our method significantly reduces failures relative to state-of-the-art latent safety filters while preserving task completion.

Abstract

Explore similar work

When World Models Lie: Adaptive Safety Analysis Under Wrong Imaginations

Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Control

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies