World Models are appearing as the next major frontier in computer vision. However, their robustness is currently largely unexplored. We identify the phenomenon of hallucination in latent World Models: given a state and an action, the predicted next latent can decode to a scene that never occurs. Because the prediction is statistically ordinary and is fed back autoregressively by the model, the error is both silent and compounding. We study whether such latent hallucination can be detected, localised, and corrected at inference time, on a frozen self-supervised world model in the absence of ground-truth error labels. We introduce Masked Empirical-Bayes Neural Denoising (MEND), a single conditional score network trained by denoising score matching on real transitions, whose score field serves three roles: its magnitude detects hallucination, its per-token field localises it to specific image patches, and it defines an inference-time correction direction. On two navigation environments MEND detects hallucination with an AUROC of up to 0.80 without using actions, exceeding a single-Gaussian density baseline while also localising the error (per-token AUPRC up to 0.87) and correcting it, all from one score field. Our correction reliably reduces single-step latent error and improves predictions. We identify that a part of the error is tangent to the data manifold, hence, we focus on detection and localisation while highlighting promises of the correction.
Figures & tables
Fig. 1: Latent hallucination grows with imagination. Free-running a frozen world model on PointMaze : the top row is ground truth, the bottom row is the model’s imagined rollout decoded to images. Early on they agree, but by step 17 the imagined agent has drifted to a part of the maze it never visits. The prediction is still a plausible latent; nothing flags the error until the true future is known.
Fig. 2: MEND’s single-iteration pipeline. The world model predicts z^t+1 from (zt,at) and the detector scores it, D(z^t+1) . Two mitigation paths follow. Option 1 (inference-time correction) uses the localiser to select the suspicious tokens ( m ) and iteratively move them toward the valid manifold while freezing the confident tokens; Option 2 (data-driven repair) logs the flagged predictions for offline fine-tuning of the world model. Bottom: over K iterations the correction restores the hallucinated tokens (red) to trusted ones (grey).
Detection AUROC
Det.
Loc.
Environment
Ours
Gauss.
AUPRC
AUPRC
Wall
0.801
0.648
0.802
0.712
PointMaze
0.691
0.631
0.687
0.874
TABLE I: MEND ( Ours ) against a diagonal-Gaussian density reference [ 15 ] . Detection is AUROC; the last two columns are detection and localisation per-token AUPRC (random 0.50 ).
Fig. 3: Localising deep-rollout hallucinations on PointMaze (step 17 of a free-running rollout, where the imagined agent has drifted far from reality). Columns: decoded ground truth, decoded prediction, ground-truth per-token error, and the D1 detector score field (per-token AUPRC annotated). Even when the whole state is far off the manifold the detector still points at the tokens that are wrong. Cyan circles mark the nine most-erroneous tokens.
Detector ( Wall )
AUROC
Conditional score net (no action)
0.801
Directly conditioned score, +action (AdaLN)
0.799
Cross-attention, +action (15M)
0.782
Inverse model — D2 action factor
0.490
Directly conditioned score, − action
0.507
TABLE II: Ablation of detector variants on Wall . Variant architectures are detailed in the supplementary material.
Environment
Δ error ↓
% improved ↑
Wall
−6.4%
98.5
PointMaze
−3.0%
99.5
TABLE III: Single-step correction. Δ error is the relative change in latent error, so a negative value indicates improvement; arrows mark the better direction.
Depth
No corr.
MEND (D1) ↓
1
1.770
1.626 ( −8.2% )
5
2.370
2.237 ( −5.6% )
9
2.830
2.732 ( −3.5% )
TABLE IV: Per-step correction on Wall : mean per-token latent error versus rollout depth (lower is better). Parentheses show the reduction relative to No corr.
Department of Mechanical and Aerospace Engineering, University of Florida, Gainesville, FL 32611, USA · Department of Civil and Coastal Engineering, University of Florida, Gainesville, FL 32611, USA