Events, Not Pixels: Where World Models Get the Return Wrong

Haoyue Liu; Xiaoying Tang

cs.AIOct 21, 2026

Events, Not Pixels: Where World Models Get the Return Wrong

Authors: Haoyue Liu, Xiaoying Tang

Organizations: School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)

Abstract

World models are increasingly used for planning, where a planner ranks candidate actions by returns imagined by the model. However, these models are judged mainly by how well they render frames, even though a visually faithful rollout can still assign the wrong return. To identify where this mismatch arises, we introduce a transition-level diagnostic that replaces imagined transitions with real ones and measures the reduction in return error against a budget-matched random control. Our analysis yields two findings. (1) Event transitions, the few steps where a reward is earned or an episode ends, are rendered as well as other steps (median pixel-error ratio 0.85), yet restoring them removes a median of 70% of the return error. In contrast, restoring transitions with the largest MSE, SSIM or LPIPS error removes less than a random choice. Across the 44 Atari-100k model--collection pairs that pass our reliability checks, including three released families, event patches outperform random patches in 39 (sign test p=1.4 x 10^{-7}), with a median gain of +44 percentage points. (2) The remaining error is structured: a complementary patch separates errors at true events from those elsewhere, while an eight-step timing tolerance recovers few events missed by our deterministic models. This structure also guides repair: a rule recorded before four repair outcomes were known predicts when a diffusion model repairs these failures. Event patches improve plan ranking, and the same concentration appears on control tasks with rare reward events.

Figures & tables

Figure 1: Return error concentrates at event transitions that pixel error does not single out. A: A saved Seaquest MSE model predicts a 60-step future from 36 real context frames and recorded actions. B: Each selected detector input substitutes both frames of a transition, keeping its action fixed. C: Event and random masks each select three of sixty steps here; complement and all-real inputs complete the controls. D: Saved return-error reductions over 400 probes. E: Five selected times show real frames, predictions, absolute RGB residuals on a shared square-root colour scale, and the frames supplied by the two substitution arms. Coloured borders mark real-pair substitutions.

Figure 2: Targeting advantage across patch budgets. 9 sweeps, three games, 3 released families. Solid/dashed curves denote event/random patches; markers identify families. Lower panels show their difference, $L$ , at matched coverage. Dotted guides mark the second DIAMOND budget; annotations give its mean patched-step count and lift. All-frame reference ranges are listed above each panel. Sweeps use one sampling seed; Table 1 reports three-seed means.

Figure 3: Task-event transitions beyond Atari. Six tasks, two real trajectory segments per task, and three frames per segment. Blue borders mark reward onset, the first rewarded step after at least four reward-free steps. Cases are selected from real events, separated by more than 96 steps within each task. Aggregate targeting and detector results are in the supplement.

Game		DIAMOND [ 2 ]				IRIS [ 36 ]				Simulus [ 12 ]
	$q$ (%)	Ev	Non	Rnd	$L$	Ev	Non	Rnd	$L$	Ev	Non	Rnd	$L$
Pong	3.5	41	46	5	+36	48	38	4	+45	63	23	4	+59
Kangaroo	4.1	79	4	5	+74	98	0	5	+92	38	$-$ 1	2	+36
Freeway	4.6	68	$-$ 6	$-$ 4	+72	35	$-$ 38	$-$ 2	+38	72	$-$ 16	1	+71
Seaquest	10.7	57	$-$ 19	9	+48	$-$ 26	43	7	$-$ 33	54	$-$ 28	8	+45
Frostbite	12.0	57	17	$-$ 181	+238	71	$-$ 18	5	+65	83	$-$ 4	16	+66

Table 1: Event targeting separates concentrated errors from diffuse failures. Percent of return error removed by patching event (Ev), complementary (Non) or budget-matched random (Rnd) transitions (mean of three seeds); shading grows with the lift

L=\mathrm{Ev}-\mathrm{Rnd}

. Rows are sorted by event coverage

q

; grey pairs are outside the counted set, and games whose detector fails the gate are omitted (supplement).

Figure 4: Testing the proposed repair across five games. Five games, each shown as ground truth, deterministic MSE prediction, and sampling prediction under the same actions. Blue borders mark the real-event time; marks show whether the repair study’s detector reads the reward there ( ✓ ), a few steps off (+3) or not ( $\scriptstyle\times$ ). Each row uses a validation window selected from real-event timing alone.

Figure 5: The event deficit of the deterministic model persists under temporal tolerance. Reward-event recall versus alignment tolerance across five games. Header values compare MSE $\rightarrow$ sampling at $\tau=0$ ; green guides mark the sampling value. Each displaced null matches its model’s predicted event rate. Kangaroo is the pre-specified no-effect case for return-error improvement.

Selection	Medians over the 44 counted pairs
Patch chosen by	Removed	$L$	$L>0$	On events
Event transitions	70%	$+43.7$	39/44	100%
Largest MSE	15%	$-1.1$	17/44	19%
Largest 1 $-$ SSIM	11%	$-1.2$	15/44	15%
Largest LPIPS	15%	$-0.8$	17/44	16%

Table 2: Pixel error does not find the event transitions. Every row patches as many transitions as the event patch (seed 0). A random patch removes 17% and hits events at rate

q

(median 16%).

Game	Split	MSE	Sample	MSE $\downarrow$	Sample $\downarrow$	$\Delta E\uparrow$	Floor	$\times$ floor	Pred. / obs.
Evaluation setting		Event recall $\uparrow$		Return error			Decision margin
Freeway	development	0.091	0.943	0.6048	0.1545	+0.4503	0.050	9.0	helps ✓
Pong	prospective	0.348	0.773	0.2981	0.1421	+0.1560	0.079	2.0	helps ✓
Breakout	prospective	0.069	0.229	1.4138	0.8822	+0.5316	0.146	3.6	helps ✓
Seaquest	prospective	0.062	0.357	1.3490	0.5688	+0.7802	0.134	5.8	helps ✓

Table 3: Selected repair comparisons.

\Delta E=E_{\rm MSE}-E_{\rm sample}

. Errors compare competent-seed MSE means (30k updates) with designated sampling runs (8k). Freeway is the development result; shaded rows are prospective tests. Red gains exceed declared seed floors; ✓ marks matched predictions. The complete original study, including the fourth prospective outcome, is in the supplement.

Panel of Figure 6 — Figure 6: Scene transitions in additional environments. Figures A–B (MiniGrid) and C–D (Crafter) show real frames and saved RSSM/Transformer rollouts under the same recorded actions at steps 1, 13 and 30. Map orientation and night/day changes make trajectory divergence visible. Each rollout starts from 36 real frames; futures may cross episode boundaries. The supplement shows additional, nonoverlapping contexts.

World Model-Based Planning

World model-based planning uses models of how environments change to compare imagined actions and choose goal-directed behavior.

See all 261 papers

Not All Errors Matter: Decision-Relevant Prediction Error Predicts Planning Quality
Sep 26, 2026Linhao Wang, Yiyan Fan, Dongjin HuangWorld Model-Based Planning Model-Based Planning
Shanghai Film Academy, Shanghai University
The Intervention Gap in Latent World Models
Aug 30, 2026Donna VakalisWorld Model-Based Planning Model-Based Planning
Mila – Quebec AI Institute · University of Montreal
When Low Prediction Error Misleads Planning: Diagnosing Representation, Dynamics, and Decision Failures in Latent World Models
Oct 4, 2026Rui Min, Xianyao Li, Fang Xu +1World Model-Based Planning Action-Conditioned World Models
Department of Mechanical and Aerospace Engineering, University of Florida, Gainesville, FL 32611, USA · Department of Civil and Coastal Engineering, University of Florida, Gainesville, FL 32611, USA

Sep 26, 2026J/K move · Enter open · S save

Not All Errors Matter: Decision-Relevant Prediction Error Predicts Planning Quality

Linhao Wang, Yiyan Fan, Dongjin Huang

Shanghai Film Academy, Shanghai University

World Model-Based Planning Model-Based Planning World Model Evaluation Model-Based RL

Open paper AskPDF arXiv

World models are typically trained and evaluated by prediction error, assuming that more accurate predictions lead to better decisions. We show that this assumption can fail because models with similar total error can differ substantially in planning performance when their errors occur on different state dimensions. We introduce Decision-Relevant Prediction Error (DRPE), which measures prediction error on the state dimensions that affect decisions. We also develop an iso-error evaluation protocol that varies error allocation while keeping total error fixed. In a factored gridworld with known state relevance and a standardized planner, we evaluate 55 controlled and learned models across different error levels and allocations. Total prediction error is weakly related to planning success (Spearman

ρ=-0.25

), whereas DRPE is strongly predictive (

ρ=-0.84

;

-0.98

within the controlled family). Models with only a 1% difference in total error can differ by 60 percentage points in planning success (97% vs 37%). The relevant error also depends on the task, with model rankings reversing across tasks at the same total error. Deeper imagination further amplifies decision-relevant errors, while learned models exhibit systematic bias on rare but decision-critical events. We formalize sufficient conditions under which DRPE correctly ranks models and total prediction error cannot.

Abstract

World Model-Based Planning

Explore similar work

Not All Errors Matter: Decision-Relevant Prediction Error Predicts Planning Quality

The Intervention Gap in Latent World Models

When Low Prediction Error Misleads Planning: Diagnosing Representation, Dynamics, and Decision Failures in Latent World Models