When Low Prediction Error Misleads Planning: Diagnosing Representation, Dynamics, and Decision Failures in Latent World Models

Rui Min; Xianyao Li; Fang Xu; Jing Du

cs.LGOct 4, 2026

When Low Prediction Error Misleads Planning: Diagnosing Representation, Dynamics, and Decision Failures in Latent World Models

Authors: Rui Min, Xianyao Li, Fang Xu, Jing Du

Organizations: Department of Mechanical and Aerospace Engineering, University of Florida, Gainesville, FL 32611, USA · Department of Civil and Coastal Engineering, University of Florida, Gainesville, FL 32611, USA

Abstract

The component that dominates a latent world model's prediction error need not be the one whose repair most improves action selection. We show this by comparing action sequences from identical physical starts and separating endpoint error into a candidate-pool center and action-relative responses. Across four model families and four tasks, a confirmation pool of 256 new starts per task and 300 shared candidates per start shows that center error dominates MSE in 14/16 model-task cells. Yet in six of these cells, an oracle that corrects only the action-relative responses yields better physical rank correlation and top-30 elite quality than one that corrects only the center, while leaving more latent MSE (family-wise corrected intervals). The preference differs across the evaluated settings: a separate LeWorldModel (LeWM) study that executes oracle-selected actions favors center repair on PushT and on Reacher with a render-matched goal. Matched-candidate tests localize ordering loss: for LeWM, encoding realized endpoints raises physical Spearman from 0.464 to 0.975 on that Reacher setting and from 0.193 to 0.631 on PushT (64 starts per task), while Cube's encoded-goal cost remains uninformative. A 72-run objective study improves selected response diagnostics, while incremental closed-loop planning gains remain unconfirmed. These results separate error magnitude from the decision effects of oracle correction and motivate evaluating representation, prediction, and planning as separate stages.

Figures & tables

Figure 1: Overview of the diagnostic framework. (a) From one restored state, pair predicted and realized outcomes of alternative action sequences. (b) Separate center error from action-relative response error, then distinguish response scale from direction. (c) On identical candidates, compare predicted- versus realized-outcome costs ( $P/E$ ), and center- versus response-only oracle repairs ( $C/S$ ). (d) Test transfer from response fidelity to physical candidate ranking and elite quality, then closed-loop planning. Dashed arrows mark transfers assessed by separate tests. Latent geometry is schematic.

Model	Task	$\Delta\rho$	$\Delta q$	$\Delta M$
PreJEPA	PushT	0.2436 [0.1668, 0.3186]	0.0982 [0.0386, 0.1662]	0.3787 [0.3504, 0.4081]
PreJEPA	TwoRoom	0.2626 [0.1816, 0.3437]	0.0647 [0.0355, 0.0989]	0.2452 [0.1906, 0.3052]
PreJEPA	Reacher	0.2900 [0.2442, 0.3379]	0.7475 [0.6279, 0.8742]	1.0953 [1.0732, 1.1188]
DINO-WM	Reacher	0.2411 [0.1983, 0.2848]	0.4916 [0.3995, 0.5930]	1.6051 [1.5822, 1.6288]
DreamerV3	PushT	0.1852 [0.1264, 0.2428]	0.1201 [0.0720, 0.1880]	0.0283 [0.0240, 0.0329]
DreamerV3	TwoRoom	0.1357 [0.0544, 0.2190]	0.0410 [0.0061, 0.0825]	0.0066 [0.0043, 0.0092]

Table 1: Six of 16 new-start cells show an MSE–decision inversion. Four models share 256 starts per task and 300 candidates per start.

\Delta\rho=\rho_{S}-\rho_{C}

and

\Delta q=q_{C}-q_{S}

favor

S

;

\Delta M=\mathrm{MSE}_{S}-\mathrm{MSE}_{C}>0

means more error under

S

. Brackets are nominal 95% family-wise intervals over 80 comparisons. All 16 cells are summarized in Table 2 ; full intervals are in Appendix F.2 .

Model	PushT	TwoRoom	Reacher	Cube
PLDM	$?\,+\,+$	$?\,+\,+$	$+\,?\,+$	$?\,?\,+$
PreJEPA	$+\,+\,+$	$+\,+\,+$	$+\,+\,+$	$?\,+\,+$
DINO-WM	$+\,+\,?$	$?\,?\,+$	$+\,+\,+$	$?\,?\,+$
DreamerV3	$+\,+\,+$	$+\,+\,+$	$?\,?\,-$	$?\,?\,+$

Table 2: All 16 new-start cells. Signs of the three contrasts of Table 1 , in the order

\rho_{S}-\rho_{C}

q_{C}-q_{S}

\mathrm{MSE}_{S}-\mathrm{MSE}_{C}

. A sign is

+

-

when the 80-comparison family-wise interval lies wholly above or below zero, and

?

when it includes zero. In the first two positions

+

means better decisions under

S

; in the third,

+

means more residual MSE under

S

, so

+\,+\,+

is the inversion. Shaded cells are the six inversion cells; no cell has

-

in a decision position.

Figure 2: Matched candidates expose a task-dependent ordering gap.

P

scores the predicted outcome; oracle

E

scores the model’s representation of the realized outcome of the same action sequence. (top) Rank correlation of each cost with physical error on identical candidates. (bottom) Reduction in the physical error of the selected top-30 elites with its family-wise interval; units are task specific, so each task has its own scale. A star marks a rank gain

E-P

whose family-wise interval excludes zero (eight comparisons for LeWM; separate 32-comparison families for the feedforward and recurrent protocols). PushT recovers on both criteria for all five model families and Cube for none. For LeWM Reacher, the goal image is rendered with the same pipeline as the candidate outcomes (Appendix C ). Exact values are in Appendix Table 7 .

Figure 3: Aggregate prediction error can hide worse action response. (a–d) Share of the initial model’s latent prediction MSE carried by the center (common-mode) term against rollout horizon; the remainder is the action-relative response term. (e) PushT, direction objective minus its same-budget base, class means over three seeds: total MSE falls because the center term falls, while the response term rises beyond the training horizon (dashed line). At H5/H20, whiskers are descriptive 95% intervals for the components and retrospectively frozen 120-comparison family-wise intervals for total MSE.

Figure 4: Response shrinkage and the closed-loop conversion boundary. (a–d) Gain

g

(solid) and direction cosine

c

(dashed) versus rollout horizon: one initial checkpoint and trained-arm means over three paired seeds on 1,024 held-out starts. The vertical line marks training H5; the initial Reacher gain (

\approx

3.8, an observation-domain shift) is above the plotted range. The identity

Q=(1-c^{2})+(g-c)^{2}

separates a direction floor (for

c\geq 0

) from amplitude excess. Squared-error fine-tuning moves gain toward cosine; pinning gain over-expands on Reacher beyond H5. (e,f) Transition minus same-budget base: H5

Q

improvement and native CEM success change, with family-wise intervals (80 and 12 comparisons). Every CEM interval includes zero; task scores are not pooled.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Study	Unit	Draws	$m$	Bootstrap seed
Preregistered replication (Appendix H )	Start	20,000	4	22092041
Exploratory repair (Appendix F.1 )	Start	100,000	96	$744021+i_{c}$
New-start repair (Appendix F.2 )	Start	100,000	80	$748321+i_{t}$
Directional continuation (Appendix I.1 )	Start	50,000	24	730021
Condition-matched directions	Group	50,000	32	735021
True-history control	Group	50,000	16	736021

Appendix

Table 3: Resampling settings for the preregistered, oracle and exploratory studies. Each row is a separate comparison family with nominal 95% family-wise coverage; the two-sided percentile interval uses tail probability

0.05/(2m)

for

m

comparisons. Starts or episodes are paired before resampling; a group contains four starts linked by donor matching.

	$\Delta\log$ total error		$\Delta Q$		$\Delta\rho_{P}$		$\Delta\rho_{E}$
Task	est.	CI	est.	CI	est.	CI	est.	CI
PushT	$-$ 1.719	[ $-$ 2.209, $-$ 1.283]	$-$ .253	[ $-$ .524, $-$ .043]	.132	[.000, .262]	.036	[ $-$ .036, .109]
TwoRoom	$-$ 1.149	[ $-$ 1.321, $-$ .974]	$-$ .888	[ $-$ 1.579, $-$ .413]	.046	[ $-$ .067, .158]	$-$ .147	[ $-$ .386, .074]
Reacher †	.192	[.082, .314]	17.773	[15.718, 20.068]	.466	[.299, .627]	.293	[.087, .487]
Cube	$-$ .330	[ $-$ .575, $-$ .122]	$-$ .017	[ $-$ .142, .104]	.022	[ $-$ .017, .070]	.093	[.011, .189]

Appendix

Table 4: Change from epoch 1 to epoch 10 on the frozen candidates, with 20-comparison family-wise intervals. Bold marks an interval that excludes zero.

\dagger

: protocol differs from the LeWM Reacher row (see text).

Figure 5: Along training, error falls mainly through the center (common-mode) term while predictor ranking lags. Top: total prediction error and its two exact components, relative to true response energy (log scale). Bottom: rank correlation with physical error for the predicted-outcome cost

P

and the encoded-outcome cost

E

; bands are descriptive 95% start-bootstrap intervals. The ten checkpoints of a run are nested, not independent runs.

\dagger

: Reacher’s encoded quantities use natively rendered endpoints and are not comparable with the LeWM Reacher row.

		Response	Response error $Q$		Gain	Cosine	Branch visibility
Model	Task	share	est.	95% CI	$g$	$c$	true	predicted
PLDM	PushT	.690	.665	[.569, .769]	.894	.634	.612	.588
	TwoRoom	.691	.067	[.041, .099]	1.005	.967	.673	.679
	Reacher	.755	.342	[.288, .401]	1.004	.830	.744	.745
	Cube	.741	.770	[.686, .854]	.832	.554	.740	.743
PreJEPA	PushT	.489	1.348	[1.292, 1.411]	.707	.107	.354	.223

Appendix

Table 5: Same-start response geometry at H5 for four further model families (64 starts,

K=4

). Response share is the action-relative fraction of total latent MSE;

Q

is the ratio-of-sums normalized response error with a descriptive 95% start-bootstrap interval; visibility is the within-start share of latent variance. PreJEPA and DINO-WM rows are conditional on zero placeholders for extra context that the branch banks did not record. Values are within-model and do not rank architectures.

Task	Initial	Base	Transition	Branch	$\Delta$	Family-wise CI
PushT	89.06	82.42	83.07	82.42	0.651	[ $-$ 2.344, 3.646]
TwoRoom	85.94	86.20	85.55	86.33	$-$ 0.651	[ $-$ 1.953, 0.521]
Reacher	58.98	58.46	56.12	59.51	$-$ 2.344	[ $-$ 8.073, 3.516]
Cube	63.67	64.58	65.76	64.58	1.172	[ $-$ 1.823, 4.297]

Appendix

Table 6: Native CEM success and the transition-minus-base contrast (

\Delta

). Base, transition, and branch average three trained seeds; initial is one checkpoint. Printed intervals are 12-comparison family-wise and all include zero. The distinct PushT base-minus-initial contrast is

-6.641

percentage points [

{-11.979},{-1.432}

] under the same family; it measures shared fine-tuning rather than an added response loss.

		Rank corr. $\rho$		Rank gain $E-P$		Elite-error reduction $P-E$
Model	Task	$P$	$E$	est.	family-wise CI	est.	family-wise CI
LeWM	PushT	.193	.631	.438	[.319, .556]	.175	[.110, .248]
	TwoRoom	.060	.300	.241	[.018, .448]	.028	[ $-$ .037, .086]
	Reacher	.464	.975	.511	[.449, .574]	1.337	[1.025, 1.681]
	Cube	.020	.033	.013	[ $-$ .095, .122]	.030	[ $-$ .103, .202]
PLDM	PushT	.267	.549	.282	[.165, .409]	.108	[.056, .180]

Appendix

Table 7: Matched-candidate contrast: predicted-outcome cost

P

versus encoded-realized-outcome cost

E

on identical candidates (64 starts, 300 candidates, top-30 elites). Gains are oriented so that positive favors

E

; elite error is in task-specific units. Bold marks a family-wise interval that excludes zero: eight comparisons for LeWM and separate prespecified 32-comparison families for feedforward and recurrent cross-model protocols.

	PushT		TwoRoom		Reacher		Cube
Model	mask	shuffle	mask	shuffle	mask	shuffle	mask	shuffle
LeWM	30.1	51.7	195	229	17.2	28.9	125	193
PLDM	47.9	74.4	115	137	12.2	19.4	106	140
PreJEPA	0.79	1.07	3.18	4.64	1.05	1.61	1.60	2.01
DreamerV3	0.20	0.34	4.60	6.53	1.61	2.38	4.34	4.96
DINO-WM	4.17	5.47	16.4	23.6	10.6	14.3	5.78	7.23

Appendix

Table 8: Action-use control at the final checkpoint: relative increase of one-step latent prediction loss (loss increase divided by true-action loss) when the action is masked or shuffled across 512 fixed probe windows. Every entry is positive; magnitudes are within-model only.

Task	$\Delta Q$	Family-wise interval	$\Delta$ cosine
PushT	$-.0263$	[ $-.0485$ , $-.0055$ ]	$+.0323$
TwoRoom	$-.0070$	[ $-.0541$ , .0314]	$+.0035$
Reacher	$-.0645$	[ $-.0748$ , $-.0545$ ]	$+.0522$
Cube	$-.0402$	[ $-.0531$ , $-.0279$ ]	$+.0217$

Appendix

Table 9: Action-block coverage control. Top: varying-action base minus repeated-action base at H5;

Q

intervals use the 80-comparison family, whereas cosine changes are descriptive three-seed means. Bottom: varying-minus-repeated native-CEM success in percentage points for each objective, with 24-comparison family-wise intervals over 256 paired source episodes per task.

Model / task	$f_{c}$	$\rho_{P}$	$\rho_{C}$	$\rho_{S}$	$\rho_{E}$	$q_{P}$	$q_{C}$	$q_{S}$	$q_{E}$
PLDM / PushT	0.580	0.277	0.350	0.354	0.549	0.408	0.377	0.368	0.297
PLDM / TwoRoom	0.668	0.094	0.094	0.234	0.322	1.689	1.702	1.652	1.640
PLDM / Reacher	0.654	0.718	0.738	0.842	0.737	1.142	1.175	1.093	1.592
PLDM / Cube	0.754	0.007	0.038	0.116	-0.052	1.290	1.298	1.234	1.373
PreJEPA / PushT	0.829	0.116	0.182	0.503	0.516	0.502	0.427	0.321	0.308
PreJEPA / TwoRoom	0.761	0.093	0.114	0.473	0.642	1.685	1.687	1.612	1.581

Appendix

Table 10: All-mode means and aggregate center-error fraction

f_{c}

. Order is

P,C,S,E

in both metric blocks. These fractions are descriptive, not causal responsibility. Elite means use task-specific physical units.

Model / task	$C-P$	$S-P$	$C-S$
PLDM / PushT	0.0738 [-0.0147, 0.1656]	0.0769 [-0.0278, 0.1676]	-0.0032 [-0.1100, 0.1069]
PLDM / TwoRoom	0.0000 [-0.0997, 0.0656]	0.1407 [-0.0782, 0.3582]	-0.1407 [-0.3637, 0.0724]
PLDM / Reacher	0.0207 [-0.0169, 0.0615]	0.1248 [0.0380, 0.2060]	-0.1041 [-0.1864, -0.0046]
PLDM / Cube	0.0310 [-0.0000, 0.0641]	0.1086 [0.0130, 0.2013]	-0.0776 [-0.1725, 0.0211]
PreJEPA / PushT	0.0664 [-0.0651, 0.1973]	0.3870 [0.2087, 0.5717]	-0.3206 [-0.4786, -0.1586]
PreJEPA / TwoRoom	0.0218 [-0.0712, 0.1008]	0.3806 [0.1966, 0.5325]	-0.3588 [-0.5301, -0.1774]

Appendix

Table 11: Center/response localization: physical Spearman, 48 of the 96 contrasts in this comparison family. Positive favors the first mode in each column; negative

C-S

favors response repair. All 16 configurations are retained.

Model / task	$C-P$	$S-P$	$C-S$
PLDM / PushT	0.0315 [0.0009, 0.0700]	0.0397 [0.0009, 0.0941]	-0.0082 [-0.0554, 0.0337]
PLDM / TwoRoom	-0.0121 [-0.0462, 0.0076]	0.0371 [-0.0229, 0.0929]	-0.0492 [-0.1192, 0.0114]
PLDM / Reacher	-0.0334 [-0.1552, 0.0782]	0.0484 [-0.2161, 0.2478]	-0.0818 [-0.2909, 0.1828]
PLDM / Cube	-0.0081 [-0.0995, 0.0595]	0.0560 [-0.0152, 0.1778]	-0.0641 [-0.1750, 0.0221]
PreJEPA / PushT	0.0755 [-0.0125, 0.2086]	0.1813 [0.0845, 0.3330]	-0.1058 [-0.1818, -0.0521]
PreJEPA / TwoRoom	-0.0013 [-0.0357, 0.0215]	0.0732 [0.0270, 0.1101]	-0.0745 [-0.1135, -0.0354]

Appendix

Table 12: Center/response localization: elite-error reduction, 48 of the 96 contrasts in this comparison family. Positive favors the first mode in each column; negative

C-S

favors response repair. All 16 configurations are retained.

Figure 6: The 16-cell new-start confirmation. Each model–task cell shows signs of the 80-comparison family-wise intervals for

\rho_{S}-\rho_{C}

(rank),

q_{C}-q_{S}

(top-30 elite improvement), and

\mathrm{MSE}_{S}-\mathrm{MSE}_{C}

(more residual MSE under

S

), in that order.

+

means an interval wholly above zero,

-

wholly below, and

?

unresolved; an outline marks all three positive (six cells). Models share 256 physical starts within each task and the same 300 LeWM-proposed candidates per start. Physical elite-error units are task-specific; exact effects and intervals for all cells are in Tables 13 – 15 .

Model / task	S-C rank	S-C elite	S-C MSE	Shape-Amp rank	Shape-Amp elite
PLDM / PushT	?	+	+	+	+
PLDM / TwoRoom	?	+	+	+	+
PLDM / Reacher	+	?	+	+	+
PLDM / Cube	?	?	+	?	?
PreJEPA / PushT	+	+	+	+	+
PreJEPA / TwoRoom	+	+	+	+	+

Appendix

Table 13: New-start confirmation: 256 starts per task, disjoint from the exploratory starts, and 300 identical LeWM-proposed candidates per start; overlap with pretraining episodes is not ruled out. Entries are the signs of the five preregistered contrasts, with intervals adjusted over all 80 comparisons, in order:

\rho_{S}-\rho_{C}

q_{C}-q_{S}

\mathrm{MSE}_{S}-\mathrm{MSE}_{C}

\rho_{\rm SHAPE}-\rho_{\rm AMP}

q_{\rm AMP}-q_{\rm SHAPE}

+

denotes an adjusted interval wholly above zero,

-

wholly below, and ? unresolved. Cells are not pooled. All repairs are oracles that use realized outcomes, not deployable methods or closed-loop results.

Model / task	$\rho_{S}-\rho_{C}$	$q_{C}-q_{S}$	$\mathrm{MSE}_{S}-\mathrm{MSE}_{C}$
PLDM / PushT	0.0567 [-0.0049, 0.1165]	0.0421 [0.0044, 0.0914]	0.0735 [0.0316, 0.1235]
PLDM / TwoRoom	0.0791 [-0.0140, 0.1714]	0.0291 [0.0023, 0.0556]	0.3166 [0.1868, 0.4504]
PLDM / Reacher	0.1336 [0.0968, 0.1709]	0.0515 [-0.0389, 0.1318]	0.1164 [0.0692, 0.1683]
PLDM / Cube	0.0333 [-0.0119, 0.0788]	0.0294 [-0.0240, 0.0919]	0.6783 [0.4983, 0.8621]
PreJEPA / PushT	0.2436 [0.1668, 0.3186]	0.0982 [0.0386, 0.1662]	0.3787 [0.3504, 0.4081]
PreJEPA / TwoRoom	0.2626 [0.1816, 0.3437]	0.0647 [0.0355, 0.0989]	0.2452 [0.1906, 0.3052]

Appendix

Table 14: Center-versus-response repair on the new starts: all 48 contrasts, with intervals adjusted over all 80 comparisons. Positive rank/elite values favor response repair S; positive residual-MSE values mean S leaves more latent error than center repair C. Bold indicates a family-wise interval excluding zero. Elite units are task-specific physical errors.

Model / task	$\rho_{\rm SHAPE}-\rho_{\rm AMP}$	$q_{\rm AMP}-q_{\rm SHAPE}$
PLDM / PushT	0.1367 [0.0838, 0.1930]	0.0817 [0.0413, 0.1382]
PLDM / TwoRoom	0.0917 [0.0120, 0.1709]	0.0280 [0.0009, 0.0560]
PLDM / Reacher	0.1890 [0.1466, 0.2329]	0.1582 [0.0577, 0.2508]
PLDM / Cube	0.0365 [-0.0088, 0.0819]	0.0173 [-0.0380, 0.0847]
PreJEPA / PushT	0.3159 [0.2471, 0.3882]	0.1633 [0.1084, 0.2338]
PreJEPA / TwoRoom	0.2987 [0.2301, 0.3664]	0.0745 [0.0504, 0.1010]

Appendix

Table 15: Response-configuration versus amplitude-only oracle repair on the new starts: all 32 contrasts, with intervals adjusted over all 80 comparisons. SHAPE keeps predicted total response energy while substituting the true normalized branch configuration; AMP rescales the predicted response to the true total energy. Positive favors SHAPE. No Cube cell supports both decision contrasts; individual contrasts may resolve.

Task	Fixed elites	Executed mean	Change	Executed $C-S$ status
PushT	$-.041220$	$-.069815$	$-.028595$	12-comparison: supported
TwoRoom	$.064196$	$.036242$	$-.027954$	12-comparison: unresolved
Reacher (render-matched)	$-.343188$	$-.489133$	$-.145945$	4-comparison: supported
Cube (full render)	$.014597$	$-.073012$	$-.087609$	12-comparison: unresolved

Appendix

Table 16: Retrospective first-replan comparison on audited LeWM oracles (

64

starts/task, final CEM iteration, 300 candidates, top 30). Fixed-elite and change columns are descriptive; the executed-mean status is inherited from the original protocols. Physical-error units differ by task.

		Base $-$ initial				Direction $-$ base
		$\Delta$ floor		$\Delta$ excess		$\Delta$ gain		$\Delta$ excess
Task	H	est.	CI	est.	CI	est.	CI	est.	CI
PushT	H5	$-$ .010	[ $-$ .033, .012]	$-$ .050	[ $-$ .071, $-$ .032]	.075	[.065, .087]	.006	[.001, .011]
PushT	H20	.007	[ $-$ .004, .019]	$-$ .274	[ $-$ .332, $-$ .222]	.155	[.137, .174]	.040	[.028, .054]
TwoRoom	H5	$-$ .020	[ $-$ .037, $-$ .007]	$-$ .002	[ $-$ .003, $-$ .001]	.002	[.001, .002]	.000	[.000, .000]
TwoRoom	H20	$-$ .246	[ $-$ .298, $-$ .196]	$-$ .091	[ $-$ .130, $-$ .062]	.003	[.001, .004]	.000	[ $-$ .001, .001]

Appendix

Table 17: Shrinkage-law contrasts (class means over three seeds; paired start bootstrap, 32-comparison family-wise intervals). Base

-

initial: change in the direction floor

1-c^{2}

and in the amplitude excess

(g-c)^{2}

under squared-error fine-tuning. Direction

-

base: change in gain and in excess when the gain is pinned. Bold marks an interval that excludes zero.

Task	H	est.	CI	est.	CI	est.	CI	est.	CI	outcome
		$\Delta$ floor		$\Delta$ regret		$\Delta$ gain		$\Delta$ excess
PushT	H5	$-$ .025	[ $-$ .043, $-$ .009]	$-$ .075	[ $-$ .169, .017]	.258	[.235, .284]	.061	[.041, .085]	direction only
PushT	H20	$-$ .016	[ $-$ .025, $-$ .007]	$-$ .137	[ $-$ .321, .039]	.535	[.494, .578]	.336	[.280, .400]	direction only
TwoRoom	H5	$-$ .003	[ $-$ .005, $-$ .000]	.000	[ $-$ .003, .005]	.008	[.006, .010]	.000	[.000, .001]	direction only
TwoRoom	H20	$-$ .004	[ $-$ .018, .010]	.001	[ $-$ .012, .015]	.015	[.010, .020]	.002	[.000, .005]	null
Reacher	H5	$-$ .110	[ $-$ .118, $-$ .102]	$-$ .094	[ $-$ .217, .022]	.437	[.425, .448]	.084	[.070, .099]	direction only

Appendix

Table 18: Direction plus log-amplitude objective minus the squared-error base (class means over three paired seeds; paired start bootstrap). Floor and regret are the primary contrasts (16-comparison family); gain and excess are secondary (40-comparison family). Bold marks an interval that excludes zero.

Horizon	Quantity	Predicted sign	Estimate	Family-wise interval	Outcome
H20	$Q$	$+$	$+.0438$	[.0327, .0558]	Predicted direction
H20	Regret	$-$	$-.0635$	[ $-.1273$ , .0007]	Unresolved
H5	$Q$	$+$	$-.0219$	[ $-.0285$ , $-.0154$ ]	Opposite direction
H5	Regret	$-$	$-.0251$	[ $-.0586$ , .0084]	Unresolved

Appendix

Table 19: All four primary contrasts of the preregistered replication, NS minus SQ. Intervals are Bonferroni-adjusted over the four prespecified comparisons. Positive

Q

favors the squared-error class; negative regret favors the non-shrinkage class.

Model / task	$\Delta$ H6 MSE	$\Delta\kappa$	$\Delta$ H20 regret
LeWM / PushT	0.09760 [0.08369, 0.11317]	0.15049 [0.13037, 0.17288]	0.40443 [0.24743, 0.57578]
LeWM / TwoRoom	0.01509 [0.01105, 0.02126]	0.47289 [0.42852, 0.51998]	0.00686 [-0.00728, 0.02228]
LeWM / Reacher	0.00239 [0.00142, 0.00339]	0.11995 [0.10766, 0.13297]	0.05231 [-0.14183, 0.25165]
LeWM / Cube	0.04657 [0.03544, 0.05851]	0.25722 [0.22777, 0.28822]	-0.00132 [-0.02446, 0.02031]
PLDM / PushT	0.17455 [0.14686, 0.20350]	0.32276 [0.27144, 0.38276]	0.18968 [0.00861, 0.38629]
PLDM / TwoRoom	0.03597 [0.02470, 0.05042]	0.46066 [0.40792, 0.52052]	0.01753 [-0.00385, 0.04778]

Appendix

Table 20: All confirmation contrasts,

P-M

: fixed donors from an earlier bank, 1,024 starts/task, complete 24-comparison family. Positive MSE/regret means the own residual is worse; positive gain means greater squared response. Physical-score units differ across tasks. Bold marks adjusted intervals excluding zero.

Model / task	$\kappa(P)-\kappa(L)$	$\kappa(L)-\kappa(X)$
LeWM / PushT	0.00637 [-0.00868, 0.02078]	0.13593 [0.11516, 0.15885]
LeWM / TwoRoom	0.09405 [0.07493, 0.11516]	0.37594 [0.33789, 0.41538]
LeWM / Reacher	0.01826 [0.01558, 0.02109]	0.10342 [0.09365, 0.11354]
LeWM / Cube	0.04302 [0.03167, 0.05474]	0.22768 [0.19922, 0.25816]
PLDM / PushT	0.04572 [0.01504, 0.07566]	0.26685 [0.21957, 0.32254]
PLDM / TwoRoom	0.08714 [0.06705, 0.11183]	0.37484 [0.33200, 0.42442]

Appendix

Table 21: Condition-matched squared directional gains (32-comparison family; 256 four-start groups). Same-start

L

exceeds cross-start

X

in all eight settings; an unresolved

P-L

contrast is not equality.

Model / task	MSE $(P)-$ MSE $(L)$	MSE $(L)-$ MSE $(X)$
LeWM / PushT	0.03390 [0.02521, 0.04346]	0.06377 [0.05404, 0.07492]
LeWM / TwoRoom	0.00514 [0.00261, 0.00902]	0.01007 [0.00781, 0.01336]
LeWM / Reacher	-0.00053 [-0.00083, -0.00024]	0.00278 [0.00189, 0.00370]
LeWM / Cube	0.00690 [0.00274, 0.01162]	0.04122 [0.03063, 0.05193]
PLDM / PushT	0.05397 [0.03888, 0.06953]	0.11496 [0.09392, 0.13698]
PLDM / TwoRoom	0.00774 [0.00418, 0.01241]	0.02878 [0.01945, 0.04063]

Appendix

Table 22: Condition-matched H6 MSE differences, retaining the same 32-comparison family. The LeWM/Reacher

P-L

reversal is reported, not removed.

Model / task	True-history $\kappa(L)-\kappa(X)$	True minus predicted history
LeWM / PushT	0.13669 [0.11662, 0.15862]	0.00076 [-0.00058, 0.00230]
LeWM / TwoRoom	0.37571 [0.34073, 0.41245]	-0.00023 [-0.00048, -0.00000]
LeWM / Reacher	0.10458 [0.09516, 0.11400]	0.00117 [0.00033, 0.00200]
LeWM / Cube	0.23265 [0.20535, 0.26142]	0.00498 [0.00214, 0.00802]
PLDM / PushT	0.25863 [0.21590, 0.30631]	-0.00821 [-0.01854, 0.00037]
PLDM / TwoRoom	0.37458 [0.33537, 0.41937]	-0.00026 [-0.00086, 0.00035]

Appendix

Table 23: True-history control (16-comparison family; 256 four-start groups). All

L-X

gains remain positive, but changes relative to predicted history do not establish equivalence.

Task	Original	Initial	Anchor	Joint	Fixed
PushT	90.62	84.90	84.64	83.46	84.77
TwoRoom	85.16	85.16	84.51	84.64	84.90
Reacher	58.59	56.90	62.11	59.24	59.90
Cube	68.36	70.44	73.57	72.27	72.53

Appendix

Table 24: Native-CEM success (%). Original is the unmodified pretrained checkpoint; other columns average three fixed fine-tuning initializations on the same 256 episodes per task. Fixed denotes fixed-reference training, with only the student deployed. These rates are descriptive, not separate significance tests.

Task	Contrast	Difference	Adjusted interval	Status
PushT	Fixed-anchor	+0.13	$[-2.47,+2.73]$	Unresolved
PushT	Fixed-joint	+1.30	$[-1.43,+4.17]$	Unresolved
TwoRoom	Fixed-anchor	+0.39	$[-1.04,+1.82]$	Unresolved
TwoRoom	Fixed-joint	+0.26	$[-0.91,+1.56]$	Unresolved
Reacher	Fixed-anchor	-2.21	$[-6.90,+2.60]$	Unresolved
Reacher	Fixed-joint	+0.65	$[-4.30,+5.47]$	Unresolved

Appendix

Table 25: All eight predeclared fixed-reference CEM success contrasts, in percentage points (positive favors Fixed). Intervals use 50,000 paired-episode bootstrap draws and a Bonferroni correction over the eight comparisons (99.375% per interval). All include zero: no confirmed benefit or equivalence.

Task	Initialization	Initial	Anchor	Joint	Fixed
PushT	94111	213	220	215	214
PushT	94112	213	215	218	220
PushT	94113	226	215	208	217
TwoRoom	94111	219	216	218	216
TwoRoom	94112	216	218	216	218
TwoRoom	94113	219	215	216	218

Appendix

Table 26: All fine-tuning-initialization CEM success counts, each out of the same 256 episodes per task. Original counts are PushT 232, TwoRoom 218, Reacher 150 and Cube 175. Repeated models do not add independent physical episodes.

Model / task	$f_{P}$	$f_{X}$	$f_{P}-f_{X}$
LeWM / PushT	0.2419	0.0769	0.164945 [0.152492, 0.177439]
LeWM / TwoRoom	0.3646	0.0527	0.311914 [0.298531, 0.325415]
LeWM / Reacher	0.2467	0.1287	0.117934 [0.107290, 0.128552]
LeWM / Cube	0.2367	0.0395	0.197243 [0.186697, 0.208054]
PLDM / PushT	0.2214	0.0592	0.162224 [0.151556, 0.173043]
PLDM / TwoRoom	0.3445	0.0301	0.314367 [0.297506, 0.331499]

Appendix

Table 27: Held-out action-secant projection fractions. Own residual

P

versus cross-start residual

X

, with the recipient branch excluded from the reference span. All eight

P-X

contrasts have positive intervals (8-comparison family). Fractions, not percentages; this finite span is not a full physical tangent space.

Model / task	$\Delta$ energy $N-A$	$\Delta$ energy $T-B$
LeWM / PushT	-0.007923 [-0.010538, -0.005528]	-0.003829 [-0.005957, -0.001738]
LeWM / TwoRoom	-0.002766 [-0.005804, -0.001479]	-0.000556 [-0.001896, 0.000238]
LeWM / Reacher	-0.000820 [-0.001018, -0.000630]	0.000136 [-0.000035, 0.000300]
LeWM / Cube	-0.007049 [-0.009365, -0.004987]	-0.001851 [-0.004239, 0.000105]
PLDM / PushT	-0.032906 [-0.041168, -0.025480]	-0.011992 [-0.017771, -0.006405]
PLDM / TwoRoom	-0.005907 [-0.009035, -0.003747]	-0.001383 [-0.003707, 0.000268]

Appendix

Table 28: Finite component interventions, response energy: all 16 contrasts of the 32-comparison family. Negative

N-A

was expected; positive

T-B

was expected but is supported in no setting. Bold denotes an adjusted interval excluding zero, irrespective of the expected sign.

Model / task	$\Delta$ MSE $N-A$	$\Delta$ MSE $T-B$
LeWM / PushT	-0.007899 [-0.010986, -0.004895]	-0.007218 [-0.010145, -0.004299]
LeWM / TwoRoom	-0.002800 [-0.005834, -0.001485]	-0.000578 [-0.001976, 0.000240]
LeWM / Reacher	0.000076 [-0.000294, 0.000449]	0.000020 [-0.000353, 0.000387]
LeWM / Cube	-0.004531 [-0.007945, -0.001152]	0.002280 [-0.001649, 0.006165]
PLDM / PushT	-0.015862 [-0.025823, -0.006149]	-0.008323 [-0.016075, -0.000516]
PLDM / TwoRoom	-0.006255 [-0.009440, -0.003940]	-0.000613 [-0.002971, 0.001222]

Appendix

Table 29: Finite component interventions, next-step MSE: the other 16 contrasts of the same 32-comparison family. PLDM/Reacher

T-B

has lower response energy but higher MSE; stronger contraction is not automatically more accurate. Raw latent units are not comparable across encoders.

Model / task	Response cross term	Nonadditive correction	Total excess
LeWM / PushT	0.016418	-0.010174	0.006244
LeWM / TwoRoom	0.002928	-0.000907	0.002020
LeWM / Reacher	0.000711	-0.000396	0.000315
LeWM / Cube	0.007868	-0.002043	0.005826
PLDM / PushT	0.048264	-0.030126	0.018138
PLDM / TwoRoom	0.006832	-0.002144	0.004688

Appendix

Table 30: Post-hoc descriptive finite-response decomposition. Cross terms are positive in all eight sample means, but nonlinear corrections are negative throughout and total excess is negative for PLDM/Reacher and Cube. No confidence intervals or additional hypothesis tests were introduced.

Model / task	Real coupling	Real minus random
LeWM / PushT	0.004066 [-0.003901, 0.011637]	-0.140713 [-0.150916, -0.131311]
LeWM / TwoRoom	0.082070 [0.064108, 0.099741]	-0.127948 [-0.144895, -0.112618]
LeWM / Reacher	0.017043 [0.011433, 0.022641]	-0.072477 [-0.076673, -0.068296]
LeWM / Cube	0.031253 [0.020827, 0.041818]	-0.086397 [-0.095481, -0.077789]
PLDM / PushT	0.044083 [0.024719, 0.065862]	-0.087057 [-0.101807, -0.073359]
PLDM / TwoRoom	0.071182 [0.055293, 0.087785]	-0.096474 [-0.111827, -0.082444]

Appendix

Table 31: Small-scale coupling versus energy-matched random orthogonal decompositions (

\lambda=1/64

; 16-comparison family). Real action-secant coupling is smaller in all eight settings. Positive real coupling alone does not establish action-specific excess coupling.

Model / task	Teacher $L-X$	Teacher minus native $L-X$
LeWM / PushT	0.112495 [0.085692, 0.143114]	-0.024197 [-0.049751, 0.005279]
LeWM / TwoRoom	0.214829 [0.184216, 0.247567]	-0.160880 [-0.188319, -0.133072]
LeWM / Reacher	0.057607 [0.051156, 0.064272]	-0.046976 [-0.055577, -0.038620]
LeWM / Cube	0.194810 [0.167430, 0.223418]	-0.037843 [-0.060604, -0.013599]
PLDM / PushT	0.210159 [0.173971, 0.252970]	-0.048474 [-0.090733, -0.008256]
PLDM / TwoRoom	0.203771 [0.174429, 0.236751]	-0.170810 [-0.204543, -0.141825]

Appendix

Table 32: True-history residual generation: all 16 contrasts (16-comparison family). The continuation map and each recipient error norm remain fixed. Bold indicates an adjusted interval excluding zero, not necessarily the hypothesized sign.

Model / task	Persistence $L-X$	Teacher minus persistence
LeWM / PushT	0.067476 [0.040936, 0.095913]	0.046232 [0.012818, 0.083248]
LeWM / TwoRoom	0.334933 [0.274951, 0.398615]	-0.120104 [-0.182330, -0.060132]
LeWM / Reacher	0.176905 [0.164643, 0.189121]	-0.119298 [-0.131259, -0.107234]
LeWM / Cube	0.188272 [0.149008, 0.228418]	0.006538 [-0.036826, 0.050508]
PLDM / PushT	0.305623 [0.245357, 0.376711]	-0.095089 [-0.163596, -0.033823]
PLDM / TwoRoom	0.409728 [0.346408, 0.477856]	-0.205957 [-0.270396, -0.143363]

Appendix

Table 33: Negative control from real cumulative change, all 16 associated contrasts of the 32-comparison family. Persistence directions are

E_{0}-E_{5}

, not learned predictor residuals. PushT uses 251 valid four-start groups for both arms; other tasks use 256.

Model / task	Centroid $L-X$	Teacher minus centroid
LeWM / PushT	-0.087113 [-0.103184, -0.070151]	0.199608 [0.168100, 0.234424]
LeWM / TwoRoom	0.078492 [0.019825, 0.141365]	0.136337 [0.074545, 0.196322]
LeWM / Reacher	0.351561 [0.333254, 0.369986]	-0.293955 [-0.310126, -0.277841]
LeWM / Cube	0.105642 [0.077123, 0.135579]	0.089168 [0.051830, 0.127332]
PLDM / PushT	-0.007828 [-0.041842, 0.026288]	0.217987 [0.167479, 0.273493]
PLDM / TwoRoom	0.221413 [0.169795, 0.273275]	-0.017642 [-0.070652, 0.036955]

Appendix

Table 34: Fixed center of an independent bank minus

E_{5}

: the remaining 16 contrasts of the 32-comparison family. All cases use 256 groups. Positive conditional directional gain is not a predictor-error-specific signature.

Figure 7: Global representation collapse makes prediction look easy while planning degrades. Removing the global regularizer lowers H5 latent prediction MSE by more than four orders of magnitude (a) while closed-loop CEM success falls on every task (b). Historical evaluation sizes are heterogeneous (mostly one 50-episode seed; healthy Reacher five seeds, bar

\pm

1 s.e.): a motivating positive control, not a paired confirmatory experiment.

Explore similar work

Sep 26, 2026cs.LG

Not All Errors Matter: Decision-Relevant Prediction Error Predicts Planning Quality

World models are typically trained and evaluated by prediction error, assuming that more accurate predictions lead to better decisions. We show that this assumption can fail because models with similar total error can differ substantially in planning performance when their errors occur on different state dimensions. We introduce Decision-Relevant Prediction Error (DRPE), which measures prediction error on the state dimensions that affect decisions. We also develop an iso-error evaluation protocol that varies error allocation while keeping total error fixed. In a factored gridworld with known state relevance and a standardized planner, we evaluate 55 controlled and learned models across different error levels and allocations. Total prediction error is weakly related to planning success (Spearman

ρ=-0.25

), whereas DRPE is strongly predictive (

ρ=-0.84

;

-0.98

within the controlled family). Models with only a 1% difference in total error can differ by 60 percentage points in planning success (97% vs 37%). The relevant error also depends on the task, with model rankings reversing across tasks at the same total error. Deeper imagination further amplifies decision-relevant errors, while learned models exhibit systematic bias on rare but decision-critical events. We formalize sufficient conditions under which DRPE correctly ranks models and total prediction error cannot.

Linhao Wang, Yiyan Fan, Dongjin Huang

Shanghai Film Academy, Shanghai University

Prediction Error World Model Planning Planning World Models

Ask PDF

Jul 11, 2026cs.LG

A Control Theory of Predictability in Latent World Models

Latent world models are trained to predict future states in a learned representation and are then deployed inside a planner that selects actions by simulating them forward. Current practice adopts the prediction error, the single- or multi-step rollout loss on held-out data, as the training and model-selection objective, on the assumption that a lower prediction error yields better control. We show that this assumption is unreliable for a structural reason: a planner does not query the model on the training distribution but on the states that its candidate actions reach, which generally leave the data manifold, so an error averaged over the data cannot by itself govern control. We therefore reframe the objective as the discrepancy between the predicted and the true plan-cost at the plan the planner commits to, and prove that the planner's suboptimality is bounded by twice this discrepancy, whereas the data-averaged prediction error neither bounds nor tracks it. Under a linear-control premise the discrepancy separates into two terms. The first is a small on-manifold residual, on which the predicted and true dynamics agree and which a spectral tax prices through the non-normality of the latent transition operator. The second is an off-manifold divergence, on which an action carries the state off the manifold and the two dynamics diverge; this divergence is the binding term and is bounded by no data-averaged error. Synthetic operators confirm the pricing formulas, and latent model-predictive control experiments confirm the decoupling: across seeds, the single-step validation error is essentially uncorrelated with control success, whereas a fidelity score on the planner-reachable measure tracks it.

Hanzhe You, Yonggang Zhang, Maohao Ran +6

University of Science and Technology of China · The Hong Kong University of Science and Technology, HKGAI · Hong Kong Baptist University

Latent World Models Prediction Error Model Predictive Control Predictability Robust Control World+1

Ask PDF

May 8, 2026cs.LG

Predictive but Not Plannable: RC-aux for Latent World Models

A latent world model may achieve accurate short-horizon prediction while still inducing a latent space that is poorly aligned with planning. A key issue is spatiotemporal mismatch: these models are often trained with local predictive supervision, but deployed for long-horizon goal-directed search in latent spaces where Euclidean distance may not reflect what is reachable within a finite action budget. We present the Reachability-Correction auxiliary objective (RC-aux), a lightweight correction for this mismatch in reconstruction-free latent world models. RC-aux keeps the world-model backbone unchanged and adds planning-aligned supervision along two axes. Along the time axis, multi-horizon open-loop prediction trains the model beyond one-step consistency. Along the space axis, budget-conditioned reachability supervision, together with temporal hard negatives, encourages the latent space to distinguish states that are eventually reachable from those reachable within the current planning horizon. At test time, the learned reachability signal can also be used by a reachability-aware planner to favor trajectories that are both goal-directed and attainable under the available budget. We instantiate RC-aux on LeWorldModel and evaluate it under both continuation-training and matched-from-scratch settings. Across goal-conditioned pixel-control tasks and a LIBERO-Goal extension, RC-aux improves LeWM-style planning with modest additional cost. These results suggest that planning with latent world models depends not only on predictive accuracy, but also on whether the learned representation encodes the temporal and geometric structure required by downstream search. The code is available at https://github.com/Guang000/RC-aux.

Wenyuan Li, Guang Li, Keisuke Maeda +2

Hokkaido University

Latent World Models World Models Short-Horizon Forecasting Reachability

Ask PDF