RolloutFaith: Auditing Persistent Internal Interventions in Visual World Model

Junchi Yao; Ziyi Wang; Youling Huang; Lijie Hu

cs.LGSep 29, 2026

RolloutFaith: Auditing Persistent Internal Interventions in Visual World Model

Authors: Junchi Yao, Ziyi Wang, Youling Huang, Lijie Hu

Organizations: UESTC · Dalian University of Technology · MBZUAI

Abstract

Interpretability methods such as probes, activation patches and learned editors are designed to reveal or modify a model's current computation. World models pose a harder requirement: because their predictions become inputs to later predictions, a useful internal correction must survive after editing stops. We therefore propose RolloutFaith, a framework that measures semantic improvement both in the prediction produced at intervention time and over later autonomous predictions under fixed events, actions, noise, and information budgets. We evaluate ten fitted editors on three world models across Crafter, Cartpole, and CoinRun. We also use Reference Activation Patching, which replaces a model activation with the paired activation computed from the real observation, to measure the correction available at the chosen interface. This reference intervention improves later predictions in all nine model and task combinations and outperforms the best fitted editor in eight, yet its sustained gain decreases with horizon in five of nine combinations. Current fitted editors recover only limited and inconsistent long term effects. By restoring individual state components to their untouched values, we find that persistent effects travel through the newest generated frame in DIAMOND, recurrent memory in DreamerV3, and both in STORM. These findings suggest that training should reward future consequences. To test this hypothesis, we propose Delayed LoReFT, which optimizes the same low rank intervention through four frozen future transitions and improves sustained intervention effects to some extent.

Figures & tables

Figure 1: Current repair and later drift. In STORM CoinRun, Reference Activation Patching (RAP) raises terrain accuracy from 72.2% to 96.4%. Eight steps later, accuracy is 70.0% edited and 61.3% untouched. Left: accuracy from 60 to 100%; right: retained gain. Details in Appendix F .

Panel of Figure 2 — Figure 2: Task examples and test composition. Real observations show each task. Sources are episodes for Crafter and Cartpole and levels for CoinRun; events are intervention locations.

		Crafter		Cartpole (DMC)		CoinRun (Procgen)
WM	Tool	ISG	SSG@8	ISG	SSG@8	ISG	SSG@8
DIAMOND	Mean	+0.11	+0.31	+0.09	+0.06	+0.06	+0.14
	InvProbe	+0.89	+0.21	+0.16	-0.13	+0.01	+0.02
	NormProbe	-0.23	-1.06	+0.01	+0.02	+3.05	+2.25
	DiffMean	-0.43	+0.09	+0.00039	+0.0040	+0.34	+0.16
	PCA + ridge	-0.16	-0.03	-0.06	+0.24	+0.88	+0.20

Table 2: Intervention results under the shared evaluation protocol. Point estimates are gains over untouched predictions in percentage points (pp), with equal source weights; all 95% source bootstrap CIs are in Appendix Tables 5 – 7 . Bold marks the best fitted result per model, task, and metric; RAP is not ranked.

Figure 3: Direction, Magnitude, and Transfer Gaps of Fitted Editors (a) Magnitude importance on DIAMOND–Crafter. (b) Direction importance on DreamerV3–Crafter. (c) Validation-to-test transfer with matched controls. (d) Validation and test SSG@8 across all fitted combinations.

Figure 4: Intervention Propagation Pathways. For a uniform legend, DIAMOND’s “latent” and “memory” denote its newest generated frame and the context formed by the preceding three frames.

		Immediate			Delayed
WM	Task	ISG	SSG@8	Final gain	ISG	SSG@8	Final gain
DIAMOND	Crafter	-0.50	+0.18	+0.31	-0.27 $\uparrow$	+0.24 $\uparrow$	+1.21 $\uparrow$
	Cartpole	+0.02	+0.32	+0.01	+0.34 $\uparrow$	+3.24 $\uparrow$	+1.20 $\uparrow$
	CoinRun	+0.66	+0.39	+2.29	-8.67	-1.17	+1.13
DreamerV3	Crafter	-0.21	-0.12	-0.16	-0.06 $\uparrow$	-0.17	-0.23
	Cartpole	+2.57	+3.22	+4.56	+2.39	+3.46 $\uparrow$	+4.89 $\uparrow$

Table 3: Immediate versus Delayed LoReFT. In the 24-step study, final gain

G_{17:24}

is the mean paired improvement over untouched predictions from steps 17–24. Delayed results that improve on their Immediate counterparts are in light blue and marked with a green

\uparrow

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Task	WM	Training	Budget	Batch layout
Cartpole	DIAMOND	Offline	20,000 updates	$32\times 5$
Cartpole	DreamerV3	Online	110,004 decisions	$16\times 64$
Cartpole	STORM	Offline	20,000 updates	$16\times 64$
Crafter	DIAMOND	Offline	20,000 updates	$32\times 5$
Crafter	DreamerV3	Online	110,000 decisions	$16\times 64$
Crafter	STORM	Offline	20,000 updates	$16\times 64$

Appendix

Table 4: Recorded training budgets and batch layouts. Online training updates the WM and policy; offline training updates prediction modules on frozen Dreamer training replay. Batch layout gives sequences times frames per sequence.

Task	Tool	ISG		SSG@8		Endpoint
		Mean	95% CI	Mean	95% CI	Mean	95% CI
Crafter	Mean training delta	+0.11	[-0.10, 0.37]	+0.31	[0.04, 0.61]	+0.51	[0.10, 0.96]
	Inverse probe	+0.89	[-1.75, 4.31]	+0.21	[-1.54, 2.46]	-0.71	[-2.85, 1.49]
	Normalized probe	-0.23	[-1.66, 1.12]	-1.06	[-2.81, 0.43]	-2.37	[-5.50, -0.03]
	Difference of means	-0.43	[-1.11, 0.08]	+0.09	[-0.14, 0.34]	+0.42	[0.08, 0.80]
	PCA + ridge	-0.16	[-0.39, 0.02]	-0.03	[-0.20, 0.15]	-0.00075	[-0.17, 0.20]

Appendix

Table 5: Complete intervention results for DIAMOND, in pp. Means and 95% confidence intervals use 10,000 bootstrap draws over independent sources. Intervals are conditional on the selected editor and do not include training-seed variation. Endpoint is the paired semantic gain at

h=8

Task	Tool	ISG		SSG@8		Endpoint
		Mean	95% CI	Mean	95% CI	Mean	95% CI
Crafter	Mean training delta	+0.11	[-0.38, 0.62]	-0.13	[-0.41, 0.14]	+0.06	[-0.31, 0.47]
	Inverse probe	+0.28	[-0.25, 0.84]	+0.19	[-0.02, 0.38]	-0.07	[-0.56, 0.39]
	Normalized probe	-0.01	[-0.59, 0.54]	+0.32	[-0.12, 0.76]	+0.26	[-0.48, 0.95]
	Difference of means	+0.06	[-0.15, 0.29]	-0.04	[-0.18, 0.08]	+0.08	[-0.24, 0.38]
	PCA + ridge	+0.11	[-0.44, 0.68]	-0.15	[-0.48, 0.17]	-0.20	[-0.80, 0.36]

Appendix

Table 6: Complete intervention results for DreamerV3, in pp. Means and 95% confidence intervals use 10,000 bootstrap draws over independent sources. Intervals are conditional on the selected editor and do not include training-seed variation. Endpoint is the paired semantic gain at

h=8

Task	Tool	ISG		SSG@8		Endpoint
		Mean	95% CI	Mean	95% CI	Mean	95% CI
Crafter	Mean training delta	-0.87	[-1.65, -0.13]	-0.03	[-0.47, 0.44]	-0.28	[-1.13, 0.56]
	Inverse probe	+2.17	[0.63, 3.70]	+2.06	[1.07, 3.12]	+0.85	[-0.24, 1.92]
	Normalized probe	+0.62	[-0.87, 2.05]	+1.44	[0.53, 2.34]	+0.11	[-1.26, 1.39]
	Difference of means	+0.24	[-0.37, 0.81]	+0.09	[-0.21, 0.38]	-0.08	[-0.86, 0.72]
	PCA + ridge	-0.33	[-1.19, 0.59]	+0.60	[-0.01, 1.33]	+0.63	[-0.51, 1.89]

Appendix

Table 7: Complete intervention results for STORM, in pp. Means and 95% confidence intervals use 10,000 bootstrap draws over independent sources. Intervals are conditional on the selected editor and do not include training-seed variation. Endpoint is the paired semantic gain at

h=8

Task	Sources	Events	$h=0$	$h=1$	$h=2$	$h=3$	$h=4$	$h=5$	$h=6$	$h=7$	$h=8$
Crafter	38	41	41	40	41	41	41	40	41	41	41
Cartpole	48	96	96	96	96	96	96	96	96	96	96
CoinRun	47	92	92	92	92	92	92	92	92	92	92

Appendix

Table 8: Scoreable event coverage at every evaluated horizon. Entries from

h=0

through

h=8

are the number of scoreable events out of the total in the Events column. The mask is determined from real references and is shared across all models, interventions, and three prediction seeds.

Figure 5: The dominant interface pattern and its exception. (a) RAP exceeds the strongest fitted editor in eight of nine cohorts spanning models and tasks. (b) DreamerV3 on Cartpole is the exception: Immediate LoReFT exceeds full RAP, RAP matched by norm, and its random test edit matched by norm. Delayed LoReFT is excluded from both panels.

Prediction seed	RAP	LoReFT I	RAP minus LoReFT I
69000	+0.76	+3.39	-2.63
69001	+1.18	+3.70	-2.53
69002	+0.84	+2.56	-1.73
All seeds	+0.92	+3.22	-2.30

Appendix

Table 9: DreamerV3 on Cartpole exception across the three registered prediction seeds. Values are SSG@8 in pp with equal source weights. The paired 95% bootstrap interval over sources and all seeds for RAP minus Immediate LoReFT is [-2.68, -1.90]. Seeds are robustness strata over the same 48 sources, not independent datasets.

WM	Task	Paired SSG@8 contrast	Mean	Pointwise 95% CI	Simultaneous 99.44% CI
DIAMOND	Crafter	RAP–Mean	+10.46	[+6.34, +15.40]	[+4.98, +17.82]
DIAMOND	Crafter	Mean–Delayed	+0.06	[-0.31, +0.53]	[-0.45, +0.75]
DIAMOND	Cartpole	RAP–Delayed	+5.77	[+4.21, +7.33]	[+3.60, +7.94]
DIAMOND	Cartpole	Delayed–Immediate	+2.92	[+1.64, +4.19]	[+1.13, +4.78]
DIAMOND	CoinRun	RAP–NormProbe	+1.88	[+0.65, +3.17]	[+0.19, +3.74]
DIAMOND	CoinRun	NormProbe–Immediate	+1.86	[+0.70, +3.10]	[+0.23, +3.59]

Appendix

Table 10: The fitted candidate set includes all eleven methods (the ten baseline editors and Delayed LoReFT). Exploratory paired bootstrap intervals over sources for observed SSG@8 leaders. Seeds and events are averaged within source before 10,000 resamples. The final column uses

\alpha=0.05/9

for the nine cohort contrasts in each contrast family. Because leaders were selected by the same point estimates, these intervals describe ranking stability rather than preregistered hypothesis tests.

WM	Task	Tool	Norm	$\Delta$ random	SSG@8 for zero request
DIAMOND	Crafter	Mean training delta	40.58	+0.27	+0.31
DIAMOND	Crafter	Normalized probe	114.76	-0.83	-2.16
DIAMOND	Crafter	RAP	107.43	+10.89	N/A
DIAMOND	Cartpole	Inverse probe	14.06	-0.14	-0.20
DIAMOND	Cartpole	LoReFT delayed	37.47	+3.27	+3.24
DIAMOND	Cartpole	RAP	23.90	+9.06	N/A

Appendix

Table 11: Intervention diagnostics for the two highest and lowest SSG@8 tools from the original nine-intervention bank in each available model and task. Ties follow the fixed method order. Norm is the mean Euclidean edit norm with equal source weights, concatenating the three edits across denoiser calls for DIAMOND; units differ across interfaces.

\Delta

random is SSG@8 above the random control matched by norm. Both gain columns are in pp. All original primary tools produce nonzero updates on every evaluated pair of events and seeds. SSG@8 for requests with zero change is evaluated for the eight fitted tools with targets equal to the descriptor of the predicted image; N/A marks RAP. Complete diagnostics for every evaluated tool accompany the results bound to each source.

ISG (pp)
	Crafter				Cartpole				CoinRun
Tool	Val.	Test	Rand.	$\Delta$	Val.	Test	Rand.	$\Delta$	Val.	Test	Rand.	$\Delta$
Mean training delta	-0.23	+0.11	-0.03	+0.14	+0.88	+0.09	+0.06	+0.03	+0.06	+0.06	+0.00	+0.06
Inverse probe	+0.99	+0.89	-0.30	+1.19	+0.98	+0.16	+0.09	+0.08	-0.01	+0.01	+0.00	+0.01
Normalized probe	-1.13	-0.23	-0.15	-0.08	+0.00	+0.01	+0.00	+0.01	+3.80	+3.05	-0.00	+3.05
Difference of means	-0.84	-0.43	-0.00	-0.43	+0.00	+0.00	+0.00	+0.00	+0.18	+0.34	-0.00	+0.34

Appendix

Table 12: Validation, test and norm matched random controls for DIAMOND.

\Delta

is the test editor effect minus its random control.

ISG (pp)
	Crafter				Cartpole				CoinRun
Tool	Val.	Test	Rand.	$\Delta$	Val.	Test	Rand.	$\Delta$	Val.	Test	Rand.	$\Delta$
Mean training delta	+0.11	+0.11	-0.06	+0.17	+0.57	+0.13	-0.05	+0.18	+0.00	+0.00	-0.00	+0.00
Inverse probe	+0.59	+0.28	+0.04	+0.23	+0.00	-0.16	-0.09	-0.07	+0.18	+0.03	-0.03	+0.06
Normalized probe	+0.14	-0.01	+0.41	-0.41	-0.55	+0.19	-0.14	+0.33	+0.24	+0.18	+0.04	+0.14
Difference of means	-0.16	+0.06	-0.14	+0.20	-0.38	-0.35	-0.12	-0.24	-0.00	+0.00	+0.00	-0.00

Appendix

Table 13: Validation, test and norm matched random controls for DreamerV3.

\Delta

is the test editor effect minus its random control.

ISG (pp)
	Crafter				Cartpole				CoinRun
Tool	Val.	Test	Rand.	$\Delta$	Val.	Test	Rand.	$\Delta$	Val.	Test	Rand.	$\Delta$
Mean training delta	-0.49	-0.87	-0.45	-0.42	-0.20	+0.22	-0.48	+0.70	-0.00	-0.01	-0.00	-0.01
Inverse probe	+3.91	+2.17	-0.55	+2.72	+0.29	-0.28	-0.21	-0.07	+0.01	+0.03	-0.29	+0.32
Normalized probe	-1.24	+0.62	-0.09	+0.71	-0.09	-0.03	-0.13	+0.10	+0.04	+0.71	-0.17	+0.88
Difference of means	-0.40	+0.24	+0.09	+0.14	+0.30	-0.13	-0.11	-0.02	+0.07	+0.05	+0.01	+0.04

Appendix

Table 14: Validation, test and norm matched random controls for STORM.

\Delta

is the test editor effect minus its random control.

		Full	Immediate		Delayed
WM	Task	RAP	LoReFT	Matched RAP	LoReFT	Matched RAP
DIAMOND	Crafter	+10.76	+0.18	+0.48	+0.24	+0.48
DIAMOND	Cartpole	+9.01	+0.32	+1.08	+3.24	+1.97
DIAMOND	CoinRun	+4.14	+0.39	+2.01	-1.17	+2.01
DreamerV3	Crafter	+1.24	-0.12	+0.38	-0.17	+0.26
DreamerV3	Cartpole	+0.92	+3.22	+1.18	+3.46	+1.18

Appendix

Table 15: SSG@8 in pp for full RAP, learned editors and RAP controls matched by norm at every edit call. Each comparison shares source, event, actions and prediction noise. Full RAP retains its native magnitude.

WM	Task	Delayed LoReFT	Delayed at RAP norm	Full RAP	Delayed norm	RAP norm
DIAMOND	Crafter	+0.24	-3.39	+10.76	22.95	107.43
DIAMOND	Cartpole	+3.24	+3.11	+9.01	37.47	23.90
DIAMOND	CoinRun	-1.17	+0.86	+4.14	580.29	272.40
DreamerV3	Crafter	-0.17	-0.33	+1.24	5.94	73.64
DreamerV3	Cartpole	+3.46	+3.20	+0.92	21.88	18.74
DreamerV3	CoinRun	+0.14	+0.13	+1.20	38.68	31.10

Appendix

Table 16: Bidirectional magnitude control on the frozen confirmation cohorts. “Delayed at RAP norm” keeps the Delayed LoReFT direction but rescales it per event and edit call to the realized full RAP norm. SSG@8 is in pp. This complements RAP evaluated at the LoReFT norm.

WM	Task	H=8 fitted leader	Leader by $H=1,2,4,8$	Fitted leader gains	RAP gains	Stable?
DIAMOND	Crafter	Mean	InvProbe/InvProbe/InvProbe/Mean	+0.60/+0.69/+0.63/+0.31	+12.81/+12.69/+12.20/+10.76	no
DIAMOND	Cartpole	Delayed	Delayed/Delayed/Delayed/Delayed	+0.91/+1.58/+2.18/+3.24	+6.00/+6.13/+6.93/+9.01	yes
DIAMOND	CoinRun	NormProbe	NormProbe/NormProbe/NormProbe/NormProbe	+4.15/+3.97/+3.36/+2.25	+7.13/+6.71/+5.77/+4.14	yes
DreamerV3	Crafter	NormProbe	InvProbe/InvProbe/NormProbe/NormProbe	+0.08/+0.16/+0.27/+0.32	+2.11/+1.92/+1.54/+1.24	no
DreamerV3	Cartpole	Delayed	Delayed/Delayed/Delayed/Delayed	+2.53/+2.69/+2.97/+3.46	+0.56/+0.69/+0.87/+0.92	yes
DreamerV3	CoinRun	Carrier	PCA+ridge/PCA+ridge/PCA+ridge/Carrier	+0.22/+0.23/+0.31/+0.38	+1.08/+1.09/+1.18/+1.20	no

Appendix

Table 17: The fitted candidate set includes all eleven methods (the ten baseline editors and Delayed LoReFT). Horizon sensitivity on the frozen confirmation reports. Each entry uses the scoreability threshold

\lceil 0.75H\rceil

and aggregation with equal source weights. Gain sequences follow

H=1,2,4,8

. Immediate and Delayed denote the two LoReFT objectives. RAP is shown separately and is not ranked with fitted editors.

WM	Task	Split	$n_{L}/n_{H}$ sources	Fitted leader $L/H$	RAP $L/H$	Same fitted leader?
DIAMOND	Crafter	lower / higher action variation	22/18	InvProbe +0.54/Mean +0.54	+10.96/+10.31	no
DIAMOND	Cartpole	lower / higher action variation	39/39	Delayed +2.69/Delayed +3.85	+10.36/+8.48	yes
DIAMOND	CoinRun	constant / switched	19/44	NormProbe +0.78/NormProbe +2.68	+2.55/+4.41	yes
DreamerV3	Crafter	lower / higher action variation	22/18	NormProbe +0.43/InvProbe +0.18	+0.63/+1.99	no
DreamerV3	Cartpole	lower / higher action variation	39/39	Delayed +3.36/Delayed +3.73	+0.82/+1.09	yes
DreamerV3	CoinRun	constant / switched	19/44	NormProbe +0.13/PCA+ridge +0.49	+0.95/+1.26	no

Appendix

Table 18: The fitted candidate set includes all eleven methods (the ten baseline editors and Delayed LoReFT). Sensitivity to action sequences on the same frozen events. Cartpole and Crafter are split at the task median of mean action variation after editing; CoinRun is split by whether its schedule of held actions over eight steps switches in the measured window.

L/H

denotes lower or higher variation, and constant or switched actions for CoinRun. Immediate and Delayed denote the two LoReFT objectives.

WM	Task	Fitted winner by seed	Fitted SSG@8 by seed	RAP SSG@8 by seed
DIAMOND	Crafter	Mean/Mean/Delayed	+0.32/+0.32/+0.30	+10.90/+10.54/+10.84
DIAMOND	Cartpole	Delayed/Delayed/Delayed	+3.16/+3.30/+3.27	+9.07/+9.01/+8.94
DIAMOND	CoinRun	NormProbe/NormProbe/NormProbe	+2.08/+2.48/+2.20	+4.04/+4.31/+4.07
DreamerV3	Crafter	InvProbe/PCA+ridge/NormProbe	+0.45/+0.03/+0.71	+1.07/+1.43/+1.21
DreamerV3	Cartpole	Delayed/Delayed/Delayed	+3.93/+3.78/+2.67	+0.76/+1.18/+0.84
DreamerV3	CoinRun	Carrier/PCA+ridge/Immediate	+0.39/+0.51/+0.27	+1.37/+1.21/+1.02

Appendix

Table 19: The fitted candidate set includes all eleven methods (the ten baseline editors and Delayed LoReFT). Sensitivity to prediction noise. Columns list the three preregistered seeds (69000/69001/69002); each seed is aggregated independently over events and sources. Immediate and Delayed denote the two LoReFT objectives. The seeds are robustness strata, not additional independent datasets.

Figure 6: Representative effects over 24 prediction steps. Source weighted means and 95% bootstrap intervals illustrate three recurring regimes: a large RAP benefit that decays, a fitted effect that emerges after the edited prediction, and harm that persists during autonomous prediction.

WM	Task	Edit	Original	Restore latent	Restore memory	$L_{z}/L_{m}$
DIAMOND	Crafter	RAP	+5.43	+0.01	+5.40	+5.42/+0.02
DIAMOND	Crafter	LoReFT D	+1.31	-0.02	+1.26	+1.33/+0.05
DIAMOND	Crafter	LoReFT I	+0.43	+0.03	+0.43	+0.39/-0.01
DIAMOND	Cartpole	RAP	+12.88	-1.20	+11.97	+14.09/+0.91
DIAMOND	Cartpole	LoReFT D	+3.32	-1.61	+4.08	+4.93/-0.76
DIAMOND	Cartpole	LoReFT I	+0.47	-0.08	+0.49	+0.55/-0.02

Appendix

Table 20: Component restoration results across all models. Original and restored gains average steps 9 to 24; losses are original minus restored gain. Positive loss means restoration removes benefit, while negative loss means restoration improves the edited rollout. Values are means in pp with equal source weights.

Figure 7: DIAMOND component pathways over 24 prediction steps. Restoration loss is shown for the newest generated frame and the three earlier context frames under RAP, Immediate LoReFT and Delayed LoReFT. Shading gives 95% source bootstrap intervals.

Figure 8: DreamerV3 component pathways over 24 prediction steps. Restoration loss is shown for the sampled latent and deterministic recurrent memory under RAP, Immediate LoReFT and Delayed LoReFT. Shading gives 95% source bootstrap intervals.

Figure 9: STORM component pathways over 24 prediction steps. Restoration loss is shown for the sampled latent and transformer cache under RAP, Immediate LoReFT and Delayed LoReFT. Shading gives 95% source bootstrap intervals.

Cohort	Edit	Sources	Original	Latent loss	Memory loss
DreamerV3 Cartpole	RAP	48	$+1.22$ $[+0.74,+1.70]$	$+0.14$ $[+0.02,+0.28]$	$+1.10$ $[+0.66,+1.54]$
DreamerV3 Cartpole	Delayed LoReFT	48	$+4.95$ $[+4.10,+5.86]$	$+0.18$ $[+0.02,+0.33]$	$+4.68$ $[+3.80,+5.58]$
STORM Cartpole	RAP	48	$+5.54$ $[+3.77,+7.34]$	$+3.10$ $[+1.70,+4.56]$	$+2.80$ $[+1.42,+4.26]$
STORM Cartpole	Delayed LoReFT	48	$+0.93$ $[-1.65,+3.45]$	$-0.28$ $[-2.03,+1.59]$	$+0.53$ $[-1.25,+2.15]$
STORM Crafter	RAP	37	$+1.88$ $[+0.44,+3.50]$	$+1.01$ $[-0.17,+2.33]$	$+0.49$ $[-0.31,+1.26]$
STORM Crafter	Delayed LoReFT	37	$-1.21$ $[-2.78,+0.29]$	$-1.32$ $[-2.49,-0.34]$	$-0.93$ $[-2.02,+0.07]$

Appendix

Table 21: Later gain and restoration loss over steps 9 to 24, in pp. Loss is edited gain minus gain after restoring the named component at step 1. Brackets show 95% source bootstrap intervals.

WM	Task	Edit	Restored	Random loss	Semantic loss	Semantic minus random loss (95% CI)
DreamerV3	Cartpole	RAP	Latent	+0.09	+0.14	+0.05 [-0.11, +0.22]
DreamerV3	Cartpole	RAP	Memory	+0.15	+1.10	+0.96 [+0.41, +1.49]
DreamerV3	Cartpole	Delayed LoReFT	Latent	-0.04	+0.18	+0.22 [+0.04, +0.39]
DreamerV3	Cartpole	Delayed LoReFT	Memory	+0.05	+4.68	+4.62 [+3.78, +5.46]
STORM	Cartpole	RAP	Latent	-0.45	+3.10	+3.55 [+2.00, +5.14]
STORM	Cartpole	RAP	Memory	-0.04	+2.80	+2.84 [+1.41, +4.38]

Appendix

Table 22: Matched random controls for hybrid state restoration, in pp. The final column subtracts the equal norm random restoration loss from the semantic restoration loss; brackets show paired 95% source bootstrap intervals.

Explore similar work

The Intervention Gap in Latent World Models
Aug 30, 2026Donna VakalisWorld Model-Based Planning Model-Based Planning
Mila – Quebec AI Institute · University of Montreal
Compact but Moving: Intervention-Relevant Geometry in Recurrent World Models
Sep 18, 2026Yuming Chen, Yang LiuWorld Models World Model Learning
College of Intelligent Robotics and Advanced Manufacturing, Fudan University Shanghai, China
Low-Rank Dynamics-Effective Latent Carriers for Counterfactual Rollout in Learned World Models
Aug 15, 2026Yang Liu, Yuming ChenWorld Model Learning Latent World Models
College of Intelligent Robotics and Advanced Manufacturing, Fudan University Shanghai, China

Aug 30, 2026J/K move · Enter open · S save

The Intervention Gap in Latent World Models

Donna Vakalis

Mila – Quebec AI Institute · University of Montreal

World Model-Based Planning Model-Based Planning Latent World Models World Model Evaluation

Open paper AskPDF arXiv

Planning-time intervention fidelity is a distinct, measurable property of a learned world model: whether the model's own open-loop transitions move task variables the way matched environment interventions do. In the settings we test, it is neither revealed by reward fit nor ensured by task-anchored training. Across released TD-MPC2 checkpoint sizes, episode return falls as an operator-error diagnostic on task observables grows, while reward-prediction error stays small and nearly flat, and a self-supervised world model trained without task signal preserves the same operator substantially better than a task-anchored model on the shared task. A capture-gated matched-intervention audit then localizes what fails. On Cheetah, three LeWorldModel checkpoints capture the current task query and support decodable real intervention effects; however, their imagined five-step effects are worse than predicting no effect and worse than an environment-endpoint oracle. The failure is task-direction rotation with excess gain, not feature collapse. This severe pattern is conditional: five PreJEPA seeds retain an oracle-relative deficit without it, Finger Spin experiments extend the deficit beyond locomotion with heterogeneous severity across seeds, and shared-bank effect geometry is both candidate- and support-dependent. We also test practice-side questions. In DreamerV3 the posterior distribution, not its sample, carries the current query; ensemble disagreement ranks error only near training support; and a frozen support-aware score degrades held-out error ranking in both tested transfer directions while native disagreement remains informative in both. We conclude that intervention fidelity must be audited directly, capture-first, on the model's native interface.

Abstract

Explore similar work

The Intervention Gap in Latent World Models

Compact but Moving: Intervention-Relevant Geometry in Recurrent World Models

Low-Rank Dynamics-Effective Latent Carriers for Counterfactual Rollout in Learned World Models