MultiEcho: An Experimental Science of Learned Worlds

Meng Zhu; Airui Zhang

cs.LGSep 27, 2026

MultiEcho: An Experimental Science of Learned Worlds

Authors: Meng Zhu, Airui Zhang

Organizations: SHIHAO XINRUI Group

Abstract

World models can be studied as experimental systems with response laws of their own. We introduce MultiEcho, a framework for estimating these laws through controlled counterfactual interventions, delimiting their applicability, and separately testing their physical correspondence. Across nine simulated physical systems and seven frozen model configurations, three-reference estimators predict complete intervention responses and recover intervention parameters. Estimator selection uses discovery data only; frozen fits are evaluated on validation and confirmation contexts. The experiments distinguish response predictability, intervention readability and physical accuracy. Responses can be locally describable yet poorly match physical effects in the same target coordinates. Event-window, visibility and camera interventions reveal conditional applicability, and paired generator configurations show reduced readability under a scene prompt with stronger guidance. Magnitude sweeps expose small image errors alongside large relative effect errors. An exact-reset material experiment separates registered visible-response success from fixed-readout failure on material-dependent futures at matched positions and velocities. Exact finite-scale identities resolve odd and even response errors; first-order remainder bounds specify when refined calibration converges. MultiEcho provides an experimental basis for studying learned-world laws independently of, and in relation to, physical laws.

Figures & tables

Figure 1: Constructing a counterfactual experiment. (A) Final input frames at diffusivities $u_{-}$ , $u_{0}$ and $u_{+}$ , one set of coarse heat references from Section 5 . (B) Each branch input $\kappa_{h}(c,u)$ passes through the same adapter $A_{W}$ , frozen model $M_{W}$ and observation map $O_{W,o}$ (Equation 1 ). (C) The paired effect $\Delta Y$ enters the forward fit, reverse readout and response statistics.

	Forward $R^{2}$		Reverse $R^{2}$		Within $\rho$		Direction
Observation	Median	Below FF	Median	Below FF	Median	Below FF	Median	Below FF
VideoSAUR enc.	0.344	91.7	0.208	100.0	0.392	100.0	0.315	75.0
C-JEPA pred.	0.363	83.3	0.455	95.8	0.524	91.7	0.427	41.7
LeWM enc.	0.619	33.3	0.659	100.0	0.667	70.8	0.508	33.3
LeWM pred.	0.742	16.7	0.760	87.5	0.782	58.3	0.475	25.0
AC enc.	0.359	95.8	0.898	83.3	0.848	45.8	0.401	41.7

Table 1: Response statistics of every observation. Entries are medians over the same 24 physical axes. Forward and reverse

R^{2}

come from the common three-reference neural procedures under six context-held-out folds. Within

\rho

is the median within-context distance association, and Direction is direction consistency. Below FF gives the percentage of axes on which the observation scores lower than simulator first-future RGB, compared before rounding. Negative scores are included.

	Validation				Confirmation
Observation	Forward	Below FF	Reverse	Below FF	Forward	Below FF	Reverse	Below FF
VideoSAUR enc.	0.351	79.2	0.056	100.0	0.332	79.2	0.050	100.0
C-JEPA pred.	0.398	66.7	0.382	91.7	0.412	75.0	0.401	87.5
LeWM enc.	0.541	37.5	0.586	100.0	0.543	33.3	0.703	91.7
LeWM pred.	0.692	20.8	0.822	91.7	0.622	16.7	0.829	87.5
AC enc.	0.363	91.7	0.904	79.2	0.359	95.8	0.877	75.0

Table 2: Forward and reverse

R^{2}

of the frozen discovery fits in eight validation and eight confirmation contexts per scene. Medians and Below FF follow Table 1 , with first-future RGB from the same stage. Table 7 adds intervals.

Figure 2: Model predictions compared with true physical futures across intervention magnitudes. Top: true effect size

\|\Delta T_{W}\|_{\rm RMS}

(dashed, no model involved), Cosmos’s own effect size

\|\Delta Z_{W}\|_{\rm RMS}

(dotted) and the error of its effect,

\|\Delta Z_{W}-\Delta T_{W}\|_{\rm RMS}

(solid). Per query, error over true effect size is

e_{W}

(bottom), in RGB or native target coordinates. Each point summarizes 144 queries. Bands are conditional 95% context-block intervals.

Figure 3: Self-description and physical-effect errors in identical latent coordinates. Solid lines give the held-out error of the neural law fitted to each model’s own responses at references

0,\pm\delta_{0}/2

, relative to its own response norm. Open markers give PL, and dashed lines the physical effect error

e_{W}

. Each point summarizes 144 queries. Bands are conditional 95% context-block intervals.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Scene	Intervention axes	Rows
Two-mass spring	Stiffness, two masses, initial amplitude (all log)	44
Collision	Two masses and speed (log), restitution	44
Bounce	Gravity and initial height (log), restitution	33
Refraction	Refractive index, slab thickness (log), incidence	33
Beam splitting	Refractive index, plate angle, incidence	33
Mechanics + optics	Gravity (log), initial height	22

Appendix

Table 3: Physical interventions in the atlas. Each axis is recorded through eleven distinct observations. Coordinates marked “log” use logarithmic physical parameters. The remaining coordinates use the recorded units.

Study	Observation set	Contexts	Units	Queries
Discovery	Nine learned interfaces	24/scene	216	41,328
Discovery	Two simulator RGB frames	24/scene	48	9,184
Validation	Eleven interfaces	8/scene	264	16,841
Confirmation	Eleven interfaces	8/scene	264	16,830
Fine scale	Seven learned interfaces	24/scene	21	15,120
Fine scale	Two simulator RGB frames	24/scene	6	4,320

Appendix

Table 4: Observation and query counts used by the revised neural analysis. Query counts exclude the two endpoint references and reference-parameter duplicates. The same physical query identities are used at every interface. External-stage counts do not multiply queries by the six frozen fits.

Interface	Training domain	Saved observation	Input and inference policy
VideoSAUR	CLEVRER object videos ( Yi et al., 2020 ) , pretrained DINOv2 ( Oquab et al., 2024 )	Last-time seven 128-D slots	Fixed slot initialization seed 0.
C-JEPA predictor	CLEVRER slot sequences	First of eight predicted slot frames	15 slot frames. Adapted 15-to-1 partition differs from original 6-to-10 training partition.
LeWM encoder	Released Reacher checkpoint ( Tassa et al., 2018 )	Final-frame projected 192-D CLS state	Same visual stimulus as its predictor.
LeWM predictor	Released Reacher checkpoint	First of eight predicted latents, before prediction projection	Three-latent history, constant zero actions.
AC encoder	Natural-video pretraining, robot transfer	Last tubelet, 256 tokens of 1,408 channels	Eight two-frame tubelets, zero action/state inputs.
AC predictor	Robot interactions, 62-hour post-training	First of four predicted tubelets, frames 16–17	Same fixed conditions. Later rollout exceeds two-step automatic setting.

Appendix

Table 5: Model access and recorded observations. The inference column specifies the history and conditioning used to produce each measurement.

Scene	Observation	Axes	Forward $R^{2}$	Reverse $R^{2}$	RGB reverse $R^{2}$
Bounce	VideoSAUR enc.	3	0.310	-0.128	-1.035
Collision	VideoSAUR enc.	4	0.396	0.413	-0.046
Refraction	VideoSAUR enc.	3	0.387	0.234	-0.068
Mechanics + optics	VideoSAUR enc.	2	0.304	0.056	-1.129
Heat	VideoSAUR enc.	2	0.219	0.048	-0.606
Fluid	VideoSAUR enc.	1	0.359	0.198	-1.085

Appendix

Table 6: Scene-resolved neural response laws. Each entry is a median over that scene’s physical axes. The last column applies the fixed future-RGB reverse to the subject’s three-reference query geometry.

Observation	Forward $R^{2}$	Own reverse $R^{2}$	RGB reverse $R^{2}$
Discovery CV
RGB first future	0.469 [0.432, 0.678]	0.972 [0.953, 0.993]	0.972 [0.953, 0.993]
RGB last input	0.433 [0.370, 0.495]	0.877 [0.458, 0.991]	0.873 [0.002, 0.991]
VideoSAUR enc.	0.344 [0.286, 0.385]	0.208 [-0.015, 0.338]	-0.334 [-0.776, -0.130]
C-JEPA pred.	0.363 [0.319, 0.409]	0.455 [0.272, 0.681]	-0.061 [-0.273, 0.190]
LeWM enc.	0.619 [0.420, 0.804]	0.659 [0.229, 0.845]	0.158 [-0.508, 0.767]

Appendix

Table 7: Response laws across stages. Entries are medians across all 24 axes with marginal 95% scene/context-block percentile intervals from 1,999 draws. Discovery uses six-fold context cross-validation and 24 contexts per scene. Each external stage has eight contexts per scene and three labeled references per context. Its six unchanged discovery fits contribute averaged squared losses. Intervals condition on saved fits and the examined cohorts, without refitting, selection correction or multiplicity adjustment. Reused physical settings are not independent dynamical replicates.

	Forward $R^{2}$			Reverse $R^{2}$
Observation	Canonical	Viewpoint	Lower	Canonical	Viewpoint	Lower
VideoSAUR enc.	0.344	0.365	41.7	0.208	0.176	58.3
C-JEPA pred.	0.363	0.374	37.5	0.455	0.449	50.0
LeWM enc.	0.619	0.560	41.7	0.659	0.456	58.3
LeWM pred.	0.742	0.687	45.8	0.760	0.852	58.3
AC enc.	0.359	0.355	33.3	0.898	0.893	62.5

Appendix

Table 8: Frozen discovery fits under the recorded camera changes. Each discovery context is rendered again with unchanged physics and scored by the fold that held it out, using the moved view’s own three references. Entries are medians over the same 24 axes. Lower gives the percentage of axes whose viewpoint score falls below the canonical score, compared before rounding.

Condition	Null $-$ text	$>0$	$\uparrow$	$\downarrow$	Null $-$ text	$>0$	$\uparrow$	$\downarrow$
	Forward $R^{2}$				Reverse $R^{2}$
Discovery CV	0.062 [-0.001, 0.146]	16	15	0	0.219 [0.048, 0.483]	20	14	0
Validation	0.056 [-0.018, 0.142]	15	13	2	0.396 [0.071, 0.584]	22	15	1
Confirmation	0.064 [-0.007, 0.205]	17	11	0	0.232 [0.018, 0.477]	21	15	1
Moved camera	0.026 [-0.016, 0.101]	15	15	0	0.294 [0.111, 0.395]	23	12	0

Appendix

Table 9: Paired Cosmos null minus text differences in neural

R^{2}

. Each estimate is the median over 24 axes of axis-wise differences, with a 95% scene/context-block percentile interval from 1,999 draws shared by both configurations.

>0

counts axes whose difference is positive, and

\uparrow

\downarrow

count axes whose own paired context-block interval lies wholly above/below zero. The configurations differ in one branch-invariant scene prompt and guidance, and each follows one fixed generation seed. Intervals condition on saved fits and the examined cohorts.

Observation	F: PL	F: N	R: PL	R: N	F gains	R gains
VideoSAUR enc.	0.311	0.344	-0.280	0.208	20/24	24/24
C-JEPA pred.	0.348	0.363	0.148	0.455	19/24	24/24
LeWM enc.	0.564	0.619	0.451	0.659	20/24	22/24
LeWM pred.	0.705	0.742	0.716	0.760	22/24	22/24
AC enc.	0.317	0.359	0.876	0.898	24/24	22/24
AC pred.	0.311	0.350	0.859	0.885	24/24	23/24

Appendix

Table 10: Equal-three-reference comparisons. PL denotes piecewise-linear response prediction or its projected inverse. N denotes the neural residual tool. Scores are axis medians. Gains count axes with a higher neural score.

Observation	$S_{F}$	$S_{R}$	$S_{RGB}$	Own vs. RGB
VideoSAUR enc.	0.056 [0.024, 0.074]	0.366 [0.297, 0.422]	-0.075 [-0.134, 0.016]	0.377 [0.342, 0.459]
C-JEPA pred.	0.032 [0.002, 0.050]	0.361 [0.245, 0.474]	-0.035 [-0.141, 0.009]	0.469 [0.358, 0.506]
LeWM enc.	0.059 [0.024, 0.265]	0.369 [0.174, 0.522]	-0.019 [-0.187, 0.142]	0.473 [0.353, 0.602]
LeWM pred.	0.059 [0.023, 0.301]	0.290 [0.166, 0.573]	-0.082 [-0.524, 0.151]	0.477 [0.343, 0.606]
AC enc.	0.058 [0.047, 0.135]	0.345 [0.223, 0.433]	-0.056 [-0.551, 0.066]	0.420 [0.290, 0.561]
AC pred.	0.054 [0.040, 0.114]	0.312 [0.138, 0.423]	0.008 [-0.510, 0.121]	0.363 [0.253, 0.487]

Appendix

Table 11: Paired gains conditional on the recorded fits. First three columns are skill relative to the matching PL control. The last is own reverse skill relative to RGB reverse. Each estimate is a median of axis-wise paired loss ratios, with 1,999 hierarchical scene/context-block draws and 95% percentile intervals. These intervals do not account for tool selection on discovery data or repeated physical trajectories. Axis-wise context-block intervals accompany the analysis records.

Interface	Axis	$\rho_{\rm m}/\widetilde{\rho}_{c}$	$C$	CV	$S_{\rm p}$	$B$	$\theta$	$\|a\|$
Spring
VideoSAUR enc.	k	0.474/-0.047	0.057	0.24	0.369	1.32	117.7	0.164
	$m_{A}$	0.871/0.507	0.095	0.21	0.241	1.14	111.4	0.169
	$m_{B}$	0.896/0.503	0.090	0.21	0.259	1.16	111.7	0.182
	amp.	0.701/0.333	0.062	0.33	0.404	1.40	106.8	0.226
C-JEPA pred.	k	0.405/0.016	0.068	0.37	0.504	1.29	117.7	0.199

Appendix

Table 12: Context-resolved response geometry for every atlas axis and interface.

\rho_{\rm m}/\widetilde{\rho}_{c}

compares mean-response and median within-context distance association.

C

is mean squared-distance retention. CV is median within-context distance standard deviation divided by mean, and

S_{\rm p}

is median normalized distance discrepancy between contexts.

B

is the median-pair amplitude factor,

\theta

the median local turning angle in degrees, and

|a|

the median absolute neighboring-secant amplitude asymmetry. Medians use defined observations. Brackets give their count when incomplete (24 contexts for

\rho

and CV, 192 positions otherwise). All ten levels are used.

Observation	Spring	Heat	Bounce
VideoSAUR enc.	0.982	1.279	0.975
C-JEPA pred.	1.020	1.832	0.989
LeWM pred.	1.009	0.562	1.014
AC pred.	0.974	1.042	0.967
Cosmos tokenizer	1.021	0.967	1.003
Cosmos null	1.007	0.970	0.991

Appendix

Table 13: Full-response median error ratios at

|s|=\log(1.5)/32

. The neural predictor is frozen from atlas fold zero and receives only three coarse references in each new context. Each cell includes 144 signed queries. A ratio of one equals the unchanged-output baseline.

Observation	Neural	PL	$q_{25}$	$q_{75}$	Reverse $R^{2}_{0}$
VideoSAUR enc.	1.279	0.972	0.985	2.752	-11.426
C-JEPA pred.	1.832	0.912	1.183	3.620	-7.608
LeWM pred.	0.562	0.551	0.484	0.616	0.990
AC pred.	1.042	0.986	1.017	1.076	-87.156
Cosmos tokenizer	0.967	0.975	0.960	0.972	-28.363
Cosmos null	0.970	0.971	0.962	0.980	0.531

Appendix

Table 14: Finest-scale heat prediction. Neural and PL columns use complete outputs and the same three references. Quartiles describe neural pointwise error ratios. Reverse uses the same frozen atlas tool and is scored against zero signed intervention. This denominator differs from the atlas training-mean score.

Scene	Subject	$\|s\|/\delta_{0}$	Future RMS	Effect error	True effect	Ratio
Spring	Cosmos null	1/2	0.0480	0.0314	0.0354	0.916
Spring	Cosmos null	1/4	0.0497	0.0288	0.0312	1.063
Spring	Cosmos null	1/8	0.0520	0.0257	0.0237	1.111
Spring	Cosmos null	1/16	0.0501	0.0181	0.0165	1.124
Spring	Cosmos null	1/32	0.0502	0.0117	0.0105	1.118
Spring	Cosmos text	1/2	0.0333	0.0393	0.0354	1.195

Appendix

Table 15: Generated futures compared directly with true first-future RGB, with no fitted response estimator. All errors use every pixel in native

[0,1]

display units. Future RMS compares complete images. Effect error compares branch-minus-reference responses. True effect is the simulator response RMS. Ratio is the median of pointwise effect-error/true-effect ratios. Each row contains 144 queries.

Scene	Subject	$\|s\|/\delta_{0}$	Future RMS	Persist. RMS	Future/persist.	Effect ratio
Spring	C-JEPA	1/2	0.4122	0.0681	5.81	1.035
Spring	C-JEPA	1/4	0.4083	0.0664	6.09	1.044
Spring	C-JEPA	1/8	0.4119	0.0673	6.18	1.028
Spring	C-JEPA	1/16	0.4052	0.0641	6.27	1.025
Spring	C-JEPA	1/32	0.4139	0.0617	6.39	1.032
Spring	LeWM	1/2	0.8279	0.0175	46.05	0.999

Appendix

Table 16: Predicted latents compared with encoded true first-future frames. Each row retains 144 queries. C-JEPA predicts from the original history in a local frozen replay, and its target continues the same float32 slot state. LeWM applies its released prediction projector before comparison with the target encoder. Future and persistence errors are in each model’s own latent units. Ratios are medians of pointwise ratios. Values above one lose to the respective baseline.

Scene	Subject	Scale	$A$	$c$	$e$ [95% interval]	$e_{\rm persist}$
Spring	C-JEPA	1/2	0.352	0.038	1.035 [1.022, 1.083]	0.560
Spring	C-JEPA	1/4	0.303	-0.008	1.044 [1.028, 1.072]	0.779
Spring	C-JEPA	1/8	0.259	0.023	1.028 [1.012, 1.048]	0.712
Spring	C-JEPA	1/16	0.251	0.016	1.025 [1.011, 1.048]	0.629
Spring	C-JEPA	1/32	0.265	0.018	1.032 [1.018, 1.083]	0.775
Spring	LeWM	1/2	0.277	0.175	0.999 [0.988, 1.008]	0.874

Appendix

Table 17: Complete-effect diagnostics at every scale.

A=\|\Delta Z\|/\|\Delta T\|

c

is their cosine,

e

is the physical-effect error ratio, and

e_{\rm persist}

reuses the last-input intervention effect in the same coordinates. Brackets give 95% paired context-block intervals. All summaries are pointwise medians with 144 queries.

Scene	Subject	Scale	Even share	True even	Future	Debiased	Persist.
Spring	C-JEPA	1/2	0.653	0.658	0.4122	0.2334	0.0681
Spring	C-JEPA	1/4	0.620	0.599	0.4083	0.2061	0.0664
Spring	C-JEPA	1/8	0.640	0.655	0.4119	0.1726	0.0673
Spring	C-JEPA	1/16	0.626	0.616	0.4052	0.1877	0.0641
Spring	C-JEPA	1/32	0.616	0.606	0.4139	0.1535	0.0617
Spring	LeWM	1/2	0.579	0.596	0.8279	0.0234	0.0175

Appendix

Table 18: Parity and constant-offset diagnostics. Even share is the median fraction of paired squared error from the even mismatch. Odd and even energies sum exactly before aggregation. True even is the median fraction of paired true-effect energy in its even part. Debiased future subtracts the reference branch’s prediction-minus-target vector, using no query fit, and therefore equals absolute effect error. The last three columns are observation-specific RMS errors. Temporal persistence is a distinct baseline from persistence of intervention effects.

Scene	Subject	Rerender RMS	Effect error	Round-only	Interval lower
Spring	Cosmos null	0.00000	0.01166	0.00000	0.01110
Spring	Cosmos text	0.00000	0.01799	0.00000	0.01718
Heat	Cosmos null	0.00329	0.00146	0.00033	0.00017
Heat	Cosmos text	0.00532	0.00359	0.00033	0.00175
Bounce	Cosmos null	0.00208	0.01122	0.00000	0.01023
Bounce	Cosmos text	0.00278	0.00558	0.00000	0.00477

Appendix

Table 19: Finest-scale RGB absolute errors and precision diagnostics. Rerender is generated-reference variation across independent input-render streams. Round-only is the error induced by rounding the true images to 8-bit. Interval lower is the remaining error after allowing one code step per contrast coordinate. All 24 same-input/same-seed repeats are identical. No future-truth rerender or encoded-target repeat is available to identify a matched noise floor.

Scene	Subject	Scale	Own: PL	Own: neural	Physical effect
Spring	C-JEPA	1/4	0.834 [0.787, 0.861]	0.827 [0.781, 0.874]	1.044 [1.028, 1.072]
Spring	C-JEPA	1/8	0.916 [0.888, 0.940]	0.922 [0.887, 0.950]	1.028 [1.012, 1.048]
Spring	C-JEPA	1/16	0.962 [0.946, 0.982]	0.958 [0.934, 0.977]	1.025 [1.011, 1.048]
Spring	C-JEPA	1/32	0.975 [0.967, 0.990]	0.969 [0.958, 0.991]	1.032 [1.018, 1.083]
Spring	LeWM	1/4	0.769 [0.702, 0.818]	0.731 [0.687, 0.770]	1.056 [1.041, 1.067]
Spring	LeWM	1/8	0.969 [0.902, 1.016]	0.898 [0.861, 0.966]	1.045 [1.036, 1.054]

Appendix

Table 20: Self-description and physical correspondence in identical target coordinates: LeWM after its prediction projection, C-JEPA paired replay. References are

0,\pm\log(1.5)/2

, the largest saved target-space offsets. Four finer magnitudes are held out by context. PL and the common neural procedure use the same references. All three ratios retain full coordinates but normalize by their own task’s response magnitude. Brackets are context-block 95% intervals conditional on the fits.

Observation	0	0.075	0.15	0.225	0.3	Native
VideoSAUR enc.	-0.002 † (0.791)	-0.030 (0.990)	0.004 (0.025)	-0.002 † (0.826)	-0.349 (1.000)	-0.538 (1.000)
C-JEPA pred.	-0.240 (0.995)	-0.287 (0.995)	0.529 (0.005)	0.590 (0.005)	0.565 (0.005)	0.616 (0.005)
LeWM enc.	-0.000 † (0.473)	0.000 † (0.015)	0.000 † (0.005)	0.000 † (0.005)	0.000 † (0.005)	0.001 † (0.020)
LeWM pred.	-0.002 † (1.000)	0.002 † (0.010)	0.015 (0.005)	0.036 (0.005)	0.076 (0.005)	0.217 (0.005)
AC enc.	0.230 (0.005)	0.707 (0.005)	0.660 (0.005)	0.674 (0.005)	0.648 (0.005)	0.615 (0.005)
AC pred.	0.193 (0.005)	0.662 (0.005)	0.610 (0.005)	0.629 (0.005)	0.614 (0.005)	0.585 (0.005)

Appendix

Table 21: Bounce visibility dose for post-contact restitution. Entries are held-out ridge

R^{2}

with within-context permutation

p

values from 200 draws in parentheses. Columns give the ball’s measured Weber contrast, with its shadow kept at every rung, and Native is the unmodified rendering. The zero rung removes the ball. Each entry uses 96 queries from 12 contexts in eight context folds. Cosmos null and text use the first generated RGB frame. † Prediction spread below 1% of the target spread.

Observation	Target PL $R^{2}$	RGB $R^{2}$	$S_{RGB}$ [95% interval]	Wins
VideoSAUR enc.	-0.280	-0.334	-0.075 [-0.134, 0.016]	10/24
C-JEPA pred.	0.148	-0.061	-0.035 [-0.141, 0.009]	8/24
LeWM enc.	0.451	0.158	-0.019 [-0.187, 0.142]	12/24
LeWM pred.	0.716	0.645	-0.082 [-0.524, 0.151]	11/24
AC enc.	0.876	0.780	-0.056 [-0.551, 0.066]	11/24
AC pred.	0.859	0.721	0.008 [-0.510, 0.121]	12/24

Appendix

Table 22: RGB-trained correction beyond the target’s own three-reference inverse. PL and RGB columns are median

R^{2}

scores. The paired skill

S_{RGB}=1-\mathrm{SSE}_{RGB}/\mathrm{SSE}_{PL}

isolates the effect of adding the frozen RGB correction. Brackets are 95% scene/context-block percentile intervals conditional on saved fits. Wins are paired axes.

Figure 4: Paired axis-wise $\Delta R^{2}$ from adding the frozen RGB correction to each target’s own PL inverse. Positive entries improve the target PL. Negative entries worsen it. The source RGB column is a self-check. All axes and negative values are retained. High absolute reverse scores alone do not establish a transferred correction.

Explore similar work

Aug 30, 2026cs.LG

The Intervention Gap in Latent World Models

Planning-time intervention fidelity is a distinct, measurable property of a learned world model: whether the model's own open-loop transitions move task variables the way matched environment interventions do. In the settings we test, it is neither revealed by reward fit nor ensured by task-anchored training. Across released TD-MPC2 checkpoint sizes, episode return falls as an operator-error diagnostic on task observables grows, while reward-prediction error stays small and nearly flat, and a self-supervised world model trained without task signal preserves the same operator substantially better than a task-anchored model on the shared task. A capture-gated matched-intervention audit then localizes what fails. On Cheetah, three LeWorldModel checkpoints capture the current task query and support decodable real intervention effects; however, their imagined five-step effects are worse than predicting no effect and worse than an environment-endpoint oracle. The failure is task-direction rotation with excess gain, not feature collapse. This severe pattern is conditional: five PreJEPA seeds retain an oracle-relative deficit without it, Finger Spin experiments extend the deficit beyond locomotion with heterogeneous severity across seeds, and shared-bank effect geometry is both candidate- and support-dependent. We also test practice-side questions. In DreamerV3 the posterior distribution, not its sample, carries the current query; ensemble disagreement ranks error only near training support; and a frozen support-aware score degrades held-out error ranking in both tested transfer directions while native disagreement remains informative in both. We conclude that intervention fidelity must be audited directly, capture-first, on the model's native interface.

Donna Vakalis

Mila – Quebec AI Institute · University of Montreal

Prediction Error

Ask PDF

Sep 29, 2026cs.CV

Do-JEPA: From Masking to Intervention in Latent World Models

Latent world models are trained to predict what happens next, so nothing in their objective separates what an action caused from what merely co-occurred with it. Object-masking models such as C-JEPA intervene on what the predictor can see; we intervene on what physically happens. From one saved simulator state we run the dynamics under an action

a

and under a reference action

a_{\varnothing}

, and train the model to predict the difference

Δz=z^{a}-z^{a_{\varnothing}}

between the two latent futures. The resulting objective, Do-JEPA, has an effect loss, a support loss (where the action enters), a propagation loss (where its effect travels) and invariance losses (what must not change). In a synthetic system with object-aligned variables, support supervision finds the directly intervened object in 99.95% of test cases, where a sparse action mask sends the action to a nuisance slot in every case, and response-onset supervision recovers the ring-shaped propagation graph (edge AUROC 0.975 vs. 0.624). From pixels, the effect loss beats a control trained on exactly the same data: it lowers latent effect error by 28.4% on an end-to-end LeWM model and physical effect error by 13.5% when trained and tested on natural action sequences, and on three independently generated CausalWorld benchmarks it lowers responsive effect error by about 20% under physics shifts and the latent context sensitivity of predicted effects by 66%. Trained from scratch it costs factual accuracy; fine-tuning an existing model with it removes this cost. Together, these results show that intervening on the world, rather than on what the model sees, helps latent world models predict what their actions cause.

Hossein Resani, Javen Qinfeng Shi

Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre

Latent World Models S-Jepa World

Ask PDF

Aug 15, 2026cs.RO

Low-Rank Dynamics-Effective Latent Carriers for Counterfactual Rollout in Learned World Models

We ask whether a small, directly addressable hidden-state intervention can place a learned world model on an intended counterfactual future and then let the model's own dynamics carry that future forward. In a controlled two-object collision environment, we study a 192-dimensional recurrent model trained on factual and locally edited counterfactual trajectories. Candidate carriers are learned from training-only counterfactual-minus-factual hidden differences, and an affine map predicts carrier coordinates from the factual state and requested edit without access to the native counterfactual hidden state at test time. For bounded single-component velocity edits, rank 4 is the smallest tested rank on the preregistered grid that satisfies the development criteria. A one-shot rank-4 patch launches a 12-transition autonomous rollout without future observations, teacher forcing, repeated hidden-state correction, or physical-state clamping. The frozen procedure satisfies the preregistered 2-of-3 fresh-checkpoint replication rule and remains reusable at nearby anchors. The same Single-derived carrier and Single-only affine map also support bounded same-object two-component requests. Across the matched training regimes, broader counterfactual support was associated mainly with better Joint rollout accuracy and more additive Joint hidden responses. Composition-related structure is enriched in the rank-4 subspace but is not confined to it, and local recurrent diagnostics show strong one-step coupling from the carrier to the rest of the hidden state. A position-edit stress test fails the required specificity controls. Together, these results support a compact dynamics-effective intervention-entry interface, not a closed four-dimensional state or an intrinsic state dimension.

Yang Liu, Yuming Chen

College of Intelligent Robotics and Advanced Manufacturing, Fudan University Shanghai, China

World Models Four-Dimensional Collision Prediction

Ask PDF