Video World Models Need Predictable Latents

Mattia Scardecchia; Sherry Yang

cs.LGOct 26, 2026

Video World Models Need Predictable Latents

Authors: Mattia Scardecchia, Sherry Yang

Organizations: New York University

Abstract

Video world models trained on large-scale robot data promise a safe and scalable approach for learning and evaluating policies that act in the physical world. However, they still struggle with long-horizon rollouts and out-of-distribution generalization, and they remain computationally demanding at inference time. Typically, video world models make predictions in the latent space of a reconstruction-trained autoencoder, which however is not directly optimized for dynamics modeling. On the other hand, world models designed for latent planning usually optimize latents to be predictable, but they forgo accurate prediction of observations in pixel space. In this paper, we show that it is possible to achieve the best of both worlds: a latent geometry optimized to make future prediction as simple as possible without sacrificing accurate prediction of future observations. More precisely, we find that temporal objectives for representation learning, paired with a past-conditioned decoder architecture, lead to consistent improvements in generative video world models at the same training budget. The gains include up to 5x cheaper inference, over 7x faster training, and 15-24% lower pixel and perceptual errors out of distribution and over long horizons. Our world models also agree more closely with real-world policy evaluation when using a vision-language model to judge task success from their pixel-space predictions. Overall, our study indicates that temporal predictability is an important design criterion for the latent space of video world models, and provides concrete guidance on how to incorporate it into the training of autoencoders.

Figures & tables

Figure 1: Method overview. (1) An autoencoder is fine-tuned on pairs of nearby video frames $x_{t}$ , $x_{t+1}$ , with a reconstruction loss and a temporal objective on the latents $z_{t}$ , $z_{t+1}$ and the action $a_{t}$ ; the decoder $D$ conditions on the past frame $x_{t}$ to reconstruct $x_{t+1}$ from its latent. (2) An action-conditioned diffusion world model is trained on the latents from the frozen encoder $E$ . (3) From a context frame $x_{1}$ and a sequence of actions $a_{1:T}$ , the world model generates latents, which $D$ decodes conditioning on $x_{1}$ as the past frame. We highlight in orange and green the two components that distinguish our recipe from a standard video world model: the temporal objective, and the past-conditioned decoder.

Figure 2: Latents can be predictable and informative in a synthetic setting. Left : Synthetic observations have a predictable component (signal, the dot) and an unpredictable noise component (the static). An encoder $E$ is trained with reconstruction, temporal contrastive learning or both, and three readouts are then trained on its frozen latents: a world model predicts the next latent from the current latent and action, a probe predicts the signal, and a decoder $D$ reconstructs the observation. Right : the error of each readout at latent size 100, divided by the error of the same readout trained on the raw observation; geometric mean and standard error over 3 seeds, lower is better. Reconstruction alone leaves the next latent as hard to predict as the raw observation and reads the signal far less accurately; contrastive learning alone loses the observation; their combination avoids both failures.

	ID $\downarrow$		OOD $\downarrow$		2-step $\downarrow$		Long-horizon $\downarrow$		Autoencoder
	MSE	LPIPS	MSE	LPIPS	MSE	LPIPS	MSE	LPIPS	Updates	Components
Reconstruction (baseline)	11.65	0.164	33.87	0.308	15.95	0.214	33.26	0.287	4.4	388
	$\pm$ 0.20	$\pm$ 0.001	$\pm$ 1.28	$\pm$ 0.008	$\pm$ 1.03	$\pm$ 0.004	$\pm$ 1.84	$\pm$ 0.008
Pixel prediction	11.43	0.159	34.23	0.300	14.53	0.199	32.63	0.279	8.9	445
	$\pm$ 0.22	$\pm$ 0.001	$\pm$ 0.67	$\pm$ 0.002	$\pm$ 0.75	$\pm$ 0.002	$\pm$ 0.69	$\pm$ 0.003
Contrastive prediction	11.17	0.160	31.23	0.303	12.38	0.185	31.50	0.277	96.0	293

Table 1: Temporal structure benefits world models at equal reconstruction quality. With the pipeline of Figure 1 , using the ordinary decoder, we compare different objectives for fine-tuning the autoencoder until a target reconstruction error is reached, training the same world model on the learned latents. We report mean and standard error over 3 seeds for the error of world model predictions against ground truth in the four evaluation conditions of Section 5 (pixel MSE

\times 10^{3}

; bold: best). Updates in thousands; Components: principal components for 95% of the latent variance. Temporal objectives, especially latent and contrastive prediction, improve performance across the board compared to the reconstruction baseline, but they require more autoencoder updates to reach the same reconstruction error.

Figure 3: The information-predictability frontier and the decoder’s reliance on the past frame. Left : reconstruction error against the unpredictable share of the latent variance, for autoencoders fine-tuned for 80,000 updates with latent prediction at weight

\lambda

, with the ordinary decoder or the past-conditioned decoder. Points are means over two training seeds at

\lambda=0

and 0.3 (faint: individual seeds) and one seed elsewhere. Raising

\lambda

trades reconstruction for predictability, and the past-conditioned decoder improves the frontier at every weight. Right : reconstruction error of each past-conditioned autoencoder with its past-frame pathway disabled (Appendix G.4 ), divided by its error with the correct past frame (log scale); same seeds as on the left. As the pressure from the temporal objective on the latent space increases, the decoder relies more on the past frame for accurate reconstructions.

Ordinary decoder
	ID $\downarrow$		OOD $\downarrow$		2-step $\downarrow$		Long-horizon $\downarrow$		Policy eval. $\uparrow$		Predictability $\downarrow$
	MSE	LPIPS	MSE	LPIPS	MSE	LPIPS	MSE	LPIPS	10 steps	2 steps	Latent MSE
Reconstruction (baseline)	14.34	0.194	36.15	0.302	18.35	0.232	47.93	0.394	0.51	0.21	0.232
	$\pm$ 0.16	$\pm$ 0.002	$\pm$ 0.98	$\pm$ 0.003	$\pm$ 0.65	$\pm$ 0.004	$\pm$ 1.51	$\pm$ 0.002	$\pm$ 0.10	$\pm$ 0.07	$\pm$ 0.008
Pretrained, no fine-tuning	14.81	0.208	34.81	0.309	19.15	0.250	46.78	0.393	0.57	0.21	0.253
	$\pm$ 0.26	$\pm$ 0.001	$\pm$ 1.97	$\pm$ 0.008	$\pm$ 0.89	$\pm$ 0.003	$\pm$ 0.83	$\pm$ 0.003	$\pm$ 0.07	$\pm$ 0.09	$\pm$ 0.001

Table 2: Past-conditioned decoding and temporal objectives improve world models at equal training budget. With the pipeline of Figure 1 , we compare decoders and objectives for fine-tuning the autoencoder for a fixed number of updates, training the same world model on the learned latents. The past-conditioned reconstruction row against the baseline gives the effect of the decoder; the two objective rows against that row give the effect of the temporal objective. We report mean and standard error over 3 seeds (pixel MSE

\times 10^{3}

; bold: best), except for policy evaluation in WorldGym (one training seed, bootstrap standard error over tasks). ID, OOD, 2-step and long-horizon as in Section 5 ; latent MSE: the error of the world model’s one-step forecast of the next latent (Section 5.3 ). Our recipe improves every metric compared to the baseline; both the past-conditioned decoder and the temporal objectives account for some of the improvements.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: The synthetic study at every latent size and weight. Errors on the observation, the signal, the next latent and the next signal against the latent size, for all six weights

\lambda

of the contrastive temporal term (

\lambda=0

: reconstruction alone;

\lambda=1

: the temporal term alone). Horizontal lines: the raw observation given directly to the same downstream networks (in the next-latent panel, hidden under

\lambda=0

). Mean and standard error over three seeds. Intermediate weights retain reconstruction information while improving signal recovery and latent predictability over reconstruction alone.

	Equal reconstruction (Section 5.1 )	Equal budget and weight sweep (Sections 5.2 and 5.3 )
Autoencoder fine-tuning
Frame pairs	consecutive, or two frames apart with probability 1/3	consecutive
Batch	24 pairs	48 pairs
Learning rate	$2.4\times 10^{-6}$	$3.46\times 10^{-6}$
Length	until the reconstruction target of Appendix F	80,000 updates; final checkpoint
Validation pairs	one per held-out video, redrawn at each check	a fixed set, one per held-out video

Appendix

Table 3: Training settings. The ablations use the settings of the equal-budget experiment unless Appendix G.9 states otherwise.

Condition	Clips	Scored prediction steps	Denoising chunk	DDIM steps	Sampling seeds
In distribution	2,000	1–9	8	10	3
Out of distribution	479	1–29	1	10	3
Sampling-step sweep	512	1–9	1	2, 4, 6, 8, 10	1
Long rollouts	716	1–59	8	10	1

Appendix

Table 4: Evaluation conditions. Each rollout starts from one real context frame, and the scored prediction steps count from the first frame after it. The denoising chunk is the largest number of future frames sampled jointly. The 2-step columns of the result tables come from the sampling-step sweep at 2 steps.

	ID			OOD
	MSE $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	MSE $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
Reconstruction (baseline)	11.65	0.164	0.801	33.87	0.308	0.675
	$\pm$ 0.20	$\pm$ 0.001	$\pm$ 0.002	$\pm$ 1.28	$\pm$ 0.008	$\pm$ 0.009
Pixel prediction	11.43	0.159	0.806	34.23	0.300	0.683
	$\pm$ 0.22	$\pm$ 0.001	$\pm$ 0.001	$\pm$ 0.67	$\pm$ 0.002	$\pm$ 0.002
Contrastive prediction	11.17	0.160	0.807	31.23	0.303	0.681

Appendix

Table 5: Complete results at equal reconstruction quality. Table 1 without its autoencoder columns, and with SSIM and the distribution metrics added; all pipelines use the ordinary decoder. Mean and standard error over three training seeds (pixel MSE

\times 10^{3}

); bold: best in each column. ID, OOD, 2-step and long-horizon as in Section 5 . Latent prediction gives the best perceptual metrics, especially with two sampling steps, and the lowest FID; contrastive prediction gives a slightly lower FVD.

Figure 5: World models at equal reconstruction quality, across sampling steps, rollout steps and training. The four pipelines of Table 1 , all with the ordinary decoder, against the number of DDIM sampling steps (top), along the long rollouts (middle) and over world-model training (bottom). Gray shading: prediction steps reached by fewer than half of the videos. Mean and standard error over three training seeds. Latent prediction gives the largest perceptual gains with few sampling steps, and the lowest LPIPS from the fourth prediction step on and throughout training.

Ordinary decoder
	ID			OOD
	MSE $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	MSE $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
Reconstruction (baseline)	14.34	0.194	0.779	36.15	0.302	0.682
	$\pm$ 0.16	$\pm$ 0.002	$\pm$ 0.001	$\pm$ 0.98	$\pm$ 0.003	$\pm$ 0.004
Pretrained, no fine-tuning	14.81	0.208	0.758	34.81	0.309	0.673
	$\pm$ 0.26	$\pm$ 0.001	$\pm$ 0.001	$\pm$ 1.97	$\pm$ 0.008	$\pm$ 0.009

Appendix

Table 6: Complete results at equal training budget. Table 2 without its policy-evaluation columns (Table 8 ), and with SSIM, the out-of-distribution latent MSE and the distribution metrics added. Mean and standard error over three training seeds (pixel MSE

\times 10^{3}

); bold: best in each column. ID, OOD, 2-step and long-horizon as in Section 5 . Both temporal objectives improve LPIPS and SSIM over both fine-tuned reconstruction pipelines in all four conditions; at

\lambda=0.7

, latent prediction has a higher FID and FVD than reconstruction with the same decoder, which lower weights avoid (Appendix G.9.1 ).

Figure 6: World models at equal training budget, across sampling steps, rollout steps and training. The pipelines of Table 2 against the number of DDIM sampling steps (top), along the long rollouts (middle) and over world-model training (bottom). Gray shading: prediction steps reached by fewer than half of the videos. Mean and standard error over three training seeds. The temporal objectives improve LPIPS and SSIM in all three rows, and inverse dynamics also gives a lower pixel MSE than either reconstruction-only pipeline.

Figure 7: Autoencoder losses over fine-tuning in the weight sweep. Reconstruction error (left) and error of the jointly trained latent predictor (right), for each temporal weight and decoder. One training seed; the predictor receives no gradient at

\lambda=0

. Past conditioning lowers both errors at every nonzero weight, and larger weights trade reconstruction for a lower prediction error.

	Reconstruction error $\downarrow$
$\lambda$	Past frame	Context disabled	Permuted past frame	Ordinary decoder
0	0.0231	0.0407	0.0394	0.0294
0.1	0.0237	0.0430	0.0424	0.0314
0.3	0.0257	0.0496	0.0481	0.0345
0.5	0.0288	0.0700	0.0565	0.0379
0.7	0.0375	0.1947	0.1072	0.0451

Appendix

Table 7: Dependence of the past-conditioned decoder on its context. Reconstruction error of the past-conditioned autoencoders of the weight sweep with three contexts; the last column gives the autoencoder trained with the ordinary decoder at the same weight. One training seed. The correct context lowers the error at every weight, and the dependence on the context is largest at

\lambda=0.7

Figure 8: Policy evaluation inside the world models. Three robot policies on the 17 Bridge tasks of WorldGym, inside the five world models of Table 2 . One training seed; error bars: standard error over the 170 starting frames (left) and bootstrap standard error over tasks (right). With 2 sampling steps, the temporal world models keep their agreement with the real world and the real ranking of the three policies, while the reconstruction-only world models lose both.

10 sampling steps
	OpenVLA		Octo		RT-1-X		Three policies pooled
	$r$ $\uparrow$	$\rho$ $\uparrow$	$r$ $\uparrow$	$\rho$ $\uparrow$	$r$ $\uparrow$	$\rho$ $\uparrow$	$r$ $\uparrow$	$\rho$ $\uparrow$	$\Delta r$ $\uparrow$	$p$
Ordinary decoder
Reconstruction (baseline)	0.40	0.27	0.58	0.53	0.11	0.19	0.51	0.45	–	–
	$\pm$ 0.20	$\pm$ 0.27	$\pm$ 0.21	$\pm$ 0.21	$\pm$ 0.24	$\pm$ 0.24	$\pm$ 0.10	$\pm$ 0.12
Pretrained, no fine-tuning	0.56	0.56	0.58	0.55	0.01	0.10	0.57	0.55	+0.06	0.261

Appendix

Table 8: Agreement with real-world policy evaluation at 100,000 updates. Pearson

r

and Spearman

\rho

between simulated and real success rates over the 17 tasks of each policy, or over the 51 policy–task pairs when pooled;

\Delta r

and

p

: paired difference from the baseline (–) and its two-sided bootstrap

p

-value. One training seed; bootstrap standard errors over tasks under the values. With two sampling steps, both temporal world models agree with real-world performance better than all reconstruction-only world models when the three policies are pooled.

Figure 9: AutoEval rollouts with video world models trained on different latent spaces. The baseline learns its latent space through reconstruction, as is standard; our recipe adds a temporal objective (latent prediction or inverse dynamics) and a past-conditioned decoder. One AutoEval trajectory, out of distribution: the recorded actions are replayed from one context frame for three times the length of the training clips, with 2 sampling steps. Figures 10 to 17 show more examples, selected and random.

			Lower than baseline (%)		Lower than all reconstruction-only (%)			Median change (%)
Setting	Steps	Clips	Latent	Inverse	Latent	Inverse	Both	Latent	Inverse
ID	10	224	92.0	93.3	80.4	79.9	69.6	$-13.1$	$-13.6$
ID	2	224	96.0	98.7	89.3	92.9	87.5	$-21.5$	$-21.1$
OOD	10	479	95.0	96.7	60.5	86.2	53.2	$-13.9$	$-22.8$
OOD	2	479	100.0	99.2	97.9	97.9	96.2	$-28.9$	$-31.8$

Appendix

Table 9: Rollout quality over all clips. Lower than baseline: percentage of clips on which the temporal world model has a lower clip-mean LPIPS than reconstruction with the ordinary decoder. Lower than all reconstruction-only: the same against all three reconstruction-only world models; Both: the two temporal world models at once. Median change: relative change of LPIPS against the baseline. ID: the 224 held-out videos with at least 60 frames, rolled out for 59 frames; OOD: the 479 AutoEval trajectories, rolled out for 29 frames; one rollout per clip. On almost every clip, the temporal world models are closer to the recording than the baseline, most consistently with 2 sampling steps.

Panel of Figure 10 — Figure 10: Out of distribution, 2 sampling steps, selected trajectories. AutoEval trajectories with the recorded actions replayed for 29 frames. Top row: the recorded frames; below: the rollout of each world model from the same context frame, with its LPIPS.

Panel of Figure 11 — Figure 11: Out of distribution, 2 sampling steps, random trajectories. AutoEval trajectories with the recorded actions replayed for 29 frames; rows as in Figure 10 .

Panel of Figure 12 — Figure 12: In distribution, 2 sampling steps, selected videos. Held-out Bridge videos rolled out for 59 frames; rows as in Figure 10 .

Panel of Figure 13 — Figure 13: In distribution, 2 sampling steps, random videos. Held-out Bridge videos rolled out for 59 frames; rows as in Figure 10 .

Panel of Figure 14 — Figure 14: Out of distribution, 10 sampling steps, selected trajectories. AutoEval trajectories with the recorded actions replayed for 29 frames; rows as in Figure 10 .

Panel of Figure 15 — Figure 15: Out of distribution, 10 sampling steps, random trajectories. AutoEval trajectories with the recorded actions replayed for 29 frames; rows as in Figure 10 .

Panel of Figure 16 — Figure 16: In distribution, 10 sampling steps, selected videos. Held-out Bridge videos rolled out for 59 frames; rows as in Figure 10 .

Panel of Figure 17 — Figure 17: In distribution, 10 sampling steps, random videos. Held-out Bridge videos rolled out for 59 frames; rows as in Figure 10 .

Panel of Figure 18 — Figure 18: OpenVLA in WorldGym with 2 sampling steps. One starting frame per panel, inside the five world models of Table 2 (one training seed), at frames 0 to 40; bold: the frames the judge sees. Beside each row: the judged outcome and the three votes.

200,000 world-model updates
	ID			OOD
	MSE $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	MSE $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
Reconstruction	12.14	0.163	0.813	34.53	0.280	0.707
	$\pm$ 0.36	$\pm$ 0.001	$\pm$ 0.002	$\pm$ 2.33	$\pm$ 0.006	$\pm$ 0.006
Inverse dynamics	11.56	0.148	0.823	31.19	0.242	0.737
	$\pm$ 0.02	$\pm$ 0.001	$\pm$ 0.001	$\pm$ 1.42	$\pm$ 0.005	$\pm$ 0.005

Appendix

Table 10: Longer world-model training. The three past-conditioned pipelines of the equal-budget experiment at 200,000 world-model updates and, for the same two training seeds, at 100,000. Mean and standard error over two training seeds (pixel MSE

\times 10^{3}

); bold: best in each column at each training length. ID, OOD, 2-step and long-horizon as in Section 5 . Inverse dynamics keeps its gains in every condition; latent prediction keeps a lower LPIPS than reconstruction in every condition and extends its out-of-distribution gain, but its in-distribution pixel MSE, FID and FVD are higher.

	OpenVLA		Octo		RT-1-X		Three policies pooled
	$r$ $\uparrow$	$\rho$ $\uparrow$	$r$ $\uparrow$	$\rho$ $\uparrow$	$r$ $\uparrow$	$\rho$ $\uparrow$	$r$ $\uparrow$	$\rho$ $\uparrow$	$\Delta r$ $\uparrow$	$p$
10 sampling steps
Reconstruction	0.50	0.45	0.55	0.53	0.24	0.41	0.61	0.61	–	–
	$\pm$ 0.16	$\pm$ 0.22	$\pm$ 0.21	$\pm$ 0.21	$\pm$ 0.23	$\pm$ 0.21	$\pm$ 0.08	$\pm$ 0.10
Inverse dynamics	0.54	0.47	0.57	0.53	0.34	0.30	0.66	0.65	+0.05	0.081
	$\pm$ 0.15	$\pm$ 0.20	$\pm$ 0.21	$\pm$ 0.21	$\pm$ 0.19	$\pm$ 0.19	$\pm$ 0.07	$\pm$ 0.10	$\pm$ 0.03

Appendix

Table 11: Agreement with real-world policy evaluation at 200,000 updates. The three past-conditioned world models of Table 10 , one training seed; columns as in Table 8 , with

\Delta r

and

p

against reconstruction (–). With two sampling steps, both temporal world models agree with real-world performance better than reconstruction when the three policies are pooled; with ten, the differences are smaller.

Figure 19: Policy evaluation after 200,000 world-model updates. The three past-conditioned world models of Table 11 , one training seed; panels, rows and error bars as in Figure 8 . With 2 steps, the temporal world models keep the real ranking of the policies and their agreement with the real world.

Equal reconstruction, three seeds
	Components for 95%	Participation ratio	Turning- angle ratio	Within / across video distance	Spectral slope	High-frequency spectral share
Reconstruction	388 $\pm$ 0.9	52.7 $\pm$ 0.03	1.100 $\pm$ 0.0001	0.423 $\pm$ 0.0004	$-$ 0.837 $\pm$ 0.0003	0.073 $\pm$ 0.00008
Latent prediction	177 $\pm$ 8	49.4 $\pm$ 1.1	1.019 $\pm$ 0.007	0.344 $\pm$ 0.005	$-$ 0.907 $\pm$ 0.010	0.061 $\pm$ 0.002
Pixel prediction	445 $\pm$ 0.3	64.4 $\pm$ 0.2	1.110 $\pm$ 0.0004	0.443 $\pm$ 0.001	$-$ 0.763 $\pm$ 0.0008	0.094 $\pm$ 0.0002
Contrastive prediction	293 $\pm$ 6	46.7 $\pm$ 0.7	1.086 $\pm$ 0.0009	0.384 $\pm$ 0.002	$-$ 0.933 $\pm$ 0.010	0.052 $\pm$ 0.002
Equal budget, past-conditioned decoder, three seeds

Appendix

Table 12: Geometry of the frozen latent spaces. Statistics of the latents in both experiments, defined in the text; lower values mean lower dimension, straighter trajectories, tighter clustering by video and smoother maps. Mean and standard error over three training seeds unless one is stated. Latent prediction reduces dimension and high-frequency content in both experiments, and inverse dynamics also smooths the latents.

Panel of Figure 20 — Figure 20: Latent maps at equal training budget. One held-out Bridge frame encoded by the encoders of the three past-conditioned pipelines, one training seed. Columns average channels 0–4, 5–9 and 10–15 of each $16\times 32\times 32$ latent. Each map is standardized separately, black low and white high, so intensities are not comparable across maps. Latent prediction and inverse dynamics produce smoother regions around the arm and the objects than reconstruction alone.

Figure 22: Spatial spectra of the frozen latents. Radial amplitude spectra of the latent channels, as a share of their total amplitude; at equal budget, solid curves use the past-conditioned decoder and dashed curves the ordinary one. Mean and standard error over three training seeds, except the pretrained encoder (one). Latent and contrastive prediction reduce the high-frequency components at equal reconstruction, and latent prediction and inverse dynamics do so at equal budget.

Figure 23: World-model improvement over reconstruction as the temporal weight varies. Latent prediction at each weight against reconstruction (

\lambda=0

, zero by construction, open marker), both with the past-conditioned decoder and the same training seed; panels have independent scales. One training seed, and two more at

\lambda=0.7

(dots). LPIPS improves at every weight in distribution, out of distribution and with two sampling steps; long-rollout gains favor larger weights.

Figure 24: Distribution metrics as the temporal weight varies. FID and FVD of latent prediction at each weight against reconstruction (

\lambda=0

), both with the past-conditioned decoder and the same training seed, in distribution; panels as in Figure 23 . One training seed, and two more at

\lambda=0.7

(dots). Weights from 0.1 to 0.5 improve both metrics; at

\lambda=0.7

, both are worse than at

\lambda=0

, for all three training seeds.

	ID			OOD
	MSE $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	MSE $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
Reconstruction	10.78	0.158	0.797	35.11	0.278	0.698
Latent prediction, $\lambda=0.1$	10.57	0.150	0.805	30.31	0.257	0.715
Latent prediction, $\lambda=0.3$	11.67	0.163	0.790	29.68	0.271	0.704
Latent prediction, $\lambda=0.7$	11.98	0.178	0.763	28.21	0.299	0.673
	2-step			Long-horizon

Appendix

Table 13: Latent prediction with a DINOv2 encoder. DINOv2 encoder and past-conditioned RAE decoder; one training seed (pixel MSE

\times 10^{3}

); bold: best in each column. ID, OOD, 2-step and long-horizon as in Section 5 . At

\lambda=0.1

, latent prediction improves every metric; larger weights lower the latent MSE further but worsen most other errors and the distribution metrics.

Figure 25: Effect of latent prediction across world-model sizes. Improvement of latent prediction over reconstruction, both with the past-conditioned decoder and the same seed: relative reductions of pixel MSE and LPIPS, and increases of SSIM. Rows of the equal-budget autoencoders show the mean and standard error over training seeds, with dots for single seeds; rows of the second autoencoder pair show one run each. The gains persist out of distribution and with two sampling steps at both widths; in distribution, the effect is mixed at width 384 and negative at width 1,024, where the long-rollout gains also shrink.

No KL penalty
	ID			OOD
	MSE $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	MSE $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
Reconstruction	13.76	0.178	0.801	31.81	0.274	0.713
Latent prediction	14.16	0.170	0.808	26.58	0.260	0.738
KL penalty, weight $10^{-6}$
Reconstruction	13.71	0.169	0.810	36.35	0.258	0.725

Appendix

Table 14: KL penalty. Reconstruction and latent prediction at

\lambda=0.7

, both with the past-conditioned decoder, with and without a KL penalty on the autoencoder; one training seed (pixel MSE

\times 10^{3}

); bold: best in each column. ID, OOD, 2-step and long-horizon as in Section 5 . With the penalty, latent prediction still improves every framewise metric and FID over reconstruction with the penalty, while its FVD is higher.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 257 papers

LaLa-World: Scaling Latent World Models with Natural Language Actions
Oct 20, 2026Junha Song, Geonmo Gu, Jeongeun Park +5World Model Learning Latent World Models
NAVER AI Lab · Georgia Institute of Technology · Brown University +1
Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence
May 15, 2026Ali J Alrasheed, Aryan Yazdan Parast, Basim Azam +2Video Representation Learning Video Prediction
The University of Melbourne · Monash University
Nano World Models: A Minimalist Implementation of Future Video Prediction
May 17, 2026Siqiao Huang, Partha Kaushik, Michael Chen +5Action-Conditioned Video Generation Long-Horizon Video Generation
Tsinghua University · Carnegie Mellon University · University of Bristol +2

Oct 20, 2026J/K move · Enter open · S save

LaLa-World: Scaling Latent World Models with Natural Language Actions

Junha Song, Geonmo Gu, Jeongeun Park, Heejeong Nam+4

NAVER AI Lab · Georgia Institute of Technology · Brown University+1

World Model Learning Latent World Models World Model-Based Planning Efficient Inference for World Action Models

Open paper AskPDF arXiv

This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning. To this end, we introduce LaLa-World, a Language-conditioned Latent World model that learns from simulation and robot trajectories together with broader image and video data. Two designs make this work. First, we describe both numerical actions and visual changes in language, enabling joint training of a single future predictor across heterogeneous datasets. Second, we build the predictor on a language model, allowing it to interpret language conditions while predicting future latent states. The resulting predictor achieves higher average MPC planning performance than existing LWMs trained separately for each domain or task. Adding image and video data further improves average planning performance and robustness to action phrasing. In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with

6\times

less training cost and

13.9\times

faster planning. Beyond planning, the pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations. LaLa-Policy performs competitively with Cosmos Policy on LIBERO and RoboCasa, with up to

5.5\times

faster inference.

Abstract

Video World Models

Explore similar work

LaLa-World: Scaling Latent World Models with Natural Language Actions

Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence

Nano World Models: A Minimalist Implementation of Future Video Prediction