Pretrained Features as Teachers, Not States: Compact Latent World Models for Visual Planning

Yanbo Wang; Nan Lin; Wendelin Böhmer; Javier Alonso-Mora

cs.CVOct 14, 2026

Pretrained Features as Teachers, Not States: Compact Latent World Models for Visual Planning

Authors: Yanbo Wang, Nan Lin, Wendelin Böhmer, Javier Alonso-Mora

Organizations: Delft University of Technology (TU Delft)

Abstract

A key design choice in visual world models is what information the latent state is encouraged to preserve. Models that learn a latent space with only a latent-prediction objective adapt to environment dynamics, but the predictive objective alone admits collapsed solutions and provides no explicit constraint on the visual information retained by the latent. Models built directly on pretrained visual features inherit a strong visual prior, but tie prediction and planning to a frozen state space. We propose Anchor-WM, which instead uses pretrained visual features as a teacher for learning a compact latent state rather than as the world-model state itself. Anchor-WM learns this compact latent with action-conditioned dynamics, while distilling the dense DINO representations through a training-only distillation head. This encourages the compact latent to retain visual information captured by the pretrained representation while discouraging representational collapse. At inference, the DINO encoder and distillation head are discarded, allowing prediction and planning to operate entirely in the learned latent space. Anchor-WM achieves the best average success rate across the four evaluated environments while enabling substantially faster planning than world models operating directly on DINO patch features. These results show that pretrained visual features can effectively guide world-model learning without defining the state space used for prediction and planning.

Figures & tables

Figure 1: Training and inference of Anchor-WM. During training, the observation encoder $\mathcal{E}_{\theta}$ , transition model $\mathcal{T}_{\phi}$ , and distillation head $\mathcal{H}_{\psi}$ are optimized jointly, while the pretrained DINO encoder remains frozen. The compact latent is trained with both a dynamics prediction objective and a distillation objective that reconstructs DINO features. At inference, the DINO encoder and distillation head are discarded, and planning is performed entirely in the learned latent space by optimizing an action sequence whose predicted terminal latent approaches the encoded goal latent.

Figure 2: Evaluation Environments. Top left: Two-Room for 2D navigation. Top right: Reacher for planar two-link arm control. Bottom left: Push-T for contact-based 2D object manipulation. Bottom right: Cube for 3D robotic cube manipulation. Further details are given in Section B .

Method	Latent Size	TwoRoom	Reacher	PushT	Cube	Average
Predictive	1 $\times$ 192	35.2 $\pm$ 2.2	10.6 $\pm$ 5.3	3.2 $\pm$ 1.1	46.5 $\pm$ 2.9	23.9
PLDM	1 $\times$ 192	93.0 $\pm$ 2.0	69.5 $\pm$ 5.8	71.8 $\pm$ 6.1	51.2 $\pm$ 1.6	71.4
LeWM	1 $\times$ 192	86.1 $\pm$ 1.8	78.4 $\pm$ 5.7	90.4 $\pm$ 4.3	64.8 $\pm$ 2.1	79.9
DINO-WM	196 $\times$ 384	99.6 $\pm$ 0.9	67.2 $\pm$ 1.8	65.2 $\pm$ 6.6	74.4 $\pm$ 2.6	76.6
DINO-WM (CLS)	1 $\times$ 384	59.3 $\pm$ 1.7	45.2 $\pm$ 3.4	37.8 $\pm$ 5.6	50.2 $\pm$ 3.8	48.1
Anchor-WM (ours)	1 $\times$ 192	99.6 $\pm$ 0.9	82.4 $\pm$ 4.3	84.8 $\pm$ 5.4	72.0 $\pm$ 4.2	84.7

Table 1: Planning Performance Comparison . Success Rate (

\uparrow

) across four environments, with the final column reporting the average across tasks. Bold indicates the highest mean success rate in each environment.

Method	TwoRoom	Reacher	PushT	Cube	Average
LeWM	79.2 $\pm$ 3.6	95.2 $\pm$ 1.8	41.6 $\pm$ 7.1	44.8 $\pm$ 8.3	65.2
DINO-WM	96.0 $\pm$ 2.5	90.0 $\pm$ 5.7	32.0 $\pm$ 2.8	53.2 $\pm$ 5.0	67.8
Anchor-WM (ours)	96.4 $\pm$ 1.7	96.4 $\pm$ 3.3	46.4 $\pm$ 3.6	45.2 $\pm$ 2.7	71.1

Table 2: Long-horizon planning performance. Planning success rates when the goal observation is selected 50 timesteps after the start observation and the evaluation budget is 100 steps.

Method	TwoRoom	Reacher	PushT	Cube	Average
LeWM	62.8 $\pm$ 8.2	56.4 $\pm$ 8.4	10.8 $\pm$ 3.0	64.0 $\pm$ 4.0	48.5
DINO-WM	98.0 $\pm$ 2.4	60.8 $\pm$ 7.0	62.8 $\pm$ 4.8	76.4 $\pm$ 3.6	74.5
Anchor-WM (ours)	99.2 $\pm$ 1.1	47.2 $\pm$ 10.8	33.2 $\pm$ 4.8	68.0 $\pm$ 6.5	61.9

Table 3: Zero-shot Planning Performance. Anchor-WM and the baselines are trained on the original environments shown in Fig. 2 . At test time, we introduce unseen visual perturbations to evaluate zero-shot generalization, such as changing the end-effector or target-object color.

Figure 3: Planning Time Comparison. Anchor-WM achieves planning times comparable to LeWM while being substantially faster than DINO-WM across all four environments.

Property	Model	Linear MSE $\downarrow$	Linear $r$ $\uparrow$	MLP MSE $\downarrow$	MLP $r$ $\uparrow$
Joint Position	LeWM	$0.216\pm.003$	$0.702\pm.002$	$0.212\pm.002$	$0.705\pm.003$
	DINO-WM	$0.193\pm.003$	$\mathbf{0.736\pm.002}$	$0.203\pm.004$	$\mathbf{0.741\pm.004}$
	Anchor-WM (ours)	$\mathbf{0.192\pm.003}$	$0.734\pm.002$	$\mathbf{0.192\pm.003}$	$0.740\pm.003$
Joint Velocity	LeWM	$0.987\pm.011$	$0.140\pm.005$	$0.997\pm.011$	$0.117\pm.009$
	DINO-WM	$\mathbf{0.883\pm.011}$	$\mathbf{0.315\pm.004}$	$0.861\pm.015$	$0.357\pm.003$
	Anchor-WM (ours)	$0.931\pm.011$	$0.254\pm.005$	$\mathbf{0.802\pm.018}$	$\mathbf{0.427\pm.006}$

Table 4: Probing the Latent Space. Probing results on OGBench-Cube. MSE (

\downarrow

) and Pearson correlation

r

(

\uparrow

) for linear and MLP probes. Best mean among all methods per metric in bold.

Figure 4: Open-Loop Rollouts on PushT and Cube. DINO-WM produces visually sharper reconstructions, whereas Anchor-WM discards some fine-grained appearance information. Nevertheless, Anchor-WM preserves the task-relevant evolution of object configurations and interactions. Considering the planning results in Table 1 and Table 2 , these examples suggest that visual reconstruction fidelity is not necessarily indicative of a world model’s utility for planning.

Ablation	Setting	TwoRoom	Reacher	PushT	Cube
Distillation Objective	MSE	99.6 $\pm$ 0.9	82.4 $\pm$ 4.3	$84.8\pm 5.4$	$72.0\pm 4.2$
Distillation Objective	Cosine	89.2 $\pm$ 1.8	78.6 $\pm$ 3.1	$79.5\pm 6.8$	$67.0\pm 3.8$
Distillation Target	Patch	99.6 $\pm$ 0.9	82.4 $\pm$ 4.3	$84.8\pm 5.4$	$72.0\pm 4.2$
Distillation Target	Global	83.8 $\pm$ 2.4	54.7 $\pm$ 3.7	$64.3\pm 3.3$	$49.6\pm 5.2$
DINO Guidance	Distillation	99.6 $\pm$ 0.9	82.4 $\pm$ 4.3	$84.8\pm 5.4$	$72.0\pm 4.2$
DINO Guidance	Alignment	50.0 $\pm$ 5.7	48.1 $\pm$ 2.3	$42.7\pm 5.1$	$50.4\pm 3.9$

Table 5: Ablation Results. We evaluate the effects of the distillation objective, distillation target, and DINO guidance strategy across all the environments by reporting planning success rate (

\uparrow

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Property	Model	Linear MSE $\downarrow$	Linear $r\uparrow$	MLP MSE $\downarrow$	MLP $r\uparrow$
Agent Location	LeWM	$0.060\pm.002$	$0.970\pm.001$	$\bm{0.005\pm.001}$	$\bm{0.998\pm.001}$
	DINO-WM	$0.031\pm.002$	$0.984\pm.001$	$0.012\pm.003$	$0.994\pm.001$
	Anchor-WM (ours)	$\bm{0.027\pm.002}$	$\bm{0.987\pm.001}$	$0.007\pm.002$	$0.997\pm.001$
Block Location	LeWM	$\bm{0.028\pm.001}$	$\bm{0.986\pm.000}$	$\bm{0.001\pm.000}$	$\bm{0.999\pm.000}$
	DINO-WM	$0.026\pm.001$	$0.987\pm.000$	$0.008\pm.001$	$0.996\pm.000$
	Anchor-WM (ours)	$0.043\pm.002$	$0.978\pm.001$	$0.005\pm.001$	$0.997\pm.000$

Appendix

Table 6: Probing results on PushT. MSE (

\downarrow

) and Pearson correlation

r

(

\uparrow

) for linear and MLP probes. Values are mean

\pm

standard deviation over five splits. Best of {Anchor-WM, LeWM} per metric in bold.

Figure 5: Additional open-loop rollout results. Additional qualitative rollouts on PushT and OGBench-Cube. Each pair of rows compares ground-truth observations with observations decoded from the open-loop latent predictions of Anchor-WM.

Figure 6: Latent-space organization in PushT. Left: a grid of physical states in PushT, with color indicating the state location. Right: a two-dimensional projection of the corresponding learned latent representations, using the same color assignment. The projected latents exhibit a smooth and coherent organization, with nearby physical states generally remaining close in the learned representation.

Figure 7: Effect of the distillation weight

\lambda

. Planning success rate of Anchor-WM on OGBench-Cube as the weight

\lambda

of the DINO distillation objective is varied. Setting

\lambda=0

removes the distillation objective. Performance improves substantially when distillation is introduced and remains relatively stable across a broad range of nonzero values, indicating that Anchor-WM is not highly sensitive to the precise choice of

\lambda

Figure 8: Effect of latent dimensionality. Planning success rate of Anchor-WM on OGBench-Cube with latent dimensions of 64, 192, and 384. Performance remains similar across all three settings, indicating that Anchor-WM is relatively insensitive to the latent dimensionality within this range.

Figure 9: Out-of-Distribution Environment Illustration. Top Left: Two-Room, we change the agent color from red to yellow. Top-Right: Reacher, we change the arm color from blue to green. Bottom Left: Push-T, we change the object color from grey to orange. Bottom Right: OGBench-Cube, we change the cube color from red to brown.

Hyperparameter	Value
Encoder	ViT-Tiny (patch size 14), from scratch
Input resolution	$224\times 224$
Latent dimension	192 (CLS token)
Predictor	Causal Transformer, 6 layers
Attention heads / head dim	16 / 64
Predictor MLP dimension	2048

Appendix

Table 7: Anchor-WM hyperparameters (shared across environments).

World Model-Based Planning

World model-based planning uses models of how environments change to compare imagined actions and choose goal-directed behavior.

See all 245 papers

Scope-WM: Scoped Computation for Efficient Visual World Models
Sep 27, 2026Chunzheng Li, Zesheng Jia, Hongda Zhang +4World Models for Robotics World Models
Soochow University · Fudan University
Learning Visual Feature-Based World Models via Residual Latent Action
May 8, 2026Xinyu Zhang, Zhengtong Xu, Yutian Tao +3Latent Action Learning World Model Learning
Rutgers University · Purdue University · University of Wisconsin-Madison
Fast LeWorldModel
Jun 24, 2026Yuntian Gao, Xiangyu XuLatent Dynamics Modeling World Model-Based Planning
Xi’an Jiaotong University

Sep 27, 2026J/K move · Enter open · S save

Scope-WM: Scoped Computation for Efficient Visual World Models

Chunzheng Li, Zesheng Jia, Hongda Zhang, Jiaying Tang+3

Soochow University · Fudan University

World Models for Robotics World Models World Model Learning World Model-Based Planning Model-Based Planning

Open paper AskPDF arXiv

Visual world models enable robotic planning by predicting future observations, but dense latent-state propagation and sample-intensive trajectory optimization incur high inference latency and peak memory usage, limiting real-time deployment on resource-constrained platforms. Existing sparse world-model acceleration methods either rely on unguided token sparsification, which may discard planning-relevant information and restrict achievable sparsity, or introduce heavy auxiliary modules and cumbersome multi-stage training pipelines. In this work, we present Scope-WM, an efficient visual world model that scopes computation to prediction-relevant latent regions and promising action sequences. Scope-WM distills prediction relevance into a lightweight action-conditioned selector and applies full dynamics prediction only to a compact subset of selected tokens. It updates the remaining tokens using a compact summary of foreground states and their changes, allowing the background to perceive foreground dynamics without costly token-to-token interactions. During planning, Scope-WM preserves and reuses high-quality action sequences discovered during the initial MPC search, focusing subsequent search under reduced rollout budgets. The resulting pipeline requires only a one-off selector distillation followed by a single joint training stage for the sparse world model. On the challenging Push-T task, Scope-WM reduces peak GPU memory usage and planning time to

18.1\%

and

14.3\%

of those of dense DINO-WM, respectively, corresponding to a

6.97\times

planning speedup, while maintaining competitive task performance. Further evaluations across five diverse visual planning tasks demonstrate the general applicability of Scope-WM. Code is available at https://github.com/ChunZheng2022/Scope-WM.

Abstract

World Model-Based Planning

Explore similar work

Scope-WM: Scoped Computation for Efficient Visual World Models

Learning Visual Feature-Based World Models via Residual Latent Action

Fast LeWorldModel