Visual world models enable robotic planning by predicting future observations, but dense latent-state propagation and sample-intensive trajectory optimization incur high inference latency and peak memory usage, limiting real-time deployment on resource-constrained platforms. Existing sparse world-model acceleration methods either rely on unguided token sparsification, which may discard planning-relevant information and restrict achievable sparsity, or introduce heavy auxiliary modules and cumbersome multi-stage training pipelines. In this work, we present Scope-WM, an efficient visual world model that scopes computation to prediction-relevant latent regions and promising action sequences. Scope-WM distills prediction relevance into a lightweight action-conditioned selector and applies full dynamics prediction only to a compact subset of selected tokens. It updates the remaining tokens using a compact summary of foreground states and their changes, allowing the background to perceive foreground dynamics without costly token-to-token interactions. During planning, Scope-WM preserves and reuses high-quality action sequences discovered during the initial MPC search, focusing subsequent search under reduced rollout budgets. The resulting pipeline requires only a one-off selector distillation followed by a single joint training stage for the sparse world model. On the challenging Push-T task, Scope-WM reduces peak GPU memory usage and planning time to 18.1% and 14.3% of those of dense DINO-WM, respectively, corresponding to a 6.97× planning speedup, while maintaining competitive task performance. Further evaluations across five diverse visual planning tasks demonstrate the general applicability of Scope-WM. Code is available at https://github.com/ChunZheng2022/Scope-WM.
Figures & tables
Figure 1: Performance–efficiency trade-off of Scope-WM on Push-T. (a) Different token budgets yield different performance–efficiency operating points under varying sparsity. (b) Scope-WM substantially reduces planning time and peak GPU memory over dense DINO-WM.
Figure 2: Overview of Scope-WM. Scope-WM scopes computation at the token, dynamics, and planning levels. At the token level, the action-conditioned DRS selector identifies a compact set of prediction-relevant foreground tokens from the encoded visual state. At the dynamics level, the sparse world-model predictor applies full dynamics prediction to the selected tokens, while FDBU uses a summarized foreground-dynamics context to update the remaining background tokens through a lightweight gated residual update, producing the complete latent rollout. At the planning level, EB-CEM performs test-time action optimization by retaining and reusing high-quality elite action sequences across MPC steps. Candidate action sequences are evaluated by a planning objective that compares the predicted latent rollout with the goal latent encoded by the frozen visual encoder.
Method
Full-dyn. tokens
SR ( ↑ )
FLOPs (G, ↓ )
Plan time (s, ↓ )
Speedup ( ↑ )
Peak Mem. (MB, ↓ )
Mem. Ratio ( ↓ )
DINO-WM (Dense)
196
90%
31.89
240.96
1.00×
18436
100.0%
Scope-WM (Fast)
98
94%
15.04
93.06
2.59×
7088
38.4%
Scope-WM (Faster)
32
88%
5.27
34.56
6.97×
3334
18.1%
Table 1: End-to-end performance–efficiency trade-off on Push-T. Fast and Faster apply full dynamics prediction to 98 and 32 of 196 visual tokens. FLOPs are reported per prediction sample, with memory ratios normalized to DINO-WM.
Method
PointMaze (SR ↑ )
Push-T (SR ↑ )
Wall (SR ↑ )
Rope (CD ↓ )
Granular (CD ↓ )
IRIS
74%
32%
4%
1.11
0.37
DreamerV3
100%
30%
100%
2.49
1.05
Sparse Imagination
100%
78.3%
95%
–
–
DINO-WM
98%
90%
96%
0.41
0.26
Scope-WM (Fast, K=98 )
100%
94%
92%
0.87
0.22
Scope-WM (Faster, K=32 )
98%
88%
92%
0.71
0.25
Table 2: Cross-task planning performance. SR ( ↑ ) is reported for PointMaze, Push-T, and Wall, and CD ( ↓ ) for Rope and Granular. Fast and Faster use 98 and 32 full-dynamics tokens.
Variant
Selection
BG update
Planner
SR ( ↑ )
w/o DRS
Random
FDBU
EB-CEM
76%
w/o FDBU
DRS
Copy
EB-CEM
56%
w/o EB-CEM
DRS
FDBU
CEM
82%
Scope-WM
DRS
FDBU
EB-CEM
88%
Table 3: Component ablation on Push-T under the 32 -token full-dynamics setting used by Scope-WM (Faster).
Figure 3: Effect of the full-dynamics token budget on Push-T. Bars show planning time and peak GPU memory normalized to dense DINO-WM, with absolute values annotated above the bars. The line shows success rate as the token budget varies across configurations.
Figure 4: Qualitative analysis of Scope-WM on Push-T. (a) DRS focuses computation on prediction-relevant regions. (b) FDBU captures background evolution without dense token-to-token interactions. (c) EB-CEM reuses promising early trajectories for subsequent replanning.
Selection
Background update
Copy
FDBU
Random
72%
76%
DRS
56%
88%
Table 4: Interaction between DRS and FDBU on Push-T under the same 32-token budget and EB-CEM planner. Entries report success rate (SR, ↑ ).
Model
Planner
Samples
SR ( ↑ )
Plan time (s, ↓ )
DINO-WM
CEM
100/100
90%
240.96
DINO-WM
EB-CEM
300/100
84%
267.36
Scope-WM
CEM
100/100
82%
30.67
Scope-WM
CEM
200/200
88%
60.96
Scope-WM
EB-CEM
300/100
88%
34.56
Table 5: Interaction between sparse prediction and planning strategy on Push-T. Scope-WM uses the 32-token setting; sample budgets denote initial/subsequent MPC steps.
CEM
EB-CEM
Early samples
100
100
300
300
Later samples
100
300
100
100
SR ( ↑ )
82%
82%
86%
88%
Table 6: Effect of sample allocation and elite reuse on Push-T. Early and later denote initial and subsequent MPC steps.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Hyperparameter
Setting
Visual representation
Encoder
Frozen DINOv2 ViT-S/14
Input resolution
224×224
Visual tokens / token dimension
196/384
Dynamics predictor
Architecture
6-layer causal Transformer
Attention heads / head dimension
16/64
Feed-forward hidden dimension
2048
Appendix
Table S1: Shared model and optimization settings. LN denotes layer normalization.
Task
History
Frameskip
Global batch
Training GPUs
Epoch budget
Train K
Eval. instances
Max MPC steps
PointMaze
3
5
512
8 V100
≤100
98 / 32
50
15
Wall
1
5
512
8 V100
≤100
98 / 32
50
15
Push-T
3
5
512
8 V100
≤100
98 / 32
50
15
Rope
1
1
64
1 RTX 2080 Ti
100
98 / 32
10
7
Granular
1
1
64
1 RTX 2080 Ti
100
98 / 32
10
7
Appendix
Table S3: Task-specific training and evaluation settings. Fast and Faster are independently trained with K=98 and K=32 , respectively.
Task
Method
K
Plan time (s, ↓ )
Speedup ( ↑ )
Peak Mem. (MB, ↓ )
Mem. Ratio ( ↓ )
SR ( ↑ )
PointMaze
DINO-WM
196
204.52
1.00×
6309
100.0%
98%
PointMaze
Scope-WM (Faster)
32
35.01
5.84×
3334
52.8%
98%
Wall
DINO-WM
196
59.44
1.00×
1542
100.0%
96%
Wall
Scope-WM (Faster)
32
19.60
3.03×
1939
125.8%
92%
Push-T
DINO-WM
196
240.96
1.00×
18436
100.0%
90%
Push-T
Scope-WM (Faster)
32
34.56
6.97×
3334
18.1%
88%
Appendix
Table S5: End-to-end efficiency on three rigid-body planning tasks. Time and peak memory use one fixed full-length run; SR uses the complete 50 -instance set. Memory ratios are normalized per task, and the peak includes the larger initial EB-CEM population.
Figure S1: Environment and decoded latent rollouts on Push-T. The upper row executes the planned actions in the environment; the lower row decodes the corresponding Scope-WM latent rollout with the dense DINO-WM decoder. Steps proceed left to right, and the highlighted final column is the goal. The decoder is used only for visualization.
Joint-Embedding Predictive Architectures (JEPAs), including recent LeWorldModel (LeWM), have become a promising foundation for reconstruction-free visual world models. For visual planning, however, LeWM evaluates candidate action sequences by repeatedly applying a local one-step latent transition model. This autoregressive rollout makes planning computationally expensive and exposes the predicted trajectory to accumulated latent errors as the horizon grows. We propose Fast LeWorldModel (Fast-LeWM), a fast latent world model that replaces repeated local rollout with action-prefix prediction. Given the current latent and a candidate action sequence, Fast-LeWM encodes its prefixes and predicts the future latents reached after executing those prefixes in parallel. By making action prefixes the basic prediction unit, Fast-LeWM directly models action effects accumulated to different extents over multiple horizons. This prefix-level supervision forces the model to learn how states continuously evolve under different action prefixes, rather than only fitting one-step state transitions. During planning, the predictor can use the prefix token from the encoded action sequence to evaluate the corresponding future latent without explicitly rolling through each intermediate imagined state. Across multiple tasks, Fast-LeWM improves average success over LeWM while substantially reducing planning time, achieving lower open-loop latent loss whose growth becomes significantly slower as the rollout horizon increases.
A key design choice in visual world models is what information the latent state is encouraged to preserve. Models that learn a latent space with only a latent-prediction objective adapt to environment dynamics, but the predictive objective alone admits collapsed solutions and provides no explicit constraint on the visual information retained by the latent. Models built directly on pretrained visual features inherit a strong visual prior, but tie prediction and planning to a frozen state space. We propose Anchor-WM, which instead uses pretrained visual features as a teacher for learning a compact latent state rather than as the world-model state itself. Anchor-WM learns this compact latent with action-conditioned dynamics, while distilling the dense DINO representations through a training-only distillation head. This encourages the compact latent to retain visual information captured by the pretrained representation while discouraging representational collapse. At inference, the DINO encoder and distillation head are discarded, allowing prediction and planning to operate entirely in the learned latent space. Anchor-WM achieves the best average success rate across the four evaluated environments while enabling substantially faster planning than world models operating directly on DINO patch features. These results show that pretrained visual features can effectively guide world-model learning without defining the state space used for prediction and planning.
World-model-based visual navigation enables action planning by predicting future states under candidate trajectories. However, long-horizon rollout remains challenging: models are often trained with ground-truth visual histories but deployed in closed-loop prediction, and evaluating many candidates with step-by-step rollout is computationally expensive. We propose SkipWorld, a latent skip planning framework for efficient visual navigation. SkipWorld introduces three components: (i) Latent AutoRoll, which trains a world model to recursively roll out future states in latent space, reducing the mismatch between training and closed-loop inference; (ii) SkipDistill, which distills the latent autoregressive model into a skip world model that directly predicts long-stride future latent states; and (iii) Skip-then-Refine Planning, a dual-model coarse-to-fine planner that first filters candidate trajectories with the skip model and then reranks selected candidates with the stronger autoregressive model. This design preserves accurate trajectory evaluation while reducing redundant rollout computation. Experiments on RECON, HuRoN, SCAND, and TartanDrive show that, compared with NWM, SkipWorld reduces the average ATE by 4.2% and RPE by 2.7%, while decreasing planning time by 62.4% with a 2.7-fold speedup.
Yifei Yang, Zehua Fan, Zike Yan +2
Institute for AI Industry Research (AIR), Tsinghua University · Hong Kong Embodied AI Lab · The Chinese University of Hong Kong +1