World models provide a powerful framework for simulating environment dynamics conditioned on actions or instructions, enabling downstream tasks such as action planning or policy learning. Recent approaches leverage world models as learned simulators, but its application to decision-time planning remains computationally prohibitive for real-time control. A key bottleneck lies in latent representations: conventional tokenizers encode each observation into hundreds of tokens, making planning both slow and resource-intensive. To address this, we propose CompACT, a discrete tokenizer that compresses each observation into as few as 8 tokens, drastically reducing computational cost while preserving essential information for planning. An action-conditioned world model that occupies CompACT tokenizer achieves competitive planning performance with orders-of-magnitude faster planning, offering a practical step toward real-world deployment of world models.
Figures & tables
Figure 1: Overview of the proposed latent world model formulation (Sec. 3.1 ). (a) An image tokenizer is first trained with a reconstruction objective to map an input image into compact latent tokens z . (Fig. 2 and Sec. 3.2 ). (b) Using the learned tokenizer, latent world model fϕ(zt,at) is trained to model the conditional distribution of the future state pϕ(zt+1∣zt,at) , where we adopt masked generative modeling (Sec. 3.3 ). (c) At test time, the learned latent world model is used for decision-time planning : An optimization procedure (e.g., MPC with CEM) searches over actions a0:H−1 to minimize the distance between the predicted final state and a goal image.
Figure 2: A tokenizer architecture detail. During training, only the latent resampler and Dcompact are updated. Eψ produces masked target tokens (training only), while Dψ is used only during inference for pixel level reconstruction.
Model
Type
#tok
rFID ↓
IS ↑
SD-VAE [ 57 ]
cont.
1024
0.64
223.8
MaskGIT-VQGAN [ 8 ]
disc.
256
1.83
186.7
TA-TiTok-VQ [ 39 ]
disc.
32
3.95
219.6
TA-TiTok-KL [ 39 ]
cont.
32
1.93
222.0
FlexTok [ 2 ]
disc.
1-256
5.60 †
114.9
CompACT
disc.
16
2.40
209.0
Table 1: Reconstruction performance of CompACT on ImageNet validation split. Metrics are computed using open-sourced checkpoints. rFID is measured using clean-fid [ 54 ] . † : Measured using 16 tokens.
Configuration
rFID ↓
#tok
Target tokenizer ( Dψ∘Eψ ) [ 8 ]
1.83
256
DINOv3-B (frozen) + latent resampler
2.40
16
Dcompact
w/o generative decoding
28.80
16
Ecompact
ViT-B (trained from scratch) + [REG]
7.28
16
DINOv3-B (finetuned) + [REG]
4.51
16
DINOv3-B (finetuned) + latent resampler
5.22
16
Table 2: Ablation on CompACT tokenizer. rFID is measured on ImageNet [ 15 ] validation split using clean-fid [ 54 ] .
Figure 3: CompACT encoder Ecompact architecture variation. (a) ViT (scratch)+ [ REG ]: Initial latent tokens are concatenated to the input patch tokens. This design follows previous transformer-based image tokenizers [ 77 , 2 , 75 ] . (b) DINOv3 [ 61 ] + [ REG ]: Similar to (a), but encoder is initialized with Dinov3. (c) DINOv3 [ 61 ] + latent resampler: latent resampler and Dinov3 initialized encoder. Dino and ViT are updated during training in these variants.
Figure 4: Attention visualization for compact latent token in latent resampler. Brighter the color, higher the attention score.
Tokenizer
#tok
L1 err ↓
R2↑
Target tokenizer ( Dψ∘Eψ ) [ 8 ]
256
0.093
0.684
CompACT
16
0.091
0.716
Table 3: Performance of Inverse Dynamics Model (IDM) trained with different tokenizers on RoboNet [ 13 ] . L1 error and R2 are measured between ground truth and predicted end effector position.
Tokenizer
#tok
RECON
SCAND
Latency ↓
ATE ↓
RPE ↓
ATE ↓
RPE ↓
SD-VAE [ 57 ]
784
1.262
0.354
1.065
0.291
178.78
FlexTok [ 2 ]
64
1.484
0.400
1.578
0.378
16.68
FlexTok [ 2 ]
16
1.625
0.446
1.503
0.362
14.48
CompACT
16
1.330
0.390
1.358
0.336
5.78
CompACT
8
1.373
0.401
1.391
0.346
4.83
Table 4: Planning performance of NWM on RECON benchmark with different tokenizers. Latency (sec) represents single trajectory optimization time using a single RTX 6000 ADA GPU.
Figure 5: Qualitative results of planning with the proposed CompACT. Best and Worst 1&2 denote the final rollouts corresponding to the simulated trajectories with the minimum and maximum cost, respectively.
Hist. mask
ATE
✓
1.330
1.480
Table 5: Effect of design choices in terms of planning accuracy on RECON. ( Left ): Effect of the history masking in the world model fθ (Sec. 3.3 ). ( Middle ): Comparison between the different cost function (Sec. 3.1 ). ( Right ): Effect of the freezing vision encoder during tokenizer training (Sec 3.2.1 ).
Model
#tok
APE
Latency (sec)
Target tokenizer ( Dψ∘Eψ ) [ 8 ]
256
0.3383
3.826
CompACT
16
0.1122
0.740
Table 6: Action-conditioned video prediction results on RoboNet [ 13 ] . Action prediction error is measured using IDM reported in Tab 3 . Latency is measure for when generating next 14 frames on a single RTX 6000 ADA GPU.
Figure 6: Qualitative results of action-conditioned video generation. Red and green boxes indicate incorrect and correct end-effector positions, respectively.
Backbone
SigLIP-2
MAE
DINOv3
rFID ↓
2.09
3.43
2.40
Table 7: Cross-backbone ablation of CompACT. Reconstruction performance of CompACT with 16 tokens on ImageNet validation split across different backbone choices.
Figure 7: Planning latency breakdown. Comparison of rollout and decoding latency between SD-VAE and CompACT on a single RTX 6000 ADA GPU. Note the log scale on the x -axis.
Tokenizer
SR ↑
Steps ↓
Target tokenizer ( Dψ∘Eψ ) [ 8 ]
56%
66.8
CompACT (ours)
56%
55.1
Table 8: Closed-loop manipulation on RoboMimic Lift.
hparams
CompACT
optimizer
AdamW
β1
0.9
β2
0.999
weight decay
0.01
lr
0.0001
lr scheduling
cosine
Table 9: Training hyperparameters for CompACT.
hparams
Latent resampler
Dcompact
depth
5
16
dim
768
1024
MLP dim
3072
4096
heads
8
8
Table 10: Model architecture hyperparameters for CompACT.
Figure 8: Dcompact architecture. The proposed Dcompact is based on MM-DiT [ 18 ] , a DiT [ 55 ] variant designed for multimodal input processing. The architecture consists of two parallel processing streams: one for compact latent tokens z and another for target latent tokens zψ , which are fused through self attention over the concatenated token sequence. For the overall depiction of CompACT, we refer to Fig. 2 in the main paper.
Figure 9: Inverse Dynamics Model (IDM) architecture. Consecutive frames are tokenized and processed through a transformer-based frame encoder, which produces a single conditioning vector via average pooling. This vector conditions an action denoiser implemented as a diffusion policy [ 11 ] , which predicts the action taken between the two frames.
hparams
Frame encoder
depth
4
dim
512
MLP dim
2048
heads
8
#params
13.5M
Table 11: Model architecture hyperparameters for IDM.
Figure 10: World model fϕ architecture for navigation. For navigation tasks, the world model is formulated as an action-conditioned generative model that autoregressively generates the next frame. The model follows the CDiT architecture proposed in NWM [ 3 ] , where actions are conditioned via adaptive layer normalization and history frames are conditioned via cross-attention.
Figure 11: World model fϕ architecture for manipulation. For manipulation tasks, the world model is formulated as a block-causal transformer that enables parallel prediction of multiple future frames conditioned on historical timesteps. Action tokens are used to condition the prediction head (AdaLN + linear layer) that unmasks latent tokens at each future timestep.
hparams
navigation
manipulation
depth
12
16
dim
768
1024
MLP dim
3072
4096
heads
12
16
# params
243M
270M
Table 12: Model architecture hyperparameters for world model.
Figure 12: Qualitative results of reconstruction with CompACT.
Figure 13: Attention visualization for compact latent tokens in latent resampler. Brighter the color, higher the attention score.
Figure 14: Plot for ATE, planning latency, and memory peak usage on RECON [ 60 ] . Latency and memory usage is mesaured for single trajectory optimization, using a single RTX 6000 ADA GPU.
Figure 15: Additional qualitative results of navigation planning with the proposed CompACT. Among candidates, worst two action sequences and best action sequences are presented together.
Figure 16: Additional qualitative results of action-conditioned video generation. Red and green boxes indicate incorrect and correct end-effector positions, respectively.
Token-based world models enable fine-grained latent planning, but repeatedly processing large spatial token grids makes action search expensive. We introduce COSTGRAD, a training-free, goal-conditioned selector that ranks spatial tokens by the gradient norm of the planning cost with respect to each input token. By deriving importance from the downstream control objective, COSTGRAD targets tokens that matter for planning rather than merely for prediction. On AdaLN-conditioned predictors at 50% sparsity, COSTGRAD matches or exceeds full-token planning on three of four continuous-control benchmarks, while giving a measured 2.6× wall-clock speedup per environment planning step. Combining token sparsity with reduced CEM search increases this to a ∼5× total speedup while still exceeding the full-token baseline. We also identify an architecture-dependent failure mode: in a matched AdaLN-vs-concat comparison, concat maintains comparable full-token performance but pure COSTGRAD loses its advantage over random selection. This difference tracks action-pathway drift: gradient-selected removal produces less drift than random removal on AdaLN, but more on concat. These results highlight selector-architecture compatibility as a design axis for sparse world-model planning. Project page and demos: https://ycxuyingchen.github.io/costgrad/
Visual world models enable robotic planning by predicting future observations, but dense latent-state propagation and sample-intensive trajectory optimization incur high inference latency and peak memory usage, limiting real-time deployment on resource-constrained platforms. Existing sparse world-model acceleration methods either rely on unguided token sparsification, which may discard planning-relevant information and restrict achievable sparsity, or introduce heavy auxiliary modules and cumbersome multi-stage training pipelines. In this work, we present Scope-WM, an efficient visual world model that scopes computation to prediction-relevant latent regions and promising action sequences. Scope-WM distills prediction relevance into a lightweight action-conditioned selector and applies full dynamics prediction only to a compact subset of selected tokens. It updates the remaining tokens using a compact summary of foreground states and their changes, allowing the background to perceive foreground dynamics without costly token-to-token interactions. During planning, Scope-WM preserves and reuses high-quality action sequences discovered during the initial MPC search, focusing subsequent search under reduced rollout budgets. The resulting pipeline requires only a one-off selector distillation followed by a single joint training stage for the sparse world model. On the challenging Push-T task, Scope-WM reduces peak GPU memory usage and planning time to 18.1% and 14.3% of those of dense DINO-WM, respectively, corresponding to a 6.97× planning speedup, while maintaining competitive task performance. Further evaluations across five diverse visual planning tasks demonstrate the general applicability of Scope-WM. Code is available at https://github.com/ChunZheng2022/Scope-WM.
Joint-Embedding Predictive Architectures (JEPAs), including recent LeWorldModel (LeWM), have become a promising foundation for reconstruction-free visual world models. For visual planning, however, LeWM evaluates candidate action sequences by repeatedly applying a local one-step latent transition model. This autoregressive rollout makes planning computationally expensive and exposes the predicted trajectory to accumulated latent errors as the horizon grows. We propose Fast LeWorldModel (Fast-LeWM), a fast latent world model that replaces repeated local rollout with action-prefix prediction. Given the current latent and a candidate action sequence, Fast-LeWM encodes its prefixes and predicts the future latents reached after executing those prefixes in parallel. By making action prefixes the basic prediction unit, Fast-LeWM directly models action effects accumulated to different extents over multiple horizons. This prefix-level supervision forces the model to learn how states continuously evolve under different action prefixes, rather than only fitting one-step state transitions. During planning, the predictor can use the prefix token from the encoded action sequence to evaluate the corresponding future latent without explicitly rolling through each intermediate imagined state. Across multiple tasks, Fast-LeWM improves average success over LeWM while substantially reducing planning time, achieving lower open-loop latent loss whose growth becomes significantly slower as the rollout horizon increases.