Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Model

Dongwon Kim; Gawon Seo; Jinsung Lee; Minsu Cho; Suha Kwak

Figure 1: Overview of the proposed latent world model formulation (Sec. 3.1 ). (a) An image tokenizer is first trained with a reconstruction objective to map an input image into compact latent tokens

{\bm{z}}

. (Fig. 2 and Sec. 3.2 ). (b) Using the learned tokenizer, latent world model

f_{\phi}({\bm{z}}_{t},{\bm{a}}_{t})

is trained to model the conditional distribution of the future state

p_{\phi}({\bm{z}}_{t+1}|{\bm{z}}_{t},{\bm{a}}_{t})

, where we adopt masked generative modeling (Sec. 3.3 ). (c) At test time, the learned latent world model is used for decision-time planning : An optimization procedure (e.g., MPC with CEM) searches over actions

{\bm{a}}_{0:H-1}

to minimize the distance between the predicted final state and a goal image.

Figure 2: A tokenizer architecture detail. During training, only the latent resampler and

\mathcal{D}_{\textrm{compact}}

are updated.

\mathcal{E}_{\psi}

produces masked target tokens (training only), while

\mathcal{D}_{\psi}

is used only during inference for pixel level reconstruction.

Model	Type	#tok	rFID $\downarrow$	IS $\uparrow$
SD-VAE [ 57 ]	cont.	1024	0.64	223.8
MaskGIT-VQGAN [ 8 ]	disc.	256	1.83	186.7
TA-TiTok-VQ [ 39 ]	disc.	32	3.95	219.6
TA-TiTok-KL [ 39 ]	cont.	32	1.93	222.0
FlexTok [ 2 ]	disc.	1-256	5.60 †	114.9
CompACT	disc.	16	2.40	209.0

Table 1: Reconstruction performance of CompACT on ImageNet validation split. Metrics are computed using open-sourced checkpoints. rFID is measured using clean-fid [ 54 ] .

\dagger

: Measured using 16 tokens.

Configuration		rFID $\downarrow$	#tok
Target tokenizer ( $\mathcal{D}_{\psi}\circ\mathcal{E}_{\psi}$ ) [ 8 ]		1.83	256
DINOv3-B (frozen) + latent resampler		2.40	16
$\mathcal{D}_{\textrm{compact}}$	w/o generative decoding	28.80	16
$\mathcal{E}_{\textrm{compact}}$	ViT-B (trained from scratch) + [REG]	7.28	16
	DINOv3-B (finetuned) + [REG]	4.51	16
	DINOv3-B (finetuned) + latent resampler	5.22	16

Table 2: Ablation on CompACT tokenizer. rFID is measured on ImageNet [ 15 ] validation split using clean-fid [ 54 ] .

Figure 3: CompACT encoder

\mathcal{E}_{\textrm{compact}}

architecture variation. (a) ViT (scratch)+ [ REG ]: Initial latent tokens are concatenated to the input patch tokens. This design follows previous transformer-based image tokenizers [ 77 , 2 , 75 ] . (b) DINOv3 [ 61 ] + [ REG ]: Similar to (a), but encoder is initialized with Dinov3. (c) DINOv3 [ 61 ] + latent resampler: latent resampler and Dinov3 initialized encoder. Dino and ViT are updated during training in these variants.

Figure 4: Attention visualization for compact latent token in latent resampler. Brighter the color, higher the attention score.

Tokenizer	#tok	L1 err $\downarrow$	$R^{2}$ $\uparrow$
Target tokenizer ( $\mathcal{D}_{\psi}\circ\mathcal{E}_{\psi}$ ) [ 8 ]	256	0.093	0.684
CompACT	16	0.091	0.716

Table 3: Performance of Inverse Dynamics Model (IDM) trained with different tokenizers on RoboNet [ 13 ] . L1 error and

R^{2}

are measured between ground truth and predicted end effector position.

Tokenizer	#tok	RECON		SCAND		Latency $\downarrow$
Tokenizer	#tok	ATE $\downarrow$	RPE $\downarrow$	ATE $\downarrow$	RPE $\downarrow$	Latency $\downarrow$
SD-VAE [ 57 ]	784	1.262	0.354	1.065	0.291	178.78
FlexTok [ 2 ]	64	1.484	0.400	1.578	0.378	16.68
FlexTok [ 2 ]	16	1.625	0.446	1.503	0.362	14.48
CompACT	16	1.330	0.390	1.358	0.336	5.78
CompACT	8	1.373	0.401	1.391	0.346	4.83

Table 4: Planning performance of NWM on RECON benchmark with different tokenizers. Latency (sec) represents single trajectory optimization time using a single RTX 6000 ADA GPU.

Figure 5: Qualitative results of planning with the proposed CompACT. Best and Worst 1&2 denote the final rollouts corresponding to the simulated trajectories with the minimum and maximum cost, respectively.

Hist. mask	ATE
✓	1.330
	1.480

Table 5: Effect of design choices in terms of planning accuracy on RECON. ( Left ): Effect of the history masking in the world model

f_{\theta}

(Sec. 3.3 ). ( Middle ): Comparison between the different cost function (Sec. 3.1 ). ( Right ): Effect of the freezing vision encoder during tokenizer training (Sec 3.2.1 ).

Model	#tok	APE	Latency (sec)
Target tokenizer ( $\mathcal{D}_{\psi}\circ\mathcal{E}_{\psi}$ ) [ 8 ]	256	0.3383	3.826
CompACT	16	0.1122	0.740

Table 6: Action-conditioned video prediction results on RoboNet [ 13 ] . Action prediction error is measured using IDM reported in Tab 3 . Latency is measure for when generating next 14 frames on a single RTX 6000 ADA GPU.

Figure 6: Qualitative results of action-conditioned video generation. Red and green boxes indicate incorrect and correct end-effector positions, respectively.

Backbone	SigLIP-2	MAE	DINOv3
rFID $\downarrow$	2.09	3.43	2.40

Table 7: Cross-backbone ablation of CompACT. Reconstruction performance of CompACT with 16 tokens on ImageNet validation split across different backbone choices.

Figure 7: Planning latency breakdown. Comparison of rollout and decoding latency between SD-VAE and CompACT on a single RTX 6000 ADA GPU. Note the log scale on the

x

-axis.

Tokenizer	SR $\uparrow$	Steps $\downarrow$
Target tokenizer ( $\mathcal{D}_{\psi}\circ\mathcal{E}_{\psi}$ ) [ 8 ]	56%	66.8
CompACT (ours)	56%	55.1

Table 8: Closed-loop manipulation on RoboMimic Lift.

hparams	CompACT
optimizer	AdamW
$\beta_{1}$	0.9
$\beta_{2}$	0.999
weight decay	0.01
lr	0.0001
lr scheduling	cosine

Table 9: Training hyperparameters for CompACT.

hparams	Latent resampler	$\mathcal{D}_{\text{compact}}$
depth	5	16
dim	768	1024
MLP dim	3072	4096
heads	8	8

Table 10: Model architecture hyperparameters for CompACT.

Figure 8:

\mathcal{D}_{\text{compact}}

architecture. The proposed

\mathcal{D}_{\text{compact}}

is based on MM-DiT [ 18 ] , a DiT [ 55 ] variant designed for multimodal input processing. The architecture consists of two parallel processing streams: one for compact latent tokens

{\bm{z}}

and another for target latent tokens

{\bm{z}}^{\psi}

, which are fused through self attention over the concatenated token sequence. For the overall depiction of CompACT, we refer to Fig. 2 in the main paper.

Figure 9: Inverse Dynamics Model (IDM) architecture. Consecutive frames are tokenized and processed through a transformer-based frame encoder, which produces a single conditioning vector via average pooling. This vector conditions an action denoiser implemented as a diffusion policy [ 11 ] , which predicts the action taken between the two frames.

hparams	Frame encoder
depth	4
dim	512
MLP dim	2048
heads	8
#params	13.5M

Table 11: Model architecture hyperparameters for IDM.

Figure 10: World model

f_{\phi}

architecture for navigation. For navigation tasks, the world model is formulated as an action-conditioned generative model that autoregressively generates the next frame. The model follows the CDiT architecture proposed in NWM [ 3 ] , where actions are conditioned via adaptive layer normalization and history frames are conditioned via cross-attention.

Figure 11: World model

f_{\phi}

architecture for manipulation. For manipulation tasks, the world model is formulated as a block-causal transformer that enables parallel prediction of multiple future frames conditioned on historical timesteps. Action tokens are used to condition the prediction head (AdaLN + linear layer) that unmasks latent tokens at each future timestep.

hparams	navigation	manipulation
depth	12	16
dim	768	1024
MLP dim	3072	4096
heads	12	16
# params	243M	270M

Table 12: Model architecture hyperparameters for world model.

Figure 12: Qualitative results of reconstruction with CompACT.

Figure 13: Attention visualization for compact latent tokens in latent resampler. Brighter the color, higher the attention score.

Figure 14: Plot for ATE, planning latency, and memory peak usage on RECON [ 60 ] . Latency and memory usage is mesaured for single trajectory optimization, using a single RTX 6000 ADA GPU.

Figure 15: Additional qualitative results of navigation planning with the proposed CompACT. Among candidates, worst two action sequences and best action sequences are presented together.

Figure 16: Additional qualitative results of action-conditioned video generation. Red and green boxes indicate incorrect and correct end-effector positions, respectively.

Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Model

Abstract

Figures & tables

World Model-Based Planning

Sparse Planning in Visual World Models via Cost Gradients

Scope-WM: Scoped Computation for Efficient Visual World Models

Fast LeWorldModel

Abstract

World Model-Based Planning

Explore similar work

Sparse Planning in Visual World Models via Cost Gradients

Scope-WM: Scoped Computation for Efficient Visual World Models

Fast LeWorldModel