World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action in world models.
Figures & tables
Figure 2: Two-stage visual state pretraining. Stage 1 trains an encoder E from scratch with four objectives on shared patch tokens. Stage 2 initializes a student from E and keeps E frozen as the teacher. Given student tokens from an augmented crop A and query tokens encoding position and relative scale, a predictor regresses standardized teacher tokens from a clean crop B . Only patches of B whose centers lie inside A contribute to the loss. After training, the predictor is discarded and the refined student serves as ALPS.
Figure 3: The LIBERO state probe. (a) A lightweight probe reads frozen tokens from the last 16 frames to estimate the current scene state from observations alone (readout) or predict the state 1 or 3 s ahead given robot actions (prediction). (b) State predictions 3 s ahead from ALPS and SigLIP 2, overlaid on the frame observed 3 s later, shown on the held-out LIBERO-Spatial episode with the median error difference between the two encoders. (c) Per-object position errors at this step, the manipulated bowl’s error over the episode, and its mean error across 50 held-out episodes.
Figure 4: Reconstruction from frozen tokens. Following the input column, each column shows the evaluation decoder’s output from one encoder’s frozen tokens, labeled with its mean ℓ1 error across the 1,000 held-out ImageNet images. The rows show the first three images from a random sample ( Fig. S7 ). ALPS, OpenVision 3, MAE, and VC-1 retain sharp edges and textures. C-RADIOv3 and DINOv3 preserve spatial layout with softer detail, while SigLIP 2, PE-Core, and CLIP mainly recover coarse structure.
Figure 5: The same Stage 2 refinement applied to ALPS (Stage 1), SigLIP 2, and CLIP , each using its own frozen copy as the teacher. Lines connect each encoder before and after refinement, and the star marks ALPS. Language is evaluated through each encoder’s unchanged head. Higher values are better in every panel, with state-probe error relative to ALPS (Stage 1) inverted in (a). Endpoint labels show the value after Stage 2 and its change ( Tab. S20 ). All three improve state information, while only ALPS preserves language alignment.
PushT
Wall
Granular
Rope
Reacher
Encoder
block (units)
agent (units)
Chamfer
Chamfer
joints (deg)
DINOv2-B/14
7.16
9.69
.246
.410
1.21
DINOv3-B/16
8.70
9.78
.222
.407
1.03
C-RADIOv3-B
5.64
9.75
.218
.372
1.06
SigLIP 2-B/16
11.96
9.91
.268
.469
2.02
ALPS (Stage 1)
9.32
9.81
.272
.424
1.42
Table 4: State read out from predicted tokens, 5 world-model steps ahead: Stage 2 reduces prediction error by 11–28% across four tasks. A probe trained on real frames reads the world-model predictions of Tab. 3 and Tab. S11 on held-out trajectories; lower is better, positions in workspace units. Bold: lowest of a column. Every step and two references: Tab. S16 ; protocol: Sec. S2 .
Figure 6: Rendered world-model predictions on PushT (window 834, largest true motion among 200 windows). The top row shows ground-truth frames; lower rows show each encoder’s evaluation-decoder outputs from real current tokens (green) and open-loop predicted tokens given recorded actions (blue). Mean errors: Tab. S15 ; full sequence: Fig. S4 .
fixed 2D sine-cosine position embedding, no final normalization
one class token, used in attention but not in any loss
image embedding
mean-pooled patch tokens, linear projection to 768
text encoder
12 layers, width 768, 32 tokens, pooled at the last token
caption decoder
12 layers, width 768, prefix over image tokens, 80 text tokens
Table S1: Settings of Stage 1. Loss weights and objectives: Secs. 3.1 and S1 .
Views
crop frame
central square of the image, side m ; square boxes of side am in integer pixels, sampled uniformly
teacher crop B
area a∼U(0.4,1) ; clean; resized to 224×224 pixels (bilinear)
student crop A
local, a∼U(0.1,0.4) , or global, a∼U(0.4,1) , p=0.5 ; resized to 224×224 pixels (bilinear)
photometric augmentation; at least 16 patch centers of B inside A , else up to 10 resamples, then A=B
token grids
14×14 patches of 16 pixels for both crops; no flip
Predictor P
Table S2: Settings of Stage 2 for ALPS. Optimization and augmentation details: Sec. S1 .
source
share (%)
source
share (%)
SEED-Data-Edit [ 25 ]
17.5
PICA-100K [ 59 ]
1.8
ScaleEdit-12M [ 14 ]
14.9
NeIn [ 10 ]
1.2
UltraEdit [ 88 ]
10.1
Pico-Banana [ 60 ]
1.1
PIPE [ 73 ]
9.7
Dress-ED [ 45 ]
1.0
AnyEdit [ 83 ]
9.5
WEAVE [ 16 ]
0.7
OmniEdit [ 74 ]
8.4
GPT-Image-Edit [ 72 ]
0.3
Table S3: Sources of the Stage 2 training images and their share of the 5M sampled images.
Frames rendered at 256 px
128 px
read-
mean
Encoder
out
+1 s
+3 s
mean
(cm)
mean
TIPSv2-B/14
.875
.896
1.047
.939
1.25
.963
DINOv2-B/14
.863
.895
1.008
.922
1.22
.942
PE-Spatial-B/16
.986
.901
.923
.937
1.24
.936
DINOv3-B/16
.880
.901
.806
.862
1.15
.870
Table S6 : State probe on frames rendered at 256×256 resolution from the stored states of the episodes of Tab. 1 , with the same splits, targets, and probe. Relative error with ALPS (Stage 1) on the same frames as 1.000; last column: the 128-px mean of Tab. 1 . Bold: lowest ViT-B mean; gray : ViT-L.
Validation
Relative error ↓
loss
read-
Encoder
Block
(mean)
out
+1 s
+3 s
mean
SigLIP 2-B/16
6
.046
1.194
.970
1.277
1.147
8
.052
1.165
1.106
1.249
1.173
10
.056
1.260
1.169
1.368
1.266
12
.059
1.149
1.202
1.130
1.160
Table S7: Intermediate layers on the LIBERO state probe. Relative error as in Tab. 1 for blocks 6, 8, and 10 and for the last block, 12, which Tab. 1 uses. Bold: the block with the lowest mean validation loss, a selection rule fixed before the runs; ALPS uses its last block.
Task
CLIP
SigLIP 2
DINOv2
DINOv3
C-RADIOv3
ALPS (Stage 1)
ALPS
LIBERO-Long
alphabet soup and tomato sauce in the basket
48
71
59
65
39
62
71
cream cheese box and butter in the basket
51
91
71
86
65
68
71
turn on the stove, moka pot on it
79
97
85
81
85
95
79
black bowl in the bottom drawer, close it
90
98
94
87
93
96
93
white mug on the left plate, yellow-white mug on the right plate
42
50
48
51
49
61
55
Table S9 : Success of every task in % for the policies of Tab. 2 with BERT, each value the mean of three evaluations (150 rollouts per task), tasks in benchmark order; the mean of a suite is the value of Tab. 2 . Bold: best of a row.
Figure S1 : Closed-loop rollouts of policies trained on frozen ALPS tokens across two tasks each from LIBERO-Spatial and LIBERO-Goal, shown with their language instructions. We display the first two tasks (in canonical task order) whose recorded rollout succeeded; overall, 15 of the 20 recorded rollouts across the two suites (one per task) succeeded.
Observation
cameras
agent view and eye-in-hand, 128×128 , flipped upright
encoder input
bilinear resize to 224×224 , normalization of the encoder
encoder precision
bfloat16 autocast, tokens cast to 32 bit
tokens per camera
196 patch tokens of the last block (DINOv2: 256)
pooling
per camera, one query from the task embedding, 4 heads, output 64
language
BERT-base (cased), pooled embedding, frozen
Table S10: Settings of policy learning ; within a suite, every encoder is trained and evaluated with matched settings.
Figure S2 : Planning on PushT with a world model over frozen tokens : start, goal, and final states for each encoder on the first four goals of Tab. 3 . Labels indicate executed simulator steps; dashed outlines mark the target block and agent poses. This visualization comes from a separate run of the same CEM planner; because action sampling is stochastic, DINOv2 misses goal 1 here despite succeeding in the main evaluation of Tab. 3 .
Figure S3 : Executed PushT rollouts for the four goals in Fig. S2 , comparing ALPS and DINOv2: six frames from the initial state to the final evaluated state, followed by the goal image (dashed outlines mark the target block and agent poses). DINOv2 misses goal 1 in this rollout ( Fig. S2 ).
Chamfer ↓ , 50 instances
Encoder
mean
median
DINOv2-B/14
.701
.446
DINOv3-B/16
.681
.406
C-RADIOv3-B
.558
.289
SigLIP 2-B/16
.848
.511
ALPS (Stage 1)
.943
.514
Table S11: Planning on Rope with the planner of Tab. 3 . The first six rows use the world-model recipe of Tab. 3 ; the last row uses the predictor and the schedule of DINO-WM with its encoder, DINOv2-S/14, as a check of the pipeline.
Shared by all encoders and tasks
Input
the 196 tokens of one frame with their 768 visual channels, standardized as for the world model; real tokens in training, real or predicted tokens in evaluation; no proprioceptive or action channels and no history frames
Architecture
linear projection to width 256 with learned token embeddings; 8 learned queries; 2 cross-attention blocks with 8 heads and an MLP of width 1,024; layer normalization and a linear layer on the concatenated queries; 1.8M parameters, 3.4M for 256 points
Data
real tokens of 20,000 frames of the training data; 2,000 other frames of the training data for model selection; evaluation on held-out trajectories
Optimization
AdamW, learning rate 3×10−4 , weight decay 0.05, cosine decay to zero without warm-up, no gradient clipping, batch 128, 6,000 updates, seed 0; encoder frozen
Selection
lowest validation loss, computed every 500 updates; every selected checkpoint lies at 5,000 to 6,000 updates
Evaluation
probe frozen and in evaluation mode; one probe per encoder and task reads the real and predicted tokens of every world model of that encoder
Table S13 : State probe of Tabs. 4 and S16 : settings shared by all encoders and tasks, and per task, with the inputs that only the world model receives. MSE: mean squared error of targets standardized with training statistics; Chamfer: symmetric Chamfer distance between the predicted points and the particles of the true state.
Encoder
ℓ1
LPIPS
DINOv2-B/14
.0065
.0091
DINOv3-B/16
.0065
.0080
C-RADIOv3-B
.0062
.0074
SigLIP 2-B/16
.0066
.0086
ALPS (Stage 1)
.0055
.0069
ALPS
.0061
.0073
Table S14 : Decoding PushT frames from frozen tokens. We use the protocol of Tab. 1 and evaluate on 1,000 frames from validation trajectories. Lower is better; bold: best encoder in each column.
token
decoded real
rendered prediction
cosine
k=5
k=1
k=5
Encoder
LPIPS
LPIPS
ℓ1
LPIPS
SigLIP 2-B/16
1.000
.0081
.0107
.0150
.0308
C-RADIOv3-B
1.000
.0070
.0088
.0103
.0176
DINOv3-B/16
1.000
.0074
.0096
.0128
.0242
ALPS (Stage 1)
1.000
.0066
.0093
.0126
.0248
Table S15 : Rendered world-model predictions on PushT (200 windows; lower is better). Token cosine: world-model vs. decoder-training tokens (DINOv2: .80, not rendered).
predicted, world-model steps ahead
references, 5 steps ahead
Encoder
1
2
3
4
5
true tokens
last obs.
PushT (block position, workspace units)
DINOv2-B/14
4.29
4.47
5.16
6.08
7.16
3.24
38.27
DINOv3-B/16
3.87
4.38
5.47
7.04
8.70
2.93
37.87
C-RADIOv3-B
3.51
3.66
4.21
4.85
5.64
2.66
38.14
SigLIP 2-B/16
6.17
6.87
8.21
9.97
11.96
3.52
38.02
Table S16: State readout from predicted tokens at every step. We evaluate the state probe of Tab. 4 on tokens predicted 1 to 5 world-model steps ahead, alongside two references scored against the true state at step 5: the true tokens at step 5 and the last observed tokens (lower is better).
true future
one-step prediction,
five-step
rollout with recorded
last observation
Encoder
tokens
true history
open-loop rollout
proprioception
repeated
ALPS
3.1
3.6
6.6
6.7
42.0
ALPS (Stage 1)
3.8
4.3
9.9
9.2
42.3
C-RADIOv3-B
2.4
2.8
5.8
5.1
41.9
DINOv3-B/16
3.0
3.2
9.1
8.4
41.7
SigLIP 2-B/16
3.7
5.2
14.1
12.1
42.0
Table S17: PushT block error read from true, one-step, and rolled-out tokens 5 world-model steps ahead , in workspace units, lower is better, on the 200 windows of Tab. S15 with the world models of Tab. 3 . One-step: every step predicted from the true history. Last row: paired difference with a 95% bootstrap interval over the 21 trajectories. Bold: lowest of a column; conditions: Sec. S2 .
Figure S4 : Rendered world-model predictions across encoders on PushT window 834 , ranked first by our motion criterion ( Sec. S2 ). Each row applies an encoder’s evaluation decoder to its real tokens at the current frame ( k=0 ) and to tokens predicted k=1 to 5 world-model steps ahead; the top row shows ground-truth frames. Row annotations report 5-step LPIPS against the true frame alongside block position and angle errors read by the probe of Tab. 4 .
Figure S5 : Rendered world-model predictions across encoders on PushT window 138 , ranked second by our motion criterion; layout and metrics follow Fig. S4 .
Figure S6 : Rendered world-model predictions across encoders on PushT window 991 , ranked third by our motion criterion; layout and metrics follow Fig. S4 .
Figure S7 : Reconstruction from frozen tokens on a random sample of held-out ImageNet images : input images alongside outputs from the evaluation decoders of ten encoders, each labeled with its mean ℓ1 error over the 1,000 held-out images (the first three rows correspond to Fig. 4 ).
Standard decoder
Larger decoder
Encoder
ℓ1
LPIPS
ℓ1
LPIPS
ALPS
.120
.372
.107
.302
C-RADIOv3-B
.161
.359
.141
.311
DINOv2-B/14
.248
.535
.218
.423
SigLIP 2-B/16
.264
.541
.220
.404
Table S18: Reconstruction with a stronger decoding and training configuration. Standard: the protocol of Tab. 1 ; larger: a wider and deeper decoder trained longer on more images; both on the same 1,000 held-out images. Every encoder improves; ALPS keeps the lowest ℓ1 and overtakes C-RADIOv3 on LPIPS.
State probe
Language alignment ↑
LIBERO
IN-1k
Flickr
COCO
Encoder
err. ↓
0-shot
I → T
I → T
ALPS (Stage 1)
1.000
74.5
93.4
70.6
ALPS
.829
73.4
93.2
70.2
change
−.171
−1.1
−0.2
−0.4
SigLIP 2-B/16
1.160
77.6
91.7
67.9
Table S20 : Numerical results for Fig. 5 : state-probe error and language alignment for each encoder before and after Stage 2, and the change (after minus before), under the same budget of 5M sampled images.
Patch matching (%) ↑
Stage 1 objectives
Stage 1
Stage 2
all four objectives
37.4
40.8
no reconstruction
34.6
38.6
no dense correspondence
31.4
38.2
no captioning
37.0
41.5
contrastive only
29.7
35.1
Table S21 : Cross-view patch matching for the ablation variants of Tab. 5 , evaluated after Stage 1 and after Stage 2. Accuracy measures the percentage of patches whose nearest token neighbor in a second view is the geometrically corresponding patch, averaged across four view-pair settings on 1,000 held-out ImageNet images.
A common approach to world-model simulation for vision-language-action (VLA) systems is to predict future RGB observations and then re-encode them into policy inputs, introducing an indirect interface between simulation and downstream policy execution. We instead investigate whether world dynamics can be modeled in a compact, policy-oriented state derived from VLM visual tokens. A key challenge is that raw VLM visual tokens are high-dimensional, making efficient and accurate autoregressive dynamics modeling challenging. To address this, we introduce Token-World, an action-conditioned world model that compresses VLM features into a compact token state, learns future dynamics in this reduced space, and maps predicted states back to the original policy-facing representation for downstream use. Across manipulation benchmarks, Token-World improves open-loop feature fidelity and policy-action consistency over recent world-model simulators, with slower degradation over long rollout horizons. In closed-loop evaluation, its simulated policy performance correlates more strongly with reference policy performance than Ctrl-World (r=0.794 vs.\ 0.583), while requiring lower simulation latency. Ablations further show that compact-representation design and dimensionality substantially affect future-state prediction. Code will be available at https://chuyaofu.github.io/Token-World/.
Chuyao Fu, Xiaowei Chi, Yuhan Rui +14
Southern University of Science and Technology · MUKA Robotics · Hong Kong University of Science and Technology +3
Vision-language-action (VLA) models are a widely adopted paradigm for embodied policies. They excel at efficient closed-loop control but do not explicitly model how physical scenes evolve as a task unfolds. Recently emerging world-action models (WAMs) leverage pretrained video world models to capture spatiotemporal evolution, yet retaining future generation or a large video backbone in the control loop substantially increases inference cost. We introduce World Tokens, an embodied policy architecture built around a World Adapter that bridges visual-language understanding, world-dynamics modeling, and action generation. It uses world modeling during training to enhance the action policy while preserving efficient deployment. Specifically, the World Adapter transforms VLM features into a fixed set of world tokens, which condition a jointly fine-tuned future-video denoiser and simultaneously serve as the action expert's sole visual-language context. This shared conditioning allows gradients from future-video denoising to directly shape the representation used for action prediction, while exclusive routing prevents the policy from bypassing that representation. At deployment, the world-model branch is removed, leaving only the VLM, World Adapter, and action expert, with no online video-model inference. With a 2B backbone and no embodied action pretraining, World Tokens is highly competitive on LIBERO, attains the best reported averages on SIMPLER, substantially improves real-world R1 Pro success over a matched action-only baseline, and generates each action chunk at VLA-level latency.
A key design choice in visual world models is what information the latent state is encouraged to preserve. Models that learn a latent space with only a latent-prediction objective adapt to environment dynamics, but the predictive objective alone admits collapsed solutions and provides no explicit constraint on the visual information retained by the latent. Models built directly on pretrained visual features inherit a strong visual prior, but tie prediction and planning to a frozen state space. We propose Anchor-WM, which instead uses pretrained visual features as a teacher for learning a compact latent state rather than as the world-model state itself. Anchor-WM learns this compact latent with action-conditioned dynamics, while distilling the dense DINO representations through a training-only distillation head. This encourages the compact latent to retain visual information captured by the pretrained representation while discouraging representational collapse. At inference, the DINO encoder and distillation head are discarded, allowing prediction and planning to operate entirely in the learned latent space. Anchor-WM achieves the best average success rate across the four evaluated environments while enabling substantially faster planning than world models operating directly on DINO patch features. These results show that pretrained visual features can effectively guide world-model learning without defining the state space used for prediction and planning.