ALPS: Visual State Encoders for World Models

Yanqing Liu; Letian Zhang; Chengzhi Liu; Zijun Wang; Jieru Mei; Hongliang Fei; Yuyin Zhou; Cihang Xie

cs.CVOct 21, 2026

ALPS: Visual State Encoders for World Models

Authors: Yanqing Liu, Letian Zhang, Chengzhi Liu, Zijun Wang, Jieru Mei, Hongliang Fei, Yuyin Zhou, Cihang Xie

Organizations: University of California, Santa Cruz · University of California, Santa Barbara · Google

Abstract

World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action in world models.

Figures & tables

Figure 2: Two-stage visual state pretraining. Stage 1 trains an encoder $E$ from scratch with four objectives on shared patch tokens. Stage 2 initializes a student from $E$ and keeps $E$ frozen as the teacher. Given student tokens from an augmented crop $A$ and query tokens encoding position and relative scale, a predictor regresses standardized teacher tokens from a clean crop $B$ . Only patches of $B$ whose centers lie inside $A$ contribute to the loss. After training, the predictor is discarded and the refined student serves as ALPS.

Figure 3: The LIBERO state probe. (a) A lightweight probe reads frozen tokens from the last 16 frames to estimate the current scene state from observations alone (readout) or predict the state 1 or 3 s ahead given robot actions (prediction). (b) State predictions 3 s ahead from ALPS and SigLIP 2, overlaid on the frame observed 3 s later, shown on the held-out LIBERO-Spatial episode with the median error difference between the two encoders. (c) Per-object position errors at this step, the manipulated bowl’s error over the episode, and its mean error across 50 held-out episodes.

Figure 4: Reconstruction from frozen tokens. Following the input column, each column shows the evaluation decoder’s output from one encoder’s frozen tokens, labeled with its mean $\ell_{1}$ error across the 1,000 held-out ImageNet images. The rows show the first three images from a random sample ( Fig. S7 ). ALPS, OpenVision 3, MAE, and VC-1 retain sharp edges and textures. C-RADIOv3 and DINOv3 preserve spatial layout with softer detail, while SigLIP 2, PE-Core, and CLIP mainly recover coarse structure.

Figure 5: The same Stage 2 refinement applied to ALPS (Stage 1), SigLIP 2, and CLIP , each using its own frozen copy as the teacher. Lines connect each encoder before and after refinement, and the star marks ALPS. Language is evaluated through each encoder’s unchanged head. Higher values are better in every panel, with state-probe error relative to ALPS (Stage 1) inverted in (a). Endpoint labels show the value after Stage 2 and its change ( Tab. S20 ). All three improve state information, while only ALPS preserves language alignment.

	PushT	Wall	Granular	Rope	Reacher
Encoder	block (units)	agent (units)	Chamfer	Chamfer	joints (deg)
DINOv2-B/14	7.16	9.69	.246	.410	1.21
DINOv3-B/16	8.70	9.78	.222	.407	1.03
C-RADIOv3-B	5.64	9.75	.218	.372	1.06
SigLIP 2-B/16	11.96	9.91	.268	.469	2.02
ALPS (Stage 1)	9.32	9.81	.272	.424	1.42

Table 4: State read out from predicted tokens, 5 world-model steps ahead: Stage 2 reduces prediction error by 11–28% across four tasks. A probe trained on real frames reads the world-model predictions of Tab. 3 and Tab. S11 on held-out trajectories; lower is better, positions in workspace units. Bold: lowest of a column. Every step and two references: Tab. S16 ; protocol: Sec. S2 .

Figure 6: Rendered world-model predictions on PushT (window 834, largest true motion among 200 windows). The top row shows ground-truth frames; lower rows show each encoder’s evaluation-decoder outputs from real current tokens (green) and open-loop predicted tokens given recorded actions (blue). Mean errors: Tab. S15 ; full sequence: Fig. S4 .

Architecture
encoder	ViT-B/16, width 768, 12 layers, 12 heads, MLP 3,072
	fixed 2D sine-cosine position embedding, no final normalization
	one class token, used in attention but not in any loss
image embedding	mean-pooled patch tokens, linear projection to 768
text encoder	12 layers, width 768, 32 tokens, pooled at the last token
caption decoder	12 layers, width 768, prefix over image tokens, 80 text tokens

Table S1: Settings of Stage 1. Loss weights and objectives: Secs. 3.1 and S1 .

Views
crop frame	central square of the image, side $m$ ; square boxes of side $\sqrt{a}\,m$ in integer pixels, sampled uniformly
teacher crop $B$	area $a\sim\mathcal{U}(0.4,1)$ ; clean; resized to $224\times 224$ pixels (bilinear)
student crop $A$	local, $a\sim\mathcal{U}(0.1,0.4)$ , or global, $a\sim\mathcal{U}(0.4,1)$ , $p=0.5$ ; resized to $224\times 224$ pixels (bilinear)
	photometric augmentation; at least 16 patch centers of $B$ inside $A$ , else up to 10 resamples, then $A=B$
token grids	$14\times 14$ patches of 16 pixels for both crops; no flip
Predictor $P$

Table S2: Settings of Stage 2 for ALPS. Optimization and augmentation details: Sec. S1 .

source	share (%)	source	share (%)
SEED-Data-Edit [ 25 ]	17.5	PICA-100K [ 59 ]	1.8
ScaleEdit-12M [ 14 ]	14.9	NeIn [ 10 ]	1.2
UltraEdit [ 88 ]	10.1	Pico-Banana [ 60 ]	1.1
PIPE [ 73 ]	9.7	Dress-ED [ 45 ]	1.0
AnyEdit [ 83 ]	9.5	WEAVE [ 16 ]	0.7
OmniEdit [ 74 ]	8.4	GPT-Image-Edit [ 72 ]	0.3

Table S3: Sources of the Stage 2 training images and their share of the 5M sampled images.

	Frames rendered at 256 px					128 px
	read-				mean
Encoder	out	+1 s	+3 s	mean	(cm)	mean
TIPSv2-B/14	.875	.896	1.047	.939	1.25	.963
DINOv2-B/14	.863	.895	1.008	.922	1.22	.942
PE-Spatial-B/16	.986	.901	.923	.937	1.24	.936
DINOv3-B/16	.880	.901	.806	.862	1.15	.870

Table S6 : State probe on frames rendered at

256\times 256

resolution from the stored states of the episodes of Tab. 1 , with the same splits, targets, and probe. Relative error with ALPS (Stage 1) on the same frames as 1.000; last column: the 128-px mean of Tab. 1 . Bold: lowest ViT-B mean; gray : ViT-L.

		Validation	Relative error $\downarrow$
		loss	read-
Encoder	Block	(mean)	out	+1 s	+3 s	mean
SigLIP 2-B/16	6	.046	1.194	.970	1.277	1.147
	8	.052	1.165	1.106	1.249	1.173
	10	.056	1.260	1.169	1.368	1.266
	12	.059	1.149	1.202	1.130	1.160

Table S7: Intermediate layers on the LIBERO state probe. Relative error as in Tab. 1 for blocks 6, 8, and 10 and for the last block, 12, which Tab. 1 uses. Bold: the block with the lowest mean validation loss, a selection rule fixed before the runs; ALPS uses its last block.

Task	CLIP	SigLIP 2	DINOv2	DINOv3	C-RADIOv3	ALPS (Stage 1)	ALPS
LIBERO-Long
alphabet soup and tomato sauce in the basket	48	71	59	65	39	62	71
cream cheese box and butter in the basket	51	91	71	86	65	68	71
turn on the stove, moka pot on it	79	97	85	81	85	95	79
black bowl in the bottom drawer, close it	90	98	94	87	93	96	93
white mug on the left plate, yellow-white mug on the right plate	42	50	48	51	49	61	55

Table S9 : Success of every task in % for the policies of Tab. 2 with BERT, each value the mean of three evaluations (150 rollouts per task), tasks in benchmark order; the mean of a suite is the value of Tab. 2 . Bold: best of a row.

Figure S1 : Closed-loop rollouts of policies trained on frozen ALPS tokens across two tasks each from LIBERO-Spatial and LIBERO-Goal, shown with their language instructions. We display the first two tasks (in canonical task order) whose recorded rollout succeeded; overall, 15 of the 20 recorded rollouts across the two suites (one per task) succeeded.

Observation
cameras	agent view and eye-in-hand, $128\times 128$ , flipped upright
encoder input	bilinear resize to $224\times 224$ , normalization of the encoder
encoder precision	bfloat16 autocast, tokens cast to 32 bit
tokens per camera	196 patch tokens of the last block (DINOv2: 256)
pooling	per camera, one query from the task embedding, 4 heads, output 64
language	BERT-base (cased), pooled embedding, frozen

Table S10: Settings of policy learning ; within a suite, every encoder is trained and evaluated with matched settings.

Figure S2 : Planning on PushT with a world model over frozen tokens : start, goal, and final states for each encoder on the first four goals of Tab. 3 . Labels indicate executed simulator steps; dashed outlines mark the target block and agent poses. This visualization comes from a separate run of the same CEM planner; because action sampling is stochastic, DINOv2 misses goal 1 here despite succeeding in the main evaluation of Tab. 3 .

Figure S3 : Executed PushT rollouts for the four goals in Fig. S2 , comparing ALPS and DINOv2: six frames from the initial state to the final evaluated state, followed by the goal image (dashed outlines mark the target block and agent poses). DINOv2 misses goal 1 in this rollout ( Fig. S2 ).

	Chamfer $\downarrow$ , 50 instances
Encoder	mean	median
DINOv2-B/14	.701	.446
DINOv3-B/16	.681	.406
C-RADIOv3-B	.558	.289
SigLIP 2-B/16	.848	.511
ALPS (Stage 1)	.943	.514

Table S11: Planning on Rope with the planner of Tab. 3 . The first six rows use the world-model recipe of Tab. 3 ; the last row uses the predictor and the schedule of DINO-WM with its encoder, DINOv2-S/14, as a check of the pipeline.

Shared by all encoders and tasks
Input	the 196 tokens of one frame with their 768 visual channels, standardized as for the world model; real tokens in training, real or predicted tokens in evaluation; no proprioceptive or action channels and no history frames
Architecture	linear projection to width 256 with learned token embeddings; 8 learned queries; 2 cross-attention blocks with 8 heads and an MLP of width 1,024; layer normalization and a linear layer on the concatenated queries; 1.8M parameters, 3.4M for 256 points
Data	real tokens of 20,000 frames of the training data; 2,000 other frames of the training data for model selection; evaluation on held-out trajectories
Optimization	AdamW, learning rate $3\times 10^{-4}$ , weight decay 0.05, cosine decay to zero without warm-up, no gradient clipping, batch 128, 6,000 updates, seed 0; encoder frozen
Selection	lowest validation loss, computed every 500 updates; every selected checkpoint lies at 5,000 to 6,000 updates
Evaluation	probe frozen and in evaluation mode; one probe per encoder and task reads the real and predicted tokens of every world model of that encoder

Table S13 : State probe of Tabs. 4 and S16 : settings shared by all encoders and tasks, and per task, with the inputs that only the world model receives. MSE: mean squared error of targets standardized with training statistics; Chamfer: symmetric Chamfer distance between the predicted points and the particles of the true state.

Encoder	$\ell_{1}$	LPIPS
DINOv2-B/14	.0065	.0091
DINOv3-B/16	.0065	.0080
C-RADIOv3-B	.0062	.0074
SigLIP 2-B/16	.0066	.0086
ALPS (Stage 1)	.0055	.0069
ALPS	.0061	.0073

Table S14 : Decoding PushT frames from frozen tokens. We use the protocol of Tab. 1 and evaluate on 1,000 frames from validation trajectories. Lower is better; bold: best encoder in each column.

	token	decoded real	rendered prediction
	cosine	$k=5$	$k=1$	$k=5$
Encoder		LPIPS	LPIPS	$\ell_{1}$	LPIPS
SigLIP 2-B/16	1.000	.0081	.0107	.0150	.0308
C-RADIOv3-B	1.000	.0070	.0088	.0103	.0176
DINOv3-B/16	1.000	.0074	.0096	.0128	.0242
ALPS (Stage 1)	1.000	.0066	.0093	.0126	.0248

Table S15 : Rendered world-model predictions on PushT (200 windows; lower is better). Token cosine: world-model vs. decoder-training tokens (DINOv2: .80, not rendered).

PushT (block position, workspace units)
	predicted, world-model steps ahead					references, 5 steps ahead
Encoder	1	2	3	4	5	true tokens	last obs.
DINOv2-B/14	4.29	4.47	5.16	6.08	7.16	3.24	38.27
DINOv3-B/16	3.87	4.38	5.47	7.04	8.70	2.93	37.87
C-RADIOv3-B	3.51	3.66	4.21	4.85	5.64	2.66	38.14
SigLIP 2-B/16	6.17	6.87	8.21	9.97	11.96	3.52	38.02

Table S16: State readout from predicted tokens at every step. We evaluate the state probe of Tab. 4 on tokens predicted 1 to 5 world-model steps ahead, alongside two references scored against the true state at step 5: the true tokens at step 5 and the last observed tokens (lower is better).

	true future	one-step prediction,	five-step	rollout with recorded	last observation
Encoder	tokens	true history	open-loop rollout	proprioception	repeated
ALPS	3.1	3.6	6.6	6.7	42.0
ALPS (Stage 1)	3.8	4.3	9.9	9.2	42.3
C-RADIOv3-B	2.4	2.8	5.8	5.1	41.9
DINOv3-B/16	3.0	3.2	9.1	8.4	41.7
SigLIP 2-B/16	3.7	5.2	14.1	12.1	42.0

Table S17: PushT block error read from true, one-step, and rolled-out tokens 5 world-model steps ahead , in workspace units, lower is better, on the 200 windows of Tab. S15 with the world models of Tab. 3 . One-step: every step predicted from the true history. Last row: paired difference with a 95% bootstrap interval over the 21 trajectories. Bold: lowest of a column; conditions: Sec. S2 .

Figure S4 : Rendered world-model predictions across encoders on PushT window 834 , ranked first by our motion criterion ( Sec. S2 ). Each row applies an encoder’s evaluation decoder to its real tokens at the current frame ( $k=0$ ) and to tokens predicted $k=1$ to 5 world-model steps ahead; the top row shows ground-truth frames. Row annotations report 5-step LPIPS against the true frame alongside block position and angle errors read by the probe of Tab. 4 .

Figure S5 : Rendered world-model predictions across encoders on PushT window 138 , ranked second by our motion criterion; layout and metrics follow Fig. S4 .

Figure S6 : Rendered world-model predictions across encoders on PushT window 991 , ranked third by our motion criterion; layout and metrics follow Fig. S4 .

Figure S7 : Reconstruction from frozen tokens on a random sample of held-out ImageNet images : input images alongside outputs from the evaluation decoders of ten encoders, each labeled with its mean $\ell_{1}$ error over the 1,000 held-out images (the first three rows correspond to Fig. 4 ).

	Standard decoder		Larger decoder
Encoder	$\ell_{1}$	LPIPS	$\ell_{1}$	LPIPS
ALPS	.120	.372	.107	.302
C-RADIOv3-B	.161	.359	.141	.311
DINOv2-B/14	.248	.535	.218	.423
SigLIP 2-B/16	.264	.541	.220	.404

Table S18: Reconstruction with a stronger decoding and training configuration. Standard: the protocol of Tab. 1 ; larger: a wider and deeper decoder trained longer on more images; both on the same 1,000 held-out images. Every encoder improves; ALPS keeps the lowest

\ell_{1}

and overtakes C-RADIOv3 on LPIPS.

	State probe	Language alignment $\uparrow$
	LIBERO	IN-1k	Flickr	COCO
Encoder	err. $\downarrow$	0-shot	I $\to$ T	I $\to$ T
ALPS (Stage 1)	1.000	74.5	93.4	70.6
ALPS	.829	73.4	93.2	70.2
change	$-.171$	$-1.1$	$-0.2$	$-0.4$
SigLIP 2-B/16	1.160	77.6	91.7	67.9

Table S20 : Numerical results for Fig. 5 : state-probe error and language alignment for each encoder before and after Stage 2, and the change (after minus before), under the same budget of 5M sampled images.

	Patch matching (%) $\uparrow$
Stage 1 objectives	Stage 1	Stage 2
all four objectives	37.4	40.8
no reconstruction	34.6	38.6
no dense correspondence	31.4	38.2
no captioning	37.0	41.5
contrastive only	29.7	35.1

Table S21 : Cross-view patch matching for the ablation variants of Tab. 5 , evaluated after Stage 1 and after Stage 2. Accuracy measures the percentage of patches whose nearest token neighbor in a second view is the geometrically corresponding patch, averaged across four view-pair settings on 1,000 held-out ImageNet images.

World Model Learning

World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.

See all 414 papers

Token-World: World Modeling in Vision-Language Model Token Space for Robot Manipulation
Sep 30, 2026Chuyao Fu, Xiaowei Chi, Yuhan Rui +14World Models Action-Conditioned World Models
Southern University of Science and Technology · MUKA Robotics · Hong Kong University of Science and Technology +3
World Tokens: Enhancing Embodied Policies with Training-Time World Modeling
Aug 10, 2026Qu Tang, Benhui Zhuang, Bo Yuan +3Efficient VLA Model Inference Efficient VLA Models
JIUTIAN Research · Zhongguancun Academy
Pretrained Features as Teachers, Not States: Compact Latent World Models for Visual Planning
Oct 14, 2026Yanbo Wang, Nan Lin, Wendelin Böhmer +1World Model-Based Planning Latent World Models
Delft University of Technology (TU Delft)

Sep 30, 2026J/K move · Enter open · S save

Token-World: World Modeling in Vision-Language Model Token Space for Robot Manipulation

Chuyao Fu, Xiaowei Chi, Yuhan Rui, Yu-kai Wang+13

Southern University of Science and Technology · MUKA Robotics · Hong Kong University of Science and Technology+3

World Models Action-Conditioned World Models Robotic Manipulation VLMs for Robotics Embodied World Models World Models for VLA Models

Open paper AskPDF arXiv

A common approach to world-model simulation for vision-language-action (VLA) systems is to predict future RGB observations and then re-encode them into policy inputs, introducing an indirect interface between simulation and downstream policy execution. We instead investigate whether world dynamics can be modeled in a compact, policy-oriented state derived from VLM visual tokens. A key challenge is that raw VLM visual tokens are high-dimensional, making efficient and accurate autoregressive dynamics modeling challenging. To address this, we introduce Token-World, an action-conditioned world model that compresses VLM features into a compact token state, learns future dynamics in this reduced space, and maps predicted states back to the original policy-facing representation for downstream use. Across manipulation benchmarks, Token-World improves open-loop feature fidelity and policy-action consistency over recent world-model simulators, with slower degradation over long rollout horizons. In closed-loop evaluation, its simulated policy performance correlates more strongly with reference policy performance than Ctrl-World (

r=0.794

vs.\

0.583

), while requiring lower simulation latency. Ablations further show that compact-representation design and dimensionality substantially affect future-state prediction. Code will be available at https://chuyaofu.github.io/Token-World/.

Abstract

World Model Learning

Explore similar work

Token-World: World Modeling in Vision-Language Model Token Space for Robot Manipulation

World Tokens: Enhancing Embodied Policies with Training-Time World Modeling

Pretrained Features as Teachers, Not States: Compact Latent World Models for Visual Planning