Chain of World: World Model Thinking in Latent Motion

Fuxiang Yang; Donglin Di; Lulu Tang; Xuancheng Zhang; Lei Fan; Hao Li; Chen Wei; Tonghua Su

cs.CVMar 3, 2026

Chain of World: World Model Thinking in Latent Motion

Authors: Fuxiang Yang, Donglin Di, Lulu Tang, Xuancheng Zhang, Lei Fan, Hao Li, Chen Wei, Tonghua Su, +1 more

Organizations: Harbin Institute of Technology · Li Auto · Beijing Academy of Artificial Intelligence (BAAI) · Peking University · University of New South Wales · Chongqing Research Institute of HIT

Abstract

Vision-Language-Action (VLA) models are a promising path toward embodied intelligence, yet they often overlook the predictive and temporal-causal structure underlying visual dynamics. World-model VLAs address this by predicting future frames, but waste capacity reconstructing redundant backgrounds. Latent-action VLAs encode frame-to-frame transitions compactly, but lack temporally continuous dynamic modeling and world knowledge. To overcome these limitations, we introduce CoWVLA (Chain-of-World VLA), a new "Chain of World" paradigm that unifies world-model temporal reasoning with a disentangled latent motion representation. First, a pretrained video VAE serves as a latent motion extractor, explicitly factorizing video segments into structure and motion latents. Then, during pre-training, the VLA learns from an instruction and an initial frame to infer a continuous latent motion chain and predict the segment's terminal frame. Finally, during co-fine-tuning, this latent dynamic is aligned with discrete action prediction by jointly modeling sparse keyframes and action sequences in a unified autoregressive decoder. This design preserves the world-model benefits of temporal reasoning and world knowledge while retaining the compactness and interpretability of latent actions, enabling efficient visuomotor learning. Extensive experiments on robotic simulation benchmarks show that CoWVLA outperforms existing world-model and latent-action approaches and achieves moderate computational efficiency, highlighting its potential as a more effective VLA pretraining paradigm. The project website can be found at https://fx-hit.github.io/cowvla-io.

Figures & tables

Figure 1: Comparison of VLA pretraining strategies. (a) World Model : It predicts future visual frames, leading to redundant background reconstruction. (b) Latent Action : It learns the frame-to-frame transition using a visual encoder $E$ , but lacks temporally continuous reasoning. (c) CoWVLA : Our method first uses a video encoder $E$ to decompose each video segment into motion and structure latents, and then trains the VLM to infer latent motion and predict the terminal frame of the segment given the instruction and the initial frame.

Figure 2: Overview of the CoWVLA framework. CoWVLA consists of two core components: a latent motion extractor and a VLA decoder. The latent motion extractor, implemented as a video VAE, disentangles each video segment into a structure latent $z_{s}$ and two directional motion latents $z_{m}^{h}$ and $z_{m}^{w}$ , which are concatenated into a unified latent motion vector $z_{m}$ . The VLA decoder performs unified autoregressive modeling over multimodal sequences. During pre-training, the model takes the instruction and initial frame as input, and uses a learnable motion query $Q$ to predict the latent motion $\hat{z}_{m}$ while reconstructing the terminal frame of the video segment. During co-fine-tuning, the input expands into alternating keyframe–action pairs; $Q$ continues to aggregate temporally continuous latent dynamics, guiding multi-step action generation under sparse visual observations.

Model	LIBERO					SimplerEnv-WidowX
Model	SPATIAL	OBJECT	GOAL	LONG	Avg.	Stack Block	Put Carrot	Put Spoon	Put Eggplant	Avg.
OpenVLA [ 24 ]	0.849	0.884	0.792	0.537	0.765	0.000	0.000	0.000	0.041	0.010
SpatialVLA [ 36 ]	0.882	0.899	0.786	0.555	0.781	0.292	0.250	0.167	1.000	0.427
CogACT [ 28 ]	0.960	0.874	0.868	0.846	0.887	0.150	0.508	0.717	0.675	0.513
Dita [ 21 ]	0.842	0.963	0.854	0.638	0.824	–	–	–	–	–
$\pi_{0}$ [ 3 ]	0.968	0.988	0.958	0.852	0.942	0.167	0.000	0.291	0.625	0.401

Table 1: Comparison of different methods on the LIBERO [ 32 ] and SimplerEnv-WidowX [ 30 ] benchmarks. The best and the second-best values for each metric are bold and underlined respectively.

Model	Reconstruction Metrics			Simulation Evaluation
Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Stack Block	Put Carrot	Put Spoon	Put Eggplant	Average
Pretrain	32.7	0.923	0.122	0.458	0.750	0.792	0.917	0.729
Finetune	33.4	0.934	0.123	0.625	0.667	0.792	0.958	0.760

Table 2: Evaluation of VAE-Reconstructed Videos and downstream fine-tuning performance on SimplerEnv-WidowX [ 30 ] .

Figure 3: Visualization of the disentangled motion and structure latents. We select two frames ( $t_{1}$ and $t_{2}$ ) and show the original (Orig.) and reconstructed (Recon.) frames. “M. Recon.” and “S. Recon.” denote the reconstructions obtained by decoding only the motion latent or only the structure latent, respectively. The structure latent preserves the global scene layout, whereas the motion latent captures motion and fine-grained temporal details.

Figure 4: Cross-reconstruction visualization. We extract the structure latent from the static video in the first row and the motion latent from the robot-arm motion video in the second row. By combining the two latents, we reconstruct the video shown in the third row. We compute the difference between the cross-reconstructed frames and the static frames to highlight the changed regions, which correspond to the robot arm’s motion.

Figure 5: Comparative visualization of future-frame prediction strategies. There are two tasks demonstrated: i) pick up the black bowl from the table center and place it on the plate, and ii) sweep into a pile. (a) The world-model approach predicts five future frames. (b) The single-goal-frame approach predicts one goal frame. (c) Our method reasons through a learned motion latent $z_{m}$ , producing more reasonable and instruction-aligned frames.

Config	Variant	Spatial	Object	Goal	Long	Average
Latent Action	w/o LA	0.622	0.146	0.694	0.328	0.448
	LAPA style	0.718	0.852	0.804	0.488	0.716
	villa-X style	0.840	0.904	0.834	0.668	0.812
	structure latent	0.856	0.898	0.822	0.692	0.817
	motion latent	0.916	0.932	0.886	0.774	0.877
World Model	UniVLA Style	0.958	0.978	0.932	0.898	0.942

Table 3: Ablation study on the LIBERO [ 32 ] benchmark.

$\lambda_{1}$	$\lambda_{2}$	Spatial	Object	Goal	Long	Average
0.0	0.0	0.922	0.962	0.862	0.742	0.872
0.1	0.0	0.960	0.980	0.922	0.882	0.936
1.0	0.0	0.958	0.970	0.950	0.902	0.945
0.1	0.05	0.954	0.972	0.944	0.914	0.946
0.1	0.01	0.970	0.964	0.958	0.926	0.955
1.0	0.01	0.970	0.956	0.934	0.922	0.946

Table 4: Ablation study of loss weights on the LIBERO [ 32 ] benchmark.

Figure 6: Comparison of pre-training efficiency and task performance on LIBERO [ 32 ] across different methods. Blue and orange circles denote world-model and latent-action baselines, respectively, while green circles denote our configurations. Circle size indicates training-time GPU memory usage. Our method balances pre-training efficiency and performance, achieving a higher success rate with moderate computational efficiency.

Dataset Name	Count
Berkeley Autolab Ur5 [ 10 ]	892
Bridgev2 [ 43 ]	24879
Cmu Play Fusion [ 9 ]	576
Fractal [ 4 ]	65530
Kuka [ 23 ]	84202
Maniskill [ 20 ]	30029

Table 1: Training datasets.

Method	Task	Tasks Completed in a Row					Avg. Len $\uparrow$
Method	Task	1	2	3	4	5	Avg. Len $\uparrow$
UniVLA † [ 50 ]	ABCD $\rightarrow$ D	0.988	0.934	0.883	0.829	0.764	4.398
Ours	ABCD $\rightarrow$ D	0.972	0.939	0.894	0.859	0.809	4.473
TLA [ 6 ]	ABC $\rightarrow$ D	0.955	0.858	0.754	0.669	0.565	3.800
Dita [ 21 ]		0.945	0.825	0.728	0.613	0.500	3.610
UniVLA † [ 50 ]		0.972	0.902	0.826	0.741	0.661	4.102

Table 2: Long-horizon robotic manipulation evaluation on the CALVIN [ 33 ] benchmark. Methods marked with

{\dagger}

are from our re-implementation.

Model	Pick	Move	Drawer	Place	Average
OpenVLA [ 24 ]	0.180	0.563	0.630	0.000	0.343
SpatialVLA [ 36 ]	0.860	0.779	0.574	0.090	0.576
MoTo [ 12 ]	0.740	0.604	0.431	0.000	0.444
villa-X [ 11 ]	0.987	0.750	0.593	0.056	0.597
UniVLA [ 50 ]	0.870	0.565	0.194	0.167	0.449
Ours	0.923	0.676	0.428	0.407	0.609

Table 3: Evaluation on SimplerEnv-Google Robot [ 30 ] across various manipulation tasks.

Figure 1: Sensitivity analysis of

N

and

l_{a}

on LIBERO.

Variant	Pre-training	Co-fine-tuning	Spatial	Object	Goal	Long	Average
Ours	latent motion + terminal frame	+ latent motion	0.948	0.974	0.958	0.906	0.947
Wan2.1 VAE [ 44 ]	latent $\mathbf{z}$ + terminal frame	+ latent $\mathbf{z}$	0.938	0.950	0.922	0.868	0.920

Table 4: Comparison between our latent motion representation and Wan 2.1 VAE latent

\mathbf{z}

on LIBERO.

Figure 3: Cross-Recon visualization on SimplerEnv [ 30 ] and Bridgev2 [ 43 ] .

Figure 4: Visualization of latent-motion clusters and corresponding video examples. (a) Unsupervised clustering results of clip-level motion trajectories. Each subplot shows the average 2D motion trajectory (obtained from the first two PCA components of the accumulated frame-wise motion deltas) for one cluster. (b) Representative video examples from clusters. Cluster 1 and 2 correspond to monotonic downward-like or upward-like motions, whereas Cluster 3 and 4 exhibit rightward-like or leftward-like behaviors.

Figure 5: Comparative visualization of future-frame prediction strategies.

Figure 6: An Intel RealSense camera and a Realman RM75B robot.

Figure 7: Comparison between data collection and real-world deployment during testing.

Explore similar work

World2Act: Latent Action Post-Training from World Model Dynamics
Mar 11, 2026An Dinh Vuong, Tuan Van Vo, Abdullah Sohail +6Vision-Language-Action Models VLMs for Robotics
MBZUAI
LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Jun 14, 2026Jialei Chen, Kai Wang, Kang Chen +9Latent Action Learning Action-Conditioned World Models
Jilin University · Zhongguancun Academy · Nankai University +4
Learning Visual Feature-Based World Models via Residual Latent Action
May 8, 2026Xinyu Zhang, Zhengtong Xu, Yutian Tao +3Latent Action Learning World Model Learning
Rutgers University · Purdue University · University of Wisconsin-Madison

Mar 11, 2026J/K move · Enter open · S save

World2Act: Latent Action Post-Training from World Model Dynamics

An Dinh Vuong, Tuan Van Vo, Abdullah Sohail, Haoran Ding+5

MBZUAI

Vision-Language-Action Models VLMs for Robotics Latent Action Learning

Open paper AskPDF arXiv

World Models (WMs) offer a promising mechanism for post-training Vision-Language-Action (VLA) policies by providing dynamics priors that improve generalization under task and scene variation. However, most WM-based post-training methods rely on pixel-space supervision, making policies sensitive to visual artifacts introduced by imperfect WM rollouts. We present World2Act, a latent-space post-training framework that transfers WM dynamics to the VLA policy without pixel-space supervision. World2Act operates in two stages: 1) it induces a shared video-action latent space by contrastively aligning WM-dynamics latents with action embeddings, and 2) it post-trains the VLA by guiding policy action representations toward WM-imagined dynamics rather than decoded pixels. Built on GR00T-N1.6, World2Act delivers absolute success-rate gains of up to +2.5% on simulation benchmarks (RoboCasa, LIBERO, Bridge-SIMPLER) and +6.7% on a real robot over finetuned VLA baselines. Notably, it outperforms pixel-space WM supervision by up to +6.0%, including on LIBERO where pixel supervision degrades the baseline, suggesting that latent WM dynamics offer a more stable WM-based post-training alternative to pixel-space transfer.

Abstract

Explore similar work

World2Act: Latent Action Post-Training from World Model Dynamics

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies

Learning Visual Feature-Based World Models via Residual Latent Action