Chain of World: World Model Thinking in Latent Motion
Authors: Fuxiang Yang, Donglin Di, Lulu Tang, Xuancheng Zhang, Lei Fan, Hao Li, Chen Wei, Tonghua Su, +1 more
Organizations: Harbin Institute of Technology · Li Auto · Beijing Academy of Artificial Intelligence (BAAI) · Peking University · University of New South Wales · Chongqing Research Institute of HIT
Vision-Language-Action (VLA) models are a promising path toward embodied intelligence, yet they often overlook the predictive and temporal-causal structure underlying visual dynamics. World-model VLAs address this by predicting future frames, but waste capacity reconstructing redundant backgrounds. Latent-action VLAs encode frame-to-frame transitions compactly, but lack temporally continuous dynamic modeling and world knowledge. To overcome these limitations, we introduce CoWVLA (Chain-of-World VLA), a new "Chain of World" paradigm that unifies world-model temporal reasoning with a disentangled latent motion representation. First, a pretrained video VAE serves as a latent motion extractor, explicitly factorizing video segments into structure and motion latents. Then, during pre-training, the VLA learns from an instruction and an initial frame to infer a continuous latent motion chain and predict the segment's terminal frame. Finally, during co-fine-tuning, this latent dynamic is aligned with discrete action prediction by jointly modeling sparse keyframes and action sequences in a unified autoregressive decoder. This design preserves the world-model benefits of temporal reasoning and world knowledge while retaining the compactness and interpretability of latent actions, enabling efficient visuomotor learning. Extensive experiments on robotic simulation benchmarks show that CoWVLA outperforms existing world-model and latent-action approaches and achieves moderate computational efficiency, highlighting its potential as a more effective VLA pretraining paradigm. The project website can be found at https://fx-hit.github.io/cowvla-io.
Figures & tables
Figure 1: Comparison of VLA pretraining strategies. (a) World Model : It predicts future visual frames, leading to redundant background reconstruction. (b) Latent Action : It learns the frame-to-frame transition using a visual encoder E , but lacks temporally continuous reasoning. (c) CoWVLA : Our method first uses a video encoder E to decompose each video segment into motion and structure latents, and then trains the VLM to infer latent motion and predict the terminal frame of the segment given the instruction and the initial frame.
Figure 2: Overview of the CoWVLA framework. CoWVLA consists of two core components: a latent motion extractor and a VLA decoder. The latent motion extractor, implemented as a video VAE, disentangles each video segment into a structure latent zs and two directional motion latents zmh and zmw , which are concatenated into a unified latent motion vector zm . The VLA decoder performs unified autoregressive modeling over multimodal sequences. During pre-training, the model takes the instruction and initial frame as input, and uses a learnable motion query Q to predict the latent motion z^m while reconstructing the terminal frame of the video segment. During co-fine-tuning, the input expands into alternating keyframe–action pairs; Q continues to aggregate temporally continuous latent dynamics, guiding multi-step action generation under sparse visual observations.
Model
LIBERO
SimplerEnv-WidowX
SPATIAL
OBJECT
GOAL
LONG
Avg.
Stack Block
Put Carrot
Put Spoon
Put Eggplant
Avg.
OpenVLA [ 24 ]
0.849
0.884
0.792
0.537
0.765
0.000
0.000
0.000
0.041
0.010
SpatialVLA [ 36 ]
0.882
0.899
0.786
0.555
0.781
0.292
0.250
0.167
1.000
0.427
CogACT [ 28 ]
0.960
0.874
0.868
0.846
0.887
0.150
0.508
0.717
0.675
0.513
Dita [ 21 ]
0.842
0.963
0.854
0.638
0.824
–
–
–
–
–
π0 [ 3 ]
0.968
0.988
0.958
0.852
0.942
0.167
0.000
0.291
0.625
0.401
Table 1: Comparison of different methods on the LIBERO [ 32 ] and SimplerEnv-WidowX [ 30 ] benchmarks. The best and the second-best values for each metric are bold and underlined respectively.
Model
Reconstruction Metrics
Simulation Evaluation
PSNR ↑
SSIM ↑
LPIPS ↓
Stack Block
Put Carrot
Put Spoon
Put Eggplant
Average
Pretrain
32.7
0.923
0.122
0.458
0.750
0.792
0.917
0.729
Finetune
33.4
0.934
0.123
0.625
0.667
0.792
0.958
0.760
Table 2: Evaluation of VAE-Reconstructed Videos and downstream fine-tuning performance on SimplerEnv-WidowX [ 30 ] .
Figure 3: Visualization of the disentangled motion and structure latents. We select two frames ( t1 and t2 ) and show the original (Orig.) and reconstructed (Recon.) frames. “M. Recon.” and “S. Recon.” denote the reconstructions obtained by decoding only the motion latent or only the structure latent, respectively. The structure latent preserves the global scene layout, whereas the motion latent captures motion and fine-grained temporal details.
Figure 4: Cross-reconstruction visualization. We extract the structure latent from the static video in the first row and the motion latent from the robot-arm motion video in the second row. By combining the two latents, we reconstruct the video shown in the third row. We compute the difference between the cross-reconstructed frames and the static frames to highlight the changed regions, which correspond to the robot arm’s motion.
Figure 5: Comparative visualization of future-frame prediction strategies. There are two tasks demonstrated: i) pick up the black bowl from the table center and place it on the plate, and ii) sweep into a pile. (a) The world-model approach predicts five future frames. (b) The single-goal-frame approach predicts one goal frame. (c) Our method reasons through a learned motion latent zm , producing more reasonable and instruction-aligned frames.
Config
Variant
Spatial
Object
Goal
Long
Average
Latent Action
w/o LA
0.622
0.146
0.694
0.328
0.448
LAPA style
0.718
0.852
0.804
0.488
0.716
villa-X style
0.840
0.904
0.834
0.668
0.812
structure latent
0.856
0.898
0.822
0.692
0.817
motion latent
0.916
0.932
0.886
0.774
0.877
World Model
UniVLA Style
0.958
0.978
0.932
0.898
0.942
Table 3: Ablation study on the LIBERO [ 32 ] benchmark.
λ1
λ2
Spatial
Object
Goal
Long
Average
0.0
0.0
0.922
0.962
0.862
0.742
0.872
0.1
0.0
0.960
0.980
0.922
0.882
0.936
1.0
0.0
0.958
0.970
0.950
0.902
0.945
0.1
0.05
0.954
0.972
0.944
0.914
0.946
0.1
0.01
0.970
0.964
0.958
0.926
0.955
1.0
0.01
0.970
0.956
0.934
0.922
0.946
Table 4: Ablation study of loss weights on the LIBERO [ 32 ] benchmark.
Figure 6: Comparison of pre-training efficiency and task performance on LIBERO [ 32 ] across different methods. Blue and orange circles denote world-model and latent-action baselines, respectively, while green circles denote our configurations. Circle size indicates training-time GPU memory usage. Our method balances pre-training efficiency and performance, achieving a higher success rate with moderate computational efficiency.
Dataset Name
Count
Berkeley Autolab Ur5 [ 10 ]
892
Bridgev2 [ 43 ]
24879
Cmu Play Fusion [ 9 ]
576
Fractal [ 4 ]
65530
Kuka [ 23 ]
84202
Maniskill [ 20 ]
30029
Table 1: Training datasets.
Method
Task
Tasks Completed in a Row
Avg. Len ↑
1
2
3
4
5
UniVLA † [ 50 ]
ABCD → D
0.988
0.934
0.883
0.829
0.764
4.398
Ours
0.972
0.939
0.894
0.859
0.809
4.473
TLA [ 6 ]
ABC → D
0.955
0.858
0.754
0.669
0.565
3.800
Dita [ 21 ]
0.945
0.825
0.728
0.613
0.500
3.610
UniVLA † [ 50 ]
0.972
0.902
0.826
0.741
0.661
4.102
Table 2: Long-horizon robotic manipulation evaluation on the CALVIN [ 33 ] benchmark. Methods marked with † are from our re-implementation.
Model
Pick
Move
Drawer
Place
Average
OpenVLA [ 24 ]
0.180
0.563
0.630
0.000
0.343
SpatialVLA [ 36 ]
0.860
0.779
0.574
0.090
0.576
MoTo [ 12 ]
0.740
0.604
0.431
0.000
0.444
villa-X [ 11 ]
0.987
0.750
0.593
0.056
0.597
UniVLA [ 50 ]
0.870
0.565
0.194
0.167
0.449
Ours
0.923
0.676
0.428
0.407
0.609
Table 3: Evaluation on SimplerEnv-Google Robot [ 30 ] across various manipulation tasks.
Figure 1: Sensitivity analysis of N and la on LIBERO.
Variant
Pre-training
Co-fine-tuning
Spatial
Object
Goal
Long
Average
Ours
latent motion + terminal frame
+ latent motion
0.948
0.974
0.958
0.906
0.947
Wan2.1 VAE [ 44 ]
latent z + terminal frame
+ latent z
0.938
0.950
0.922
0.868
0.920
Table 4: Comparison between our latent motion representation and Wan 2.1 VAE latent z on LIBERO.
Figure 2: Cross-Recon visualization on LIBERO [ 32 ] . The first six columns show temporally sampled frames from three rows: Structure (top), Motion (middle), and Cross-Recon (bottom). The Cross-Recon videos are generated by combining the static appearance from the Structure video with the motion representation extracted from the Motion video, revealing the transferred motion patterns. Each Cross-Recon frame is overlaid with a motion heatmap to highlight dynamic regions. The last column presents three summary maps: motion heatmaps obtained by averaging and maximizing per-frame absolute differences between Cross-Recon and Structure, and the end-effector trajectory estimated from the motion regions.
Figure 3: Cross-Recon visualization on SimplerEnv [ 30 ] and Bridgev2 [ 43 ] .
Figure 4: Visualization of latent-motion clusters and corresponding video examples. (a) Unsupervised clustering results of clip-level motion trajectories. Each subplot shows the average 2D motion trajectory (obtained from the first two PCA components of the accumulated frame-wise motion deltas) for one cluster. (b) Representative video examples from clusters. Cluster 1 and 2 correspond to monotonic downward-like or upward-like motions, whereas Cluster 3 and 4 exhibit rightward-like or leftward-like behaviors.
Figure 5: Comparative visualization of future-frame prediction strategies.
Figure 6: An Intel RealSense camera and a Realman RM75B robot.
Figure 7: Comparison between data collection and real-world deployment during testing.
World Models (WMs) offer a promising mechanism for post-training Vision-Language-Action (VLA) policies by providing dynamics priors that improve generalization under task and scene variation. However, most WM-based post-training methods rely on pixel-space supervision, making policies sensitive to visual artifacts introduced by imperfect WM rollouts. We present World2Act, a latent-space post-training framework that transfers WM dynamics to the VLA policy without pixel-space supervision. World2Act operates in two stages: 1) it induces a shared video-action latent space by contrastively aligning WM-dynamics latents with action embeddings, and 2) it post-trains the VLA by guiding policy action representations toward WM-imagined dynamics rather than decoded pixels. Built on GR00T-N1.6, World2Act delivers absolute success-rate gains of up to +2.5% on simulation benchmarks (RoboCasa, LIBERO, Bridge-SIMPLER) and +6.7% on a real robot over finetuned VLA baselines. Notably, it outperforms pixel-space WM supervision by up to +6.0%, including on LIBERO where pixel supervision degrades the baseline, suggesting that latent WM dynamics offer a more stable WM-based post-training alternative to pixel-space transfer.
Vision-Language-Action models (VLAs) leverage large-scale vision-language pretraining for semantic robot control, but often lack explicit foresight into how robot actions change the scene. World-Action Models (WAMs) address this limitation by conditioning policies on predicted futures, yet existing approaches typically rely on computationally expensive video generation with substantial pixel-level redundancy. We present LaWAM, a Latent World Action Model that exposes predictive dynamics to robot policies through compact latent visual subgoals instead of reconstructed future video. At the core of LaWAM is a latent-action-conditioned Latent World Model (LaWM). We obtain LaWM by training a latent action model in the latent space of a pretrained vision foundation model and repurposing its forward decoder to predict future observation features for scene evolution. LaWAM then conditions action generation on these predicted latent visual subgoals to enable dynamics-aware robot control. LaWAM achieves state-of-the-art or competitive success rates (SRs) across LIBERO (98.6% SR), RoboTwin (91.22% SR), and real-world manipulation tasks while retaining low-latency inference. LaWAM runs in 187 ms per action-chunk prediction and achieves up to 24x lower wall-clock latency than pixel-space WAMs.
Jialei Chen, Kai Wang, Kang Chen +9
Jilin University · Zhongguancun Academy · Nankai University +4
World models predict future transitions from observations and actions. Existing works predominantly focus on image generation only. Visual feature-based world models, on the other hand, predict future visual features instead of raw video pixels, offering a promising alternative that is more efficient and less prone to hallucination. However, current feature-based approaches rely on direct regression, which leads to blurry or collapsed predictions in complex interactions, while generative modeling in high-dimensional feature spaces still remains challenging. In this work, we discover that a new type of latent action representation, which we refer to as Residual Latent Action (RLA), can be easily learned from DINO residuals. We also show that RLA is predictive, generalizable, and encodes temporal progression. Building on RLA, we propose RLA World Model (RLA-WM), which predicts RLA values via flow matching. RLA-WM outperforms both state-of-the-art feature-based and video-diffusion world models on simulation and real-world datasets, while being orders of magnitude faster than video diffusion. Furthermore, we develop two robot learning techniques that use RLA-WM to improve policy learning. The first one is a minimalist world action model with RLA that learns from actionless videos, and improves VLA on LIBERO and real robot. The second one is a visual RL framework trained entirely inside a world model learned from offline videos only, using a video-aligned reward and no online interactions. Project page: https://mlzxy.github.io/rla-wm
Xinyu Zhang, Zhengtong Xu, Yutian Tao +3
Rutgers University · Purdue University · University of Wisconsin-Madison