World2Act: Latent Action Post-Training from World Model Dynamics

An Dinh Vuong; Tuan Van Vo; Abdullah Sohail; Haoran Ding; Liang Ma; Xiaodan Liang; Anqing Duan; Ivan Laptev

cs.CVMar 11, 2026

World2Act: Latent Action Post-Training from World Model Dynamics

Authors: An Dinh Vuong, Tuan Van Vo, Abdullah Sohail, Haoran Ding, Liang Ma, Xiaodan Liang, Anqing Duan, Ivan Laptev, +1 more

Organizations: MBZUAI

Abstract

World Models (WMs) offer a promising mechanism for post-training Vision-Language-Action (VLA) policies by providing dynamics priors that improve generalization under task and scene variation. However, most WM-based post-training methods rely on pixel-space supervision, making policies sensitive to visual artifacts introduced by imperfect WM rollouts. We present World2Act, a latent-space post-training framework that transfers WM dynamics to the VLA policy without pixel-space supervision. World2Act operates in two stages: 1) it induces a shared video-action latent space by contrastively aligning WM-dynamics latents with action embeddings, and 2) it post-trains the VLA by guiding policy action representations toward WM-imagined dynamics rather than decoded pixels. Built on GR00T-N1.6, World2Act delivers absolute success-rate gains of up to +2.5% on simulation benchmarks (RoboCasa, LIBERO, Bridge-SIMPLER) and +6.7% on a real robot over finetuned VLA baselines. Notably, it outperforms pixel-space WM supervision by up to +6.0%, including on LIBERO where pixel supervision degrades the baseline, suggesting that latent WM dynamics offer a more stable WM-based post-training alternative to pixel-space transfer.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Problem: Prior WM $\rightarrow$ VLA post-training relies on imagined pixel rollouts, making policies vulnerable to visual artifacts. (b) Solution: We bypass this by directly aligning VLA action representation with WM latent space. (c) Post-training on GR00T-N1.6 [ 14 ] indicates that our latent post-training yields consistent improvements, overcoming the instability of pixel-space methods.

Figure 2: World2Act overview. (a) Stage 1: Latent alignment. We train video and action adapters (

\mathcal{B}_{\text{v}},\mathcal{B}_{\text{a}}

) with reconstruction and contrastive objectives. (b) Stage 2: VLA post-training. We freeze the VLA and learn a residual policy guided by WM-induced latent dynamics.

Table 3

Panel of Figure 3 — Figure 3: Post-training scaling and generalization. Left: Cosine similarity correlates with downstream success rate. Right: Unseen-task performance as seen-task diversity increases.

Table 5

Figure 7: Qualitative results of our WM-to-VLA framework. Top: A trajectory generated by the WM. Bottom: The corresponding real-robot execution of the same task utilizing GR00T-N1.6-ft + World2Act . $t$ denotes the timestep for each frame.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Baselines	Spatial	Object	Goal	Long	Average
$\pi_{0}$ [ 6 ]	0.968	0.988	0.958	0.852	0.942
$\pi_{0.5}$ [ 24 ]	0.988	0.982	0.980	0.924	0.969
OpenVLA-OFT [ 28 ]	0.976	0.984	0.979	0.945	0.971
UniVLA [ 8 ]	0.965	0.968	0.956	0.920	0.952
CoWVLA [ 56 ]	0.972	0.978	0.946	0.928	0.956
Cosmos Policy [ 29 ]	0.981	1.000	0.982	0.976	0.985

Appendix

Table 7: Detailed LIBERO results.

Baselines	SR
Cosmos Policy [ 29 ]	0.657
Cosmos Policy [ 29 ] + World2Act (Ours)	0.663
GR00T-N1.6-ft [ 14 ]	0.701
GR00T-N1.6-ft [ 14 ] + DreamGen [ 26 ]	0.705
GR00T-N1.6-ft [ 14 ] + World2Act (Ours)	0.726

Appendix

Table 8: Extra RoboCasa results.

Baselines	Put Spoon on Towel	Stack Green on Yellow	Put Carrot on Plate	Put Eggplant in Basket	Open Drawer	Close Drawer	Put Eggplant in Sink	Average
$\pi_{0}$ [ 6 ]	0.292	0.000	0.166	0.624	0.676	0.480	0.124	0.337
OpenVLA [ 30 ]	0.000	0.000	0.000	0.042	0.644	0.036	0.000	0.103
OpenVLA-OFT [ 28 ]	0.342	0.308	0.292	0.726	0.338	0.742	0.320	0.438
GR00T-N1.6-ft [ 14 ]	0.734	0.044	0.624	0.372	0.916	0.984	0.358	0.576
GR00T-N1.6-ft [ 14 ] + DreamGen [ 26 ]	0.718	0.048	0.646	0.424	0.928	0.976	0.342	0.583
GR00T-N1.6-ft [ 14 ] + World2Act (Ours)	0.768	0.042	0.664	0.386	0.916	0.990	0.362	0.590

Appendix

Table 9: Detailed Bridge-SIMPLER results.

Figure 8: Stage 1 pretraining losses. Training curves for

\mathcal{L}_{\text{recon.}}

and

\mathcal{L}_{\text{contrastive}}

(defined in subsection 3.2 of our main paper) over 30K steps. Both losses converge stably, demonstrating effective cross-modal alignment between the video and action representations.

Figure 9: Post-training rollout synthesis. Given an initial frame and a high-level instruction, an LLM decomposes the instruction into an ordered list of atomic skill prompts. The world model generates one video segment per skill, using the final frame of each segment as the visual condition for the next skill. The resulting video segments are concatenated to form a long-horizon synthetic rollout. We retain the pre-decoding video latents from the world model as post-training supervision for World2Act . The atomic-skill data used to train the world model are described in Appendix D .

Train Tasks	Test Tasks
CoffeeSetupMug	PnPCounterToCab
CoffeeServeMug	PnPCabToCounter
CoffeePressButton	PnPCounterToSink
OpenSingleDoor	PnPSinkToCounter
OpenDoubleDoor	PnPCounterToMicrowave
CloseSingleDoor	PnPMicrowaveToCounter

Appendix

Table 10: Cross-task generalization experiment splits.

Figure 10: Additional t-SNE visualizations. We provide extra qualitative results on PnPCounterToCab and TurnOnStove. Consistent with the OpenDrawer results in Figure 4 , World2Act yields predicted-action latent distributions that better overlap with the reference-action and target-WM latent regions, indicating improved alignment while remaining compatible with the expert action manifold.

Model	Inference Speed (Hz) ( $\uparrow$ )	Per Action (ms) ( $\downarrow$ )
GR00T-N1.6-ft [ 14 ]	274.1	3.6
GR00T-N1.6-ft + World2Act (Ours)	251.9	4.0
Cosmos Policy [ 29 ]	20.8	48.1
Cosmos Policy + World2Act (Ours)	20.5	48.8

Appendix

Table 11: Inference speed comparison. We report the inference speed on RoboCasa using an NVIDIA RTX 4090. We measure speed by the time required to predict an action from an observation.

Module	Tensor Mapping	Architecture
Action Adapter $\mathcal{B}_{\text{a}}$	$\mathbb{R}^{A\times M}\rightarrow\mathbb{R}^{D}$	Flatten, FC( $A\times M$ , 128), GELU
(Action Encoder)		FC(128, 64), GELU
		FC(64, $D$ )
Video Adapter $\mathcal{B}_{\text{v}}$	$\mathbb{R}^{C\times H\times W}\rightarrow\mathbb{R}^{D}$	Conv2D( $C$ , 64, $k=3,s=2,p=1$ ), GroupNorm, GELU
(Video Encoder)		Conv2D(64, 128, $k=3,s=2,p=1$ ), GroupNorm, GELU
		AdaptiveAvgPool2D((1, 1)), Flatten, FC(128, $D$ )

Appendix

Table 12: Architectural details of the policy components. In our implementation,

A=12

for RoboCasa and

A=7

for LIBERO, Bridge-SIMPLER and real Franka Robot,

M=4

D=32

, and

D_{\text{proprio}}=53

. Input images are

256\times 256

. The Attention Module processes

N_{\text{src}}=3

tokens: the visual feature, the latent base action from

\mathcal{B}_{\text{a}}

, and the proprioception state. The video latent resolution is

16\times 60\times 104

, following the Cosmos-Predict-2 [ 1 ] implementation.

Figure 11: Atomic-skill data construction for world-model training. We segment each demonstration using gripper-state changes, decompose the global instruction into atomic skill prompts with an LLM, and synchronize sub-videos with prompts through schema-based chronological alignment.

Panel of Figure 12 — Figure 12: Video-length distribution. Across RoboCasa and LIBERO, decomposed skills exhibit shorter mean length and a more concentrated distribution than full sequences, improving stability for world-model training on variable-duration robotic demonstrations.

Panel of Figure 13 — Figure 13: Multi-view stitching strategies. To preserve geometric context, observations from multiple cameras are spatially arranged into a unified $2\times 2$ grid. For each simulation environment, we illustrate the camera mapping template on the left and an example stitched observation on the right. (a) In LIBERO, the primary and wrist cameras occupy the top row, with the bottom row black-padded. (b) In RoboCasa, the left, right, and wrist views are mapped to the top-left, top-right, and bottom-left quadrants, respectively.

Video Diffusion Backbone	IF ( $\uparrow$ )	PA ( $\uparrow$ )
LTX-Video [ 20 ]	7.7	40.1
HunyuanVideo-1.5 [ 53 ]	8.3	44.8
Wan2.2 [ 50 ]	20.5	56.9
Cosmos-Predict2 [ 1 ] (Base data)	29.6	60.5
Cosmos-Predict2 (Ours, atomic-skill data)	35.3	65.4

Appendix

Table 13: Quantitative comparison of video diffusion backbones. The Cosmos-Predict2 model trained on our atomic-skill data significantly outperforms baselines in both instruction following and physical alignment. Our results are consistent with Jang et al. [ 26 ] .

Figure 14: Qualitative comparison of task completion: Skill-WM vs. Base-WM. We display the final frames of execution videos generated from the same prompts and initial states. While the Base-WM (left) fails to complete the trajectory and stalls mid-task, the WM trained on atomic-skill data (right) successfully places the object on the counter. Each example depicts three synchronized camera perspectives: left, wrist-mounted, and right.

Method	World Model	SR
Cosmos Policy [ 29 ] + World2Act	Base-WM	0.661
	Skill-WM (Ours)	0.663
GR00T-N1.6-ft [ 14 ] + World2Act	Base-WM	0.715
	Skill-WM (Ours)	0.726

Appendix

Table 14: Impact of WM on World2Act . Our Skill-WM improves downstream success rate across both policies.

Panel of Figure 15 — Figure 15: Visual artifacts examples. We illustrate representative failure cases across different environments. The RoboCasa, LIBERO, and real-world setups utilize three, two, and one camera views, respectively.

IDM input video	MSE $\downarrow$	MAE $\downarrow$
Ground-truth video	0.082	0.167
Decoded WM rollout	0.097	0.199
WM / ground-truth	$1.18{\times}$	$1.19{\times}$

Appendix

Table 15: Frozen-IDM action error on LIBERO-Object. Errors are computed against temporally aligned ground-truth actions over 50 held-out scenes. Decoded WM rollouts increase both MSE and MAE under the same IDM.

Panel of Figure 16 — Figure 16: Failure analysis of WM to VLA transfer. The WM generates a visually plausible completion of the task, but the VLA fails to physically ground these dynamics, causing a distribution shift during actual deployment.

Explore similar work

Token-World: World Modeling in Vision-Language Model Token Space for Robot Manipulation
Sep 30, 2026Chuyao Fu, Xiaowei Chi, Yuhan Rui +14World Models Action-Conditioned World Models
Southern University of Science and Technology · MUKA Robotics · Hong Kong University of Science and Technology +3
Chain of World: World Model Thinking in Latent Motion
Mar 3, 2026Fuxiang Yang, Donglin Di, Lulu Tang +6Vision-Language-Action Models Reasoning for VLA Models
Harbin Institute of Technology · Li Auto · Beijing Academy of Artificial Intelligence (BAAI) +3
LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Jun 14, 2026Jialei Chen, Kai Wang, Kang Chen +9Latent Action Learning Action-Conditioned World Models
Jilin University · Zhongguancun Academy · Nankai University +4

Sep 30, 2026J/K move · Enter open · S save

Token-World: World Modeling in Vision-Language Model Token Space for Robot Manipulation

Chuyao Fu, Xiaowei Chi, Yuhan Rui, Yu-kai Wang+13

Southern University of Science and Technology · MUKA Robotics · Hong Kong University of Science and Technology+3

World Models Action-Conditioned World Models Robotic Manipulation VLMs for Robotics Embodied World Models

Open paper AskPDF arXiv

A common approach to world-model simulation for vision-language-action (VLA) systems is to predict future RGB observations and then re-encode them into policy inputs, introducing an indirect interface between simulation and downstream policy execution. We instead investigate whether world dynamics can be modeled in a compact, policy-oriented state derived from VLM visual tokens. A key challenge is that raw VLM visual tokens are high-dimensional, making efficient and accurate autoregressive dynamics modeling challenging. To address this, we introduce Token-World, an action-conditioned world model that compresses VLM features into a compact token state, learns future dynamics in this reduced space, and maps predicted states back to the original policy-facing representation for downstream use. Across manipulation benchmarks, Token-World improves open-loop feature fidelity and policy-action consistency over recent world-model simulators, with slower degradation over long rollout horizons. In closed-loop evaluation, its simulated policy performance correlates more strongly with reference policy performance than Ctrl-World (

r=0.794

vs.\

0.583

), while requiring lower simulation latency. Ablations further show that compact-representation design and dimensionality substantially affect future-state prediction. Code will be available at https://chuyaofu.github.io/Token-World/.

Abstract

Explore similar work

Token-World: World Modeling in Vision-Language Model Token Space for Robot Manipulation

Chain of World: World Model Thinking in Latent Motion

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies