Video world models promise general-purpose interactive simulators, but their training is limited by action supervision: action labels are scarce, fragmented across incompatible specifications, and do not scale with the unlabeled video pretraining corpus. A common workaround jointly trains an action encoder to compress consecutive frames into a latent action and a decoder to reconstruct the next frame from the previous frame and that latent, forcing the latent to capture only new information. This has two limitations: learning the action space from scratch can limit generalization to unseen domains, and the bottleneck trades expressiveness for separation, with tighter settings losing nuance and looser ones risking scene leakage under domain shifts. We argue for a different starting point: rather than learning actions from scratch, we start from a pretrained foundation model (V-JEPA 2), whose embeddings of a frame pair already carry substantial action information after large-scale, diverse video pretraining. The remaining challenge is to extract the action while discarding the scene. We propose a label-free contrastive adaptation framework that pulls together frame pairs sharing the same motion under a different scene and pushes apart pairs from the same scene with different motion, reshaping the geometry of the pretrained space without imposing a reconstruction bottleneck. We introduce a unified evaluation protocol along three axes (action expressiveness, scene invariance, and action-conditioned video generation quality) across robot manipulation, camera navigation, game, and driving benchmarks, most of which are unseen during adaptation. Compared with annotation-free latent-action methods, our representations are substantially more expressive while matching their scene invariance, and they yield world models that transfer demonstrated actions to new scenes more faithfully.
Figures & tables
Figure 1: OnlyAct: an unlabeled action representation learning framework builds on a pretrained foundation model to extract expressive and transferable action representations, keeping only the action while discarding the scene in which it occurs. Actions from diverse domains are mapped into a single shared action space, inheriting broad spatio-temporal priors from large-scale video pretraining. These action embeddings can be derived from video frames alone and used to generate videos with the same actions from different initial images across diverse domains. Real actions can also be adaptively mapped into the same action space for real-world control.
Figure 2: Overview of the proposed learning pipeline. Left: Contrastive action adaptation. An anchor frame pair (ot,ot+k) is encoded as a two-frame clip by the OnlyAct encoder ϕθ , a pretrained foundation model with low-rank adapters whose weights are shared across all branches. The positive keeps the action but changes the scene: the moving region of the anchor, found by optical flow, is cut out and pasted at a shifted location onto the background of a different scene (for moving cameras, a different photometric augmentation of the anchor is used instead). The hard negative is drawn from a non-overlapping window (ot′,ot′+k) of the same video, sharing the scene but corresponding to a different action. In the action space, the anchor is pulled towards its positive and pushed away from its hard negative. Right: Action-conditioned video world model. The frozen OnlyAct encoder ϕθ maps consecutive frame pairs into the learned action space, yielding an action stream α1:t that conditions an autoregressive video world model Gψ . Given past frames o1:t and the action stream, Gψ predicts the next frame ot+1 , and is rolled out autoregressively to generate subsequent frames.
Figure 3: Two control modes. After pretraining the video world model on latent actions from the OnlyAct encoder, the learned action space supports two modes. First, action transfer extracts an action stream from a demonstration video and uses it to condition the world model from a new initial frame, reproducing the demonstrated action with no labels. Second, real-action control learns a lightweight interface from real actions onto the learned latent-action space on a small action-labeled set, so that real actions drive the world model directly.
Table 1: Linear probing of action information. A linear layer predicts the ground-truth action from each frozen representation. We report test R2 on held-out scenes (LIBERO, RLBench: held-out demonstrations of training scenes; Procgen: macro-F1 over 15 discrete actions), averaged over 3 probe seeds (std ≤ 0.03). MetaWorld actions are clipped to [−1,1] . ∘ Videos seen by our encoder during training, without action labels; all other datasets are entirely unseen. † Retrained on the same 168K training pairs as our encoder. V-JEPA 2 (frozen): our backbone, embedded identically but without contrastive adaptation. Avg.: mean R2 over the 12 continuous-action datasets. Δ : ours minus the best other method. Bold : best; underline : second best.
World Models (WMs) offer a promising mechanism for post-training Vision-Language-Action (VLA) policies by providing dynamics priors that improve generalization under task and scene variation. However, most WM-based post-training methods rely on pixel-space supervision, making policies sensitive to visual artifacts introduced by imperfect WM rollouts. We present World2Act, a latent-space post-training framework that transfers WM dynamics to the VLA policy without pixel-space supervision. World2Act operates in two stages: 1) it induces a shared video-action latent space by contrastively aligning WM-dynamics latents with action embeddings, and 2) it post-trains the VLA by guiding policy action representations toward WM-imagined dynamics rather than decoded pixels. Built on GR00T-N1.6, World2Act delivers absolute success-rate gains of up to +2.5% on simulation benchmarks (RoboCasa, LIBERO, Bridge-SIMPLER) and +6.7% on a real robot over finetuned VLA baselines. Notably, it outperforms pixel-space WM supervision by up to +6.0%, including on LIBERO where pixel supervision degrades the baseline, suggesting that latent WM dynamics offer a more stable WM-based post-training alternative to pixel-space transfer.
Learning world models across diverse domains calls for a unified control space, yet ac- tion interfaces vary across domains and action annotations are often scarce. Latent ac- tions learned directly from videos offer a promising way to establish such a space without explicit action supervision. However, learning shared latent actions across domains re- mains challenging, as variations in appearance, embodiment, and dynamics can entangle domain-specific visual cues with transferable transition structure. To this end, we intro- duce OneLA, a simple framework that learns continuous latent actions through a shared orthogonal dictionary and a common predictive objective, without task-specific designs or cross-domain alignment losses during latent action pretraining. These latent actions pro- vide a shared conditioning interface for world models, with lightweight adapters mapping native control commands into the same space. Conditioned on latent actions and the cur- rent visual state, the world model predicts context-dependent transitions for controllable rollout. OneLA improves action readout across domains and reduces action cycle-transfer rotation error by 24.8% over the strongest evaluated baseline. A joint-training comparison further shows a 20.5% improvement in Human Preference Score v3 (HPSv3) for real- scene generation. These results suggest a path toward general-purpose world models that learn from diverse videos and support flexible control through latent action space.
Video world models are increasingly used to provide predictive visual representations, yet it remains unclear which pretraining signals induce action-relevant structure in their latent spaces. We study this question through a unified probe-based evaluation across diverse encoder families, including image-only self-supervision, video pretraining with and without latent prediction, reconstruction-based autoencoders, diffusion models, and shortcut-forcing dynamics models. Using a common inverse-dynamics probing objective, we find that action-relevant structure is driven primarily by temporal video pretraining rather than pixel reconstruction fidelity: models with strong pixel decoding quality can exhibit near-zero action recoverability, while video-pretrained self-supervised encoders consistently achieve the best Pareto trade-off between visual fidelity and action prediction. Comparing V-JEPA and VideoMAE further shows that most gains arise from natural-video temporal context, with feature-level latent prediction providing a smaller additional benefit. These trends transfer across robotic benchmarks, though CALVIN reveals that static-environment tasks can partially mask the importance of temporal structure by allowing strong image priors to suffice. Finally, inverse-dynamics supervision substantially improves robustness to visual corruption, suggesting that action-aware objectives regularize latent geometry beyond clean-setting performance. Our results identify temporal predictive structure -- not reconstruction fidelity -- as the primary ingredient underlying action-relevant video representations.
Jewon Yeom, Hanseul Kim, Jeongjae Park +3
Graduate School of Data Science, Seoul National University