Videos reveal how the world changes, but rarely the actions that cause those changes. Latent action learning recovers such supervision from visual transitions, yet often learns the underlying dynamics largely from scratch. We ask whether a pretrained video generator can instead provide the prior for both inverse and forward dynamics. We introduce O2LaF, built on a simple principle: one prior, two dynamics. We initialize both models from the same pretrained video DiT and adapt them to their distinct roles: a feed-forward transition encoder with learnable action queries for inverse dynamics, and a causal, action-conditioned video generator for forward dynamics. By controlling information access and optimizing both dynamics end-to-end through a compact bottleneck, O2LaF learns transferable actions from unlabeled videos. Across navigation and robotic manipulation, O2LaF sets a new state of the art in few-shot action probing and enables stronger zero-shot action transfer across visual contexts and embodiments. Beyond generative video priors, we further extend our framework to predictive representation priors such as V-JEPA2, achieving competitive performance. Our findings suggest a broader paradigm for latent-action learning: repurposing pretrained video models to discover transferable actions and predict their consequences.
Figures & tables
Figure 2: Paradigms for self-supervised latent-action learning. Existing methods learn latent actions (a) from scratch [ 8 , 85 , 20 ] , (b) on frozen visual features [ 9 , 21 ] , (c) with a pretrained video generator as the FDM only [ 62 , 76 ] , or (d) with an external video prior as auxiliary supervision [ 35 ] . Ours repurposes one pretrained video DiT as both the IDM and the FDM.
Figure 3: Action-aware response in a pretrained video DiT. Cross-attention follows the active hands and manipulated regions as the interaction changes.
Figure 4: Action-query attention in IDM.
Figure 5: Overview of O 2 LaF. Two independently adapted video DiTs share a pretrained prior: the IDM infers latent actions, while the FDM predicts their visual consequences. Both employ frame-causal attention and role-specific noise sampling, with end-to-end training through latent actions.
Figure 6: Few-shot linear probing of frozen latent actions. O 2 LaF consistently outperforms prior latent-action representations on both settings across labeled-data budgets.
Method
1st-P
3rd-P
Visual Quality
Camera Motion
Visual Quality
Camera Motion
Img. Q. ↑
Temp. C. ↑
Trans. ↓
Rot. ↓
Img. Q. ↑
Temp. C. ↑
Trans. ↓
Rot. ↓
AdaWorld [ 20 ]
0.639
0.899
2.539
2.055
0.616
0.927
1.342
1.217
Olaf-World [ 35 ]
0.690
0.910
1.627
1.300
0.648
0.930
0.985
0.701
Ours
0.698
0.900
1.289
0.786
0.682
0.919
0.511
0.330
Ours (w/o CFG)
0.672
0.910
1.364
0.830
0.662
0.927
0.513
0.364
Table 1: Zero-shot cross-context camera navigation. We report VBench imaging quality (Img. Q.) and temporal consistency (Temp. C.), and ViPE-based translation NRMSE and rotation RPE. Methods above the separator use action CFG =3 . Best and second-best results are bold and underlined , respectively.
Figure 7: Zero-shot cross-context camera navigation. O 2 LaF better reproduces reference camera and agent action while preserving the target scene.
Method
Human
Robot
Self
Cross-cycle
Self
Cross-cycle
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
DreamDojo [ 19 ]
23.38
0.871
0.223
22.61
0.859
0.244
18.73
0.769
0.159
18.35
0.754
0.165
Ours
26.72
0.903
0.148
24.30
0.878
0.195
19.92
0.787
0.140
18.85
0.770
0.163
Table 2: Cross-embodiment action transfer via cycle reconstruction. Self reconstructs the source video directly, while Cross-cycle reconstructs it after transferring latent actions to the opposite embodiment and back ( H→R→H and R→H→R ). Best results are in bold .
Figure 8: Zero-shot cross-embodiment manipulation O 2 LaF better preserves object-directed motion while adapting to the target embodiment and visual context.
Table 3: Design choices for the two dynamics. MIND joint exact-match accuracy ( % , ↑ ) at k∈{1,50} . Panels (a,c) use Wan2.2-5B, while (b) studies Wan2.1 FDM scaling with a fixed 1.3B IDM.
Figure 9: Role-specific noise levels. IDM linear probing across a 5×5 grid of IDM and FDM flow times. The non-monotonic trends reveal distinct preferred noise configurations across motion domains.
IDM Backbone
Probing
1st-P World Model
3rd-P World Model
MIND ↑
Color R2L / a / b
VBench ↑
ViPE ↓
VBench ↑
ViPE ↓
1st-P
3rd-P
Img.
Temp.
Trans.
Rot.
Img.
Temp.
Trans.
Rot.
ST-Transformer
58.79
79.70
−0.419/−0.190/−0.530
0.666
0.901
1.449
1.022
0.636
0.924
0.620
0.478
VideoChat3
36.96
43.86
0.411/−0.147/0.364
–
–
–
–
–
–
–
–
V-JEPA2
88.62
96.14
0.514/0.816/0.742
0.657
0.901
1.349
0.847
0.664
0.916
0.439
0.337
Wan2.2
87.95
94.37
−0.297/0.081/0.223
0.698
0.900
1.289
0.786
0.682
0.919
0.511
0.330
Table 4: Comparison of O 2 LaF IDM backbones. Color R2 measures the linear decodability of mean reference-frame CIELAB ( L/a/b ) from latent actions. Positive values indicate appearance leakage, while non-positive values indicate no improvement over the mean predictor.
Figure 10: Cross-context action transfer with IDM priors. V-JEPA2 exhibits source-appearance leakage, whereas video DiT better preserves target appearance.
Configuration
1st-P
3rd-P
Probing ↑
VBench ↑
ViPE ↓
Probing ↑
VBench ↑
ViPE ↓
FDM noise
IDM grad.
k=1
k=50
Img.
Temp.
Trans.
Rot.
k=1
k=50
Img.
Temp.
Trans.
Rot.
End-to-end learning (causal FDM)
Mixed
All
72.81
87.95
0.649
0.895
1.937
0.993
52.08
94.37
0.654
0.913
0.845
0.481
Mixed
High
72.23
86.79
0.654
0.885
1.959
0.943
60.38
95.11
0.660
0.902
0.918
0.484
High
High
72.28
85.76
0.647
0.836
2.419
1.168
60.75
94.86
0.646
0.853
1.618
0.765
Table 5: Trade-offs between action learning and generation. End-to-end models vary FDM noise sampling ( Mixed : standard and high; High : high only) and IDM gradient routing ( All : all timesteps; High : high-noise only). Two-stage models freeze the Mixed/All IDM and compare pretrained versus E2E FDM initialization. Gray probing scores are shared and excluded from ranking.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
End-to-end
Two-stage FDM
Backbone
Wan2.2-5B ×2
Wan2.2-5B
Trainable weights
LoRA-128 + heads
Full FDM
IDM
Jointly trained
Frozen @ 5k
Attention
Frame-causal
Bidirectional FDM
RGB frames
41
81
Latent actions
10
20
Appendix
Table 6: Training configurations.
Figure 11: Where does the pairwise action query attend? For transition xt→xt+1 , action query qt jointly attends to spatial video tokens in both endpoint frames.
Training data
MIND Exact Match ↑
In-lab EEF R2↑
1st-P
3rd-P
Left arm
Right arm
Mira only
87.95
94.37
0.135
0.257
EgoDex only
76.70
86.60
0.192
0.363
EgoDexMix
76.70
91.66
0.227
0.386
Mira / EgoDexMix (70/30)
85.94
93.96
0.163
0.341
Appendix
Table 7: Effect of latent-action training data. All checkpoints are compared at training step 5k.
Model
Pretraining
Adaptation
Translation (Dolly/Pedestal/Truck)
Zoom
Rotation (Pan/Tilt/Roll)
Static
Avg.
ΔLA
In
Out
Up
Down
Right
Left
In
Out
Right
Left
Up
Down
CW
CCW
CameraBench reference
Qwen2.5-VL-72B
VLM
SFT
86.8
41.3
30.5
50.6
70.7
52.6
57.1
78.5
81.9
86.6
79.1
70.4
35.0
43.8
76.6
62.8
–
Without latent-action pretraining
Scratch IDM (270M)
None
Full FT- h
45.9
12.8
9.8
15.4
32.1
21.2
14.2
12.7
35.4
32.1
29.4
22.1
15.4
14.8
66.9
25.3
–
Video DiT IDM (5B)
Video
LoRA- h
93.9
72.4
54.5
73.5
81.4
77.3
68.4
76.1
91.7
90.2
81.9
71.1
88.7
90.3
87.7
79.9
–
Appendix
Table 8: Camera-motion annotation on CameraBench. Per-class AP ( ↑ ) on the official test set. Video / LA denote video-generative/latent-action pretraining. Frozen- μ / Frozen- h use MLP probes on latent action means/pre-Gaussian features. LoRA- h / Full FT- h additionally adapt the IDM. Blue and red rows highlight matched supervised comparisons at 270M and 5B scales, respectively. ΔLA reports the AP gain from action pretraining over matched baselines. Our results ensemble three independently trained seeds.
Figure 12: The FDM as a policy verifier on a GR-1 humanoid. Each case shows one trajectory in the physics simulator rebuilt from a single photo (top), the same motion with objects removed, from which latent actions are extracted (middle), and the FDM rollout from the real photo (bottom).
Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by 300× (1,200→4 tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.
Latent action models (LAMs) learn action representations from unlabeled videos by inferring latent actions from visual transitions and reconstructing future states. However, we identify a fundamental reconstruction-action mismatch: lower reconstruction error does not necessarily yield better latent dynamics or downstream performance. We attribute this mismatch to two underconstrained aspects of reconstruction-based latent dynamics modeling: (i) the inverse dynamics model (IDM) is not explicitly encouraged to distinguish action-related transitions from nuisance appearance, and (ii) the forward dynamics model (FDM) can underutilize the inferred latent action by exploiting predictive shortcuts from the current state. To address both limitations, we propose ACT-LAM, a lightweight action-centric framework that strengthens both action extraction and action utilization. Specifically, its Action Query IDM (AQ-IDM) employs learnable action queries and gated aggregation to selectively extract rich action-related transition cues without strong information bottlenecks. And its Action Token FDM (AT-FDM) projects latent actions into action tokens that progressively interact with evolving state representations, enabling continuous state-aware action conditioning. ACT-LAM further streamlines feature processing to concentrate model capacity on latent dynamics modeling. Extensive experiments on several robotic datasets and the VP2 benchmark demonstrate stronger latent action consistency, forward dynamics, and downstream visual planning performance with fewer trainable parameters and lower computational overhead. In particular, ACT-LAM surpasses the previous state of the art by \textbf{7.6%} on the aggregated VP2 success rate. Codes at url.
Dingjie Fu, Dianxing Shi, Yangyang Xu +1
Harbin Institute of Technology (Shenzhen) · Beihang University
Learning world models across diverse domains calls for a unified control space, yet ac- tion interfaces vary across domains and action annotations are often scarce. Latent ac- tions learned directly from videos offer a promising way to establish such a space without explicit action supervision. However, learning shared latent actions across domains re- mains challenging, as variations in appearance, embodiment, and dynamics can entangle domain-specific visual cues with transferable transition structure. To this end, we intro- duce OneLA, a simple framework that learns continuous latent actions through a shared orthogonal dictionary and a common predictive objective, without task-specific designs or cross-domain alignment losses during latent action pretraining. These latent actions pro- vide a shared conditioning interface for world models, with lightweight adapters mapping native control commands into the same space. Conditioned on latent actions and the cur- rent visual state, the world model predicts context-dependent transitions for controllable rollout. OneLA improves action readout across domains and reduces action cycle-transfer rotation error by 24.8% over the strongest evaluated baseline. A joint-training comparison further shows a 20.5% improvement in Human Preference Score v3 (HPSv3) for real- scene generation. These results suggest a path toward general-purpose world models that learn from diverse videos and support flexible control through latent action space.