Videos reveal how the world changes, but rarely the actions that cause those changes. Latent action learning recovers such supervision from visual transitions, yet often learns the underlying dynamics largely from scratch. We ask whether a pretrained video generator can instead provide the prior for both inverse and forward dynamics. We introduce O2LaF, built on a simple principle: one prior, two dynamics. We initialize both models from the same pretrained video DiT and adapt them to their distinct roles: a feed-forward transition encoder with learnable action queries for inverse dynamics, and a causal, action-conditioned video generator for forward dynamics. By controlling information access and optimizing both dynamics end-to-end through a compact bottleneck, O2LaF learns transferable actions from unlabeled videos. Across navigation and robotic manipulation, O2LaF sets a new state of the art in few-shot action probing and enables stronger zero-shot action transfer across visual contexts and embodiments. Beyond generative video priors, we further extend our framework to predictive representation priors such as V-JEPA2, achieving competitive performance. Our findings suggest a broader paradigm for latent-action learning: repurposing pretrained video models to discover transferable actions and predict their consequences.
Figures & tables
Figure 2: Paradigms for self-supervised latent-action learning. Existing methods learn latent actions (a) from scratch [ 8 , 85 , 20 ] , (b) on frozen visual features [ 9 , 21 ] , (c) with a pretrained video generator as the FDM only [ 62 , 76 ] , or (d) with an external video prior as auxiliary supervision [ 35 ] . Ours repurposes one pretrained video DiT as both the IDM and the FDM.
Figure 3: Action-aware response in a pretrained video DiT. Cross-attention follows the active hands and manipulated regions as the interaction changes.
Figure 4: Action-query attention in IDM.
Figure 5: Overview of O 2 LaF. Two independently adapted video DiTs share a pretrained prior: the IDM infers latent actions, while the FDM predicts their visual consequences. Both employ frame-causal attention and role-specific noise sampling, with end-to-end training through latent actions.
Figure 6: Few-shot linear probing of frozen latent actions. O 2 LaF consistently outperforms prior latent-action representations on both settings across labeled-data budgets.
Method
1st-P
3rd-P
Visual Quality
Camera Motion
Visual Quality
Camera Motion
Img. Q. ↑
Temp. C. ↑
Trans. ↓
Rot. ↓
Img. Q. ↑
Temp. C. ↑
Trans. ↓
Rot. ↓
AdaWorld [ 20 ]
0.639
0.899
2.539
2.055
0.616
0.927
1.342
1.217
Olaf-World [ 35 ]
0.690
0.910
1.627
1.300
0.648
0.930
0.985
0.701
Ours
0.698
0.900
1.289
0.786
0.682
0.919
0.511
0.330
Ours (w/o CFG)
0.672
0.910
1.364
0.830
0.662
0.927
0.513
0.364
Table 1: Zero-shot cross-context camera navigation. We report VBench imaging quality (Img. Q.) and temporal consistency (Temp. C.), and ViPE-based translation NRMSE and rotation RPE. Methods above the separator use action CFG =3 . Best and second-best results are bold and underlined , respectively.
Figure 7: Zero-shot cross-context camera navigation. O 2 LaF better reproduces reference camera and agent action while preserving the target scene.
Method
Human
Robot
Self
Cross-cycle
Self
Cross-cycle
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
DreamDojo [ 19 ]
23.38
0.871
0.223
22.61
0.859
0.244
18.73
0.769
0.159
18.35
0.754
0.165
Ours
26.72
0.903
0.148
24.30
0.878
0.195
19.92
0.787
0.140
18.85
0.770
0.163
Table 2: Cross-embodiment action transfer via cycle reconstruction. Self reconstructs the source video directly, while Cross-cycle reconstructs it after transferring latent actions to the opposite embodiment and back ( H→R→H and R→H→R ). Best results are in bold .
Figure 8: Zero-shot cross-embodiment manipulation O 2 LaF better preserves object-directed motion while adapting to the target embodiment and visual context.
Table 3: Design choices for the two dynamics. MIND joint exact-match accuracy ( % , ↑ ) at k∈{1,50} . Panels (a,c) use Wan2.2-5B, while (b) studies Wan2.1 FDM scaling with a fixed 1.3B IDM.
Figure 9: Role-specific noise levels. IDM linear probing across a 5×5 grid of IDM and FDM flow times. The non-monotonic trends reveal distinct preferred noise configurations across motion domains.
IDM Backbone
Probing
1st-P World Model
3rd-P World Model
MIND ↑
Color R2L / a / b
VBench ↑
ViPE ↓
VBench ↑
ViPE ↓
1st-P
3rd-P
Img.
Temp.
Trans.
Rot.
Img.
Temp.
Trans.
Rot.
ST-Transformer
58.79
79.70
−0.419/−0.190/−0.530
0.666
0.901
1.449
1.022
0.636
0.924
0.620
0.478
VideoChat3
36.96
43.86
0.411/−0.147/0.364
–
–
–
–
–
–
–
–
V-JEPA2
88.62
96.14
0.514/0.816/0.742
0.657
0.901
1.349
0.847
0.664
0.916
0.439
0.337
Wan2.2
87.95
94.37
−0.297/0.081/0.223
0.698
0.900
1.289
0.786
0.682
0.919
0.511
0.330
Table 4: Comparison of O 2 LaF IDM backbones. Color R2 measures the linear decodability of mean reference-frame CIELAB ( L/a/b ) from latent actions. Positive values indicate appearance leakage, while non-positive values indicate no improvement over the mean predictor.
Figure 10: Cross-context action transfer with IDM priors. V-JEPA2 exhibits source-appearance leakage, whereas video DiT better preserves target appearance.
Configuration
1st-P
3rd-P
Probing ↑
VBench ↑
ViPE ↓
Probing ↑
VBench ↑
ViPE ↓
FDM noise
IDM grad.
k=1
k=50
Img.
Temp.
Trans.
Rot.
k=1
k=50
Img.
Temp.
Trans.
Rot.
End-to-end learning (causal FDM)
Mixed
All
72.81
87.95
0.649
0.895
1.937
0.993
52.08
94.37
0.654
0.913
0.845
0.481
Mixed
High
72.23
86.79
0.654
0.885
1.959
0.943
60.38
95.11
0.660
0.902
0.918
0.484
High
High
72.28
85.76
0.647
0.836
2.419
1.168
60.75
94.86
0.646
0.853
1.618
0.765
Table 5: Trade-offs between action learning and generation. End-to-end models vary FDM noise sampling ( Mixed : standard and high; High : high only) and IDM gradient routing ( All : all timesteps; High : high-noise only). Two-stage models freeze the Mixed/All IDM and compare pretrained versus E2E FDM initialization. Gray probing scores are shared and excluded from ranking.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
End-to-end
Two-stage FDM
Backbone
Wan2.2-5B ×2
Wan2.2-5B
Trainable weights
LoRA-128 + heads
Full FDM
IDM
Jointly trained
Frozen @ 5k
Attention
Frame-causal
Bidirectional FDM
RGB frames
41
81
Latent actions
10
20
Appendix
Table 6: Training configurations.
Figure 11: Where does the pairwise action query attend? For transition xt→xt+1 , action query qt jointly attends to spatial video tokens in both endpoint frames.
Training data
MIND Exact Match ↑
In-lab EEF R2↑
1st-P
3rd-P
Left arm
Right arm
Mira only
87.95
94.37
0.135
0.257
EgoDex only
76.70
86.60
0.192
0.363
EgoDexMix
76.70
91.66
0.227
0.386
Mira / EgoDexMix (70/30)
85.94
93.96
0.163
0.341
Appendix
Table 7: Effect of latent-action training data. All checkpoints are compared at training step 5k.
Model
Pretraining
Adaptation
Translation (Dolly/Pedestal/Truck)
Zoom
Rotation (Pan/Tilt/Roll)
Static
Avg.
ΔLA
In
Out
Up
Down
Right
Left
In
Out
Right
Left
Up
Down
CW
CCW
CameraBench reference
Qwen2.5-VL-72B
VLM
SFT
86.8
41.3
30.5
50.6
70.7
52.6
57.1
78.5
81.9
86.6
79.1
70.4
35.0
43.8
76.6
62.8
–
Without latent-action pretraining
Scratch IDM (270M)
None
Full FT- h
45.9
12.8
9.8
15.4
32.1
21.2
14.2
12.7
35.4
32.1
29.4
22.1
15.4
14.8
66.9
25.3
–
Video DiT IDM (5B)
Video
LoRA- h
93.9
72.4
54.5
73.5
81.4
77.3
68.4
76.1
91.7
90.2
81.9
71.1
88.7
90.3
87.7
79.9
–
Appendix
Table 8: Camera-motion annotation on CameraBench. Per-class AP ( ↑ ) on the official test set. Video / LA denote video-generative/latent-action pretraining. Frozen- μ / Frozen- h use MLP probes on latent action means/pre-Gaussian features. LoRA- h / Full FT- h additionally adapt the IDM. Blue and red rows highlight matched supervised comparisons at 270M and 5B scales, respectively. ΔLA reports the AP gain from action pretraining over matched baselines. Our results ensemble three independently trained seeds.
Figure 12: The FDM as a policy verifier on a GR-1 humanoid. Each case shows one trajectory in the physics simulator rebuilt from a single photo (top), the same motion with objects removed, from which latent actions are extracted (middle), and the FDM rollout from the real photo (bottom).