cs.CVNov 19, 2026

One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning

Authors: Yuxin Jiang, Wenqiang Sun, Jiaqi Li, Yuchao Gu, Lvmin Zhang, Ivor Tsang, Mike Zheng Shou

Organizations: Show Lab, National University of Singapore · A*STAR, Singapore · Tencent · Stanford University

Abstract

Videos reveal how the world changes, but rarely the actions that cause those changes. Latent action learning recovers such supervision from visual transitions, yet often learns the underlying dynamics largely from scratch. We ask whether a pretrained video generator can instead provide the prior for both inverse and forward dynamics. We introduce O2^{2}LaF, built on a simple principle: one prior, two dynamics. We initialize both models from the same pretrained video DiT and adapt them to their distinct roles: a feed-forward transition encoder with learnable action queries for inverse dynamics, and a causal, action-conditioned video generator for forward dynamics. By controlling information access and optimizing both dynamics end-to-end through a compact bottleneck, O2^{2}LaF learns transferable actions from unlabeled videos. Across navigation and robotic manipulation, O2^{2}LaF sets a new state of the art in few-shot action probing and enables stronger zero-shot action transfer across visual contexts and embodiments. Beyond generative video priors, we further extend our framework to predictive representation priors such as V-JEPA2, achieving competitive performance. Our findings suggest a broader paradigm for latent-action learning: repurposing pretrained video models to discover transferable actions and predict their consequences.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling

    Sep 14, 2026Dingjie Fu, Dianxing Shi, Yangyang Xu +1Latent Action LearningLatent Dynamics Modeling

  2. OneLA: Learning Many Worlds from Latent Actions

    Oct 14, 2026Ziyu Zhu, Zhaoxu Sun, Fan Jiang +3World Model LearningAction-Conditioned World Models