Latent-action world models learn actions from video without labels, and recent methods aim to share these actions across bodies. Transfer between 2D characters with different body structures remains underexplored. In this paper, we study whether an action learned from one character can make a character with a different body perform the same action. A cat, a blob and a human can all jump, but each moves its body in its own way. The usual transfer gain mixes body motion with screen movement and can assign a perfect score to a model that ignores its action. To address this, we introduce a transfer score that compares the correct action with wrong ones from the same source. We test five training objectives on a new synthetic benchmark of 2D cartoon characters with six body plans, the standard Sprites benchmark, and 431 selected animal and human animations from DeformingThings4D. With a fixed camera, a baseline that only copies the source's screen movement scores 0.71, as high as most learned models. In contrast, with a camera that follows the body, the best model reaches only 0.46 for an unseen animal family and 0.20 for humans. With this camera, a linear probe identifies the action class of unseen animal families with 66% to 86% balanced accuracy (chance 17%). The learned actions therefore retain information about which action is performed, but the correct action closes only part of the prediction error gap for a new character.
Figures & tables
Figure 1: A jump transferred from a quadruped to a blob not seen in training. The table lists the three parts of an action. The numbers in the last column summarise our tests on the animator-made animations and do not describe this example.
Figure 2: The latent-action model. Top, training without action labels. Bottom, the transfer test. f is run three times from the same target frame bt , once with each kind of action, and each prediction is compared with the same real frame bt+3 . The three errors give the score S of Section 3.4.
Figure 3: One run of a bear with the fixed camera (top) and the body camera (bottom). The body camera removes the movement across the image, and the motion that remains is centred on the body.
Figure 4: One jump on the six body plans, every second frame. From top to bottom, quadruped, biped, blob, limbless, multi-limbed and disjointed.
Figure 5: Four Sprites characters with the same body and different skin, clothing and hair. From top to bottom, walk to the right, spellcard to the front, slash to the left and slash to the right. Art from the Liberated Pixel Cup, CC BY-SA 3.0.
Figure 6: Animator-made animations drawn as 2D side views with the body camera, every second frame. From top to bottom, a bear, a dog, a deer and a rabbit jumping, a deer falling and a human walking.
Figure 7: The usual gain G and our score S for seen to held-out bodies on the synthetic benchmark, mean and standard deviation over 9 runs. A model that ignores its action (right) gets G=1.00 , and its S is undefined.
Figure 8: Transfer on the synthetic benchmark with the fixed camera (left) and the body camera (right), mean and standard deviation over held-out body plans and seeds.
Figure 9: Transfer on the animator-made animations, mean and standard deviation over the 5 held-out families.
Figure 10: Body-centred transfer of M4 from each held-out source family (rows) to each target family (columns). The diagonal is transfer within a family.
Figure 11: Action class read by a linear probe (blue) and body-centred transfer S (orange) for held-out bodies. Probe values are balanced accuracy on the animator-made animations and plain accuracy on the synthetic benchmark. The dotted line is the probe’s chance level.
Figure 12: A deer’s jump transferred to a bear (top) and a human (bottom). From top to bottom, the source, the target’s real jump, and the predictions of M1 to M5.
Figure 13: Body leak in the action (x axis) against body-centred transfer (y axis). Small points are runs and large points are model means.
Learning world models across diverse domains calls for a unified control space, yet ac- tion interfaces vary across domains and action annotations are often scarce. Latent ac- tions learned directly from videos offer a promising way to establish such a space without explicit action supervision. However, learning shared latent actions across domains re- mains challenging, as variations in appearance, embodiment, and dynamics can entangle domain-specific visual cues with transferable transition structure. To this end, we intro- duce OneLA, a simple framework that learns continuous latent actions through a shared orthogonal dictionary and a common predictive objective, without task-specific designs or cross-domain alignment losses during latent action pretraining. These latent actions pro- vide a shared conditioning interface for world models, with lightweight adapters mapping native control commands into the same space. Conditioned on latent actions and the cur- rent visual state, the world model predicts context-dependent transitions for controllable rollout. OneLA improves action readout across domains and reduces action cycle-transfer rotation error by 24.8% over the strongest evaluated baseline. A joint-training comparison further shows a 20.5% improvement in Human Preference Score v3 (HPSv3) for real- scene generation. These results suggest a path toward general-purpose world models that learn from diverse videos and support flexible control through latent action space.
Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by 300× (1,200→4 tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.
Despite the central role of action in embodied intelligence, learning transferable action representations from visual transitions remains a fundamental challenge, particularly when world models must generalize across embodiments under limited data. We argue that action is not merely an auxiliary conditioning signal, but a distinct representational factor that decouples the controllable change from embodiment-specific actuation. In this work, we propose SCAR, a joint inverse-forward dynamics framework for learning unified action representations across embodiments from visual transitions. Built on a pretrained generative backbone, SCAR uses an inverse dynamics model (IDM) to infer latent actions from latent observation pairs and a forward dynamics model (FDM) to predict future dynamics conditioned on them. To make the latent space transferable rather than a generic visual bottleneck, we regularize the latent action posterior toward a standard Gaussian prior to limit arbitrary visual encoding, and introduce adversarial invariance to suppress embodiment- and environment-specific nuisance factors. Experiments on the Procgen and Robotwin dataset show that the learned unified latent action representation serves as a stronger conditioning interface for world modeling than embodiment-specific raw actions, yielding improved cross-embodiment low-data adaptation and cross-task transfer. Taken together, these results suggest that action can be learned as a shared representation of controllable change across embodiments, providing an interface for more transferable and generalizable world models.
Hongjia Liu, Fan Feng, Minghao Fu +3
University of California, San Diego · KTH Royal Institute of Technology