OneLA: Learning Many Worlds from Latent Actions

Ziyu Zhu; Zhaoxu Sun; Fan Jiang; Chiyu Wang; Junfeng Ni; Mu Xu

cs.CVOct 14, 2026

OneLA: Learning Many Worlds from Latent Actions

Authors: Ziyu Zhu, Zhaoxu Sun, Fan Jiang, Chiyu Wang, Junfeng Ni, Mu Xu

Organizations: AMAP, Alibaba Group · Tsinghua University

Abstract

Learning world models across diverse domains calls for a unified control space, yet ac- tion interfaces vary across domains and action annotations are often scarce. Latent ac- tions learned directly from videos offer a promising way to establish such a space without explicit action supervision. However, learning shared latent actions across domains re- mains challenging, as variations in appearance, embodiment, and dynamics can entangle domain-specific visual cues with transferable transition structure. To this end, we intro- duce OneLA, a simple framework that learns continuous latent actions through a shared orthogonal dictionary and a common predictive objective, without task-specific designs or cross-domain alignment losses during latent action pretraining. These latent actions pro- vide a shared conditioning interface for world models, with lightweight adapters mapping native control commands into the same space. Conditioned on latent actions and the cur- rent visual state, the world model predicts context-dependent transitions for controllable rollout. OneLA improves action readout across domains and reduces action cycle-transfer rotation error by 24.8% over the strongest evaluated baseline. A joint-training comparison further shows a 20.5% improvement in Human Preference Score v3 (HPSv3) for real- scene generation. These results suggest a path toward general-purpose world models that learn from diverse videos and support flexible control through latent action space.

Figures & tables

Figure 1: OneLA across many worlds. A shared latent action interface connects video prediction to the different control signals of driving, robotics, games, and navigation. The inset boxes at the bottom of each panel show frames from model-generated rollouts.

Figure 2: Video distribution for latent action pretraining. (a) Clip counts grouped by video category. (b) Example frame pairs from games, navigation, driving, and robotics.

Figure 3: Overview of OneLA . (a) Orthogonal latent action learning from frame pairs. (b) Joint video and action prediction conditioned on inferred latent actions or mapped native controls.

(a) Linear probe (all dimensions)
Method	Game	Sekai	MIND	nuScenes	RT-1	AgiBot
OLaF [ 35 ]	61.78	53.84	81.41	87.22	60.02	48.90
AdaWorld [ 23 ]	58.95	50.63	79.68	80.84	61.58	45.41
OLaF w/o align (VAE) [ 35 ]	56.43	48.02	75.57	78.39	58.59	43.87
DiLA [ 66 ]	40.55	34.90	57.94	59.05	56.32	46.53
RLA-WM [ 67 ]	45.26	26.33	64.48	49.58	52.20	39.09

Table 1: Action information in frozen LAM representations. Macro-F1 (%,

\uparrow

) for the three readout settings. Boldface and underlining indicate the best and second-best scores.

	nuScenes			RT-1
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
OLaF [ 35 ]	15.92	0.601	0.525	18.78	0.785	0.321
AdaWorld [ 23 ]	10.08	0.208	0.801	17.16	0.743	0.367
DiLA [ 66 ]	15.49	0.604	0.538	17.28	0.734	0.370
OneLA	16.13	0.625	0.533	17.78	0.771	0.338

Table 2: Video prediction and action cycle transfer. Video prediction uses mapped native commands; cycle transfer uses video-derived latent actions. SSIM [ 57 ] and LPIPS [ 65 ] measure fidelity; IQ is MUSIQ-SPAQ [ 36 ] ; RPE denotes translation (

t

) and rotation (

r

, degrees) error. Boldface and underlining indicate the best and second-best scores.

Figure 4: Cross-domain action prototypes with and without QR. Cosine similarity between action-conditioned posterior means, across MIND viewpoints (left) and AgiBot/RT-1 robots (right). Rows and columns denote action classes.

Figure 5: World-model training comparisons. (a–c) With identical data and training procedures, w/ and w/o latent action use LAM-inferred and zero action vectors for unlabeled videos, respectively. Real-scene HPSv3 averages 16 generated frames per video over 30 scenes with fixed per-case target-scene text captions. (d) Latent-action prediction loss, with 50 cases per setting.

Layers	Params (M)	Game	nuScenes	RT-1
6	180.9	55.94	71.29	57.81
12	357.2	62.38	81.59	59.16
24	709.8	65.35	84.02	60.05

Table 3: Model size and data amount. Linear-probe macro-F1 (%,

\uparrow

). Data amount denotes cumulative sampled frame pairs. Boldface and underlining mark best and second-best scores.

Training sources				Evaluation datasets
G Games	V Real videos	D Driving	R Robotics	Game	Sekai	MIND	nuScenes	RT-1	AgiBot
$\checkmark$	$\times$	$\times$	$\times$	58.41	44.60	71.16	73.64	58.79	39.18
$\checkmark$	$\checkmark$	$\times$	$\times$	64.82	54.44	82.49	83.84	58.76	49.79
$\checkmark$	$\checkmark$	$\checkmark$	$\times$	65.14	55.60	83.74	84.93	58.89	50.79
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	66.34	53.87	84.58	87.49	61.08	54.42

Table 4: Training-data composition. Linear-probe macro-F1 (%,

\uparrow

). Boldface and underlining indicate the best and second-best scores.

Figure 6: Reference-based action transfer. Source (top) and Transfer (bottom) show matched relative frame indices; each transfer rollout starts from its first displayed frame.

Figure 7: Qualitative rollouts across four domains. Each row shows an input frame followed by five generated frames.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Source	Retained (%)	Clips	Share (%)
Egocentric	Ego4D	100	242,712	13.53
	EgoDex	100	7,136	0.40
	Daily activities	5	159,833	8.91
	In the wild	100	15,203	0.85
Games	ABot-World	100	333,201	18.58
Driving	NVIDIA driving	100	306,912	17.11

Appendix

Table 5: Latent action pretraining video composition. Retained clips and their shares after source-specific subsampling.

Source	Sample count	Labeled count
ABot-World	42,422	8,484
AgiBot	16,204	3,241
nuScenes	1,250	250
RT-1	4,490	898
DL3DV	1,674	0
EgoDex	2,548	0

Appendix

Table 6: World-model training data and native-label usage. Expected sample counts over 6,000 updates with batch size 16, rounded to integers. Labeled counts are before action dropout.

Dataset	Fit	Test	Action labels
Game	800	1,600	8 binary keyboard controls
Sekai	607	1,222	6 evaluated motion labels
MIND	1,500	1,500	8 actions; first-/third-person views
nuScenes	2,342	2,441	4 binary motion directions
RT-1	1,640	1,650	14 signed motion/gripper labels
AgiBot	1,600	1,600	28 signed labels across two arms

Appendix

Table 7: Frozen-representation evaluation sets. Counts refer to frame pairs. Binary labels are predicted independently. Sekai macro-F1 averages the six labels with positive test examples; I/K have no positive examples.

Figure 8: Additional OneLA scene-control rollouts. Six frames per case illustrate motion under the indicated keyboard controls.

Figure 9: Additional OneLA scene-control rollouts in natural and built environments. Each row follows the indicated keyboard control.

Figure 10: Additional OneLA rollouts in games. Examples include first- and third-person views, with six frames sampled from each rollout.

Figure 11: Additional OneLA driving rollouts. Examples cover vehicle following, open roads, a road bend, and an industrial passage.

Figure 12: Additional OneLA navigation rollouts. Examples show camera motion through outdoor and indoor scenes.

Figure 13: Additional OneLA robot rollouts. Examples show interactions with a drawer, a can, a snack bag, and a sponge.

Figure 14: Additional OneLA reference-action transfer results. Each case pairs a source video (top) with a generated rollout in a different scene (bottom). Columns show matched temporal positions.

Figure 15: Enlarged view of Figure 4 . Diagonal annotations give the cosine similarity between prototypes with matching action labels.

World Model Learning

World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.

See all 414 papers

LaLa-World: Scaling Latent World Models with Natural Language Actions
Oct 20, 2026Junha Song, Geonmo Gu, Jeongeun Park +5World Model Learning Latent World Models
NAVER AI Lab · Georgia Institute of Technology · Brown University +1
World2Act: Latent Action Post-Training from World Model Dynamics
Mar 11, 2026An Dinh Vuong, Tuan Van Vo, Abdullah Sohail +6VLA Model Post-Training Vision-Language-Action Models
MBZUAI
WLA $^3$ : World Latent Action Modeling for Semantics, Dynamics, and Kinematics
Sep 14, 2026Peidong Liu, Zhiyuan Xiang, Mingyang Li +4Latent Action Learning World Models for Robotics
Joy Future Academy, JD Group

Oct 20, 2026J/K move · Enter open · S save

LaLa-World: Scaling Latent World Models with Natural Language Actions

Junha Song, Geonmo Gu, Jeongeun Park, Heejeong Nam+4

NAVER AI Lab · Georgia Institute of Technology · Brown University+1

World Model Learning Latent World Models World Model-Based Planning Efficient Inference for World Action Models

Open paper AskPDF arXiv

This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning. To this end, we introduce LaLa-World, a Language-conditioned Latent World model that learns from simulation and robot trajectories together with broader image and video data. Two designs make this work. First, we describe both numerical actions and visual changes in language, enabling joint training of a single future predictor across heterogeneous datasets. Second, we build the predictor on a language model, allowing it to interpret language conditions while predicting future latent states. The resulting predictor achieves higher average MPC planning performance than existing LWMs trained separately for each domain or task. Adding image and video data further improves average planning performance and robustness to action phrasing. In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with

6\times

less training cost and

13.9\times

faster planning. Beyond planning, the pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations. LaLa-Policy performs competitively with Cosmos Policy on LIBERO and RoboCasa, with up to

5.5\times

faster inference.

Abstract

World Model Learning

Explore similar work

LaLa-World: Scaling Latent World Models with Natural Language Actions

World2Act: Latent Action Post-Training from World Model Dynamics

WLA3^33: World Latent Action Modeling for Semantics, Dynamics, and Kinematics

WLA $^3$ : World Latent Action Modeling for Semantics, Dynamics, and Kinematics