LaLa-World: Scaling Latent World Models with Natural Language Actions

Junha Song; Geonmo Gu; Jeongeun Park; Heejeong Nam; Lucas Maes; Byeongho Heo; Dongyoon Han; Sangdoo Yun

Figure A: Comparison with representative world models and policy models. (a) LaLa-World uses natural language conditions, enabling training on control trajectories together with image and video transitions. The RAEv2-based WM is our generative world model baseline under matched settings in Section 4.2 . (b) Fine-tuned from LaLa-World, LaLa-Policy jointly predicts actions and future latents, whereas VLAs predict actions alone and Cosmos Policy predicts actions and the future with visual details.

Figure B: ImageNet training pairs and transformation retrieval. Left: a source image and a transformed view form a training pair, with the transformation described in language. Right: the predicted representation is used to rank 256 transformed views of the same validation image; a subset is illustrated. The outlined view is the target specified by the language action.

LaLa-World
Method	MRR $\uparrow$
IWM (default) †	$0.79$
IWM (strong jitter) †	$0.85$
Source latent	$0.024$
Predicted latents, empty language	$0.026$
Predicted latents, language action	$\mathbf{0.856}$

Appendix

Table A: ImageNet transformation retrieval. We report mean reciprocal rank (MRR) over 256 transformed views of each source image. The LaLa-World rows compare retrieval using the source latent with predicted latents conditioned on empty language or a language action. † Published IWM results ( Garrido et al., 2024 ) use different training and transformation settings.

Figure C: Communicable world modeling. An LLM takes the place of CEM and talks to LaLa-World in language. It reads the initial and goal observations, writes

N

candidate plans as language actions (top message), and receives only the goal-distance cost of each plan (bottom message), which is the same cost that CEM minimizes. After

K

rounds, the lowest-cost plan is executed. The LLM needs two orders of magnitude fewer world-model rollouts than CEM.

Planner	Wall	PointMaze	RoboCasa-R	DROID
CEM	$95.8\,(9{,}000)$	56.2 $\,(9{,}000)$	$40.6\,(54{,}000)$	$40.2\,(4{,}500)$
Claude Opus 5	$\mathbf{100.0}\,(30{\times}3{=}90)$	$31.2\,(30{\times}3{=}90)$	$\mathbf{53.1}\,(10{\times}2{\times}4{=}80)$	$36.2\,(20{\times}2{=}40)$
GPT-6 Astra	$91.3\,(8{\times}1{=}8)$	$26.1\,(8{\times}3{=}24)$	$17.4\,(8{\times}1{\times}12{=}96)$	$\mathbf{44.7}\,(16{\times}3{=}48)$

Appendix

Table B: MPC planning with an LLM actor in place of CEM. We report success rates (%), except for the DROID action score, with the number of world-model rollouts per trajectory in parentheses. For the LLM actors, the rollout count is written as

N\times K

, e.g.,

30\times 3=90

, where

N=30

is the number of candidate action sequences proposed per round and

K=3

is the number of rounds; for RoboCasa-R, the third factor is the number of replanning steps per trajectory. The LLM actors use 8–96 rollouts per trajectory, compared with 4,500–54,000 for CEM.

Figure D: Qualitative Results of Communicable World Modeling with an LLM Actor. Each row shows the initial and goal observations and the simulator observations after executing the lowest-cost sequence written by the LLM, with its language action below each frame. Below the goal observation, we list the cost of the best sequence at each round, which decreases as the LLM revises its plans from the returned costs.

Figure E: Language sensitivity across training data and predictor backbones. (a) Language conditions used to evaluate the same target transition; red words mark semantic swaps. (b) Expected score pattern for a predictor that follows language. (c) Effect of adding image and video transitions to control data. (d) Comparison of language model backbones.

Figure F: Predictor backbone, initialization, and architecture ablations. (a) Validation loss for language model backbones, including pretrained and randomly initialized Qwen3.5. (b) Language reliance

S(z)-S(\varnothing)

for positional encoding and readout variants. 3D input positional embeddings with learned queries yield the highest reliance.

Figure G: Predictor positional encoding and readout. (a) Projected state patches, language tokens and copies of one shared learned query vector receive additive 3D positional embeddings. The prediction head maps query outputs to future patch representations. (b) Native 1D uses sequence indices for visual and text tokens, handled by the backbone’s positional mechanism. MRoPE uses 3D coordinates to rotate attention queries and keys inside the backbone. Inplace readout uses state-token hidden states.

Initialization	Training scope	Wall	PointMaze	Push-T	DROID	RoboCasa-R	RoboCasa-P	Average
Pretrained LLM	Single-domain	$\mathbf{97.9}$ $(1.5)$	$\mathbf{66.7}$ $(4.8)$	$70.8$ $(4.6)$	$38.3$ $(4.5)$	$21.9$ $(7.3)$	$31.2$ $(8.2)$	$54.5$ $(2.3)$
Random init.	Joint	$\mathbf{97.9}$ $(1.5)$	$50.0$ $(5.1)$	$79.2$ $(4.1)$	$31.6$ $(3.8)$	$31.2$ $(8.2)$	$43.8$ $(8.8)$	$55.6$ $(2.4)$
Pretrained	Joint	$95.8$ $(2.0)$	$56.2$ $(5.1)$	$\mathbf{81.2}$ $(4.0)$	$\mathbf{40.2}$ $(5.2)$	$\mathbf{40.6}$ $(8.7)$	$\mathbf{46.9}$ $(8.8)$	$\mathbf{60.2}$ $(2.5)$

Appendix

Table C: LaLa-World Predictor initialization and joint training. We compare pretrained and random initialization, and single-domain and joint training. We report success rates (%), except for the DROID action score, with standard errors in parentheses. Average is the unweighted mean of the six benchmark scores.

Vision encoder	Wall	PointMaze	Push-T	MetaW.-R	MetaW.-RW	Rope	Granular	RoboC.-R	RoboC.-P	DROID	Average
JEPA-WMs
DINOv2-S/14	$\mathbf{79.2}$	$\mathbf{60.4}$	$\mathbf{68.8}$	$20.8$	$4.2$	$54.2$	$\mathbf{50.0}$	$\mathbf{37.5}$	$\mathbf{31.2}$	$40.1$	$44.6$
DINOv3-L/16	$76.0$	$56.2$	$47.9$	$\mathbf{66.7}$	$\mathbf{47.9}$	$\mathbf{62.5}$	$41.7$	$21.9$	$28.1$	$\mathbf{43.0}$	$\mathbf{49.2}$
Table 1 (reference)	$79.2$	$60.4$	$68.8$	$66.7$	$47.9$	$62.5$	$50.0$	$37.5$	$31.2$	$43.0$	$54.7$
LaLa-World
DINOv2-S/14	$95.8$	$45.8$	$\mathbf{81.2}$	$8.3$	$4.2$	$58.3$	$58.3$	$40.6$	$43.8$	$40.2$	$47.7$

Appendix

Table D: Vision encoder ablations. We compare frozen vision encoders for JEPA-WMs and LaLa-World. DINOv2 + DINOv3 uses both DINOv2-S/14 and DINOv3-L/16. We report success rates (%), except for the DROID action score. Average is the unweighted mean of the ten benchmark scores. Bold denotes the highest score among encoder configurations within each method; reference rows reproduce the main planning results and are excluded from this comparison.

Language action	Wall	PointMaze	Push-T	MetaWorld-R	MetaWorld-RW	RoboCasa-R	RoboCasa-P
Qualitative	$83.3$ $(3.8)$	$47.9$ $(5.1)$	$4.2$ $(2.0)$	$70.8$ $(6.6)$	$60.4$ $(7.1)$	$28.1$ $(7.9)$	$25.0$ $(7.7)$
Numerical	$95.8$ $(2.0)$	$56.2$ $(5.1)$	$81.2$ $(4.0)$	$75.0$ $(6.2)$	$52.1$ $(7.2)$	$40.6$ $(8.7)$	$46.9$ $(8.8)$

Appendix

Table E: MPC planning with numerical and qualitative language actions. Qualitative language actions replace numerical magnitudes with slightly , no modifier, or far . Lower success rates are expected when exact magnitudes are omitted, yet it is still interesting that LaLa-World can plan successfully in several domains from qualitative language actions. Only the language conditions supplied to the predictor change; CEM samples numerical actions, and the simulator executes the selected actions in their original numerical form.

Phrasing	Wall	PointMaze	MetaWorld-R	RoboCasa-R / DROID
Original	move the agent	push the ball with force	move the gripper	move gripper, rotate roll
Rephrased	push the dot	push the sphere	shift the robot hand	move robot hand, turn roll

Appendix

Table F: Robustness to action phrasing. We report planning results with original and paraphrased action templates (bottom) and the substitutions used at evaluation (top). All data combines control trajectories with image and video transitions. We report success rates (%), except for the DROID action score, with standard errors in parentheses.

Figure H: Domain-level comparison of latent and generative world models for MPC planning. We compare LaLa-World with RAEv2 using one and five denoising steps under the matched setup in Section 4.2 . We report success rates (%) for simulation tasks and the action score for DROID.

Dataset	Training data size
Wall	86,296
PointMaze	186,675
Push-T	2,199,488
MetaWorld	1,162,310
Rope	18,525
Granular	18,582

Appendix

Table G: Data used to train LaLa-World. We count target transitions for control and video data and editing pairs for OmniEdit and ShareGPT-4o-Image. Counts exclude validation data and count each control transition once across its two action description formats.

Benchmark	Evaluation trajectories	Trajectory length	Planning horizon $H$	Executed steps per plan	Substeps per step	CEM settings	Evaluation criterion
Wall	96	6	6	6	5	$30\times 300\times 10$	Position error $<4.5$
PointMaze	96	6	6	6	5	$30\times 300\times 10$	Position error $<0.5$
Push-T	96	6	6	6	5	$30\times 300\times 10$	Position error $<20$ and angle error $<20^{\circ}$
MetaWorld-R	48	20	6	3	5	$15\times 300\times 10$	Success at any step
MetaWorld-RW	48	20	6	3	5	$15\times 300\times 10$	Success at any step
Rope	24	5	5	1	1	$30\times 300\times 30$	Chamfer distance $<0.675$

Appendix

Table H: MPC planning settings and evaluation criteria. One step corresponds to one world-model transition, with the listed number of action updates (substeps). CEM settings are iterations

\times

candidate sequences per iteration

\times

retained sequences. For benchmarks requiring replanning, we execute the listed number of steps, observe the environment, and re-optimize the action sequence with CEM. Position and angle errors are measured at the final step, using

L_{2}

distance for position. DROID is evaluated offline without simulator execution.

Figure I: MPC planning horizons and action execution. One step corresponds to one world-model transition, with the listed number of action updates (substeps). For benchmarks requiring replanning, we obtain a new observation after executing the prefix and optimize a new sequence with CEM.

Figure J: Substep aggregation for language conditioning and action execution. (a) Summing across substeps reduces the action dimension per step from 10 to 2 for language conditioning. (b) Within MPC, CEM optimizes these sums, which are distributed equally across five substeps for execution. This discards within-step variation. Yet, it is notable that LaLa-World achieves accurate prediction and high planning success rates.

Figure K: MPC planning with qualitative language actions. (a) Each magnitude maps to one of three equal-width bins, and the direction is kept. (b) LaLa-World evaluates CEM samples through their qualitative descriptions, while the simulator executes the selected actions with their numerical values. Thus, LaLa-World plans without exact action magnitudes.

Figure L: LaLa-Policy architecture. The predictor, fine-tuned from LaLa-World, receives encoded observations, a task instruction, and learned action and future queries. It predicts future visual latents and provides conditioning features for an action generator, which produces an action chunk from Gaussian noise. Proprio and intermediate projections are omitted for clarity.

Figure M: Numerical language actions on Wall. Initial and goal observations bracket decoded future predictions (top) and PCA visualizations of predicted latents (bottom), with the selected language actions below each frame.

Figure N: Numerical language actions on PointMaze. Each row shows initial and goal observations and decoded future predictions under the selected actions, which specify pushing forces and directions.

Figure O: Numerical language actions on Push-T. Initial and goal observations bracket decoded future predictions under the selected pusher movements, with numerical displacements described below each frame.

Figure P: Numerical language actions on MetaWorld. Decoded predictions are shown before and after replanning from the simulator observation

x_{t+3\Delta}

, alongside initial and goal observations.

Figure Q: Numerical language actions on DROID. Each row shows an initial observation, three decoded future predictions, and the goal observation, with language actions below.

Figure R: Numerical language actions on RoboCasa Reach. Decoded future predictions under selected gripper actions are shown between initial and goal observations. RoboCasa is excluded from world-model training.

Figure S: Numerical language actions on RoboCasa Place. Initial and goal observations bracket decoded future predictions during object placement, with the selected gripper actions below each frame.

Figure T: Qualitative language actions on Wall. Simulator observations show the agent passing through the opening toward the goal. Actions describe directions and relative movement magnitudes.

Figure U: Qualitative language actions on PointMaze. Simulator observations show the ball moving through the maze under MPC, with pushing directions and qualitative force descriptions below the frames.

Figure V: Qualitative language actions on Push-T. Initial and goal observations bracket simulator states during MPC, with the pusher’s direction and relative displacement described below each frame.

Figure W: Qualitative language actions on MetaWorld. Simulator observations show reaching trajectories planned using descriptions of gripper motion and grip adjustment without exact numerical magnitudes.

Figure X: Qualitative language actions on RoboCasa Reach. Simulator observations show zero-shot MPC execution toward the goal, conditioned on qualitative descriptions of gripper translation, rotation, and opening or closing.

Figure Y: Qualitative language actions on RoboCasa Place. Simulator observations show an object-placement trajectory under MPC, with qualitative gripper actions and the goal observation for reference.

Figure Z: LIBERO Spatial: bowl on a ramekin. LaLa-Policy moves the bowl from the ramekin onto the plate. Primary and wrist views compare simulator observations (upper rows) with decoded future predictions (lower rows).

Figure AA: LIBERO Spatial: bowl next to a cookie box. LaLa-Policy moves the specified bowl onto the plate. Simulator observations and decoded future predictions are paired for the primary and wrist views.

Figure AB: LIBERO Object: alphabet soup. LaLa-Policy picks up the alphabet soup and places it in the basket. Each camera view pairs observed states with decoded future predictions.

Figure AC: LIBERO Object: salad dressing. LaLa-Policy transfers the salad dressing to the basket. Primary and wrist views show simulator observations above the corresponding decoded future predictions.

Figure AD: LIBERO Goal: bowl on the cabinet. LaLa-Policy places the bowl on top of the cabinet. Primary and wrist views compare observed states with decoded future predictions throughout execution.

Figure AE: LIBERO Goal: bowl inside the drawer. LaLa-Policy opens the top drawer and places the bowl inside. Both camera views pair simulator observations with decoded future predictions.

Figure AF: LIBERO Long: placing two objects in a basket. LaLa-Policy transfers the cream cheese box and butter to the basket. Observations and decoded future predictions are shown for both camera views.

Figure AG: LIBERO Long: stove and moka pot. LaLa-Policy turns on the stove and places the moka pot on it. Primary and wrist views compare simulator observations with decoded future predictions.

Figure AH: RoboCasa: pressing the coffee-machine button. Simulator observations (upper rows) and decoded future predictions (lower rows) show LaLa-Policy pressing the button, from two external views and the wrist view.

Figure AI: RoboCasa: serving a mug. LaLa-Policy moves the mug from beneath the coffee-machine dispenser to the counter. Each of the three views compares observed states with decoded future predictions.

Figure AJ: RoboCasa: opening a microwave. Simulator observations and decoded future predictions show LaLa-Policy opening the microwave door, from the primary, secondary, and wrist views.

Figure AK: RoboCasa: opening a drawer. LaLa-Policy opens the left drawer. The three camera views pair simulator observations with decoded future predictions during reaching and drawer opening.

Figure AL: RoboCasa: counter-to-cabinet transfer. LaLa-Policy picks up the bar from the counter and places it in the cabinet. Each camera view compares observed states with decoded future predictions.

Figure AM: RoboCasa: pan-to-plate transfer. LaLa-Policy moves the cheese from the pan onto the plate. Observations and decoded future predictions are shown for two external views and the wrist view.

Figure AN: RoboCasa: turning on a stove burner. Simulator observations and decoded future predictions show LaLa-Policy turning on the front-left burner, from the primary, secondary, and wrist views.

Method	Joint-training	Wall	PointMaze	Push-T	MetaWorld-R	MetaWorld-RW	Average
DreamerV3 ( 2025a )	✗	$39.6$ $(5.0)$	$43.8$ $(5.1)$	$32.3$ $(4.8)$	$25.0$ $(6.2)$	$10.4$ $(4.4)$	$30.2$ $(2.3)$
TD-MPC2 ( 2024 )	✗	$28.1$ $(4.6)$	$8.3$ $(2.8)$	$16.7$ $(3.8)$	$12.5$ $(4.8)$	$22.9$ $(6.1)$	$17.7$ $(2.0)$
IRIS ( 2023 )	✗	$30.2$ $(4.7)$	$41.7$ $(5.0)$	$31.2$ $(4.7)$	$20.8$ $(5.9)$	$6.2$ $(3.5)$	$26.0$ $(2.2)$
LeWorldModel ( 2026a )	✗	$24.0$ $(4.4)$	$24.0$ $(4.4)$	$59.4$ $(5.0)$	$16.7$ $(5.4)$	$16.7$ $(5.4)$	$28.1$ $(2.2)$
DINO-WM ( 2025 )	✗	$90.6$ $(3.0)$	$59.4$ $(5.0)$	$65.6$ $(4.8)$	$52.1$ $(7.2)$	$43.8$ $(7.2)$	$62.3$ $(2.5)$
JEPA-WMs ( 2025 )	✗	$79.2$ $(4.1)$	$\mathbf{60.4}$ $(5.0)$	$68.8$ $(4.7)$	$66.7$ $(6.8)$	$47.9$ $(7.2)$	$64.6$ $(2.6)$

	LIBERO	RoboCasa
Cosmos Policy	$4.45$ s	$11.71$ s
LaLa-Policy	$\mathbf{0.86}$ s	$\mathbf{2.12}$ s
LaLa-Policy	( $\times 5.2$ )	( $\times 5.5$ )

LaLa-World: Scaling Latent World Models with Natural Language Actions

Abstract

Figures & tables

Appendix

World Model Learning

Grounded World Model: Latent Planning with Language Goals

RoboJEPA: Scaling Robotic Latent World Models

World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

Abstract

World Model Learning

Explore similar work

Grounded World Model: Latent Planning with Language Goals

RoboJEPA: Scaling Robotic Latent World Models

World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models