cs.CVOct 20, 2026
SaveLaLa-World: Scaling Latent World Models with Natural Language Actions
Organizations: NAVER AI Lab · Georgia Institute of Technology · Brown University · Mila & Université de Montréal
Abstract
This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning. To this end, we introduce LaLa-World, a Language-conditioned Latent World model that learns from simulation and robot trajectories together with broader image and video data. Two designs make this work. First, we describe both numerical actions and visual changes in language, enabling joint training of a single future predictor across heterogeneous datasets. Second, we build the predictor on a language model, allowing it to interpret language conditions while predicting future latent states. The resulting predictor achieves higher average MPC planning performance than existing LWMs trained separately for each domain or task. Adding image and video data further improves average planning performance and robustness to action phrasing. In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with less training cost and faster planning. Beyond planning, the pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations. LaLa-Policy performs competitively with Cosmos Policy on LIBERO and RoboCasa, with up to faster inference.
Figures & tables
Figure 1: Overview of LaLa-World. (a) Language provides a shared interface for numerical actions and visual change, enabling joint training on control and video data. (b) A single predictor built on a language model uses encoded observations and language conditions to predict future visual representations. Fine-tuning the pretrained LaLa-World yields LaLa-Policy, which predicts actions as well as future visual representations. (c) The jointly trained predictor achieves higher average MPC planning performance than both separately trained LWM baselines and the generative world model baseline, RAEv2. (d) LaLa-Policy performs competitively with our adapted Cosmos Policy baseline on RoboCasa, while reducing average inference time per trajectory by a factor of .
Figure 2: MPC planning with language actions. LaLa-World imagines future representations under candidate actions with MPC and selects the most promising action toward the goal. Because all actions share one language interface, a single LaLa-World predictor can plan across multiple domains, despite differences in action dimensionality and physical meaning. The predictor also supports language-based actions without numerical values (bottom two rows), as detailed in Section D.4.3 . The visualized observations result from executing the selected actions in the simulator.
Figure 3: Examples of language-conditioned transition data. Each example consists of past observations, a language condition, and the next observation. Language conditions describe either numerical control actions from simulation and robotics or visual changes in images and videos. This shared interface enables training a single LWM across different action spaces and data sources.
| Method | Joint-training | Wall | PointMaze | Push-T | MetaWorld-R | MetaWorld-RW | Average |
|---|---|---|---|---|---|---|---|
| DreamerV3 ( 2025a ) | ✗ | ||||||
| TD-MPC2 ( 2024 ) | ✗ | ||||||
| IRIS ( 2023 ) | ✗ | ||||||
| LeWorldModel ( 2026a ) | ✗ | ||||||
| DINO-WM ( 2025 ) | ✗ | ||||||
| JEPA-WMs ( 2025 ) | ✗ |
Table 1: MPC planning results. We compare LaLa-World with existing world-model baselines across simulation and robot domains. LaLa-World achieves the highest average score in both tables. We report success rates (%), except for the DROID action score ( Terver et al., 2025 ) , with standard errors in parentheses. RoboCasa is evaluated zero-shot; -R, -RW, and -P denote Reach, Reach-Wall, and Place, respectively. Joint-training denotes a single predictor trained across domains. Additionally, scores in Figure 1 (c) are averaged over the benchmarks used to compare LaLa-World with the generative world model baseline ( Section 4.2 ).
Figure 4: Benefits of image /video data inclusion.
Figure 5: Latent vs. generative world models for MPC planning under matched setups. We match the vision encoder, language conditions, and training data to compare the two world models. (a) We track planning performance over training time for full-depth predictors under a 12-hour training budget. (b) We compare planning performance and average planning time per trajectory across four predictor depths, with all models trained on the same number of transition data. These results show that LaLa-World achieves higher planning performance than RAEv2 with lower training cost and planning time.
Figure 6: Visualization of predicted future states during MPC planning on DROID. We visualize decoded predictions from LaLa-World and RAEv2, using one or five denoising steps for RAEv2. Each example shows the initial and goal observations, the predicted rollout under the selected language actions, and planning time. Decoded images from LaLa-World may have lower visual quality because it is not trained for pixel reconstruction. Both models predict plausible future states, while LaLa-World requires less planning time.
Figure 7: Predicted future observations on LIBERO and RoboCasa. Each example shows the initial observation and task instruction, followed by simulator observations during policy execution (upper row) and decoded future predictions (lower row). LaLa-Policy predicts , and the simulator executes to yield ; see Equation 3 . Since LaLa-Policy is not trained for pixel reconstruction, decoded images may contain visual inaccuracies.
| LIBERO | RoboCasa | |
|---|---|---|
| Cosmos Policy | s | s |
| LaLa-Policy | s | s |
| ( ) | ( ) |
Table 4: Avg. inference time per trajectory and speedup.
Appendix figures & tables48 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A: Comparison with representative world models and policy models. (a) LaLa-World uses natural language conditions, enabling training on control trajectories together with image and video transitions. The RAEv2-based WM is our generative world model baseline under matched settings in Section 4.2 . (b) Fine-tuned from LaLa-World, LaLa-Policy jointly predicts actions and future latents, whereas VLAs predict actions alone and Cosmos Policy predicts actions and the future with visual details.
Figure B: ImageNet training pairs and transformation retrieval. Left: a source image and a transformed view form a training pair, with the transformation described in language. Right: the predicted representation is used to rank 256 transformed views of the same validation image; a subset is illustrated. The outlined view is the target specified by the language action.
| Method | MRR |
|---|---|
| IWM (default) † | |
| IWM (strong jitter) † | |
| LaLa-World | |
| Source latent | |
| Predicted latents, empty language | |
| Predicted latents, language action | |
Table A: ImageNet transformation retrieval. We report mean reciprocal rank (MRR) over 256 transformed views of each source image. The LaLa-World rows compare retrieval using the source latent with predicted latents conditioned on empty language or a language action. † Published IWM results ( Garrido et al., 2024 ) use different training and transformation settings.
Figure C: Communicable world modeling. An LLM takes the place of CEM and talks to LaLa-World in language. It reads the initial and goal observations, writes candidate plans as language actions (top message), and receives only the goal-distance cost of each plan (bottom message), which is the same cost that CEM minimizes. After rounds, the lowest-cost plan is executed. The LLM needs two orders of magnitude fewer world-model rollouts than CEM.
| Planner | Wall | PointMaze | RoboCasa-R | DROID |
|---|---|---|---|---|
| CEM | 56.2 | |||
| Claude Opus 5 | ||||
| GPT-6 Astra |
Table B: MPC planning with an LLM actor in place of CEM. We report success rates (%), except for the DROID action score, with the number of world-model rollouts per trajectory in parentheses. For the LLM actors, the rollout count is written as , e.g., , where is the number of candidate action sequences proposed per round and is the number of rounds; for RoboCasa-R, the third factor is the number of replanning steps per trajectory. The LLM actors use 8–96 rollouts per trajectory, compared with 4,500–54,000 for CEM.
Figure D: Qualitative Results of Communicable World Modeling with an LLM Actor. Each row shows the initial and goal observations and the simulator observations after executing the lowest-cost sequence written by the LLM, with its language action below each frame. Below the goal observation, we list the cost of the best sequence at each round, which decreases as the LLM revises its plans from the returned costs.
Figure E: Language sensitivity across training data and predictor backbones. (a) Language conditions used to evaluate the same target transition; red words mark semantic swaps. (b) Expected score pattern for a predictor that follows language. (c) Effect of adding image and video transitions to control data. (d) Comparison of language model backbones.
Figure F: Predictor backbone, initialization, and architecture ablations. (a) Validation loss for language model backbones, including pretrained and randomly initialized Qwen3.5. (b) Language reliance for positional encoding and readout variants. 3D input positional embeddings with learned queries yield the highest reliance.
Figure G: Predictor positional encoding and readout. (a) Projected state patches, language tokens and copies of one shared learned query vector receive additive 3D positional embeddings. The prediction head maps query outputs to future patch representations. (b) Native 1D uses sequence indices for visual and text tokens, handled by the backbone’s positional mechanism. MRoPE uses 3D coordinates to rotate attention queries and keys inside the backbone. Inplace readout uses state-token hidden states.
| Initialization | Training scope | Wall | PointMaze | Push-T | DROID | RoboCasa-R | RoboCasa-P | Average |
|---|---|---|---|---|---|---|---|---|
| Pretrained LLM | Single-domain | |||||||
| Random init. | Joint | |||||||
| Pretrained | Joint |
Table C: LaLa-World Predictor initialization and joint training. We compare pretrained and random initialization, and single-domain and joint training. We report success rates (%), except for the DROID action score, with standard errors in parentheses. Average is the unweighted mean of the six benchmark scores.
| Vision encoder | Wall | PointMaze | Push-T | MetaW.-R | MetaW.-RW | Rope | Granular | RoboC.-R | RoboC.-P | DROID | Average |
| JEPA-WMs | |||||||||||
| DINOv2-S/14 | |||||||||||
| DINOv3-L/16 | |||||||||||
| Table 1 (reference) | |||||||||||
| LaLa-World | |||||||||||
| DINOv2-S/14 |
Table D: Vision encoder ablations. We compare frozen vision encoders for JEPA-WMs and LaLa-World. DINOv2 + DINOv3 uses both DINOv2-S/14 and DINOv3-L/16. We report success rates (%), except for the DROID action score. Average is the unweighted mean of the ten benchmark scores. Bold denotes the highest score among encoder configurations within each method; reference rows reproduce the main planning results and are excluded from this comparison.
| Language action | Wall | PointMaze | Push-T | MetaWorld-R | MetaWorld-RW | RoboCasa-R | RoboCasa-P |
|---|---|---|---|---|---|---|---|
| Qualitative | |||||||
| Numerical |
Table E: MPC planning with numerical and qualitative language actions. Qualitative language actions replace numerical magnitudes with slightly , no modifier, or far . Lower success rates are expected when exact magnitudes are omitted, yet it is still interesting that LaLa-World can plan successfully in several domains from qualitative language actions. Only the language conditions supplied to the predictor change; CEM samples numerical actions, and the simulator executes the selected actions in their original numerical form.
| Phrasing | Wall | PointMaze | MetaWorld-R | RoboCasa-R / DROID |
|---|---|---|---|---|
| Original | move the agent | push the ball with force | move the gripper | move gripper, rotate roll |
| Rephrased | push the dot | push the sphere | shift the robot hand | move robot hand, turn roll |
Table F: Robustness to action phrasing. We report planning results with original and paraphrased action templates (bottom) and the substitutions used at evaluation (top). All data combines control trajectories with image and video transitions. We report success rates (%), except for the DROID action score, with standard errors in parentheses.
Figure H: Domain-level comparison of latent and generative world models for MPC planning. We compare LaLa-World with RAEv2 using one and five denoising steps under the matched setup in Section 4.2 . We report success rates (%) for simulation tasks and the action score for DROID.
| Dataset | Training data size |
|---|---|
| Wall | 86,296 |
| PointMaze | 186,675 |
| Push-T | 2,199,488 |
| MetaWorld | 1,162,310 |
| Rope | 18,525 |
| Granular | 18,582 |
Table G: Data used to train LaLa-World. We count target transitions for control and video data and editing pairs for OmniEdit and ShareGPT-4o-Image. Counts exclude validation data and count each control transition once across its two action description formats.
| Benchmark | Evaluation trajectories | Trajectory length | Planning horizon | Executed steps per plan | Substeps per step | CEM settings | Evaluation criterion |
|---|---|---|---|---|---|---|---|
| Wall | 96 | 6 | 6 | 6 | 5 | Position error | |
| PointMaze | 96 | 6 | 6 | 6 | 5 | Position error | |
| Push-T | 96 | 6 | 6 | 6 | 5 | Position error and angle error | |
| MetaWorld-R | 48 | 20 | 6 | 3 | 5 | Success at any step | |
| MetaWorld-RW | 48 | 20 | 6 | 3 | 5 | Success at any step | |
| Rope | 24 | 5 | 5 | 1 | 1 | Chamfer distance |
Table H: MPC planning settings and evaluation criteria. One step corresponds to one world-model transition, with the listed number of action updates (substeps). CEM settings are iterations candidate sequences per iteration retained sequences. For benchmarks requiring replanning, we execute the listed number of steps, observe the environment, and re-optimize the action sequence with CEM. Position and angle errors are measured at the final step, using distance for position. DROID is evaluated offline without simulator execution.
Figure I: MPC planning horizons and action execution. One step corresponds to one world-model transition, with the listed number of action updates (substeps). For benchmarks requiring replanning, we obtain a new observation after executing the prefix and optimize a new sequence with CEM.
Figure J: Substep aggregation for language conditioning and action execution. (a) Summing across substeps reduces the action dimension per step from 10 to 2 for language conditioning. (b) Within MPC, CEM optimizes these sums, which are distributed equally across five substeps for execution. This discards within-step variation. Yet, it is notable that LaLa-World achieves accurate prediction and high planning success rates.
Figure K: MPC planning with qualitative language actions. (a) Each magnitude maps to one of three equal-width bins, and the direction is kept. (b) LaLa-World evaluates CEM samples through their qualitative descriptions, while the simulator executes the selected actions with their numerical values. Thus, LaLa-World plans without exact action magnitudes.
Figure L: LaLa-Policy architecture. The predictor, fine-tuned from LaLa-World, receives encoded observations, a task instruction, and learned action and future queries. It predicts future visual latents and provides conditioning features for an action generator, which produces an action chunk from Gaussian noise. Proprio and intermediate projections are omitted for clarity.
Figure M: Numerical language actions on Wall. Initial and goal observations bracket decoded future predictions (top) and PCA visualizations of predicted latents (bottom), with the selected language actions below each frame.
Figure N: Numerical language actions on PointMaze. Each row shows initial and goal observations and decoded future predictions under the selected actions, which specify pushing forces and directions.
Figure O: Numerical language actions on Push-T. Initial and goal observations bracket decoded future predictions under the selected pusher movements, with numerical displacements described below each frame.
Figure P: Numerical language actions on MetaWorld. Decoded predictions are shown before and after replanning from the simulator observation , alongside initial and goal observations.
Figure Q: Numerical language actions on DROID. Each row shows an initial observation, three decoded future predictions, and the goal observation, with language actions below.
Figure R: Numerical language actions on RoboCasa Reach. Decoded future predictions under selected gripper actions are shown between initial and goal observations. RoboCasa is excluded from world-model training.
Figure S: Numerical language actions on RoboCasa Place. Initial and goal observations bracket decoded future predictions during object placement, with the selected gripper actions below each frame.
Figure T: Qualitative language actions on Wall. Simulator observations show the agent passing through the opening toward the goal. Actions describe directions and relative movement magnitudes.
Figure U: Qualitative language actions on PointMaze. Simulator observations show the ball moving through the maze under MPC, with pushing directions and qualitative force descriptions below the frames.
Figure V: Qualitative language actions on Push-T. Initial and goal observations bracket simulator states during MPC, with the pusher’s direction and relative displacement described below each frame.
Figure W: Qualitative language actions on MetaWorld. Simulator observations show reaching trajectories planned using descriptions of gripper motion and grip adjustment without exact numerical magnitudes.
Figure X: Qualitative language actions on RoboCasa Reach. Simulator observations show zero-shot MPC execution toward the goal, conditioned on qualitative descriptions of gripper translation, rotation, and opening or closing.
Figure Y: Qualitative language actions on RoboCasa Place. Simulator observations show an object-placement trajectory under MPC, with qualitative gripper actions and the goal observation for reference.
Figure Z: LIBERO Spatial: bowl on a ramekin. LaLa-Policy moves the bowl from the ramekin onto the plate. Primary and wrist views compare simulator observations (upper rows) with decoded future predictions (lower rows).
Figure AA: LIBERO Spatial: bowl next to a cookie box. LaLa-Policy moves the specified bowl onto the plate. Simulator observations and decoded future predictions are paired for the primary and wrist views.
Figure AB: LIBERO Object: alphabet soup. LaLa-Policy picks up the alphabet soup and places it in the basket. Each camera view pairs observed states with decoded future predictions.
Figure AC: LIBERO Object: salad dressing. LaLa-Policy transfers the salad dressing to the basket. Primary and wrist views show simulator observations above the corresponding decoded future predictions.
Figure AD: LIBERO Goal: bowl on the cabinet. LaLa-Policy places the bowl on top of the cabinet. Primary and wrist views compare observed states with decoded future predictions throughout execution.
Figure AE: LIBERO Goal: bowl inside the drawer. LaLa-Policy opens the top drawer and places the bowl inside. Both camera views pair simulator observations with decoded future predictions.
Figure AF: LIBERO Long: placing two objects in a basket. LaLa-Policy transfers the cream cheese box and butter to the basket. Observations and decoded future predictions are shown for both camera views.
Figure AG: LIBERO Long: stove and moka pot. LaLa-Policy turns on the stove and places the moka pot on it. Primary and wrist views compare simulator observations with decoded future predictions.
Figure AH: RoboCasa: pressing the coffee-machine button. Simulator observations (upper rows) and decoded future predictions (lower rows) show LaLa-Policy pressing the button, from two external views and the wrist view.
Figure AI: RoboCasa: serving a mug. LaLa-Policy moves the mug from beneath the coffee-machine dispenser to the counter. Each of the three views compares observed states with decoded future predictions.
Figure AJ: RoboCasa: opening a microwave. Simulator observations and decoded future predictions show LaLa-Policy opening the microwave door, from the primary, secondary, and wrist views.
Figure AK: RoboCasa: opening a drawer. LaLa-Policy opens the left drawer. The three camera views pair simulator observations with decoded future predictions during reaching and drawer opening.
Figure AL: RoboCasa: counter-to-cabinet transfer. LaLa-Policy picks up the bar from the counter and places it in the cabinet. Each camera view compares observed states with decoded future predictions.
Figure AM: RoboCasa: pan-to-plate transfer. LaLa-Policy moves the cheese from the pan onto the plate. Observations and decoded future predictions are shown for two external views and the wrist view.
Figure AN: RoboCasa: turning on a stove burner. Simulator observations and decoded future predictions show LaLa-Policy turning on the front-left burner, from the primary, secondary, and wrist views.
More in this topic
World Model Learning
World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.
See all 411 papers