World models enable planning through internal simulation, while natural language provides intuitive task specification. Yet, language-conditioned world modeling remains fragmented across domains with disparate action spaces, datasets, and evaluation protocols. We introduce LCWM, a shared task that maps an initial agent-view observation and a language instruction to an action sequence that fulfills the instruction. Its accompanying benchmark, LCWM-Bench, integrates 15 datasets across navigation, robotics, 3D gaming, and driving under a shared action interface and evaluation protocol, pairing 142,987 episodes with 428,961 instructions in three styles. We further propose Tiresias, an agent that plans via selective imagination: a language-conditioned diffusion policy generates candidate trajectories, a trajectory ranker shortlists them, and a video-pretrained world model imagines their visual consequences. A visual ranker evaluates these rollouts, complementing trajectory scores to select the final plan. Tiresias achieves competitive planning and imagination performance on LCWM-Bench. Ablations demonstrate the complementary benefits of trajectory ranking and visual verification, while experiments with alternative proposal methods show that the trained rankers remain effective beyond our diffusion policy. Together, LCWM-Bench and Tiresias provide a unified evaluation setting and an effective starting point for studying language-guided planning through world models.
Figures & tables
Figure 1: Overview of LCWM, LCWM-Bench, and Tiresias . (a) From an initial observation and instruction, Tiresias plans through imagination (A ↦ D); colored markers identify landmarks and highlighted arrows indicate actions. (b) LCWM-Bench spans 15 datasets, four domains, and 142,987 episodes in three instruction styles. (c) Comparison of planning and imagination performance across evaluated methods on LCWM-Bench.
Figure 2: Construction and statistics of LCWM-Bench. (a) Dataset collection, action and observation standardization, and instruction annotation (Sec. 3.2 ). (b–d) Dataset sample counts, trajectory-length distributions, and average instruction lengths for each of the three styles (Sec. 3.3 ).
Figure 3: Overview of Tiresias . (a) The LDP learns to denoise action sequences given instructions. (b) The Wan2.2-based world model predicts future observations given the initial frame, trajectory, and instruction. (c) At inference, the Tr-Ranker shortlists LDP proposals for imagination; fused trajectory and visual scores select the final plan.
Figure 4: Qualitative results on Navigation, Robotics, Driving, and 3D gaming. For each case, we show the initial observation and instruction, the imagined rollout of the selected plan against the ground-truth future, and a comparison of the predicted trajectory with the corresponding reference trajectory.
L C W M -Bench
Navigation
Robotics
3D Gaming
Driving
Method
Score (%)
SR
nDTW
ATE
RPE
SR
nDTW
ATE
RPE
SR
nDTW
ATE
RPE
SR
nDTW
ATE
RPE
Val Seen
GNM (lang)
36
0.24
0.44
1.10
0.34
0.25
0.42
0.74
0.33
0.25
0.39
3.18
0.48
0.36
0.52
1.29
0.27
NoMaD (lang)
40
0.28
0.48
0.95
0.29
0.29
0.45
0.68
0.31
0.27
0.41
3.05
0.45
0.42
0.57
1.14
0.24
Diamond
33
0.19
0.40
1.31
0.40
0.20
0.37
0.82
0.36
0.31
0.45
2.88
0.42
0.29
0.46
1.48
0.30
NWM
45
0.31
0.50
0.72
0.23
0.33
0.49
0.64
0.29
0.40
0.53
2.30
0.35
0.46
0.60
1.04
0.22
NWM (lang)
48
0.33
0.52
0.69
0.25
0.35
0.51
0.59
0.27
0.43
0.55
2.14
0.33
0.49
0.63
0.95
0.20
Table 1: Planning comparison on val seen and val unseen splits of LCWM-Bench. The score is the mean of SR and nDTW macro-averaged over four domains; higher is better for Score, SR, and nDTW, lower for ATE and RPE.
One-step ( @1 )
Rollout ( @10 )
Method
FVD @32↓
SSIM ↑
PSNR ↑
LPIPS ↓
DreamSim ↓
SSIM ↑
PSNR ↑
LPIPS ↓
DreamSim ↓
Val Seen
Diamond
648.7
0.336
10.642
0.401
0.128
0.118
4.731
0.668
0.301
NWM
382.1
0.382
12.364
0.326
0.111
0.184
6.218
0.621
0.242
NWM (lang)
365.3
0.396
12.578
0.319
0.108
0.178
6.176
0.598
0.236
LCVN-WM
321.6
0.452
18.724
0.268
0.101
0.289
11.204
0.471
0.219
LCVN-Uni
344.8
0.438
14.018
0.287
0.096
0.224
7.562
0.492
0.214
Table 2: Imagination comparison on the val seen and val unseen splits of LCWM-Bench, averaged over the four domains. @n metrics score n -step open-loop rollouts under ground-truth actions against the real frames. Best results are in bold.
Table 7Figure 8
Figure 6: Ablations on candidate sampling and selection on val seen. (a) Test-time scaling with pool size N at fixed k=8 ; Oracle-of- N provides the SR upper bound over each candidate pool. (b) Effect of shortlist size k for different pool sizes. (c) Ranker fusion weights λtr:λv under the default 128→8 funnel. (d) Random versus Tr-Ranker shortlisting with different imagination budgets at N=128 . Shaded bands and error bars in (b–d) indicate ± one standard deviation.
Table 10
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
List of listings A1 Annotation prompt used to generate navigation instructions.
List of listings A2 Annotation prompt used to generate robotics instructions.
List of listings A3 Annotation prompt used to generate 3D gaming instructions (Minecraft source).
List of listings A4 Annotation prompt used to generate 3D gaming instructions (CS:GO source).
List of listings A5 Annotation prompt used to generate driving instructions.
Style
Instruction
Navigation
Concise
Move forward, turn left and continue straight until you reach the open area, then stop.
Intricate
Start by moving forward through the open space with a yellow archway on the left and a blue seating area ahead. Follow the path as it curves gently to the left, then continue past the blue seating area and a white pillar. The floor transitions from carpet to a shiny white surface. Proceed into the larger open area with tables and chairs visible in the distance, then stop.
Landmark
Walk forward past the yellow archway and blue seating area. Follow the path as it curves left, then continue past the white pillar onto the shiny white floor. Proceed into the open area with tables and stop.
Robotics
Concise
From an external third-person view, grasp and lift one corner of the blue cloth, then fold it diagonally.
Intricate
From an external third-person view, the robot arm approaches a light blue cloth spread flat on a wooden table near several colorful toy vegetables. The gripper targets the bottom-right corner, closes around the fabric, and lifts it upward. The arm then moves diagonally across the table, drawing the corner toward the top-left side. As the cloth is pulled, it folds over itself into a triangular shape. The gripper releases the fabric, leaving the cloth partially folded on the table.
Landmark
From an external third-person view, move to the bottom-right corner of the blue cloth on the wooden table, grasp it, and pull it diagonally toward the top-left corner to fold the fabric.
Appendix
Table A1: Multi-style instruction samples from LCWM-Bench. One episode per domain, each paired with all three instruction styles. The styles share the same trajectory and differ only in the level of specificity and the degree of visual anchoring, enabling a controlled evaluation of language grounding across domains.
Figure A1 : Word clouds of the released instructions, by domain and instruction style. Word size reflects frequency within the corresponding style-domain subset of LCWM-Bench.
Domain
Dataset
Source
Action variant
Dim.
#Episodes
#Instructions
Navigation
GoStanford
Real
(u,ϕ)
3
10,080
30,240
RECON
Real
(u,ϕ)
3
10,392
31,176
SCAND
Real
(u,ϕ)
3
5,427
16,281
HuRoN
Real
(u,ϕ)
3
9,346
28,038
TartanDrive
Real
(u,ϕ)
3
5,896
17,688
Robotics
BridgeData V2
Real
+z,ρ,ϑ,g
7
10,000
30,000
Appendix
Table A2 : Composition of LCWM-Bench. Action variant follows the base parameterization (ut,ϕt) of Sec. 3.1, with domain-specific channels ( zt , vt , or ρt,ϑt,gt ) appended where applicable. Every episode is paired with three instruction styles (concise, intricate, landmark), yielding three instruction-conditioned samples per episode.
Figure A2 : Architectures of two trained modules of Tiresias . (a) The world model’s multimodal diffusion Transformer (Sec. B.3 ). (b) LDP’s language-conditioned diffusion Transformer (Sec. B.2 ).
Navigation
System : You are a language-conditioned navigation world model. One initial observation is provided together with the route instruction and the full action sequence. Generate the complete future agent-view video in one shot. Treat the instruction as the route semantics and the action sequence as transition control for timing, direction, and motion scale. Keep motion physically plausible, temporally continuous, and faithful to both instruction and action. Instruction : {language instruction}
Robotics
System : You are a language-conditioned robot-manipulation world model. One initial observation is provided together with the manipulation instruction and the full 7-DoF end-effector action sequence. Generate the complete future video in one shot. Preserve the viewpoint stated at the beginning of the instruction: a wrist-mounted first-person view must remain wrist-mounted first-person, and an external third-person view must remain external third-person. In an external third-person view, camera extrinsics are fixed: never pan, tilt, zoom, dolly, orbit, or track the robot. In a wrist-mounted first-person view, the camera moves only because it is rigidly attached to the robot wrist. The actor is a robotic arm with a mechanical gripper; never replace it with a human hand, human arm, skin, or fingers. Preserve whether the initial scene is real or simulated rather than changing visual domain. Treat the instruction as task and contact semantics, and the action sequence as transition control for end-effector translation, rotation, and gripper opening or closing. Maintain robot identity, geometry, kinematic continuity, object permanence, contact, grasp, lift, transport, placement, and release. Instruction : {language instruction}
3D Gaming
System : You are a language-conditioned first-person gaming world model. One initial observation is provided together with the gameplay instruction and the full ego-action sequence. Generate the complete future first-person gaming video in one shot. Treat the instruction as task semantics and the action sequence as transition control for forward motion, lateral strafing, yaw rotation (looking left/right), and vertical movement (jumping, falling, stairs). Keep scene geometry, player motion, lighting, and object appearance physically plausible and temporally continuous. Be faithful to both instruction and action. Instruction : {language instruction}
Driving
System : You are a language-conditioned self-driving world model. One initial front-facing observation is provided together with the route instruction and the full ego-action sequence. Generate the complete future driving video in one shot. Treat the instruction as route semantics and the action sequence as transition control for speed, direction, and motion scale. Keep road geometry, ego-motion, traffic, and lighting physically plausible and temporally continuous. Be faithful to both instruction and action. Instruction : {language instruction}
Appendix
Table A3: System prompts used to condition the language-conditioned world model. For each domain, the system prompt and the episode instruction are concatenated into one text sequence, encoded by the frozen umT5 encoder, and supplied to the world model. Prompts are fixed per domain and identical at training and inference time.
Figure A3 : More qualitative results on val_seen and val_unseen navigation scenarios. For each case, we show the initial observation and instruction, the imagined rollout of the selected plan against the ground-truth future, and the predicted trajectory against the reference.
Figure A4 : More qualitative results on driving scenarios. For each case, we show the initial observation and instruction, the imagined rollout of the selected plan against ground-truth future, and predicted trajectory against the reference.
Figure A5 : More qualitative results on driving scenarios. For each case, we show the initial observation and instruction, the imagined rollout of the selected plan against ground-truth future, and predicted trajectory against the reference.
Figure A6 : More qualitative results on 3D gaming scenarios. For each case, we show the initial observation and instruction, the imagined rollout of the selected plan against ground-truth future, and predicted trajectory against the reference.
Figure A7 : More qualitative results on first-person-view robotics scenarios. For each case, we show initial observation and instruction, imagined rollout of selected plan against ground-truth future, and predicted trajectory against reference.
Figure A8 : More qualitative results on third-person-view robotics scenarios. For each case, we show initial observation and instruction, imagined rollout of selected plan against ground-truth future, and predicted trajectory against reference.
Setting
SR ↑
nDTW ↑
ATE ↓
RPE ↓
Funnel shape N→k→1 (stage one: Tr-Ranker)
16→1
0.508 ± 0.022
0.604 ± 0.005
0.797 ± 0.010
0.155 ± 0.001
16→4
0.545 ± 0.016
0.622 ± 0.005
0.791 ± 0.011
0.152 ± 0.003
16→8
0.560 ± 0.032
0.636 ± 0.002
0.766 ± 0.001
0.146 ± 0.001
32→1
0.516 ± 0.002
0.610 ± 0.007
0.786 ± 0.013
0.153 ± 0.002
32→4
0.552 ± 0.005
0.628 ± 0.006
0.783 ± 0.009
0.150 ± 0.002
Appendix
Table A4: Ablation on the candidate funnel on val seen. Top : N→k→1 samples N candidates, imagines the Tr-Ranker top- k , and selects one trajectory. Middle : Tr-Ranker versus random shortlisting at matched imagination budgets. Bottom : ranker score fusion with the default 128→8 funnel. Bold marks the default configuration.
Method
Configuration
SR ↑
Latency (s) ↓
NoMaD (lang)
–
0.32
0.7
LCVN-Uni
–
0.47
85
Tiresias
128→8→1 (default)
0.60
36
Tiresias
128→2→1
0.55
9
Appendix
Table A5: Planning performance and inference latency on val seen. Latency measures the time to produce the final action trajectory, including candidate generation and selection where applicable. For Tiresias , N→k→1 imagines k shortlisted candidates before selecting one.
This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning. To this end, we introduce LaLa-World, a Language-conditioned Latent World model that learns from simulation and robot trajectories together with broader image and video data. Two designs make this work. First, we describe both numerical actions and visual changes in language, enabling joint training of a single future predictor across heterogeneous datasets. Second, we build the predictor on a language model, allowing it to interpret language conditions while predicting future latent states. The resulting predictor achieves higher average MPC planning performance than existing LWMs trained separately for each domain or task. Adding image and video data further improves average planning performance and robustness to action phrasing. In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with 6× less training cost and 13.9× faster planning. Beyond planning, the pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations. LaLa-Policy performs competitively with Cosmos Policy on LIBERO and RoboCasa, with up to 5.5× faster inference.
Junha Song, Geonmo Gu, Jeongeun Park +5
NAVER AI Lab · Georgia Institute of Technology · Brown University +1
World models such as DINO-WM and LeWM specify the goal with an image, which is difficult to obtain in advance for novel tasks. We present the Grounded World Model (GWM), a latent world model that enables zero-shot planning in the real world from language goals alone. Given a candidate action sequence and the current observation, GWM predicts the future in the visual space of a pretrained video-language embedding model. The frozen readout of this embedding model maps this imagined future and the task description into the same embedding space, where their negative cosine similarity serves as the planning cost. Training GWM requires only offline and task-agnostic video-action pairs and no language labels. In simulated experiments on WISER, planning with GWM, which executes the candidate action of lowest cost, solves 87% of 288 tasks with unseen instructions and visual signals, while ten fine-tuned VLAs average 22%. We then scale GWM up with real robot data, and use it for zero-shot planning in realistic simulation and real scenes. In the IsaacSim evaluation, planning with GWM completes all 70 trials across 14 tasks that require reasoning over referring expressions, matching a modular planner grounded by a frontier VLM, while pi0.5 reaches 37/70. Deployed on a real Franka, the same stack completes 55/60 separately evaluated pick-and-place sub-tasks, comparable to the modular planner's 52/60, with the full system running locally on a single consumer GPU. Project website: https://quanyili.github.io/gwm-wiser/.
Building generalizable agents for diverse applications remains a fundamental challenge. While imitation learning-based policies succeed in specific training environments, they often fail to generalize to novel scenes and tasks. In this work, we propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs) and the physical grounding of a multi-task pose-image conditioned world model. Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts. We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios, significantly outperforming state-of-the-art end-to-end policy models such as VLAs and WAMs. Project website at worldactionplanner.github.io