Tiresias: Towards Language-Conditioned World Models
Organizations: University of Washington · National University of Singapore · Microsoft Research · Carnegie Mellon University
Abstract
World models enable planning through internal simulation, while natural language provides intuitive task specification. Yet, language-conditioned world modeling remains fragmented across domains with disparate action spaces, datasets, and evaluation protocols. We introduce LCWM, a shared task that maps an initial agent-view observation and a language instruction to an action sequence that fulfills the instruction. Its accompanying benchmark, LCWM-Bench, integrates 15 datasets across navigation, robotics, 3D gaming, and driving under a shared action interface and evaluation protocol, pairing 142,987 episodes with 428,961 instructions in three styles. We further propose Tiresias, an agent that plans via selective imagination: a language-conditioned diffusion policy generates candidate trajectories, a trajectory ranker shortlists them, and a video-pretrained world model imagines their visual consequences. A visual ranker evaluates these rollouts, complementing trajectory scores to select the final plan. Tiresias achieves competitive planning and imagination performance on LCWM-Bench. Ablations demonstrate the complementary benefits of trajectory ranking and visual verification, while experiments with alternative proposal methods show that the trained rankers remain effective beyond our diffusion policy. Together, LCWM-Bench and Tiresias provide a unified evaluation setting and an effective starting point for studying language-guided planning through world models.
Figures & tables
| L C W M -Bench | Navigation | Robotics | 3D Gaming | Driving | ||||||||||||||
| Method | Score (%) | SR | nDTW | ATE | RPE | SR | nDTW | ATE | RPE | SR | nDTW | ATE | RPE | SR | nDTW | ATE | RPE | |
| Val Seen | GNM (lang) | 36 | 0.24 | 0.44 | 1.10 | 0.34 | 0.25 | 0.42 | 0.74 | 0.33 | 0.25 | 0.39 | 3.18 | 0.48 | 0.36 | 0.52 | 1.29 | 0.27 |
| NoMaD (lang) | 40 | 0.28 | 0.48 | 0.95 | 0.29 | 0.29 | 0.45 | 0.68 | 0.31 | 0.27 | 0.41 | 3.05 | 0.45 | 0.42 | 0.57 | 1.14 | 0.24 | |
| Diamond | 33 | 0.19 | 0.40 | 1.31 | 0.40 | 0.20 | 0.37 | 0.82 | 0.36 | 0.31 | 0.45 | 2.88 | 0.42 | 0.29 | 0.46 | 1.48 | 0.30 | |
| NWM | 45 | 0.31 | 0.50 | 0.72 | 0.23 | 0.33 | 0.49 | 0.64 | 0.29 | 0.40 | 0.53 | 2.30 | 0.35 | 0.46 | 0.60 | 1.04 | 0.22 | |
| NWM (lang) | 48 | 0.33 | 0.52 | 0.69 | 0.25 | 0.35 | 0.51 | 0.59 | 0.27 | 0.43 | 0.55 | 2.14 | 0.33 | 0.49 | 0.63 | 0.95 | 0.20 | |
| One-step ( ) | Rollout ( ) | |||||||||
| Method | FVD | SSIM | PSNR | LPIPS | DreamSim | SSIM | PSNR | LPIPS | DreamSim | |
| Val Seen | Diamond | 648.7 | 0.336 | 10.642 | 0.401 | 0.128 | 0.118 | 4.731 | 0.668 | 0.301 |
| NWM | 382.1 | 0.382 | 12.364 | 0.326 | 0.111 | 0.184 | 6.218 | 0.621 | 0.242 | |
| NWM (lang) | 365.3 | 0.396 | 12.578 | 0.319 | 0.108 | 0.178 | 6.176 | 0.598 | 0.236 | |
| LCVN-WM | 321.6 | 0.452 | 18.724 | 0.268 | 0.101 | 0.289 | 11.204 | 0.471 | 0.219 | |
| LCVN-Uni | 344.8 | 0.438 | 14.018 | 0.287 | 0.096 | 0.224 | 7.562 | 0.492 | 0.214 | |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Style | Instruction | |
| Navigation | Concise | Move forward, turn left and continue straight until you reach the open area, then stop. |
| Intricate | Start by moving forward through the open space with a yellow archway on the left and a blue seating area ahead. Follow the path as it curves gently to the left, then continue past the blue seating area and a white pillar. The floor transitions from carpet to a shiny white surface. Proceed into the larger open area with tables and chairs visible in the distance, then stop. | |
| Landmark | Walk forward past the yellow archway and blue seating area. Follow the path as it curves left, then continue past the white pillar onto the shiny white floor. Proceed into the open area with tables and stop. | |
| Robotics | Concise | From an external third-person view, grasp and lift one corner of the blue cloth, then fold it diagonally. |
| Intricate | From an external third-person view, the robot arm approaches a light blue cloth spread flat on a wooden table near several colorful toy vegetables. The gripper targets the bottom-right corner, closes around the fabric, and lifts it upward. The arm then moves diagonally across the table, drawing the corner toward the top-left side. As the cloth is pulled, it folds over itself into a triangular shape. The gripper releases the fabric, leaving the cloth partially folded on the table. | |
| Landmark | From an external third-person view, move to the bottom-right corner of the blue cloth on the wooden table, grasp it, and pull it diagonally toward the top-left corner to fold the fabric. |
| Domain | Dataset | Source | Action variant | Dim. | #Episodes | #Instructions |
| Navigation | GoStanford | Real | 3 | 10,080 | 30,240 | |
| RECON | Real | 3 | 10,392 | 31,176 | ||
| SCAND | Real | 3 | 5,427 | 16,281 | ||
| HuRoN | Real | 3 | 9,346 | 28,038 | ||
| TartanDrive | Real | 3 | 5,896 | 17,688 | ||
| Robotics | BridgeData V2 | Real | 7 | 10,000 | 30,000 |
| Navigation | System : You are a language-conditioned navigation world model. One initial observation is provided together with the route instruction and the full action sequence. Generate the complete future agent-view video in one shot. Treat the instruction as the route semantics and the action sequence as transition control for timing, direction, and motion scale. Keep motion physically plausible, temporally continuous, and faithful to both instruction and action. Instruction : {language instruction} |
| Robotics | System : You are a language-conditioned robot-manipulation world model. One initial observation is provided together with the manipulation instruction and the full 7-DoF end-effector action sequence. Generate the complete future video in one shot. Preserve the viewpoint stated at the beginning of the instruction: a wrist-mounted first-person view must remain wrist-mounted first-person, and an external third-person view must remain external third-person. In an external third-person view, camera extrinsics are fixed: never pan, tilt, zoom, dolly, orbit, or track the robot. In a wrist-mounted first-person view, the camera moves only because it is rigidly attached to the robot wrist. The actor is a robotic arm with a mechanical gripper; never replace it with a human hand, human arm, skin, or fingers. Preserve whether the initial scene is real or simulated rather than changing visual domain. Treat the instruction as task and contact semantics, and the action sequence as transition control for end-effector translation, rotation, and gripper opening or closing. Maintain robot identity, geometry, kinematic continuity, object permanence, contact, grasp, lift, transport, placement, and release. Instruction : {language instruction} |
| 3D Gaming | System : You are a language-conditioned first-person gaming world model. One initial observation is provided together with the gameplay instruction and the full ego-action sequence. Generate the complete future first-person gaming video in one shot. Treat the instruction as task semantics and the action sequence as transition control for forward motion, lateral strafing, yaw rotation (looking left/right), and vertical movement (jumping, falling, stairs). Keep scene geometry, player motion, lighting, and object appearance physically plausible and temporally continuous. Be faithful to both instruction and action. Instruction : {language instruction} |
| Driving | System : You are a language-conditioned self-driving world model. One initial front-facing observation is provided together with the route instruction and the full ego-action sequence. Generate the complete future driving video in one shot. Treat the instruction as route semantics and the action sequence as transition control for speed, direction, and motion scale. Keep road geometry, ego-motion, traffic, and lighting physically plausible and temporally continuous. Be faithful to both instruction and action. Instruction : {language instruction} |
| Setting | SR | nDTW | ATE | RPE |
| Funnel shape (stage one: Tr-Ranker) | ||||
| 0.508 0.022 | 0.604 0.005 | 0.797 0.010 | 0.155 0.001 | |
| 0.545 0.016 | 0.622 0.005 | 0.791 0.011 | 0.152 0.003 | |
| 0.560 0.032 | 0.636 0.002 | 0.766 0.001 | 0.146 0.001 | |
| 0.516 0.002 | 0.610 0.007 | 0.786 0.013 | 0.153 0.002 | |
| 0.552 0.005 | 0.628 0.006 | 0.783 0.009 | 0.150 0.002 | |
| Method | Configuration | SR | Latency (s) |
| NoMaD (lang) | – | 0.32 | 0.7 |
| LCVN-Uni | – | 0.47 | 85 |
| Tiresias | (default) | 0.60 | 36 |
| Tiresias | 0.55 | 9 |
World Model-Based Planning
World model-based planning uses models of how environments change to compare imagined actions and choose goal-directed behavior.
See all 302 papers