Planning with world models offers a powerful paradigm for robotic control. Conventional approaches train a model to predict future frames conditioned on current frames and actions, which can then be used for planning. However, the objective of predicting future pixels is often at odds with the actual planning objective; strong pixel reconstruction does not always correlate with good planning decisions. This paper posits that instead of reconstructing future frames as pixels, world models only need to predict task-relevant semantic information about the future. For such prediction the paper poses world modeling as a visual question answering problem about semantic information in future frames. This perspective allows world modeling to be approached with the same tools underlying vision language models. Thus, vision-language models can be trained as "semantic" world models through a supervised finetuning process on image-action-text data, enabling planning for decision-making while inheriting many of the generalization and robustness properties from the pretrained vision-language models. In this paper, we introduce this new method and validate it on simulated robotic tasks leading to improvements over typical paradigms of reconstruction-based action-conditional world modeling. Website available at https://weirdlabuw.github.io/swm.
Figures & tables
Figure 1: Comparison between Vision-Language Models, Video World Models, and Semantic World Models . While Vision-Language Models answer questions about static observations and Video World Models predict future observations given actions, Semantic World Models take observations and actions as input to directly answer questions about the future outcomes of those actions.
Figure 2: Overview of Semantic World Models. SWM is a VLM adapted to answer questions about the future realized by the actions used to condition the model. Using a set of questions and desired answers, its predictions can be converted into a planning signal and iteratively refine the action sequence.
Figure 3: Example state entry in the SAQA dataset with two action horizons and six QA pairs.
Figure 4: Examples of each evaluation task. The top frame represents the initialization, and the bottom frame represents task completion. The first three tasks are for LangTable and the last two are for OGBench.
Figure 5: Policy Improvement across LangTable and OGBench across multiple tasks. The average success rates of the base policies ( 14.4% on LangTable and 45.33% on OGBench) increase to 81.6% and 76.0% , respectively. SWM further outperforms the IDQL and AVD baselines across all evaluated tasks and environments. Reported success rates over n=50 seeds with 95% confidence intervals (normal approximation).
LangTable
OGBench
Dataset Type
Expert Data
Expert Data OOD
Expert Data
Expert Data OOD
Sub Optimal
85.98±0.33
81.99±1.46
90.83±0.39
85.56±1.10
Expert
91.27±0.79
86.49±0.39
96.53±0.13
87.33±2.13
Combined
92.92±0.34
88.32±2.10
96.86±0.13
88.16±1.54
Table 1: Future QA Performance. Accuracy of answers on future QA evaluated on expert SAQA datasets generated by experts on test time seeds in both in-domain and out-of-domain block combinations. Reported standard deviation across 3 model training seeds.
Task
Base Policy
AVD
SWM (Ours)
MS1
6% ± 6.6
8% ± 7.5
50% ± 13.9
MS2
4% ± 5.4
2% ± 3.9
66% ± 13.1
MS3
4% ± 5.4
2% ± 3.9
54% ± 13.8
MS4
2% ± 3.9
4% ± 5.4
54% ± 13.8
Table 2: Multi-Step Results. SWM model improvement results on four different multi-step compositional tasks. The tasks are as follows: MS1 - red pentagon to blue moon, yellow pentagon to red moon. MS2 - yellow star to blue cube, yellow pentagon to red moon. MS3 - yellow star to blue cube, red pentagon to blue moon. MS4 - green cube to blue moon, yellow pentagon to red moon. Reported success rates over n=50 seeds with 95% confidence intervals (normal approximation).
Table 4: Out-of-Distribution Improvement Results. SWM model improvement results on tasks in LangTable and OGBench on out-of-distribution scenes. SWM is able to show policy improvement and outperform AVD across both environments. Reported success rates over n=50 seeds with 95% confidence intervals (normal approximation).
Figure 6: Visualization of the attention map from language tokens to image patches in the 4th transformer layer. The language tokens correctly attend to the task-relevant locations in the image depending on the prompt.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Architecture of Semantic World Model
Diffusion
Batch size
128
Epochs
100
Action horizon
16
Observation horizon
2
Diffusion iters
100
Eval diffusion iters
10
Appendix
Table 5: Hyperparameters for MPPI, IDQL, Diffusion, and AVD Model
Figure 8: Out-of-distribution configurations for the evaluation tasks. LangTable is configured to have OOD block/color combinations. OGBench is configured to have a different color background.
Type
Example
LangTable
Block touching
Is the red star touching the blue cube?
Peg to block
Is the green cube next to the peg?
Block board position
Is the red star in the center of the board?
Peg block relative direction
Is the peg above the red cube block?
Block to block relative direction
Is the red star to the right of the blue cube?
Appendix
Table 6: Question types and examples for LangTable and OGBench
Task
Question
Weight
Desired Answer
Reaching LT
Is the robotic peg touching the {target_block}?
0.8
Yes
Is the robotic peg closer to the {target_block}?
0.2
Yes
Reaching OG
Is the robotic gripper touching the {target_block}?
0.8
Yes
Is the robotic gripper closer to the {target_block}?
0.2
Yes
Separate Blocks
Is the robotic peg touching the {center_block}?
0.6
Yes
Is the {avoid block} touching the {center block}?
0.4
No
Appendix
Table 7: QA pairs used for task rewards
Task
Base Policy
IDQL
AVD
SWM
Push Green Cube to Blue Moon
6% ± 6.6
8% ± 7.5
48% ± 13.8
78% ± 11.5
Push Red Moon to Green Star
18% ± 10.6
8% ± 7.5
44% ± 13.8
80% ± 11.1
Push Red Pentagon to Blue Moon
14% ± 9.6
12% ± 9.0
38% ± 13.5
80% ± 11.1
Push Yellow Pentagon to Red Moon
18% ± 10.6
8% ± 7.5
34% ± 13.1
86% ± 9.6
Push Yellow Star to Blue Cube
16% ± 10.2
10% ± 8.3
62% ± 13.5
84% ± 10.2
Stack Blue Cube on Yellow Cube
52% ± 13.8
8% ± 7.5
50% ± 13.9
82% ± 10.6
Appendix
Table 8: Improvement Results. SWM model improvement results on planning tasks in LangTable and OGBench on in-distribution scenes. Reported success rates over n=50 seeds with 95% confidence intervals (normal approximation). The top tasks are LangTable and the bottom tasks are OGBench.
Task
SWM (with weights)
SWM (no weights)
Push Green Cube to Blue Moon
78% ± 11.5
72% ± 12.4
Push Red Moon to Green Star
80% ± 11.1
78% ± 11.4
Push Red Pentagon to Blue Moon
80% ± 11.1
82% ± 10.6
Push Yellow Pentagon to Red Moon
86% ± 9.6
88% ± 9.0
Push Yellow Star to Blue Cube
84% ± 10.2
76% ± 11.8
Stack Blue Cube on Yellow Cube
82% ± 10.6
82% ± 10.6
Appendix
Table 9: Ablation on question weights. Success rates for SWM with weights vs. SWM without weights on LangTable and OGBench tasks. Reported over n=50 seeds with 95% confidence intervals.
Task
Base Policy
ID 1
ID 2
OOD 1
OOD 2
Push Red Moon to Green Star
18% ± 10.6
72% ± 12.4
88% ± 9.0
84% ± 10.2
78% ± 11.5
Push Yellow Star to Blue Cube
16% ± 10.2
78 % ± 11.5
86% ± 9.6
84% ± 10.2
72% ± 12.4
Push Yellow Pentagon to Red Moon
18% ± 10.6
88% ± 9.0
88% ± 9.0
86% ± 9.6
76% ± 11.8
Appendix
Table 10: Ablation on Question Phrasing. Success rates of SWM under in-distribution (ID) and out-of-distribution (OOD) task phrasings. Reported with 95% confidence intervals.
Table 19
Task
Base Policy
SWM+grad
SWM+grad + LLM planning
MS1
6
50
62
MS2
4
66
42
MS3
4
54
52
MS4
2
54
44
Average
4
56
50
Appendix
Table 14: Success rate (%) on long-horizon LangTable tasks with and without LLM-derived task decompositions over 50 seeds. MS1 - red pentagon to blue moon, yellow pentagon to red moon. MS2 - yellow star to blue cube, yellow pentagon to red moon. MS3 - yellow star to blue cube, red pentagon to blue moon. MS4 - green cube to blue moon, yellow pentagon to red moon.
Figure 9: Visualization of gradient-based planning on the LangTable - Red Pentagon to Blue Moon task. The initially proposed action sequence is on the left, and updates to this action sequence go progressively to the right, approaching the optimal trajectory over successive gradient steps.
Figure 10: Attention maps in different layers of SWM. Question: “Is the red moon touching the blue block?”
Figure 11: Attention maps for different trajectories. Question: “Is the red moon touching the blue block?”
Method
Time per action chunk (s)
MPPI
4.48
Gradient-based
1.56
Appendix
Table 15: Planning speed comparison between MPPI and gradient-based planning, measured on a single GPU in the LangTable environment.
World models such as DINO-WM and LeWM specify the goal with an image, which is difficult to obtain in advance for novel tasks. We present the Grounded World Model (GWM), a latent world model that enables zero-shot planning in the real world from language goals alone. Given a candidate action sequence and the current observation, GWM predicts the future in the visual space of a pretrained video-language embedding model. The frozen readout of this embedding model maps this imagined future and the task description into the same embedding space, where their negative cosine similarity serves as the planning cost. Training GWM requires only offline and task-agnostic video-action pairs and no language labels. In simulated experiments on WISER, planning with GWM, which executes the candidate action of lowest cost, solves 87% of 288 tasks with unseen instructions and visual signals, while ten fine-tuned VLAs average 22%. We then scale GWM up with real robot data, and use it for zero-shot planning in realistic simulation and real scenes. In the IsaacSim evaluation, planning with GWM completes all 70 trials across 14 tasks that require reasoning over referring expressions, matching a modular planner grounded by a frontier VLM, while pi0.5 reaches 37/70. Deployed on a real Franka, the same stack completes 55/60 separately evaluated pick-and-place sub-tasks, comparable to the modular planner's 52/60, with the full system running locally on a single consumer GPU. Project website: https://quanyili.github.io/gwm-wiser/.
World models are increasingly used in embodied intelligence and generative simulation, yet their scope remains ambiguous across communities. This tutorial presents a design-space view of world models as action-conditioned predictive models that estimate the future evolution of task-relevant observations or states. We categorize existing methods into observation-space and state-space world models, comparing their trade-offs in visual fidelity, spatial structure, physical interpretability, and control usability. We further introduce world action models, which connect predicted futures with executable robot actions, and summarize four representative paradigms: imagine-then-execute, video-feature-conditioned action prediction, joint video-action modeling, and auxiliary video prediction for policy learning. The goal of this tutorial is to clarify the conceptual scope of world (action) models and provide a structured taxonomy for embodied prediction and control.
Xiaoxiong Zhang, Xiong Zeng, Wei Zhang
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · LimX Dynamics, Shenzhen, China
World models are becoming central to robotic planning and control by predicting future state transitions. Existing approaches mainly rely on visual, latent, or language prediction, which can be difficult to ground in executable robot actions and prone to compounding errors over long horizons. In contrast, traditional robotic task and motion planning enables structured long-horizon reasoning through compact symbolic representations of world transitions, but typically lacks synchronized visual prediction. We propose Hierarchical World Model (H-WM), which jointly predicts logical and visual state transitions by combining a high-level logical world model with a low-level visual world model. The predicted logical actions and latent visual state transitions are jointly incorporated into Vision-Language-Action (VLA) models as intermediate state guidance for long-horizon task execution. Experiments on three long-horizon benchmarks and real robots show that H-WM consistently improves VLA's performance by stabilizing long-horizon execution and mitigating error accumulation. We also construct LIBERO-Logic, a frame-level aligned dataset that pairs visual observations and continuous robot states with logical actions and predicate-based logical states.
Jinbang Huang, Wenyuan Chen, Zhiyuan Li +9
Huawei Noah’s Ark Lab · University of Toronto · University of British Columbia +1