Semantic World Models

Jacob Berg; Chuning Zhu; Yanda Bao; Ishan Durugkar; Abhishek Gupta

cs.LGOct 22, 2025

Semantic World Models

Authors: Jacob Berg, Chuning Zhu, Yanda Bao, Ishan Durugkar, Abhishek Gupta

Organizations: University of Washington · Sony AI

Abstract

Planning with world models offers a powerful paradigm for robotic control. Conventional approaches train a model to predict future frames conditioned on current frames and actions, which can then be used for planning. However, the objective of predicting future pixels is often at odds with the actual planning objective; strong pixel reconstruction does not always correlate with good planning decisions. This paper posits that instead of reconstructing future frames as pixels, world models only need to predict task-relevant semantic information about the future. For such prediction the paper poses world modeling as a visual question answering problem about semantic information in future frames. This perspective allows world modeling to be approached with the same tools underlying vision language models. Thus, vision-language models can be trained as "semantic" world models through a supervised finetuning process on image-action-text data, enabling planning for decision-making while inheriting many of the generalization and robustness properties from the pretrained vision-language models. In this paper, we introduce this new method and validate it on simulated robotic tasks leading to improvements over typical paradigms of reconstruction-based action-conditional world modeling. Website available at https://weirdlabuw.github.io/swm.

Figures & tables

Figure 1: Comparison between Vision-Language Models, Video World Models, and Semantic World Models . While Vision-Language Models answer questions about static observations and Video World Models predict future observations given actions, Semantic World Models take observations and actions as input to directly answer questions about the future outcomes of those actions.

Figure 2: Overview of Semantic World Models. SWM is a VLM adapted to answer questions about the future realized by the actions used to condition the model. Using a set of questions and desired answers, its predictions can be converted into a planning signal and iteratively refine the action sequence.

Figure 3: Example state entry in the SAQA dataset with two action horizons and six QA pairs.

Figure 4: Examples of each evaluation task. The top frame represents the initialization, and the bottom frame represents task completion. The first three tasks are for LangTable and the last two are for OGBench.

Figure 5: Policy Improvement across LangTable and OGBench across multiple tasks. The average success rates of the base policies (

14.4\%

on LangTable and

45.33\%

on OGBench) increase to

81.6\%

and

76.0\%

, respectively. SWM further outperforms the IDQL and AVD baselines across all evaluated tasks and environments. Reported success rates over

n=50

seeds with 95% confidence intervals (normal approximation).

	LangTable		OGBench
Dataset Type	Expert Data	Expert Data OOD	Expert Data	Expert Data OOD
Sub Optimal	$85.98\pm 0.33$	$81.99\pm 1.46$	$90.83\pm 0.39$	$85.56\pm 1.10$
Expert	$91.27\pm 0.79$	$86.49\pm 0.39$	$96.53\pm 0.13$	$87.33\pm 2.13$
Combined	$\textbf{92.92}\pm 0.34$	$\textbf{88.32}\pm 2.10$	$\textbf{96.86}\pm 0.13$	$\textbf{88.16}\pm 1.54$

Table 1: Future QA Performance. Accuracy of answers on future QA evaluated on expert SAQA datasets generated by experts on test time seeds in both in-domain and out-of-domain block combinations. Reported standard deviation across 3 model training seeds.

Task	Base Policy	AVD	SWM (Ours)
MS1	6% $\pm$ 6.6	8% $\pm$ 7.5	50% $\pm$ 13.9
MS2	4% $\pm$ 5.4	2% $\pm$ 3.9	66% $\pm$ 13.1
MS3	4% $\pm$ 5.4	2% $\pm$ 3.9	54% $\pm$ 13.8
MS4	2% $\pm$ 3.9	4% $\pm$ 5.4	54% $\pm$ 13.8

Table 2: Multi-Step Results. SWM model improvement results on four different multi-step compositional tasks. The tasks are as follows: MS1 - red pentagon to blue moon, yellow pentagon to red moon. MS2 - yellow star to blue cube, yellow pentagon to red moon. MS3 - yellow star to blue cube, red pentagon to blue moon. MS4 - green cube to blue moon, yellow pentagon to red moon. Reported success rates over

n=50

seeds with 95% confidence intervals (normal approximation).

Task	SWM
LT Reach Block	100%
LT Separate Blocks	100%
OG Reach Cube	97%

Table 3: Planning Results MPPI planning success rates over 100 seeds.

Task	Base Policy	AVD	SWM (Ours)
Push Blue Star to Red Cube	54% $\pm$ 13.8	66% $\pm$ 13.1	86% $\pm$ 9.6
Push Yellow Moon to Purple Cube	54% $\pm$ 13.8	56% $\pm$ 13.8	78% $\pm$ 11.5
Stack Red to Green OOD Background	62% $\pm$ 13.5	28% $\pm$ 12.4	72% $\pm$ 12.4
Stack Blue to Yellow OOD Background	50% $\pm$ 13.9	50% $\pm$ 13.9	70% $\pm$ 12.7

Table 4: Out-of-Distribution Improvement Results. SWM model improvement results on tasks in LangTable and OGBench on out-of-distribution scenes. SWM is able to show policy improvement and outperform AVD across both environments. Reported success rates over

n=50

seeds with 95% confidence intervals (normal approximation).

Figure 6: Visualization of the attention map from language tokens to image patches in the 4th transformer layer. The language tokens correctly attend to the task-relevant locations in the image depending on the prompt.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Architecture of Semantic World Model

Diffusion
Batch size	128
Epochs	100
Action horizon	16
Observation horizon	2
Diffusion iters	100
Eval diffusion iters	10

Appendix

Table 5: Hyperparameters for MPPI, IDQL, Diffusion, and AVD Model

Figure 8: Out-of-distribution configurations for the evaluation tasks. LangTable is configured to have OOD block/color combinations. OGBench is configured to have a different color background.

Type	Example
LangTable
Block touching	Is the red star touching the blue cube?
Peg to block	Is the green cube next to the peg?
Block board position	Is the red star in the center of the board?
Peg block relative direction	Is the peg above the red cube block?
Block to block relative direction	Is the red star to the right of the blue cube?

Appendix

Table 6: Question types and examples for LangTable and OGBench

Task	Question	Weight	Desired Answer
Reaching LT	Is the robotic peg touching the {target_block}?	0.8	Yes
Reaching LT	Is the robotic peg closer to the {target_block}?	0.2	Yes
Reaching OG	Is the robotic gripper touching the {target_block}?	0.8	Yes
Reaching OG	Is the robotic gripper closer to the {target_block}?	0.2	Yes
Separate Blocks	Is the robotic peg touching the {center_block}?	0.6	Yes
Separate Blocks	Is the {avoid block} touching the {center block}?	0.4	No

Appendix

Table 7: QA pairs used for task rewards

Task	Base Policy	IDQL	AVD	SWM
Push Green Cube to Blue Moon	6% $\pm$ 6.6	8% $\pm$ 7.5	48% $\pm$ 13.8	78% $\pm$ 11.5
Push Red Moon to Green Star	18% $\pm$ 10.6	8% $\pm$ 7.5	44% $\pm$ 13.8	80% $\pm$ 11.1
Push Red Pentagon to Blue Moon	14% $\pm$ 9.6	12% $\pm$ 9.0	38% $\pm$ 13.5	80% $\pm$ 11.1
Push Yellow Pentagon to Red Moon	18% $\pm$ 10.6	8% $\pm$ 7.5	34% $\pm$ 13.1	86% $\pm$ 9.6
Push Yellow Star to Blue Cube	16% $\pm$ 10.2	10% $\pm$ 8.3	62% $\pm$ 13.5	84% $\pm$ 10.2
Stack Blue Cube on Yellow Cube	52% $\pm$ 13.8	8% $\pm$ 7.5	50% $\pm$ 13.9	82% $\pm$ 10.6

Appendix

Table 8: Improvement Results. SWM model improvement results on planning tasks in LangTable and OGBench on in-distribution scenes. Reported success rates over

n=50

seeds with 95% confidence intervals (normal approximation). The top tasks are LangTable and the bottom tasks are OGBench.

Task	SWM (with weights)	SWM (no weights)
Push Green Cube to Blue Moon	78% $\pm$ 11.5	72% $\pm$ 12.4
Push Red Moon to Green Star	80% $\pm$ 11.1	78% $\pm$ 11.4
Push Red Pentagon to Blue Moon	80% $\pm$ 11.1	82% $\pm$ 10.6
Push Yellow Pentagon to Red Moon	86% $\pm$ 9.6	88% $\pm$ 9.0
Push Yellow Star to Blue Cube	84% $\pm$ 10.2	76% $\pm$ 11.8
Stack Blue Cube on Yellow Cube	82% $\pm$ 10.6	82% $\pm$ 10.6

Appendix

Table 9: Ablation on question weights. Success rates for SWM with weights vs. SWM without weights on LangTable and OGBench tasks. Reported over

n=50

seeds with 95% confidence intervals.

Task	Base Policy	ID 1	ID 2	OOD 1	OOD 2
Push Red Moon to Green Star	18% $\pm$ 10.6	72% $\pm$ 12.4	88% $\pm$ 9.0	84% $\pm$ 10.2	78% $\pm$ 11.5
Push Yellow Star to Blue Cube	16% $\pm$ 10.2	78 % $\pm$ 11.5	86% $\pm$ 9.6	84% $\pm$ 10.2	72% $\pm$ 12.4
Push Yellow Pentagon to Red Moon	18% $\pm$ 10.6	88% $\pm$ 9.0	88% $\pm$ 9.0	86% $\pm$ 9.6	76% $\pm$ 11.8

Appendix

Table 10: Ablation on Question Phrasing. Success rates of SWM under in-distribution (ID) and out-of-distribution (OOD) task phrasings. Reported with 95% confidence intervals.

Table 19

Task	Base Policy	SWM+grad	SWM+grad + LLM planning
MS1	6	50	62
MS2	4	66	42
MS3	4	54	52
MS4	2	54	44
Average	4	56	50

Appendix

Table 14: Success rate (%) on long-horizon LangTable tasks with and without LLM-derived task decompositions over 50 seeds. MS1 - red pentagon to blue moon, yellow pentagon to red moon. MS2 - yellow star to blue cube, yellow pentagon to red moon. MS3 - yellow star to blue cube, red pentagon to blue moon. MS4 - green cube to blue moon, yellow pentagon to red moon.

Figure 9: Visualization of gradient-based planning on the LangTable - Red Pentagon to Blue Moon task. The initially proposed action sequence is on the left, and updates to this action sequence go progressively to the right, approaching the optimal trajectory over successive gradient steps.

Figure 10: Attention maps in different layers of SWM. Question: “Is the red moon touching the blue block?”

Figure 11: Attention maps for different trajectories. Question: “Is the red moon touching the blue block?”

Method	Time per action chunk (s)
MPPI	4.48
Gradient-based	1.56

Appendix

Table 15: Planning speed comparison between MPPI and gradient-based planning, measured on a single GPU in the LangTable environment.

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 384 papers

Grounded World Model: Latent Planning with Language Goals
Apr 13, 2026Quanyi Li, Lan Feng, Haonan Zhang +4Language-Conditioned Robot Planning World Models for Robotics
EPFL · NUS
From World Models to World Action Models: A Concise Tutorial for Robotics
Jul 1, 2026Xiaoxiong Zhang, Xiong Zeng, Wei ZhangWorld Model Learning World Models for Robotics
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · LimX Dynamics, Shenzhen, China
H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model
Feb 11, 2026Jinbang Huang, Wenyuan Chen, Zhiyuan Li +9Long-Horizon Robotic Manipulation World Models for Robotics
Huawei Noah’s Ark Lab · University of Toronto · University of British Columbia +1

Apr 13, 2026J/K move · Enter open · S save

Grounded World Model: Latent Planning with Language Goals

Quanyi Li, Lan Feng, Haonan Zhang, Wuyang Li+3

EPFL · NUS

Language-Conditioned Robot Planning World Models for Robotics World Model-Based Planning Latent World Models Zero-Shot Robotic Manipulation Action-Conditioned World Models+8

Open paper AskPDF arXiv

World models such as DINO-WM and LeWM specify the goal with an image, which is difficult to obtain in advance for novel tasks. We present the Grounded World Model (GWM), a latent world model that enables zero-shot planning in the real world from language goals alone. Given a candidate action sequence and the current observation, GWM predicts the future in the visual space of a pretrained video-language embedding model. The frozen readout of this embedding model maps this imagined future and the task description into the same embedding space, where their negative cosine similarity serves as the planning cost. Training GWM requires only offline and task-agnostic video-action pairs and no language labels. In simulated experiments on WISER, planning with GWM, which executes the candidate action of lowest cost, solves 87% of 288 tasks with unseen instructions and visual signals, while ten fine-tuned VLAs average 22%. We then scale GWM up with real robot data, and use it for zero-shot planning in realistic simulation and real scenes. In the IsaacSim evaluation, planning with GWM completes all 70 trials across 14 tasks that require reasoning over referring expressions, matching a modular planner grounded by a frontier VLM, while pi0.5 reaches 37/70. Deployed on a real Franka, the same stack completes 55/60 separately evaluated pick-and-place sub-tasks, comparable to the modular planner's 52/60, with the full system running locally on a single consumer GPU. Project website: https://quanyili.github.io/gwm-wiser/.

Abstract

World Models for Robotics

Explore similar work

Grounded World Model: Latent Planning with Language Goals

From World Models to World Action Models: A Concise Tutorial for Robotics

H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model