Long-horizon tasks in embodied and web environments involve extended decision sequences and delayed feedback, making accurate planning crucial for success. World models tackle this by learning latent dynamics for imagination-based planning. However, existing approaches merely rely on dense pixel-level observations, which forces the model to absorb redundant information while overlooking relations among task-relevant entities. Consequently, small errors may accumulate over long imagined rollouts and gradually corrupt the predictions. In this paper, we explore world models that imagine over sparse state abstractions, which poses two key challenges: 1) how to remove redundant information and form a compact state abstraction; 2) how to incorporate the relational structure to learn reliable long-horizon dynamics. To address these challenges, we propose Embodied Graph, a relational state abstraction that supports structured dynamics modeling and long-horizon imagination, and the Relational World Model (RWM), which performs imagination directly over graph states. RWM consists of (i) a Task-Aware Embodied Graph Constructor that filters redundant details and maintains graph memory through online updates, and (ii) an Embodied Graph Recurrent State Space Model that learns node-wise latent dynamics while capturing inter-node dependencies. The imagined graph states provide predictions of action consequences to guide decision-making. Experiments on Minecraft and Crafter demonstrate the effectiveness of RWM in embodied environments. Notably, in Minecraft, RWM is the only method that obtains diamonds within the given budget. Experiments on web interaction further illustrate the applicability of relational state prediction to the decision making process of web agents.
Figures & tables
Figure 1. Imagination drift in a pixel-level world model. A minor early inaccuracy (a smoothed sandbar) can accumulate into a severe deviation (a hallucinated oasis) that would lead the agent toward an implausible state.
Figure 2. Overall framework of RWM. A Task-Aware Embodied Graph Constructor builds and continuously updates a compact embodied graph, while an Embodied Graph Recurrent State Space Model learns action-conditioned, node-wise dynamics through relational propagation. The graph imagination model predicts future node attributes and structural outcomes, providing forecasts of action consequences to guide decision-making in dynamic environments.
Steps
Model
Stone
Iron
Gold
Diamond
SR
SL
SR
SL
SR
SL
SR
SL
1M
PPO
92.40 ± 3.99
358 ± 8
28.76 ± 4.68
2628 ± 540
0.00 ± 0.00
–
0.00 ± 0.00
–
Rainbow
95.05 ± 0.70
663 ± 324
29.80 ± 3.77
2320 ± 635
0.00 ± 0.00
–
0.00 ± 0.00
–
DreamerV3
97.03 ± 0.80
264 ± 18
30.82 ± 4.66
1875 ± 255
2.58 ± 2.21
1190 ± 92
0.00 ± 0.00
–
LS-Imagine
96.41 ± 1.78
279 ± 78
25.11 ± 4.64
2103 ± 184
1.89 ± 1.37
5522 ± 721
0.00 ± 0.00
–
RWM (Ours)
95.57 ± 2.76
420 ± 138
32.89 ± 4.62
2115 ± 113
4.19 ± 0.53
3054 ± 1351
1.47 ± 1.11
8198 ± 708
Table 1. Minecraft results after 1M–5M environment steps: success rate (SR ↑ , %) and success length (SL ↓ , steps, “– if none) are reported mean ± std over 3 seeds. Best in bold, second best underlined. Ranks are computed among methods with non-zero SR.
Model
Metric
1M
2M
3M
4M
5M
RWM
SR
1.47 ± 1.11
1.55 ± 1.12
2.97 ± 1.80
5.72 ± 1.78
6.94 ± 2.12
SL
8198 ± 708
5401 ± 297
5306 ± 236
4740 ± 295
4512 ± 1014
Variant 1
SR
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
SL
–
–
–
–
–
Variant 2
SR
0.00 ± 0.00
1.33 ± 0.62
1.79 ± 0.71
1.66 ± 0.69
2.48 ± 0.95
SL
–
6750 ± 294
5730 ± 201
5971 ± 337
3945 ± 883
Table 2. Ablation study in the Minecraft diamond task. Variant 1: w/o admission scorer. Variant 2: w/o relational propagation. Success rate (SR, %) and success length (SL, steps) are reported by mean ± std over 3 seeds.
Model
PPO
Rainbow
DreamerV3
RWM
Score (%)
4.6 ± 0.3
4.3 ± 0.2
14.5 ± 1.6
15.8 ± 2.0
Reward
4.2 ± 1.2
6.0 ± 1.3
11.7 ± 1.9
12.2 ± 1.7
Table 3. Experimental results in Crafter after 1M environment steps (mean ± std).
Figure 3. Open-loop imagination drift under different imagination lengths. Compared with DreamerV3, RWM maintains a nearly stable prediction error over long rollouts, showing that our proposed method effectively reduces compounding imagination drift.
Figure 4. Case study on CMS35. Top: Case 1, typing a phone number into the search box. Bottom: Case 2, selecting the report period. Each case is enlarged around the changed element and shows (a) the current screen, (b) the actual next screen after the verifier’s choice, (c) the image RSSM’s prediction for the verifier’s choice, and (d) its prediction for WebRL’s proposal. RWM forecasts the new values exactly ( 8015551212 in the search box and month in the Period drop-down), whereas the image RSSM’s predictions are blurred and nearly identical for the two candidates.
Method
CMS SR
GPT-4-Turbo
14.3
GPT-4o
20.0
AutoWebGLM
28.6
Llama3.1-8B + DigiRL
37.1
GLM-4-9B + WebRL
48.6
WebRL-Llama-3.1-8B
54.3
Table 4. Success rate (%) on WebArena-Lite CMS35. Upper block: published results ( Qi et al., 2025 ) . Lower block: our runs, in which WebRL-Llama-3.1-8B proposes the candidates and the same verifier selects one. The variants differ only in the forecasts given to the verifier.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5. Perception front-end in Minecraft. Each row corresponds to a 64×64 observation (left), the segmentation (middle), and the resulting label map (right).
Hyperparameter
Value
Hyperparameter
Value
Macro-action decision interval ( H )
50
Decoder layers
3
Max nodes in the embodied graph ( N )
128
Policy layers
2
Max observation candidates ( M )
256
Value layers
3
node matching threshold ( δ )
2.0
Imagination length ( K )
15
KNN neighbors ( Knn )
4
Loss scale (dynamics) ( λdyn )
1.0
Replay size
5×105
Loss scale (representation) ( λrep )
0.1
Appendix
Table 5. Key hyperparameters for RWM. Unless otherwise specified, shared hyperparameters follow the default DreamerV3 configuration ( Hafner and et al., 2025 ) .
Figure 6. Agent ability spectrum for Crafter (ordered left to right by number of prerequisites for an achievement). The y-axis is shown in log scale. RWM achieves comparable performance on common early-stage achievements and stronger performance on harder long-horizon achievements involving resource collection and tool crafting.
School of Computing Technologies, RMIT University, Australia · Faculty of Information Technology, Monash University, Australia · KStelrix Star Dynamics Lab, KStelrix, China +1
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Amazon AGI SF Lab · Institute for Artificial Intelligence, Peking University