Spatial reasoning requires understanding how objects and observers are arranged in a scene and how their relations change with viewpoint or movement. Vision-language models (VLMs) typically perform both steps through model-internal reasoning, forming an implicit scene interpretation from partial views and imagining the consequences of spatial transformations. A misread scene can lead to a wrong answer, while even a correct interpretation does not ensure that imagined transformations satisfy spatial constraints. We introduce Spatial Harness, which represents scenes as explicit and executable world states to ground spatial reasoning in physics-engine feedback. A Perceiver integrates scene constraints extracted from the question with specialized visual estimates to place objects and cameras in a shared coordinate frame. The resulting state is instantiated in a physics engine, where a Reasoner executes viewpoint changes and uses computed measurements and rendered views to guide reasoning. We evaluate on MindCube, SAT, MMSI-Bench, and SpatialRGPT-Bench with open- and closed-weight VLMs. On MindCube, Spatial Harness improves Qwen3.5-27B from 52.70% to 71.14%, surpassing pySpatial by 7.52 percentage points. Given identical annotated object and camera poses, it reaches 98.25% versus 71.81% for model-internal reasoning. Code is publicly available at: https://github.com/tmlr-group/Spatial-Harness
Figures & tables
Figure 1: From model-internal reasoning to executable world states. Both approaches proceed from scene representation through spatial transformation to answer derivation. Top: a VLM implicitly represents the scene and imagines the requested turn, leaving the resulting spatial relation unchecked. Bottom: Spatial Harness constructs an explicit world state and executes the turn in a physics engine, grounding the answer in computed feedback. This makes both the estimated layout and the consequences of the transformation available for inspection. Execution enforces consistency with this state, whose accuracy still depends on the available language constraints and visual estimates.
Figure 2: Spatial reasoning through iterative execution and feedback. Given three views linked by 90∘ camera rotations, the Perceiver constructs a shared world state and instantiates it in MuJoCo (steps 1–3). The Reasoner executes the requested 180∘ turn and renders the new view (steps 4–6). The window is now in front, but the question asks what is behind. The Reasoner therefore turns again to inspect the opposite direction, identifies the grey-green decorative wall, and selects C (steps 7–10).
Figure 3: Separating scene information from its use in reasoning. MindCube accuracy across seven VLMs. Gray and blue bars denote direct reasoning (Base) and Spatial Harness, respectively. Solid bars use images and questions without a ground-truth (GT) map, while hatched bars supply both methods with the same GT layout. With identical GT layouts, Spatial Harness outperforms Base on every backbone, demonstrating gains from executing the supplied layout.
Figure 4: Overview of Spatial Harness. The Perceiver combines question-derived constraints with specialized visual estimates, aligning camera frames and fusing observations into a JSON world state s=(L,Ω,C) : a shared layout, per-view observations, and stated constraints. The Reasoner instantiates L in MuJoCo and issues Python tool calls for viewpoint control, measurement, and rendering. Computed measurements and rendered views update its context and guide subsequent operations and the final answer. The JSON and code are illustrative excerpts.
Figure 5: World state and its engine instantiation. Camera poses align per-view observations into a shared layout (Eq. 2 ), instantiated as labeled proxies. For display, the first-camera origin is shifted to (4,6) , with y increasing downward; Ti combines camera rotation with this grid-axis convention.
Figure 6: Engine interface exposed to the Reasoner. Representative mujoco_sim calls control the camera pose, return geometric measurements, and render views of the same scene. The pose persists across calls. Appendix C.3 gives the full signatures, argument defaults, and return values.
Model
Method
Rotation
Among
Around
Overall
Qwen2.5-VL-7B-Instruct
Base (w/o map)
35.83
33.39
40.67
35.59
Base (w map)
35.50
27.56
32.93
30.35
Spatial Harness (w map)
34.33
58.17
70.27
56.51 (+26.16)
Qwen2.5-VL-32B-Instruct
Base (w/o map)
38.83
38.17
46.00
40.16
Base (w map)
40.83
40.11
65.73
46.35
Spatial Harness (w map)
69.00
88.39
84.53
83.78 (+37.43)
Table 1: MindCube accuracy (%) in the GT-map setting. Base (w/o map) answers from the images and question; Base (w map) and Spatial Harness (w map) receive the same GT map. Parentheses give the gain over Base (w map). Bold: best overall accuracy for each model; underline: second best.
Model
Method
Rotation
Among
Around
Overall
Qwen2.5-VL-72B-Instruct
Base
40.83
41.06
42.93
41.46
MindJourney
42.00
40.50
49.20
42.86
pySpatial
30.50
55.50
53.60
50.29
Spatial Harness
49.50
58.00
53.60
55.33
Qwen3.5-27B
Base
77.17
41.72
59.47
52.70
MindJourney
82.00
42.00
65.60
55.24
Table 2: End-to-end accuracy (%) on MindCube. Spatial Harness reasons over the world state estimated by its Perceiver. Bold: best overall accuracy for each model; underline: second best.
Model
Method
SAT
MMSI-Bench
SpatialRGPT-Bench
Avg
Qwen2.5-VL-72B-Instruct
Base
68.60
29.60
40.72
46.31
MindJourney
68.60
30.60
39.12
46.11
pySpatial
55.40
33.20
31.64
40.08
Spatial Harness
71.60
29.00
84.11
61.57
Qwen3.5-27B
Base
81.80
33.40
48.20
54.47
MindJourney
78.40
33.00
45.66
52.35
Table 3: End-to-end results (%) on SAT, MMSI-Bench, and SpatialRGPT-Bench. SAT and MMSI-Bench report answer accuracy, and SpatialRGPT-Bench reports the fraction of predictions within its error tolerance. Avg is the mean over the three benchmarks. Per-category results are in Tables 6 – 8 . Bold: best result for each model and column; underline: second best.
Map source
Rotation
Among
Around
Overall
VLM-generated map
54.00
28.83
61.20
41.33
Perceiver without text channel
71.00
63.00
79.20
68.38
Perceiver without visual channel
72.50
49.67
45.20
52.95
Full Perceiver (Spatial Harness)
81.50
68.00
70.40
71.14
GT map (reference)
96.83
98.06
99.87
98.25
Table 4: Ablation on world state construction on MindCube with Qwen3.5-27B (accuracy in %). The Reasoner is fixed, and only the source of the map changes; the GT map serves as a reference.
Figure 7: Channel ablations on MindCube with Qwen3.5-27B. (a) Each condition removes one module from the text channel; the entity–relation module extracts entities and relations jointly. (b) Each condition replaces one specialized model with the VLM, which produces the same output fields.
Feedback condition
Rotation
Among
Around
Overall
Base
83.83
60.94
88.27
71.81
Multi-turn self-review
96.00
66.67
90.00
77.81
Numerical feedback only
93.00
95.00
99.20
95.62
Rendered feedback only
95.00
92.33
96.00
93.71
Spatial Harness
96.83
98.06
99.87
98.25
Table 5: Ablation on grounded feedback on MindCube with Qwen3.5-27B (accuracy in %). All conditions reason over the same GT map and differ only in the feedback available to the Reasoner.
Figure 8: Quality of the world states estimated by the Perceiver with Qwen3.5-27B on MindCube. Maps are scored against the GT maps with the MindCube cognitive-map metrics after name alignment (Appendix D.5 ); scores lie in [0,1] (higher is better).
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Base prompt template on MindCube in the end-to-end setting.
Figure 10: Base prompt template in the GT-map setting.
Figure 11: Top-level prompt of the Reasoner.
Figure 12: MuJoCo API specification given to the Reasoner.
Figure 13: Tool-use guide given to the Reasoner.
Figure 14: Tool-use guide given to the Reasoner (continued).
Figure 15: Scene note given to the Reasoner: it describes MuJoCo’s stylized renderings (colored cylinders with labels rather than photo-realistic objects) and the camera initialization from the map.
Figure 16: Prompt used to elicit a cognitive map from a VLM.
Figure 17: LLM-as-judge prompt for scoring free-form answers.
Figure 18: An example from the MindCube dataset. The top shows the question, the left presents four viewpoint images, and the right visualizes the corresponding cognitive map (see Figure 19 ).
Figure 19: Example cognitive map. Five objects on a 10×10 bird’s-view grid ( [0,0] at the top-left, [9,9] at the bottom-right; +x right, +y down), and four camera views around the central black sneaker ; facing is one of { up , down , left , right }.
Model
Method
EgoM
ObjM
Act
Goal
Pers
Overall
Qwen2.5-VL-72B-Instruct
Base
84.00
70.00
56.00
71.00
62.00
68.60
MindJourney
86.00
61.00
57.00
90.00
49.00
68.60
pySpatial
42.00
41.00
56.00
73.00
65.00
55.40
Spatial Harness
76.00
70.00
65.00
82.00
65.00
71.60
Qwen3.5-27B
Base
95.00
81.00
71.00
91.00
71.00
81.80
MindJourney
99.00
75.00
74.00
89.00
55.00
78.40
Appendix
Table 6: End-to-end accuracy (%) on the sampled synthetic split of SAT. EgoM: ego movement; ObjM: object movement; Act: action consequence; Goal: goal aim; Pers: perspective.
Model
Method
Pos.
Attr.
Motion
Multi-step
Overall
Qwen2.5-VL-72B-Instruct
Base
30.84
34.62
24.00
27.27
29.60
MindJourney
31.42
35.38
28.00
27.27
30.60
pySpatial
33.33
36.15
34.00
30.30
33.20
Spatial Harness
29.89
34.62
23.33
27.27
29.00
Qwen3.5-27B
Base
35.44
34.62
32.67
27.78
33.40
MindJourney
32.76
38.46
26.00
35.35
33.00
Appendix
Table 7: End-to-end accuracy (%) on MMSI-Bench. Pos.: positional relationship; Attr.: attribute; Multi-step: multi-step reasoning. Overall aggregates all questions.
Model
Method
Dir.
Dist.
Height
H-dist.
V-dist.
Width
Overall
Qwen2.5-VL-72B-Instruct
Base
60.75
25.00
58.65
33.61
26.42
42.11
40.72
MindJourney
65.42
25.00
55.64
31.15
19.81
39.85
39.12
pySpatial
54.21
14.86
42.86
22.95
22.64
36.09
31.64
Spatial Harness
98.13
89.86
75.19
88.52
86.79
69.17
84.11
Qwen3.5-27B
Base
60.75
31.76
73.68
35.25
36.79
51.88
48.20
MindJourney
63.55
28.38
66.17
36.89
28.30
51.88
45.66
Appendix
Table 8: Quantitative success rate (%) on SpatialRGPT-Bench. H-dist. and V-dist. denote horizontal and vertical distance. Scores use the benchmark metric and direction tolerances.
Model
Method
Rotation
Among
Around
Overall
Qwen2.5-VL-72B-Instruct
Base
40.83
41.06
42.93
41.46
GCA
39.00
32.67
31.20
33.52
Spatial Harness
49.50
58.00
53.60
55.33
Qwen3.5-27B
Base
77.17
41.72
59.47
52.70
GCA
81.50
51.33
73.20
62.29
Spatial Harness
81.50
68.00
70.40
71.14
Appendix
Table 9: End-to-end accuracy (%) on MindCube compared with GCA . Bold: best overall accuracy for each model; underline: second best.
Corruption
Low
Mid
High
One object position
1 cell 92.13
2 cells 89.40
3 cells 87.78
One camera yaw
(0,30∘] 98.10
(30∘,90∘] 84.89
(90∘,180∘] 76.67
Object facing ( 90∘ rotation)
1 object 96.06
N/A
2 objects 94.32
Appendix
Table 10: MindCube accuracy (%) under single-dimension corruption of the GT map, with Qwen3.5-27B. The uncorrupted reference accuracy is 98.25 . Each row changes only the specified field. Object facing is evaluated on the 364 and 129 questions with at least one and two annotated facings, whose uncorrupted accuracies are 97.53 and 98.45 .
Model
Method
Rotation
Among
Around
Overall
Qwen2.5-VL-7B-Instruct
Base (w/o map)
35.83 ± 0.76
33.39 ± 0.10
40.67 ± 0.83
35.59 ± 0.29
Base (w map)
35.50 ± 0.50
27.56 ± 0.35
32.93 ± 0.92
30.35 ± 0.29
Spatial Harness (w map)
34.33 ± 4.31
58.17 ± 0.44
70.27 ± 2.20
56.51 ± 1.10
Qwen2.5-VL-32B-Instruct
Base (w/o map)
38.83 ± 0.29
38.17 ± 0.29
46.00 ± 0.40
40.16 ± 0.22
Base (w map)
40.83 ± 2.47
40.11 ± 0.51
65.73 ± 0.61
46.35 ± 0.38
Spatial Harness (w map)
69.00 ± 1.00
88.39 ± 0.54
84.53 ± 1.15
83.78 ± 0.71
Appendix
Table 11: Run-to-run variation on MindCube in the GT-map setting. Entries are mean ± sample standard deviation of accuracy (%) over three runs.
Figure 20: The success case (item among_group603_q3_5_1 ). The top shows the question, and the bottom presents the four images, numbered as in the question. The reference answer is B.
Figure 21: View rendered by capture_image() in the first turn of the success case.
Figure 22: The failure case (item among_group738_q2_5_1 ). The top shows the question, and the bottom presents the four images, numbered as in the question. The reference answer is A.
Figure 23: Views rendered by capture_image() in the three tool-calling turns of the failure case, in order.
Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intelligence demands predictive spatial reasoning: constructing a scene from observations, anticipating how an intervention changes it, and reasoning about the unseen outcome. We introduce SpaceCast-Bench, the first benchmark to directly and diagnostically evaluate this capability. Built around an observe-transform-infer framework, its 3,862 questions from 182 real-world scenes span 16 task types at three levels: static perception, local prediction, and global prediction, progressively requiring scene understanding, spatial state updating, and relational inference over unobserved outcomes. Evaluating 21 models exposes a stark gap: the strongest model reaches only 58.0% against 87.2% human performance, while spatially specialized models remain near random chance. Controlled analyses further reveal that bridge views are critical for integrating distributed observations, and that explicit 3D evidence benefits models more reliably than generated outcome images or videos. Fine-tuning on our programmatically generated data lifts Qwen3-VL-4B from 34.0% to 65.7% with macro-average gains across six out-of-domain benchmarks.
Vision-Language Models (VLMs) have continuous deficiency in spatial reasoning tasks that require mental spatial transformation, such as reasoning under hypothetical changes in viewpoint or position, known as perspective-taking tasks. Existing approaches either scale linguistic reasoning or augment VLMs with static geometric representations, but neither directly enables active transformation of the represented scene. World models offer this capability, yet existing systems typically communicate through synthesized RGB images, introducing a latent-to-pixel-to-latent bottleneck. We introduce WorldMind, a framework for test-time scaling of spatial reasoning through latent world simulation. Given a query and observation, the VLM selects useful spatial transformations, while a world model executes them and exposes aligned intermediate latent features directly to the VLM. The resulting latent evidence is incorporated into subsequent reasoning steps, enabling iterative spatial simulation without repeatedly decoding imagined states into pixels. To optimize this interaction, WorldMind further employs process-verifiable reinforcement learning that rewards reasoning progress only when the associated action and latent transition are valid. WorldMind thus scales spatial reasoning by allocating additional inference-time computation to active world simulation rather than generating more language or pixels. Across VSI-Bench, MMSI-Bench, and MindCube-Tiny, WorldMind achieves an average improvement of 16.0% over the base VLM and 8.3% over the strongest RGB-based world simulation.
Vision-language models (VLMs) have shown strong performance on static visual understanding, yet they still struggle with dynamic spatial reasoning that requires imagining how scenes evolve under egocentric motion. Recent efforts address this limitation either by scaling spatial supervision with synthetic data or by coupling VLMs with world models at inference time. However, the former often lacks explicit modeling of motion-conditioned state transitions, while the latter incurs substantial computational overhead. In this work, we propose World2VLM, a training framework that distills spatial imagination from a generative world model into a vision-language model. Given an initial observation and a parameterized camera trajectory, we use a view-consistent world model to synthesize geometrically aligned future views and derive structured supervision for both forward (action-to-outcome) and inverse (outcome-to-action) spatial reasoning. We post-train the VLM with a two-stage recipe on a compact dataset generated by this pipeline and evaluate it on multiple spatial reasoning benchmarks. World2VLM delivers consistent improvements over the base model across diverse benchmarks, including SAT-Real, SAT-Synthesized, VSI-Bench, and MindCube. It also outperforms the test-time world-model-coupled methods while eliminating the need for expensive inference-time generation. Our results suggest that world models can serve not only as inference-time tools, but also as effective training-time teachers, enabling VLMs to internalize spatial imagination in a scalable and efficient manner.
Wanyue Zhang, Wenxiang Wu, Wang Xu +6
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Harbin Institute of Technology +2