Spatial reasoning requires understanding how objects and observers are arranged in a scene and how their relations change with viewpoint or movement. Vision-language models (VLMs) typically perform both steps through model-internal reasoning, forming an implicit scene interpretation from partial views and imagining the consequences of spatial transformations. A misread scene can lead to a wrong answer, while even a correct interpretation does not ensure that imagined transformations satisfy spatial constraints. We introduce Spatial Harness, which represents scenes as explicit and executable world states to ground spatial reasoning in physics-engine feedback. A Perceiver integrates scene constraints extracted from the question with specialized visual estimates to place objects and cameras in a shared coordinate frame. The resulting state is instantiated in a physics engine, where a Reasoner executes viewpoint changes and uses computed measurements and rendered views to guide reasoning. We evaluate on MindCube, SAT, MMSI-Bench, and SpatialRGPT-Bench with open- and closed-weight VLMs. On MindCube, Spatial Harness improves Qwen3.5-27B from 52.70% to 71.14%, surpassing pySpatial by 7.52 percentage points. Given identical annotated object and camera poses, it reaches 98.25% versus 71.81% for model-internal reasoning. Code is publicly available at: https://github.com/tmlr-group/Spatial-Harness
Figures & tables
Figure 1: From model-internal reasoning to executable world states. Both approaches proceed from scene representation through spatial transformation to answer derivation. Top: a VLM implicitly represents the scene and imagines the requested turn, leaving the resulting spatial relation unchecked. Bottom: Spatial Harness constructs an explicit world state and executes the turn in a physics engine, grounding the answer in computed feedback. This makes both the estimated layout and the consequences of the transformation available for inspection. Execution enforces consistency with this state, whose accuracy still depends on the available language constraints and visual estimates.
Figure 2: Spatial reasoning through iterative execution and feedback. Given three views linked by 90∘ camera rotations, the Perceiver constructs a shared world state and instantiates it in MuJoCo (steps 1–3). The Reasoner executes the requested 180∘ turn and renders the new view (steps 4–6). The window is now in front, but the question asks what is behind. The Reasoner therefore turns again to inspect the opposite direction, identifies the grey-green decorative wall, and selects C (steps 7–10).
Figure 3: Separating scene information from its use in reasoning. MindCube accuracy across seven VLMs. Gray and blue bars denote direct reasoning (Base) and Spatial Harness, respectively. Solid bars use images and questions without a ground-truth (GT) map, while hatched bars supply both methods with the same GT layout. With identical GT layouts, Spatial Harness outperforms Base on every backbone, demonstrating gains from executing the supplied layout.
Figure 4: Overview of Spatial Harness. The Perceiver combines question-derived constraints with specialized visual estimates, aligning camera frames and fusing observations into a JSON world state s=(L,Ω,C) : a shared layout, per-view observations, and stated constraints. The Reasoner instantiates L in MuJoCo and issues Python tool calls for viewpoint control, measurement, and rendering. Computed measurements and rendered views update its context and guide subsequent operations and the final answer. The JSON and code are illustrative excerpts.
Figure 5: World state and its engine instantiation. Camera poses align per-view observations into a shared layout (Eq. 2 ), instantiated as labeled proxies. For display, the first-camera origin is shifted to (4,6) , with y increasing downward; Ti combines camera rotation with this grid-axis convention.
Figure 6: Engine interface exposed to the Reasoner. Representative mujoco_sim calls control the camera pose, return geometric measurements, and render views of the same scene. The pose persists across calls. Appendix C.3 gives the full signatures, argument defaults, and return values.
Model
Method
Rotation
Among
Around
Overall
Qwen2.5-VL-7B-Instruct
Base (w/o map)
35.83
33.39
40.67
35.59
Base (w map)
35.50
27.56
32.93
30.35
Spatial Harness (w map)
34.33
58.17
70.27
56.51 (+26.16)
Qwen2.5-VL-32B-Instruct
Base (w/o map)
38.83
38.17
46.00
40.16
Base (w map)
40.83
40.11
65.73
46.35
Spatial Harness (w map)
69.00
88.39
84.53
83.78 (+37.43)
Table 1: MindCube accuracy (%) in the GT-map setting. Base (w/o map) answers from the images and question; Base (w map) and Spatial Harness (w map) receive the same GT map. Parentheses give the gain over Base (w map). Bold: best overall accuracy for each model; underline: second best.
Model
Method
Rotation
Among
Around
Overall
Qwen2.5-VL-72B-Instruct
Base
40.83
41.06
42.93
41.46
MindJourney
42.00
40.50
49.20
42.86
pySpatial
30.50
55.50
53.60
50.29
Spatial Harness
49.50
58.00
53.60
55.33
Qwen3.5-27B
Base
77.17
41.72
59.47
52.70
MindJourney
82.00
42.00
65.60
55.24
Table 2: End-to-end accuracy (%) on MindCube. Spatial Harness reasons over the world state estimated by its Perceiver. Bold: best overall accuracy for each model; underline: second best.
Model
Method
SAT
MMSI-Bench
SpatialRGPT-Bench
Avg
Qwen2.5-VL-72B-Instruct
Base
68.60
29.60
40.72
46.31
MindJourney
68.60
30.60
39.12
46.11
pySpatial
55.40
33.20
31.64
40.08
Spatial Harness
71.60
29.00
84.11
61.57
Qwen3.5-27B
Base
81.80
33.40
48.20
54.47
MindJourney
78.40
33.00
45.66
52.35
Table 3: End-to-end results (%) on SAT, MMSI-Bench, and SpatialRGPT-Bench. SAT and MMSI-Bench report answer accuracy, and SpatialRGPT-Bench reports the fraction of predictions within its error tolerance. Avg is the mean over the three benchmarks. Per-category results are in Tables 6 – 8 . Bold: best result for each model and column; underline: second best.
Map source
Rotation
Among
Around
Overall
VLM-generated map
54.00
28.83
61.20
41.33
Perceiver without text channel
71.00
63.00
79.20
68.38
Perceiver without visual channel
72.50
49.67
45.20
52.95
Full Perceiver (Spatial Harness)
81.50
68.00
70.40
71.14
GT map (reference)
96.83
98.06
99.87
98.25
Table 4: Ablation on world state construction on MindCube with Qwen3.5-27B (accuracy in %). The Reasoner is fixed, and only the source of the map changes; the GT map serves as a reference.
Figure 7: Channel ablations on MindCube with Qwen3.5-27B. (a) Each condition removes one module from the text channel; the entity–relation module extracts entities and relations jointly. (b) Each condition replaces one specialized model with the VLM, which produces the same output fields.
Feedback condition
Rotation
Among
Around
Overall
Base
83.83
60.94
88.27
71.81
Multi-turn self-review
96.00
66.67
90.00
77.81
Numerical feedback only
93.00
95.00
99.20
95.62
Rendered feedback only
95.00
92.33
96.00
93.71
Spatial Harness
96.83
98.06
99.87
98.25
Table 5: Ablation on grounded feedback on MindCube with Qwen3.5-27B (accuracy in %). All conditions reason over the same GT map and differ only in the feedback available to the Reasoner.
Figure 8: Quality of the world states estimated by the Perceiver with Qwen3.5-27B on MindCube. Maps are scored against the GT maps with the MindCube cognitive-map metrics after name alignment (Appendix D.5 ); scores lie in [0,1] (higher is better).
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Base prompt template on MindCube in the end-to-end setting.
Figure 10: Base prompt template in the GT-map setting.
Figure 11: Top-level prompt of the Reasoner.
Figure 12: MuJoCo API specification given to the Reasoner.
Figure 13: Tool-use guide given to the Reasoner.
Figure 14: Tool-use guide given to the Reasoner (continued).
Figure 15: Scene note given to the Reasoner: it describes MuJoCo’s stylized renderings (colored cylinders with labels rather than photo-realistic objects) and the camera initialization from the map.
Figure 16: Prompt used to elicit a cognitive map from a VLM.
Figure 17: LLM-as-judge prompt for scoring free-form answers.
Figure 18: An example from the MindCube dataset. The top shows the question, the left presents four viewpoint images, and the right visualizes the corresponding cognitive map (see Figure 19 ).
Figure 19: Example cognitive map. Five objects on a 10×10 bird’s-view grid ( [0,0] at the top-left, [9,9] at the bottom-right; +x right, +y down), and four camera views around the central black sneaker ; facing is one of { up , down , left , right }.
Model
Method
EgoM
ObjM
Act
Goal
Pers
Overall
Qwen2.5-VL-72B-Instruct
Base
84.00
70.00
56.00
71.00
62.00
68.60
MindJourney
86.00
61.00
57.00
90.00
49.00
68.60
pySpatial
42.00
41.00
56.00
73.00
65.00
55.40
Spatial Harness
76.00
70.00
65.00
82.00
65.00
71.60
Qwen3.5-27B
Base
95.00
81.00
71.00
91.00
71.00
81.80
MindJourney
99.00
75.00
74.00
89.00
55.00
78.40
Appendix
Table 6: End-to-end accuracy (%) on the sampled synthetic split of SAT. EgoM: ego movement; ObjM: object movement; Act: action consequence; Goal: goal aim; Pers: perspective.
Model
Method
Pos.
Attr.
Motion
Multi-step
Overall
Qwen2.5-VL-72B-Instruct
Base
30.84
34.62
24.00
27.27
29.60
MindJourney
31.42
35.38
28.00
27.27
30.60
pySpatial
33.33
36.15
34.00
30.30
33.20
Spatial Harness
29.89
34.62
23.33
27.27
29.00
Qwen3.5-27B
Base
35.44
34.62
32.67
27.78
33.40
MindJourney
32.76
38.46
26.00
35.35
33.00
Appendix
Table 7: End-to-end accuracy (%) on MMSI-Bench. Pos.: positional relationship; Attr.: attribute; Multi-step: multi-step reasoning. Overall aggregates all questions.
Model
Method
Dir.
Dist.
Height
H-dist.
V-dist.
Width
Overall
Qwen2.5-VL-72B-Instruct
Base
60.75
25.00
58.65
33.61
26.42
42.11
40.72
MindJourney
65.42
25.00
55.64
31.15
19.81
39.85
39.12
pySpatial
54.21
14.86
42.86
22.95
22.64
36.09
31.64
Spatial Harness
98.13
89.86
75.19
88.52
86.79
69.17
84.11
Qwen3.5-27B
Base
60.75
31.76
73.68
35.25
36.79
51.88
48.20
MindJourney
63.55
28.38
66.17
36.89
28.30
51.88
45.66
Appendix
Table 8: Quantitative success rate (%) on SpatialRGPT-Bench. H-dist. and V-dist. denote horizontal and vertical distance. Scores use the benchmark metric and direction tolerances.
Model
Method
Rotation
Among
Around
Overall
Qwen2.5-VL-72B-Instruct
Base
40.83
41.06
42.93
41.46
GCA
39.00
32.67
31.20
33.52
Spatial Harness
49.50
58.00
53.60
55.33
Qwen3.5-27B
Base
77.17
41.72
59.47
52.70
GCA
81.50
51.33
73.20
62.29
Spatial Harness
81.50
68.00
70.40
71.14
Appendix
Table 9: End-to-end accuracy (%) on MindCube compared with GCA . Bold: best overall accuracy for each model; underline: second best.
Corruption
Low
Mid
High
One object position
1 cell 92.13
2 cells 89.40
3 cells 87.78
One camera yaw
(0,30∘] 98.10
(30∘,90∘] 84.89
(90∘,180∘] 76.67
Object facing ( 90∘ rotation)
1 object 96.06
N/A
2 objects 94.32
Appendix
Table 10: MindCube accuracy (%) under single-dimension corruption of the GT map, with Qwen3.5-27B. The uncorrupted reference accuracy is 98.25 . Each row changes only the specified field. Object facing is evaluated on the 364 and 129 questions with at least one and two annotated facings, whose uncorrupted accuracies are 97.53 and 98.45 .
Model
Method
Rotation
Among
Around
Overall
Qwen2.5-VL-7B-Instruct
Base (w/o map)
35.83 ± 0.76
33.39 ± 0.10
40.67 ± 0.83
35.59 ± 0.29
Base (w map)
35.50 ± 0.50
27.56 ± 0.35
32.93 ± 0.92
30.35 ± 0.29
Spatial Harness (w map)
34.33 ± 4.31
58.17 ± 0.44
70.27 ± 2.20
56.51 ± 1.10
Qwen2.5-VL-32B-Instruct
Base (w/o map)
38.83 ± 0.29
38.17 ± 0.29
46.00 ± 0.40
40.16 ± 0.22
Base (w map)
40.83 ± 2.47
40.11 ± 0.51
65.73 ± 0.61
46.35 ± 0.38
Spatial Harness (w map)
69.00 ± 1.00
88.39 ± 0.54
84.53 ± 1.15
83.78 ± 0.71
Appendix
Table 11: Run-to-run variation on MindCube in the GT-map setting. Entries are mean ± sample standard deviation of accuracy (%) over three runs.
Figure 20: The success case (item among_group603_q3_5_1 ). The top shows the question, and the bottom presents the four images, numbered as in the question. The reference answer is B.
Figure 21: View rendered by capture_image() in the first turn of the success case.
Figure 22: The failure case (item among_group738_q2_5_1 ). The top shows the question, and the bottom presents the four images, numbered as in the question. The reference answer is A.
Figure 23: Views rendered by capture_image() in the three tool-calling turns of the failure case, in order.
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Harbin Institute of Technology +2