Multi-modal Large Language Models (MLLMs) have demonstrated strong capabilities in general-purpose perception and reasoning, but they still struggle with tasks that require spatial understanding of the 3D world. To address this, we introduce pySpatial, a visual programming framework that equips MLLMs with the ability to interface with spatial tools via Python code generation. Given an image sequence and a natural-language query, the model composes function calls to spatial tools including 3D reconstruction, camera-pose recovery, novel-view rendering, etc. These operations convert raw 2D inputs into an explorable 3D scene, enabling MLLMs to reason explicitly over structured spatial representations. Notably, pySpatial requires no gradient-based fine-tuning and operates in a fully zero-shot setting. Experimental evaluations on the challenging MindCube and Omni3D-Bench benchmarks demonstrate that our framework pySpatial consistently surpasses strong MLLM baselines; for instance, it outperforms GPT-4.1-mini by 12.94% on MindCube. Furthermore, we conduct real-world indoor navigation experiments where the robot can successfully traverse complex environments using route plans generated by pySpatial, highlighting the practical effectiveness of our approach.
Figures & tables
Figure 1: Comparing our pySpatial with spatial mental models for multi-view spatial reasoning tasks . Unlike spatial mental models ( Yin et al., 2025 ) , which rely on the implicit imagination of MLLMs to construct a 2D cognitive map, we introduce pySpatial , a visual programming framework that flexibly composes spatial tools ( e.g. , 3D reconstruction, camera movements, and novel view synthesis) to enable MLLMs to explicitly reason in 3D space for diverse spatial reasoning tasks.
Method
Reference
Overall
Rotation
Among
Around
Baseline
Random (chance)
-
32.35
36.36
32.29
30.66
Random (frequency)
-
33.02
38.30
32.66
35.79
Open-Weight Multi-Image Models
LLaVA-OneVision-7B
Li et al. (2025)
47.43
36.45
48.42
44.09
LLaVA-Video-Qwen-7B
Zhang et al. (2025c)
41.96
35.71
43.55
30.12
Table 1: Performance comparison on the full MindCube ( Yin et al., 2025 ) dataset . The best results are shown in bold , and the second-best are underlined . Note that we implement pySpatial using GPT-4.1-mini as the code agent for this dataset due to budget constraints.
Method
Reference
Overall
Rotation
Among
Around
Baseline Models
Qwen2.5-VL-3B-Instruct
Bai et al. (2025)
37.81
34.00
36.00
45.20
GPT-4o
OpenAI (2024)
42.29
35.00
43.00
46.40
Spatial Mental Models
Chain-of-Thought
40.48
32.00
36.00
58.00
View Interpolation
Yin et al. (2025)
37.81
35.50
36.50
42.80
Table 2: Performance comparison on the MindCube -1k ( Yin et al., 2025 ) dataset . The evaluated mental models ( Yin et al., 2025 ) are based on Qwen2.5-VL-3B-Instruct ( Bai et al., 2025 ) . VADAR w/ Recon. denotes that we implement VADAR with our 3D reconstruction module. The best results are highlighted in bold , and the second-best are underlined .
Method
Reference
numeric (ct)
numeric (other)
y/n
multi-choice
Total
Baseline Models
GPT-4o
OpenAI (2024)
28.1
35.5
66.7
57.2
42.9
Claude3.5-Sonnet
Anthropic (2024)
22.4
20.6
62.2
50.6
32.2
Llama-3.2
Meta (2024)
24.3
19.3
47.5
27.4
25.6
Gemini1.5-Pro
Google (2024)
25.2
28.1
46.2
37.6
32.0
SpaceMantis
Chen et al. (2024a)
20.0
21.7
50.6
48.2
30.3
Table 3: Performance comparison on Omni3D-Bench . Following VADAR ( Marsili et al., 2025 ) , We report mean relative accuracy ( Yang et al., 2025 ) for the numeric (other) and accuracy for the other category. The best results are shown in bold , and the second-best are underlined .
Figure 2: Qualitative results on four representative examples from MindCube . We show that pySpatial enables MLLMs to explicitly reason within the reconstructed explorable 3D scene, effectively addressing diverse spatial reasoning tasks through interpretable and executable 3D visual programs. Figure A1 further illustrates that pySpatial is capable of composing executable 3D visual programs with control flow constructs ( e.g. , for -loops), allowing it to robustly address a wide range of spatial reasoning tasks. Best viewed when zoomed in.
Figure 3: Qualitative results on real-world robot navigation. We deploy pySpatial on a Unitree-Go1 robot to navigate toward a target object (mushroom toy) using limited views as input. Compared to the GPT-4.1 baseline, which fails due to an incorrect initial turn and produces a collision-prone trajectory, pySpatial generates a geometrically consistent plan that successfully reaches the goal.
Method
Overall
Rotation
Among
Around
GPT-4o
42.29
35.00
43.00
46.40
+ pySpatial
62.67
41.00
65.00
66.33
GPT-4.1-mini
43.34
36.00
45.00
44.80
+ pySpatial
58.19
37.50
62.00
65.60
GPT-4.1
44.67
35.50
45.33
50.40
+ pySpatial
62.35
41.83
64.89
72.67
Table 4: Ablation study on the code agent . We report the accuracy on the MindCube -1k dataset.
Figure 4: Failure case study. We manually examine the error sources in about 100 samples from MindCube .
Method
Overall
Rotation
Among
Around
GPT-4o
42.29
35.00
43.00
46.20
pySpatial w/ GPT-4o
62.67
41.00
66.33
71.20
pySpatial w/ Qwen3-Coder
62.10
40.00
64.50
74.00
pySpatial w/ DeepSeek-v3
61.05
40.50
65.33
64.80
Table A2: Performance comparison of our pySpatial framework with different code agents.
Method
Overall
Rotation
Among
Around
GPT-4o
42.29
35.00
43.00
46.20
pySpatial w/ VGGT
62.67
41.00
66.33
71.20
pySpatial w/ Pi3
63.33
43.50
65.66
72.33
pySpatial w/ CUT3R
61.05
40.50
64.33
69.60
Table A3: Performance comparison of pySpatial using different 3D reconstruction backbones.
Method
Overall
Rotation
Among
Around
GPT-4o
42.29
35.00
43.00
46.20
pySpatial w/ 0 examples
53.62
32.50
55.16
66.80
pySpatial w/ 2 examples
62.67
41.00
66.33
71.20
pySpatial w/ 4 examples
63.14
46.00
64.50
73.60
Table A4: Comparison of pySpatial with different numbers of in-context learning examples.
Figure A1: More qualitative examples from MindCube . We show that pySpatial enables MLLMs to explicitly reason within a reconstructed, explorable 3D scene, allowing the model not only to interpret spatial structure but also to compose executable 3D visual programs with control flow, such as for-loops to robustly solve diverse spatial reasoning tasks.
Figure A2: An additional qualitative real-world example in a challenging dynamic scene.