Multi-modal Large Language Models (MLLMs) have demonstrated strong capabilities in general-purpose perception and reasoning, but they still struggle with tasks that require spatial understanding of the 3D world. To address this, we introduce pySpatial, a visual programming framework that equips MLLMs with the ability to interface with spatial tools via Python code generation. Given an image sequence and a natural-language query, the model composes function calls to spatial tools including 3D reconstruction, camera-pose recovery, novel-view rendering, etc. These operations convert raw 2D inputs into an explorable 3D scene, enabling MLLMs to reason explicitly over structured spatial representations. Notably, pySpatial requires no gradient-based fine-tuning and operates in a fully zero-shot setting. Experimental evaluations on the challenging MindCube and Omni3D-Bench benchmarks demonstrate that our framework pySpatial consistently surpasses strong MLLM baselines; for instance, it outperforms GPT-4.1-mini by 12.94% on MindCube. Furthermore, we conduct real-world indoor navigation experiments where the robot can successfully traverse complex environments using route plans generated by pySpatial, highlighting the practical effectiveness of our approach.
Figures & tables
Figure 1: Comparing our pySpatial with spatial mental models for multi-view spatial reasoning tasks . Unlike spatial mental models ( Yin et al., 2025 ) , which rely on the implicit imagination of MLLMs to construct a 2D cognitive map, we introduce pySpatial , a visual programming framework that flexibly composes spatial tools ( e.g. , 3D reconstruction, camera movements, and novel view synthesis) to enable MLLMs to explicitly reason in 3D space for diverse spatial reasoning tasks.
Method
Reference
Overall
Rotation
Among
Around
Baseline
Random (chance)
-
32.35
36.36
32.29
30.66
Random (frequency)
-
33.02
38.30
32.66
35.79
Open-Weight Multi-Image Models
LLaVA-OneVision-7B
Li et al. (2025)
47.43
36.45
48.42
44.09
LLaVA-Video-Qwen-7B
Zhang et al. (2025c)
41.96
35.71
43.55
30.12
Table 1: Performance comparison on the full MindCube ( Yin et al., 2025 ) dataset . The best results are shown in bold , and the second-best are underlined . Note that we implement pySpatial using GPT-4.1-mini as the code agent for this dataset due to budget constraints.
Method
Reference
Overall
Rotation
Among
Around
Baseline Models
Qwen2.5-VL-3B-Instruct
Bai et al. (2025)
37.81
34.00
36.00
45.20
GPT-4o
OpenAI (2024)
42.29
35.00
43.00
46.40
Spatial Mental Models
Chain-of-Thought
40.48
32.00
36.00
58.00
View Interpolation
Yin et al. (2025)
37.81
35.50
36.50
42.80
Table 2: Performance comparison on the MindCube -1k ( Yin et al., 2025 ) dataset . The evaluated mental models ( Yin et al., 2025 ) are based on Qwen2.5-VL-3B-Instruct ( Bai et al., 2025 ) . VADAR w/ Recon. denotes that we implement VADAR with our 3D reconstruction module. The best results are highlighted in bold , and the second-best are underlined .
Method
Reference
numeric (ct)
numeric (other)
y/n
multi-choice
Total
Baseline Models
GPT-4o
OpenAI (2024)
28.1
35.5
66.7
57.2
42.9
Claude3.5-Sonnet
Anthropic (2024)
22.4
20.6
62.2
50.6
32.2
Llama-3.2
Meta (2024)
24.3
19.3
47.5
27.4
25.6
Gemini1.5-Pro
Google (2024)
25.2
28.1
46.2
37.6
32.0
SpaceMantis
Chen et al. (2024a)
20.0
21.7
50.6
48.2
30.3
Table 3: Performance comparison on Omni3D-Bench . Following VADAR ( Marsili et al., 2025 ) , We report mean relative accuracy ( Yang et al., 2025 ) for the numeric (other) and accuracy for the other category. The best results are shown in bold , and the second-best are underlined .
Figure 2: Qualitative results on four representative examples from MindCube . We show that pySpatial enables MLLMs to explicitly reason within the reconstructed explorable 3D scene, effectively addressing diverse spatial reasoning tasks through interpretable and executable 3D visual programs. Figure A1 further illustrates that pySpatial is capable of composing executable 3D visual programs with control flow constructs ( e.g. , for -loops), allowing it to robustly address a wide range of spatial reasoning tasks. Best viewed when zoomed in.
Figure 3: Qualitative results on real-world robot navigation. We deploy pySpatial on a Unitree-Go1 robot to navigate toward a target object (mushroom toy) using limited views as input. Compared to the GPT-4.1 baseline, which fails due to an incorrect initial turn and produces a collision-prone trajectory, pySpatial generates a geometrically consistent plan that successfully reaches the goal.
Method
Overall
Rotation
Among
Around
GPT-4o
42.29
35.00
43.00
46.40
+ pySpatial
62.67
41.00
65.00
66.33
GPT-4.1-mini
43.34
36.00
45.00
44.80
+ pySpatial
58.19
37.50
62.00
65.60
GPT-4.1
44.67
35.50
45.33
50.40
+ pySpatial
62.35
41.83
64.89
72.67
Table 4: Ablation study on the code agent . We report the accuracy on the MindCube -1k dataset.
Figure 4: Failure case study. We manually examine the error sources in about 100 samples from MindCube .
Method
Overall
Rotation
Among
Around
GPT-4o
42.29
35.00
43.00
46.20
pySpatial w/ GPT-4o
62.67
41.00
66.33
71.20
pySpatial w/ Qwen3-Coder
62.10
40.00
64.50
74.00
pySpatial w/ DeepSeek-v3
61.05
40.50
65.33
64.80
Table A2: Performance comparison of our pySpatial framework with different code agents.
Method
Overall
Rotation
Among
Around
GPT-4o
42.29
35.00
43.00
46.20
pySpatial w/ VGGT
62.67
41.00
66.33
71.20
pySpatial w/ Pi3
63.33
43.50
65.66
72.33
pySpatial w/ CUT3R
61.05
40.50
64.33
69.60
Table A3: Performance comparison of pySpatial using different 3D reconstruction backbones.
Method
Overall
Rotation
Among
Around
GPT-4o
42.29
35.00
43.00
46.20
pySpatial w/ 0 examples
53.62
32.50
55.16
66.80
pySpatial w/ 2 examples
62.67
41.00
66.33
71.20
pySpatial w/ 4 examples
63.14
46.00
64.50
73.60
Table A4: Comparison of pySpatial with different numbers of in-context learning examples.
Figure A1: More qualitative examples from MindCube . We show that pySpatial enables MLLMs to explicitly reason within a reconstructed, explorable 3D scene, allowing the model not only to interpret spatial structure but also to compose executable 3D visual programs with control flow, such as for-loops to robustly solve diverse spatial reasoning tasks.
Figure A2: An additional qualitative real-world example in a challenging dynamic scene.
Although Multimodal Large Language Models have achieved remarkable progress, they still struggle with complex 3D spatial reasoning due to the reliance on 2D visual priors. Existing approaches typically mitigate this limitation either through computationally expensive post-training procedures on limited 3D datasets or through rigid tool-calling mechanisms that lack explicit geometric understanding and viewpoint flexibility. To address these challenges, we propose a training-free framework that introduces a Visual Chain-of-Thought mechanism grounded in explicit 3D reconstruction. The proposed pipeline first reconstructs a high-fidelity 3D mesh from a single image using MLLM-guided keyword extraction and mask generation at multiple granularities. Subsequently, the framework leverages an external knowledge base to iteratively compute optimal camera extrinsic parameters and synthesize novel views, thereby emulating human perspective-taking. Extensive experiments demonstrate that the proposed approach significantly enhances spatial comprehension. Specifically, the framework outperforms specialized spatial models and general-purpose MLLMs, including GPT-5.2 and Gemini-2.5-Flash, on major benchmarks such as 3DSRBench and Rel3D.
Jiahua Chen, Qihong Tang, Weinong Wang +1
Tsinghua University · Nanjing University · Tencent
Precise spatial understanding from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs), as their visual representations are predominantly semantic and lack explicit geometric grounding. While existing approaches augment visual tokens with geometric cues from visual geometry models, their MLLM is still required to implicitly infer the underlying 3D structure of the scene from these augmented tokens, limiting its spatial reasoning capability. To address this issue, we introduce Cog3DMap, a framework that recurrently constructs an explicit 3D memory from multi-view images, where each token is grounded in 3D space and possesses both semantic and geometric information. By feeding these tokens into the MLLM, our framework enables direct reasoning over a spatially structured 3D map, achieving state-of-the-art performance on various spatial reasoning benchmarks. Code will be made publicly available.
While Multimodal Large Language Models (MLLMs) have achieved remarkable success in 2D visual understanding, their ability to reason about 3D space remains limited. To address this gap, we introduce geometrically referenced 3D scene representations (GR3D). Given a set of input images, GR3D annotates objects in the images with unique IDs and encodes their 3D geometric attributes as textual references indexed by these IDs. This representation enables MLLMs to interpret 3D cues using their advanced language-based skills in mathematical reasoning, while concurrently analyzing 2D visual features in a tightly coupled way. We present a simple yet effective approach based on GR3D, which requires no additional training and is readily applicable to different MLLMs. Implemented in a zero-shot setting, our approach yields substantial improvements on challenging spatial reasoning benchmarks, boosting GPT-5 performance by 9% on VSI-Bench and 12% on MindCube. Qualitative studies further demonstrate that GR3D empowers MLLMs to perform complex spatial reasoning with highly sparse input views.