SpatialMind: Adaptive Multi-Level Geometric Fusion for Spatial Reasoning in Multimodal Large Language Models
Authors: Shibin Zhao, Hao Tan, Fan Yang, Haixin Wang, Feilong Chen, Yidong Ma, Chaoyang Zhao, Ming Tang, Jinqiao Wang
Organizations: School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Xiaohongshu
Multimodal Large Language Models (MLLMs) demonstrate promising capabilities in 2D visual understanding, yet remain limited in 3D spatial reasoning. Recent studies explore integrating 3D reconstruction models into MLLMs. However, these approaches either use a single geometric granularity across all tasks or adopt a fixed multi-level fusion scheme, regardless of task-specific reasoning requirements. Through systematic experiments on multi-level geometric fusion, we find that spatial tasks at different cognitive levels (e.g., distance estimation vs. route planning) require geometric representations of different granularities, revealing that fixed fusion cannot accommodate diverse spatial reasoning requirements. To bridge this gap, we propose SpatialMind, a task-adaptive framework for dynamically integrating multi-level geometric representations. SpatialMind employs GeoFus, a multi-layer geometric fusion module built around a gating network jointly conditioned on textual instructions, visual semantics, and geometric representations. This network adaptively modulates the contributions of shallow, intermediate, and deep geometric features across the layers of the LLM decoder. Furthermore, we introduce a negative-entropy regularization loss to mitigate gate collapse. Gate visualizations indicate that GeoFus adapts geometric granularity to task requirements. Experiments demonstrate that SpatialMind achieves competitive performance across spatial benchmarks, attaining a state-of-the-art score of 72.3 on VSI-Bench. The source code is publicly available at https://github.com/zhaosb-star/SpatialMind.
Figures & tables
Figure 2: Visualization of Multi-Level Geometric Attention across 2D Views and 3D Point Clouds. Features at different levels capture spatial information at distinct scales, progressing from fine-grained details to global layouts.
Figure 3: Effects of Multi-Level Geometric Features. Shallow features contribute more to low-level tasks, while deep features are more important for high-level tasks, demonstrating hierarchical specialization and cross-level complementarity.
Figure 4: Overview of the SpatialMind architecture. Our framework adopts a decoupled fusion design and introduces GeoFus for task-adaptive integration of multi-level geometric features with visual tokens. By jointly conditioning on textual, visual, and geometric contexts, GeoFus dynamically adjusts the contributions of geometric representations at different LLM injection layers according to task requirements.
Methods
Rank
Avg.
Numerical Answer
Multiple-Choice Answer
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
Proprietary Models (API)
Doubao-Seed-2.0-Pro ( Seed 2026 )
3
49.3
55.1
25.3
69.8
29.1
60.6
43.2
43.8
67.6
GPT-5.6-Luna ( OpenAI 2026 )
2
52.2
49.6
26.2
71.2
53.2
56.6
47.2
47.4
66.2
Gemini-3.5-Flash ( Google DeepMind 2026 )
1
56.5
55.5
41.3
74.3
58.0
66.6
50.3
38.1
67.8
Open-source Models
Table 1: Evaluation on VSI-Bench. Bold and underlined values indicate the best and second-best results within each open-source spatial intelligence model scale group, respectively. † denotes models trained using the same training data.
Methods
Rank
Avg.
SparBench
VSI-Bench
SPBench
BLINK
MindCube
SQA3D
Proprietary Models (API)
GPT-5.6-Luna
3
52.4
47.7
52.2
54.2
65.6
56.4
38.1
Doubao-Seed-2.0-Pro
2
54.1
55.5
49.3
50.3
50.6
61.4
57.7
Gemini-3.5-Flash
1
54.3
57.2
56.5
61.8
33.9
71.6
45.0
Open-Source Models
Qwen2.5-VL-3B
39.1
29.5
27.4
40.5
48.9
41.8
46.6
Table 2: Evaluation across spatial intelligence benchmarks. Bold and underlined values indicate the best and second-best results within each open-source spatial intelligence model scale group, respectively. † denotes models trained using the same training data.
Gating Conditions
Cross-Layer Selection
Neg.-Entropy Reg.
Avg.
Numerical Answer
Multiple-Choice Answer
Text
Semantic + Spatial
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
×
×
×
×
69.6
70.7
60.6
76.7
73.1
66.3
88.9
46.9
73.3
✓
×
×
×
70.1
71.5
60.2
76.8
72.5
68.9
86.2
43.8
81.4
✓
✓
×
×
70.2
71.0
61.2
77.1
73.4
69.0
84.9
46.4
78.8
✓
✓
✓
×
70.9
69.8
65.2
76.5
76.9
70.3
91.5
44.8
72.3
✓
✓
✓
✓
71.5
71.8
65.0
77.1
77.7
70.7
91.8
45.9
72.0
Table 3: Ablation Study of Components on VSI-Bench. Checkmarks indicate enabled components. The complete GeoFus configuration achieves the best average score of 71.5% .
Figure 5: Visualization of Adaptive Gating Weights. Across the L0–L2 injection layers, low-level tasks favor shallow geometric features, whereas high-level tasks emphasize deep features, demonstrating task-adaptive selection of geometric representations.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Stage 1: Fixed-Fusion Training
Stage 2: Adaptive GeoFus Training
Base checkpoint
Qwen3.5-4B / Qwen3.5-9B
Stage-1 fixed-fusion checkpoint
Geometry encoder
VGGT-1B
VGGT-1B
Trainable modules
LLM and geometry-fusion modules
GeoFus modules only
Optimizer and precision
AdamW, BF16
AdamW, BF16
Learning rate
1×10−5
5×10−5
Epochs
1
1
Appendix
Table 4: Training configurations for the two-stage optimization strategy.
Methods
Rank
Overall
Among
Around
Rotation
Proprietary Models (API)
GPT-5.6-Luna
3
56.44
41.69
72.40
80.00
Doubao-Seed-2.0-Pro
2
61.35
47.29
67.20
95.50
Gemini-3.5-Flash
1
71.63
61.53
79.20
92.00
Open-Source Models
Qwen2.5-VL-3B-Instruct
41.83
42.20
48.40
32.50
Appendix
Table 5: Evaluation on MindCubeBench. Bold and underlined values indicate the best and second-best results within each specialized-model size group, respectively. † denotes models trained using the same training data.
Methods
Rank
Avg. EM
EM by Question Type
What
Is
How
Can
Which
Others
Open-Source Models
Qwen2.5-VL-3B-Instruct
46.43
40.28
53.53
43.44
53.25
47.29
48.59
Qwen2.5-VL-7B-Instruct
45.69
41.41
54.91
33.12
56.51
47.01
46.82
Qwen3.5-4B
48.05
40.80
60.43
43.66
54.14
45.87
49.82
Qwen3-VL-4B-Instruct
3
51.58
46.21
62.42
47.74
61.83
49.29
48.41
Appendix
Table 6: Evaluation on SQA3D. Bold and underlined values indicate the best and second-best results within each specialized-model size group, respectively. † denotes models trained using the same training data.
Figure 6: Visualization of Multi-Level Geometric Representations. We visualize similarity maps derived from shallow, intermediate, and deep geometric features across representative video frames and the reconstructed point cloud.
Methods
Rank
Avg.
Art Style
Counting
Forensic
Functional
IQ
Jigsaw
Multi-view
Localization
Rel. Depth
Rel. Reflect.
Semantic
Spatial Rel.
Visual Corr.
Visual Sim.
Proprietary Models (API)
GPT-5.6-Luna
1
65.60
57.26
71.67
71.21
44.62
20.00
74.67
45.11
86.89
86.29
65.67
63.31
77.62
91.28
61.48
Doubao-Seed-2.0-Pro
2
50.60
58.97
75.83
11.36
5.38
8.67
35.33
24.81
72.95
91.13
61.94
30.22
84.62
80.81
69.63
Gemini-3.5-Flash
3
33.88
81.20
82.50
44.70
0.00
1.33
50.67
48.12
13.11
0.00
0.00
0.00
92.31
9.30
62.96
Open-Source Models
Qwen2.5-VL-3B-Instruct
3
48.87
63.25
65.00
28.79
21.54
22.67
49.33
44.36
47.54
74.19
42.54
35.97
82.52
45.35
67.41
Appendix
Table 7: Evaluation on BLINK. Bold and underlined values indicate the best and second-best results within each specialized-model size group, respectively. † denotes models trained using the same training data.
Model
Abs. Distance
Rel. Distance
Object Size
Room Size
Rel. Direction
Avg.
Change (pp)
Full model
55.70
66.76
69.18
68.09
84.51
68.85
–
w/o 11–0
51.07
66.90
69.67
67.77
81.71
67.42
−1.43
w/o 17–1
53.50
65.77
67.36
67.36
83.92
67.52
−1.33
w/o 23–2
54.19
65.21
69.41
69.06
83.12
68.19
−0.66
Appendix
Table 8: Ablation results on low-level spatial tasks (%).
Model
Appearance Order
Object Counting
Route Planning
Avg.
Change (pp)
Full model
83.33
70.86
42.26
65.48
–
w/o 11–0
83.17
69.98
42.78
65.31
−0.17
w/o 17–1
81.55
69.43
40.72
63.90
−1.58
w/o 23–2
81.22
70.88
38.14
63.41
−2.07
Appendix
Table 9: Ablation results on high-level spatial tasks (%).
Methods
Rank
Avg.
Low
Depth-OC
Depth-OC-MV
Depth-OO
Depth-OO-MV
Dist-OC
Dist-OC-MV
Dist-OO
Dist-OO-MV
Medium
PosMatch
CamMotion
ViewChg
High
Dist-C-OO
Dist-C-OO-MV
ObjRel-OC-MV
ObjRel-OO
ObjRel-OO-MV
SphImag-OC
SphImag-OC-MV
SphImag-OO
SphImag-OO-MV
Proprietary Models (API)
GPT-5.6-Luna
3
47.65
46.51
60.22
48.91
27.98
26.32
58.65
59.32
51.10
39.60
41.17
74.55
22.75
26.21
50.82
71.47
61.90
46.50
66.76
41.83
45.16
42.44
40.73
40.62
Doubao-Seed-2.0-Pro
2
55.50
46.85
74.22
57.84
31.61
24.41
69.13
63.81
29.70
24.11
50.26
86.26
35.25
29.28
64.92
82.94
80.65
62.00
79.12
69.53
53.76
51.45
47.68
57.14
Gemini-3.5-Flash
1
57.23
55.98
75.25
63.19
34.22
32.85
74.66
62.38
63.28
42.02
32.10
45.55
25.25
25.49
66.72
87.65
76.19
69.25
77.20
73.41
62.90
54.94
53.31
45.66
Open-Source Models
Qwen2.5-VL-3B-Instruct
29.53
25.66
38.58
40.53
18.25
17.72
16.62
21.32
29.97
22.26
28.15
43.77
27.00
13.67
33.43
57.06
58.63
39.75
30.49
20.78
19.89
26.74
22.85
24.65
Appendix
Table 10: Evaluation on SPAR-Bench. Bold and underlined values indicate the best and second-best results within each specialized-model size group, respectively. † denotes models trained using the same training data.
Methods
Rank
SPBench-SI
SPBench-MV
Overall
Overall
Abs. Dist.
Direction
Rel. Dist.
Size
Overall
Abs. Dist.
Counting
Direction
Rel. Dist.
Size
Mean
Proprietary Models (API)
GPT-5.6-Luna
2
54.72
41.01
33.66
79.12
65.08
53.66
32.33
50.86
29.55
88.24
67.34
54.19
Doubao-Seed-2.0-Pro
3
48.06
2.55
31.70
90.11
67.88
52.49
19.67
55.57
25.00
94.12
68.10
50.28
Gemini-3.5-Flash
1
58.19
33.62
30.07
92.31
76.76
65.34
42.33
56.57
56.82
94.12
76.84
61.77
Open-Source Models
Appendix
Table 11: Evaluation on SPBench. Bold and underlined values indicate the best and second-best results within each specialized-model size group, respectively. † denotes models trained using the same training data.
Figure 7: VSI-Bench Task Taxonomy. The eight VSI-Bench subtasks are grouped into low-level perception and high-level reasoning according to their dominant spatial operations
Figure 8: Per-Task Adaptive Gating Distributions on VSI-Bench. Softmax-normalized gating weights assigned to shallow, intermediate, and deep geometric representations at the three LLM injection positions, L0 , L1 , and L2 . Each row reports the average gating distribution over 50 randomly sampled examples from the corresponding VSI-Bench subtask. The upper five rows correspond to low-level perception tasks, while the lower three rows correspond to high-level reasoning tasks.
Figure 9: Task-Dependent Geometric Response Visualization on VSI-Bench. The low-level absolute-distance task emphasizes fine-grained local geometry, whereas the high-level route-planning task produces broader responses over global scene structures.
Figure 10: Qualitative Results on Low-Level VSI-Bench Tasks. Representative comparisons on absolute-distance estimation, relative-direction reasoning, and room-size estimation. SpatialMind produces predictions consistent with or closer to the ground truth, whereas the baseline exhibits substantial metric and directional errors.
Figure 11: Qualitative Results on High-Level VSI-Bench Tasks. Representative comparisons on appearance-order reasoning, cross-view object counting, and route planning. SpatialMind correctly integrates information across views and performs global or multi-step spatial reasoning, while the baseline produces incorrect predictions.
Multimodal large language models (MLLMs) are increasingly being applied to spatial cognition tasks, where they are expected to understand and interact with complex environments. Most existing works improve spatial reasoning by introducing 3D priors or geometric supervision, which enhances performance but incurs substantial data preparation and alignment costs. In contrast, purely 2D approaches often struggle with multi-frame spatial reasoning due to their limited ability to capture cross-frame spatial relationships. To address these limitations, we propose EgoMind, a Chain-of-Thought framework that enables geometry-free spatial reasoning through Role-Play Caption, which jointly constructs a coherent linguistic scene graph across frames, and Progressive Spatial Analysis, which progressively reasons toward task-specific questions. With only 5K auto-generated SFT samples and 20K RL samples, EgoMind achieves competitive results on VSI-Bench, SPAR-Bench, SITE-Bench, and SPBench, demonstrating its effectiveness in strengthening the spatial reasoning capabilities of MLLMs and highlighting the potential of linguistic reasoning for spatial cognition. Code and data are released at https://github.com/Hyggge/EgoMind.
Zhenghao Chen, Huiqun Wang, Di Huang
State Key Laboratory of Complex and Critical Software Environment, Beihang University · School of Computer Science and Engineering, Beihang University
While Multimodal Large Language Models (MLLMs) have achieved remarkable success in 2D visual understanding, their ability to reason about 3D space remains limited. To address this gap, we introduce geometrically referenced 3D scene representations (GR3D). Given a set of input images, GR3D annotates objects in the images with unique IDs and encodes their 3D geometric attributes as textual references indexed by these IDs. This representation enables MLLMs to interpret 3D cues using their advanced language-based skills in mathematical reasoning, while concurrently analyzing 2D visual features in a tightly coupled way. We present a simple yet effective approach based on GR3D, which requires no additional training and is readily applicable to different MLLMs. Implemented in a zero-shot setting, our approach yields substantial improvements on challenging spatial reasoning benchmarks, boosting GPT-5 performance by 9% on VSI-Bench and 12% on MindCube. Qualitative studies further demonstrate that GR3D empowers MLLMs to perform complex spatial reasoning with highly sparse input views.
Multimodal large language models (MLLMs) have exhibited remarkable performance in various visual tasks, yet still struggle with spatial reasoning. Recent efforts mitigate this by injecting geometric features from 3D foundation models, but rely on static single-layer extractions. We identify that such an approach induces a task misalignment bias: the geometric features naturally evolve towards 3D pretraining objectives, which may contradict the heterogeneous spatial demands of MLLMs, rendering any single layer fundamentally insufficient. To resolve this, we propose GeoAlign, a novel framework that dynamically aggregates multi-layer geometric features to realign with the actual demands. GeoAlign constructs a hierarchical geometric feature bank and leverages the MLLM's original visual tokens as content-aware queries to perform layer-wise sparse routing, adaptively fetching the suitable geometric features for each patch. Extensive experiments on VSI-Bench, ScanQA, and SQA3D demonstrate that our compact 4B model effectively achieves state-of-the-art performance, even outperforming larger existing MLLMs.
Zhaochen Liu, Limeng Qiao, Guanglu Wan +1
National Engineering Research Center of Visual Technology, National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · Meituan Inc. · National Biomedical Imaging Center, Peking University