Large Vision-Language Models (LVLMs) have achieved strong performance on diverse visual tasks, yet their ability to reconstruct and reason about the 3D structure of the scene depicted in 2D images -- referred to as spatial intelligence -- remains limited. Existing approaches attempt to address this gap by using real-scene spatial question answering datasets that require dense geometric annotations. However, constructing such labels is costly, time-consuming, and often noisy due to reliance on external perception modules. In this work, we propose a novel paradigm inspired by human cognitive development: learning foundational spatial skills through structured block-manipulation tasks. We introduce SpatialBlock-15k, a synthetic dataset of 15,000 block-stacking problems covering 3D-to-2D projection, viewpoint transformation, and structural combination. The dataset further incorporates controlled color modulation as visual cues to encourage anchor-based reasoning in visually complex conditions. Experiments demonstrate that LVLMs trained on our dataset through either direct answering or reasoning-based prediction significantly outperform baselines and generalize to real-world spatial tasks, despite the dataset's synthetic and compact nature. Code and data are available at https://github.com/rsoohyun/SpatialBlock.
Figures & tables
Figure 1: Performance Gap Between Humans and LVLMs in Block-Stacking Problem . The top row shows a block-stacking task that requires predicting a 2D projection from a given 3D structure. While humans achieve near-perfect accuracy, state-of-the-art LVLMs struggle even with basic spatial reasoning problems. Leading proprietary models exhibit flawed spatial logic, often producing incorrect interpretations of the structure.
Figure 2: Spatial Abilities Underlying Block-Stacking Reasoning . Solving block-stacking tasks involves (a) inferring latent 3D composition, (b) reasoning under transformation, and (c) integrating multiple components into a coherent structure.
Figure 3: SpatialBlock-15k . (a) Three block-stacking tasks: 3D-to-2D projection (Q1), viewpoint transformation (Q2), and structural combination (Q3). (b) Visual cue-based extension with controlled color variations, where colors encode depth ordering (Q1), preserved structural roles across transformations (Q2), and correspondence cues for integration (Q3).
Model
# Data
In-domain
Out-of-domain
SB-Bench
MindCube
MMSI-Bench
SPBench
MMMU
Overall
Proprietary Models
GPT-5 ( Singh et al., 2025 )
-
43.8
56.7
42.8
43.0
77.6
55.0
Gemini-2.5-flash ( Comanici et al., 2025 )
-
36.4
51.0
32.2
44.3
58.8
46.6
Claude-Sonnet-4.5 ( Anthropic, 2025 )
-
46.3
34.0
32.7
43.9
69.2
43.9
Open-Source Models
Table 1: Main Results . For each dataset, bold and underlined indicate the best and second-best performance, respectively. * represents in-domain results where the model was trained on the benchmark. We re-evaluated all previous works for fair comparison.
Figure 4: Synthetic-Real Dataset . Widely used spatial question types, (a) relative direction and (b) relative distance, are adapted into a block-based environment while preserving their semantic structure.
In-domain
Out-of-domain
Train Dataset
Relative Direction
Relative Distance
MindCube
MMSI-Bench
MMMU
Baseline
31.4
74.1
39.1
26.1
45.6
Synthetic-Real
43.4
76.9
32.8
25.5
43.6
SpatialBlock
31.4
67.4
49.1
28.1
46.4
Table 3: Comparison of Training on SpatialBlock and Synthetic-Real . While Synthetic-Real improves in-domain performance, training on SpatialBlock leads to better generalization on out-of-domain benchmark datasets. All experiments are conducted on Qwen2.5-VL-3B ( Bai et al., 2025b ) , and in-domain performance is evaluated on SPBench.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Instructions for Reinforcement Learning . The model is instructed to provide a step-by-step thinking process, and then the final answer between answer tags.
Figure 6: Instructions for Reasoning Quality Evaluation . We evaluate the quality of model-generated reasoning by measuring its alignment with each step of the corresponding ground-truth rationale.
Model
Absolute distance
Object size
Object counting
Average
Qwen2.5-VL-3B-Instruct
30.9
17.6
35.1
27.9
SpatialBlock-3B-direct
23.2
19.9
72.1
38.4
SpatialBlock-3B-reason
21.4
38.3
43.3
34.3
Appendix
Table 4: Results on Numerical Tasks . Results on tasks from SPBench.
Figure 7: Performance Across Viewing Angles . (a) The same block structures are rendered from different viewing angles. (b) Performance of models across viewing angles; the bars correspond to results under 45∘ , 30∘ , and 60∘ views, respectively.
Figure 8: Generated Cold-Start Data Sample . The reasoning trace extracted from Qwen2.5-VL-7B includes incorrect information about the given 3D structure, even though the model answers correctly.
Figure 9: Qualitative Comparison of Reasoning Traces . (a) Baseline only describes a single view. (b) Training without visual cues attempts multi-image reasoning but fails to identify the shared object. (c) Our model uses the “yellow container” as an anchor to integrate views and reach the correct answer.
Figure 10: Additional Qualitative Result on Reasoning Traces . The figure shows additional qualitative results on reasoning traces of the baseline model, our model trained without visual cues, and our final model, SpatialBlock-3B-reason.
Achieving human-like spatial intelligence for vision-language models (VLMs) requires inferring 3D structures from 2D observations, recognizing object properties and relations in 3D space, and performing high-level spatial reasoning. In this paper, we propose a principled hierarchical framework that decomposes the learning of 3D spatial understanding in VLMs into four progressively complex levels, from geometric perception to abstract spatial reasoning. Guided by this framework, we construct an automated pipeline that processes approximately 5M images with over 45M objects to generate 3D spatial VQA pairs across diverse tasks and scenes for VLM supervised fine-tuning. We also develop an RGB-D VLM incorporating metric-scale point maps as auxiliary inputs to further enhance spatial understanding. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on multiple spatial understanding and reasoning benchmarks, surpassing specialized spatial models and large proprietary systems such as Gemini-2.5-pro and GPT-5. Moreover, our analysis reveals clear dependencies among hierarchical task levels, offering new insights into how multi-level task design facilitates the emergence of 3D spatial intelligence.
Huizhi Liang, Yichao Shen, Yu Deng +5
Tsinghua University · Microsoft Research Asia · Xi’an Jiaotong University +1
Recent advancements in Large Vision-Language Models (VLMs) have demonstrated exceptional semantic understanding, yet these models consistently struggle with spatial reasoning, often failing at fundamental geometric tasks such as depth ordering and precise coordinate grounding. Recent efforts introduce spatial supervision from scene-centric datasets (e.g., multi-view scans or indoor video), but are constrained by the limited number of underlying scenes. As a result, the scale and diversity of such data remain significantly smaller than those of web-scale 2D image collections. To address this limitation, we propose SpatialForge, a scalable data synthesis pipeline that transforms in-the-wild 2D images into spatial reasoning supervision. Our approach decomposes spatial reasoning into perception and relation, and constructs structured supervision signals covering depth, layout, and viewpoint-dependent reasoning, with automatic verification to ensure data quality. Based on this pipeline, we build SpatialForge-10M, a large-scale dataset containing 10 million spatial QA pairs. Extensive experiments across multiple spatial reasoning benchmarks demonstrate that training on SpatialForge-10M significantly improves the spatial reasoning ability of standard VLMs, highlighting the effectiveness of scaling 2D data for 3D-aware spatial reasoning.
Vision-Language Models (VLMs) perform well on commonsense reasoning tasks but struggle with visual spatial reasoning. Most existing solutions introduce extra 3D prior inputs or external spatial encoders, which increase complexity and degrade the underlying VLMs' general-purpose capabilities after spatial fine-tuning. To this end, we propose a parameter-efficient Spatio-vision Language Models (SpatioLM), that enhances spatial intelligence without extra 3D prior inputs or third-party spatial encoders. Concretely, we design a plug-and-play and non-invasive spatio-vision module that elicits the spatial knowledge inherent in VLMs. Furthermore, we innovatively leverage pseudo depth and camera information as supervision to guide the model in learning physically coherent representations. Extensive experiments show that SpatioLM achieves significant improvements in diverse tasks, including spatial perception and understanding while effectively limiting the degradation of general capabilities. Notably, the model achieves an impressive score of 71.6 on the VSI-Bench (the first model to surpass 70). In addition, it attains competitive performance when transferred to embodied manipulation tasks. Code is available at \faGithub~spatio-lm.
Jing Wu, Jianhua Wu, Jiayi Guan +5
Xiaomi EV, Beijing, China · College of Automotive and Energy Engineering, Tongji University, Shanghai, China · Independent Researcher