Organizations: Shanghai AI Lab · Northwestern Polytechnical University · Nanyang Technological University · Beihang University · Peking University · Sichuan University · Tsinghua University · The Chinese University of Hong Kong · Fudan University · Hong Kong University of Science and Technology · Shanghai Jiao Tong University
The pursuit of spatial intelligence fundamentally relies on access to large-scale, fine-grained 3D data. However, existing approaches predominantly construct spatial understanding benchmarks by generating question-answer (QA) pairs from a limited number of manually annotated datasets, rather than systematically annotating new large-scale 3D scenes from raw web data. As a result, their scalability is severely constrained, and model performance is further hindered by domain gaps inherent in these narrowly curated datasets. In this work, we propose Holi-Spatial, the first fully automated, large-scale, spatially-aware multimodal dataset, constructed from raw video inputs without human intervention, using the proposed data curation pipeline. Holi-Spatial supports multi-level spatial supervision, ranging from geometrically accurate 3D Gaussian Splatting (3DGS) reconstructions with rendered depth maps to object-level and relational semantic annotations, together with corresponding spatial Question-Answer (QA) pairs. Following a principled and systematic pipeline, we further construct Holi-Spatial-4M, the first large-scale, high-quality 3D semantic dataset, containing 12K optimized 3DGS scenes, 1.3M 2D masks, 320K 3D bounding boxes, 320K instance captions, 1.2M 3D grounding instances, and 1.2M spatial QA pairs spanning diverse geometric, relational, and semantic reasoning tasks. Holi-Spatial demonstrates exceptional performance in data curation quality, significantly outperforming existing feed-forward and per-scene optimized methods on datasets such as ScanNet, ScanNet++, and DL3DV. Furthermore, fine-tuning Vision-Language Models (VLMs) on spatial reasoning tasks using this dataset has also led to substantial improvements in model performance.
Figures & tables
Figure 2: Comparison of our refined annotations with the official annotations on the ScanNet dataset [ 28 ] . Our method achieves more accurate and sharper segmentation masks, as well as improved category recognition.
Method
Inputs
Outputs
Images
Point Clouds
Depth
2D Seg.
3D Det.
Grounding
Spatial QA
2D-VLM Methods
SAM3
✓
✗
✗
✓
✗
✗
✗
SA2VA
✓
✗
✗
✓
✗
✓
✓
3D-VLM Methods
SpatialLM
✗
✓
✗
✗
✓
✓
✓
Table 1: Overview of Pipeline Capabilities. We compare input modalities and output tasks across different paradigms, ranging from 2D VLMs and 3D-VLMs to 3D-GS-based understanding methods. Holi-Spatial serves as a unified framework supporting diverse spatial tasks without relying on 3D priors.
Figure 3: Overview of the Holi-Spatial data curation pipeline. The framework operates in three progressive stages: (1) Geometric Optimization distills high-fidelity 3D structure from video streams using 3DGS; (2) Image-level Perception lifts 2D VLM and SAM3 predictions into initial 3D proposals; and (3) Scene-level Refinement employs a coarse-to-fine strategy to merge, verify, and caption instances, yielding dense, high-quality spatial annotations. Finally, leveraging the generated Holi-Spatial-4M dataset, we directly fine-tune the Qwen-VL family for downstream tasks ( e.g. , 3D grounding and spatial reasoning).
Figure 4: Pipeline of 2D-to-3D OBB Generation. We transform 2D object masks into initial 3D OBBs via depth projection, utilizing a four-step strategy to mitigate the impact of depth floaters. (1) We obtain an initial object depth map by combining 3DGS rendering with SAM3 instance segmentation. (2) To mitigate 2D boundary errors from SAM3, we erode the object mask near its contour and keep only the reliable interior region. (3) To remove 3D outliers caused by depth discontinuities, we use a multi-view-consistent mesh depth as guidance and filter inconsistent pixels in the 3DGS depth. (4) Finally, we estimate the initial 3D OBB from the refined point cloud, while preserving the associated 2D mask, confidence score, and source image index.
Figure 5: Floor-aligned OBB post-processing pipeline. Starting from the input instance OBBs produced after 2D-to-3D lifting and initial OBB estimation, we (1) detect a floor (or fallback planar structure) to infer a global up-axis, (2) re-align each instance OBB under a yaw-lock constraint with optional PCA fallback and update extents, and (3) apply a validation check to output gravity-/floor-consistent OBBs for downstream scene-level refinement.
Figure 6: Comprehensive Statistics of Holi-Spatial-4M. (1) Object Diversity: Word cloud showing the long-tailed distribution of open-vocabulary categories. (2) Dataset Composition: The inner ring displays source scenes (ScanNet [ 28 ] , ScanNet++ [ 27 ] , DL3DV [ 31 ] ), while the outer ring details the breakdown of over 4M generated spatial annotations. (3) Spatial QA Taxonomy: Distribution of 1.25M spatial QA pairs, categorized into Camera-centric tasks ( e.g. , rotation, movement) and Object-centric tasks ( e.g. , distance, direction).
Method
ScanNet
ScanNet++
DL3DV
Depth ( ↑ )
2D Seg. ( ↑ )
3D Det. ( ↑ )
Depth ( ↑ )
2D Seg. ( ↑ )
3D Det. ( ↑ )
Depth ( ↑ )
2D Seg. ( ↑ )
3D Det. ( ↑ )
F1
IoU
AP 25
AP 50
F1
IoU
AP 25
AP 50
F1
IoU
AP 25
AP 50
2D-VLM Methods
SAM3
–
0.63
–
–
–
0.50
–
–
–
0.66
–
–
SA2VA
–
0.64
–
–
–
0.25
–
–
–
0.44
–
–
3D-VLM Methods
Table 2: Quantitative results of 3D spatial understanding on ScanNet, ScanNet++ and DL3DV datasets. We group methods by their input modality and task definition. 3D Det : 3D Object Detection (AP@25, AP@50); 2D Seg : 2D Object Segmentation (IoU); Depth : Depth Estimation (F1-score). Bold indicates best results. ↑ : higher is better. ‘—’ denotes that the corresponding metric is not available.
Figure 7: Qualitative comparison of multi-view depth on ScanNet++. We visualize depth maps from GT and baselines (LangSplat, M3-Spatial, Depth-Anything-V3) versus Holi-Spatial. Right: point clouds obtained by multi-view back-projection, where Holi-Spatial produces cleaner geometry with substantially fewer ghosting artifacts and floaters.
Figure 8: Qualitative comparison of open-vocabulary 2D instance segmentation on the same scenes. Holi-Spatial yields sharper boundaries, more complete masks under occlusion, and more accurate fine-grained categories.
Figure 9: Qualitative comparison of 3D object detection on ScanNet++. We visualize predicted 3D bounding boxes from SceneScript, LLaVA-3D, SpatialLM, and Holi-Spatial. Holi-Spatial produces tighter boxes and more correct categories.
Model
MMSI-Bench
MindCube
VST-SFT-3B [ 26 ]
30.2
35.9
Cambrian-S-3B [ 3 ]
25.2
32.5
VST-SFT-7B [ 26 ]
32.0
39.7
Cambrian-S-7B [ 3 ]
25.8
39.6
SpaceR-SFT-7B [ 55 ]
27.4
37.9
Intern3-VL-8B [ 57 ]
28.0
41.5
Table 3: Quantitative results of Spatial Understanding QA tasks. Here we caompare different models [ 55 , 57 , 25 ] on MMSI-bench [ 18 ] and MindCube [ 19 ] benchmarks. Bold indicates the best performance.
Method
AP 15
AP 25
AP 50
VST-7B-SFT [ 26 ]
17.29
14.50
11.20
Qwen3-VL-8B [ 13 ]
19.82
16.80
13.50
Qwen3-VL-8B + Ours
35.52
31.94
27.98
Table 4: Quantitative results of 3D Grounding on ScanNet++ [ 27 ] dataset.
Figure 10: Stage-wise visualization of scene-level refinement. Detailed discussion is include in Section 5.3
Figure 11: Given the same input images and queried objects, our predicted boxes align better with ground truth, indicating improved spatial localization.
ID
DA3 Depth
3DGS Training
Conf. Filter
Agent Recall
P25
R25
Step 1: Geometric Optimization
1
✓
✗
✓
✓
0.13
0.31
2
✓
✓
✓
✓
0.81
0.89
Step 3: Scene-Level Refinement
3
✓
✓
✗
✗
0.35
0.74
4
✓
✓
✓
✗
0.67
0.69
Table 5: Ablation study on depth refinement, confidence filtering, and agent recall. ID. 1–2 (Step 1) compare depth sources, while ID. 3–5 (Step 3) evaluate post-processing modules. P25 and R25 denote precision and recall under IoU 25%.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Ablation study on 3D multi-view merging and depth refinement. (a) 3D geometric clustering corrects the fragmentation caused by occlusions in SAM3 image-level instance predictions, merging separated segments into unified object instances. (b) Using GS-refined depth constraints enables more accurate multi-view lifting, which avoids false merges and preserves the integrity of spatial object instances.
Figure 13: Prompt template used for Agent verification.
Figure 14: Prompt template used for object grounding in 2D image.
Figure 15: Additional 3D detection visualizations. We show Holi-Spatial predictions across diverse indoor scenes, demonstrating multi-view consistent localization and robust detection.
Figure 16: Examples of 10 types of curated spatial QA pairs in Holi-Spatial.
Figure 17: Training with our curated QA data improves the three types of spatial reasoning tasks in MindCube [ 19 ] (upper row) as well as in MMSI-Bench [ 18 ] (lower row).
Spatial intelligence requires scalable 3D environments that are both photorealistic and semantically grounded, yet existing indoor datasets are dominated by room-scale captures and costly LiDAR acquisition. Online real-estate platforms offer abundant panoramas of diverse multi-room homes, but sparse viewpoints and texture-poor regions make reconstruction severely under-constrained. We introduce Holi-Spatial 360, a generative reconstruction and data curation framework for converting these panoramas into annotated multi-room 3D environments. Because reconstruction quality bounds subsequent generation, our Correspondence-Guided Geometric Optimization converts feed-forward reconstruction priors into reliable cross-view constraints and builds a coherent 3D Gaussian Splatting representation as 3D spatial memory. Adapted single-image and video priors then restore appearance and complete continuous trajectories while preserving observed content. The reconstructed scenes support arbitrary RGB-D rendering and an automated curation pipeline that combines end-to-end 3D grounding with VLM-based verification. Holi360 contains over 1K multi-room 3DGS scenes, more than 100K instance annotations, and 2M spatial question-answer pairs. We also introduce the Holi360 benchmark to evaluate the spatial understanding of VLMs in real-world, multi-room indoor environments. Experiments demonstrate superior reconstruction and curation quality under sparse panoramic inputs, while fine-tuning VLMs on Holi360 substantially improves spatial reasoning capabilities.
Yuanyuan Gao, Zhirui Liu, Yuning Gong +9
The Hong Kong University of Science and Technology · ShanghaiTech University · Tencent +4
Achieving human-like spatial intelligence for vision-language models (VLMs) requires inferring 3D structures from 2D observations, recognizing object properties and relations in 3D space, and performing high-level spatial reasoning. In this paper, we propose a principled hierarchical framework that decomposes the learning of 3D spatial understanding in VLMs into four progressively complex levels, from geometric perception to abstract spatial reasoning. Guided by this framework, we construct an automated pipeline that processes approximately 5M images with over 45M objects to generate 3D spatial VQA pairs across diverse tasks and scenes for VLM supervised fine-tuning. We also develop an RGB-D VLM incorporating metric-scale point maps as auxiliary inputs to further enhance spatial understanding. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on multiple spatial understanding and reasoning benchmarks, surpassing specialized spatial models and large proprietary systems such as Gemini-2.5-pro and GPT-5. Moreover, our analysis reveals clear dependencies among hierarchical task levels, offering new insights into how multi-level task design facilitates the emergence of 3D spatial intelligence.
Huizhi Liang, Yichao Shen, Yu Deng +5
Tsinghua University · Microsoft Research Asia · Xi’an Jiaotong University +1
Recent advancements in Large Vision-Language Models (VLMs) have demonstrated exceptional semantic understanding, yet these models consistently struggle with spatial reasoning, often failing at fundamental geometric tasks such as depth ordering and precise coordinate grounding. Recent efforts introduce spatial supervision from scene-centric datasets (e.g., multi-view scans or indoor video), but are constrained by the limited number of underlying scenes. As a result, the scale and diversity of such data remain significantly smaller than those of web-scale 2D image collections. To address this limitation, we propose SpatialForge, a scalable data synthesis pipeline that transforms in-the-wild 2D images into spatial reasoning supervision. Our approach decomposes spatial reasoning into perception and relation, and constructs structured supervision signals covering depth, layout, and viewpoint-dependent reasoning, with automatic verification to ensure data quality. Based on this pipeline, we build SpatialForge-10M, a large-scale dataset containing 10 million spatial QA pairs. Extensive experiments across multiple spatial reasoning benchmarks demonstrate that training on SpatialForge-10M significantly improves the spatial reasoning ability of standard VLMs, highlighting the effectiveness of scaling 2D data for 3D-aware spatial reasoning.