Organizations: Shanghai AI Lab · Northwestern Polytechnical University · Nanyang Technological University · Beihang University · Peking University · Sichuan University · Tsinghua University · The Chinese University of Hong Kong · Fudan University · Hong Kong University of Science and Technology · Shanghai Jiao Tong University
The pursuit of spatial intelligence fundamentally relies on access to large-scale, fine-grained 3D data. However, existing approaches predominantly construct spatial understanding benchmarks by generating question-answer (QA) pairs from a limited number of manually annotated datasets, rather than systematically annotating new large-scale 3D scenes from raw web data. As a result, their scalability is severely constrained, and model performance is further hindered by domain gaps inherent in these narrowly curated datasets. In this work, we propose Holi-Spatial, the first fully automated, large-scale, spatially-aware multimodal dataset, constructed from raw video inputs without human intervention, using the proposed data curation pipeline. Holi-Spatial supports multi-level spatial supervision, ranging from geometrically accurate 3D Gaussian Splatting (3DGS) reconstructions with rendered depth maps to object-level and relational semantic annotations, together with corresponding spatial Question-Answer (QA) pairs. Following a principled and systematic pipeline, we further construct Holi-Spatial-4M, the first large-scale, high-quality 3D semantic dataset, containing 12K optimized 3DGS scenes, 1.3M 2D masks, 320K 3D bounding boxes, 320K instance captions, 1.2M 3D grounding instances, and 1.2M spatial QA pairs spanning diverse geometric, relational, and semantic reasoning tasks. Holi-Spatial demonstrates exceptional performance in data curation quality, significantly outperforming existing feed-forward and per-scene optimized methods on datasets such as ScanNet, ScanNet++, and DL3DV. Furthermore, fine-tuning Vision-Language Models (VLMs) on spatial reasoning tasks using this dataset has also led to substantial improvements in model performance.
Figures & tables
Figure 2: Comparison of our refined annotations with the official annotations on the ScanNet dataset [ 28 ] . Our method achieves more accurate and sharper segmentation masks, as well as improved category recognition.
Method
Inputs
Outputs
Images
Point Clouds
Depth
2D Seg.
3D Det.
Grounding
Spatial QA
2D-VLM Methods
SAM3
✓
✗
✗
✓
✗
✗
✗
SA2VA
✓
✗
✗
✓
✗
✓
✓
3D-VLM Methods
SpatialLM
✗
✓
✗
✗
✓
✓
✓
Table 1: Overview of Pipeline Capabilities. We compare input modalities and output tasks across different paradigms, ranging from 2D VLMs and 3D-VLMs to 3D-GS-based understanding methods. Holi-Spatial serves as a unified framework supporting diverse spatial tasks without relying on 3D priors.
Figure 3: Overview of the Holi-Spatial data curation pipeline. The framework operates in three progressive stages: (1) Geometric Optimization distills high-fidelity 3D structure from video streams using 3DGS; (2) Image-level Perception lifts 2D VLM and SAM3 predictions into initial 3D proposals; and (3) Scene-level Refinement employs a coarse-to-fine strategy to merge, verify, and caption instances, yielding dense, high-quality spatial annotations. Finally, leveraging the generated Holi-Spatial-4M dataset, we directly fine-tune the Qwen-VL family for downstream tasks ( e.g. , 3D grounding and spatial reasoning).
Figure 4: Pipeline of 2D-to-3D OBB Generation. We transform 2D object masks into initial 3D OBBs via depth projection, utilizing a four-step strategy to mitigate the impact of depth floaters. (1) We obtain an initial object depth map by combining 3DGS rendering with SAM3 instance segmentation. (2) To mitigate 2D boundary errors from SAM3, we erode the object mask near its contour and keep only the reliable interior region. (3) To remove 3D outliers caused by depth discontinuities, we use a multi-view-consistent mesh depth as guidance and filter inconsistent pixels in the 3DGS depth. (4) Finally, we estimate the initial 3D OBB from the refined point cloud, while preserving the associated 2D mask, confidence score, and source image index.
Figure 5: Floor-aligned OBB post-processing pipeline. Starting from the input instance OBBs produced after 2D-to-3D lifting and initial OBB estimation, we (1) detect a floor (or fallback planar structure) to infer a global up-axis, (2) re-align each instance OBB under a yaw-lock constraint with optional PCA fallback and update extents, and (3) apply a validation check to output gravity-/floor-consistent OBBs for downstream scene-level refinement.
Figure 6: Comprehensive Statistics of Holi-Spatial-4M. (1) Object Diversity: Word cloud showing the long-tailed distribution of open-vocabulary categories. (2) Dataset Composition: The inner ring displays source scenes (ScanNet [ 28 ] , ScanNet++ [ 27 ] , DL3DV [ 31 ] ), while the outer ring details the breakdown of over 4M generated spatial annotations. (3) Spatial QA Taxonomy: Distribution of 1.25M spatial QA pairs, categorized into Camera-centric tasks ( e.g. , rotation, movement) and Object-centric tasks ( e.g. , distance, direction).
Method
ScanNet
ScanNet++
DL3DV
Depth ( ↑ )
2D Seg. ( ↑ )
3D Det. ( ↑ )
Depth ( ↑ )
2D Seg. ( ↑ )
3D Det. ( ↑ )
Depth ( ↑ )
2D Seg. ( ↑ )
3D Det. ( ↑ )
F1
IoU
AP 25
AP 50
F1
IoU
AP 25
AP 50
F1
IoU
AP 25
AP 50
2D-VLM Methods
SAM3
–
0.63
–
–
–
0.50
–
–
–
0.66
–
–
SA2VA
–
0.64
–
–
–
0.25
–
–
–
0.44
–
–
3D-VLM Methods
Table 2: Quantitative results of 3D spatial understanding on ScanNet, ScanNet++ and DL3DV datasets. We group methods by their input modality and task definition. 3D Det : 3D Object Detection (AP@25, AP@50); 2D Seg : 2D Object Segmentation (IoU); Depth : Depth Estimation (F1-score). Bold indicates best results. ↑ : higher is better. ‘—’ denotes that the corresponding metric is not available.
Figure 7: Qualitative comparison of multi-view depth on ScanNet++. We visualize depth maps from GT and baselines (LangSplat, M3-Spatial, Depth-Anything-V3) versus Holi-Spatial. Right: point clouds obtained by multi-view back-projection, where Holi-Spatial produces cleaner geometry with substantially fewer ghosting artifacts and floaters.
Figure 8: Qualitative comparison of open-vocabulary 2D instance segmentation on the same scenes. Holi-Spatial yields sharper boundaries, more complete masks under occlusion, and more accurate fine-grained categories.
Figure 9: Qualitative comparison of 3D object detection on ScanNet++. We visualize predicted 3D bounding boxes from SceneScript, LLaVA-3D, SpatialLM, and Holi-Spatial. Holi-Spatial produces tighter boxes and more correct categories.
Model
MMSI-Bench
MindCube
VST-SFT-3B [ 26 ]
30.2
35.9
Cambrian-S-3B [ 3 ]
25.2
32.5
VST-SFT-7B [ 26 ]
32.0
39.7
Cambrian-S-7B [ 3 ]
25.8
39.6
SpaceR-SFT-7B [ 55 ]
27.4
37.9
Intern3-VL-8B [ 57 ]
28.0
41.5
Table 3: Quantitative results of Spatial Understanding QA tasks. Here we caompare different models [ 55 , 57 , 25 ] on MMSI-bench [ 18 ] and MindCube [ 19 ] benchmarks. Bold indicates the best performance.
Method
AP 15
AP 25
AP 50
VST-7B-SFT [ 26 ]
17.29
14.50
11.20
Qwen3-VL-8B [ 13 ]
19.82
16.80
13.50
Qwen3-VL-8B + Ours
35.52
31.94
27.98
Table 4: Quantitative results of 3D Grounding on ScanNet++ [ 27 ] dataset.
Figure 10: Stage-wise visualization of scene-level refinement. Detailed discussion is include in Section 5.3
Figure 11: Given the same input images and queried objects, our predicted boxes align better with ground truth, indicating improved spatial localization.
ID
DA3 Depth
3DGS Training
Conf. Filter
Agent Recall
P25
R25
Step 1: Geometric Optimization
1
✓
✗
✓
✓
0.13
0.31
2
✓
✓
✓
✓
0.81
0.89
Step 3: Scene-Level Refinement
3
✓
✓
✗
✗
0.35
0.74
4
✓
✓
✓
✗
0.67
0.69
Table 5: Ablation study on depth refinement, confidence filtering, and agent recall. ID. 1–2 (Step 1) compare depth sources, while ID. 3–5 (Step 3) evaluate post-processing modules. P25 and R25 denote precision and recall under IoU 25%.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Ablation study on 3D multi-view merging and depth refinement. (a) 3D geometric clustering corrects the fragmentation caused by occlusions in SAM3 image-level instance predictions, merging separated segments into unified object instances. (b) Using GS-refined depth constraints enables more accurate multi-view lifting, which avoids false merges and preserves the integrity of spatial object instances.
Figure 13: Prompt template used for Agent verification.
Figure 14: Prompt template used for object grounding in 2D image.
Figure 15: Additional 3D detection visualizations. We show Holi-Spatial predictions across diverse indoor scenes, demonstrating multi-view consistent localization and robust detection.
Figure 16: Examples of 10 types of curated spatial QA pairs in Holi-Spatial.
Figure 17: Training with our curated QA data improves the three types of spatial reasoning tasks in MindCube [ 19 ] (upper row) as well as in MMSI-Bench [ 18 ] (lower row).