Holi-Spatial 360: Generative Reconstruction from Sparse Panoramas for Spatial Rendering and Intelligence

Yuanyuan Gao; Zhirui Liu; Yuning Gong; Yifei Liu; Yifan Zhan; Kuixuan Jiao; Xiaolong Zhou; Yifan Yang

cs.CVOct 14, 2026

Holi-Spatial 360: Generative Reconstruction from Sparse Panoramas for Spatial Rendering and Intelligence

Authors: Yuanyuan Gao, Zhirui Liu, Yuning Gong, Yifei Liu, Yifan Zhan, Kuixuan Jiao, Xiaolong Zhou, Yifan Yang,

Organizations: The Hong Kong University of Science and Technology · ShanghaiTech University · Tencent · Shanghai Artificial Intelligence Laboratory ·

Abstract

Spatial intelligence requires scalable 3D environments that are both photorealistic and semantically grounded, yet existing indoor datasets are dominated by room-scale captures and costly LiDAR acquisition. Online real-estate platforms offer abundant panoramas of diverse multi-room homes, but sparse viewpoints and texture-poor regions make reconstruction severely under-constrained. We introduce Holi-Spatial 360, a generative reconstruction and data curation framework for converting these panoramas into annotated multi-room 3D environments. Because reconstruction quality bounds subsequent generation, our Correspondence-Guided Geometric Optimization converts feed-forward reconstruction priors into reliable cross-view constraints and builds a coherent 3D Gaussian Splatting representation as 3D spatial memory. Adapted single-image and video priors then restore appearance and complete continuous trajectories while preserving observed content. The reconstructed scenes support arbitrary RGB-D rendering and an automated curation pipeline that combines end-to-end 3D grounding with VLM-based verification. Holi360 contains over 1K multi-room 3DGS scenes, more than 100K instance annotations, and 2M spatial question-answer pairs. We also introduce the Holi360 benchmark to evaluate the spatial understanding of VLMs in real-world, multi-room indoor environments. Experiments demonstrate superior reconstruction and curation quality under sparse panoramic inputs, while fine-tuning VLMs on Holi360 substantially improves spatial reasoning capabilities.

Figures & tables

Figure 1: Overview of Holi-Spatial 360. Sparse panoramas are reconstructed into geometrically multi-room 3DGS, densified with image and video generative priors, and converted into RGB-D, 3D grounding, and spatial-reasoning supervision.

Figure 2: Framework overview. DA3 correspondences anchor 3DGS optimization; adapted DiFix and Wan-VACE ( Jiang et al., 2025 ) densify trajectories; verified RGB-D groundings yield 3D instances, captions, and spatial QA.

Figure 3: Panorama-derived supervision. Left: Held-out panoramas provide image pairs and fixed-center rotation sequences for adapting DiFix and Wan-VACE; Right: An example shows the train–test split and co-visibility graph.

Figure 4: Interpolated-view generation from trajectory endpoints. Ours better preserves scene identity and temporal consistency than existing reconstruction and restoration methods.

	Room-Random Batch 1			Room-Random Batch 2			Room-Random Batch 3
Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Reconstruction Baselines
3DGS	19.68	0.797	0.365	18.63	0.738	0.410	19.54	0.777	0.407
2DGS	16.99	0.752	0.393	17.47	0.725	0.420	17.93	0.768	0.417
PGSR	17.32	0.726	0.439	16.71	0.646	0.483	17.51	0.684	0.479
FSGS	20.01	0.820	0.345	18.83	0.756	0.391	19.57	0.801	0.397

Table 1: Novel-view synthesis on three room-random batches.

\uparrow/\downarrow

indicate better. ∗ denotes models fine-tuned on our data.

Figure 5: Gaussian-rendered out-of-distribution novel views. Columns compare methods; rows share target viewpoints.

Figure 6: OOD novel-view depth over 0.2–10.0 m. Ours produces more complete and coherent geometry than 3DGS, 2DGS, and PGSR.

Table 8

Figure 7: 3D grounding on real-world scenes. Qualitative comparison against pseudo ground truth, SpatialLM ( Mao et al., 2025 ) , SceneScript ( Avetisyan et al., 2024 ) , and LLaVA-3D ( Zhu et al., 2024 ) . Our verified boxes align more closely with the target objects across diverse categories.

Model	Classification Macro $\uparrow$	Numerical [email protected] $\uparrow$	Numerical MAE $\downarrow$	13-Class Macro $\uparrow$	Scene Macro $\uparrow$
VST-SFT-3B	48.04	30.65	1.2444	44.02	37.90
InternVL3-8B	43.10	28.95	1.2257	39.83	34.54
Qwen3.8-27B	53.15	49.46	0.7389	52.30	49.51
Qwen3.5-4B	46.77	37.59	0.9767	44.65	42.16
Qwen3.5-4B+Ours	85.25	76.34	0.3898	83.19	81.27

Table 4: Spatial reasoning on the Holi360 benchmark.

Table 11

Figure 8: Ablation study on DiFix and Wan; DiFix and Wan finetuned denote DiFix and Wan models after our finetuning. W/O depth denotes only GS RGB input without depth.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Qualitative comparison between pseudo 3D bounding-box ground truth and our predicted annotations.

Figure 11: Ablation of the correspondence-guided module in our LRM. All visualized correspondences have reprojection errors below 1 pixel.

Figure 12: Extra ablation study on DiFix and Wan.

Figure 13: Ablation on our cross-view merge and VLM verifier.

Figure 14: Additional qualitative results. Reference views, 3DGS inputs, and generated trajectories across diverse indoor scenes.

Method	2 Frames	4 Frames	8 Frames
DA3	114.6	87.1	79.0
VGGT	46.8	22.7	15.5
i3dgs	44.0	44.0	44.0
COLMAP	29.0	29.0	29.0

Appendix

Table 7: Forward–backward-consistent matches (

<1

px) versus input-view count.

Figure 15: Holi360 benchmark QA visualization.

Indoor Scene Reconstruction

Indoor scene reconstruction builds 3D models of rooms and homes from images, video, depth, or sensor scans for mapping, simulation, and robotics.

See all 39 papers

Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence
Mar 8, 2026Yuanyuan Gao, Hao Li, Yifei Liu +143D Scene Understanding 3D Spatial Reasoning
Shanghai AI Lab · Northwestern Polytechnical University · Nanyang Technological University +8
Rein3D: Reinforced 3D Indoor Scene Generation with Panoramic Video Diffusion Models
Apr 12, 2026Dehui Wang, Rong Wei, Yue Shi +9Indoor Scene Generation Panoramic 3D Reconstruction
Shanghai Jiao Tong University · Manycore Tech Inc. · The University of Hong Kong +1
PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning
Feb 25, 2026Zekai Lin, Xu Zheng3D Spatial Reasoning 3D VQA
University of Glasgow · HKUST(GZ)

Mar 8, 2026J/K move · Enter open · S save

Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence

Yuanyuan Gao, Hao Li, Yifei Liu, Xinhao Ji+13

Shanghai AI Lab · Northwestern Polytechnical University · Nanyang Technological University+8

3D Scene Understanding 3D Spatial Reasoning 3D VQA 3D Visual Grounding

Open paper AskPDF arXiv

The pursuit of spatial intelligence fundamentally relies on access to large-scale, fine-grained 3D data. However, existing approaches predominantly construct spatial understanding benchmarks by generating question-answer (QA) pairs from a limited number of manually annotated datasets, rather than systematically annotating new large-scale 3D scenes from raw web data. As a result, their scalability is severely constrained, and model performance is further hindered by domain gaps inherent in these narrowly curated datasets. In this work, we propose Holi-Spatial, the first fully automated, large-scale, spatially-aware multimodal dataset, constructed from raw video inputs without human intervention, using the proposed data curation pipeline. Holi-Spatial supports multi-level spatial supervision, ranging from geometrically accurate 3D Gaussian Splatting (3DGS) reconstructions with rendered depth maps to object-level and relational semantic annotations, together with corresponding spatial Question-Answer (QA) pairs. Following a principled and systematic pipeline, we further construct Holi-Spatial-4M, the first large-scale, high-quality 3D semantic dataset, containing 12K optimized 3DGS scenes, 1.3M 2D masks, 320K 3D bounding boxes, 320K instance captions, 1.2M 3D grounding instances, and 1.2M spatial QA pairs spanning diverse geometric, relational, and semantic reasoning tasks. Holi-Spatial demonstrates exceptional performance in data curation quality, significantly outperforming existing feed-forward and per-scene optimized methods on datasets such as ScanNet, ScanNet++, and DL3DV. Furthermore, fine-tuning Vision-Language Models (VLMs) on spatial reasoning tasks using this dataset has also led to substantial improvements in model performance.

Abstract

Indoor Scene Reconstruction

Explore similar work

Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence

Rein3D: Reinforced 3D Indoor Scene Generation with Panoramic Video Diffusion Models

PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning