Existing 3D open-vocabulary scene understanding methods mostly emphasize distilling language features from 2D foundation models into 3D feature fields, but largely overlook the synergy among scene appearance, semantics, and geometry. As a result, scene understanding often deviates from the underlying geometric structure of scenes and becomes decoupled from the reconstruction process. In this work, we propose a novel approach that leverages language and geometry grounded sparse voxel representations to comprehensively model appearance, semantics, and geometry within a unified framework. Specifically, we use 3D sparse voxels as primitives and employ an appearance field, a density field, a feature field, and a confidence field to holistically represent a 3D scene. To promote synergy among the appearance, density, and feature fields, we construct a feature modulation module and distill language features from a 2D foundation model into our 3D scene model. In addition, we integrate geometric distillation into feature field distillation to transfer geometric knowledge from a geometry foundation model to our 3D scene representations via depth correlation regularization and pattern consistency regularization. These components work together to synergistically model the appearance, semantics, and geometry of the 3D scene within a unified framework. Extensive experiments demonstrate that our approach achieves superior overall performance compared with state-of-the-art methods in holistic scene understanding and reconstruction.
Figures & tables
Figure 2: An overview of the proposed approach. Our approach optimizes language and geometry grounded sparse voxels as the 3D scene representations to comprehensively model the appearance, semantics, and geometry of the 3D scene in a unified framework.
Figure 3: The illustrations of (a) the proposed feature modulation module and (b) the architecture of the employed autoencoder. We show the hidden dimension of each layer above the corresponding layer.
Method
LERF dataset
Mip-NeRF360 dataset
Understanding
Reconstruction
Understanding
Reconstruction
mIoU ↑
mAcc ↑
PSNR ↑
LPIPS ↓
mIoU ↑
mAcc ↑
PSNR ↑
LPIPS ↓
LERF [ 5 ]
37.4
73.6
20.75
0.264
-
-
-
-
Feature-3DGS [ 37 ]
38.3
70.1
23.98
0.222
-
-
-
-
LangSplat ⋆ [ 20 ]
51.4
84.3
23.89
0.218
54.7
73.1
29.38
0.183
LEGaussians [ 24 ]
24.5
67.4
-
-
29.1
65.2
-
-
Table 1: Quantitative comparison with the state-of-the-arts on the LERF and Mip-NeRF360 datasets. We report average results across four scenes for each dataset. ⋆ : 3D reconstruction results for these two-stage methods are borrowed from 3DGS, as they use pre-trained 3DGS for feature distillation. † : Lifting fine-grained features to 3D primitives. Some results are cited from existing work.
Figure 4: Qualitative comparison with state-of-the-art methods in 3D semantic segmentation on the LERF and Mip-NeRF360 datasets. The segmented regions are highlighted.
Figure 5: Qualitative comparison with state-of-the-art methods in 3D object localization on the LERF and Mip-NeRF360 datasets. Red dots denote the positions with the highest localization responses and black dashed boxes represent the ground-truth localizations.
Figure 6: Qualitative comparison with state-of-the-art methods in novel view synthesis on the LERF and Mip-NeRF360 datasets. Red dashed boxes highlight some fine-grained details.
Method
Scene Understanding
Scene Reconstruction
Speed †
GPU
Seg.(mIoU ↑ )
Loc.(mAcc ↑ )
PSNR ↑
LPIPS ↓
(Image FPS ↑ )
Memory ↓ ⋆
SVRaster
46.1
61.1
23.88
0.215
69
12
Ours
62.1
84.4
24.02
0.212
35
14
Table 2: Effectiveness and efficiency against SVRaster on the LERF dataset. † Results are obtained with a V100 GPU. ⋆ Training GPU memory in terms of GB.
Figure 7: Visualizations of the rendered depth maps, depth to normal maps, normal maps, and extracted meshes on the LERF dataset.
Variant
Understanding
Reconstruction
mIoU ↑
mAcc ↑
PSNR ↑
LPIPS ↓
Ours w/ VGGT [ 26 ]
62.0
84.4
24.06
0.212
Ours w/ Depth-Anything-V2 [ 32 ]
62.1
84.4
24.02
0.212
Table 3: Evaluation of using different geometric foundation models in the proposed approach. Results on the LERF dataset are reported.
Figure 8: Evaluation of the effectiveness of (a) geometric distillation, (b) feature modulation, and (c) latent feature dimension k in the proposed method. Results on the LERF dataset are reported.
Variant
mAcc ↑
PSNR ↑
Ours
84.4
24.02
Ours w/o conf.
82.2
24.07
Table 4: Analysis of the confidence field (results on the LERF dataset).
Figure 9: An example of a failure case. All evaluated methods failed to locate the corn in the bowl since the corn in the bowl is very small.
Recent advancements in open-vocabulary 3D scene understanding heavily rely on 3D Gaussian Splatting (3DGS) to register vision-language features into 3D space. However, we identify two critical limitations in these approaches: the spatial ambiguity arising from unstructured, overlapping Gaussians which necessitates probabilistic feature registration, and the multi-level semantic ambiguity caused by pooling features over object-level masks, which dilutes fine-grained details. To address these challenges, we present a novel framework that leverages Sparse Voxel Rasterization (SVRaster) as a structured, disjoint geometry representation. By regularizing SVRaster with monocular depth and normal priors, we establish a stable geometric foundation. This enables a deterministic, confidence-aware feature registration process and suppresses the semantic bleeding artifact common in 3DGS. Furthermore, we resolve multi-level ambiguity by exploiting the emerging dense alignment properties of the AM-RADIO foundation model, avoiding the computational overhead of hierarchical training methods. Our approach achieves state-of-the-art performance on Open Vocabulary Point Cloud Understanding, and highly competitive results on 3D and 2D Object Retrieval benchmarks.
Fusang Wang, Nathan Piasco, Moussab Bennehar +3
Huawei Noah’s Ark Lab · CAOR, Mines Paris-PSL, France
Reconstructing the complete geometry of a scene from a single RGB image remains challenging, especially when inferring hidden structures where visual evidence is incomplete. We introduce VolFill, a generative framework that predicts the 3D structure of the complete scene rather than relying on traditional pixel-aligned regression. Our method utilizes a hybrid 3D VAE to compress sparse truncated unsigned distance function grids into a compact latent space, paired with a latent Diffusion Transformer that denoises this representation to recover the complete scene. We condition the generation on a geometry foundation model, leveraging rich spatial priors for robust reasoning. Unlike existing methods limited by per-ray constraints or unstructured point-cloud queries, VolFill provides a structured representation that supports direct surface extraction and occupancy queries at scale. Extensive experiments on the SCRREAM and NRGB-D datasets demonstrate that our approach significantly outperforms current amodal baselines, providing a robust foundation for holistic spatial understanding.
Recently, several studies have combined Gaussian Splatting to obtain scene representations with language embeddings for open-vocabulary 3D scene understanding. While these methods perform well, they essentially require very dense multi-view inputs, limiting their applicability in real-world scenarios. In this work, we propose SparseLGS to address the challenge of 3D scene understanding with pose-free and sparse view input images. Our method leverages a learning-based dense stereo model to handle pose-free and sparse inputs, and a three-step region matching approach to address the multi-view semantic inconsistency problem, which is especially important for sparse inputs. Different from directly learning high-dimensional CLIP features, we extract low-dimensional information and build bijections to avoid excessive learning and storage costs. We introduce a reconstruction loss during semantic training to improve Gaussian positions and shapes. To the best of our knowledge, we are the first to address the 3D semantic field problem with sparse pose-free inputs. Experimental results show that SparseLGS achieves comparable quality when reconstructing semantic fields with fewer inputs (3-4 views) compared to previous SOTA methods with dense input. Besides, when using the same sparse input, SparseLGS leads significantly in quality and heavily improves the computation speed (5×speedup). Project page: https://ustc3dv.github.io/SparseLGS
Jun Hu, Zhang Chen, Zhong Li +2
School of Mathematical Science, University of Science and Technology of China · Meta · Apple Inc. +1