Language and Geometry Grounded Sparse Voxel Representations for Holistic Scene Understanding

Guile Wu; David Huang; Bingbing Liu; Dongfeng Bai

cs.CVFeb 17, 2026

Language and Geometry Grounded Sparse Voxel Representations for Holistic Scene Understanding

Authors: Guile Wu, David Huang, Bingbing Liu, Dongfeng Bai

Organizations: Huawei Noah’s Ark Lab · University of Toronto

Abstract

Existing 3D open-vocabulary scene understanding methods mostly emphasize distilling language features from 2D foundation models into 3D feature fields, but largely overlook the synergy among scene appearance, semantics, and geometry. As a result, scene understanding often deviates from the underlying geometric structure of scenes and becomes decoupled from the reconstruction process. In this work, we propose a novel approach that leverages language and geometry grounded sparse voxel representations to comprehensively model appearance, semantics, and geometry within a unified framework. Specifically, we use 3D sparse voxels as primitives and employ an appearance field, a density field, a feature field, and a confidence field to holistically represent a 3D scene. To promote synergy among the appearance, density, and feature fields, we construct a feature modulation module and distill language features from a 2D foundation model into our 3D scene model. In addition, we integrate geometric distillation into feature field distillation to transfer geometric knowledge from a geometry foundation model to our 3D scene representations via depth correlation regularization and pattern consistency regularization. These components work together to synergistically model the appearance, semantics, and geometry of the 3D scene within a unified framework. Extensive experiments demonstrate that our approach achieves superior overall performance compared with state-of-the-art methods in holistic scene understanding and reconstruction.

Figures & tables

Figure 2: An overview of the proposed approach. Our approach optimizes language and geometry grounded sparse voxels as the 3D scene representations to comprehensively model the appearance, semantics, and geometry of the 3D scene in a unified framework.

Figure 3: The illustrations of (a) the proposed feature modulation module and (b) the architecture of the employed autoencoder. We show the hidden dimension of each layer above the corresponding layer.

Method	LERF dataset				Mip-NeRF360 dataset
	Understanding		Reconstruction		Understanding		Reconstruction
	mIoU $\uparrow$	mAcc $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	mAcc $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
LERF [ 5 ]	37.4	73.6	20.75	0.264	-	-	-	-
Feature-3DGS [ 37 ]	38.3	70.1	23.98	0.222	-	-	-	-
LangSplat ⋆ [ 20 ]	51.4	84.3	23.89	0.218	54.7	73.1	29.38	0.183
LEGaussians [ 24 ]	24.5	67.4	-	-	29.1	65.2	-	-

Table 1: Quantitative comparison with the state-of-the-arts on the LERF and Mip-NeRF360 datasets. We report average results across four scenes for each dataset. ⋆ : 3D reconstruction results for these two-stage methods are borrowed from 3DGS, as they use pre-trained 3DGS for feature distillation. † : Lifting fine-grained features to 3D primitives. Some results are cited from existing work.

Figure 4: Qualitative comparison with state-of-the-art methods in 3D semantic segmentation on the LERF and Mip-NeRF360 datasets. The segmented regions are highlighted.

Figure 5: Qualitative comparison with state-of-the-art methods in 3D object localization on the LERF and Mip-NeRF360 datasets. Red dots denote the positions with the highest localization responses and black dashed boxes represent the ground-truth localizations.

Figure 6: Qualitative comparison with state-of-the-art methods in novel view synthesis on the LERF and Mip-NeRF360 datasets. Red dashed boxes highlight some fine-grained details.

Method	Scene Understanding		Scene Reconstruction		Speed †	GPU
Method	Seg.(mIoU $\uparrow$ )	Loc.(mAcc $\uparrow$ )	PSNR $\uparrow$	LPIPS $\downarrow$	(Image FPS $\uparrow$ )	Memory $\downarrow$ ⋆
SVRaster	46.1	61.1	23.88	0.215	69	12
Ours	62.1	84.4	24.02	0.212	35	14

Table 2: Effectiveness and efficiency against SVRaster on the LERF dataset. † Results are obtained with a V100 GPU. ⋆ Training GPU memory in terms of GB.

Figure 7: Visualizations of the rendered depth maps, depth to normal maps, normal maps, and extracted meshes on the LERF dataset.

Variant	Understanding		Reconstruction
Variant	mIoU $\uparrow$	mAcc $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
Ours w/ VGGT [ 26 ]	62.0	84.4	24.06	0.212
Ours w/ Depth-Anything-V2 [ 32 ]	62.1	84.4	24.02	0.212

Table 3: Evaluation of using different geometric foundation models in the proposed approach. Results on the LERF dataset are reported.

Figure 8: Evaluation of the effectiveness of (a) geometric distillation, (b) feature modulation, and (c) latent feature dimension

k

in the proposed method. Results on the LERF dataset are reported.

Variant	mAcc $\uparrow$	PSNR $\uparrow$
Ours	84.4	24.02
Ours w/o conf.	82.2	24.07

Table 4: Analysis of the confidence field (results on the LERF dataset).

Figure 9: An example of a failure case. All evaluated methods failed to locate the corn in the bowl since the corn in the bowl is very small.

3D Scene Understanding

3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.

See all 295 papers

LESV: Language Embedded Sparse Voxel Fusion for Open-Vocabulary 3D Scene Understanding
Apr 1, 2026Fusang Wang, Nathan Piasco, Moussab Bennehar +3Point Cloud Understanding 3D Scene Understanding
Huawei Noah’s Ark Lab · CAOR, Mines Paris-PSL, France
VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching
May 29, 2026Tuan Duc Ngo, Chuang Gan, Evangelos Kalogerakis3D Reconstruction 3D Scene Representation
UMass Amherst · TU Crete
SparseLGS: Sparse View Language Embedded Gaussian Splatting
Dec 3, 2024Jun Hu, Zhang Chen, Zhong Li +23D Gaussian Splatting Language-Embedded 3DGS
School of Mathematical Science, University of Science and Technology of China · Meta · Apple Inc. +1

Apr 1, 2026J/K move · Enter open · S save

LESV: Language Embedded Sparse Voxel Fusion for Open-Vocabulary 3D Scene Understanding

Fusang Wang, Nathan Piasco, Moussab Bennehar, Luis Roldão+2

Huawei Noah’s Ark Lab · CAOR, Mines Paris-PSL, France

Point Cloud Understanding 3D Scene Understanding 3D VLMs

Open paper AskPDF arXiv

Recent advancements in open-vocabulary 3D scene understanding heavily rely on 3D Gaussian Splatting (3DGS) to register vision-language features into 3D space. However, we identify two critical limitations in these approaches: the spatial ambiguity arising from unstructured, overlapping Gaussians which necessitates probabilistic feature registration, and the multi-level semantic ambiguity caused by pooling features over object-level masks, which dilutes fine-grained details. To address these challenges, we present a novel framework that leverages Sparse Voxel Rasterization (SVRaster) as a structured, disjoint geometry representation. By regularizing SVRaster with monocular depth and normal priors, we establish a stable geometric foundation. This enables a deterministic, confidence-aware feature registration process and suppresses the semantic bleeding artifact common in 3DGS. Furthermore, we resolve multi-level ambiguity by exploiting the emerging dense alignment properties of the AM-RADIO foundation model, avoiding the computational overhead of hierarchical training methods. Our approach achieves state-of-the-art performance on Open Vocabulary Point Cloud Understanding, and highly competitive results on 3D and 2D Object Retrieval benchmarks.

Abstract

3D Scene Understanding

Explore similar work

LESV: Language Embedded Sparse Voxel Fusion for Open-Vocabulary 3D Scene Understanding

VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching

SparseLGS: Sparse View Language Embedded Gaussian Splatting