Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes

Changqing Zhou; Yueru Luo; Han Zhang; Zeyu Jiang; Changhao Chen

cs.CVFeb 26, 2026

Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes

Authors: Changqing Zhou, Yueru Luo, Han Zhang, Zeyu Jiang, Changhao Chen

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen

Abstract

Open-vocabulary 3D occupancy is vital for embodied agents, which need to understand complex indoor environments where semantic categories are abundant and evolve beyond fixed taxonomies. While recent work has explored open-vocabulary occupancy in outdoor driving scenarios, such methods transfer poorly indoors, where geometry is denser, layouts are more intricate, and semantics are far more fine-grained. To address these challenges, we adopt a geometry-only supervision paradigm that uses only binary occupancy labels (occupied vs free). Our framework builds upon 3D Language-Embedded Gaussians, which serve as a unified intermediate representation coupling fine-grained 3D geometry with a language-aligned semantic embedding. On the geometry side, we find that existing Gaussian-to-Occupancy operators fail to converge under such weak supervision, and we introduce an opacity-aware, Poisson-based approach that stabilizes volumetric aggregation. On the semantic side, direct alignment between rendered features and open-vocabulary segmentation features suffers from feature mixing; we therefore propose a Progressive Temperature Decay schedule that gradually sharpens opacities during splatting, strengthening Gaussian-language alignment. On Occ-ScanNet, our framework achieves 59.50 IoU and 21.05 mIoU in the open-vocabulary setting, surpassing all existing occupancy methods in IoU and outperforming prior open-vocabulary approaches by a large margin in mIoU. Code will be released at https://github.com/JuIvyy/LegoOcc.

Figures & tables

Figure 1: Closed- vs. open-vocabulary occupancy. Prior methods [ 50 , 47 ] trained under a closed vocabulary can label only the categories predefined at training time, which restricts real-world deployment. Our open-vocabulary approach aligns language with 3D occupancy and supports text queries for arbitrary categories. Right column (Random Class): text-conditioned per-voxel scores are visualized as heatmaps; darker red indicates higher likelihood for the queried category.

Figure 2: LegoOcc Framework Overview. From a monocular image, a feed-forward Gaussian model produces Language-Embedded Gaussians . Training proceeds along two paths: Semantic learning , we differentiably render Gaussian features to the image with Progressive Temperature Decay and align them to a training-free open-vocabulary segmenter via a cosine objective $L_{\text{feat}}$ ; Geometry learning , we convert Gaussians to occupancy using an opacity-aware Poisson-based Gaussian-to-Occupancy operator and supervise binary occupancy with $L_{\text{occ}}$ . At inference, the Language-Embedded Occupancy supports text-driven queries by computing cosine similarity between voxel embeddings and prompt embeddings, yielding open-vocabulary semantic occupancy without dense voxel-level semantic labels during training.

Figure 3: Comparison of temperature schedules. Linear decay decreases

\tau

uniformly, whereas our exponential schedule rapidly approaches

T_{\min}

, allocating more iterations for the model to adapt to the low-temperature regime.

Method	IoU	$\blacksquare$ ceiling	$\blacksquare$ floor	$\blacksquare$ wall	$\blacksquare$ window	$\blacksquare$ chair	$\blacksquare$ bed	$\blacksquare$ sofa	$\blacksquare$ table	$\blacksquare$ tvs	$\blacksquare$ furniture	$\blacksquare$ objects	mIoU
Closed-vocabulary (full annotation)
TPVFormer [ 10 ]	33.39	6.96	32.97	14.41	9.10	24.01	41.49	45.44	28.61	10.66	35.37	25.31	24.94
GaussianFormer [ 11 ]	40.91	20.70	42.00	23.40	17.40	27.0	44.30	44.80	32.70	15.30	36.70	25.00	29.93
MonoScene [ 4 ]	41.60	15.17	44.71	22.41	12.55	26.11	27.03	35.91	28.32	6.57	32.16	19.84	24.62
ISO [ 50 ]	42.16	19.88	41.88	22.37	16.98	29.09	42.43	42.00	29.60	10.62	36.36	24.61	28.71
Surroundocc [ 45 ]	42.52	18.90	49.30	24.80	18.00	26.80	42.00	44.10	32.90	18.60	36.80	26.90	30.83

Table 1: Monocular results on Occ-ScanNet. † We re-implement POP-3D [ 41 ] and LOcc [ 53 ] for the indoor monocular setting, and supervise geometry with ground-truth binary occupancy instead of the authors’ LiDAR-derived pseudo labels. We also adopt DepthAnything-v2 [ 48 ] as the backbone and use the same input resolution and training recipe as ours for fair comparison. For POP-3D, we project voxel centers to the image plane and sample language-aligned features at those locations to form a 3D grid of text-aligned embeddings for open-vocabulary reasoning. For LOcc, we prompt Qwen2.5-VL-7B [ 2 ] to extract object names and use Trident [ 35 ] to obtain training-free open-vocabulary segmentations, which yield stronger 2D masks on Occ-ScanNet.

Operation	mIoU	IoU
GaussianFormer2	51.88	56.96
Bernoulli	51.68	58.06
Poisson	52.80	59.24
GaussianFormer2	0.00	0.00
Bernoulli	17.25	46.65
Poisson	21.05	59.50

Table 2: Ablation on the Gaussian-to-Occupancy operator. We compare three aggregation rules: GaussianFormer2, Bernoulli, and Poisson, under both closed- and open- vocabulary settings.

$T_{min}$	$T_{max}$	$\tau_{\text{test}}$	scheduler	mIoU	IoU
1.0	1.0	1	$\times$	18.15	59.19
1.0	1.0	1e-3	$\times$	12.83	32.52
1e-3	1e-3	1e-3	$\times$	0.00	0.00
1e-3	1.0	1e-3	linear	2.30	7.60
1e-2	1.0	1e-2	exp	20.85	59.25
1e-3	1.0	1e-3	exp	21.05	59.50

Table 3: Ablation on temperature scheduling. We vary the temperature range

(T_{\min},T_{\max})

, the test-time temperature

\tau_{\text{test}}

, and the scheduling strategy. “

\times

” denotes no schedule.

Model	mIoU	IoU	FPS	#parameters
ISO [ 50 ]	28.71	42.16	3.81	303.05M
EmbodiedOcc [ 47 ]	45.15	53.55	11.48	231.45M
POP-3D [ 41 ]	5.96	35.32	10.21	115.31M
LOcc [ 53 ]	9.25	36.70	8.93	113.50M
Ours	21.05	59.50	22.47	99.04M

Table 4: Model profile. FPS is measured on the same machine with a single RTX 4090 GPU, averaged over 1,000 runs after 100 warm-up runs.

Figure 4: Qualitative results on Occ-ScanNet. From top to bottom: (a) input images; (b) ground-truth semantic occupancy; (c) results from our re-implemented LOcc [ 53 ] ; (d) our method. Both (c) and (d) are trained with geometry-only annotations and evaluated on the closed-vocabulary annotation of Occ-ScanNet.

Figure 5: Open-vocabulary qualitative results. Legends list the VLM-extracted object nouns used as text queries. (a) Input image. (b) Open-vocabulary 2D segmentation for queried nouns. (c) Our 3D open-vocabulary occupancy colored by the same categories.

Explore similar work

Apr 30, 2026

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction

Existing learning-based occupancy prediction methods rely on large-scale 3D annotations and generalize poorly across environments. We present FreeOcc, a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences. Unlike prior approaches that require voxel-level supervision and ground-truth camera poses, FreeOcc operates without 3D annotations, pose ground truth, or any learning stage. FreeOcc incrementally builds a globally consistent occupancy map via a four-layer pipeline: a SLAM backbone estimates poses and sparse geometry; a geometrically consistent Gaussian update constructs dense 3D Gaussian maps; open-vocabulary semantics from off-the-shelf vision-language models are associated with Gaussian primitives; and a probabilistic Gaussian-to-occupancy projection produces dense voxel occupancy. Despite being entirely training-free and pose-agnostic, FreeOcc achieves over

2\times

improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods. We further introduce ReplicaOcc, a benchmark for indoor open-vocabulary occupancy prediction, and show that FreeOcc transfers zero-shot to novel environments, substantially outperforming both supervised and self-supervised baselines. Project page: https://the-masses.github.io/freeocc-web/.

Zeyu Jiang, Changqing Zhou, Xingxing Zuo +1

The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI

Semantic Occupancy Mapping 3D Occupancy Prediction

Ask PDF

Mar 12, 2026

O3N: Omnidirectional Open-Vocabulary Occupancy Prediction for Embodied Intelligent Robotics

The rapid evolution of consumer electronics toward embodied intelligence has accelerated the emergence of Consumer Embodied Intelligent Robotics (CEIRs), where intelligent devices are expected to perceive, understand, and interact with complex real-world environments. Understanding and reconstructing the 3D world through omnidirectional perception is therefore becoming increasingly important for CEIRs operating in complex and dynamic environments. However, existing vision-based 3D occupancy prediction methods are constrained by limited perspective inputs and a predefined training distribution, making them difficult to support embodied intelligent systems that require comprehensive and safe perception of scenes in open-world exploration. To address this, we present O3N, the first framework for open-vocabulary occupancy prediction from a single omnidirectional RGB image. O3N embeds omnidirectional voxels in a polar-spiral topology via the Polar-spiral Mamba (PsM) module, enabling continuous spatial representation and long-range context modeling across 360°. The Occupancy Cost Aggregation (OCA) module introduces a principled mechanism for unifying geometric and semantic supervision within the voxel space, ensuring consistency between reconstructed geometry and underlying semantic structure. Moreover, Natural Modality Alignment (NMA) establishes a gradient-free alignment pathway that harmonizes visual features, voxel embeddings, and text semantics, forming a consistent pixel-voxel-text representation triad for open-world perception. Extensive experiments on multiple models demonstrate that our method not only achieves state-of-the-art performance on QuadOcc and Human360Occ benchmarks but also exhibits remarkable cross-scene generalization and semantic scalability. The source code will be made publicly available at https://github.com/MengfeiD/O3N.

Mengfei Duan, Hao Shi, Fei Teng +4

School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha, China · Ant Group, Hangzhou, China · State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University, Hangzhou, China

3D Occupancy Prediction 3D Scene Understanding

Ask PDF

Jul 15, 2026

GPOcc++: Unified Sparse Gaussian Occupancy Prediction with Visual Geometry Priors

Accurate 3D scene understanding is fundamental to embodied intelligence and autonomous driving, where 3D occupancy provides a unified representation of objects, structures, and free space. However, recovering such a complete volumetric representation from visual observations remains challenging, particularly in occluded and unobserved regions. Visual geometry priors offer strong and generalizable geometric cues for addressing this challenge, but their outputs are inherently surface-centric, whereas occupancy prediction requires reasoning about volumetric interiors and free space. To bridge this gap, we introduce GPOcc, which transforms visual geometry priors into occupancy-aware sparse Gaussian representations for efficient and expressive volumetric scene modeling. Building on GPOcc, GPOcc++ models multi-view observations and temporal sequences within a unified framework, allowing spatial and temporal evidence to be handled through the same representation. We further extend GPOcc++ from indoor scenes to outdoor occupancy prediction. Extensive experiments on both indoor and outdoor benchmarks demonstrate consistently strong performance across both multi-view and temporal settings, together with favorable efficiency and generalization. Code will be released at https://github.com/JuIvyy/GPOcc.

Changqing Zhou, Yueru Luo, Yulan Guo +3

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · The Chinese University of Hong Kong, Shenzhen, China · The School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China +3

3D Occupancy Prediction 3D Scene Representation

Ask PDF