Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes

Changqing Zhou; Yueru Luo; Han Zhang; Zeyu Jiang; Changhao Chen

cs.CVFeb 26, 2026

Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes

Authors: Changqing Zhou, Yueru Luo, Han Zhang, Zeyu Jiang, Changhao Chen

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen

Abstract

Open-vocabulary 3D occupancy is vital for embodied agents, which need to understand complex indoor environments where semantic categories are abundant and evolve beyond fixed taxonomies. While recent work has explored open-vocabulary occupancy in outdoor driving scenarios, such methods transfer poorly indoors, where geometry is denser, layouts are more intricate, and semantics are far more fine-grained. To address these challenges, we adopt a geometry-only supervision paradigm that uses only binary occupancy labels (occupied vs free). Our framework builds upon 3D Language-Embedded Gaussians, which serve as a unified intermediate representation coupling fine-grained 3D geometry with a language-aligned semantic embedding. On the geometry side, we find that existing Gaussian-to-Occupancy operators fail to converge under such weak supervision, and we introduce an opacity-aware, Poisson-based approach that stabilizes volumetric aggregation. On the semantic side, direct alignment between rendered features and open-vocabulary segmentation features suffers from feature mixing; we therefore propose a Progressive Temperature Decay schedule that gradually sharpens opacities during splatting, strengthening Gaussian-language alignment. On Occ-ScanNet, our framework achieves 59.50 IoU and 21.05 mIoU in the open-vocabulary setting, surpassing all existing occupancy methods in IoU and outperforming prior open-vocabulary approaches by a large margin in mIoU. Code will be released at https://github.com/JuIvyy/LegoOcc.

Figures & tables

Figure 1: Closed- vs. open-vocabulary occupancy. Prior methods [ 50 , 47 ] trained under a closed vocabulary can label only the categories predefined at training time, which restricts real-world deployment. Our open-vocabulary approach aligns language with 3D occupancy and supports text queries for arbitrary categories. Right column (Random Class): text-conditioned per-voxel scores are visualized as heatmaps; darker red indicates higher likelihood for the queried category.

Figure 2: LegoOcc Framework Overview. From a monocular image, a feed-forward Gaussian model produces Language-Embedded Gaussians . Training proceeds along two paths: Semantic learning , we differentiably render Gaussian features to the image with Progressive Temperature Decay and align them to a training-free open-vocabulary segmenter via a cosine objective $L_{\text{feat}}$ ; Geometry learning , we convert Gaussians to occupancy using an opacity-aware Poisson-based Gaussian-to-Occupancy operator and supervise binary occupancy with $L_{\text{occ}}$ . At inference, the Language-Embedded Occupancy supports text-driven queries by computing cosine similarity between voxel embeddings and prompt embeddings, yielding open-vocabulary semantic occupancy without dense voxel-level semantic labels during training.

Figure 3: Comparison of temperature schedules. Linear decay decreases

\tau

uniformly, whereas our exponential schedule rapidly approaches

T_{\min}

, allocating more iterations for the model to adapt to the low-temperature regime.

Method	IoU	$\blacksquare$ ceiling	$\blacksquare$ floor	$\blacksquare$ wall	$\blacksquare$ window	$\blacksquare$ chair	$\blacksquare$ bed	$\blacksquare$ sofa	$\blacksquare$ table	$\blacksquare$ tvs	$\blacksquare$ furniture	$\blacksquare$ objects	mIoU
Closed-vocabulary (full annotation)
TPVFormer [ 10 ]	33.39	6.96	32.97	14.41	9.10	24.01	41.49	45.44	28.61	10.66	35.37	25.31	24.94
GaussianFormer [ 11 ]	40.91	20.70	42.00	23.40	17.40	27.0	44.30	44.80	32.70	15.30	36.70	25.00	29.93
MonoScene [ 4 ]	41.60	15.17	44.71	22.41	12.55	26.11	27.03	35.91	28.32	6.57	32.16	19.84	24.62
ISO [ 50 ]	42.16	19.88	41.88	22.37	16.98	29.09	42.43	42.00	29.60	10.62	36.36	24.61	28.71
Surroundocc [ 45 ]	42.52	18.90	49.30	24.80	18.00	26.80	42.00	44.10	32.90	18.60	36.80	26.90	30.83

Table 1: Monocular results on Occ-ScanNet. † We re-implement POP-3D [ 41 ] and LOcc [ 53 ] for the indoor monocular setting, and supervise geometry with ground-truth binary occupancy instead of the authors’ LiDAR-derived pseudo labels. We also adopt DepthAnything-v2 [ 48 ] as the backbone and use the same input resolution and training recipe as ours for fair comparison. For POP-3D, we project voxel centers to the image plane and sample language-aligned features at those locations to form a 3D grid of text-aligned embeddings for open-vocabulary reasoning. For LOcc, we prompt Qwen2.5-VL-7B [ 2 ] to extract object names and use Trident [ 35 ] to obtain training-free open-vocabulary segmentations, which yield stronger 2D masks on Occ-ScanNet.

Operation	mIoU	IoU
GaussianFormer2	51.88	56.96
Bernoulli	51.68	58.06
Poisson	52.80	59.24
GaussianFormer2	0.00	0.00
Bernoulli	17.25	46.65
Poisson	21.05	59.50

Table 2: Ablation on the Gaussian-to-Occupancy operator. We compare three aggregation rules: GaussianFormer2, Bernoulli, and Poisson, under both closed- and open- vocabulary settings.

$T_{min}$	$T_{max}$	$\tau_{\text{test}}$	scheduler	mIoU	IoU
1.0	1.0	1	$\times$	18.15	59.19
1.0	1.0	1e-3	$\times$	12.83	32.52
1e-3	1e-3	1e-3	$\times$	0.00	0.00
1e-3	1.0	1e-3	linear	2.30	7.60
1e-2	1.0	1e-2	exp	20.85	59.25
1e-3	1.0	1e-3	exp	21.05	59.50

Table 3: Ablation on temperature scheduling. We vary the temperature range

(T_{\min},T_{\max})

, the test-time temperature

\tau_{\text{test}}

, and the scheduling strategy. “

\times

” denotes no schedule.

Model	mIoU	IoU	FPS	#parameters
ISO [ 50 ]	28.71	42.16	3.81	303.05M
EmbodiedOcc [ 47 ]	45.15	53.55	11.48	231.45M
POP-3D [ 41 ]	5.96	35.32	10.21	115.31M
LOcc [ 53 ]	9.25	36.70	8.93	113.50M
Ours	21.05	59.50	22.47	99.04M

Table 4: Model profile. FPS is measured on the same machine with a single RTX 4090 GPU, averaged over 1,000 runs after 100 warm-up runs.

Figure 4: Qualitative results on Occ-ScanNet. From top to bottom: (a) input images; (b) ground-truth semantic occupancy; (c) results from our re-implemented LOcc [ 53 ] ; (d) our method. Both (c) and (d) are trained with geometry-only annotations and evaluated on the closed-vocabulary annotation of Occ-ScanNet.

Figure 5: Open-vocabulary qualitative results. Legends list the VLM-extracted object nouns used as text queries. (a) Input image. (b) Open-vocabulary 2D segmentation for queried nouns. (c) Our 3D open-vocabulary occupancy colored by the same categories.

Explore similar work

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction
Apr 30, 2026Zeyu Jiang, Changqing Zhou, Xingxing Zuo +1Semantic Occupancy Mapping 3D Occupancy Prediction
The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI
O3N: Omnidirectional Open-Vocabulary Occupancy Prediction for Embodied Intelligent Robotics
Mar 12, 2026Mengfei Duan, Hao Shi, Fei Teng +43D Occupancy Prediction 3D Scene Understanding
School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha, China · Ant Group, Hangzhou, China · State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University, Hangzhou, China
GPOcc++: Unified Sparse Gaussian Occupancy Prediction with Visual Geometry Priors
Jul 15, 2026Changqing Zhou, Yueru Luo, Yulan Guo +33D Occupancy Prediction 3D Scene Representation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · The Chinese University of Hong Kong, Shenzhen, China · The School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China +3

Apr 30, 2026J/K move · Enter open · S save

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction

Zeyu Jiang, Changqing Zhou, Xingxing Zuo, Changhao Chen

The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI

Semantic Occupancy Mapping 3D Occupancy Prediction

Open paper AskPDF arXiv

Existing learning-based occupancy prediction methods rely on large-scale 3D annotations and generalize poorly across environments. We present FreeOcc, a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences. Unlike prior approaches that require voxel-level supervision and ground-truth camera poses, FreeOcc operates without 3D annotations, pose ground truth, or any learning stage. FreeOcc incrementally builds a globally consistent occupancy map via a four-layer pipeline: a SLAM backbone estimates poses and sparse geometry; a geometrically consistent Gaussian update constructs dense 3D Gaussian maps; open-vocabulary semantics from off-the-shelf vision-language models are associated with Gaussian primitives; and a probabilistic Gaussian-to-occupancy projection produces dense voxel occupancy. Despite being entirely training-free and pose-agnostic, FreeOcc achieves over

2\times

improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods. We further introduce ReplicaOcc, a benchmark for indoor open-vocabulary occupancy prediction, and show that FreeOcc transfers zero-shot to novel environments, substantially outperforming both supervised and self-supervised baselines. Project page: https://the-masses.github.io/freeocc-web/.

Abstract

Explore similar work

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction

O3N: Omnidirectional Open-Vocabulary Occupancy Prediction for Embodied Intelligent Robotics

GPOcc++: Unified Sparse Gaussian Occupancy Prediction with Visual Geometry Priors