OVM-Occ: Open-Vocabulary Monocular Occupancy Prediction via Geometry-Aware Vision-Language Distillation

Shuyang Yu; Haibo Qin; Shaohan Hu; Wei Shan; Xiaoqing Guan; You Wang; Guang Li

cs.CVOct 26, 2026

OVM-Occ: Open-Vocabulary Monocular Occupancy Prediction via Geometry-Aware Vision-Language Distillation

Authors: Shuyang Yu, Haibo Qin, Shaohan Hu, Wei Shan, Xiaoqing Guan, You Wang, Guang Li

Organizations: Zhejiang University. · AgiBot.

Abstract

Monocular occupancy prediction in indoor scenes aims to recover a scene's 3D geometry and semantics from a single RGB image. Existing methods are restricted to a closed set of predefined categories, limiting their ability to generalize to unseen categories. This limitation is particularly critical in indoor scenes, where objects exhibit highly diverse and fine-grained semantic categories. We study open-vocabulary monocular occupancy prediction for indoor scenes and present OVM-Occ, a Gaussian-based occupancy framework with an instance-aware vision-language teacher. Specifically, we adopt Language-Embedded Gaussians as the intermediate scene representation, propose a geometry-aware deformable attention module to refine Gaussian semantic features, and construct instance-level vision-language embeddings to provide reliable semantic supervision. We further introduce geometry-aware decoupled semantic distillation to mitigate the interference between geometry learning and semantic learning. To evaluate open-vocabulary occupancy prediction performance, we introduce OpenOcc-ScanNet and OpenOcc-Replica, two extended benchmarks with more diverse and fine-grained semantic categories. Experiments show that our method achieves state-of-the-art performance in open-vocabulary monocular occupancy prediction for indoor scenes, outperforming the previous state-of-the-art approach by 12.2 mIoU and 0.6 IoU on the Occ-ScanNet dataset, with consistent gains under new open-vocabulary evaluation protocols.

Figures & tables

Panel of Figure 1 — Fig. 1: Illustration of open-vocabulary occupancy prediction and its potential downstream applications. (a) Given a monocular image and arbitrary semantic text queries, open-vocabulary occupancy prediction infers a query-conditioned 3D occupancy representation. (b) The predicted 3D occupancy representation can be further projected into a semantic BEV map. (c) The representation supports language-guided navigation, such as planning a path toward targets specified by language instructions, e.g., ”throw away the rubbish”.

Fig. 2: Overview of OVM-Occ. Given a monocular RGB image, the Gaussian-based model predicts Language-Embedded Gaussians (LE-Gaussians) and refines their features using geometry-aware deformable attention. During training, the LE-Gaussians are rendered into camera views for 2D semantic and depth supervision, while being aggregated into voxel representations for 3D geometric occupancy supervision. At inference, the LE-Gaussians are aggregated into Language-Embedded Occupancy (LE-Occ), where the occupancy features are matched with query text embeddings via cosine similarity to produce open-vocabulary occupancy predictions.

Fig. 3: Comparison of different teacher embeddings and their corresponding open-vocabulary semantic segmentations. Top: PCA visualization of teacher embeddings. Each method is visualized independently; therefore, colors are only comparable within each visualization. Bottom: Corresponding open-vocabulary segmentation results. The color legend is provided in Table I .

Fig. 4: Construction of the instance-aware vision-language teacher. A large vision-language model (LVLM) extracts semantic categories from the input image, which are then fed into an open-vocabulary instance segmentation model (OV-ISM) to generate instance masks and into a text encoder to produce text embeddings. In parallel, a visual encoder extracts dense visual embeddings from the input image. The text and visual embeddings are fused, guided by the instance masks, to obtain instance-aware teacher embeddings.

Closed-vocabulary: supervised by full annotations.
Method	IoU	Ceiling	Floor	Wall	Window	Chair	Bed	Sofa	Table	TVs	Furniture	Objects	mIoU
TPVFormer [ 6 ]	33.39	6.96	32.97	14.41	9.10	24.01	41.49	45.44	28.61	10.66	35.37	25.31	24.94
GaussianFormer [ 8 ]	40.91	20.70	42.00	23.40	17.40	27.00	44.30	44.80	32.70	15.30	36.70	25.00	29.93
MonoScene [ 12 ]	41.60	15.17	44.71	22.41	12.55	26.11	27.03	35.91	28.32	6.57	32.16	19.84	24.62
ISO [ 13 ]	42.16	19.88	41.88	22.37	16.98	29.09	42.43	42.00	29.60	10.62	36.36	24.61	28.71
SurroundOcc [ 7 ]	42.52	18.90	49.30	24.80	18.00	26.80	42.00	44.10	32.90	18.60	36.80	26.90	30.83

TABLE I: Comparison of monocular occupancy prediction performance on the Occ-ScanNet benchmark. Closed-vocabulary methods are trained with both semantic and occupancy ground-truth annotations, whereas open-vocabulary methods use only occupancy ground truth. The best result among open-vocabulary methods is highlighted in bold.

Fig. 5: Qualitative comparison of the results from LegoOcc and our proposed OVM-Occ on the Occ-ScanNet benchmark. The color legend is provided in Table I .

		per-class IoU										mIoU
Method	IoU	Cabinet	Counter	Pillow	Books	Towel	Mirror	Fridge	Toilet	Sink	Bathtub	11-class	20-class	40-class
POP-3D [ 20 ]	48.94	0.28	4.54	0.11	0.00	9.28	0.00	7.97	9.79	11.02	1.07	13.22	11.37	6.19
LOcc [ 22 ]	44.70	17.30	26.17	0.00	0.00	0.00	0.00	17.39	14.25	9.84	37.87	18.89	21.86	12.91
LegoOcc [ 23 ]	56.22	8.73	9.10	9.13	10.63	6.15	0.00	4.96	13.82	17.13	33.41	23.77	20.64	12.93
OVM-Occ (Ours)	56.82	22.77	21.44	28.82	19.68	18.62	24.06	19.56	32.06	38.18	57.38	32.40	30.83	21.79

TABLE II: Comparison of monocular occupancy prediction performance on the OpenOcc-ScanNet benchmark. Per-class IoU is reported for a subset of semantic categories under the OpenOcc-ScanNet40 protocol.

Method	From Scratch 1.1K frames	Pre-trained 100 frames	Pre-trained 1.1K frames
LegoOcc	62.85 / 9.27	47.76 / 8.12	73.65 / 14.90
Ours	64.21 / 14.62	50.09 / 14.44	77.88 / 23.28

TABLE III: Open-vocabulary occupancy prediction results on OpenOcc-Replica35 under different training settings. Each entry reports IoU / mIoU.

Fig. 6: Qualitative results on the OpenOcc-ScanNet and OpenOcc-Replica benchmarks with more diverse semantic categories.

Student	Teacher	IoU	mIoU
LegoOcc	Trident Visual Embedding [ 31 ]	59.50	21.05
LegoOcc	Instance-aware VL Embedding	59.71	29.50
Ours	Trident Visual Embedding [ 31 ]	60.07	24.54
Ours	Instance-aware VL Embedding	60.10	33.25

TABLE IV: Ablation of different teacher embeddings and student frameworks. VL denotes vision-language.

Teacher Construction	IoU	mIoU
Pixel-level Vision	60.20	28.16
Category-level Language	60.09	31.28
Instance-aware Vision-Language	60.10	33.25

TABLE V: Ablation of teacher construction.

Refinement Module	IoU	mIoU
None	59.57	31.98
Deformable Attention	60.02	32.43
Geometry-aware Deformable Attention	60.10	33.25

TABLE VI: Ablation of feature refinement modules.

Spatial SG	Opacity SG	Depth Mask	IoU	mIoU
			58.38	31.11
✓			58.30	32.12
	✓		59.39	31.55
✓	✓		59.86	32.51
✓	✓	✓	60.10	33.25

TABLE VII: Ablation of geometry-semantic decoupling strategies. Spatial SG and Opacity SG denote stop-gradient operations on Gaussian geometry attributes (mean, rotation, and scale) and opacity, respectively.

$N$	#Gauss.	Mem. (GB)	Time (ms)	IoU	mIoU
1	4,144	1.59	32.93	58.99	31.26
2	8,288	1.65	47.50	59.77	32.11
4	16,576	2.22	65.12	59.99	32.59
8	33,152	4.00	81.26	60.10	33.25
16	66,304	7.56	108.84	59.76	32.84

TABLE VIII: Runtime and memory analysis under different numbers of sampled Gaussians

N

per ray.

3D Occupancy Prediction

3D occupancy prediction estimates the geometry and semantics of occupied and free space around an agent from camera, LiDAR, or radar data.

See all 113 papers

Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes
Feb 26, 2026Changqing Zhou, Yueru Luo, Han Zhang +23D Occupancy Prediction Semantic Occupancy Mapping
The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen
LegoOcc++: Towards Consistent Open-Vocabulary 3D Occupancy Prediction
Oct 25, 2026Changqing Zhou, Yueru Luo, Han Zhang +23D Occupancy Prediction 3D Gaussian Splatting
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · The Chinese University of Hong Kong, Shenzhen, China · Division of Emerging Interdisciplinary Areas, The Hong Kong University of Science and Technology, Hong Kong SAR 999077, China
FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction
Apr 30, 2026Zeyu Jiang, Changqing Zhou, Xingxing Zuo +1Semantic Occupancy Mapping 3D Occupancy Prediction
The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI

Feb 26, 2026J/K move · Enter open · S save

Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes

Changqing Zhou, Yueru Luo, Han Zhang, Zeyu Jiang+1

The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen

3D Occupancy Prediction Semantic Occupancy Mapping Language-Embedded 3DGS

Open paper AskPDF arXiv

Open-vocabulary 3D occupancy is vital for embodied agents, which need to understand complex indoor environments where semantic categories are abundant and evolve beyond fixed taxonomies. While recent work has explored open-vocabulary occupancy in outdoor driving scenarios, such methods transfer poorly indoors, where geometry is denser, layouts are more intricate, and semantics are far more fine-grained. To address these challenges, we adopt a geometry-only supervision paradigm that uses only binary occupancy labels (occupied vs free). Our framework builds upon 3D Language-Embedded Gaussians, which serve as a unified intermediate representation coupling fine-grained 3D geometry with a language-aligned semantic embedding. On the geometry side, we find that existing Gaussian-to-Occupancy operators fail to converge under such weak supervision, and we introduce an opacity-aware, Poisson-based approach that stabilizes volumetric aggregation. On the semantic side, direct alignment between rendered features and open-vocabulary segmentation features suffers from feature mixing; we therefore propose a Progressive Temperature Decay schedule that gradually sharpens opacities during splatting, strengthening Gaussian-language alignment. On Occ-ScanNet, our framework achieves 59.50 IoU and 21.05 mIoU in the open-vocabulary setting, surpassing all existing occupancy methods in IoU and outperforming prior open-vocabulary approaches by a large margin in mIoU. Code will be released at https://github.com/JuIvyy/LegoOcc.

Abstract

3D Occupancy Prediction

Explore similar work

Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes

LegoOcc++: Towards Consistent Open-Vocabulary 3D Occupancy Prediction

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction