Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · The Chinese University of Hong Kong, Shenzhen, China · Division of Emerging Interdisciplinary Areas, The Hong Kong University of Science and Technology, Hong Kong SAR 999077, China
Open-vocabulary 3D occupancy prediction is important for embodied agents operating in complex environments whose semantic categories extend beyond fixed taxonomies. While recent studies have explored open-vocabulary occupancy in outdoor driving scenarios, indoor environments remain challenging due to dense occlusions, complex geometry, and fine-grained long-tailed semantics. To reduce the need for expensive voxel-level semantic annotations, we study a geometry-only supervision setting requiring only binary occupancy labels. We build upon Language-Embedded Gaussians as a unified representation coupling 3D geometry with language-aligned semantics. For geometry learning, we introduce an opacity-aware Poisson-based Gaussian-to-Occupancy formulation that enables stable volumetric aggregation under binary occupancy supervision. For semantic learning, we render Gaussian features into images and align them with a frozen open-vocabulary model, together with Progressive Temperature Decay to reduce feature mixing during Gaussian splatting. Building on this framework, we further introduce Instance-Level Semantic Consistency to alleviate fragmented object predictions, and Covisibility-Aware Support-View Selection to provide informative and less redundant multi-view semantic supervision. We also extend our framework to outdoor driving scenarios to evaluate its generality across substantially different environments. Extensive experiments on indoor and outdoor benchmarks demonstrate that our method achieves strong open-vocabulary occupancy performance, validating the effectiveness and generality of our framework. Code will be available at https://github.com/JuIvyy/LegoOcc.
Figures & tables
Fig. 1: Closed- vs. open-vocabulary occupancy. Conventional closed-vocabulary methods [ 5 , 6 ] can recognize only categories predefined during training. In contrast, our open-vocabulary formulation aligns language with 3D occupancy and supports text queries for arbitrary categories. Right column (Random Class): text-conditioned voxel scores are visualized as heatmaps, where darker red indicates a higher response to the queried category.
Fig. 2: Framework Overview. From a monocular image, a feed-forward Gaussian model produces Language-Embedded Gaussians . Training proceeds along two paths: Semantic learning , we differentiably render Gaussian features to the image with Progressive Temperature Decay and align them to a training-free open-vocabulary segmenter via a cosine objective Lfeat ; Geometry learning , we convert Gaussians to occupancy using an opacity-aware Poisson-based Gaussian-to-Occupancy operator and supervise binary occupancy with Locc . At inference, the Language-Embedded Occupancy supports text-driven queries by computing cosine similarity between voxel embeddings and prompt embeddings, yielding open-vocabulary semantic occupancy without dense voxel-level semantic labels during training.
Fig. 3: Comparison of temperature schedules. Linear decay decreases τ uniformly, whereas our exponential schedule rapidly approaches Tmin , allocating more iterations for the model to adapt to the low-temperature regime.
Fig. 4: Instance-level semantic consistency. Given an input image (left), a frozen open-vocabulary teacher (e.g., Trident [ 36 ] ) provides semantic cues in image space (middle), which can be spatially fragmented and incomplete for the same object. In contrast, SAM produces category-agnostic instance masks (right) that more coherently capture complete object extents. We aggregate teacher features within each reliable SAM mask to form an instance-level semantic prototype, and encourage rendered LE-Gaussian features inside the same instance to align with this shared prototype.
Fig. 5: Covisibility-aware support-view selection. Each example shows a reference view (left) and four support views. Fixed temporal selection often produces highly redundant observations (top). Our method filters views with either insufficient overlap or near-duplicate content according to geometric covisibility, and selects pose-diverse views from the remaining candidates (bottom), providing complementary observations while preserving shared scene geometry.
Method
IoU
■ ceiling
■ floor
■ wall
■ window
■ chair
■ bed
■ sofa
■ table
■ tvs
■ furniture
■ objects
mIoU
Closed-vocabulary (full annotation)
TPVFormer [ 72 ]
33.39
6.96
32.97
14.41
9.10
24.01
41.49
45.44
28.61
10.66
35.37
25.31
24.94
GaussianFormer [ 16 ]
40.91
20.70
42.00
23.40
17.40
27.0
44.30
44.80
32.70
15.30
36.70
25.00
29.93
MonoScene [ 44 ]
41.60
15.17
44.71
22.41
12.55
26.11
27.03
35.91
28.32
6.57
32.16
19.84
24.62
ISO [ 5 ]
42.16
19.88
41.88
22.37
16.98
29.09
42.43
42.00
29.60
10.62
36.36
24.61
28.71
Surroundocc [ 15 ]
42.52
18.90
49.30
24.80
18.00
26.80
42.00
44.10
32.90
18.60
36.80
26.90
30.83
TABLE I: Monocular results on Occ-ScanNet. † We re-implement POP-3D [ 58 ] and LOcc [ 23 ] for the indoor monocular setting, and supervise geometry with ground-truth binary occupancy instead of the authors’ LiDAR-derived pseudo labels. We also adopt DepthAnything-v2 [ 70 ] as the backbone and use the same input resolution and training recipe as ours for fair comparison. For POP-3D, we project voxel centers to the image plane and sample language-aligned features at those locations to form a 3D grid of text-aligned embeddings for open-vocabulary reasoning. For LOcc, we prompt Qwen2.5-VL-7B [ 71 ] to extract object names and use Trident [ 36 ] to obtain training-free open-vocabulary segmentations.
Method
Frames
mIoU
■ barrier
■ bicycle
■ bus
■ car
■ cons. veh.
■ motorcycle
■ pedestrian
■ traffic cone
■ trailer
■ truck
■ drive. surf.
■ sidewalk
■ terrain
■ manmade
■ vegetation
SelfOcc [ 60 ]
1
10.54
0.15
0.66
5.46
12.54
0.00
0.80
2.10
0.00
0.00
8.25
55.49
26.30
26.54
14.22
5.60
OccNeRF [ 57 ]
1
10.81
0.83
0.82
5.13
12.49
3.50
0.23
3.10
1.84
0.52
3.90
52.62
20.81
24.75
18.45
13.19
GaussianOcc [ 73 ]
1
11.26
1.79
5.82
14.58
13.55
1.30
2.82
7.95
9.76
0.56
9.61
44.59
20.10
17.58
8.61
10.29
LangOcc [ 24 ]
2
12.04
2.70
7.20
5.80
13.90
0.50
10.80
6.40
8.70
3.20
11.00
42.10
12.50
27.20
14.10
14.50
GaussTR [ 74 ]
1
13.25
2.09
5.22
14.07
20.43
5.70
7.08
5.12
3.93
0.92
13.36
39.44
15.68
22.89
21.17
21.87
PGOcc [ 41 ]
8
15.15
4.33
7.31
23.63
26.42
11.38
6.33
2.74
5.79
3.07
17.29
37.81
19.29
20.85
19.02
21.92
TABLE II: Quantitative performance of 3D occupancy methods on the Occ3D-nuScenes dataset. We compare methods that do not rely on LiDAR supervision. Frames denotes the number of temporal frames required at inference, and methods requiring more than one frame are shown in gray. IoU scores for “others” and “other flat” classes are consistently zero and thus omitted. “Cons. veh.” denotes construction vehicles, and “drive. surf.” denotes drivable surfaces.
Fig. 6: Qualitative results on Occ-ScanNet. From top to bottom: (a) input images; (b) ground-truth semantic occupancy; (c) results from our re-implemented LOcc [ 23 ] ; (d) LegoOcc; and (e) LegoOcc++. All compared open-vocabulary methods are trained with geometry-only supervision and evaluated using the closed-vocabulary annotations of Occ-ScanNet.
Fig. 7: Open-vocabulary qualitative comparison. Legends list the VLM-extracted object nouns used as text queries. (a) Input image. (b) Open-vocabulary 2D segmentation for the queried nouns. (c) LegoOcc 3D open-vocabulary occupancy, colored by the same categories. (d) LegoOcc++ 3D open-vocabulary occupancy, colored by the same categories.
Fig. 8: Qualitative results on Occ3D-nuScenes. We show surround-view input images and semantic occupancy predictions from GaussTR and LegoOcc++, together with the ground truth. LegoOcc++ better preserves the overall road layout and long-range scene structure, yielding more complete occupancy predictions.
Operation
mIoU
IoU
GaussianFormer2
51.88
56.96
Bernoulli
51.68
58.06
Poisson
52.80
59.24
GaussianFormer2
0.00
0.00
Bernoulli
17.25
46.65
Poisson
21.05
59.50
TABLE III: Ablation on the Gaussian-to-Occupancy operator. We compare three aggregation rules: GaussianFormer2, Bernoulli, and Poisson, under both closed- and open- vocabulary settings.
Tmin
Tmax
τtest
scheduler
mIoU
IoU
1.0
1.0
1
×
18.15
59.19
1.0
1.0
1e-3
×
12.83
32.52
1e-3
1e-3
1e-3
×
0.00
0.00
1e-3
1.0
1e-3
linear
2.30
7.60
1e-2
1.0
1e-2
exp
20.85
59.25
1e-3
1.0
1e-3
exp
21.05
59.50
TABLE IV: Ablation on temperature scheduling. We vary the temperature range (Tmin,Tmax) , the test-time temperature τtest , and the scheduling strategy. “ × ” denotes no schedule.
Method
Covis.
SAM-Inst.
mIoU
IoU
LegoOcc
21.05
59.50
LegoOcc + Covis.
✓
23.17
60.31
LegoOcc++
✓
✓
24.22
61.73
TABLE V: Comparison of LegoOcc and LegoOcc++. Starting from LegoOcc baseline, we progressively introduce covisibility-aware support-view selection (Covis.) and SAM-based instance-level semantic consistency (SAM-Inst.).
Model
mIoU
IoU
FPS
Params
ISO [ 5 ]
28.71
42.16
3.81
303.05M
EmbodiedOcc [ 6 ]
45.15
53.55
11.48
231.45M
POP-3D [ 58 ]
5.96
35.32
10.21
115.31M
LOcc [ 23 ]
9.25
36.70
8.93
113.50M
LegoOcc
21.05
59.50
22.47
99.04M
LegoOcc++
24.22
61.73
22.47
99.04M
TABLE VI: Model profile. FPS is measured on the same machine with a single RTX 4090 GPU, averaged over 1,000 runs after 100 warm-up runs.
Model
Frames
mIoU
FPS
Params (Core / E2E)
GaussTR [ 74 ]
1
13.25
0.42
13.74M / 575.59M
PG-Occ [ 41 ]
8
15.15
0.24
90.03M / 501.97M
LegoOcc++
1
15.14
6.22
161.58M / 161.58M
TABLE VII: Model profile. FPS is measured on the same machine using a single RTX 4090 GPU.
Open-vocabulary 3D occupancy is vital for embodied agents, which need to understand complex indoor environments where semantic categories are abundant and evolve beyond fixed taxonomies. While recent work has explored open-vocabulary occupancy in outdoor driving scenarios, such methods transfer poorly indoors, where geometry is denser, layouts are more intricate, and semantics are far more fine-grained. To address these challenges, we adopt a geometry-only supervision paradigm that uses only binary occupancy labels (occupied vs free). Our framework builds upon 3D Language-Embedded Gaussians, which serve as a unified intermediate representation coupling fine-grained 3D geometry with a language-aligned semantic embedding. On the geometry side, we find that existing Gaussian-to-Occupancy operators fail to converge under such weak supervision, and we introduce an opacity-aware, Poisson-based approach that stabilizes volumetric aggregation. On the semantic side, direct alignment between rendered features and open-vocabulary segmentation features suffers from feature mixing; we therefore propose a Progressive Temperature Decay schedule that gradually sharpens opacities during splatting, strengthening Gaussian-language alignment. On Occ-ScanNet, our framework achieves 59.50 IoU and 21.05 mIoU in the open-vocabulary setting, surpassing all existing occupancy methods in IoU and outperforming prior open-vocabulary approaches by a large margin in mIoU. Code will be released at https://github.com/JuIvyy/LegoOcc.
Changqing Zhou, Yueru Luo, Han Zhang +2
The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen
Monocular occupancy prediction in indoor scenes aims to recover a scene's 3D geometry and semantics from a single RGB image. Existing methods are restricted to a closed set of predefined categories, limiting their ability to generalize to unseen categories. This limitation is particularly critical in indoor scenes, where objects exhibit highly diverse and fine-grained semantic categories. We study open-vocabulary monocular occupancy prediction for indoor scenes and present OVM-Occ, a Gaussian-based occupancy framework with an instance-aware vision-language teacher. Specifically, we adopt Language-Embedded Gaussians as the intermediate scene representation, propose a geometry-aware deformable attention module to refine Gaussian semantic features, and construct instance-level vision-language embeddings to provide reliable semantic supervision. We further introduce geometry-aware decoupled semantic distillation to mitigate the interference between geometry learning and semantic learning. To evaluate open-vocabulary occupancy prediction performance, we introduce OpenOcc-ScanNet and OpenOcc-Replica, two extended benchmarks with more diverse and fine-grained semantic categories. Experiments show that our method achieves state-of-the-art performance in open-vocabulary monocular occupancy prediction for indoor scenes, outperforming the previous state-of-the-art approach by 12.2 mIoU and 0.6 IoU on the Occ-ScanNet dataset, with consistent gains under new open-vocabulary evaluation protocols.
Existing learning-based occupancy prediction methods rely on large-scale 3D annotations and generalize poorly across environments. We present FreeOcc, a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences. Unlike prior approaches that require voxel-level supervision and ground-truth camera poses, FreeOcc operates without 3D annotations, pose ground truth, or any learning stage. FreeOcc incrementally builds a globally consistent occupancy map via a four-layer pipeline: a SLAM backbone estimates poses and sparse geometry; a geometrically consistent Gaussian update constructs dense 3D Gaussian maps; open-vocabulary semantics from off-the-shelf vision-language models are associated with Gaussian primitives; and a probabilistic Gaussian-to-occupancy projection produces dense voxel occupancy. Despite being entirely training-free and pose-agnostic, FreeOcc achieves over 2× improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods. We further introduce ReplicaOcc, a benchmark for indoor open-vocabulary occupancy prediction, and show that FreeOcc transfers zero-shot to novel environments, substantially outperforming both supervised and self-supervised baselines. Project page: https://the-masses.github.io/freeocc-web/.
Zeyu Jiang, Changqing Zhou, Xingxing Zuo +1
The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI