LegoOcc++: Towards Consistent Open-Vocabulary 3D Occupancy Prediction

Changqing Zhou; Yueru Luo; Han Zhang; Zeyu Jiang; Changhao Chen

cs.CVOct 25, 2026

LegoOcc++: Towards Consistent Open-Vocabulary 3D Occupancy Prediction

Authors: Changqing Zhou, Yueru Luo, Han Zhang, Zeyu Jiang, Changhao Chen

Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · The Chinese University of Hong Kong, Shenzhen, China · Division of Emerging Interdisciplinary Areas, The Hong Kong University of Science and Technology, Hong Kong SAR 999077, China

Abstract

Open-vocabulary 3D occupancy prediction is important for embodied agents operating in complex environments whose semantic categories extend beyond fixed taxonomies. While recent studies have explored open-vocabulary occupancy in outdoor driving scenarios, indoor environments remain challenging due to dense occlusions, complex geometry, and fine-grained long-tailed semantics. To reduce the need for expensive voxel-level semantic annotations, we study a geometry-only supervision setting requiring only binary occupancy labels. We build upon Language-Embedded Gaussians as a unified representation coupling 3D geometry with language-aligned semantics. For geometry learning, we introduce an opacity-aware Poisson-based Gaussian-to-Occupancy formulation that enables stable volumetric aggregation under binary occupancy supervision. For semantic learning, we render Gaussian features into images and align them with a frozen open-vocabulary model, together with Progressive Temperature Decay to reduce feature mixing during Gaussian splatting. Building on this framework, we further introduce Instance-Level Semantic Consistency to alleviate fragmented object predictions, and Covisibility-Aware Support-View Selection to provide informative and less redundant multi-view semantic supervision. We also extend our framework to outdoor driving scenarios to evaluate its generality across substantially different environments. Extensive experiments on indoor and outdoor benchmarks demonstrate that our method achieves strong open-vocabulary occupancy performance, validating the effectiveness and generality of our framework. Code will be available at https://github.com/JuIvyy/LegoOcc.

Figures & tables

Fig. 1: Closed- vs. open-vocabulary occupancy. Conventional closed-vocabulary methods [ 5 , 6 ] can recognize only categories predefined during training. In contrast, our open-vocabulary formulation aligns language with 3D occupancy and supports text queries for arbitrary categories. Right column (Random Class): text-conditioned voxel scores are visualized as heatmaps, where darker red indicates a higher response to the queried category.

Fig. 2: Framework Overview. From a monocular image, a feed-forward Gaussian model produces Language-Embedded Gaussians . Training proceeds along two paths: Semantic learning , we differentiably render Gaussian features to the image with Progressive Temperature Decay and align them to a training-free open-vocabulary segmenter via a cosine objective $L_{\text{feat}}$ ; Geometry learning , we convert Gaussians to occupancy using an opacity-aware Poisson-based Gaussian-to-Occupancy operator and supervise binary occupancy with $L_{\text{occ}}$ . At inference, the Language-Embedded Occupancy supports text-driven queries by computing cosine similarity between voxel embeddings and prompt embeddings, yielding open-vocabulary semantic occupancy without dense voxel-level semantic labels during training.

Fig. 3: Comparison of temperature schedules. Linear decay decreases

\tau

uniformly, whereas our exponential schedule rapidly approaches

T_{\min}

, allocating more iterations for the model to adapt to the low-temperature regime.

Fig. 4: Instance-level semantic consistency. Given an input image (left), a frozen open-vocabulary teacher (e.g., Trident [ 36 ] ) provides semantic cues in image space (middle), which can be spatially fragmented and incomplete for the same object. In contrast, SAM produces category-agnostic instance masks (right) that more coherently capture complete object extents. We aggregate teacher features within each reliable SAM mask to form an instance-level semantic prototype, and encourage rendered LE-Gaussian features inside the same instance to align with this shared prototype.

Fig. 5: Covisibility-aware support-view selection. Each example shows a reference view (left) and four support views. Fixed temporal selection often produces highly redundant observations (top). Our method filters views with either insufficient overlap or near-duplicate content according to geometric covisibility, and selects pose-diverse views from the remaining candidates (bottom), providing complementary observations while preserving shared scene geometry.

Method	IoU	$\blacksquare$ ceiling	$\blacksquare$ floor	$\blacksquare$ wall	$\blacksquare$ window	$\blacksquare$ chair	$\blacksquare$ bed	$\blacksquare$ sofa	$\blacksquare$ table	$\blacksquare$ tvs	$\blacksquare$ furniture	$\blacksquare$ objects	mIoU
Closed-vocabulary (full annotation)
TPVFormer [ 72 ]	33.39	6.96	32.97	14.41	9.10	24.01	41.49	45.44	28.61	10.66	35.37	25.31	24.94
GaussianFormer [ 16 ]	40.91	20.70	42.00	23.40	17.40	27.0	44.30	44.80	32.70	15.30	36.70	25.00	29.93
MonoScene [ 44 ]	41.60	15.17	44.71	22.41	12.55	26.11	27.03	35.91	28.32	6.57	32.16	19.84	24.62
ISO [ 5 ]	42.16	19.88	41.88	22.37	16.98	29.09	42.43	42.00	29.60	10.62	36.36	24.61	28.71
Surroundocc [ 15 ]	42.52	18.90	49.30	24.80	18.00	26.80	42.00	44.10	32.90	18.60	36.80	26.90	30.83

TABLE I: Monocular results on Occ-ScanNet. † We re-implement POP-3D [ 58 ] and LOcc [ 23 ] for the indoor monocular setting, and supervise geometry with ground-truth binary occupancy instead of the authors’ LiDAR-derived pseudo labels. We also adopt DepthAnything-v2 [ 70 ] as the backbone and use the same input resolution and training recipe as ours for fair comparison. For POP-3D, we project voxel centers to the image plane and sample language-aligned features at those locations to form a 3D grid of text-aligned embeddings for open-vocabulary reasoning. For LOcc, we prompt Qwen2.5-VL-7B [ 71 ] to extract object names and use Trident [ 36 ] to obtain training-free open-vocabulary segmentations.

Method	Frames	mIoU	$\blacksquare$ barrier	$\blacksquare$ bicycle	$\blacksquare$ bus	$\blacksquare$ car	$\blacksquare$ cons. veh.	$\blacksquare$ motorcycle	$\blacksquare$ pedestrian	$\blacksquare$ traffic cone	$\blacksquare$ trailer	$\blacksquare$ truck	$\blacksquare$ drive. surf.	$\blacksquare$ sidewalk	$\blacksquare$ terrain	$\blacksquare$ manmade	$\blacksquare$ vegetation
SelfOcc [ 60 ]	1	10.54	0.15	0.66	5.46	12.54	0.00	0.80	2.10	0.00	0.00	8.25	55.49	26.30	26.54	14.22	5.60
OccNeRF [ 57 ]	1	10.81	0.83	0.82	5.13	12.49	3.50	0.23	3.10	1.84	0.52	3.90	52.62	20.81	24.75	18.45	13.19
GaussianOcc [ 73 ]	1	11.26	1.79	5.82	14.58	13.55	1.30	2.82	7.95	9.76	0.56	9.61	44.59	20.10	17.58	8.61	10.29
LangOcc [ 24 ]	2	12.04	2.70	7.20	5.80	13.90	0.50	10.80	6.40	8.70	3.20	11.00	42.10	12.50	27.20	14.10	14.50
GaussTR [ 74 ]	1	13.25	2.09	5.22	14.07	20.43	5.70	7.08	5.12	3.93	0.92	13.36	39.44	15.68	22.89	21.17	21.87
PGOcc [ 41 ]	8	15.15	4.33	7.31	23.63	26.42	11.38	6.33	2.74	5.79	3.07	17.29	37.81	19.29	20.85	19.02	21.92

TABLE II: Quantitative performance of 3D occupancy methods on the Occ3D-nuScenes dataset. We compare methods that do not rely on LiDAR supervision. Frames denotes the number of temporal frames required at inference, and methods requiring more than one frame are shown in gray. IoU scores for “others” and “other flat” classes are consistently zero and thus omitted. “Cons. veh.” denotes construction vehicles, and “drive. surf.” denotes drivable surfaces.

Fig. 6: Qualitative results on Occ-ScanNet. From top to bottom: (a) input images; (b) ground-truth semantic occupancy; (c) results from our re-implemented LOcc [ 23 ] ; (d) LegoOcc; and (e) LegoOcc++. All compared open-vocabulary methods are trained with geometry-only supervision and evaluated using the closed-vocabulary annotations of Occ-ScanNet.

Fig. 7: Open-vocabulary qualitative comparison. Legends list the VLM-extracted object nouns used as text queries. (a) Input image. (b) Open-vocabulary 2D segmentation for the queried nouns. (c) LegoOcc 3D open-vocabulary occupancy, colored by the same categories. (d) LegoOcc++ 3D open-vocabulary occupancy, colored by the same categories.

Fig. 8: Qualitative results on Occ3D-nuScenes. We show surround-view input images and semantic occupancy predictions from GaussTR and LegoOcc++, together with the ground truth. LegoOcc++ better preserves the overall road layout and long-range scene structure, yielding more complete occupancy predictions.

Operation	mIoU	IoU
GaussianFormer2	51.88	56.96
Bernoulli	51.68	58.06
Poisson	52.80	59.24
GaussianFormer2	0.00	0.00
Bernoulli	17.25	46.65
Poisson	21.05	59.50

TABLE III: Ablation on the Gaussian-to-Occupancy operator. We compare three aggregation rules: GaussianFormer2, Bernoulli, and Poisson, under both closed- and open- vocabulary settings.

$T_{min}$	$T_{max}$	$\tau_{\text{test}}$	scheduler	mIoU	IoU
1.0	1.0	1	$\times$	18.15	59.19
1.0	1.0	1e-3	$\times$	12.83	32.52
1e-3	1e-3	1e-3	$\times$	0.00	0.00
1e-3	1.0	1e-3	linear	2.30	7.60
1e-2	1.0	1e-2	exp	20.85	59.25
1e-3	1.0	1e-3	exp	21.05	59.50

TABLE IV: Ablation on temperature scheduling. We vary the temperature range

(T_{\min},T_{\max})

, the test-time temperature

\tau_{\text{test}}

, and the scheduling strategy. “

\times

” denotes no schedule.

Method	Covis.	SAM-Inst.	mIoU	IoU
LegoOcc			21.05	59.50
LegoOcc + Covis.	$\checkmark$		23.17	60.31
LegoOcc++	$\checkmark$	$\checkmark$	24.22	61.73

TABLE V: Comparison of LegoOcc and LegoOcc++. Starting from LegoOcc baseline, we progressively introduce covisibility-aware support-view selection (Covis.) and SAM-based instance-level semantic consistency (SAM-Inst.).

Model	mIoU	IoU	FPS	Params
ISO [ 5 ]	28.71	42.16	3.81	303.05M
EmbodiedOcc [ 6 ]	45.15	53.55	11.48	231.45M
POP-3D [ 58 ]	5.96	35.32	10.21	115.31M
LOcc [ 23 ]	9.25	36.70	8.93	113.50M
LegoOcc	21.05	59.50	22.47	99.04M
LegoOcc++	24.22	61.73	22.47	99.04M

TABLE VI: Model profile. FPS is measured on the same machine with a single RTX 4090 GPU, averaged over 1,000 runs after 100 warm-up runs.

Model	Frames	mIoU	FPS	Params (Core / E2E)
GaussTR [ 74 ]	1	13.25	0.42	13.74M / 575.59M
PG-Occ [ 41 ]	8	15.15	0.24	90.03M / 501.97M
LegoOcc++	1	15.14	6.22	161.58M / 161.58M

TABLE VII: Model profile. FPS is measured on the same machine using a single RTX 4090 GPU.

3D Occupancy Prediction

3D occupancy prediction estimates the geometry and semantics of occupied and free space around an agent from camera, LiDAR, or radar data.

See all 113 papers

Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes
Feb 26, 2026Changqing Zhou, Yueru Luo, Han Zhang +23D Occupancy Prediction Semantic Occupancy Mapping
The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen
OVM-Occ: Open-Vocabulary Monocular Occupancy Prediction via Geometry-Aware Vision-Language Distillation
Oct 26, 2026Shuyang Yu, Haibo Qin, Shaohan Hu +43D Occupancy Prediction 3D Scene Understanding
Zhejiang University. · AgiBot.
FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction
Apr 30, 2026Zeyu Jiang, Changqing Zhou, Xingxing Zuo +1Semantic Occupancy Mapping 3D Occupancy Prediction
The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI

Feb 26, 2026J/K move · Enter open · S save

Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes

Changqing Zhou, Yueru Luo, Han Zhang, Zeyu Jiang+1

The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen

3D Occupancy Prediction Semantic Occupancy Mapping Language-Embedded 3DGS

Open paper AskPDF arXiv

Open-vocabulary 3D occupancy is vital for embodied agents, which need to understand complex indoor environments where semantic categories are abundant and evolve beyond fixed taxonomies. While recent work has explored open-vocabulary occupancy in outdoor driving scenarios, such methods transfer poorly indoors, where geometry is denser, layouts are more intricate, and semantics are far more fine-grained. To address these challenges, we adopt a geometry-only supervision paradigm that uses only binary occupancy labels (occupied vs free). Our framework builds upon 3D Language-Embedded Gaussians, which serve as a unified intermediate representation coupling fine-grained 3D geometry with a language-aligned semantic embedding. On the geometry side, we find that existing Gaussian-to-Occupancy operators fail to converge under such weak supervision, and we introduce an opacity-aware, Poisson-based approach that stabilizes volumetric aggregation. On the semantic side, direct alignment between rendered features and open-vocabulary segmentation features suffers from feature mixing; we therefore propose a Progressive Temperature Decay schedule that gradually sharpens opacities during splatting, strengthening Gaussian-language alignment. On Occ-ScanNet, our framework achieves 59.50 IoU and 21.05 mIoU in the open-vocabulary setting, surpassing all existing occupancy methods in IoU and outperforming prior open-vocabulary approaches by a large margin in mIoU. Code will be released at https://github.com/JuIvyy/LegoOcc.

Abstract

3D Occupancy Prediction

Explore similar work

Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes

OVM-Occ: Open-Vocabulary Monocular Occupancy Prediction via Geometry-Aware Vision-Language Distillation

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction