GCSR-OCC: Geometry Calibration and Semantic Routing for Open-Vocabulary Occupancy Prediction
Organizations: Alibaba Amap
Abstract
Open-vocabulary 3D occupancy prediction represents scene geometry and semantics in a unified voxel grid, enabling recognition beyond predefined categories. One approach reconstructs scenes with 3D Gaussians and converts them into voxel occupancy. However, Gaussian scale inflation and excessive overlap can distort voxel occupancy predictions, while semantic mixing can arise within Gaussians spanning multiple categories and during 3D--2D semantic feature alignment. We propose GCSR-OCC, which combines geometry calibration and multi-mode query-conditioned semantic routing for indoor occupancy prediction from a single RGB image. The Evidence-Conserving Gaussian-to-Occupancy operator (EC-G2O) uses kernel mass normalization, -norm aggregation, and a learnable gain to suppress scale inflation and excessive local evidence accumulation while calibrating occupancy responses. Multi-mode Query-First Gaussian Semantic Routing (MM-QGSR) matches multiple semantic modes per Gaussian to text queries and aggregates query-specific evidence into voxels through a learnable text-conditioned router. Fine-grained subclass queries and smoothed semantic aggregation further refine voxel-level predictions. Without 3D semantic annotations, GCSR-OCC achieves state-of-the-art performance in open-vocabulary occupancy prediction on Occ-ScanNet, with 63.31% geometric IoU and 27.44% semantic mIoU. It also reduces geometric aggregation latency from 19.76 ms to 12.88 ms. Our code will be made publicly available.
Figures & tables
| Method | IoU | ceiling | floor | wall | window | chair | bed | sofa | table | TV | furniture | objects | mIoU |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Closed-vocabulary (full annotation) | |||||||||||||
| TPVFormer [ 13 ] | 33.39 | 6.96 | 32.97 | 14.41 | 9.10 | 24.01 | 41.49 | 45.44 | 28.61 | 10.66 | 35.37 | 25.31 | 24.94 |
| GaussianFormer [ 14 ] | 40.91 | 20.70 | 42.00 | 23.40 | 17.40 | 27.0 | 44.30 | 44.80 | 32.70 | 15.30 | 36.70 | 25.00 | 29.93 |
| MonoScene [ 2 ] | 41.60 | 15.17 | 44.71 | 22.41 | 12.55 | 26.11 | 27.03 | 35.91 | 28.32 | 6.57 | 32.16 | 19.84 | 24.62 |
| ISO [ 45 ] | 42.16 | 19.88 | 41.88 | 22.37 | 16.98 | 29.09 | 42.43 | 42.00 | 29.60 | 10.62 | 36.36 | 24.61 | 28.71 |
| SurroundOcc [ 40 ] | 42.52 | 18.90 | 49.30 | 24.80 | 18.00 | 26.80 | 42.00 | 44.10 | 32.90 | 18.60 | 36.80 | 26.90 | 30.83 |
| Method | Gaussian-voxel pairs | Aggregation latency (ms) | Full forward latency (ms) |
|---|---|---|---|
| Baseline | 19.76 | 60.16 | |
| EC-G2O | 12.88 | 50.88 |
| Configuration | IoU | mIoU |
|---|---|---|
| Poisson baseline | 59.50 | 21.05 |
| + Mass normalization | 61.46 | 22.82 |
| + -norm aggregation | 63.31 | 22.64 |
| + MM-QGSR | 63.31 | 23.79 |
| + Smooth aggregation | 63.31 | 24.73 |
| + Fine-grained subclass queries | 63.31 | 27.44 |
| Parent | Count | Retained subclass queries |
|---|---|---|
| furniture | 6 | cabinet, bookshelf, dresser, chest of drawers, ottoman, filing cabinet |
| objects | 12 | box, book, bag, lamp, bottle, cup, clothes, potted plant, trash can, basket, backpack, vase |
| Configuration | IoU | mIoU |
|---|---|---|
| LegoOcc | 59.50 | 21.05 |
| + Subclass queries | 59.50 | 24.63 |
| Ours | 63.31 | 27.44 |
| Method | Mean (m) | Median (m) |
|---|---|---|
| Poisson baseline | 0.0479 | 0.0458 |
| EC-G2O | 0.0349 | 0.0341 |
3D Occupancy Prediction
3D occupancy prediction estimates the geometry and semantics of occupied and free space around an agent from camera, LiDAR, or radar data.
See all 113 papers