3D semantic occupancy prediction is a cornerstone for embodied AI, enabling agents to perceive dense scene geometry and semantics incrementally from monocular video streams. However, current online frameworks face two critical bottlenecks: the inherent depth ambiguity of monocular estimation that causes "feature bleeding" at object boundaries , and the "cold start" instability where uninitialized temporal fusion layers distort high-quality spatial priors during early training stages. In this paper, we propose SGR-OCC (Soft-Gating and Ray-refinement Occupancy), a unified framework driven by the philosophy of "Inheritance and Evolution". To perfectly inherit monocular spatial expertise, we introduce a Soft-Gating Feature Lifter that explicitly models depth uncertainty via a Gaussian gate to probabilistically suppress background noise. Furthermore, a Dynamic Ray-Constrained Anchor Refinement module simplifies complex 3D displacement searches into efficient 1D depth corrections along camera rays, ensuring sub-voxel adherence to physical surfaces. To ensure stable evolution toward temporal consistency, we employ a Two-Phase Progressive Training Strategy equipped with identity-initialized fusion, effectively resolving the cold start problem and shielding spatial priors from noisy early gradients. Extensive experiments on the EmbodiedOcc-ScanNet and Occ-ScanNet benchmarks demonstrate that SGR-OCC achieves state-of-the-art performance. In local prediction tasks, SGR-OCC achieves a completion IoU of 58.55% and a semantic mIoU of 49.89%, surpassing the previous best method, EmbodiedOcc++, by 3.65% and 3.69% respectively. In challenging embodied prediction tasks, our model reaches 55.72% SC-IoU and 46.22% mIoU. Qualitative results further confirm our model's superior capability in preserving structural integrity and boundary sharpness in complex indoor environments.
Figures & tables
Figure 1: The overall architecture of our proposed framework. The pipeline consists of a Soft-Gating Feature Lifter for robust 2D-to-3D projection and a Dynamic Ray-Constrained Anchor Refinement module for sub-voxel geometric correction. The Semantic-Adaptive GRM (right) enforces category-specific geometric constraints. The bottom-right panel illustrates our Two-Phase Progressive Training Strategy: Stage 1 freezes the monocular backbone to initialize temporal alignment (training only the Global Head), while Stage 2 unfreezes the entire network for global co-adaptation using both Local and Global supervision.
Figure 2: Overview of the core geometric modules. (a) The Soft-Gating mechanism evaluates depth consistency ( dproj vs. dpred ) within a sampling window. It applies Gaussian weights to foreground regions while truncating conflicting background noise, ensuring robust feature aggregation. (b) The Refinement module restricts anchor optimization ( Pinit→Prefined ) to a 1D depth residual Δd strictly along the camera ray r , adhering points to the physical surface and converting a complex 3D search into an efficient 1D correction.
Method
Input
SC-IoU
■ ceiling
■ floor
■ wall
■ window
■ chair
■ bed
■ sofa
■ table
■ tvs
■ furniture
■ objects
mIoU
TPVFormer ( Huang et al. (2023) )
xrgb
33.39
6.96
32.97
14.41
9.10
24.01
41.49
45.44
28.61
10.66
35.37
25.31
24.94
GaussianFormer ( Huang et al. (2024) )
xrgb
40.91
20.70
42.00
23.40
17.40
27.0
44.30
44.80
32.70
15.30
36.70
25.00
29.93
MonoScene ( Cao and de Charette (2022) )
xrgb
41.60
15.17
44.71
22.41
12.55
26.11
27.03
35.91
28.32
6.57
32.16
19.84
24.62
ISO ( Yu et al. (2024a) )
xrgb
42.16
19.88
41.88
22.37
16.98
29.09
42.43
42.00
29.60
10.62
36.36
24.61
28.71
Surroundocc ( Wei et al. (2023) )
xrgb
42.52
18.90
49.30
24.80
18.00
26.80
42.00
44.10
32.90
18.60
36.80
26.90
30.83
EmbodiedOcc ( Park et al. (2025) )
xrgb
53.55
39.60
50.40
41.40
31.70
40.90
55.00
61.40
44.00
36.10
53.90
42.20
45.15
Table 1: Local Prediction Performance on the Occ-ScanNet dataset.
Figure 3: Qualitative comparison of local occupancy prediction on Occ-ScanNet. From left to right: Input RGB images, EmbodiedOcc Wu et al. (2025) , EmbodiedOcc++ Wang et al. (2025a) , our proposed method, and the Ground Truth. Our approach exhibits superior boundary sharpness and structural integrity, particularly in complex geometries like the sink (row 3) and table edges (row 2), thanks to the Soft-Gating mechanism and Ray-Constrained Refinement.
Method
Dataset
SC-IoU
■ ceiling
■ floor
■ wall
■ window
■ chair
■ bed
■ sofa
■ table
■ tvs
■ furniture
■ objects
mIoU
TPVFormer ( Huang et al. (2023) )
EmbodiedOcc
35.88
1.62
30.54
12.03
13.22
35.47
51.39
49.79
25.63
3.60
43.15
16.23
25.70
SurroundOcc ( Wei et al. (2023) )
EmbodiedOcc
37.04
12.70
31.80
22.50
22.00
29.90
44.70
36.50
24.60
11.50
34.40
18.20
26.27
GaussianFormer ( Huang et al. (2024) )
EmbodiedOcc
38.02
17.00
33.60
21.50
21.70
29.40
47.80
37.10
24.30
15.50
36.20
16.80
27.36
SplicingOcc
EmbodiedOcc
49.01
31.60
38.80
35.50
36.30
47.10
54.50
57.20
34.40
32.50
51.20
29.10
40.74
EmbodiedOcc ( Park et al. (2025) )
EmbodiedOcc
51.52
22.70
44.60
37.40
38.00
50.10
56.70
59.70
35.40
38.40
52.00
32.90
42.53
EmbodiedOcc++ ( Wang et al. (2025a) )
EmbodiedOcc
52.20
27.90
43.90
38.70
40.60
49.00
57.90
59.20
36.80
37.80
53.50
34.10
43.60
Table 2: Embodied prediction performance on the EmbodiedOcc-ScanNet dataset.
Figure 4: Qualitative results of embodied occupancy prediction on EmbodiedOcc-ScanNet. The rows display (top to bottom): the ground truth occupancy maps, our SGR-OCC predictions, and the underlying Gaussian Memory accumulation. The columns represent the progressive reconstruction of the global scene across a 30-frame sequence. Our model successfully integrates sequential observations into a coherent Gaussian Memory (bottom row), allowing for stable and high-fidelity global occupancy prediction (middle row) that demonstrates strong structural consistency with the ground truth.
Table 7
Figure 5: Effectiveness of the proposed SGR-OCC components. (a) Qualitative comparison of anchor refinement: Our Ray-Constrained Refinement ( R1 ) reduces geometric drift compared to the unconstrained baseline ( R3 ), forcing anchors to adhere strictly to physical surfaces (indicated by cooler colors). (b) Quantitative analysis of training stability: Compared to E2E (End-to-End) baselines, our Two-Phase Progressive Training effectively bypasses the cold start drop, leading to faster convergence and a superior mIoU of 46.22%.
Figure 6: Visualization of the Semantic-Adaptive GRM. Unlike the uniform regularization of EmbodiedOcc++ (solid red), our method generates semantic-aware weight maps. It applies strict planar constraints to structural elements like walls (red) while adaptively relaxing constraints for complex furniture (blue/grey), effectively preventing geometric over-smoothing.
Table 10
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Tolerance ( σ )
SC-IoU (%)
mIoU (%)
Boundary F1 (%)
0.1
52.54
42.74
42.10
0.2
53.63
44.80
44.30
0.5 (SGR-OCC)
58.55
49.89
48.36
1.0
56.80
47.05
46.12
2.0
54.72
45.47
45.60
Appendix
Table 7: Sensitivity Analysis of Gaussian Gating Tolerance ( σ ). We evaluate the impact of the depth tolerance parameter on overall geometric completion (SC-IoU), semantic accuracy (mIoU), and structural sharpness (Boundary F1-score). The inverted-U trend demonstrates that σ=0.5 optimally balances background noise suppression and foreground feature recall, strictly preventing "feature bleeding" at depth discontinuities.
Figure 7: Qualitative Visualization of Real Gating Weights on EmbodiedOcc-ScanNet. We extract the Bird’s Eye View (BEV) cross-sectional heatmaps of our Gaussian gating weights ( G ) during the feature lifting phase. An overly strict tolerance ( σ=0.1 ) leads to severe feature starvation, while an over-relaxed tolerance ( σ=2.0 ) acts as an all-pass filter, causing feature bleeding. Our optimal setting ( σ=0.5 ) perfectly encapsulates the physical boundaries of complex objects (e.g., the chair) while explicitly suppressing occluded background noise.
Figure 8: Pareto Frontier of Accuracy vs. Efficiency. The plot illustrates the trade-off between mIoU and inference speed for varying K . The solid blue line traces the optimal Pareto frontier, while the grey dashed line exposes a sub-optimal collapse caused by spatial over-smoothing at denser sampling rates ( K≥24 ). The red star explicitly marks K=16 as our definitive sweet spot.
Sampling Points ( K )
SC-IoU (%)
mIoU (%)
TV (%)
Objects (%)
GFLOPs
FPS
4
56.20
48.15
39.50
41.20
18.5
12.5
8
57.65
49.20
42.15
44.50
21.0
10.2
16 (SGR-OCC)
58.55
49.89
44.30
46.10
25.2
8.5
24
58.70
49.05
44.75
45.45
33.0
6.8
32
57.75
47.12
42.90
45.60
41.5
3.5
Appendix
Table 8: Ablation on Deformable Sampling Points ( K ). We evaluate the trade-off between computational efficiency (GFLOPs, FPS) and geometric granularity. While increasing K initially boosts the recall of small objects (e.g., TVs, Objects), an excessively dense sampling grid ( K≥24 ) introduces spatial over-smoothing, leading to a degradation in mIoU. Ultimately, K=16 emerges as the absolute Pareto optimum, maximizing semantic accuracy while maintaining a viable inference speed.
Depth Prior
Params
Local Task
Embodied Task
SC-IoU (%)
mIoU (%)
SC-IoU (%)
mIoU (%)
DA2 (Default)
0.09B
58.55
49.89
55.72
46.22
DA3METRIC-Large
0.35B
58.11
48.92
55.89
44.24
Appendix
Table 9: Impact of Scaling the Monocular Depth Prior. We compare our default backbone (DA2) against the fine-tuned DA3METRIC-Large (0.35B). Despite a 4× increase in parameters, the saturated indoor depth prior yields no downstream improvements and even slight degradation, highlighting the inherent depth-error tolerance and decoupling capability of our SGR-OCC architecture.
Dataset
Method
Input
SC-IoU
■ ceiling
■ floor
■ wall
■ window
■ chair
■ bed
■ sofa
■ table
■ tvs
■ furniture
■ objects
mIoU
Occ-ScanNet-mini
MonoScene Cao and de Charette (2022)
xrgb
41.90
17.00
46.20
23.90
12.70
27.00
29.10
34.80
29.10
9.70
34.50
20.40
25.90
ISO Yu et al. (2024a)
xrgb
42.90
21.10
42.70
24.60
15.10
30.80
41.00
43.30
32.20
12.10
35.90
25.10
29.40
EmbodiedOcc Wu et al. (2025)
xrgb
53.80
29.10
48.70
42.30
38.70
42.00
62.70
60.60
48.20
33.80
58.00
46.50
46.40
EmbodiedOcc++ Wang et al. (2025a)
xrgb
55.70
23.30
51.00
42.80
39.30
43.50
65.60
64.00
50.70
40.70
60.30
48.90
48.20
SGR-OCC (Ours)
xrgb
59.85
37.20
54.40
50.60
41.10
45.30
67.10
64.20
48.50
46.20
57.50
48.80
50.99
EmbodiedOcc-mini
EmbodiedOcc Wu et al. (2025)
xrgb
50.70
21.50
44.50
38.30
27.90
46.90
64.70
55.30
42.70
35.80
52.50
27.50
41.60
Appendix
Table 10: Quantitative Results on Mini-Splits. We evaluate SGR-OCC alongside representative baselines on the lightweight Occ-ScanNet-mini (Local) and EmbodiedOcc-mini (Temporal) benchmarks. While all methods experience degradation due to data scarcity, SGR-OCC exhibits the strongest resilience, maintaining a significant performance gap over the baselines.
Figure 9: Qualitative Comparison on the Occ-ScanNet Dataset. Compared to EmbodiedOcc and EmbodiedOcc++, our SGR-OCC produces significantly sharper semantic boundaries and explicitly eliminates the "feature bleeding" artifacts common in purely implicit lifting methods. The reconstructed structures (e.g., furniture, walls) exhibit high fidelity to the Ground Truth.
Figure 10: Sequential Evolution of Embodied Perception. As the agent navigates the scene (Input Image), SGR-OCC incrementally aggregates multi-view geometry into the Gaussian Memory. Empowered by our Soft-Gating mechanism, which ensures only geometrically valid features are lifted, this memory acts as a robust, noise-free spatial anchor, enabling the decoding of a highly stable and structurally accurate Global Occupancy map over time.
Figure 11: Architecture of the Language-Aware Geometric Refinement. By integrating a frozen Qwen2.5-VL-3B Bai et al. (2025) backbone, we extract both global scene context and localized attribute semantics. These multi-modal embeddings are concatenated with the visual geometric features and processed by a trainable Attribute MLP (Language-Aware Predictor), which dynamically predicts the constraint coefficient κ for the downstream 1D ray-constrained refinement.
Dataset
Method
SC-IoU
■ ceiling
■ floor
■ wall
■ window
■ chair
■ bed
■ sofa
■ table
■ tvs
■ furniture
■ objects
mIoU
Occ-ScanNet
SGR-OCC (w/o VLM)
58.55
44.30
54.50
50.80
40.10
44.40
57.60
63.60
46.40
44.30
56.70
46.10
49.89
SGR-OCC + VLM
58.92
44.10
54.80
49.15
41.50
45.30
58.20
64.40
45.80
44.80
58.30
48.25
50.14
EmbodiedOcc
SGR-OCC (w/o VLM)
55.72
39.60
52.50
48.80
34.30
42.90
51.10
58.50
44.80
40.30
52.40
43.30
46.22
SGR-OCC + VLM
54.85
39.20
53.20
48.85
34.60
41.20
48.40
56.90
42.90
42.80
51.90
42.40
45.12
Appendix
Table 11: Quantitative Impact of Language-Aware Geometric Refinement. We evaluate the integration of our VLM-driven attribute reasoning across the full Occ-ScanNet Yu et al. (2024a) and EmbodiedOcc Wu et al. (2025) Compared to the baseline SGR-OCC (which relies on categorical heuristics), the VLM-integrated variant ( SGR-OCC + VLM ) delivers consistent improvements across almost all 11 semantic categories. The most pronounced gains are observed in objects with high geometric variance (e.g., Furniture, Objects) and strict planar structures (e.g., Wall, Floor).
Existing learning-based occupancy prediction methods rely on large-scale 3D annotations and generalize poorly across environments. We present FreeOcc, a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences. Unlike prior approaches that require voxel-level supervision and ground-truth camera poses, FreeOcc operates without 3D annotations, pose ground truth, or any learning stage. FreeOcc incrementally builds a globally consistent occupancy map via a four-layer pipeline: a SLAM backbone estimates poses and sparse geometry; a geometrically consistent Gaussian update constructs dense 3D Gaussian maps; open-vocabulary semantics from off-the-shelf vision-language models are associated with Gaussian primitives; and a probabilistic Gaussian-to-occupancy projection produces dense voxel occupancy. Despite being entirely training-free and pose-agnostic, FreeOcc achieves over 2× improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods. We further introduce ReplicaOcc, a benchmark for indoor open-vocabulary occupancy prediction, and show that FreeOcc transfers zero-shot to novel environments, substantially outperforming both supervised and self-supervised baselines. Project page: https://the-masses.github.io/freeocc-web/.
Zeyu Jiang, Changqing Zhou, Xingxing Zuo +1
The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI
Accurate 3D scene understanding is essential for embodied intelligence, with occupancy prediction emerging as a key task for reasoning about both objects and free space. Existing approaches largely rely on depth priors (e.g., DepthAnything) but make only limited use of 3D cues, restricting performance and generalization. Recently, visual geometry models such as VGGT have shown strong capability in providing rich 3D priors, but similar to monocular depth foundation models, they still operate at the level of visible surfaces rather than volumetric interiors, motivating us to explore how to more effectively leverage these increasingly powerful geometry priors for 3D occupancy prediction. We present GPOcc, a framework that leverages generalizable visual geometry priors (GPs) for monocular occupancy prediction. Our method extends surface points inward along camera rays to generate volumetric samples, which are represented as Gaussian primitives for probabilistic occupancy inference. To handle streaming input, we further design a training-free incremental update strategy that fuses per-frame Gaussians into a unified global representation. Experiments on Occ-ScanNet and EmbodiedOcc-ScanNet demonstrate significant gains: GPOcc improves mIoU by +9.99 in the monocular setting and +11.79 in the streaming setting over prior state of the art. Under the same depth prior, it achieves +6.73 mIoU while running 2.65× faster. These results highlight that GPOcc leverages geometry priors more effectively and efficiently. Code will be released at https://github.com/JuIvyy/GPOcc.
Changqing Zhou, Yueru Luo, Changhao Chen
The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen
Embodied tasks demand accurate, flexible, and semantically rich 3D scene representations. 3D semantic occupancy is well suited to this requirement, as it can model holistic 3D spaces by encoding geometric occupancy along with semantic categories. However, existing occupancy prediction methods struggle to meet practical deployment requirements, such as adapting to varying computing budgets, sensor setups, and observation views. In this paper, we propose a point-based Adaptive 3D Occupancy Prediction method, called AdaOcc, tailored for embodied scenarios. To accommodate heterogeneous sensor inputs, AdaOcc uses an adaptive geometry-guided dual-branch encoder that can support RGB images in various numbers of views with (estimated) depth maps or LiDAR scans. AdaOcc represents occupied regions via sparse semantic points trained with a progressive query learning strategy, allowing the prediction computational budget to be flexibly adjusted through query point numbers and decoder layers. To facilitate high-fidelity geometric modeling for lightweight point-based occupancy learning, we further propose a novel containment loss that regularizes predicted points to reside within valid occupied regions. Extensive experiments show that our method achieves a new state-of-the-art on Occ-ScanNet with considerable performance improvements over previous methods. Moreover, our framework demonstrates strong practical applicability as an adaptive 3D perception module in real-world embodied systems.
Jinglong Wang, Yunjie Wang, Zhiyang Zhang +4
Beihang University · Beijing Academy of Artificial Intelligence · Hebei University of Technology +3