SGR-OCC: Evolving Monocular Priors for Embodied 3D Occupancy Prediction via Soft-Gating Lifting and Semantic-Adaptive Geometric Refinement

Yiran Guo; Simone Mentasti; Xiaofeng Jin; Matteo Frosi; Matteo Matteucci

cs.CVMar 14, 2026

SGR-OCC: Evolving Monocular Priors for Embodied 3D Occupancy Prediction via Soft-Gating Lifting and Semantic-Adaptive Geometric Refinement

Authors: Yiran Guo, Simone Mentasti, Xiaofeng Jin, Matteo Frosi, Matteo Matteucci

Organizations: Department of Electronics Information and Bioengineering, Politecnico di Milano

Abstract

3D semantic occupancy prediction is a cornerstone for embodied AI, enabling agents to perceive dense scene geometry and semantics incrementally from monocular video streams. However, current online frameworks face two critical bottlenecks: the inherent depth ambiguity of monocular estimation that causes "feature bleeding" at object boundaries , and the "cold start" instability where uninitialized temporal fusion layers distort high-quality spatial priors during early training stages. In this paper, we propose SGR-OCC (Soft-Gating and Ray-refinement Occupancy), a unified framework driven by the philosophy of "Inheritance and Evolution". To perfectly inherit monocular spatial expertise, we introduce a Soft-Gating Feature Lifter that explicitly models depth uncertainty via a Gaussian gate to probabilistically suppress background noise. Furthermore, a Dynamic Ray-Constrained Anchor Refinement module simplifies complex 3D displacement searches into efficient 1D depth corrections along camera rays, ensuring sub-voxel adherence to physical surfaces. To ensure stable evolution toward temporal consistency, we employ a Two-Phase Progressive Training Strategy equipped with identity-initialized fusion, effectively resolving the cold start problem and shielding spatial priors from noisy early gradients. Extensive experiments on the EmbodiedOcc-ScanNet and Occ-ScanNet benchmarks demonstrate that SGR-OCC achieves state-of-the-art performance. In local prediction tasks, SGR-OCC achieves a completion IoU of 58.55 $\%$ and a semantic mIoU of 49.89 $\%$ , surpassing the previous best method, EmbodiedOcc++, by 3.65 $\%$ and 3.69 $\%$ respectively. In challenging embodied prediction tasks, our model reaches 55.72 $\%$ SC-IoU and 46.22 $\%$ mIoU. Qualitative results further confirm our model's superior capability in preserving structural integrity and boundary sharpness in complex indoor environments.

Figures & tables

Figure 1: The overall architecture of our proposed framework. The pipeline consists of a Soft-Gating Feature Lifter for robust 2D-to-3D projection and a Dynamic Ray-Constrained Anchor Refinement module for sub-voxel geometric correction. The Semantic-Adaptive GRM (right) enforces category-specific geometric constraints. The bottom-right panel illustrates our Two-Phase Progressive Training Strategy: Stage 1 freezes the monocular backbone to initialize temporal alignment (training only the Global Head), while Stage 2 unfreezes the entire network for global co-adaptation using both Local and Global supervision.

Panel of Figure 2 — Figure 2: Overview of the core geometric modules. (a) The Soft-Gating mechanism evaluates depth consistency ( $d_{proj}$ vs. $d_{pred}$ ) within a sampling window. It applies Gaussian weights to foreground regions while truncating conflicting background noise, ensuring robust feature aggregation. (b) The Refinement module restricts anchor optimization ( $\mathbf{P}_{init}\to\mathbf{P}_{refined}$ ) to a 1D depth residual $\Delta d$ strictly along the camera ray $\vec{\mathbf{r}}$ , adhering points to the physical surface and converting a complex 3D search into an efficient 1D correction.

Method	Input	SC-IoU	$\blacksquare$ ceiling	$\blacksquare$ floor	$\blacksquare$ wall	$\blacksquare$ window	$\blacksquare$ chair	$\blacksquare$ bed	$\blacksquare$ sofa	$\blacksquare$ table	$\blacksquare$ tvs	$\blacksquare$ furniture	$\blacksquare$ objects	mIoU
TPVFormer ( Huang et al. (2023) )	$x^{\text{rgb }}$	33.39	6.96	32.97	14.41	9.10	24.01	41.49	45.44	28.61	10.66	35.37	25.31	24.94
GaussianFormer ( Huang et al. (2024) )	$x^{\text{rgb }}$	40.91	20.70	42.00	23.40	17.40	27.0	44.30	44.80	32.70	15.30	36.70	25.00	29.93
MonoScene ( Cao and de Charette (2022) )	$x^{\text{rgb }}$	41.60	15.17	44.71	22.41	12.55	26.11	27.03	35.91	28.32	6.57	32.16	19.84	24.62
ISO ( Yu et al. (2024a) )	$x^{\text{rgb }}$	42.16	19.88	41.88	22.37	16.98	29.09	42.43	42.00	29.60	10.62	36.36	24.61	28.71
Surroundocc ( Wei et al. (2023) )	$x^{\text{rgb }}$	42.52	18.90	49.30	24.80	18.00	26.80	42.00	44.10	32.90	18.60	36.80	26.90	30.83
EmbodiedOcc ( Park et al. (2025) )	$x^{\mathrm{rgb}}$	53.55	39.60	50.40	41.40	31.70	40.90	55.00	61.40	44.00	36.10	53.90	42.20	45.15

Table 1: Local Prediction Performance on the Occ-ScanNet dataset.

Figure 3: Qualitative comparison of local occupancy prediction on Occ-ScanNet. From left to right: Input RGB images, EmbodiedOcc Wu et al. (2025) , EmbodiedOcc++ Wang et al. (2025a) , our proposed method, and the Ground Truth. Our approach exhibits superior boundary sharpness and structural integrity, particularly in complex geometries like the sink (row 3) and table edges (row 2), thanks to the Soft-Gating mechanism and Ray-Constrained Refinement.

Method	Dataset	SC-IoU	$\blacksquare$ ceiling	$\blacksquare$ floor	$\blacksquare$ wall	$\blacksquare$ window	$\blacksquare$ chair	$\blacksquare$ bed	$\blacksquare$ sofa	$\blacksquare$ table	$\blacksquare$ tvs	$\blacksquare$ furniture	$\blacksquare$ objects	mIoU
TPVFormer ( Huang et al. (2023) )	EmbodiedOcc	35.88	1.62	30.54	12.03	13.22	35.47	51.39	49.79	25.63	3.60	43.15	16.23	25.70
SurroundOcc ( Wei et al. (2023) )	EmbodiedOcc	37.04	12.70	31.80	22.50	22.00	29.90	44.70	36.50	24.60	11.50	34.40	18.20	26.27
GaussianFormer ( Huang et al. (2024) )	EmbodiedOcc	38.02	17.00	33.60	21.50	21.70	29.40	47.80	37.10	24.30	15.50	36.20	16.80	27.36
SplicingOcc	EmbodiedOcc	49.01	31.60	38.80	35.50	36.30	47.10	54.50	57.20	34.40	32.50	51.20	29.10	40.74
EmbodiedOcc ( Park et al. (2025) )	EmbodiedOcc	51.52	22.70	44.60	37.40	38.00	50.10	56.70	59.70	35.40	38.40	52.00	32.90	42.53
EmbodiedOcc++ ( Wang et al. (2025a) )	EmbodiedOcc	52.20	27.90	43.90	38.70	40.60	49.00	57.90	59.20	36.80	37.80	53.50	34.10	43.60

Table 2: Embodied prediction performance on the EmbodiedOcc-ScanNet dataset.

Figure 4: Qualitative results of embodied occupancy prediction on EmbodiedOcc-ScanNet. The rows display (top to bottom): the ground truth occupancy maps, our SGR-OCC predictions, and the underlying Gaussian Memory accumulation. The columns represent the progressive reconstruction of the global scene across a 30-frame sequence. Our model successfully integrates sequential observations into a coherent Gaussian Memory (bottom row), allowing for stable and high-fidelity global occupancy prediction (middle row) that demonstrates strong structural consistency with the ground truth.

Table 7

Panel of Figure 5 — Figure 5: Effectiveness of the proposed SGR-OCC components. (a) Qualitative comparison of anchor refinement: Our Ray-Constrained Refinement ( $R^{1}$ ) reduces geometric drift compared to the unconstrained baseline ( $R^{3}$ ), forcing anchors to adhere strictly to physical surfaces (indicated by cooler colors). (b) Quantitative analysis of training stability: Compared to E2E (End-to-End) baselines, our Two-Phase Progressive Training effectively bypasses the cold start drop, leading to faster convergence and a superior mIoU of 46.22%.

Figure 6: Visualization of the Semantic-Adaptive GRM. Unlike the uniform regularization of EmbodiedOcc++ (solid red), our method generates semantic-aware weight maps. It applies strict planar constraints to structural elements like walls (red) while adaptively relaxing constraints for complex furniture (blue/grey), effectively preventing geometric over-smoothing.

Table 10

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Tolerance ( $\sigma$ )	SC-IoU (%)	mIoU (%)	Boundary F1 (%)
0.1	52.54	42.74	42.10
0.2	53.63	44.80	44.30
0.5 (SGR-OCC)	58.55	49.89	48.36
1.0	56.80	47.05	46.12
2.0	54.72	45.47	45.60

Appendix

Table 7: Sensitivity Analysis of Gaussian Gating Tolerance (

\sigma

). We evaluate the impact of the depth tolerance parameter on overall geometric completion (SC-IoU), semantic accuracy (mIoU), and structural sharpness (Boundary F1-score). The inverted-U trend demonstrates that

\sigma=0.5

optimally balances background noise suppression and foreground feature recall, strictly preventing "feature bleeding" at depth discontinuities.

Figure 7: Qualitative Visualization of Real Gating Weights on EmbodiedOcc-ScanNet. We extract the Bird’s Eye View (BEV) cross-sectional heatmaps of our Gaussian gating weights ( $\mathcal{G}$ ) during the feature lifting phase. An overly strict tolerance ( $\sigma=0.1$ ) leads to severe feature starvation, while an over-relaxed tolerance ( $\sigma=2.0$ ) acts as an all-pass filter, causing feature bleeding. Our optimal setting ( $\sigma=0.5$ ) perfectly encapsulates the physical boundaries of complex objects (e.g., the chair) while explicitly suppressing occluded background noise.

Figure 8: Pareto Frontier of Accuracy vs. Efficiency. The plot illustrates the trade-off between mIoU and inference speed for varying

K

. The solid blue line traces the optimal Pareto frontier, while the grey dashed line exposes a sub-optimal collapse caused by spatial over-smoothing at denser sampling rates (

K\geq 24

). The red star explicitly marks

K=16

as our definitive sweet spot.

Sampling Points ( $K$ )	SC-IoU (%)	mIoU (%)	TV (%)	Objects (%)	GFLOPs	FPS
4	56.20	48.15	39.50	41.20	18.5	12.5
8	57.65	49.20	42.15	44.50	21.0	10.2
16 (SGR-OCC)	58.55	49.89	44.30	46.10	25.2	8.5
24	58.70	49.05	44.75	45.45	33.0	6.8
32	57.75	47.12	42.90	45.60	41.5	3.5

Appendix

Table 8: Ablation on Deformable Sampling Points (

K

). We evaluate the trade-off between computational efficiency (GFLOPs, FPS) and geometric granularity. While increasing

K

initially boosts the recall of small objects (e.g., TVs, Objects), an excessively dense sampling grid (

K\geq 24

) introduces spatial over-smoothing, leading to a degradation in mIoU. Ultimately,

K=16

emerges as the absolute Pareto optimum, maximizing semantic accuracy while maintaining a viable inference speed.

Depth Prior	Params	Local Task		Embodied Task
Depth Prior	Params	SC-IoU (%)	mIoU (%)	SC-IoU (%)	mIoU (%)
DA2 (Default)	0.09B	58.55	49.89	55.72	46.22
DA3METRIC-Large	0.35B	58.11	48.92	55.89	44.24

Appendix

Table 9: Impact of Scaling the Monocular Depth Prior. We compare our default backbone (DA2) against the fine-tuned DA3METRIC-Large (0.35B). Despite a

4\times

increase in parameters, the saturated indoor depth prior yields no downstream improvements and even slight degradation, highlighting the inherent depth-error tolerance and decoupling capability of our SGR-OCC architecture.

Dataset	Method	Input	SC-IoU	$\blacksquare$ ceiling	$\blacksquare$ floor	$\blacksquare$ wall	$\blacksquare$ window	$\blacksquare$ chair	$\blacksquare$ bed	$\blacksquare$ sofa	$\blacksquare$ table	$\blacksquare$ tvs	$\blacksquare$ furniture	$\blacksquare$ objects	mIoU
Occ-ScanNet-mini	MonoScene Cao and de Charette (2022)	$x^{\text{rgb}}$	41.90	17.00	46.20	23.90	12.70	27.00	29.10	34.80	29.10	9.70	34.50	20.40	25.90
	ISO Yu et al. (2024a)	$x^{\text{rgb}}$	42.90	21.10	42.70	24.60	15.10	30.80	41.00	43.30	32.20	12.10	35.90	25.10	29.40
	EmbodiedOcc Wu et al. (2025)	$x^{\text{rgb}}$	53.80	29.10	48.70	42.30	38.70	42.00	62.70	60.60	48.20	33.80	58.00	46.50	46.40
	EmbodiedOcc++ Wang et al. (2025a)	$x^{\text{rgb}}$	55.70	23.30	51.00	42.80	39.30	43.50	65.60	64.00	50.70	40.70	60.30	48.90	48.20
	SGR-OCC (Ours)	$x^{\text{rgb}}$	59.85	37.20	54.40	50.60	41.10	45.30	67.10	64.20	48.50	46.20	57.50	48.80	50.99
EmbodiedOcc-mini	EmbodiedOcc Wu et al. (2025)	$x^{\text{rgb}}$	50.70	21.50	44.50	38.30	27.90	46.90	64.70	55.30	42.70	35.80	52.50	27.50	41.60

Appendix

Table 10: Quantitative Results on Mini-Splits. We evaluate SGR-OCC alongside representative baselines on the lightweight Occ-ScanNet-mini (Local) and EmbodiedOcc-mini (Temporal) benchmarks. While all methods experience degradation due to data scarcity, SGR-OCC exhibits the strongest resilience, maintaining a significant performance gap over the baselines.

Figure 9: Qualitative Comparison on the Occ-ScanNet Dataset. Compared to EmbodiedOcc and EmbodiedOcc++, our SGR-OCC produces significantly sharper semantic boundaries and explicitly eliminates the "feature bleeding" artifacts common in purely implicit lifting methods. The reconstructed structures (e.g., furniture, walls) exhibit high fidelity to the Ground Truth.

Figure 10: Sequential Evolution of Embodied Perception. As the agent navigates the scene (Input Image), SGR-OCC incrementally aggregates multi-view geometry into the Gaussian Memory. Empowered by our Soft-Gating mechanism, which ensures only geometrically valid features are lifted, this memory acts as a robust, noise-free spatial anchor, enabling the decoding of a highly stable and structurally accurate Global Occupancy map over time.

Figure 11: Architecture of the Language-Aware Geometric Refinement. By integrating a frozen Qwen2.5-VL-3B Bai et al. (2025) backbone, we extract both global scene context and localized attribute semantics. These multi-modal embeddings are concatenated with the visual geometric features and processed by a trainable Attribute MLP (Language-Aware Predictor), which dynamically predicts the constraint coefficient $\kappa$ for the downstream 1D ray-constrained refinement.

Dataset	Method	SC-IoU	$\blacksquare$ ceiling	$\blacksquare$ floor	$\blacksquare$ wall	$\blacksquare$ window	$\blacksquare$ chair	$\blacksquare$ bed	$\blacksquare$ sofa	$\blacksquare$ table	$\blacksquare$ tvs	$\blacksquare$ furniture	$\blacksquare$ objects	mIoU
Occ-ScanNet	SGR-OCC (w/o VLM)	58.55	44.30	54.50	50.80	40.10	44.40	57.60	63.60	46.40	44.30	56.70	46.10	49.89
Occ-ScanNet	SGR-OCC + VLM	58.92	44.10	54.80	49.15	41.50	45.30	58.20	64.40	45.80	44.80	58.30	48.25	50.14
EmbodiedOcc	SGR-OCC (w/o VLM)	55.72	39.60	52.50	48.80	34.30	42.90	51.10	58.50	44.80	40.30	52.40	43.30	46.22
EmbodiedOcc	SGR-OCC + VLM	54.85	39.20	53.20	48.85	34.60	41.20	48.40	56.90	42.90	42.80	51.90	42.40	45.12

Appendix

Table 11: Quantitative Impact of Language-Aware Geometric Refinement. We evaluate the integration of our VLM-driven attribute reasoning across the full Occ-ScanNet Yu et al. (2024a) and EmbodiedOcc Wu et al. (2025) Compared to the baseline SGR-OCC (which relies on categorical heuristics), the VLM-integrated variant ( SGR-OCC + VLM ) delivers consistent improvements across almost all 11 semantic categories. The most pronounced gains are observed in objects with high geometric variance (e.g., Furniture, Objects) and strict planar structures (e.g., Wall, Floor).

3D Occupancy Prediction

3D occupancy prediction estimates the geometry and semantics of occupied and free space around an agent from camera, LiDAR, or radar data.

See all 113 papers

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction
Apr 30, 2026Zeyu Jiang, Changqing Zhou, Xingxing Zuo +1Semantic Occupancy Mapping 3D Occupancy Prediction
The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI
Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Prediction
Feb 25, 2026Changqing Zhou, Yueru Luo, Changhao Chen3D Occupancy Prediction
The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen
AdaOcc: Adaptive 3D Occupancy Prediction for Embodied Tasks
Sep 30, 2026Jinglong Wang, Yunjie Wang, Zhiyang Zhang +43D Occupancy Prediction Adaptive Computation
Beihang University · Beijing Academy of Artificial Intelligence · Hebei University of Technology +3

Apr 30, 2026J/K move · Enter open · S save

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction

Zeyu Jiang, Changqing Zhou, Xingxing Zuo, Changhao Chen

The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI

Semantic Occupancy Mapping 3D Occupancy Prediction

Open paper AskPDF arXiv

Existing learning-based occupancy prediction methods rely on large-scale 3D annotations and generalize poorly across environments. We present FreeOcc, a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences. Unlike prior approaches that require voxel-level supervision and ground-truth camera poses, FreeOcc operates without 3D annotations, pose ground truth, or any learning stage. FreeOcc incrementally builds a globally consistent occupancy map via a four-layer pipeline: a SLAM backbone estimates poses and sparse geometry; a geometrically consistent Gaussian update constructs dense 3D Gaussian maps; open-vocabulary semantics from off-the-shelf vision-language models are associated with Gaussian primitives; and a probabilistic Gaussian-to-occupancy projection produces dense voxel occupancy. Despite being entirely training-free and pose-agnostic, FreeOcc achieves over

2\times

improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods. We further introduce ReplicaOcc, a benchmark for indoor open-vocabulary occupancy prediction, and show that FreeOcc transfers zero-shot to novel environments, substantially outperforming both supervised and self-supervised baselines. Project page: https://the-masses.github.io/freeocc-web/.

Abstract

3D Occupancy Prediction

Explore similar work

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction

Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Prediction

AdaOcc: Adaptive 3D Occupancy Prediction for Embodied Tasks