Dr.Occ: Depth- and Region-Guided 3D Occupancy from Surround-View Cameras for Autonomous Driving

Xubo Zhu; Haoyang Zhang; Fei He; Rui Wu; Yanhu Shan; Wen Yang; Huai Yu

cs.CVMar 1, 2026

Dr.Occ: Depth- and Region-Guided 3D Occupancy from Surround-View Cameras for Autonomous Driving

Authors: Xubo Zhu, Haoyang Zhang, Fei He, Rui Wu, Yanhu Shan, Wen Yang, Huai Yu

Organizations: School of Electronic Information, Wuhan University · Horizon Robotics · Wuhan University Shenzhen Research Institute

Abstract

3D semantic occupancy prediction is crucial for autonomous driving perception, offering comprehensive geometric scene understanding and semantic recognition. However, existing methods struggle with geometric misalignment in view transformation due to the lack of pixel-level accurate depth estimation, and severe spatial class imbalance where semantic categories exhibit strong spatial anisotropy. To address these challenges, we propose Dr. Occ, a depth- and region-guided occupancy prediction framework. Specifically, we introduce a depth-guided 2D-to-3D View Transformer (D $^2$ -VFormer) that effectively leverages high-quality dense depth cues from MoGe-2 to construct reliable geometric priors, thereby enabling precise geometric alignment of voxel features. Moreover, inspired by the Mixture-of-Experts (MoE) framework, we propose a region-guided Expert Transformer (R/R $^2$ -EFormer) that adaptively allocates region-specific experts to focus on different spatial regions, effectively addressing spatial semantic variations. Thus, the two components make complementary contributions: depth guidance ensures geometric alignment, while region experts enhance semantic learning. Experiments on the Occ3D--nuScenes benchmark demonstrate that Dr. Occ improves the strong baseline BEVDet4D by 7.43% mIoU and 3.09% IoU under the full vision-only setting.

Figures & tables

Figure 2: The overall architecture of Dr.Occ. T consecutive surround‑view images are processed by MoGe‑2 [ 28 ] to estimate depth maps, which provide geometric cues for D 2 -VFormer to construct dense, low‑cost, and geometrically accurate voxel features. These features are then refined in R 2 -EFormer through recursive semantic decoding. Finally, the refined features are decoded by the OCC Decoder to produce the occupancy prediction.

Figure 3: Depth-guided Dual Projection View Transformer.

Figure 4: (a–b) Spatial semantic distribution reveals strong anisotropy across height and distance. (c–d) R-EFormer partitions 3D space into a 3×3 spatial grid along range (near, mid, far) and height (low, mid, high) dimensions, assigning each region a dedicated expert. (e) R 2 -EFormer adaptively refines regions through recursive masking.

Method	Backbone	Input Size	mIoU( $\%$ )	$\blacksquare$ others	$\blacksquare$ barrier	$\blacksquare$ bicycle	$\blacksquare$ bus	$\blacksquare$ car	$\blacksquare$ cons. veh.	$\blacksquare$ motor.	$\blacksquare$ pedes.	$\blacksquare$ tfc. cone	$\blacksquare$ trailer	$\blacksquare$ truck	$\blacksquare$ drv. surf.	$\blacksquare$ other flat	$\blacksquare$ sidewalk	$\blacksquare$ terrain	$\blacksquare$ manmade	$\blacksquare$ vegetation
BEVFormer [ 16 ]	R101	1600 $\times$ 900	26.9	5.9	37.8	17.9	40.4	42.4	7.4	23.9	21.8	21.0	22.4	30.7	55.4	28.4	36.0	28.1	20.0	17.7
CTF-Occ [ 25 ]	R101	1600 $\times$ 900	28.5	8.1	39.3	20.6	38.3	42.2	16.9	24.5	22.7	21.1	23.0	31.1	53.3	33.8	38.0	33.2	20.8	18.0
TPVFormer [ 9 ]	R101	1600 $\times$ 900	27.8	7.2	38.9	13.7	40.8	45.9	17.2	20.0	18.9	14.3	26.7	34.2	55.7	35.5	37.6	30.7	19.4	16.8
SparseOcc [ 20 ]	R50	704 $\times$ 256	30.9	10.6	39.2	20.2	32.9	43.3	19.4	23.8	23.4	29.3	21.4	29.3	67.7	36.3	44.6	40.9	22.0	21.9
OSP ∗ [ 24 ]	R101	1600 $\times$ 900	41.2	10.9	49.0	27.7	50.2	55.9	22.9	31.0	30.9	30.3	35.6	31.2	82.1	42.6	51.9	55.1	44.8	38.2
FlashOcc ∗ [ 33 ]	R50	704 $\times$ 256	37.8	9.1	46.3	17.7	42.7	50.6	23.7	20.1	22.3	24.1	30.3	37.4	81.7	40.1	52.3	56.5	47.7	40.6

Table 1: Quantitative comparison of mIoU (%) on Occ3D-nuScenes [ 25 ] . ∗ indicates models trained with camera mask supervision. ‡ indicates methods re-trained in this work as official pre-trained weights are not available.

Geometric	Semantic		Metric
D 2 -VFormer	R-EFormer	R 2 -EFormer	IoU( $\%$ ) $\uparrow$	mIoU( $\%$ ) $\uparrow$
			70.36	36.01
✓			71.29	41.45
✓	✓		73.45	43.03
✓		✓	72.87	43.43

Table 2: Ablation study on the each component.

Figure 5: Benefits of depth-guided dual projection. Our D 2 -VFormer generates more complete and detailed occupancy predictions (b) by learning geometry-aware features (a), as evidenced by the closer alignment with ground truth.

Figure 6: Qualitative visualization of region-guided semantic enhancement. Our method produces more accurate and human-interpretable occupancy predictions by adaptively partitioning regions and allocating specialized experts.

Explore similar work

VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction
Mar 3, 2026A. Enes Doruk, Hasan F. Ates3D Occupancy Prediction Autonomous Driving Perception
SparseOcc++: Geometry-Aware Sparse Latent Representation for Semantic Occupancy Prediction
Jul 6, 2026Pin Tang, Zhongdao Wang, Guoqing Wang +23D Reconstruction Autonomous Driving Perception
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200240, China · Huawei Noah’s Ark Lab, China
Semantic Occupancy Prediction with Dual Range-Voxel Representation
Jun 30, 2026Sitao Chen, Zhuangwei Zhuang, Hui Luo +3Autonomous Driving Perception Semantic Occupancy Mapping
School of Software Engineering, South China University of Technology, Guangzhou 510006, China · State Key Laboratory of Optical Field Manipulation Science and Technology, Institute of Optics and Electronics, CAS, Chengdu 610209, China

Mar 3, 2026J/K move · Enter open · S save

VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction

A. Enes Doruk, Hasan F. Ates

3D Occupancy Prediction Autonomous Driving Perception Camera-LiDAR Fusion VLMs for Autonomous Driving

Open paper AskPDF arXiv

This paper introduces VLMFusionOcc3D, a robust multimodal framework for dense 3D semantic occupancy prediction in autonomous driving. Current voxel-based occupancy models often struggle with semantic ambiguity in sparse geometric grids and performance degradation under adverse weather conditions. To address these challenges, we leverage the rich linguistic priors of Vision-Language Models (VLMs) to anchor ambiguous voxel features to stable semantic concepts. Our framework initiates with a dual-branch feature extraction pipeline that projects multi-view images and LiDAR point clouds into a unified voxel space. We propose Instance-driven VLM Attention (InstVLM), which utilizes gated cross-attention and LoRA-adapted CLIP embeddings to inject high-level semantic and geographic priors directly into the 3D voxels. Furthermore, we introduce Weather-Aware Adaptive Fusion (WeathFusion), a dynamic gating mechanism that utilizes vehicle metadata and weather-conditioned prompts to re-weight sensor contributions based on real-time environmental reliability. To ensure structural consistency, a Depth-Aware Geometric Alignment (DAGA) loss is employed to align dense camera-derived geometry with sparse, spatially accurate LiDAR returns. Extensive experiments on the nuScenes and SemanticKITTI datasets demonstrate that our plug-and-play modules consistently enhance the performance of state-of-the-art voxel-based baselines. Notably, our approach achieves significant improvements in challenging weather scenarios, offering a scalable and robust solution for complex urban navigation.

Abstract

Explore similar work

VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction

SparseOcc++: Geometry-Aware Sparse Latent Representation for Semantic Occupancy Prediction

Semantic Occupancy Prediction with Dual Range-Voxel Representation