$M^2$-Occ: Resilient 3D Semantic Occupancy Prediction for Autonomous Driving with Incomplete Camera Inputs

Kaixin Lin; Kunyu Peng; Di Wen; Yufan Chen; Ruiping Liu; Kailun Yang

cs.CVMar 10, 2026

$M^2$ -Occ: Resilient 3D Semantic Occupancy Prediction for Autonomous Driving with Incomplete Camera Inputs

Authors: Kaixin Lin, Kunyu Peng, Di Wen, Yufan Chen, Ruiping Liu, Kailun Yang

Abstract

Semantic occupancy prediction enables dense 3D geometric and semantic understanding for autonomous driving. However, existing camera-based approaches implicitly assume complete surround-view observations, an assumption that rarely holds in real-world deployment due to occlusion, hardware malfunction, or communication failures. We study semantic occupancy prediction under incomplete multi-camera inputs and introduce $M^2$ -Occ, a framework designed to preserve geometric structure and semantic coherence when views are missing. $M^2$ -Occ addresses two complementary challenges. First, a Multi-view Masked Reconstruction (MMR) module leverages the spatial overlap among neighboring cameras to recover missing-view representations directly in the feature space. Second, a Feature Memory Module (FMM) introduces a learnable memory bank that stores class-level semantic prototypes. By retrieving and integrating these global priors, the FMM refines ambiguous voxel features, ensuring semantic consistency even when observational evidence is incomplete. We introduce a systematic missing-view evaluation protocol on the nuScenes-based SurroundOcc benchmark, encompassing both deterministic single-view failures and stochastic multi-view dropout scenarios. Under the safety-critical missing back-view setting, $M^2$ -Occ improves the IoU by 4.36%. As the number of missing cameras increases, the robustness gap further widens; for instance, under the setting with five missing views, our method boosts the IoU by 6.67%. These gains are achieved without compromising full-view performance. The source code will be publicly released at https://github.com/qixi7up/M2-Occ.

Figures & tables

Fig. 1 : Existing methods fail with missing camera inputs ( e.g. , FRONT-view). Our $M^{2}$ -Occ reconstructs missing features from adjacent overlaps and stabilizes semantics via global memory, achieving superior performance.

Fig. 2 : Overview of $M^{2}$ -Occ. Multi-view images are processed by a shared backbone. MMR reconstructs missing views from adjacent overlaps. Features are lifted to a unified 3D volume. FMM refines voxel features via global semantic prototypes before generating dense 3D occupancy.

Fig. 3 : An overview of Multi-view Masked Reconstruction (MMR). The MMR module extracts overlapping boundary features from adjacent unmasked views and concatenates them with a central learnable mask token. A lightweight transformer decoder then processes this structural prior to reconstruct the missing view’s representations, preserving spatial continuity.

Fig. 4 : A comparison between the single-proto strategy and the multi-proto strategy. While the single-proto approach maintains one global centroid per semantic class, the multi-proto strategy captures intra-class variance by learning multiple sub-prototypes and dynamically retrieving them based on feature similarity.

Setting	Method	IoU	mIoU	barrier	bicycle	bus	car	const.veh.	motorcycle	pedestrian	traffic cone	trailer	truck	drive.surf.	other flat	sidewalk	terrain	manmade	vegetation
Standard	SurroundOcc (Wei et al. 2023 [ 7 ] )	31.18	17.23	16.16	8.15	19.07	29.28	10.22	9.11	10.59	8.19	11.09	18.54	38.04	19.20	23.35	21.57	12.68	20.44
Standard	Ours (SurroundOcc)	31.17	18.03	17.56	7.65	23.86	29.54	9.92	12.63	11.31	7.96	12.03	20.31	38.78	18.29	24.10	21.42	12.63	20.56
Front	SurroundOcc (Wei et al. 2023 [ 7 ] )	23.96	13.35	13.90	7.83	15.91	25.02	9.90	6.01	8.93	8.04	9.55	16.77	14.56	14.12	16.41	17.39	11.34	17.96
Front	Ours (SurroundOcc)	29.30	15.19	14.64	4.03	18.64	25.26	9.40	7.48	7.41	6.92	10.27	17.56	35.35	14.09	21.57	19.24	11.91	19.33
Front Right	SurroundOcc (Wei et al. 2023 [ 7 ] )	29.89	15.92	15.12	7.28	17.66	27.23	8.65	8.21	9.40	7.49	10.29	17.43	36.58	17.82	21.45	19.29	11.80	19.01
Front Right	Ours (SurroundOcc)	30.34	17.02	16.32	6.99	23.64	27.58	8.71	11.75	9.83	7.38	11.03	19.26	37.79	17.31	23.03	20.38	11.82	19.54

TABLE I : Single-view missing results on SurroundOcc (IoU, mIoU, as well as per-category voxel-level IoU).

Setting	Method	mIoU (vox)	mIoU (pts)	barrier	bicycle	bus	car	const.veh.	motorcycle	pedestrian	traffic cone	trailer	truck	drive.surf.	other flat	sidewalk	terrain	manmade	vegetation
Standard	TPVFormer (Huang et al. 2023 [ 8 ] )	47.84	26.92	35.07	12.71	48.70	48.51	29.52	20.04	19.61	8.90	38.32	46.09	15.45	13.82	19.82	20.83	27.29	25.99
Standard	Ours (TPVFormer)	50.26	30.11	36.42	11.30	50.71	53.86	35.98	32.90	28.90	10.62	41.51	53.28	17.41	12.75	19.18	21.45	28.49	27.06
Front	TPVFormer (Huang et al. 2023 [ 8 ] )	43.82	25.45	34.17	12.04	39.19	47.20	29.71	17.60	17.41	8.50	32.28	44.52	18.04	13.42	19.66	21.73	26.94	24.80
Front	Ours (TPVFormer)	46.89	27.94	34.43	10.92	40.95	52.07	34.78	32.18	27.38	10.33	36.86	48.97	16.44	11.32	17.39	20.69	27.41	24.98
Front Right	TPVFormer (Huang et al. 2023 [ 8 ] )	42.00	23.48	32.16	11.96	43.43	45.64	20.79	17.72	13.84	8.36	33.24	40.37	14.72	11.28	16.55	19.07	23.48	23.06
Front Right	Ours (TPVFormer)	44.36	26.09	31.40	10.33	45.91	49.06	26.16	23.61	26.88	9.35	35.34	45.71	16.48	10.25	17.21	20.45	25.12	24.15

TABLE II : Single-view missing results on TPVFormer (voxel-level and point-level mIoU, as well as per-category IoU).

Missing Views	Method	IoU (%)	mIoU (%)
0	Baseline	31.18	17.23
1	Baseline	27.70	14.95
	+ MMR	29.70	15.75
	+ MMR + FMM	29.77	16.25
3	Baseline	20.00	10.04
	+ MMR	26.09	11.11

TABLE III : Robustness analysis under different numbers of missing views. We compare the baseline, our method with MMR only, and our full model with both MMR and FMM.

Missing	MMR	FMM	IoU	mIoU
$\times$	$\times$	$\times$	31.18	17.23
✓	$\times$	$\times$	27.70	14.95
✓	✓	$\times$	29.70	15.75
✓	✓	✓	29.77	16.25

TABLE IV : Ablation study on the modules in the presented framework.

Fig. 5 : Visualizations on the nuScenes validation set [ 2 ] . Our method achieves promising results in the reconstruction of the missing $F$ view across various scenarios, even under weak lighting conditions.

Fig. 6 : Visualization of feature maps. Our reconstructed features can largely replace the original features.

Method	Number of Missing Views	Latency(s)	Memory(G)
Baseline	-	0.50	5.927
Ours	1	0.77	6.077
	2	0.91
	3	1.00
	4	1.11
	5	1.25

TABLE V : Experimental results of computation overhead.

Explore similar work

Jul 6, 2026cs.CV

SparseOcc++: Geometry-Aware Sparse Latent Representation for Semantic Occupancy Prediction

Vision-based 3D semantic occupancy prediction is essential for autonomous driving, yet dense voxel representations waste computation on largely empty space, while BEV and TPV projections compromise fine-grained 3D structure. Fully sparse representations offer an attractive alternative, but existing methods, including SparseOcc, entangle scene completion with semantic prediction by indiscriminately propagating high-dimensional features into empty regions and applying voxel-wise classification. This creates excessive activations, computational overhead, and geometric ambiguity. We present SparseOcc++, a geometry-aware sparse framework that explicitly decouples scene completion from semantic segmentation. SparseOcc++ reformulates completion as signed-distance regression on sparse anchor voxels through a scene completion field (SCF). To model complex outdoor geometry robustly, it combines orthogonal decomposition with discretized distance learning. A geometry-guided propagation module then converts the SCF into a complete volumetric scene and restricts semantic segmentation to geometrically verified regions. Experiments establish new state of the art: SparseOcc++ improves IoU by 2.3 points and is 3.9x faster than SparseOcc on nuScenes, while achieving a 5.9x speedup over OccFormer on SemanticKITTI.

Pin Tang, Zhongdao Wang, Guoqing Wang +2

MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200240, China · Huawei Noah’s Ark Lab, China

3D Semantic Occupancy Prediction Voxel Completion Propagation

Ask PDF

Jun 2, 2026cs.CV

UnsOcc: 3D Semantic Occupancy Prediction in Unstructured Scene via Rendering Fusion

Unstructured scenes present unique challenges for autonomous driving, as irregular obstacles and sparse scene layouts undermine the effectiveness of traditional perception methods such as 3D object detection. 3D semantic occupancy prediction has emerged as a prominent focus due to its ability to provide dense spatial representations by assigning semantic labels to individual voxels in 3D space. However, directly applying 3D semantic occupancy prediction to unstructured scenes remains challenging because scene sparsity hinders effective cross-modal fusion and the more severe long-tail distribution in these scenarios further degrades prediction performance. To validate the effectiveness of our approach, we construct a dedicated dataset of unstructured scenes collected from open-pit mines. Based on this, we propose UnsOcc, a multi-modal 3D semantic occupancy prediction framework that improves robustness in unstructured environments. At its core, we introduce a rendering-based fusion module, RenderFusion, which enhances cross-modal feature alignment through bidirectional rendering supervision. Furthermore, we propose GSRefinement, a detail-aware auxiliary supervision method based on Gaussian Splatting that projects sparse 3D occupancy predictions into dense 2D semantic segmentation maps, enabling effective supervision for long-tail categories. Extensive experiments on both the open-pit mine dataset and the nuScenes dataset demonstrate that our method significantly outperforms existing state-of-the-art approaches.

Ye Wu, Ruiqi Song, Baiyong Ding +3

School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Waytous Inc., Beijing, China

3D Semantic Occupancy Prediction 3D Object Detection Autonomous Driving 3D Generation

Ask PDF

Jun 30, 2026cs.CV

Semantic Occupancy Prediction with Dual Range-Voxel Representation

LiDAR-based 3D semantic occupancy prediction, which aims to provide accurate and comprehensive scene representation, is crucial for autonomous driving systems. As point clouds suffer from sparsity and incompleteness, leading to insufficient semantic learning and difficult occupancy perception, existing methods often stack multi-sweep point clouds to obtain dense spatial information. However, such a naive strategy also results in efficiency (e.g., additional computational burden) and robustness (e.g., pose transformation noise) concerns, which hinder their practical applications. In this work, we propose a Dual Range-Voxel Representation (DRVR) that leverages the range-view context and voxel-view geometry of single-sweep point clouds for 3D semantic occupancy prediction, eliminating the concerns associated with the multi-sweeps. Specifically, we use the range-view encoder to extract the compact context of the scene. To fully exploit the spatial information, we design a geometry-aware voxel-view encoder that extracts multi-scale voxel-view features separately and combines them for better geometric occupancy prediction. Moreover, we propose a range-voxel fusion module to cooperate range- and voxel-view features via voxel-to-range and range-to-voxel fusions. Extensive experiments on nuScenes-Occupancy, SemanticKITTI and SemanticPOSS show the superiority of our method. Especially on nuScenes-Occupancy, our single-sweep DRVR achieves 5.4% improvement in mIoU and 2.1x acceleration compared to the multi-sweep method.

Sitao Chen, Zhuangwei Zhuang, Hui Luo +3

School of Software Engineering, South China University of Technology, Guangzhou 510006, China · State Key Laboratory of Optical Field Manipulation Science and Technology, Institute of Optics and Electronics, CAS, Chengdu 610209, China

3D Semantic Occupancy Prediction Voxel Point Clouds Autonomous Driving

Ask PDF