SpikeOcc: Towards Resilient Multimodal 3D Occupancy Perception via Spiking Experts

Yiding Sun; Zijie Xu; Hongqiang Lin; Jiajun Cao; Dongxu Zhang; Kai Tang; Jihua Zhu; Wenxuan Liu

cs.CVOct 11, 2026

SpikeOcc: Towards Resilient Multimodal 3D Occupancy Perception via Spiking Experts

Authors: Yiding Sun, Zijie Xu, Hongqiang Lin, Jiajun Cao, Dongxu Zhang, Kai Tang, Jihua Zhu, Wenxuan Liu, +2 more

Organizations: Peking University · Xi’an Jiaotong University · Zhejiang University · Tsinghua University

Abstract

Multimodal 3D occupancy prediction underpins safe autonomous driving through comprehensive scene understanding. However, existing methods have largely pursued higher accuracy through sophisticated fusion, leaving resilience to missing sensors underexplored. In this paper, we investigate spiking neural networks (SNNs) as a means of building more resilient multimodal perception. Still, the discrete nature of spiking neurons makes optimization challenging. To overcome this difficulty, SpikeOcc is introduced with two complementary components, a Spiking Mixture-of-Experts (SMoE) and Sensor Availability Learning (SAL). Specifically, SMoE organizes RGB, LiDAR, and joint fusion into specialized pathways, enabling full-width unimodal representations while retaining the original multimodal features via a bounded residual. Beyond sensor dropout, SAL employs a staged curriculum with explicit routing supervision, progressively improving robustness under severe sensing conditions. Remarkably, SAL also yields notable gains for conventional models, highlighting its broader potential for resilient multimodal learning. Extensive experiments demonstrate that SpikeOcc maintains competitive full-sensor accuracy while markedly raising the performance floor under sensor outages. SpikeOcc illustrates the promise of neuromorphic computing, pointing toward more reliable and energy-efficient autonomous driving.

Figures & tables

Figure 1: Full-sensor accuracy does not guarantee resilience to sensor failures. Existing MMOMs degrade sharply under modality outages, whereas SpikeOcc remains more resilient.

Figure 2: Naive spiking fusion exhibits instability. Firing balance is the lower-rate modality’s normalized share and reaches 50% for equal rates.

Figure 3: Overview of SpikeOcc. Masked camera and LiDAR inputs are encoded and processed by specialized experts. Router-weighted features pass through the spiking fusion module and occupancy head, while SAL progressively introduces missing-sensor cases during training.

Method	Input Modality	IoU	mIoU	barrier	bicycle	bus	car	const. veh.	motorcycle	pedestrian	traffic cone	trailer	truck	drive surf.	other_flat	sidewalk	terrain	manmade	vegetation
All Sensors Available
MonoScene ( Cao and de Charette, 2022 )	C	18.4	6.9	7.1	3.9	9.3	7.2	5.6	3.0	5.9	4.4	4.9	4.2	14.9	6.3	7.9	7.4	10.0	7.6
JS3C-Net ( Yan et al., 2021 )	L	30.2	12.5	14.2	3.4	13.6	12.0	7.2	4.3	7.3	6.8	9.2	9.1	27.9	15.3	14.9	16.2	14.0	24.9
3DSketch ( Chen et al., 2020 )	C&D	25.6	10.7	12.0	5.1	10.7	12.4	6.5	4.0	5.0	6.3	8.0	7.2	21.8	14.8	13.0	11.8	12.0	21.2
M-CONet ( Wang et al., 2023b )	C&L	29.5	20.1	23.3	13.3	21.2	24.3	15.3	15.9	18.0	13.3	15.3	20.7	33.2	21.0	22.5	21.5	19.6	23.2
Co-Occ ( Pan et al., 2024a )	C&L	30.6	21.9	26.5	16.8	22.3	27.0	10.1	20.9	20.7	14.5	16.4	21.6	36.9	23.5	5.5	23.7	20.5	23.5

Table 1: Quantitative comparisons on the OpenOccupancy validation set. C, D, and L denote camera, depth, and LiDAR, respectively. Within each sensor setting, the best , second-best , and remaining results are shown in bold, underlined, and gray, respectively. For fair comparison, results marked with

\dagger

are reproduced by us and averaged over 3 seeds. Same for subsequent tables.

Method	Input Modality	IoU	mIoU	road	sidewalk	parking	other ground	building	car	truck	bicycle	motorcycle	other vehicle	vegetation	trunk	terrain	person	bicyclist	motorcyclist	fence	pole	traffic sign
All Sensors Available
SurroundOcc ( Wei et al., 2023 )	C	34.7	11.9	56.9	28.3	30.2	6.8	15.2	20.6	1.4	1.6	1.2	4.4	14.9	3.4	19.3	1.4	2.0	0.1	11.3	3.9	2.4
OccFormer ( Zhang et al., 2023 )	C	34.5	12.3	55.9	30.3	31.5	6.5	15.7	21.6	1.2	1.5	1.7	3.2	16.8	3.9	21.3	2.2	1.1	0.2	11.9	3.8	3.7
RenderOcc ( Pan et al., 2024b )	C	N/A	12.8	57.2	28.4	16.1	0.9	18.2	24.9	6.0	0.4	0.3	3.7	26.2	4.9	3.6	1.9	3.1	0.0	9.1	6.2	3.4
M-CONet ( Wang et al., 2023b )	C&L	N/A	20.4	60.6	36.1	29.0	13.0	38.4	33.8	4.7	3.0	2.2	5.9	41.5	20.5	35.1	0.8	2.3	0.6	26.0	18.7	15.7
OccGen † ( Wang et al., 2024a )	C&L	59.9	21.1	63.1	33.0	17.3	0.3	38.1	44.1	17.4	3.8	2.9	7.9	41.4	20.5	46.6	4.1	2.1	0.0	13.8	28.6	16.2

Table 2: Performance on the SemanticKITTI test set. Within each sensor setting, the best , second-best , and remaining results are shown in bold, underlined, and gray, respectively.

Figure 6: Generalization to unseen sensor configurations. B: Back, BL: Back left, BR: Back right.

Table 8

Figure 9: More qualitative comparison with OccMamba. SpikeOcc preserves more complete scene geometry and semantics under sensor outages. Numbers denote (IoU, mIoU).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Full		All Cam. Missing		LiDAR Missing
Method	IoU	mIoU	IoU	mIoU	IoU	mIoU
OccMamba + SAL	36.2 (+0.5)	26.4 (+0.2)	29.3 (+10.5)	16.1 (+6.7)	19.2 (+11.4)	8.5 (+6.9)
SPSC + SAL	35.4 (+0.2)	26.2 (+0.1)	26.7 (+4.9)	13.9 (+5.2)	16.5 (+16.3)	6.0 (+5.9)
SpikeOcc	35.8 (+0.6)	25.9 (+1.2)	36.8 (+5.3)	22.7 (+7.1)	19.7 (+2.3)	9.2 (+1.1)

Appendix

Table 9: Comparison with SAL-equipped baselines on OpenOccupancy. Values in parentheses denote the change relative to the corresponding model trained without SAL.

Explore similar work

$M^2$ -Occ: Resilient 3D Semantic Occupancy Prediction for Autonomous Driving with Incomplete Camera Inputs
Mar 10, 2026Kaixin Lin, Kunyu Peng, Di Wen +3Multi-View Learning Autonomous Driving Perception
School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China · Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany
Neuromorphic LiDAR-based Bird's Eye View Object Detection using Energy-efficient Spiking Neural Networks
May 24, 2026Sambit Mohapatra, Senthil Yogamani, Heinrich Gotzig +1Lidar-Based 3D Object Detection Autonomous Driving Perception
Valeo, Germany · TU Ilmenau, Germany · Valeo, Ireland
VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction
Mar 3, 2026A. Enes Doruk, Hasan F. Ates3D Occupancy Prediction Autonomous Driving Perception

Mar 10, 2026J/K move · Enter open · S save

$M^2$ -Occ: Resilient 3D Semantic Occupancy Prediction for Autonomous Driving with Incomplete Camera Inputs

Kaixin Lin, Kunyu Peng, Di Wen, Yufan Chen+2

School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China · Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany

Multi-View Learning Autonomous Driving Perception 3D Occupancy Prediction

Open paper AskPDF arXiv

Semantic occupancy prediction enables dense 3D geometric and semantic understanding for autonomous driving. However, existing camera-based approaches implicitly assume complete surround-view observations, an assumption that rarely holds in real-world deployment due to occlusion, hardware malfunction, or communication failures. We study semantic occupancy prediction under incomplete multi-camera inputs and introduce

M^2

-Occ, a framework designed to preserve geometric structure and semantic coherence when views are missing.

M^2

-Occ addresses two complementary challenges. First, a Multi-view Masked Reconstruction (MMR) module leverages the spatial overlap among neighboring cameras to recover missing-view representations directly in the feature space. Second, a Feature Memory Module (FMM) introduces a learnable memory bank that stores class-level semantic prototypes. By retrieving and integrating these global priors, the FMM refines ambiguous voxel features, ensuring semantic consistency even when observational evidence is incomplete. We introduce a systematic missing-view evaluation protocol on the nuScenes-based SurroundOcc benchmark, encompassing both deterministic single-view failures and stochastic multi-view dropout scenarios. Under the safety-critical missing back-view setting,

M^2

-Occ improves the IoU by 4.36%. As the number of missing cameras increases, the robustness gap further widens; for instance, under the setting with five missing views, our method boosts the IoU by 6.67%. These gains are achieved without compromising full-view performance. The source code will be publicly released at https://github.com/qixi7up/M2-Occ.

Abstract

Explore similar work

M2M^2M2-Occ: Resilient 3D Semantic Occupancy Prediction for Autonomous Driving with Incomplete Camera Inputs

Neuromorphic LiDAR-based Bird's Eye View Object Detection using Energy-efficient Spiking Neural Networks

VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction

$M^2$ -Occ: Resilient 3D Semantic Occupancy Prediction for Autonomous Driving with Incomplete Camera Inputs