VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction

A. Enes Doruk; Hasan F. Ates

cs.CVMar 3, 2026

VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction

Authors: A. Enes Doruk, Hasan F. Ates

Abstract

This paper introduces VLMFusionOcc3D, a robust multimodal framework for dense 3D semantic occupancy prediction in autonomous driving. Current voxel-based occupancy models often struggle with semantic ambiguity in sparse geometric grids and performance degradation under adverse weather conditions. To address these challenges, we leverage the rich linguistic priors of Vision-Language Models (VLMs) to anchor ambiguous voxel features to stable semantic concepts. Our framework initiates with a dual-branch feature extraction pipeline that projects multi-view images and LiDAR point clouds into a unified voxel space. We propose Instance-driven VLM Attention (InstVLM), which utilizes gated cross-attention and LoRA-adapted CLIP embeddings to inject high-level semantic and geographic priors directly into the 3D voxels. Furthermore, we introduce Weather-Aware Adaptive Fusion (WeathFusion), a dynamic gating mechanism that utilizes vehicle metadata and weather-conditioned prompts to re-weight sensor contributions based on real-time environmental reliability. To ensure structural consistency, a Depth-Aware Geometric Alignment (DAGA) loss is employed to align dense camera-derived geometry with sparse, spatially accurate LiDAR returns. Extensive experiments on the nuScenes and SemanticKITTI datasets demonstrate that our plug-and-play modules consistently enhance the performance of state-of-the-art voxel-based baselines. Notably, our approach achieves significant improvements in challenging weather scenarios, offering a scalable and robust solution for complex urban navigation.

Figures & tables

Fig. 1: Comparison of feature fusion architectures. Standard approaches include Addition Fusion for element-wise summation, Concatenation Fusion for channel-wise stacking, and 3D Convolution Fusion for learned spatial refinement. Our proposed WeathFusion introduces a language-aware gating mechanism. It utilizes a frozen CLIP encoder to process environmental descriptions, modulating input features through learned gating weights before final summation.

Fig. 2: Overall architecture of VLMFusionOcc3D. Our pipeline integrates multi-view camera images and LiDAR point clouds through a dual-branch architecture. High-level semantic priors are injected via Instance-driven VLM Attention (InstVLM), followed by dynamic modality integration using the Weather-Aware Adaptive Fusion (WeathFusion) module. The final dense occupancy grid is optimized using the Depth-Aware Geometric Alignment (DAGA) loss.

Fig. 3: Architecture of the Instance-driven VLM Attention (InstVLM) module. The module employs a gated cross-attention mechanism to anchor 3D voxel features to continuous text embeddings from a LoRA-adapted CLIP encoder. The gating mechanism ensures that semantic information is selectively fused into spatial voxels.

Fig. 4: Examples of structured instance prompts. These prompts encapsulate category-specific information and geographic context to provide the VLM with rich environmental priors. During inference, a recursive strategy is employed to maintain temporal semantic stability.

Fig. 5: Detailed schematic of the Weather-Aware Adaptive Fusion (WeathFusion) module. The gating head processes weather context prompts to compute dynamic reliability weights for each modality. This process enables the framework to robustly transition between sensors in response to environmental degradation, such as LiDAR scattering or low-light camera noise.

Fig. 6: Environmental context prompts for weather adaptation. These prompts represent various operational domains. These embeddings supervise the gating head to prioritize robust sensors, such as favoring LiDAR in low-visibility or Cameras in clear conditions.

Method	Input Modality	IoU	mIoU	barrier	bicycle	bus	car	const. veh.	motorcycle	pedestrian	traffic cone	trailer	truck	drive surf.	other_flat	sidewalk	terrain	manmade	vegetation
MonoScene [ 22 ]	C	18.4	6.9	7.1	3.9	9.3	7.2	5.6	3.0	5.9	4.4	4.9	4.2	14.9	6.3	7.9	7.4	10.0	7.6
TPVFormer [ 23 ]	C	15.3	7.8	9.3	4.1	11.3	10.1	5.2	4.3	5.9	5.3	6.8	6.5	13.6	9.0	8.3	8.0	9.2	8.2
SparseOcc [ 24 ]	C	21.8	14.1	16.1	9.3	15.1	18.6	7.3	9.4	11.2	9.4	7.2	13.0	31.8	21.7	20.7	18.8	6.1	10.6
3DSketch [ 25 ]	C&D	25.6	10.7	12.0	5.1	10.7	12.4	6.5	4.0	5.0	6.3	8.0	7.2	21.8	14.8	13.0	11.8	12.0	21.2
AICNet [ 26 ]	C&D	23.8	10.6	11.5	4.0	11.8	12.3	5.1	3.8	6.2	6.0	8.2	7.5	24.1	13.0	12.8	11.5	11.6	20.2
LMSCNet [ 27 ]	L	27.3	11.5	12.4	4.2	12.8	12.1	6.2	4.7	6.2	6.3	8.8	7.2	24.2	12.3	16.6	14.1	13.9	22.2

TABLE I: Quantitative comparisons on the nuScenes-OpenOccupancy validation set. C, D, L denote camera, depth and LiDAR, respectively. The best and second-best are in bold and underlined, respectively.

Fig. 7: Qualitative results under adverse weather and lighting conditions. We compare the baseline OccMamba (middle) with our VLMFusionOcc3D (middle) across rainy and nighttime scenes from the nuScenes-OpenOccupancy dataset. Better viewed when zoomed in.

Method	Input Modality	mIoU
MonoScene [ 22 ]	C	11.1
SurroundOcc [ 32 ]	C	11.9
OccFormer [ 33 ]	C	12.3
RenderOcc [ 34 ]	C	12.8
LMSCNet [ 27 ]	L	17.0
JS3C-Net [ 28 ]	L	23.8

TABLE II: Performance on SemanticKITTI test set. The best and second-best are in bold and underlined, respectively.

InstVLM	WeathFusion	DAGA	IoU	mIoU
			34.7	25.2
✓			35.6	26.6
	✓		35.1	26.2
		✓	34.9	25.7
✓	✓		36.6	26.0
✓	✓	✓	37.0	26.6

TABLE III: Ablation study of different components on the nuScenes-OpenOccupancy validation set with the OccMamba baseline.

Method	IoU	mIoU	Latency (ms)
Addition	34.2	24.9	0.02
Concatenation	34.7	25.2	0.03
3D Convolution	34.6	25.0	0.65
ACLF [ 15 ]	35.6	25.5	3.21
WeathFusion (Ours)	37.0	26.6	2.14

TABLE IV: Performance analysis of different fusion techniques on nuScenes-OpenOccupancy validation set with the OccMamba baseline.

Method	Memory Usage (GiB)		mIoU
Method	Training	Inference	(%)
M-CONet [ 8 ]	37.3	12.1	20.1
M-CONet [ 8 ] + Ours	38.9	12.7	22.4
OccMamba [ 7 ]	23.1	6.8	25.2
OccMamba [ 29 ] + Ours	24.7	7.3	26.6

TABLE V: Efficiency Analysis on nuScenes-OpenOccupancy Validation Set.

		IoU $\uparrow$			mIoU $\uparrow$
Method	Modality	Rainy	Day	Night	Rainy	Day	Night
OccMamba [ 7 ]	C&L	28.3	36.8	12.6	24.1	26.3	11.8
OccMamba [ 7 ] + Ours	C&L	32.1	40.2	20.3	29.3	27.8	17.3

TABLE VI: Performance comparison on the nuScenes-OpenOccupancy validation set under adverse conditions.

Explore similar work

May 15, 2026

WeatherOcc3D: VLM-Assisted Adverse Weather Aware 3D Semantic Occupancy Prediction

While multi-modal 3D semantic occupancy prediction typically enhances robustness by fusing camera and LiDAR inputs, its effectiveness is fundamentally constrained by environmental variability. Specifically, camera sensors suffer from severe low-light degradation, while LiDAR sensors encounter significant backscatter noise during heavy precipitation. These adverse conditions create a modality trust problem, as static fusion strategies fail to adaptively re-weight inputs when a specific sensor becomes unreliable. To address this, we propose a VLM-assisted framework leveraging the pre-trained CLIP latent space to guide multi-sensor integration via linguistic environmental cues. We utilize a parameter-efficient adapter to align weather-specific text embeddings with sensor features, coupled with a gating strategy that decomposes environmental uncertainty into two factors: visibility and illumination. This enables the model to dynamically modulate the fusion ratio - prioritizing semantic camera features in clear daylight and shifting to geometric LiDAR priors during rainy nights. Evaluations on the nuScenes dataset demonstrate the versatility of our approach, as implementing our proposed framework on the OccMamba and M-CONet architectures achieves mIoU scores of 26.3 and 21.1, respectively, significantly outperforming their traditional baselines.

A. Enes Doruk, Abdelaziz Hussein, Hasan F. Ates

Department of Artificial Intelligence and Data Engineering Ozyegin University Istanbul, Türkiye

Multimodal Robustness Camera-LiDAR Fusion 3D Occupancy Prediction Multimodal Fusion

Ask PDF

Mar 24, 2026

Gau-Occ: Geometry-Completed Gaussians for Multi-Modal 3D Occupancy Prediction

3D semantic occupancy prediction is crucial for autonomous driving. While multi-modal fusion improves accuracy over vision-only methods, it typically relies on computationally expensive dense voxel or BEV tensors. We present Gau-Occ, a multi-modal framework that bypasses dense volumetric processing by modeling the scene as a compact collection of semantic 3D Gaussians. To ensure geometric completeness, we propose a LiDAR Completion Diffuser (LCD) that recovers missing structures from sparse LiDAR to initialize robust Gaussian anchors. Furthermore, we introduce Gaussian Anchor Fusion (GAF), which efficiently integrates multi-view image semantics via geometry-aligned 2D sampling and cross-modal alignment. By refining these compact Gaussian descriptors, Gau-Occ captures both spatial consistency and semantic discriminability. Extensive experiments across challenging benchmarks demonstrate that Gau-Occ achieves state-of-the-art performance with significant computational efficiency.

Chengxin Lv, Yihui Li, Hongyu Yang +1

State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China · School of Artificial Intelligence, Beihang University, Beijing, China

3D Occupancy Prediction Camera-LiDAR Fusion 3D Gaussian Splatting

Ask PDF

Jun 30, 2026

Semantic Occupancy Prediction with Dual Range-Voxel Representation

LiDAR-based 3D semantic occupancy prediction, which aims to provide accurate and comprehensive scene representation, is crucial for autonomous driving systems. As point clouds suffer from sparsity and incompleteness, leading to insufficient semantic learning and difficult occupancy perception, existing methods often stack multi-sweep point clouds to obtain dense spatial information. However, such a naive strategy also results in efficiency (e.g., additional computational burden) and robustness (e.g., pose transformation noise) concerns, which hinder their practical applications. In this work, we propose a Dual Range-Voxel Representation (DRVR) that leverages the range-view context and voxel-view geometry of single-sweep point clouds for 3D semantic occupancy prediction, eliminating the concerns associated with the multi-sweeps. Specifically, we use the range-view encoder to extract the compact context of the scene. To fully exploit the spatial information, we design a geometry-aware voxel-view encoder that extracts multi-scale voxel-view features separately and combines them for better geometric occupancy prediction. Moreover, we propose a range-voxel fusion module to cooperate range- and voxel-view features via voxel-to-range and range-to-voxel fusions. Extensive experiments on nuScenes-Occupancy, SemanticKITTI and SemanticPOSS show the superiority of our method. Especially on nuScenes-Occupancy, our single-sweep DRVR achieves 5.4% improvement in mIoU and 2.1x acceleration compared to the multi-sweep method.

Sitao Chen, Zhuangwei Zhuang, Hui Luo +3

School of Software Engineering, South China University of Technology, Guangzhou 510006, China · State Key Laboratory of Optical Field Manipulation Science and Technology, Institute of Optics and Electronics, CAS, Chengdu 610209, China

Autonomous Driving Perception Semantic Occupancy Mapping 3D Occupancy Prediction Feature Fusion

Ask PDF