UP-Fuse: Uncertainty-guided LiDAR-Camera Fusion for 3D Panoptic Segmentation

Rohit Mohan; Florian Drews; Yakov Miron; Daniele Cattaneo; Abhinav Valada

cs.CVFeb 22, 2026

UP-Fuse: Uncertainty-guided LiDAR-Camera Fusion for 3D Panoptic Segmentation

Authors: Rohit Mohan, Florian Drews, Yakov Miron, Daniele Cattaneo, Abhinav Valada

Organizations: University of Freiburg, Germany · Robert Bosch GmbH, Germany · University of Haifa

Abstract

LiDAR-camera fusion enhances 3D panoptic segmentation by leveraging camera images to complement sparse LiDAR scans, but it also introduces a critical failure mode. Under adverse conditions, degradation or failure of the camera sensor can significantly compromise the reliability of the perception system. To address this problem, we introduce UP-Fuse, a novel uncertainty-aware fusion framework in the 2D range-view that remains robust under camera sensor degradation, calibration drift, and sensor failure. Raw LiDAR data is first projected into the range-view and encoded by a LiDAR encoder, while camera features are simultaneously extracted and projected into the same shared space. At its core, UP-Fuse employs an uncertainty-guided fusion module that dynamically modulates cross-modal interaction using predicted uncertainty maps. These maps are learned by quantifying representational divergence under diverse visual degradations, ensuring that only reliable visual cues influence the fused representation. The fused range-view features are decoded by a novel hybrid 2D-3D transformer that mitigates spatial ambiguities inherent to the 2D projection and directly predicts 3D panoptic segmentation masks. Extensive experiments on Panoptic nuScenes, SemanticKITTI, and our introduced Panoptic Waymo benchmark demonstrate the efficacy and robustness of UP-Fuse, which maintains strong performance even under severe visual corruption or misalignment, making it well suited for robotic perception in safety-critical settings.

Figures & tables

Fig. 1 : Visualization of 3D panoptic segmentation: green indicates correct and red denotes errors. LiDAR-camera fusion significantly enhances segmentation over LiDAR-only methods, accurately detecting previously missed vehicles. However, camera sensor failure scenarios reveal a critical vulnerability, with fusion-based performance falling below LiDAR-only baselines and failing to detect previously identified objects. This highlights the crucial need for both relevance and reliability in multi-modal perception.

Fig. 2 : Illustration of the proposed UP-Fuse architecture. LiDAR and multi-view camera images are fused onto a shared space of range-view feature representations. The Uncertainty-Aware Fusion Module adaptively integrates modalities via uncertainty-weighted deformable cross-modal interaction to attenuate unreliable visual cues. Finally, a Hybrid 2D-3D Panoptic Decoder generates 3D predictions. Paths and blocks shown in brown are used only during training.

Panel of Figure 3 — Fig. 3 : Illustration of our uncertainty module on a Panoptic nuScenes sample. The right column shows predicted uncertainty under increasing synthetic degradations. The jet colormap marks low uncertainty in blue and high uncertainty in red . Mild distortions (b) keep uncertainty low, while strong distortions (c) and sensor dropout (d) produce high uncertainty. Black regions indicate areas without a camera to range-view (RV) mapping.

Method	Modality	PQ	PQ †	SQ	RQ	PQ ${}^{\text{th}}$	PQ ${}^{\text{st}}$	FPS
CPSeg HR [ 14 ]	L	71.1	75.6	82.5	85.5	71.5	70.6	—
Panoptic-FusionNet [ 39 ]	L	72.7	75.4	86.4	84.8	71.2	75.1	—
LCPS [ 50 ]	L	72.9	77.6	88.4	82.0	72.8	73.0	—
Panoptic-PHNet [ 15 ]	L	74.7	77.7	88.2	84.2	74.0	75.9	—
PUPS [ 40 ]	L	74.7	77.3	89.4	83.3	75.4	73.6	—
UP-Fuse (Ours)	L	74.9	78.2	87.8	85.1	75.1	74.5	—

TABLE I : Comparison of 3D panoptic segmentation performance on the Panoptic nuScenes validation set.

Method	Modality	PQ	PQ †	SQ	RQ	PQ ${}^{\text{th}}$	PQ ${}^{\text{st}}$
MaskPLS [ 21 ]	L	61.1	64.3	86.8	68.5	54.3	72.4
EfficientLPS [ 37 ]	L	62.4	66.0	83.7	74.1	57.2	71.1
Panopitc-PolarNet [ 51 ]	L	63.6	67.1	84.3	75.1	59.0	71.3
LCPS [ 50 ]	L	72.8	76.3	88.6	81.7	72.4	73.5
CPSeg [ 14 ]	L	73.2	76.3	88.1	82.7	72.9	74.0
Panoptic-PHNet [ 51 ]	L	80.1	82.8	91.1	87.6	82.1	76.6

TABLE II : Comparison of 3D panoptic segmentation results on the Panoptic nuScenes test set.

Method	Modality	PQ	PQ †	SQ	RQ
LCPS [ 50 ]	L	55.7	65.2	65.8	74.0
EfficientLPS [ 37 ]	L	59.2	65.1	69.8	75.0
UP-Fuse (Ours)	L	60.4	67.8	74.9	71.7
Panoptic-PHNet [ 15 ]	L	61.7	—	—	—
IAL [ 32 ]	L	62.0	65.1	76.0	71.9
CenterLPS [ 22 ]	L	62.1	67.0	72.0	80.7

TABLE III : Comparison of 3D panoptic segmentation results on the SemanticKITTI validation set.

Method	Modality	PQ	PQ †	SQ	RQ	PQ ${}^{\text{th}}$	PQ ${}^{\text{st}}$
Panoptic-FusionNet [ 39 ]	L	53.9	63.2	77.4	68.2	55.1	53.4
LCPS [ 50 ]	L	54.9	63.1	77.8	69.2	58.2	53.6
EfficientLPS [ 37 ]	L	56.3	64.5	78.3	70.6	59.7	54.9
UP-Fuse (Ours)	L	57.4	65.1	78.8	71.5	61.3	55.8
IAL [ 32 ]	L	58.2	66.1	79.2	72.1	62.5	56.4
P3Former [ 45 ]	L	59.3	67.4	79.3	73.5	63.8	57.5

TABLE IV : Comparison of 3D panoptic segmentation results on the Panoptic Waymo validation set.

Method	Modality	PQ	PQ ${}^{\text{th}}$	PQ ${}^{\text{st}}$	$\Delta$ PQ
Panoptic-FusionNet [ 39 ]	L	72.7	71.2	75.1	—
LCPS [ 50 ]	L	72.9	72.8	73.0	—
UP-Fuse (Ours)	L	74.9	75.1	74.5	—
IAL [ 32 ]	L	77.0	77.8	75.7	—
Panoptic-FusionNet [ 39 ]	LC	76.6	77.1	76.0	+3.9
LCPS [ 50 ]	LC	79.3	81.7	75.3	+6.4

TABLE V : Robust performance comparison on the Panoptic nuScenes validation set under missing camera modality conditions.

\Delta

PQ denotes the change compared to the corresponding LiDAR-only variant.

Fig. 4 : Robust performance comparison on the Panoptic nuScenes validation set under increasing calibration drift between LiDAR and camera over rotation magnitudes from

0^{\circ}

5^{\circ}

. UP-Fuse (red) outperforms all baselines, dropping only

4.4\%

in PQ compared to

>8\%

for state-of-the-art methods. Refer to the supplementary material for visualization of the projection shifts.

Method	Modality	PQ	PQ ${}^{\text{th}}$	PQ ${}^{\text{st}}$	$\Delta$ PQ
Panoptic-FusionNet [ 39 ]	L	63.1	61.3	64.8	—
LCPS [ 50 ]	L	63.3	63.1	63.4	—
UP-Fuse (Ours)	L	64.2	63.9	64.4	—
IAL [ 32 ]	L	65.6	65.5	65.7	—
Panoptic-FusionNet [ 39 ]	LC	60.0	56.2	63.7	-3.1
LCPS [ 50 ]	LC	60.6	58.3	62.9	-2.7

TABLE VI : Comparison of robustness to visual domain shift on the Panoptic nuScenes validation set. Models trained on day scenes are evaluated on the night split to assess performance under corrupted visual features.

Model Variant	Modality	PQ	PQ ${}^{\text{th}}$	PQ ${}^{\text{st}}$
M1 (2D-3D Post Processing [ 24 ] )	L	72.8	72.2	73.8
M2 (Hybrid 2D-3D Decoder)	L	74.9	75.1	74.5
M2 + Concat Fusion (VT- $\mathcal{D}_{\text{sparse}}$ )	LC	76.4	77.2	75.1
M2 + Concat Fusion (VT- $\mathcal{D}_{\text{dense}}$ )	LC	77.2	78.4	75.2
M2 + D-CMI (VT- $\mathcal{D}_{\text{dense}}$ )	LC	79.6	81.6	76.3
+ Sensor Degradation Augs.	LC	78.9	80.7	75.9

TABLE VII : Ablation study of our UP-Fuse architecture on the Panoptic nuScenes validation set. VT denotes View-Transformation. D-CMI denotes Deformable Cross-Modal Interaction. Augs. denotes Augmentations.

Panel of Figure 12 — Fig. 5 : Ablation studies on the key hyperparameters of our Hybrid 2D-3D Panoptic Decoder.

Degradation Type	$\rho(d_{\text{gt}},d_{\text{pred}})$
Brightness Shift	0.6907
Color Jitter	0.6369
Out-of-Domain	0.7802
Fog	0.7258
Sensor Dropout	0.9595

TABLE VIII : Pearson correlation

\rho(d_{\text{gt}},d_{\text{pred}})

under camera sensor degradation on the Panoptic nuScenes validation set. The predicted instability

d_{\text{pred}}

is consistently positively correlated with the ground-truth feature instability

d_{\text{gt}}

, with the strongest correlation under sensor dropout.

Panel of Figure 14 — Fig. 6 : Qualitative robustness comparison of 3D panoptic segmentation between UP-Fuse and the strongest baseline IAL on the Panoptic nuScenes validation set. (a) IAL fails to resolve geometric ambiguity, misclassifying concrete barriers as man-made structures. (b) Under calibration drift, IAL incorrectly merges two trucks into a single instance and mislabels vehicle tops as background. (c) Nighttime domain shift leads to similar errors. In contrast, UP-Fuse (right) suppresses unreliable visual cues and maintains correct semantic and instance predictions across all scenarios. red : incorrect prediction, blue : correct prediction (Best viewed at $4\times$ zoom).

Panel of Figure 15 — Fig. 7 : Visualization of LiDAR-to-camera projection shifts under increasing calibration drift (rotation magnitude) from $0^{\circ}$ to $5^{\circ}$ , referenced in Fig. 4 of the main paper.

Panel of Figure 16 — Fig. 8 : Dataset statistics of Panoptic Waymo.

Fig. 9 : Examples from the Panoptic Waymo dataset showing the multi-view camera images, corresponding LiDAR scan, and their panoptic ground truth. For clarity, the semantic and instance components of the panoptic labels are visualized separately.

Fig. 10 : Qualitative 3D Panoptic Segmentation results of our proposed UP-Fuse network versus the baseline LCPS architecture, on the Panoptic Waymo val set. red : incorrect prediction, blue : correct prediction.

Panel of Figure 19 — Fig. 11 : Visualization of 3D panoptic segmentation predictions of UP-Fuse on real-world scenes. A model trained on the Panoptic Waymo dataset is deployed on sensor inputs from our in-house autonomous vehicle, highlighting its behavior under domain shift, differences in sensor configuration, and missing camera views compared to the original Panoptic Waymo setup.

Fig. 12 : Our in-house autonomous driving vehicle used to demonstrate the robustness of the UP-Fuse architecture on real-world scenes.

Camera-LiDAR Fusion

Camera–LiDAR fusion combines image semantics with LiDAR geometry for 3D perception, depth estimation, localization, and mapping.

See all 106 papers

Post Fusion Bird's Eye View Feature Stabilization for Robust Multimodal 3D Detection
Mar 5, 2026Trung Tien Dong, Dev Thakkar, Arman Sargolzaei +1Multimodal Robustness 3D Object Detection
College of Engineering, University of South Florida, Tampa, FL 33620, USA
EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation
Jun 21, 2021Mingkui Tan, Zhuangwei Zhuang, Sitao Chen +4Camera-LiDAR Fusion Autonomous Driving Perception
School of Software Engineering, South China University of Technology · Pazhou Laboratory, Guangzhou, China · School of Electronic and Information Engineering, South China University of Technology +2
SARFusion: Scene-Aware Routing Fusion for Robust Camera-LiDAR 3D Object Detection
Sep 24, 2026Yuting Zhao, Ziyi Zheng, Shuxiao LiCamera-LiDAR Fusion 3D Object Detection
Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Wuhan College, Wuhan, China

Mar 5, 2026J/K move · Enter open · S save

Post Fusion Bird's Eye View Feature Stabilization for Robust Multimodal 3D Detection

Trung Tien Dong, Dev Thakkar, Arman Sargolzaei, Xiaomin Lin

College of Engineering, University of South Florida, Tampa, FL 33620, USA

Multimodal Robustness 3D Object Detection Camera-LiDAR Fusion Distribution Shift

Open paper AskPDF arXiv

Camera-LiDAR fusion is widely used in autonomous driving to enable accurate 3D object detection. However, bird's-eye view (BEV) fusion detectors can degrade significantly under domain shift and sensor failures, limiting reliability in real-world deployment. Existing robustness approaches often require modifying the fusion architecture or retraining specialized models, making them difficult to integrate into already deployed systems. We propose a Post Fusion Stabilizer (PFS), a lightweight module that operates on intermediate BEV representations of existing detectors and produces a refined feature map for the original detection head. The design stabilizes feature statistics under domain shift, suppresses spatial regions affected by sensor degradation, and adaptively restores weakened cues through residual correction. Designed as a near-identity transformation, PFS preserves performance while improving robustness under diverse camera and LiDAR corruptions. Evaluations on the nuScenes benchmark demonstrate that PFS achieves state-of-the-art results in several failure modes, notably improving camera dropout robustness by +1.2% and low-light performance by +4.4% mAP while maintaining a lightweight footprint of only 3.3 M parameters.

Abstract

Camera-LiDAR Fusion

Explore similar work

Post Fusion Bird's Eye View Feature Stabilization for Robust Multimodal 3D Detection

EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation

SARFusion: Scene-Aware Routing Fusion for Robust Camera-LiDAR 3D Object Detection