Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation

Hu Cao; Yifeng Cheng; Mu He; Mengyu Li; Yinlong Liu; Alois Knoll

cs.CVOct 13, 2026

Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation

Authors: Hu Cao, Yifeng Cheng, Mu He, Mengyu Li, Yinlong Liu, Alois Knoll

Organizations: School of Automation, Southeast University, Sipailou 2, Nanjing, 210096, China. · Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Boltzmannstraße 15, Munich, 85748, Germany.

Abstract

Multi-modal semantic segmentation adds an auxiliary sensing stream to RGB images (RGB-X), such as thermal, LiDAR, depth, event, polarization, or light field. This can improve robustness in poor lighting and under motion blur. But different sensors often produce features that do not match well, and fusion can become unstable. To alleviate these issues, we propose a bidirectional polarity-aware cross-modality fusion (BPCF) module that effectively captures complementary cues while enhancing feature alignment. We evaluate the framework on six modality pairings: RGB-Thermal, RGB-LiDAR, RGB-Depth, RGB-Event, RGB-Polarization, and RGB-Light Field. Our model achieve state-of-the-art (SOTA) results on eight public datasets, including MFNet, KITTI-360, DELIVER, DDD17, DSEC, MCubeS, ZJU, and UrbanLF. The codes and trained models will be released upon acceptance.

Figures & tables

Figure 1: Performance comparison between our proposed method (Ours) and CMNeXt Zhang et al. (2023) across different modality combinations, including RGB-Thermal (MFNet), RGB-LiDAR (KITTI360), RGB-Depth (DELIVER), RGB-Event (DSEC and DDD17), RGB-Polarization (MCubeS), and RGB-Light Field (UrbanLF-Real and UrbanLF-Syn).

Figure 2: Overview of the proposed multi-modal semantic segmentation framework. Each modality is independently encoded using a dedicated Transformer backbone. RGB and auxiliary modality features are then progressively fused through the proposed fusion module in a stage-by-stage manner. Finally, the fused multi-scale features are passed to the segmentation head to generate predictions.

Figure 3: The overall structure of the BPCF module. It consists of a bidirectional polarity-aware linear cross-attention (BPLCA) part and a dual feature consistency constraint (DFCC) part.

Figure 4: Illustration of BPLCA. Features from RGB and auxiliary modality are deeply fused through the BPLCA component.

Figure 5: Illustration of DFCC. The features from BPLCA are further refined to obtain the final fused output.

Figure 6: Qualitative results across multiple modalities. Our model consistently produces sharper boundaries and more accurate object recognition under challenging conditions.

Figure 7: BPLCA component visualization. RGB-Thermal sample selected from MFNet dataset Ha et al. (2017) while RGB-Event sample from DELIVER dataset Zhang et al. (2023) .

Figure 8: Here is the visualization of ScoreNet on ZJU dataset Xiang et al. (2021) . Red regions indicate higher modality selection scores, while blue regions indicate lower scores.

Explore similar work

SegRGB-X: General RGB-X Semantic Segmentation Model
Mar 30, 2026Jiong Liu, Yingjie Xu, Xingcheng Zhou +4Semantic Segmentation Multimodal Fusion
Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Munich, Germany
RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation
Apr 14, 2026Guoan Xu, Yang Xiao, Guangwei Gao +3Semantic Segmentation Multimodal Fusion
Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW 2007, Australia · PCA Lab, Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Bionic Vision Systems Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 200050, China +2
RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation
Mar 10, 2026Kunyu Tan, Mingjian LiangSemantic Segmentation Multimodal Robustness
Kunyu Tan and Mingjian Liang are independent researchers.

Mar 30, 2026J/K move · Enter open · S save

SegRGB-X: General RGB-X Semantic Segmentation Model

Jiong Liu, Yingjie Xu, Xingcheng Zhou, Rui Song+3

Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Munich, Germany

Semantic Segmentation Multimodal Fusion Cross-Modal Representation Learning

Open paper AskPDF arXiv

Semantic segmentation across arbitrary sensor modalities faces significant challenges due to diverse sensor characteristics, and the traditional configurations for this task result in redundant development efforts. We address these challenges by introducing a universal arbitrary-modal semantic segmentation framework that unifies segmentation across multiple modalities. Our approach features three key innovations: (1) the Modality-aware CLIP (MA-CLIP), which provides modality-specific scene understanding guidance through LoRA fine-tuning; (2) Modality-aligned Embeddings for capturing fine-grained features; and (3) the Domain-specific Refinement Module (DSRM) for dynamic feature adjustment. Evaluated on five diverse datasets with different complementary modalities (event, thermal, depth, polarization, and light field), our model surpasses specialized multi-modal methods and achieves state-of-the-art performance with a mIoU of 65.03%. The codes will be released upon acceptance.

Abstract

Explore similar work

SegRGB-X: General RGB-X Semantic Segmentation Model

RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation

RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation