RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation

Guoan Xu; Yang Xiao; Guangwei Gao; Dongchen Zhu; Guo-Jun Qi; Wenjing Jia

cs.CVApr 14, 2026

RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation

Authors: Guoan Xu, Yang Xiao, Guangwei Gao, Dongchen Zhu, Guo-Jun Qi, Wenjing Jia

Organizations: Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW 2007, Australia · PCA Lab, Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Bionic Vision Systems Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 200050, China · Research Center for Industries of the Future and the School of Engineering, Westlake University, Hangzhou 310024, China · OPPO Research, Seattle, WA 98101 USA

Abstract

Multimodal semantic segmentation has emerged as a powerful paradigm for enhancing scene understanding by leveraging complementary information from multiple sensing modalities (e.g., RGB, depth, and thermal). However, existing cross-modal fusion methods often implicitly assume that all modalities are equally reliable, which can lead to feature degradation when auxiliary modalities are noisy, misaligned, or incomplete. In this paper, we revisit cross-modal fusion from the perspective of modality reliability and propose a novel framework termed the Reliability-aware Self-Gated State Space Model (RSGMamba). At the core of our method is the Reliability-aware Self-Gated Mamba Block (RSGMB), which explicitly models modality reliability and dynamically regulates cross-modal interactions through a self-gating mechanism. Unlike conventional fusion strategies that indiscriminately exchange information across modalities, RSGMB enables reliability-aware feature selection and enhancing informative feature aggregation. In addition, a lightweight Local Cross-Gated Modulation (LCGM) is incorporated to refine fine-grained spatial details, complementing the global modeling capability of RSGMB. Extensive experiments demonstrate that RSGMamba achieves state-of-the-art performance on both RGB-D and RGB-T semantic segmentation benchmarks, resulting 58.8% / 54.0% mIoU on NYUDepth V2 and SUN-RGBD (+0.4% / +0.7% over prior best), and 61.1% / 88.9% mIoU on MFNet and PST900 (up to +1.6%), with only 48.6M parameters, thereby validating the effectiveness and superiority of the proposed approach.

Figures & tables

Fig. 1: For each RGB–X sample, we visualize the input image, the corresponding reliability mask, the cross-modal response heatmap, and the modality-wise value distribution. Green circles highlight regions where the modality provides reliable and consistent cues, while red circles denote unreliable or noisy regions. As shown, RGB luminance exhibits a smooth and continuous distribution, while depth and thermal signals present highly skewed and discrete histograms, with large portions of unreliable values. This pronounced statistical heterogeneity motivates the development of a more robust cross-modal fusion strategy that adaptively regulates cross-state readout based on modality uncertainty and cross-modal consistency.

Fig. 2: Overall architecture of our RSGMamba for RGB–X segmentation. RGB and auxiliary modalities are encoded through two weight-sharing encoders to produce multi-scale representations $E^{rgb}_{i}$ and $E^{x}_{i}$ . At each stage, a fusion module combines cross-modal features using the proposed Local Cross-Gated Modulation (LCGM) and Reliability-aware Self-Gated Mamba Block (RSGMB), generating fused features $F_{i}$ . Finally, the four-stage fusion features $F_{1}$ , $F_{2}$ , $F_{3}$ , and $F_{4}$ are fed into the decoder module for final segmentation mask generation.

Fig. 3: (a) Overall architecture of the proposed RSGMB, where RGB and auxiliary-modal features are processed in parallel and fused via a reliability-controlled cross-state mechanism. (b) Illustration of the proposed RSG-SS module. For simplicity, certain implementation details are omitted, while the key computational components that distinguish our method from prior designs are highlighted. (c) Internal design of the reliability self-gating module (RSG), which estimates modality-wise uncertainty and cross-modal consistency to dynamically regulate cross-state readout in the Mamba-based state-space model.

Fig. 4: Structure of the Local Cross-Gated Modulation (LCGM). The RGB and auxiliary features are first aligned through linear projection and cross-modal gating. The resulting features are then processed with depthwise convolutions to capture local spatial structures, yielding the enhanced feature representation $E^{loc}$ .

Model	Publication	Backbone	Params. $\downarrow$	NYUDepthv2			SUN-RGBD
Model	Publication	Backbone	Params. $\downarrow$	Input size	Flops $\downarrow$	mIoU $\uparrow$	Input size	Flops $\downarrow$	mIoU $\uparrow$
TokenFusion [ 33 ]	CVPR’22	MiT-B3	45.9M	$480\times 640$	94.4G	54.2	$530\times 730$	122.1G	51.0
CMX [ 21 ]	TITS’23	MiT-B4	139.9M	$480\times 640$	134.3G	56.3	$530\times 730$	173.8G	52.1
DFNet [ 34 ]	TII’24	MiT-B2	41.2M	$480\times 640$	35.1G	56.1	$530\times 730$	-	51.6
DCANet [ 22 ]	PR’25	VMamba	123.8M	$480\times 640$	138.5G	53.3	$530\times 730$	-	49.6
ECMRN [ 35 ]	KBS’25	DFormer	68.6M	$480\times 640$	135.3G	54.6	$530\times 730$	178.5G	50.5

TABLE I: Qualitative comparisons on NYUDepth V2 and SUN-RGBD datasets. All the backbones are pre-trained on ImageNet-1K. The best and second best results for each metric are color-coded.

Panel of Figure 6 — Fig. 5: Qualitative comparisons on the NYUDepth V2 [ 37 ] (first two rows) and SUN-RGBD [ 38 ] (last two rows) datasets. The highlighted regions (orange circles) emphasize the key differences among the methods. Compared with previous approaches, our RSGMamba produces more accurate predictions with clearer boundaries and better preservation of fine structures, especially in challenging regions with ambiguous depth or small objects. The mIoU (%) for each method is reported below the corresponding result.

Method	Publication	Backbone	Unlabeled	Car	Person	Bike	Curve	Stop	Guardrail	Cone	Bump	mIoU
ABMDRNet [ 41 ]	CVPR’21	ResNet-50	98.6	84.8	69.6	60.3	45.1	33.1	5.1	47.4	50.0	54.8
GMNet [ 26 ]	TIP’21	ResNet-50	97.5	86.5	73.1	61.7	44.0	42.3	14.5	48.7	47.4	57.3
LASNet [ 42 ]	TCSVT’23	RGB-T	97.4	84.2	67.1	56.9	41.1	39.6	18.9	48.8	40.1	54.9
CMX [ 21 ]	TITS’23	MiT-B4	98.3	90.1	75.2	64.5	50.2	35.3	8.5	54.2	60.6	59.7
EAEFNet [ 43 ]	RAL’23	ResNet-152	97.6	87.6	72.6	63.8	48.6	35.0	14.2	52.4	58.3	58.9
CACFNet [ 44 ]	TIV’23	ConvNeXt-B	98.1	89.2	69.5	63.3	46.6	32.4	7.9	54.9	58.3	57.8

TABLE II: Quantitative comparisons on the MFNet dataset. The best and second best results for each metric are color-coded.

Method	Publication	Bg	HD	Bp	FE	Sv	mIoU
DSGBINet [ 53 ]	TCSVT’22	99.4	75.0	85.1	79.3	75.6	82.9
LASNet [ 42 ]	TCSVT’23	99.5	82.8	86.5	77.8	75.5	84.4
FDCNet [ 54 ]	TCSVT’23	99.2	70.4	72.2	71.5	72.4	77.1
SGFNet [ 55 ]	TCSVT’23	99.4	76.7	85.4	75.6	76.7	82.8
EAEFNet [ 43 ]	RAL’23	99.5	83.9	87.7	80.4	75.6	85.4
DPLNet [ 56 ]	IROS’24	-	-	-	-	-	86.7

TABLE III: Quantitative comparisons on PST900 dataset. Bg: Background, HD: Hand-Drill, Bp: Backpack, FE: Fire-Extinguisher, and Sv: Survivor. The best and second best results are color-coded.

Panel of Figure 9 — Fig. 6: Qualitative comparisons on the MFNet [ 24 ] (first two rows) and PST900 [ 39 ] (last two rows) datasets. The highlighted regions (orange circles) emphasize the key differences among methods under challenging conditions, such as low illumination and thermal ambiguity. Compared with previous approaches, our RSGMamba produces more accurate and consistent predictions, with clearer boundaries and better detection of small or hard-to-distinguish objects. The mIoU (%) for each method is reported below the corresponding result.

ID	RSGMB	LCGM	Params. $\downarrow$	GFLOPs $\downarrow$	mIoU $\uparrow$	$\boldsymbol{\Delta}$
1	✗	✗	24.31	53.97	53.6	–
2	✗	✓	25.57	55.19	54.9	+1.3
3	✓	✗	26.83	57.22	55.3	+1.7
4	✓	✓	28.10	58.45	56.4	+2.8

TABLE IV: Effectiveness of RSGMB and LCGM.

ID	Projection	Params. $\downarrow$	GFLOPs $\downarrow$	mIoU $\uparrow$	$\Delta$
1	Linear	29.41	59.13	56.2	-
2	LORA	28.10	58.45	56.4	+0.2

TABLE V: Effectiveness of LORA projection.

ID	Unc $g_{u}$	Con $g_{c}$	Params. $\downarrow$	GFLOPs $\downarrow$	mIoU $\uparrow$	$\Delta$
1	✗	✗	28.10	56.53	54.9	-
2	✓	✗	28.10	57.30	55.4	+0.5
3	✗	✓	28.10	57.68	55.6	+0.7
4	✓	✓	28.10	58.45	56.4	+1.5

TABLE VI: Effectiveness of the reliability modeling in RSGMB.

g_{u}

denotes the uncertainty-aware reliability estimation, and

g_{c}

denotes the cross-modal consistency modeling.

ID	Fusion	Params. $\downarrow$	GFLOPs $\downarrow$	mIoU $\uparrow$	$\Delta$
1	Concat	30.96	60.83	54.7	-
2	Add	24.31	53.97	55.3	+0.6
3	Cross-Attention	35.42	135.06	54.0	-0.5
4	Cross-Mamba	26.83	55.31	55.1	+0.4
5	RSGMB	26.83	57.22	56.4	+1.7

TABLE VII: Comparisons of different fusion methods under the same backbone and decoder.

Panel of Figure 14 — Fig. 7: Qualitative comparisons of reliability modeling in our framework. The highlighted regions show that the uncertainty-aware module ( $g_{u}$ ) suppresses noisy depth-induced artifacts, while the consistency module ( $g_{c}$ ) enhances cross-modal semantic consistency. Their combination ( $g_{u}$ & $g_{c}$ = RSGMB ) produces more reliable and consistent predictions. Furthermore, with the integration of LCGM , our final model achieves sharper boundaries and better preservation of fine structures.

Panel of Figure 15 — Fig. 8: Visualization of feature responses under different fusion mechanisms: Addition, Cross-Attention, Cross-Mamba, and our proposed RSGMB. The zoomed-in regions indicate that our RSGMB produces clearer structural patterns and stronger activations in semantically meaningful regions while effectively suppressing noise, demonstrating its superior ability to model cross-modal interactions.

Panel of Figure 16 — Fig. 9: Visual comparisons of RGB-X semantic segmentation results. From left to right: RGB image, auxiliary modality, and segmentation predictions obtained by Addition, Cross-Attention, Cross-Mamba, and RSGMB (ours). The highlighted regions demonstrate that RSGMB better preserves structural details and thin objects while reducing noisy predictions.

Explore similar work

Oct 13, 2026

Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation

Multi-modal semantic segmentation adds an auxiliary sensing stream to RGB images (RGB-X), such as thermal, LiDAR, depth, event, polarization, or light field. This can improve robustness in poor lighting and under motion blur. But different sensors often produce features that do not match well, and fusion can become unstable. To alleviate these issues, we propose a bidirectional polarity-aware cross-modality fusion (BPCF) module that effectively captures complementary cues while enhancing feature alignment. We evaluate the framework on six modality pairings: RGB-Thermal, RGB-LiDAR, RGB-Depth, RGB-Event, RGB-Polarization, and RGB-Light Field. Our model achieve state-of-the-art (SOTA) results on eight public datasets, including MFNet, KITTI-360, DELIVER, DDD17, DSEC, MCubeS, ZJU, and UrbanLF. The codes and trained models will be released upon acceptance.

Hu Cao, Yifeng Cheng, Mu He +3

School of Automation, Southeast University, Sipailou 2, Nanjing, 210096, China. · Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Boltzmannstraße 15, Munich, 85748, Germany.

Multimodal Fusion Semantic Segmentation Cross-Modal Alignment

Ask PDF

Apr 3, 2026

CrossWeaver: Cross-modal Weaving for Arbitrary-Modality Semantic Segmentation

Multimodal semantic segmentation has shown great potential in leveraging complementary information across diverse sensing modalities. However, existing approaches often rely on carefully designed fusion strategies that either use modality-specific adaptations or rely on loosely coupled interactions, thereby limiting flexibility and resulting in less effective cross-modal coordination. Moreover, these methods often struggle to balance efficient information exchange with preserving the unique characteristics of each modality across different modality combinations. To address these challenges, we propose CrossWeaver, a simple yet effective multimodal fusion framework for arbitrary-modality semantic segmentation. Its core is a Modality Interaction Block (MIB), which enables selective and reliability-aware cross-modal interaction within the encoder, while a lightweight Seam-Aligned Fusion (SAF) module further aggregates the enhanced features. Extensive experiments on multiple multimodal semantic segmentation benchmarks demonstrate that our framework achieves state-of-the-art performance with minimal additional parameters and strong generalization to unseen modality combinations.

Zelin Zhang, Kedi Li, Huiqi Liang +3

The University of Sydney · University of Technology Sydney

Multimodal Fusion Semantic Segmentation Multimodal Learning

Ask PDF

Jul 14, 2026

UMSS: Towards Unsupervised Multi-modal Semantic Segmentation

Multimodal semantic segmentation (MSS) is essential for robust perception in complex environments, yet its potential remains largely untapped because of the prohibitive cost of human annotations. While unsupervised semantic segmentation (USS) has achieved strong results on a single RGB modality, its naive extension to multimodal data is often hindered by fusion degradation. This occurs because, without explicit supervision, existing frameworks struggle to reconcile the heterogeneous structural patterns captured by different sensors and therefore fail to effectively exploit their complementary information. In this paper, we make the first attempt to address the novel problem of Unsupervised Multimodal Semantic Segmentation (UMSS), aiming to effectively exploit complementary sensor information in a fully label free setting. To this end, we propose UniM2 (Unified Multimodal), a novel framework built on DINOv3 that transforms conventional fusion methods into consistent performance gains. Our key idea is to learn a unified latent space driven by Cross Modal Correspondence Synergy (CMCS) to extract intrinsic shared semantic cues, bypassing the need for label guided adaptive fusion. To mitigate inherent intermodal conflicts, we introduce a Cross Modal Harmonizer (CMH) that designates RGB as a stable reference, effectively suppressing inconsistent relational supervision while guiding the model to exploit complementary structural features. Extensive experimental results on NYU Depth v2 and MFNet show that UniM2 improves mIoU by 6.4% and 9.8%, respectively, demonstrating clear advantages over existing frameworks for UMSS.

Haitian Zhang, Thai Duy Nguyen, Xiangyuan Wang +2

EmPACT Lab, School of EEE, Nanyang Technological University, Singapore · The University of Hong Kong, Hong Kong SAR, China

Cross-Modal Representation Learning Semantic Segmentation Unsupervised Image Segmentation Multimodal Sensor Fusion

Ask PDF