RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation

Guoan Xu; Yang Xiao; Guangwei Gao; Dongchen Zhu; Guo-Jun Qi; Wenjing Jia

Fig. 1: For each RGB–X sample, we visualize the input image, the corresponding reliability mask, the cross-modal response heatmap, and the modality-wise value distribution. Green circles highlight regions where the modality provides reliable and consistent cues, while red circles denote unreliable or noisy regions. As shown, RGB luminance exhibits a smooth and continuous distribution, while depth and thermal signals present highly skewed and discrete histograms, with large portions of unreliable values. This pronounced statistical heterogeneity motivates the development of a more robust cross-modal fusion strategy that adaptively regulates cross-state readout based on modality uncertainty and cross-modal consistency.

Fig. 2: Overall architecture of our RSGMamba for RGB–X segmentation. RGB and auxiliary modalities are encoded through two weight-sharing encoders to produce multi-scale representations

E^{rgb}_{i}

and

E^{x}_{i}

. At each stage, a fusion module combines cross-modal features using the proposed Local Cross-Gated Modulation (LCGM) and Reliability-aware Self-Gated Mamba Block (RSGMB), generating fused features

F_{i}

. Finally, the four-stage fusion features

F_{1}

,

F_{2}

,

F_{3}

, and

F_{4}

are fed into the decoder module for final segmentation mask generation.

Fig. 3: (a) Overall architecture of the proposed RSGMB, where RGB and auxiliary-modal features are processed in parallel and fused via a reliability-controlled cross-state mechanism. (b) Illustration of the proposed RSG-SS module. For simplicity, certain implementation details are omitted, while the key computational components that distinguish our method from prior designs are highlighted. (c) Internal design of the reliability self-gating module (RSG), which estimates modality-wise uncertainty and cross-modal consistency to dynamically regulate cross-state readout in the Mamba-based state-space model.

Fig. 4: Structure of the Local Cross-Gated Modulation (LCGM). The RGB and auxiliary features are first aligned through linear projection and cross-modal gating. The resulting features are then processed with depthwise convolutions to capture local spatial structures, yielding the enhanced feature representation

E^{loc}

.

Model	Publication	Backbone	Params. $\downarrow$	NYUDepthv2			SUN-RGBD
Model	Publication	Backbone	Params. $\downarrow$	Input size	Flops $\downarrow$	mIoU $\uparrow$	Input size	Flops $\downarrow$	mIoU $\uparrow$
TokenFusion [ 33 ]	CVPR’22	MiT-B3	45.9M	$480\times 640$	94.4G	54.2	$530\times 730$	122.1G	51.0
CMX [ 21 ]	TITS’23	MiT-B4	139.9M	$480\times 640$	134.3G	56.3	$530\times 730$	173.8G	52.1
DFNet [ 34 ]	TII’24	MiT-B2	41.2M	$480\times 640$	35.1G	56.1	$530\times 730$	-	51.6
DCANet [ 22 ]	PR’25	VMamba	123.8M	$480\times 640$	138.5G	53.3	$530\times 730$	-	49.6
ECMRN [ 35 ]	KBS’25	DFormer	68.6M	$480\times 640$	135.3G	54.6	$530\times 730$	178.5G	50.5

TABLE I: Qualitative comparisons on NYUDepth V2 and SUN-RGBD datasets. All the backbones are pre-trained on ImageNet-1K. The best and second best results for each metric are color-coded.

Fig. 5: Qualitative comparisons on the NYUDepth V2 [ 37 ] (first two rows) and SUN-RGBD [ 38 ] (last two rows) datasets. The highlighted regions (orange circles) emphasize the key differences among the methods. Compared with previous approaches, our RSGMamba produces more accurate predictions with clearer boundaries and better preservation of fine structures, especially in challenging regions with ambiguous depth or small objects. The mIoU (%) for each method is reported below the corresponding result.

Method	Publication	Backbone	Unlabeled	Car	Person	Bike	Curve	Stop	Guardrail	Cone	Bump	mIoU
ABMDRNet [ 41 ]	CVPR’21	ResNet-50	98.6	84.8	69.6	60.3	45.1	33.1	5.1	47.4	50.0	54.8
GMNet [ 26 ]	TIP’21	ResNet-50	97.5	86.5	73.1	61.7	44.0	42.3	14.5	48.7	47.4	57.3
LASNet [ 42 ]	TCSVT’23	RGB-T	97.4	84.2	67.1	56.9	41.1	39.6	18.9	48.8	40.1	54.9
CMX [ 21 ]	TITS’23	MiT-B4	98.3	90.1	75.2	64.5	50.2	35.3	8.5	54.2	60.6	59.7
EAEFNet [ 43 ]	RAL’23	ResNet-152	97.6	87.6	72.6	63.8	48.6	35.0	14.2	52.4	58.3	58.9
CACFNet [ 44 ]	TIV’23	ConvNeXt-B	98.1	89.2	69.5	63.3	46.6	32.4	7.9	54.9	58.3	57.8

TABLE II: Quantitative comparisons on the MFNet dataset. The best and second best results for each metric are color-coded.

Method	Publication	Bg	HD	Bp	FE	Sv	mIoU
DSGBINet [ 53 ]	TCSVT’22	99.4	75.0	85.1	79.3	75.6	82.9
LASNet [ 42 ]	TCSVT’23	99.5	82.8	86.5	77.8	75.5	84.4
FDCNet [ 54 ]	TCSVT’23	99.2	70.4	72.2	71.5	72.4	77.1
SGFNet [ 55 ]	TCSVT’23	99.4	76.7	85.4	75.6	76.7	82.8
EAEFNet [ 43 ]	RAL’23	99.5	83.9	87.7	80.4	75.6	85.4
DPLNet [ 56 ]	IROS’24	-	-	-	-	-	86.7

TABLE III: Quantitative comparisons on PST900 dataset. Bg: Background, HD: Hand-Drill, Bp: Backpack, FE: Fire-Extinguisher, and Sv: Survivor. The best and second best results are color-coded.

Fig. 6: Qualitative comparisons on the MFNet [ 24 ] (first two rows) and PST900 [ 39 ] (last two rows) datasets. The highlighted regions (orange circles) emphasize the key differences among methods under challenging conditions, such as low illumination and thermal ambiguity. Compared with previous approaches, our RSGMamba produces more accurate and consistent predictions, with clearer boundaries and better detection of small or hard-to-distinguish objects. The mIoU (%) for each method is reported below the corresponding result.

ID	RSGMB	LCGM	Params. $\downarrow$	GFLOPs $\downarrow$	mIoU $\uparrow$	$\boldsymbol{\Delta}$
1	✗	✗	24.31	53.97	53.6	–
2	✗	✓	25.57	55.19	54.9	+1.3
3	✓	✗	26.83	57.22	55.3	+1.7
4	✓	✓	28.10	58.45	56.4	+2.8

TABLE IV: Effectiveness of RSGMB and LCGM.

ID	Projection	Params. $\downarrow$	GFLOPs $\downarrow$	mIoU $\uparrow$	$\Delta$
1	Linear	29.41	59.13	56.2	-
2	LORA	28.10	58.45	56.4	+0.2

TABLE V: Effectiveness of LORA projection.

ID	Unc $g_{u}$	Con $g_{c}$	Params. $\downarrow$	GFLOPs $\downarrow$	mIoU $\uparrow$	$\Delta$
1	✗	✗	28.10	56.53	54.9	-
2	✓	✗	28.10	57.30	55.4	+0.5
3	✗	✓	28.10	57.68	55.6	+0.7
4	✓	✓	28.10	58.45	56.4	+1.5

TABLE VI: Effectiveness of the reliability modeling in RSGMB.

g_{u}

denotes the uncertainty-aware reliability estimation, and

g_{c}

denotes the cross-modal consistency modeling.

ID	Fusion	Params. $\downarrow$	GFLOPs $\downarrow$	mIoU $\uparrow$	$\Delta$
1	Concat	30.96	60.83	54.7	-
2	Add	24.31	53.97	55.3	+0.6
3	Cross-Attention	35.42	135.06	54.0	-0.5
4	Cross-Mamba	26.83	55.31	55.1	+0.4
5	RSGMB	26.83	57.22	56.4	+1.7

TABLE VII: Comparisons of different fusion methods under the same backbone and decoder.

Fig. 7: Qualitative comparisons of reliability modeling in our framework. The highlighted regions show that the uncertainty-aware module (

g_{u}

) suppresses noisy depth-induced artifacts, while the consistency module (

g_{c}

) enhances cross-modal semantic consistency. Their combination (

g_{u}

&

g_{c}

= RSGMB ) produces more reliable and consistent predictions. Furthermore, with the integration of LCGM , our final model achieves sharper boundaries and better preservation of fine structures.

Fig. 8: Visualization of feature responses under different fusion mechanisms: Addition, Cross-Attention, Cross-Mamba, and our proposed RSGMB. The zoomed-in regions indicate that our RSGMB produces clearer structural patterns and stronger activations in semantically meaningful regions while effectively suppressing noise, demonstrating its superior ability to model cross-modal interactions.

Fig. 9: Visual comparisons of RGB-X semantic segmentation results. From left to right: RGB image, auxiliary modality, and segmentation predictions obtained by Addition, Cross-Attention, Cross-Mamba, and RSGMB (ours). The highlighted regions demonstrate that RSGMB better preserves structural details and thin objects while reducing noisy predictions.

RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation

Abstract

Figures & tables

Explore similar work

Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation

CrossWeaver: Cross-modal Weaving for Arbitrary-Modality Semantic Segmentation

UMSS: Towards Unsupervised Multi-modal Semantic Segmentation