Organizations: the College of Land Science and Technology, China Agricultural University, and also with the Key Laboratory of Remote Sensing for Agri-Hazards, Ministry of Agriculture and Rural Affairs, Beijing 100083, China · the School of Artificial Intelligence, Sun Yat-Sen University, Zhuhai, China · Henan Polytechnic University, and also with the Key Laboratory of Spatio-Temporal Information and Ecological Restoration of Mines, Ministry of Natural Resources of the People’s Republic of China, Jiaozuo 454003, China · the Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · the National Supercomputing Center in Shenzhen, Shenzhen, China · the Ministry of Education Key Laboratory for Earth System Modeling and the Department of Earth System Science, Tsinghua University, Beijing, China · the Faculty of Geosciences and Engineering, Southwest Jiaotong University, Chengdu 611756, China
Multimodal remote sensing semantic segmentation enhances scene interpretation by exploiting complementary physical cues from heterogeneous data. Although pretrained Vision Foundation Models (VFMs) provide strong general-purpose representations, adapting them to multimodal tasks often incurs substantial computational overhead and is prone to modality imbalance, where the contribution of auxiliary modalities is suppressed during optimization. To address these challenges, we propose MoBaNet, a parameter-efficient and modality-balanced symmetric fusion framework. Built upon a largely frozen VFM backbone, MoBaNet adopts a symmetric dual-stream architecture to preserve generalizable representations while minimizing the number of trainable parameters. Specifically, we design a Cross-modal Prompt-Injected Adapter (CPIA) to enable deep semantic interaction by generating shared prompts and injecting them into bottleneck adapters under the frozen backbone. To obtain compact and discriminative multimodal representations for decoding, we further introduce a Difference-Guided Gated Fusion Module (DGFM), which adaptively fuses paired stage features by explicitly leveraging cross-modal discrepancy to guide feature selection. Furthermore, we propose a Modality-Conditional Random Masking (MCRM) strategy to mitigate modality imbalance by masking one modality only during training and imposing hard-pixel auxiliary supervision on modality-specific branches. Extensive experiments on the ISPRS Vaihingen and Potsdam benchmarks demonstrate that MoBaNet achieves state-of-the-art performance with significantly fewer trainable parameters than full fine-tuning, validating its effectiveness for robust and balanced multimodal fusion. The source code in this work is available at https://github.com/sauryeo/MoBaNet.
Figures & tables
Fig. 1: Overall framework of the proposed multimodal segmentation model with MCRM training. A pretrained vision foundation model initializes a shared frozen ViT backbone. RGB and DSM inputs are first tokenized by modality-specific embeddings, then processed by a stage-wise dual-stream encoder, where CPIA is inserted before each selected stage and DGFM is applied after each selected stage to produce multi-level fused features for the decoder. During training, MCRM performs modality-conditional random masking and hard-pixel auxiliary supervision on one main branch (S1) and two auxiliary modality branches (S2, S3), while only the main branch is used for inference.
Fig. 2: Structure of the CPIA module. A shared semantic base is generated from paired RGB and auxiliary tokens, transformed into modality-specific prompts by TFT, and injected into lightweight bottleneck adapters for symmetric cross-modal semantic modulation before each selected stage.
Fig. 3: Structure of the DGFM module. Reduced RGB and DSM features, together with their discrepancy cue, are fed into a lightweight gate network to predict an adaptive gating map, which dynamically fuses the two modalities into a compact multimodal representation.
Fig. 4: Visualized comparisons on the Vaihingen test set with the size of 512 × 512. (a) U-Net, (b) ABCNet, (c) DC-Swin, (d) UNetFormer, (e) FTransUNet, (f) MANet, and (g) proposed MoBaNet. To highlight the differences, some purple boxes are added to all subfigures.
Fig. 5: Visualized comparisons on the Potsdam test set with the size of 512 × 512. (a) U-Net, (b) ABCNet, (c) DC-Swin, (d) UNetFormer, (e) FTransUNet, (f) MANet, and (g) proposed MoBaNet. To highlight the differences, some purple boxes are added to all subfigures.
Method
Backbone
Imp.
Bui.
Low.
Tre.
Car
OA
mF1 (%)
mIoU (%)
U-Net [ 48 ]
–
86.63
90.56
76.84
83.75
72.80
80.73
82.12
70.17
SegNet [ 49 ]
–
86.72
91.31
78.82
84.87
69.66
81.55
82.27
70.55
FuseNet [ 50 ]
VGG16
91.66
96.28
78.98
90.28
81.37
90.51
87.71
78.71
PSPNet [ 51 ]
ResNet101
92.79
95.46
84.51
89.94
88.61
90.85
90.26
82.58
ABCNet [ 52 ]
ResNet18
96.67
95.44
83.67
89.61
87.19
91.14
90.51
83.04
DC-Swin [ 53 ]
Swin-B
96.68
95.85
84.38
89.59
86.31
91.33
90.56
83.13
TABLE I: Quantitative comparison on the Vaihingen dataset. Best results are shown in bold, and second-best results are underlined(%).
Method
Backbone
Imp.
Bui.
Low.
Tre.
Car
OA
mF1 (%)
mIoU (%)
U-Net [ 48 ]
–
86.62
91.96
81.52
84.09
83.55
81.93
85.55
74.92
SegNet [ 49 ]
–
86.77
92.43
79.59
81.79
82.97
81.14
84.71
73.75
FuseNet [ 50 ]
VGG16
92.64
97.48
87.31
85.14
96.10
90.58
91.60
84.86
PSPNet [ 51 ]
ResNet101
93.36
96.97
87.75
88.50
95.42
91.08
92.40
84.88
ABCNet [ 52 ]
ResNet18
92.26
95.13
85.82
87.96
94.80
89.70
91.19
84.03
DC-Swin [ 53 ]
Swin-B
93.65
96.31
87.16
88.31
95.27
90.86
92.14
85.64
TABLE II: Quantitative comparison on the Potsdam dataset. Best results are shown in bold, and second-best results are underlined(%).
Method
Params (M)
Latency (ms) ↓
FPS ↑
Memory (MB) ↓
mIoU (%) ↑
ABCNet
13.67
3.55
281.61
194.59
84.03
DC-Swin
118.39
14.59
68.56
603.83
85.64
UNetFormer
11.72
3.58
279.51
186.92
85.03
CMGFNet
43.68
13.13
76.18
357.94
85.18
FTransUNet
203.40
38.90
25.71
930.35
85.56
MANet
20.42
105.36
9.49
1741.83
85.08
TABLE III: Computational complexity analysis measured using a 256×256 image on a single NVIDIA GeForce RTX 4090 GPU. For MoBaNet, the reported results are obtained using the DINOv2-ViT-B backbone. mIoU values are reported on the Potsdam dataset. A horizontal rule separates optical-only and multimodal methods. Bold values indicate the best results.
Variant
CPIA
DGFM
MCRM
Params (M)
OA (%)
mF1 (%)
mIoU (%)
Base
1.91
90.47
91.74
84.95
Base + CPIA
✓
3.99
91.43
92.57
86.33
Base + CPIA + DGFM
✓
✓
5.00
91.73
93.02
87.15
Base + CPIA + DGFM + MCRM (Full)
✓
✓
✓
6.18
92.11
93.40
87.80
TABLE IV: Ablation study of the proposed key designs on the ISPRS Potsdam dataset. “Base” denotes the frozen VFM dual-stream baseline with a trainable auxiliary patch embedding and the same decoder head, without any of the proposed designs. “Params” reports the number of trainable parameters (M). The best results are in bold and the second-best are underlined.
Dataset
Input
MoBaNet (Full)
Reference Baseline
OA (%)
mF1 (%)
mIoU (%)
OA (%)
mF1 (%)
mIoU (%)
Potsdam
RGB + DSM
92.11
93.40
87.80
91.79
93.09
87.26
RGB-only
91.36
92.74
86.63
72.38
76.97
64.20
Drop ( ↓ )
0.75
0.66
1.17
19.41
16.12
23.06
TABLE V: Modality-missing evaluation at inference time. “RGB-only” sets DSM to zeros. All models are trained with full modalities unless otherwise specified. “Drop ( ↓ )” is computed as the performance difference between the RGB+DSM and RGB-only settings.
Fig. 6: Qualitative comparison of feature response visualizations on representative Potsdam samples for the Buildings category. (a) RGB image. (b) DSM. (c) Base . (d) Full model.
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University · Hangzhou Institute of Technology, Xidian University · School of Artificial Intelligence, Wuhan University
Department of Computer Engineering, Ankara University, Ankara, Türkiye · Department of Modeling and Simulation, Graduate School of Informatics, METU, Ankara, Türkiye
Faculty of Geosciences and Engineering, Southwest Jiaotong University, Chengdu 611756, China · State-Province Joint Engineering Laboratory of Spatial Information Technology for High-Speed Railway Safety, Southwest Jiaotong University, Chengdu 611756, China · School of Artificial Intelligence, Wuhan University, Wuhan 430072, China +1