MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing

Yimin Wei; Aoran Xiao; Hongruixuan Chen; Junshi Xia; Naoto Yokoya

cs.CVMar 18, 2026

MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing

Authors: Yimin Wei, Aoran Xiao, Hongruixuan Chen, Junshi Xia, Naoto Yokoya

Organizations: The University of Tokyo · RIKEN AIP

Abstract

Open-vocabulary segmentation enables pixel-level recognition from an open set of textual categories, allowing generalization beyond fixed classes. Despite great potential in remote sensing, progress in this area remains largely limited to clear-sky optical data and struggles under cloudy or haze-contaminated conditions. We present MM-OVSeg, a multimodal Optical-SAR fusion framework for resilient open-vocabulary segmentation under adverse weather conditions. MM-OVSeg leverages the complementary strengths of the two modalities--optical imagery provides rich spectral semantics, while synthetic aperture radar (SAR) offers cloud-penetrating structural cues. To address the cross-modal domain gap and the limited dense prediction capability of current vision-language models, we propose two key designs: a cross-modal unification process for multi-sensor representation alignment, and a dual-encoder fusion module that integrates hierarchical features from multiple vision foundation models for text-aligned multimodal segmentation. Extensive experiments demonstrate that MM-OVSeg achieves superior robustness and generalization across diverse cloud conditions. The source dataset and code are available at https://github.com/Jimmyxichen/MM-OVSeg.

Figures & tables

Figure 1: Existing unimodal OVS methods fail in cloudy environments due to severely degraded optical inputs. By incorporating SAR, which penetrates clouds and haze, MM-OVSeg produces significantly more accurate and consistent segmentation results.

Figure 2: Overall optimization framework of MM-OVSeg. The training pipeline consists of two stages. (1) In the Cross-Modal Unification stage, the SAR DINO encoder is trained to align SAR features with the fixed RGB DINO features using the CMU-Data collection of 25,087 RGB and SAR image pairs. (2) In the full MM-OVSeg training stage, the model jointly processes optical and SAR inputs for multimodal open-vocabulary segmentation. The Dual-Encoder Fusion module integrates RGB and SAR dense features and aligns them with CLIP text embeddings, after which a linear classifier predicts the final segmentation map.

Index	Datasets ( train → test )	weather	cloud cover	cloud type	generalization
①	PIE-cloud → PIE-cloud	cloudy	varied	synthetic	intra-domain
②	DDHR-SK → DDHR-SK	cloudy	varied	synthetic	intra-domain
③	OEM-thick → OEM-thick	cloudy	thick	synthetic	intra-domain
④	OEM-thin → OEM-thin	cloudy	thin	synthetic	intra-domain
⑤	PIE-clean → PIE-clean	clear sky	none	none	intra-domain
⑥	DDHR-SK → DDHR-CH	cloudy	varied	synthetic	cross-domain

Table 1: Evaluation settings for MM-OVSeg. The experiments cover clear sky and cloudy weather, synthetic cloud cover with different opacity levels (thin or thick or varied), and both intra-domain and cross-domain generalization scenarios.

Method	Publication	①	②	③	④	⑤	⑥	Mean
CAT-Seg [ 5 ]	CVPR’24	54.5	54.2	33.8	29.5	55.8	27.8	42.6
EBSeg [ 38 ]	CVPR’24	50.8	51.1	27.2	25.6	51.0	26.7	38.7
GSNet [ 51 ]	AAAI’25	57.0	55.0	35.2	37.0	57.2	32.4	45.6
SegEarth-OV [ 22 ]	CVPR’25	45.1	17.6	28.9	18.5	51.8	24.2	31.0
FGAseg [ 20 ]	arXiv’25	51.6	51.6	26.0	32.8	52.1	40.6	42.5
MM-OVSeg (ours)	–	57.7	73.1	36.6	40.2	59.7	42.6	51.7

Table 2: Comparison of OVS methods across all evaluation settings defined in Table 1 . The table reports mIoU scores for each setting and the overall mean. Settings correspond to: ①: PIE-cloud→PIE-cloud; ②: DDHR-SK→DDHR-SK; ③: OEM-thick→OEM-thick; ④: OEM-thin→OEM-thin; ⑤: PIE-clean→PIE-clean; ⑥: DDHR-SK→DDHR-CH. MM-OVSeg achieves the highest accuracy in all settings and obtains the best overall mean score, demonstrating strong robustness under cloudy conditions and superior cross-domain generalization.

Figure 3: IoU performance for each individual class under the six evaluation settings defined in Table 1 . Purple bars and blue bars represent seen and unseen classes, respectively.

Figure 4: Visualization of OVS results. From left to right: input RGB image, input SAR image, ground truth, and segmentation outputs from CAT-Seg, EBSeg, GSNet, SegEarth-OV, and our MM-OVSeg. In the legend, underlined categories represent unseen classes and the remaining categories are seen classes.

Model Variant	Forest	City	Farmland	Road	Water	mIoU
MM-OVSeg	87.3	86.8	85.6	21.2	84.8	73.1
w/o CMU	57.2	83.7	81.2	16.8	81.4	64.1
w/o CMU&DEF	80.0	90.3	79.0	6.8	19.1	55.0

Table 3: Ablation study of MM-OVSeg on the DDHR-SK→DDHR-SK segmentation task under cloudy conditions. The proposed DEF module enables effective multimodal fusion, substantially improving over the single-modality baseline. In combination with CMU, the full model achieves the best performance, demonstrating that CMU and DEF are complementary.

Figure 5: Visualization of different stages in multimodal fusion within DEF. DEF produces finer spatial localization and stronger alignment between dense and global visual features and text representations.

Loss type	mIoU
baseline ( w/o CMU)	64.1
MSE	67.7
L1	69.0
InfoNCE	73.1

Table 4: Comparison of loss functions used in the CMU stage.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Backbone	Param.	Batch.	Lr
CAT-Seg [ 5 ]	ViT-B/16	25M	4	2e-4
FGAseg [ 20 ]	ViT-B/16	33M	8	2e-4
GSNet [ 51 ]	ViT-B/16	29M	4	2e-4
EBSeg [ 38 ]	ViT-B/16	26M	16	1e-4
MM-OVSeg	ViT-B/16	31M	8	2.5e-4

Appendix

Table A1 : Model implementations. The table reports trainable parameters, batch size and learning rate (Lr).

	Time (ms)		CO 2 (kg)		Parameters	mIoU
	Train	Infer	Train	Infer	Parameters	mIoU
CMU module	343.7	31.6	1.05	0.006	85M	–
DEF module	377.2	46.8	1.14	0.008	92M	–
full MM-OVSeg	955.7	116.9	3.90	0.014	331M	73.1
GSNet [ 51 ]	700.6	83.2	3.48	0.011	244M	55.0

Appendix

Table A2 : Efficiency and sustainability comparison.

Method	Publication	①		②		③		④		⑤		⑥		Mean
		Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen
CAT-Seg [ 5 ]	CVPR’24	14.9	74.2	41.1	62.9	32.2	34.6	17.0	36.9	17.1	75.1	7.5	41.5	21.6	54.2
EBSeg [ 38 ]	CVPR’24	2.5	74.8	26.7	67.4	12.7	35.9	5.5	37.6	0.8	76.0	8.5	38.8	9.5	55.1
GSNet [ 51 ]	AAAI’25	18.6	76.1	13.0	83.1	32.0	37.1	32.8	39.4	18.0	76.8	8.8	48.1	20.5	60.1
SegEarth-OV [ 22 ]	CVPR’25	19.3	57.9	7.9	24.1	33.1	26.3	26.9	13.5	23.0	66.2	16.4	29.4	21.1	36.2
FGAseg [ 20 ]	arXiv’25	13.4	70.6	47.5	54.4	7.2	37.2	23.6	38.2	13.4	71.4	18.7	55.1	20.6	54.5

Appendix

Table A3 : Performance splits for unseen and seen classes. The table reports mIoU scores for each setting and the overall mean.

Method	Backbone	Publication	①	②	③	④	⑤	⑥	Mean
CAT-Seg [ 5 ]	ViT-L/14	CVPR’24	60.7	69.8	38.1	40.2	62.2	41.5	52.1
EBSeg [ 38 ]	ViT-L/14	CVPR’24	48.9	50.6	23.6	26.9	50.7	31.5	38.7
FGAseg [ 20 ]	ViT-L/14	arXiv’25	57.1	48.8	23.3	36.5	52.3	41.5	43.3
MM-OVSeg (ours)	ViT-L/14	–	64.5	75.5	39.4	41.2	66.0	43.5	55.0

Appendix

Table A4 : Comparison of OVS methods with ViT-L/14 backbone across all evaluation settings as illustrated in Table 1 of the main paper. The table reports mIoU scores for each setting and the overall mean. Settings correspond to: ①: PIE-cloud→PIE-cloud; ②: DDHR-SK→DDHR-SK; ③: OEM-thick→OEM-thick; ④: OEM-thin→OEM-thin; ⑤: PIE-clean→PIE-clean; ⑥: DDHR-SK→DDHR-CH. MM-OVSeg achieves the highest accuracy in all settings and obtains the best overall mean score, demonstrating strong robustness under cloudy conditions and superior cross-domain generalization.

Figure A1 : Visualization of OVS results. From left to right: input RGB image, input SAR image, ground truth, and segmentation outputs from MM-OVSeg (ViT-B/16) and MM-OVSeg (ViT-L/14). In the legend, underlined categories represent unseen classes and the remaining categories are seen classes.

Model index	CLIP	DINO	mIoU
#1	✗	✗	64.1
#2	✓	✗	65.4
#3	✗	✓	73.1
#4	✓	✓	66.5

Appendix

Table A5 : Ablation on applying CMU to different visual encoders on the ②: DDHR-SK→DDHR-SK segmentation task.

Figure A2 : Intra-domain visualization of OVS results, including ②: DDHR-SK→DDHR-SK and ⑤: PIE-clean → PIE-clean. From left to right: input RGB image, input SAR image, ground truth, and segmentation outputs from CAT-Seg, EBSeg, GSNet, SegEarth-OV, and our MM-OVSeg. In the legend, underlined categories represent unseen classes and the remaining categories are seen classes.

Figure A3 : Cross-domain visualization of OVS results for ⑥: DDHR-SK→DDHR-CH. From left to right: input RGB image, input SAR image, ground truth, and segmentation outputs from CAT-Seg, EBSeg, GSNet, SegEarth-OV, and our MM-OVSeg. In the legend, underlined categories represent unseen classes and the remaining categories are seen classes.

Remote Sensing Image Segmentation

Remote sensing image segmentation labels land cover and objects in satellite, aerial, radar, and multispectral imagery.

See all 213 papers

Open-Vocabulary Semantic Segmentation Network Integrating Object-Level Label and Scene-Level Semantic Features for Multimodal Remote Sensing Images
Apr 27, 2026Jinkun Dai, Yuanxin Ye, Peng Tang +4Remote Sensing Image Segmentation Text-Guided Image Segmentation
Faculty of Geosciences and Engineering, Southwest Jiaotong University, Chengdu 611756, China · State-Province Joint Engineering Laboratory of Spatial Information Technology for High-Speed Railway Safety, Southwest Jiaotong University, Chengdu 611756, China · School of Artificial Intelligence, Wuhan University, Wuhan 430072, China +1
Lean Learning Beyond Clouds: Efficient Discrepancy-Conditioned Optical-SAR Fusion for Semantic Segmentation
Mar 21, 2026Chenxing Meng, Wuzhou Quan, Yingjie Cai +3Efficient Semantic Segmentation Cloud Removal
School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
ConInfer: Context-Aware Inference for Training-Free Open-Vocabulary Remote Sensing Segmentation
Mar 31, 2026Wenyang Chen, Zhanxuan Hu, Yaping Zhang +2Image Segmentation Remote Sensing Image Segmentation
Yunnan Normal University · Xi’an University of Posts & Telecommunications

Apr 27, 2026J/K move · Enter open · S save

Open-Vocabulary Semantic Segmentation Network Integrating Object-Level Label and Scene-Level Semantic Features for Multimodal Remote Sensing Images

Jinkun Dai, Yuanxin Ye, Peng Tang, Tengfeng Tang+3

Faculty of Geosciences and Engineering, Southwest Jiaotong University, Chengdu 611756, China · State-Province Joint Engineering Laboratory of Spatial Information Technology for High-Speed Railway Safety, Southwest Jiaotong University, Chengdu 611756, China · School of Artificial Intelligence, Wuhan University, Wuhan 430072, China+1

Remote Sensing Image Segmentation Text-Guided Image Segmentation Open-Vocabulary Semantic Segmentation Multimodal Fusion Semantic Segmentation Multimodal Semantic Segmentation

Open paper AskPDF arXiv

Semantic segmentation of multi-modal remote sensing imagery plays a pivotal role in land use/land cover (LULC) mapping, environmental monitoring, and precision earth observation. Current multi-modal approaches mainly focus on integrating complementary visual modalities, yet neglect the incorporating of non-visual textual data - a rich source of knowledge that can bridge semantic gaps between visual patterns and real-world concepts. To address this limitation, we propose TSMNet, a text supervised multi-modal open vocabulary semantic segmentation network that synergistically integrates textual supervision with visual representation for open-vocabulary semantic segmentation. Unlike conventional multi-modal segmentation frameworks, TSMNet introduces a dual-branch text encoder to extract both scene-level semantic and object-level label information from various textual data, enabling dynamic cross-modal fusion. These text-derived features dynamically interact with visual embeddings through the proposed text-guided visual semantic fusion module, enabling domain-aware feature refinement and human-interpretable decision-making. To verify our method, we innovatively construct two new multi-modal datasets, and carry out extensive experiments to make a comprehensive comparison between the proposed method and other state-of-the-art (SOTA) semantic segmentation models. Results demonstrate that TSMNet achieves superior segmentation accuracy while exhibiting robust generalization capabilities across diverse geographical and sensor-specific scenarios. This work establishes a new paradigm for explainable remote sensing analysis, demonstrating that textual knowledge integration significantly enhances model generalizability. The source code will be available at https://github.com/yeyuanxin110/TSMNet

Abstract

Remote Sensing Image Segmentation

Explore similar work

Open-Vocabulary Semantic Segmentation Network Integrating Object-Level Label and Scene-Level Semantic Features for Multimodal Remote Sensing Images

Lean Learning Beyond Clouds: Efficient Discrepancy-Conditioned Optical-SAR Fusion for Semantic Segmentation

ConInfer: Context-Aware Inference for Training-Free Open-Vocabulary Remote Sensing Segmentation