Lean Learning Beyond Clouds: Efficient Discrepancy-Conditioned Optical-SAR Fusion for Semantic Segmentation

Chenxing Meng; Wuzhou Quan; Yingjie Cai; Liqun Cao; Liyan Zhang; Mingqiang Wei

cs.CVMar 21, 2026

Lean Learning Beyond Clouds: Efficient Discrepancy-Conditioned Optical-SAR Fusion for Semantic Segmentation

Authors: Chenxing Meng, Wuzhou Quan, Yingjie Cai, Liqun Cao, Liyan Zhang, Mingqiang Wei

Organizations: School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China

Abstract

Cloud occlusion severely degrades the semantic integrity of optical remote sensing imagery. While incorporating Synthetic Aperture Radar (SAR) provides complementary observations, achieving efficient global modeling and reliable cross-modal fusion under cloud interference remains challenging. Existing methods rely on dense global attention to capture long-range dependencies, yet such aggregation indiscriminately propagates cloud-induced noise. Improving robustness typically entails enlarging model capacity, which further increases computational overhead. Given the large-scale and high-resolution nature of remote sensing applications, such computational demands hinder practical deployment, leading to an efficiency-reliability trade-off. To address this dilemma, we propose EDC, an efficiency-oriented and discrepancy-conditioned optical-SAR semantic segmentation framework. A tri-stream encoder with Carrier Tokens enables compact global context modeling with reduced complexity. To prevent noise contamination, we introduce a Discrepancy-Conditioned Hybrid Fusion (DCHF) mechanism that selectively suppresses unreliable regions during global aggregation. In addition, an auxiliary cloud removal branch with teacher-guided distillation enhances semantic consistency under occlusion. Extensive experiments demonstrate that EDC achieves superior accuracy and efficiency, improving mIoU by 0.56% and 0.88% on M3M-CR and WHU-OPT-SAR, respectively, while reducing the number of parameters by 46.7% and accelerating inference by 1.98 $\times$ . Our implementation is available at https://github.com/mengcx0209/EDC.

Figures & tables

Fig. 1: Efficiency–accuracy comparison on M3M-CR. Bubble size indicates model parameter scale. The annotation beside each bubble gives the triplet (throughput, mIoU, parameters). EDC lies on the favorable Pareto frontier, achieving the best mIoU and the highest inference throughput simultaneously.

Fig. 2: Overall framework of the proposed EDC. The StudentNet takes the SAR image $I_{\text{SAR}}$ and the cloudy optical image $I_{\text{Cloudy}}$ as input. At each stage $i$ , the encoder produces an optical-corrected feature $Opt_{i}$ and a fused semantic feature $CM_{i}^{\text{Fuse}}$ at the corresponding resolution. Note that $CM_{1}^{\text{Fuse}}$ is initialized as an empty input (i.e., no prior fused feature is available at the first stage). The cloud-removal (CR) decoder aggregates $\{Opt_{i}\}_{i=1}^{4}$ to reconstruct the cloud-removed optical image, while the semantic-segmentation (SS) decoder aggregates $\{CM_{i}^{\text{Fuse}}\}_{i=1}^{4}$ to predict the land-cover map. During training, a TeacherNet with the same architecture is fed with $I_{\text{SAR}}$ and the cloud-free optical image $I_{\text{Cloudfree}}$ and provides supervision to the StudentNet via a cloud-mask-guided distillation loss.

Fig. 3: Framework of the proposed Efficiency-Oriented Multi-Scale Encoder.

Fig. 4: Architecture of the proposed DCHF module. It uses discrepancy-guided attention map $\mathbf{A}$ to perform weighted GAP for robust channel recalibration and fusion, producing $CM_{i}^{\text{Fuse}}$ and refined $\{Opt_{i},SAR_{i}\}$ from $\{Opt_{i-1},SAR_{i-1},CM_{i-1}^{\text{Fuse}}\}$ (with $CM_{1}^{\text{Fuse}}$ empty). $\oplus/\ominus/\otimes$ : element-wise ops; $\mathrm{C}$ : concatenation.

Method	M3M-CR						WHU-OPT-SAR
	Cloudy		Cloud-Free		Overall		Cloudy		Cloud-Free		Overall
	mPA	mIoU	mPA	mIoU	mPA	mIoU	mPA	mIoU	mPA	mIoU	mPA	mIoU
DCSA-Net [ 43 ]	50.35	39.52	51.21	40.21	51.04	40.09	46.67	36.76	47.88	38.48	47.28	37.63
AMM-FuseNet [ 44 ]	59.17	46.18	63.74	49.95	61.77	48.29	51.68	32.21	55.83	38.19	53.97	35.24
MCANet [ 17 ]	61.17	49.16	64.75	51.52	63.29	50.62	41.04	31.39	41.34	31.92	41.19	31.66
FTransUNet [ 15 ]	60.57	48.89	66.49	54.60	63.37	50.63	56.12	45.10	58.79	47.43	57.47	46.28

TABLE I: Segmentation performance comparison under cloudy, cloud-free, and overall splits on M3M-CR and WHU-OPT-SAR between EDC and state-of-the-art methods. Best results are in bold and the second-best are underlined.

Fig. 5: Visualization of land cover mapping for 4 different scenes. The first two scenes are from the M3M-CR dataset, while the latter two are from the WHU-OPT-SAR dataset. For each scene, from top-left to bottom-right are respectively the cloudy image, the cloud-free image, the SAR image, the ground truth, the result from AMM-FuseNet, MCANet, DCSA-Net, FTransUNet, CMX, CloudSeg and our EDC.

Method	M3M-CR			WHU-OPT-SAR
Method	SSIM $\uparrow$	PSNR $\uparrow$ ( dB )	MAE $\downarrow$ ( $10^{-2}$ )	SSIM $\uparrow$	PSNR $\uparrow$ ( $dB$ )	MAE $\downarrow$ ( $10^{-2}$ )
HS2P [ 45 ]	0.8809	29.61	2.72	0.8093	28.73	2.96
USSRN-CR [ 46 ]	0.8942	29.99	2.64	0.8122	28.94	2.87
CloudSeg [ 14 ]	0.8907	29.56	2.69	0.8167	28.61	2.89
Align-CR [ 42 ]	0.9050	29.12	2.57	0.8192	29.42	2.65
GLF-CR [ 47 ]	0.9076	30.25	2.43	0.8215	29.92	2.56

TABLE II: Quantitative comparisons between the proposed EDC and the current state-of-the-art methods in the field of cloud removal. The results with the highest performance are shown in bold, while the second-best results are underlined.

Method	M3M-CR		WHU-OPT-SAR
Method	mPA	mIoU	mPA	mIoU
w/o DCHF	66.84	53.35	60.06	48.91
w/o DCGD	67.09	53.44	60.57	48.99
w/o Distillation	65.43	52.54	61.00	48.90
w/o Cloud Removal	65.97	52.85	61.21	49.26
EDC-naive	64.88	52.01	59.37	47.29

TABLE III: Ablation study of different architectural components and training strategies on M3M-CR and WHU-OPT-SAR datasets.

Fig. 6: Intermediate outputs of the DCHF module are shown from left to right as follows: (a) cloudy optical image, (b) SAR image, (c) discrepancy map between cloud-free optical image and SAR image, (d) discrepancy map between cloudy optical image and SAR image, (e) discrepancy attention map, (f) reliability map, (g) cloudmask image.

Method	Para (M)	GMAC	Thrp.	M3M-CR	WHU-OPT-SAR
Method	Para (M)	GMAC	Thrp.	mIoU	mIoU
DCSA-Net	80.01	8.17	275.30	40.09	37.63
AMM-FuseNet	62.30	43.21	464.3	48.29	35.24
MCANet	131.76	11.90	952.3	50.62	31.66
FTransUNet	203.40	21.90	410.39	50.63	46.28
CMX	66.57	5.58	968.3	50.58	41.15

TABLE IV: Model complexity, computation, inference throughput, and segmentation accuracy comparison on M3M-CR and WHU-OPT-SAR. Thrp. denotes inference throughput in images per second (img/s), measured under the benchmarking protocol in Sec. IV-D .

Method	stage4_cm			cm_decoder_out
Method	erank $\uparrow$	nerank $\uparrow$	sparsity ( $\|x\|<10^{-3}$ )	$C_{d}$	CKA $\uparrow$
CMX	8.70	0.017	0.004	512	0.609
CloudSeg	23.58	0.046	0.667	16	0.821
EDC (Ours)	131.97	0.258	0.119	32	0.930

TABLE V: Representation efficiency diagnostics. We report effective-rank statistics and activation sparsity at the deepest cross-modal encoder stage ( stage4_cm ,

C{=}512

), and the channel dimension

C_{d}

and linear CKA with logits at the cross-modal decoder output ( cm_decoder_out ).

Fig. 7: Residual calibration results on the M3M-CR dataset. The top row shows residual calibration plots of all compared methods, while the bottom row zooms in on the top-performing models for fine-grained comparison. From left to right are the All Regions, Cloud-Free, and Cloudy subsets, where negative indicates over-confidence and positive indicates under-confidence.

Method	ECE (All) $\downarrow$	ECE (CF) $\downarrow$	ECE (CY) $\downarrow$
EDC (Ours)	0.0046	0.0065	0.0053
CloudSeg	0.0126	0.0133	0.0120
FTransUNet	0.0161	0.0163	0.0159
CMX	0.0420	0.0420	0.0419

TABLE VI: ECE of strong baselines on M3M-CR. ALL/CF/CY denote the all/cloud-free/cloudy regions.

Efficient Semantic Segmentation

See all 15 papers

MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing
Mar 18, 2026Yimin Wei, Aoran Xiao, Hongruixuan Chen +2Remote Sensing Image Segmentation Open-Vocabulary Semantic Segmentation
The University of Tokyo · RIKEN AIP
Heterogeneous SAR-optical fusion for near-real-time land use and land cover mapping under cloud contamination: A novel framework and global benchmark dataset
Jun 16, 2026Jiangong Xu, Weibao Xue, Xiaoyu Yu +3Land Cover Classification Remote Sensing
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China · Oriental Space Port Research Institute, Yantai, 265100, China · Hubei Luojia Laboratory, Wuhan, 430079, China +3
IB-HFN: Information Bottleneck-Driven SAR-Optical Fusion Network for High-Fidelity Cloud Removal
Jun 8, 2026Haojun Guo, Fan Feng, Ziquan Wang +2Remote Sensing Multimodal Fusion
Institute of Geospatial Information, Information Engineering University, Zhengzhou 450001, China

Mar 18, 2026J/K move · Enter open · S save

MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing

Yimin Wei, Aoran Xiao, Hongruixuan Chen, Junshi Xia+1

The University of Tokyo · RIKEN AIP

Remote Sensing Image Segmentation Open-Vocabulary Semantic Segmentation Multimodal Semantic Segmentation Multimodal Robustness Multimodal Sensor Fusion Multi-Source Image Fusion

Open paper AskPDF arXiv

Open-vocabulary segmentation enables pixel-level recognition from an open set of textual categories, allowing generalization beyond fixed classes. Despite great potential in remote sensing, progress in this area remains largely limited to clear-sky optical data and struggles under cloudy or haze-contaminated conditions. We present MM-OVSeg, a multimodal Optical-SAR fusion framework for resilient open-vocabulary segmentation under adverse weather conditions. MM-OVSeg leverages the complementary strengths of the two modalities--optical imagery provides rich spectral semantics, while synthetic aperture radar (SAR) offers cloud-penetrating structural cues. To address the cross-modal domain gap and the limited dense prediction capability of current vision-language models, we propose two key designs: a cross-modal unification process for multi-sensor representation alignment, and a dual-encoder fusion module that integrates hierarchical features from multiple vision foundation models for text-aligned multimodal segmentation. Extensive experiments demonstrate that MM-OVSeg achieves superior robustness and generalization across diverse cloud conditions. The source dataset and code are available at https://github.com/Jimmyxichen/MM-OVSeg.

Abstract

Efficient Semantic Segmentation

Explore similar work

MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing

Heterogeneous SAR-optical fusion for near-real-time land use and land cover mapping under cloud contamination: A novel framework and global benchmark dataset

IB-HFN: Information Bottleneck-Driven SAR-Optical Fusion Network for High-Fidelity Cloud Removal