To Remove or Not to Remove Clouds: A Comparative Analysis and Fusion of Raw SAR and Synthetic NDWI for Overcast Water Segmentation

Saleh Sakib Ahmed; Sara Nowreen; M. Sohel Rahman

cs.CVAug 18, 2026

To Remove or Not to Remove Clouds: A Comparative Analysis and Fusion of Raw SAR and Synthetic NDWI for Overcast Water Segmentation

Authors: Saleh Sakib Ahmed, Sara Nowreen, M. Sohel Rahman

Organizations: Computer Science and Engineering, Bangladesh University of Engineering and Technology, Palashi, Dhaka, Bangladesh. · Institute of Water and Flood Management, Bangladesh University of Engineering and Technology, Palashi, Dhaka, Bangladesh.

Abstract

Persistent clouds blind optical satellites during floods. While Synthetic Aperture Radar (SAR) penetrates clouds, its raw data is noisy and lacks clear contrast. To mitigate this, recent studies utilize deep learning models to translate SAR into cloud-free synthetic optical imagery for downstream tasks like water body segmentation. However, because raw SAR is the original source for both of these operations, a critical methodological dilemma arises: during complete overcast should segmentation models process the raw SAR directly, or rely on a translated synthetic Normalized Difference Water Index (NDWI) proxy? This study resolves the debate by demonstrating that synthetic NDWI yields better results, as the translation process acts as a powerful filter against radar noise. This raises a natural second question: what if we utilize both? Building on our findings, we introduce a Combined Framework that integrates both raw SAR and synthetic NDWI into a unified model. By fusing the sharp physical boundaries of raw SAR with the high contrast of synthetic NDWI, this hybrid approach consistently outperforms all standalone methods.

Figures & tables

Figure 1: Architectural paradigms for cloud-resilient water segmentation. The figure depicts the main motivation of our research. It first presents the scenario and where this research is relevant, illustrating the challenge of surface monitoring under complete overcast conditions where Sentinel-2 optical imagery is completely blocked. Then, it presents the two conventional options: a direct raw SAR pipeline that bypasses cloud removal (Option 1) and a synthetic NDWI proxy pipeline that translates radar features into the optical domain (Option 2). Finally, our proposed method is presented at the bottom, showcasing a Multi-View Learning framework that simultaneously fuses raw SAR backscatter with the translated synthetic NDWI map to achieve robust, dual-stream feature extraction.

Figure 2: Global spatial and temporal distribution of the 18 flood events from the Microsoft Cloud2Street flood dataset (2016–2020). The map utilizes distinct colored geometric markers representing the location and specific year of each flood occurrence.

		128 $\times$ 128 Regime		256 $\times$ 256 Regime
Category	Attribute	Tile Count	Percentage (%)	Tile Count	Percentage (%)
Continent	North America	2,401	34.49	593	38.21
	Africa	2,027	29.12	409	26.35
	Europe	944	13.56	233	15.01
	Asia	674	9.68	134	8.63
	South America	654	9.40	141	9.09

Table 1: Macro-level spatiotemporal distribution of the final dataset across the high-data (

128\times 128

) and high-context (

256\times 256

) regimes.

Model	Params (M)	MSE	MAE	R 2	SSIM	wMAPE (%)	Remarks
— Models Trained from Scratch —
U-Net
(depth=3, groups=4)	7.70	0.0107	0.0710	0.7560	0.6899	27.77	Ronneberger et al. Ronneberger et al. (2015)
U-Net							Ronneberger et al. Ronneberger et al. (2015)
(depth=4, groups=8)	31.04	0.0107	0.0700	0.7550	0.6933	27.38
U-Net

Table 2: Comparison of synthetic NDWI generation performance. The evaluation isolates architectures trained from scratch against variants utilizing cross-domain pretraining. Bold text denotes the best overall metric, and underlined text denotes the second best.

Metric	128 $\times$ 128 Tiles (High Data)			256 $\times$ 256 Tiles (High Context)
Metric	mit-b0	mit-b3	mit-b5	mit-b0	mit-b3
$R^{2}$ Score $\uparrow$	0.7835 $\pm$ 0.01	0.7956 $\pm$ 0.02	0.8079 $\pm$ 0.01	0.7390 $\pm$ 0.04	0.7531 $\pm$ 0.05
MSE $\downarrow$	0.0094 $\pm$ 0.00	0.0089 $\pm$ 0.00	0.0084 $\pm$ 0.00	0.0090 $\pm$ 0.00	0.0086 $\pm$ 0.00
MAE $\downarrow$	0.0642 $\pm$ 0.00	0.0608 $\pm$ 0.00	0.0593 $\pm$ 0.00	0.0653 $\pm$ 0.01	0.0626 $\pm$ 0.01
SSIM $\uparrow$	0.6614 $\pm$ 0.01	0.6689 $\pm$ 0.01	0.6702 $\pm$ 0.01	0.6739 $\pm$ 0.01	0.6818 $\pm$ 0.00
wMAPE (%) $\downarrow$	25.17 $\pm$ 0.53	23.82 $\pm$ 0.69	23.21 $\pm$ 0.65	23.47 $\pm$ 1.08	22.54 $\pm$ 1.64

Table 3: Cross-validation regression scaling performance metrics categorized across spatial data resolutions.

Figure 3: Comprehensive performance evaluation of the proposed fusion and translation strategies. (a) Modality gap analysis illustrating the baseline performance disparity between S1 (Baseline) and S2 (True Reference) across key segmentation metrics. (b) Evaluation of IoU and F1-Scores showing averaged Cross-Validation (CV) results. Two data splitting paradigms are compared: Event-stratified (where data is split by distinct discrete events to evaluate generalization) and Mixed (where data is uniformly shuffled across all events prior to splitting). The dashed line tracks the performance trend, with golden stars explicitly denoting the highest-scoring configuration for each metric. (c) Normalized radar plot providing a multidimensional comparison of Accuracy, Precision, Recall, and Specificity to highlight trade-offs between the strategies. (d, e) Normalized heatmaps detailing the relative Global performance (metrics computed over the entire aggregated test set rather than averaged per-fold) of each strategy under Event (d) and Mixed (e) data splitting paradigms; darker shades of blue indicate superior relative performance for a given metric.

Input Modality	IoU		F1-Score
Input Modality	Global	CV (Mean $\pm$ SD)	Global	CV (Mean $\pm$ SD)
Panel A: Event Stratified Splitting Performance
S1 Only	0.7124	0.6334 $\pm$ 0.1710	0.8320	0.7615 $\pm$ 0.1352
S1 PCA	0.7263	0.6514 $\pm$ 0.1192	0.8414	0.7830 $\pm$ 0.0820
S1 PCA + Raw S1	0.7239	0.6429 $\pm$ 0.1733	0.8398	0.7683 $\pm$ 0.1372
Synthetic S2 (Reg)	0.7316	0.6559 $\pm$ 0.1395	0.8450	0.7840 $\pm$ 0.0975

Table 4: Global and cross-validation segmentation performance alongside paired statistical significance. The segmentation network utilizes SegFormer (MiT-B0), and the synthetic representations are generated using the optimal MiT-B5 regression model. Best results among the proposed strategies are bolded , and second-best are underlined (excluding the Original S2 Oracle). Panels C and D report Wilcoxon signed-rank tests with a defined significance threshold of

p<0.05

Figure 4: Impact of regression quality on segmentation performance. Scatter plot illustrating the relationship between the synthetic image regression quality (

R^{2}

) and the resulting segmentation accuracy (IoU). The purple circles represent the Synthetic S2-only framework, while the blue diamonds represent the Proposed Combined model. Solid trendlines indicate the polynomial fit for each approach. The horizontal dashed orange line denotes the Global S1 Baseline (

\text{IoU}\approx.7988

), and the optimal combined configuration is explicitly annotated.

Panel A: Event-Stratified Cross-Validation
Tile Size	Configuration	Input Modality	IoU		F1-Score
Tile Size	Configuration	Input Modality	CV (Mean $\pm$ Std)	Global	CV (Mean $\pm$ Std)	Global
$128\times 128$	Reg-B0 / Seg-B0	S1 Only	0.6529 $\pm$ 0.1608	0.7285	0.7782 $\pm$ 0.1220	0.8429
		Syn S2 Only	$0.5724\pm 0.2633$	$0.7197$	$0.6847\pm 0.2624$	$0.8370$
		Combined	0.6738 $\pm$ 0.1253	0.7503	0.7988 $\pm$ 0.0845	0.8574
	Reg-B3 / Seg-B0	S1 Only	$0.6150\pm 0.1613$	$0.7018$	$0.7491\pm 0.1254$	$0.8248$

Table 5: Comprehensive ablation study on architectural capacity and input tile size under both Event-Stratified and Mixed cross-validation frameworks. Performance is evaluated using Intersection over Union (IoU) and F1-Score across Cross-Validation (CV) and Global Micro-aggregated metrics. The configuration is denoted as Reg-[X] / Seg-[Y] , indicating the capacity of the regression and segmentation backbones, respectively. Optimal metrics are bolded ; second-best values are underlined .

Panel A: Event-Stratified Cross-Validation
Tile Size	Configuration	Global Scenes (p-values)
Tile Size	Configuration	Comb $>$ S1	Syn S2 $>$ S1	Comb $>$ Syn S2
$128\times 128$	Reg-B0 / Seg-B0	$\mathbf{1.47}\times\mathbf{10}^{\mathbf{-78}}$	$1.000$	$\mathbf{3.32}\times\mathbf{10}^{\mathbf{-149}}$
	Reg-B3 / Seg-B0	$\mathbf{1.88}\times\mathbf{10}^{\mathbf{-73}}$	$\mathbf{5.69}\times\mathbf{10}^{\mathbf{-13}}$	$\mathbf{8.78}\times\mathbf{10}^{\mathbf{-27}}$
	Reg-B3 / Seg-B3	$1.000$	$1.000$	$\mathbf{2.43}\times\mathbf{10}^{\mathbf{-43}}$
	Reg-B5 / Seg-B0	$\mathbf{1.49}\times\mathbf{10}^{\mathbf{-113}}$	$\mathbf{3.15}\times\mathbf{10}^{\mathbf{-17}}$	$\mathbf{1.11}\times\mathbf{10}^{\mathbf{-35}}$

Table 6: Statistical significance testing (Wilcoxon signed-rank test) for image-wise Intersection over Union (IoU). P-values indicate whether the performance improvements of alternative input modalities over baselines are significant (

p<0.05

, marked in bold ).

Figure 5: Qualitative evaluation of synthetic NDWI generation. The top row displays the colorbar scales for the input Sentinel-1 SAR data (left) and the Normalized Difference Water Index (NDWI) continuum (right). The image rows track representative test patches, demonstrating how radar backscatter maps to continuous water index values. The columns show the Input SAR (first and second column), Predicted NDWI (third column), and Target NDWI (fourth column). Bounding boxes display patch-level Mean Squared Error (MSE) and Coefficient of Determination ( $R^{2}$ ). This high-fidelity translation provides the essential spectral mapping required to guide the downstream binary water segmentation task.

Figure 6: Comparative analysis of downstream water segmentation performance. The top row provides a legend defining the binary classification mapping for target surface water (dark blue) and terrestrial land background context (slate gray). The image matrix evaluates model segmentation performance across representative test patches. The columns show predictions from the Synthetic-Only model (first column), raw SAR-Only model (second column), Combined translation-segmentation model (third column), and the Ground Truth target mask (fourth column). Bounding boxes display patch-level Intersection over Union (IoU) metrics, illustrating how integrating synthetic index translation mitigates backscatter noise and stabilizes surface feature extraction.

Figure S1 : High-performance NDWI predictions from SAR data. These examples display the model’s ability to synthesize optical water indices directly from Sentinel-1 inputs. The columns show the Input SAR (first and second column), Target NDWI (third column), and Predicted NDWI (fourth column). The consistently low MSE values ( $0.0026-0.0074$ ) and the visual preservation of water boundaries highlight the model’s capacity to suppress speckle noise while recovering structural details.

Input Modality	Accuracy	Precision	Recall	Specificity
Panel A: Event Stratified Splitting (Cross-Validation Mean $\pm$ SD)
S1 Only	0.9699 $\pm$ 0.0103	0.8011 $\pm$ 0.0857	0.7456 $\pm$ 0.1856	0.9857 $\pm$ 0.0064
S1 PCA	0.9695 $\pm$ 0.0080	0.7835 $\pm$ 0.0887	0.7910 $\pm$ 0.1049	0.9825 $\pm$ 0.0073
S1 PCA + Raw S1	0.9716 $\pm$ 0.0104	0.8229 $\pm$ 0.0745	0.7361 $\pm$ 0.1835	0.9878 $\pm$ 0.0045
Synthetic S2 (Reg)	0.9717 $\pm$ 0.0094	0.8161 $\pm$ 0.0912	0.7576 $\pm$ 0.1108	0.9868 $\pm$ 0.0058
Combo (Synth S2 + S1)	0.9741 $\pm$ 0.0070	0.8153 $\pm$ 0.0764	0.7972 $\pm$ 0.1108	0.9859 $\pm$ 0.0048

Table S1 : Supplementary: Extended performance metrics (Accuracy, Precision, Recall, and Specificity) separated into Cross-Validation and Global results. The segmentation network utilizes SegFormer (MiT-B0), and the synthetic representations are generated using the optimal MiT-B5 regression model. Best results among the proposed strategies are bolded , and second-best are underlined (excluding the Original S2 Oracle).

Panel A: Event-Stratified Cross-Validation
Tile Size	Configuration	Input Modality	Specificity		Recall
Tile Size	Configuration	Input Modality	CV (Mean $\pm$ Std)	Global	CV (Mean $\pm$ Std)	Global
$128\times 128$	Reg-B0 / Seg-B0	S1 Only	0.9859 $\pm$ 0.0043	0.9854	0.7698 $\pm$ 0.1551	0.8234
		Syn S2 Only	$0.9853\pm 0.0042$	$0.9850$	$0.6710\pm 0.2817$	$0.8162$
		Combined	0.9860 $\pm$ 0.0062	0.9851	0.7904 $\pm$ 0.0964	0.8503
	Reg-B3 / Seg-B0	S1 Only	$0.9848\pm 0.0071$	0.9848	0.7218 $\pm$ 0.1696	0.7971

Table S2 : Comprehensive ablation study on architectural capacity and input tile size under both Event-Stratified and Mixed cross-validation frameworks. Performance is evaluated using Specificity and Recall across Cross-Validation (CV) and Global Micro-aggregated metrics. The configuration is denoted as Reg-[X] / Seg-[Y] , indicating the capacity of the regression and segmentation backbones, respectively. Optimal metrics are bolded ; second-best values are underlined .

Panel A: Event-Stratified Cross-Validation
Tile Size	Configuration	Input Modality	Precision		Accuracy
Tile Size	Configuration	Input Modality	CV (Mean $\pm$ Std)	Global	CV (Mean $\pm$ Std)	Global
$128\times 128$	Reg-B0 / Seg-B0	S1 Only	0.7957 $\pm$ 0.0913	0.8633	0.9716 $\pm$ 0.0101	0.9691
		Syn S2 Only	$0.7111\pm 0.2346$	$0.8589$	$0.9683\pm 0.0055$	$0.9680$
		Combined	0.8119 $\pm$ 0.0895	0.8645	0.9733 $\pm$ 0.0074	0.9715
	Reg-B3 / Seg-B0	S1 Only	$0.7963\pm 0.0808$	$0.8544$	$0.9681\pm 0.0098$	$0.9659$

Table S3 : Comprehensive ablation study on architectural capacity and input tile size under both Event-Stratified and Mixed cross-validation frameworks. Performance is evaluated using Precision and Accuracy across Cross-Validation (CV) and Global Micro-aggregated metrics. The configuration is denoted as Reg-[X] / Seg-[Y] , indicating the capacity of the regression and segmentation backbones, respectively. Optimal metrics are bolded ; second-best values are underlined .

Remote Sensing Image Segmentation

Remote sensing image segmentation labels land cover and objects in satellite, aerial, radar, and multispectral imagery.

See all 220 papers

MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing
Mar 18, 2026Yimin Wei, Aoran Xiao, Hongruixuan Chen +2Remote Sensing Image Segmentation Open-Vocabulary Semantic Segmentation
The University of Tokyo · RIKEN AIP
Lean Learning Beyond Clouds: Efficient Discrepancy-Conditioned Optical-SAR Fusion for Semantic Segmentation
Mar 21, 2026Chenxing Meng, Wuzhou Quan, Yingjie Cai +3Efficient Semantic Segmentation Cloud Removal
School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
IB-HFN: Information Bottleneck-Driven SAR-Optical Fusion Network for High-Fidelity Cloud Removal
Jun 8, 2026Haojun Guo, Fan Feng, Ziquan Wang +2Remote Sensing Multimodal Fusion
Institute of Geospatial Information, Information Engineering University, Zhengzhou 450001, China

Mar 18, 2026J/K move · Enter open · S save

MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing

Yimin Wei, Aoran Xiao, Hongruixuan Chen, Junshi Xia+1

The University of Tokyo · RIKEN AIP

Remote Sensing Image Segmentation Open-Vocabulary Semantic Segmentation Multimodal Semantic Segmentation Multimodal Robustness Multimodal Sensor Fusion Multi-Source Image Fusion

Open paper AskPDF arXiv

Open-vocabulary segmentation enables pixel-level recognition from an open set of textual categories, allowing generalization beyond fixed classes. Despite great potential in remote sensing, progress in this area remains largely limited to clear-sky optical data and struggles under cloudy or haze-contaminated conditions. We present MM-OVSeg, a multimodal Optical-SAR fusion framework for resilient open-vocabulary segmentation under adverse weather conditions. MM-OVSeg leverages the complementary strengths of the two modalities--optical imagery provides rich spectral semantics, while synthetic aperture radar (SAR) offers cloud-penetrating structural cues. To address the cross-modal domain gap and the limited dense prediction capability of current vision-language models, we propose two key designs: a cross-modal unification process for multi-sensor representation alignment, and a dual-encoder fusion module that integrates hierarchical features from multiple vision foundation models for text-aligned multimodal segmentation. Extensive experiments demonstrate that MM-OVSeg achieves superior robustness and generalization across diverse cloud conditions. The source dataset and code are available at https://github.com/Jimmyxichen/MM-OVSeg.

Abstract

Remote Sensing Image Segmentation

Explore similar work

MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing

Lean Learning Beyond Clouds: Efficient Discrepancy-Conditioned Optical-SAR Fusion for Semantic Segmentation

IB-HFN: Information Bottleneck-Driven SAR-Optical Fusion Network for High-Fidelity Cloud Removal