Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation

Siyuan Liu; Xuze Zhang; Yongshun Wang; Licong Pan; Hang Liu; Huihui Li

cs.CVSep 4, 2026

Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation

Authors: Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, Huihui Li

Organizations: School of Automation, Northwestern Polytechnical University, Xi’an 710129, China · School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710129, China

Abstract

Synthetic Aperture Radar (SAR) images have all-weather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images. Generative models for SAR-to-optical (S2O) conversion can improve visual interpretability, but existing methods often ignore the constraints on semantic structure, which are necessary for downstream tasks, for the sake of visual effects. We propose a unified collaborative dual-task learning framework, termed BMT (Bridging Modalities and Tasks), that jointly optimizes S2O image translation and semantic segmentation through a shared hierarchical Vision Transformer. The framework integrates: (1) a LocalViTBlock that fuses global self-attention with spatial depthwise convolution through a learnable gating mechanism; (2) an enhanced output module combining multi-scale refinement processing, color correction and anti-aliasing, which calibrates channel-level color statistics through feature fusion; (3) a ControlNet-style conditional injection mechanism that encodes SAR wavelet features and segmentation labels into a multi-scale feature pyramid and injects them at each encoder layer through zero-initialized convolution; (4) a bounded Kendall uncertainty weighting scheme that prevents either task from dominating the shared representation. We evaluate the framework under both paired and unpaired translation settings, on the public WHU-OPT-SAR paired dataset and a self-constructed unpaired ship dataset built from HRSID and DIOR, respectively. The experimental results show that the proposed method achieves competitive S2O translation quality and semantic segmentation performance. The dataset and source code have been publicly released at https://github.com/Lewisyuaner/BMT-S2O-main.

Figures & tables

Fig. 1: Overall architecture of the proposed framework for unpaired S2O image translation and semantic segmentation. The shared hierarchical ViT encoder processes the input through four stages. A ControlNet-style condition injection mechanism encodes SAR wavelet features and segmentation labels into a multi-scale feature pyramid, injected at every encoder level via zero-initialized convolutions. The bottleneck features are fed to dual decoders: a generation decoder with multi-scale color correction and a segmentation decoder with ViT decoder stages.

Fig. 2: Visual comparison of SAR-to-optical translation results on the WHU-OPT-SAR dataset.

Fig. 3: Visual comparison of semantic segmentation results (obtained with a pre-trained FCN-32s) on the optical images generated by different S2O methods on the WHU-OPT-SAR dataset.

Method	SSIM $\uparrow$ (%)	PSNR $\uparrow$ (dB)	FID $\downarrow$	KID $\downarrow$ ( $\times 100$ )
CycleGAN	39.63	19.82	174.18	7.72
Pix2pix	39.25	21.54	192.21	10.98
Pix2pixHD	38.89	22.17	229.32	16.54
StegoGAN	23.98	17.47	203.19	10.28
PatchGCL	37.91	19.73	190.72	10.54
S-CycleGAN	35.21	21.96	311.45	28.15

TABLE I: Comparison on the WHU-OPT-SAR dataset.

Method	Results		Category(mIoU)
Method	mIoU $\uparrow$	mPA $\uparrow$	Farmland	City	Village	Water	Forest	Road	Others
Optical	40.83	53.34	51.65	54.28	22.29	65.18	67.57	6.46	18.38
SAR Image	34.06	43.34	45.49	46.08	15.18	62.40	65.47	1.99	1.83
CycleGAN	25.89	35.05	34.53	41.06	8.62	47.09	48.50	0.62	0.86
Pix2pix	22.41	30.46	38.29	14.97	8.05	35.61	58.92	0.33	0.67
Pix2pixHD	27.20	36.33	39.88	24.47	14.78	47.37	60.98	1.22	1.69

TABLE II: SEGMENTATION QUANTITATIVE EVALUATION RESULTS OF DIFFERENT METHODS USING FCN ON THE WHU-OPT-SAR DATASET

Fig. 4: Visual comparison of S2O translation results on the HRSID-DIOR ship dataset.

Fig. 5: Visual comparison of semantic segmentation results on the HRSID-DIOR ship dataset.

Method	FID $\downarrow$	KID ( $\times 100$ ) $\downarrow$
CycleGAN	176.757	7.357
U-GAT-IT	166.900	8.096
CUT	145.782	5.741
F-LSeSim	165.002	7.888
NICE-GAN	156.878	6.831
StegoGAN	182.549	9.624

TABLE III: Quantitative comparison on the S2O translation task.

Method	Overall		Category IoU		Category Acc
Method	mIoU $\uparrow$	mPA $\uparrow$	Bg.	Ship	Bg.	Ship
U-Net	88.41	91.80	99.40	77.43	99.80	83.90
Mask R-CNN	82.33	89.19	98.99	65.67	99.51	78.88
PSPNet	83.90	89.55	99.11	68.68	99.61	79.49
HRNet	88.25	91.62	99.39	77.04	99.79	83.45
SegFormer	82.79	90.17	99.01	66.57	99.47	80.87

TABLE IV: Quantitative comparison on the semantic segmentation task.

Branchs		Quantitative Metrics
Seg.	Trans.	SSIM	PSNR	FID	KID	mIoU	mPA
$\checkmark$	$\times$	-	-	-	-	39.08	49.34
$\times$	$\checkmark$	40.15	20.80	178.54	8.19	-	-
$\checkmark$	$\checkmark$	42.25	22.10	159.20	6.47	41.17	51.36

TABLE V: Ablation study of task branchs and quantitative metrics on WHU-OPT-SAR.

Modules			Quantitative Metrics
Local.	Cond.	perc.	SSIM	PSNR	FID	KID	mIoU	mPA
$\times$	$\times$	$\times$	42.25	22.10	159.20	5.47	41.17	51.36
$\checkmark$	$\times$	$\times$	43.73	22.30	158.46	5.21	41.78	52.02
$\checkmark$	$\checkmark$	$\times$	44.90	22.30	151.48	5.47	44.04	54.39
$\checkmark$	$\checkmark$	$\checkmark$	45.54	22.40	156.31	6.18	44.09	54.73

TABLE VI: Ablation study of module configurations and quantitative metrics on WHU-OPT-SAR.

Modules		Quantitative Metrics
Local.	Cond.	mIoU $\uparrow$	mPA $\uparrow$	FID $\downarrow$	KID ( $\times 100$ ) $\downarrow$
$\times$	$\times$	84.54	90.44	174.49	6.62
$\surd$	$\times$	88.01	93.07	173.73	6.34
$\surd$	$\surd$	87.62	92.72	164.21	6.23

TABLE VII: Ablation study of the LocalViTBlock and Condition Module on HRSID-DIOR.

Loss Functions			Quantitative Metrics
$\mathcal{L}_{\text{Seg}}^{\text{cyc}}$	$\mathcal{L}_{\text{color}}$	$\mathcal{L}_{\text{hist}}$	mIoU $\uparrow$	mPA $\uparrow$	FID $\downarrow$	KID ( $\times 100$ ) $\downarrow$
$\times$	$\times$	$\times$	87.62	92.72	164.21	6.23
$\surd$	$\times$	$\times$	88.32	92.25	168.20	6.58
$\surd$	$\surd$	$\times$	88.91	93.03	165.25	6.19
$\surd$	$\surd$	$\surd$	88.51	93.07	161.81	5.81

TABLE VIII: Ablation study of the loss functions on HRSID-DIOR.

Image-to-Image Translation

Image-to-image translation maps images between visual domains or modalities while preserving relevant content, structure, and semantics.

See all 115 papers

Oct 7, 2026

SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

Synthetic aperture radar (SAR) and electro-optical (EO) imagery provide complementary observations: SAR enables day-and-night, weather-resilient sensing, whereas EO provides rich appearance and fine-grained semantic cues. We introduce SAREO-FM, which avoids forcing a single token stream to serve two distinct roles: modality tokens preserve how each sensor observes the scene through masked reconstruction, while learnable semantic queries capture what the scene contains under guidance from a pretrained vision foundation model (VFM). By jointly encoding these queries with SAR and EO tokens, the queries acquire modality-grounded semantic context, while the modality-token outputs remain the explicit targets of masked reconstruction. This design assigns semantic and reconstruction supervision to separate token streams while preserving their interaction within the shared encoder. Pretrained on the million-scale SAR-1M corpus, SAREO-FM achieves strong unimodal transfer for both SAR-only and EO-only inputs, while delivering substantial gains from joint SAR--EO observations on tasks that benefit from complementary sensing.

Jeonghyeok Do, Munchurl Kim

Korea Advanced Institute of Science and Technology (KAIST)

Remote Sensing Image Understanding Multimodal Pretraining Multimodal Foundation Models Cross-Modal Representation Learning Masked Image Modeling

Ask PDF

Mar 18, 2026

MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing

Open-vocabulary segmentation enables pixel-level recognition from an open set of textual categories, allowing generalization beyond fixed classes. Despite great potential in remote sensing, progress in this area remains largely limited to clear-sky optical data and struggles under cloudy or haze-contaminated conditions. We present MM-OVSeg, a multimodal Optical-SAR fusion framework for resilient open-vocabulary segmentation under adverse weather conditions. MM-OVSeg leverages the complementary strengths of the two modalities--optical imagery provides rich spectral semantics, while synthetic aperture radar (SAR) offers cloud-penetrating structural cues. To address the cross-modal domain gap and the limited dense prediction capability of current vision-language models, we propose two key designs: a cross-modal unification process for multi-sensor representation alignment, and a dual-encoder fusion module that integrates hierarchical features from multiple vision foundation models for text-aligned multimodal segmentation. Extensive experiments demonstrate that MM-OVSeg achieves superior robustness and generalization across diverse cloud conditions. The source dataset and code are available at https://github.com/Jimmyxichen/MM-OVSeg.

Yimin Wei, Aoran Xiao, Hongruixuan Chen +2

The University of Tokyo · RIKEN AIP

Remote Sensing Image Segmentation Open-Vocabulary Semantic Segmentation Multimodal Semantic Segmentation Multimodal Robustness Multimodal Sensor Fusion Multi-Source Image Fusion

Ask PDF

Sep 26, 2026

GeoSET: Generalist Foundation Model for SAR-to-EO Image Translation

Paired synthetic aperture radar (SAR) and electro-optical (EO) imagery is increasingly available across sensors, resolutions, and geographic regions. Yet existing SAR-to-EO image translation (SET) methods are typically trained on a single, limited-scale dataset, producing models specialized to particular sensing conditions. We introduce GeoSET, the first generalist model for SET, built around a single pretrained parent that is adapted to downstream datasets under a common protocol. We curate over 3 million high-quality SAR--EO pairs from a collection of more than 10 million SAR observations, spanning diverse sensors, spatial resolutions, and ground sampling distances. To bridge the modality gap between SAR observations and a pretrained image generator, we develop a speckle-robust SAR encoder and pretrain the conditional generator on this heterogeneous corpus. The resulting parent supports efficient adaptation across downstream datasets through low-rank adaptation (LoRA), updating only 0.60% of the generator parameters and requiring approximately one hour per dataset. Across six downstream benchmarks, GeoSET achieves state-of-the-art results in FID and DISTS with full fine-tuning or LoRA, demonstrating effective transfer across heterogeneous SAR-EO domains.

Jeonghyeok Do, Munchurl Kim

Korea Advanced Institute of Science and Technology (KAIST)

Image-to-Image Translation Geospatial Foundation Models Synthetic Aperture Radar

Ask PDF

Abstract

Image-to-Image Translation

Explore similar work

SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing

GeoSET: Generalist Foundation Model for SAR-to-EO Image Translation