SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

Jeonghyeok Do; Munchurl Kim

cs.CVOct 7, 2026

SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

Authors: Jeonghyeok Do, Munchurl Kim

Organizations: Korea Advanced Institute of Science and Technology (KAIST)

Abstract

Synthetic aperture radar (SAR) and electro-optical (EO) imagery provide complementary observations: SAR enables day-and-night, weather-resilient sensing, whereas EO provides rich appearance and fine-grained semantic cues. We introduce SAREO-FM, which avoids forcing a single token stream to serve two distinct roles: modality tokens preserve how each sensor observes the scene through masked reconstruction, while learnable semantic queries capture what the scene contains under guidance from a pretrained vision foundation model (VFM). By jointly encoding these queries with SAR and EO tokens, the queries acquire modality-grounded semantic context, while the modality-token outputs remain the explicit targets of masked reconstruction. This design assigns semantic and reconstruction supervision to separate token streams while preserving their interaction within the shared encoder. Pretrained on the million-scale SAR-1M corpus, SAREO-FM achieves strong unimodal transfer for both SAR-only and EO-only inputs, while delivering substantial gains from joint SAR--EO observations on tasks that benefit from complementary sensing.

Figures & tables

Figure 1: Motivation of SAREO-FM. Prior methods directly align image tokens with EO-derived VFM features, coupling semantic alignment with pixel reconstruction. SAREO-FM instead supervises learnable semantic queries while reconstructing SAR and EO from modality tokens, separating the destinations of the two objectives within a unified encoder.

Figure 2: Overview of SAREO-FM. Visible SAR and EO tokens interact with learnable semantic queries through a unified encoder. The modality-token outputs are decoded independently for masked pixel reconstruction, whereas the semantic-query outputs are aligned patch-wise with features from a frozen pretrained VFM. Semantic queries are excluded from the reconstruction decoders, separating the destinations of pixel-level and semantic supervision. Modality masking exposes the encoder to SAR-only, EO-only, and joint SAR–EO inputs; during downstream transfer, task heads selectively consume modality-token outputs, semantic-query outputs, or both.

Method	Venue	Backbone	FUSAR-Ship		MSTAR		SAR-ACD
Method	Venue	Backbone	40-shot	30%	40-shot	30%	40-shot	30%
ResNet-50 He et al. (2016)	CVPR’16	ResNet-50	–	58.41	–	89.94	–	59.70
Swin Transformer Liu et al. (2021)	ICCV’21	Swin-B	–	60.79	–	82.97	–	67.50
BEiT Bao et al. (2021)	ICLR’22	ViT-B	59.70	71.13	40.70	69.75	–	79.77
CROMA Fuller et al. (2023)	NeurIPS’23	ViT-B	83.71	–	–	–	–	88.99
SAR-JEPA Li et al. (2023)	ISPRS JPRS’24	ViT-B	85.80	–	91.60	–	75.50	–

Table 1: SAR target classification with end-to-end fine-tuning. The upper block reports published OA (%) under the source-specific protocol of each method and is included only as context. The lower block reports our controlled evaluation with identical data splits and downstream pipelines; mean and standard deviation over three seeds are shown. ∗ denotes our re-evaluation of the official SARMAE checkpoint. Bold denotes the best controlled result.

Method / Feature	SAR				EO
Method / Feature	1-shot	5-shot	10-shot	40-shot	1-shot	5-shot	10-shot	40-shot
DINOv3 (teacher)	38.17 $\,\pm\,4.61$	50.72 $\,\pm\,2.68$	56.01 $\,\pm\,1.81$	64.35 $\,\pm\,1.56$	59.51 $\,\pm\,3.53$	75.96 $\,\pm\,3.15$	83.34 $\,\pm\,1.42$	90.88 $\,\pm\,0.65$
SARMAE ∗	37.22 $\,\pm\,5.71$	48.64 $\,\pm\,3.00$	53.97 $\,\pm\,1.39$	64.72 $\,\pm\,1.12$	43.56 $\,\pm\,6.57$	59.57 $\,\pm\,3.05$	66.49 $\,\pm\,1.97$	78.36 $\,\pm\,1.07$
CoDe-MAE	–	–	59.88 †	–	–	–	81.18 †	–
SAREO-FM ( $\mathrm{both}$ )	39.07 $\,\pm\,5.98$	53.54 $\,\pm\,2.21$	59.36 $\,\pm\,1.46$	69.24 $\,\pm\,0.96$	64.41 $\,\pm\,5.05$	81.09 $\,\pm\,1.61$	86.76 $\,\pm\,1.24$	92.58 $\,\pm\,0.75$

Table 2: Frozen few-shot EO transfer. We report accuracy (%) for SAR and EO inputs over five support draws. CoDe-MAE values use its separate 10-shot protocol and are included only as context.

Input	Method	Feature	Benchmark
Input	Method	Feature	So2Sat LCZ42	BigEarthNet-MM
SAR+EO	Random initialization	$\mathrm{mod}$	40.56 $\,\pm\,2.25$	54.29 $\,\pm\,2.30$
EO	DINOv3 (teacher)	$\mathrm{mod}$	54.25 $\,\pm\,3.87$	64.73 $\,\pm\,1.43$
EO	SAREO-FM	$\mathrm{mod}$	58.14 $\,\pm\,2.90$	64.42 $\,\pm\,1.44$
SAR	SARMAE ∗	$\mathrm{mod}$	28.96 $\,\pm\,1.68$	57.44 $\,\pm\,0.69$
SAR	SAREO-FM	$\mathrm{mod}$	29.59 $\,\pm\,2.64$	60.47 $\,\pm\,0.79$

Table 3: Transfer on paired SAR–EO benchmarks. We report OA (%) on 5-shot So2Sat LCZ42 and micro-AP (%) on 5-shot BigEarthNet-MM.

Figure 5: Qualitative transfer on BRIGHT Chen et al. (2025a) . Given pre-event EO and post-event SAR observations, SAREO-FM fusion recovers destroyed buildings that the corresponding single-modality models predict as intact or background.

Pretraining configuration	MSTAR (OA $\uparrow$ )	FUSAR (OA $\uparrow$ )	Avg. (OA $\uparrow$ )	$\Delta$
Random initialization	26.26 $\,\pm\,1.86$	47.86 $\,\pm\,4.89$	37.06	–
Plain MAE He et al. (2022)	66.16 $\,\pm\,3.15$	76.46 $\,\pm\,2.52$	71.31	$+34.25$
$+$ DINOv3 supervision	73.49 $\,\pm\,2.31$	79.90 $\,\pm\,2.49$	76.70	$+5.39$
$+$ Decoupled semantic supervision	77.11 $\,\pm\,2.14$	82.55 $\,\pm\,2.15$	79.83	$+3.13$
$+$ Raw EO modality ( SAREO-FM )	78.41 $\,\pm\,2.14$	84.00 $\,\pm\,2.88$	81.21	$+1.38$

Table 4: Cumulative component ablation. All variants use matched pretraining and evaluation protocols. We report 10-shot SAR linear-probing performance. Avg. denotes the arithmetic mean across the two benchmarks, while

\Delta

measures the gain over the preceding configuration.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Section	Contents
Section A	Main-paper results and additional controls
Section B	Masking modes and training objectives
Section C	Pre-training corpus, datasets, and evaluation scope
Section D	Architecture and pre-training details
Section E	Feature readouts and downstream protocols
Section F	Inference cost

Appendix

Table 5: Overview of the Appendix .

Dataset	Method / Feature	1-shot	2-shot	5-shot	10-shot	20-shot	40-shot
	SARMAE ∗	54.19 $\,\pm\,8.69$	64.95 $\,\pm\,6.93$	70.03 $\,\pm\,3.48$	78.18 $\,\pm\,3.28$	83.44 $\,\pm\,2.03$	88.35 $\,\pm\,0.83$
FUSAR-Ship	SAREO-FM ( $\mathrm{both}$ )	55.56 $\,\pm\,7.50$	64.79 $\,\pm\,3.81$	72.84 $\,\pm\,3.97$	80.46 $\,\pm\,1.81$	85.32 $\,\pm\,1.79$	89.81 $\,\pm\,0.70$
	SAREO-FM ( $\mathrm{sem}$ )	55.79 $\,\pm\,9.07$	68.54 $\,\pm\,4.35$	75.74 $\,\pm\,2.46$	81.65 $\,\pm\,2.91$	85.50 $\,\pm\,1.71$	89.56 $\,\pm\,1.22$
	SARMAE ∗	35.09 $\,\pm\,7.77$	39.56 $\,\pm\,6.52$	47.12 $\,\pm\,3.16$	51.68 $\,\pm\,4.08$	60.30 $\,\pm\,2.86$	70.11 $\,\pm\,1.82$
	SAREO-FM ( $\mathrm{both}$ )	33.81 $\,\pm\,6.30$	36.31 $\,\pm\,6.91$	44.78 $\,\pm\,4.56$	51.87 $\,\pm\,4.12$	60.69 $\,\pm\,2.75$	71.35 $\,\pm\,2.01$
SAR-ACD	SAREO-FM ( $\mathrm{sem}$ )	33.64 $\,\pm\,6.95$	36.87 $\,\pm\,5.83$	48.10 $\,\pm\,3.02$	53.65 $\,\pm\,3.19$	61.65 $\,\pm\,3.35$	71.72 $\,\pm\,1.74$

Appendix

Table 6: Exact frozen few-shot SAR classification results corresponding to Fig. 3 of the main paper. We freeze the pretrained encoder and optimize only a linear downstream classifier. We report mean OA (%) and sample standard deviation over five matched support draws.

\mathrm{sem}

uses semantic-query features, whereas

\mathrm{both}

concatenates modality-token and semantic-query features. SARMAE ∗ denotes the official checkpoint evaluated with the same downstream protocol. Bold and underline denote the best and second-best values within each dataset and shot count, respectively; shaded rows are ours.

Configuration	SAR	EO	Queries	Total
Independent/shared masking	64	64	256	384
Complete EO dropping	64	0	256	320
Complete SAR dropping	0	64	256	320
Unpaired SAR training	64	0	256	320
SAR-only inference	256	0	256	512
EO-only inference	0	256	256	512

Appendix

Table 7: Shared-encoder sequence composition. Training counts use the 75% patch-mask ratio; downstream inference uses all available image patches.

Sample / mode	Prob.	Student encoder image streams	Teacher input	Active objective
Paired, independent masks	0.35	Visible SAR + visible EO	Clean EO	$\mathcal{L}_{\mathrm{pix}}^{s}+\mathcal{L}_{\mathrm{pix}}^{e}+0.5\mathcal{L}_{\mathrm{align}}$
Paired, shared mask	0.35	Visible SAR + visible EO	Clean EO	$\mathcal{L}_{\mathrm{pix}}^{s}+\mathcal{L}_{\mathrm{pix}}^{e}+0.5\mathcal{L}_{\mathrm{align}}$
Paired, complete EO dropping	0.15	Visible SAR only	Clean EO, teacher only	$\mathcal{L}_{\mathrm{pix}}^{s}+0.5\mathcal{L}_{\mathrm{align}}$
Paired, complete SAR dropping	0.15	Visible EO only	Clean EO	$\mathcal{L}_{\mathrm{pix}}^{e}+0.5\mathcal{L}_{\mathrm{align}}$
Unpaired SAR	–	Visible SAR only	Unavailable	$\mathcal{L}_{\mathrm{pix}}^{s}$
Unpaired EO	–	Not sampled in this work	–	–

Appendix

Table 8: Student inputs, teacher availability, and active losses. Retained image streams are masked by 75%, and all semantic queries remain active. The loss coefficients are

\lambda_{s}=1

\lambda_{e}=1

, and

\lambda_{\mathrm{align}}=0.5

Dataset	Input	Classes	Metric	Protocol	SAR-1M relation	Role in this work
MSTAR	SAR	10	OA	1–40-shot / 30%	In corpus	In-corpus target transfer
FUSAR-Ship	SAR	10	OA	1–40-shot / 30%	In corpus	In-corpus ship transfer
SAR-ACD	SAR	5	OA	1–40-shot / 30%	Out of corpus	Primary out-of-corpus SAR test
EuroSAT protocol	EO or SAR	10	OA	1/5/10/40-shot	No out-of-corpus claim	Unimodal scene transfer
NWPU-RESISC45	EO	45	OA	Few-shot / higher supervision	No out-of-corpus claim	EO scene transfer
AID	EO	30	OA	Few-shot / higher supervision	No out-of-corpus claim	EO scene transfer

Appendix

Table 9: Downstream datasets and interpretation. OA denotes overall accuracy; micro-AP denotes micro-averaged average precision. The paired EuroSAT protocol uses geospatially matched SAR observations in addition to standard EO imagery; standard EuroSAT itself is EO-only.

Component	Specification
Input resolution	$256\times 256$
Patch size / grid	$16\times 16$ / $16\times 16$
SAR / EO tokens	256 per available modality
Shared encoder	ViT-B/16, 12 blocks, $D=768$
Attention / MLP	12 heads / ratio 4
Semantic queries	256 learnable, spatially indexed

Appendix

Table 10: Architecture summary.

Component	Pretraining	Downstream
Shared SAREO encoder	Trainable	Used
Semantic queries	Trainable	Used
SAR/EO patch embeddings	Trainable	Used as available
SAR/EO reconstruction decoders	Trainable	Discarded
Semantic projector	Trainable	Discarded
DINOv3-7B teacher	Frozen	Discarded

Appendix

Table 11: Lifecycle of SAREO-FM components.

Available input	$\mathrm{mod}$	$\mathrm{sem}$	$\mathrm{both}$
SAR or EO	768	768	1,536
SAR + EO	1,536	768	2,304

Appendix

Table 12: Image-level readout dimensions.

Explore similar work

Cross-modal learning for SAR target recognition using optical vision foundation models
Sep 7, 2026Lucas Hirsch, James R. Hopgood, Javid Khan +2Cross-Modal Alignment Transfer Learning
School of Engineering, University of Edinburgh, United Kingdom · Leonardo UK, United Kingdom · School of Engineering and Physical Sciences, Heriot-Watt University, United Kingdom
ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation
Sep 1, 2026Jeonghyeok Do, Seungchul Lee, Munchurl KimFlow Matching Image-to-Image Translation
KAIST · Stellarvision Inc.
TAR: Text Semantic Assisted Cross-modal Image Registration Framework for Optical and SAR Images
May 12, 2026Zhuoyu Cai, Dou Quan, Ning Huyan +3Remote Sensing Synthetic Aperture Radar
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education of China, School of Artificial Intelligence, Xidian University, Xi’an 710071, China

Sep 7, 2026, cs.CVJ/K move · Enter open · S save

Cross-modal learning for SAR target recognition using optical vision foundation models

Lucas Hirsch, James R. Hopgood, Javid Khan, Yoann Altmann+1

School of Engineering, University of Edinburgh, United Kingdom · Leonardo UK, United Kingdom · School of Engineering and Physical Sciences, Heriot-Watt University, United Kingdom

Cross-Modal Alignment Transfer Learning Synthetic Aperture Radar

Open paper AskPDF arXiv

Synthetic Aperture Radar (SAR) is an important modality in a wide range of imaging applications due to its versatile, long range and near all weather operating capabilities. However, Automatic Target Recognition (ATR) remains a challenging problem due to limited labelled data, the strong speckle in SAR images and the significant domain gap between SAR and more abundant optical imagery. In contrast, electro-optical (EO) imagery benefits from massive datasets, clearer visual structure and powerful foundation models. In this work, we investigate how vision foundation models trained on optical data can provide class level supervision for SAR classification. We propose a cross-modal EO to SAR prototype alignment framework in which a frozen EO encoder, based on a DINOv3 vision foundation model, is used to construct class level optical prototypes without requiring strict EO/SAR pairs. A SAR model is then trained to classify SAR images while aligning its embeddings to the corresponding EO class prototype. At inference time, the SAR model operates independently, without access to optical imagery. We evaluate our approach on the UNICORNv2 dataset, an EO and SAR dataset of civilian vehicles with heavily speckled images and severe class imbalance. EO prototype alignment improves SAR classification accuracy over frozen DINOv3, SAR only finetuning and unpaired distribution alignment baselines, and t-SNE visualizations provide qualitative evidence of clearer separation among classes in the trained SAR embedding space. These results suggest that optical vision foundation models, despite being trained on visible spectrum imagery, provide transferable information for SAR image classification, offering a practical method for using large scale pretrained vision foundation models across challenging sensing modalities.

Abstract

Explore similar work

Cross-modal learning for SAR target recognition using optical vision foundation models

ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation

TAR: Text Semantic Assisted Cross-modal Image Registration Framework for Optical and SAR Images