Preserving the Manifold: Frozen Expert Alignment for Label-Efficient SAR Scene Understanding

Eli A. Underwood; I. Taylor McKechnie; Leslie M. Collins

cs.CVOct 18, 2026

Preserving the Manifold: Frozen Expert Alignment for Label-Efficient SAR Scene Understanding

Authors: Eli A. Underwood, I. Taylor McKechnie, Leslie M. Collins

Organizations: Duke University

Abstract

Cross-modal alignment of synthetic aperture radar (SAR) imagery to natural language is constrained by two distinct failures. The first is modality mismatch: text encoders pre-trained on optical imagery, such as CLIP, cannot be efficiently fine-tuned on SAR, because active microwave backscatter is physically incompatible with passive optical reflectance and the available SAR caption datasets are orders of magnitude too small to bridge this gap without substantially degrading the learned semantic manifold. The second is physical forgetting: SAR encoders pre-trained on large-scale self-supervised objectives encode radar physics that are progressively erased when the encoder is fine-tuned for text alignment on limited caption data. A single design principle addresses both failures: keep both encoders frozen and learn only a constrained, structure-preserving bridge. Our Frozen Expert framework aligns a frozen CROMA SAR encoder with a frozen CLIP text encoder via a lightweight constrained residual bridge. We formally define physical forgetting via Procrustes disparity and show our bridge preserves SAR geometry significantly better than fine-tuning (disparity: 0.210 vs. 0.351). We validate generality across multiple SAR backbones and text encoders. On zero-shot SAR scene classification, our approach outperforms fine-tuned baselines using up to 8x less training data: DFC2020 (+10.3% R@1, +15.5% mAP), EuroSAT-SAR (+15.7% R@1, +12.2% mAP), BigEarthNet-MM (+14.7% R@1). These results establish frozen manifold alignment as a principled, data-efficient alternative to fine-tuning in high-disparity cross-modal settings.

Figures & tables

Figure 1: Cross-modal sensor comparison. Sentinel-2 multispectral (top) and Sentinel-1 SAR (bottom) across four land-cover types: (a) Urban, (b) Farmland, (c) Grassland, (d) Barren [ 30 ] . The gap between passive optical reflectance and active microwave backscatter is visually apparent across all categories.

Figure 2: Manifold distortion under full co-training.

t

-SNE of the text encoder embedding space for pretrained CLIP (left) vs. fully fine-tuned SAR-CLIP [ 13 ] (right). Joint fine-tuning on SAR data induces geometric warping and inter-class entanglement across the semantic clusters, consistent with impaired open-vocabulary zero-shot generalization.

Figure 3: Overview of our Frozen Expert framework. A constrained structure-preserving bridge $\mathcal{P}_{\phi}$ maps frozen CROMA SAR representations into the frozen CLIP text manifold. The bridge (Eq. 2 ) combines a two-layer MLP with a linear residual path, and its $\ell_{2}$ -normalized output $\mathbf{z}$ is aligned with the caption embedding $\mathbf{t}$ on the shared unit hypersphere via a symmetric InfoNCE loss (Eq. 4 ). Only $\phi$ receives gradient updates; both encoders remain parameter-frozen throughout training.

	Comparison	$\Delta\!\downarrow$
A	CROMA frozen $\leftrightarrow$ Bridge (Ours)	0.210
B	CROMA frozen $\leftrightarrow$ CROMA fine-tuned	0.351
C	CROMA frozen $\leftrightarrow$ Bridge end-to-end	0.388

Table 1: Procrustes disparity validates structure preservation.

\Delta\!\downarrow

measures normalized geometric distortion relative to frozen CROMA representations. Our bridge distorts SAR geometry less than fine-tuning distorts CROMA’s own manifold.

Head Type	$d_{h}$	R@1	R@2	R@3	R@4	R@5	R@6	R@7	R@8	mAP
Linear	N/A	23.14	30.34	64.44	85.75	93.10	96.82	98.22	100.00	15.91
MLP	256	59.92	72.15	82.03	89.98	95.20	96.90	98.66	100.00	33.57
MLP	512	66.30	79.41	88.28	93.56	97.42	98.73	99.75	100.00	36.56
MLP	1024	64.04	75.54	84.75	89.52	96.06	99.24	99.90	100.00	29.61
MLP	2048	44.00	65.19	76.19	81.54	85.23	95.26	98.74	100.00	23.49

Table 2: Effect of projection head capacity (DFC2020). All models use frozen CROMA and frozen CLIP. Bold: best per column.

Visual Enc.	Text Enc.	Strategy	R@1	R@2	R@3	R@4	R@5	R@6	R@7	R@8	mAP
CLIP (Optical)	CLIP	Frozen	34.41	54.60	70.40	82.46	90.30	92.95	96.10	100.00	17.71
CROMA	CLIP	Unfrozen Text	10.35	16.40	22.79	65.79	82.51	89.33	93.10	100.00	15.29
CROMA	CLIP	Unfrozen Image	52.69	54.66	60.08	65.94	74.67	81.11	89.82	100.00	15.27
CROMA	CLIP	Frozen (Ours)	66.30	79.41	88.28	93.56	97.42	98.73	99.75	100.00	36.56

Table 3: Impact of expert selection and freezing strategy (DFC2020). Bold: best per column. Domain-specific SAR expertise and strict encoder freezing are both essential.

Text Encoder	R@1	R@2	R@3	mAP
CLIP (Ours)	66.30	79.41	88.28	36.56
GeoRSCLIP [ 43 ]	52.97	64.92	76.72	31.19
RemoteCLIP [ 18 ]	38.82	65.82	80.09	27.45

Table 4: Text encoder generality (DFC2020). Bridge retrained from scratch for each encoder. RS fine-tuning degrades open-vocabulary transfer.

SAR Backbone	R@1	mAP
CROMA [ 7 ] (Ours)	66.30	36.56
SAR-RS-CLIP enc. [ 10 ]	63.17	21.42
FG-MAE [ 36 ] (SAR-only SSL)	54.06	24.50
JEPA-SAR enc. [ 16 ] (target recog.)	13.84	16.35

Table 5: SAR backbone generality (DFC2020). Bridge retrained from scratch for each backbone; CLIP text encoder frozen throughout. Cross-modal grounding drives performance, not backbone scale.

Dataset	Model	R@1	R@2	R@3	mAP
DFC2020	SAR-RS-CLIP [ 10 ]	55.90%	58.94%	74.35%	17.29%
	SARCLIP [ 13 ]	56.01%	67.37%	87.82%	21.05%
	Ours ( $d_{h}\!=\!512$ )	66.30%	79.41%	88.28%	36.56%
	Ours ( $d_{h}\!=\!1024$ )	64.04%	75.54%	84.75%	29.61%
EuroSAT-SAR	SAR-RS-CLIP [ 10 ]	14.72%	25.77%	38.28%	22.87%
	SARCLIP [ 13 ]	17.79%	32.80%	43.28%	22.05%

Table 6: Zero-shot classification performance. Our framework vs. SAR-RS-CLIP and SARCLIP across three benchmarks. Bold: best per dataset per metric.

Remote Sensing VLMs

See all 41 papers

SARLO-80: Worldwide Slant SAR Language Optic Dataset 80cm
Jun 18, 2026Solène Debuysère, Nicolas Trouvé, Nathan Letheule +2Image-Text Retrieval Cross-Modal Retrieval
1DEMR-ONERA – The French Aerospace Lab, Université Paris-Saclay, Palaiseau, France · 2DTIS-ONERA – The French Aerospace Lab, Université Paris-Saclay, Palaiseau, France · 3Hugging Face, London
TAR: Text Semantic Assisted Cross-modal Image Registration Framework for Optical and SAR Images
May 12, 2026Zhuoyu Cai, Dou Quan, Ning Huyan +3Remote Sensing Synthetic Aperture Radar
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education of China, School of Artificial Intelligence, Xidian University, Xi’an 710071, China
Cross-modal learning for SAR target recognition using optical vision foundation models
Sep 7, 2026Lucas Hirsch, James R. Hopgood, Javid Khan +2Cross-Modal Alignment Cross-Modal Representation Transfer
School of Engineering, University of Edinburgh, United Kingdom · Leonardo UK, United Kingdom · School of Engineering and Physical Sciences, Heriot-Watt University, United Kingdom

Jun 18, 2026J/K move · Enter open · S save

SARLO-80: Worldwide Slant SAR Language Optic Dataset 80cm

Solène Debuysère, Nicolas Trouvé, Nathan Letheule, Elise Colin+1

1DEMR-ONERA – The French Aerospace Lab, Université Paris-Saclay, Palaiseau, France · 2DTIS-ONERA – The French Aerospace Lab, Université Paris-Saclay, Palaiseau, France · 3Hugging Face, London

Image-Text Retrieval Cross-Modal Retrieval Synthetic Aperture Radar Remote Sensing Benchmarks

Open paper AskPDF arXiv

Multimodal foundation models have advanced rapidly thanks to large optical benchmarks, but comparable resources for synthetic aperture radar (SAR) remain limited. Existing SAR--optical datasets largely rely on low-resolution, intensity-only Ground Range Detected~(GRD) products and do not preserve complex-valued SAR measurements or native acquisition geometry, which restricts physically grounded multimodal learning. In particular, large-scale public datasets combining very-high-resolution (VHR) SAR SLC, aligned optical imagery, and natural-language descriptions are still lacking. We present a VHR SAR--optical--text dataset built from open-access Umbra spotlight acquisitions distributed as Sensor Independent Complex Data (SICD). From around 2,500 worldwide scenes (VV/HH, 20cm--2m native resolution), we standardize all SAR data to an 80cm slant-range grid via band-limited FFT resampling and tile the imagery into 1024 by 1024 patches. For each SAR patch, we retrieve a high-resolution optical tile and warp it into the SAR grid using local coordinate correspondences for local pixel-level alignment. We further generate three caption variants (SHORT/MID/LONG) per sample to support vision--language training and evaluation. Our dataset contains 119,566 triplets (complex and amplitude slant-range SAR patch, aligned optical patch, natural-language description) covering 257 locations across 72 countries and a broad range of land types and infrastructures. We release fixed train/validation/test splits and the full preprocessing and baseline code to enable reproducible benchmarks for multimodal alignment on cross-modal retrieval and conditional generation in native SAR geometry. The dataset is publicly available on the Hugging Face Hub at https://huggingface.co/datasets/ONERA/SARLO-80.

Abstract

Remote Sensing VLMs

Explore similar work

SARLO-80: Worldwide Slant SAR Language Optic Dataset 80cm

TAR: Text Semantic Assisted Cross-modal Image Registration Framework for Optical and SAR Images

Cross-modal learning for SAR target recognition using optical vision foundation models