CroCoMAE: Cross-modal self-supervised collaborative representation learning for SAR-optical remote sensing data
Organizations: Department of Computer Science, University of Bari Aldo Moro, Bari, Italy · INRIA, EVERGREEN, Univ. of Montpellier, Montpellier, France · Consorzio Interuniversitario Nazionale per l’Informatica (CINI), Bari, Italy
Abstract
Remote sensing increasingly captures the Earth's surface through complementary sensors such as SAR and optical, yet ground-truth annotations remain scarce, limiting supervised approaches. Self-supervised learning (SSL) addresses the label scarcity, and cross-modal SSL further exploits the complementarity between co-located sensors, but existing cross-modal frameworks align modalities mostly through global image-level representations or treat them as largely redundant, leaving their fine-grained complementary structure underexploited. We propose CroCoMAE, a cross-modal SSL framework for SAR-optical remote sensing that unifies contrastive learning and masked image modeling within a ViT-based architecture, operating at two complementary levels. At the image level, an intra-modality contrastive loss on the class token yields globally discriminative, augmentation-invariant representations. At the patch level, masked image modeling drives fine-grained per-modality representations, while a cross-modal contrastive objective aligns a shared subspace across SAR and optical patch tokens, leaving the complementary subspace unconstrained to retain modality-specific information. On two SAR-optical benchmarks covering scene classification and semantic segmentation, CroCoMAE improves over uni-modal SSL baselines and outperforms state-of-the-art cross-modal frameworks in most configurations, with gains in both single-modality and multi-modal evaluation across varying label regimes and the largest improvements on semantic segmentation.
Figures & tables
| 1% | 10% | |||||
|---|---|---|---|---|---|---|
| SAR | Opt | MM | SAR | Opt | MM | |
| CLS | 77.1 | 96.5 | 96.4 | 82.7 | 97.8 | 97.8 |
| Patch | 81.8 | 95.3 | 94.3 | 84.6 | 97.0 | 97.1 |
| Patch(S) | 80.6 | 80.3 | 81.5 | 81.9 | 81.9 | 83.0 |
| Patch(U) | 78.8 | 92.1 | 95.1 | 84.3 | 96.6 | 97.2 |
| CLS+Patch | 80.5 | 96.3 | 96.1 | 84.6 | 97.6 | 97.7 |
Cross-Modal Representation Learning
Cross-modal representation learning connects information across data types such as images, text, audio, biosignals, and sensor measurements.
See all 221 papers