cs.CVNov 10, 2026

Relaxing Cross-Modal Correspondence for Multimodal Remote Sensing Pretraining

Authors: Xianping Ma, Yu Zhang, Mingzhe Jiang, Wanxin Xie, Qing Zhu, Nan Xu, Xiaokang Zhang, Yuanxin Ye

Organizations: Faculty of Geosciences and Engineering, Southwest Jiaotong University, Chengdu 611756, China · School of Architecture and Urban Planning, Shenzhen University, Shenzhen 518060, China · School of Artificial Intelligence, Wuhan University, Wuhan 430072, China

Abstract

Multimodal remote sensing pretraining commonly relies on exact instance-level correspondence across modalities, implicitly treating strictly paired observations as a prerequisite for learning transferable representations. In this work, we revisit this assumption by relaxing cross-modal correspondence from the instance level to the semantic level. We introduce Semantic Pairing (SemPair), a pairing-relaxed pretraining strategy that replaces exact instance pairs with class-consistent cross-instance pairs. To characterize the effect of correspondence relaxation, we formulate a latent-factor model that separates cross-modal dependence into shared semantic structure and instance-specific correlation, showing that SemPair preserves the former while marginalizing instance-specific cross-modal correlations in expectation. We further introduce Mixed Pairing (MixPair) as a controlled interpolation between semantic and exact pairing to study correspondence granularity along a continuous spectrum. Extensive experiments consistently show that SemPair achieves transfer performance comparable to exact pairing (ExaPair), while MixPair reveals a non-monotonic relationship between downstream transfer and the instance-pairing ratio. These results demonstrate that exact instance correspondence is not a prerequisite for effective multimodal remote sensing pretraining and establish cross-modal correspondence granularity as an important design dimension for transferable multimodal representation learning. More broadly, this finding suggests a practical route toward scaling multimodal pretraining with reduced dependence on costly instance-aligned data.

Figures & tables

Explore similar work

CardsList
  1. Any2Any: Unified Arbitrary Modality Translation for Remote Sensing

    Mar 4, 2026Haoyang Chen, Jing Zhang, Hebaixu Wang +7Remote Sensing Image UnderstandingImage-to-Image Translation

  2. GeoMeld: Toward Semantically Grounded Foundation Models for Remote Sensing

    Apr 12, 2026Maram Hasan, Md Aminur Hossain, Savitra Roy +6Multimodal Foundation ModelsMultimodal Grounding

  3. CR-JEPA: Cross-Modal Joint-Embedding Predictive Learning for Remote Sensing Image Retrieval

    May 30, 2026Md Aminur Hossain, Ayush V. Patel, Nitant Dube +1Cross-Modal Representation LearningJoint-Embedding Predictive Architecture