Relaxing Cross-Modal Correspondence for Multimodal Remote Sensing Pretraining
Authors: Xianping Ma, Yu Zhang, Mingzhe Jiang, Wanxin Xie, Qing Zhu, Nan Xu, Xiaokang Zhang, Yuanxin Ye
Organizations: Faculty of Geosciences and Engineering, Southwest Jiaotong University, Chengdu 611756, China · School of Architecture and Urban Planning, Shenzhen University, Shenzhen 518060, China · School of Artificial Intelligence, Wuhan University, Wuhan 430072, China
Multimodal remote sensing pretraining commonly relies on exact instance-level correspondence across modalities, implicitly treating strictly paired observations as a prerequisite for learning transferable representations. In this work, we revisit this assumption by relaxing cross-modal correspondence from the instance level to the semantic level. We introduce Semantic Pairing (SemPair), a pairing-relaxed pretraining strategy that replaces exact instance pairs with class-consistent cross-instance pairs. To characterize the effect of correspondence relaxation, we formulate a latent-factor model that separates cross-modal dependence into shared semantic structure and instance-specific correlation, showing that SemPair preserves the former while marginalizing instance-specific cross-modal correlations in expectation. We further introduce Mixed Pairing (MixPair) as a controlled interpolation between semantic and exact pairing to study correspondence granularity along a continuous spectrum. Extensive experiments consistently show that SemPair achieves transfer performance comparable to exact pairing (ExaPair), while MixPair reveals a non-monotonic relationship between downstream transfer and the instance-pairing ratio. These results demonstrate that exact instance correspondence is not a prerequisite for effective multimodal remote sensing pretraining and establish cross-modal correspondence granularity as an important design dimension for transferable multimodal representation learning. More broadly, this finding suggests a practical route toward scaling multimodal pretraining with reduced dependence on costly instance-aligned data.
Figures & tables
Fig. 1: Motivation for relaxing exact cross-modal correspondence in multimodal pretraining. ExaPair enforces same-instance correspondence, coupling semantic consistency with instance-specific dependence. Different instances from the same semantic class can still share common semantic structure despite variations in appearance and geometry. This motivates SemPair, which preserves semantic consistency while relaxing exact instance identity.
Fig. 2: Overview of pairing-relaxed multimodal pretraining and downstream transfer. Modality- a anchors are paired with modality- b samples according to the correspondence strategy. Only the cross-modal association varies among ExaPair, SemPair, and MixPair, while the multimodal encoder, pretraining objective, optimization budget, and downstream transfer protocol remain fixed.
Fig. 3: ExaPair, SemPair, and MixPair for controlling cross-modal correspondence during pretraining. ExaPair preserves exact instance correspondence. SemPair keeps the modality- a anchor fixed and replaces modality b with a different sample from the same semantic class. MixPair assigns a fraction ρ of training anchors to ExaPair and the remaining fraction 1−ρ to SemPair, with the assignment reshuffled globally at every epoch.
Fig. 4: Representative samples from the two pretraining domains. ImageNet-RS90 provides an object-centric natural-image source, whereas NWPU-RGBD-45 provides overhead remote-sensing scenes. Both are organized as RGB-pseudo-depth inputs for controlled correspondence experiments.
Dataset
Source domain
Classes
Pairs
Train
Val.
Input
ImageNet-RS90
ImageNet subset
90
116300
110483
5817
224×224
NWPU-RGBD-45
Remote-sensing scenes
45
31500
29925
1575
224×224
TABLE I: Summary of the pretraining datasets. Pseudo-depth maps are generated from RGB images using the same monocular depth estimation pipeline.
Dataset
Resolution (m)
Classes
Train
Validation
Test
Tile size
Crop size
Potsdam
0.05
6
18
6
14
6000×6000
512×512
Vaihingen
0.09
6
12
4
17
about 2500×2000
512×512
MMHunan
10
7
300
50
150
256×256
256×256
GeoNRW
1
6
82
13
42
1000×1000
512×512
TABLE II: Summary of the downstream multimodal segmentation benchmarks. GeoNRW uses the Gelsenkirchen subset.
Fig. 5: Representative multimodal samples and semantic annotations from the downstream benchmarks: (a) Potsdam, (b) Vaihingen, (c) MMHunan, and (d) GeoNRW. Each example shows the optical image, the corresponding elevation representation, and the ground-truth semantic mask.
Dataset
Pretraining
Val. loss ↓
Acc@1 ↑
Acc@5 ↑
RGB Acc@1 ↑
Depth Acc@1 ↑
ImageNet-RS90
RGB-only
0.8060
82.50
97.04
82.12
/
ExaPair
0.8398
81.80
96.80
81.78
80.85
MixPair
0.8509
81.08
96.42
81.20
79.84
SemPair
0.9086
79.25
95.65
79.13
78.53
NWPU-RGBD-45
RGB-only
0.5563
90.04
99.18
89.85
/
ExaPair
0.4783
91.88
99.18
91.62
91.37
TABLE III: Pretraining validation results. All metrics in a row are taken from the epoch with the highest fused-branch Acc@1.
Adaptation
Pretraining
OA
mF1
Kappa
Class-wise IoU
mIoU
Imp.
Build.
Low veg.
Tree
Car
Scratch
None
78.74
76.83
71.79
72.78
87.19
54.53
43.58
60.61
63.74
Frozen
RGB-only
84.36
85.39
79.31
76.41
90.31
63.62
64.55
80.49
75.07
ExaPair
85.68
86.58
81.15
79.41
91.08
65.21
65.41
83.53
76.93
MixPair
85.81
86.77
81.27
79.58
90.94
65.68
65.63
84.26
77.22
SemPair
85.72
86.67
81.16
79.72
91.03
64.69
66.06
83.79
77.06
TABLE IV: Primary transfer results on Potsdam. Imp., Build., and Low veg. denote impervious surfaces, buildings, and low vegetation, respectively.
Adaptation
Pretraining
OA
mF1
Kappa
Class-wise IoU
mIoU
Imp.
Build.
Low veg.
Tree
Car
Scratch
None
82.67
72.19
76.69
71.81
81.12
50.02
74.51
21.71
59.83
Frozen
RGB-only
84.03
79.61
78.50
71.22
81.72
52.66
78.90
51.23
67.14
ExaPair
85.80
81.93
80.89
74.35
82.74
58.13
80.33
55.25
70.16
MixPair
86.63
82.88
82.01
75.89
84.67
59.00
80.96
57.12
71.53
SemPair
86.20
82.22
81.42
74.83
83.49
59.40
80.80
54.51
70.61
TABLE V: Primary transfer results on Vaihingen. Imp., Build., and Low veg. denote impervious surfaces, buildings, and low vegetation, respectively.
Adaptation
Pretraining
OA
mF1
Kappa
Class-wise IoU
mIoU
Crop.
Forest
Grass.
Wet.
Water
Unused
Built-up
Scratch
None
68.04
43.90
55.01
45.69
68.35
10.68
0.00
60.70
22.33
19.07
32.40
Frozen
RGB-only
72.96
50.72
61.72
47.79
71.91
13.24
0.32
65.36
51.77
21.52
38.84
ExaPair
74.14
53.46
63.54
51.48
71.97
14.06
4.91
70.54
51.65
23.24
41.12
MixPair
74.64
53.13
64.27
52.04
72.47
13.33
4.62
71.03
52.54
21.30
41.05
SemPair
74.36
52.95
63.80
51.46
72.19
13.38
3.60
71.00
53.14
21.84
40.94
TABLE VI: Primary transfer results on MMHunan. Crop., Grass., Wet., and Unused denote cropland, grassland, wetland, and unused land, respectively.
Adaptation
Pretraining
OA
mF1
Kappa
Class-wise IoU
mIoU
Forest
Agric.
Built-up
Grass.
Road
Building
Scratch
None
67.39
54.01
52.15
31.07
70.26
60.89
12.21
14.92
54.72
40.68
Frozen
RGB-only
69.15
57.94
54.10
38.25
74.65
62.69
19.41
18.00
50.10
43.85
ExaPair
70.68
61.22
57.44
39.04
76.52
63.96
20.62
26.02
55.01
46.86
MixPair
71.69
61.80
58.76
39.86
77.72
65.05
20.99
25.54
56.40
47.60
SemPair
71.42
62.14
58.41
40.51
76.72
64.49
21.71
25.91
57.59
47.82
TABLE VII: Primary transfer results on GeoNRW. Agric. and Grass. denote agricultural areas and grassland, respectively.
Fig. 6: Qualitative segmentation results on test set from Potsdam ( 2048×2048 ) and GeoNRW ( 512×512 ). Each row shows the optical image, DSM, ground truth, and predictions from the end-to-end trained Scratch baseline and four pretrained models under frozen transfer.
ρ
Strategy
Potsdam
Vaihingen
Frozen
Fine-tuning
Frozen
Fine-tuning
mF1
mIoU
mF1
mIoU
mF1
mIoU
mF1
mIoU
0
SemPair
86.67
77.06
89.03
80.64
82.22
70.61
84.49
73.94
0.25
MixPair
86.68
77.10
88.98
80.59
82.32
70.76
84.77
74.40
0.50
MixPair
86.77
77.22
89.05
80.68
82.88
71.53
84.95
74.52
0.75
MixPair
86.58
76.94
88.98
80.58
81.75
69.92
84.09
73.40
TABLE VIII: Effect of continuous correspondence granularity on downstream mF1 and mIoU (%).
Fig. 7: Effect of the instance-pairing ratio ρ on downstream mIoU for Potsdam and Vaihingen under frozen transfer and end-to-end fine-tuning.
Pretraining domain
Strategy
Inst. R@1
Excl-inst. Sem. R@1
Excl-inst. Sem. R@5
ImageNet-RS90
ExaPair
97.89
74.20
90.41
MixPair
97.09
72.98
89.77
SemPair
97.59
72.80
88.95
NWPU-RGBD-45
ExaPair
97.40
86.86
96.06
MixPair
96.44
87.75
95.81
SemPair
96.13
85.46
94.79
TABLE IX: Representation-level RGB-to-depth retrieval on the fixed pretraining validation split. Inst. R@1 uses the full depth gallery. Excl-inst. Sem. R@1 and R@5 exclude the same-instance depth sample from the gallery. All metrics are reported in percentages.
Fig. 8: Qualitative cross-branch RGB-to-depth retrieval on the fixed validation splits of (a) ImageNet-RS90 and (b) NWPU-RGBD-45. Each example shows the RGB query and its same-instance depth reference once, followed by the top-three retrieved depth samples for ExaPair, MixPair, and SemPair. The reference depth is excluded from the retrieval gallery.
Pretraining domain
Strategy
Potsdam
Vaihingen
MMHunan
GeoNRW
mF1
mIoU
mF1
mIoU
mF1
mIoU
mF1
mIoU
ImageNet-RS90
RGB-only
88.84
80.37
83.48
72.49
53.70
41.39
63.34
49.01
ExaPair
88.97
80.57
84.52
74.13
54.28
41.74
63.67
49.48
MixPair
89.05
80.68
84.95
74.52
55.46
42.96
63.88
49.80
SemPair
89.03
80.64
84.49
73.94
54.69
42.41
64.43
50.06
NWPU-RGBD-45
RGB-only
86.20
76.36
79.09
67.52
49.51
38.15
59.08
45.07
TABLE X: Generalization across pretraining domains measured by downstream mF1 and mIoU (%).
Initialization
Potsdam
Vaihingen
P1
P3
P6
Full
V3
V6
Full
mF1
mIoU
mF1
mIoU
mF1
mIoU
mF1
mIoU
mF1
mIoU
mF1
mIoU
mF1
mIoU
Scratch
70.91
56.85
72.83
58.82
75.60
62.03
76.83
63.74
67.84
54.85
71.23
58.48
72.19
59.83
RGB-only
82.89
71.33
85.07
74.46
86.52
76.69
88.84
80.37
79.44
67.04
82.67
71.51
83.48
72.49
ExaPair
83.42
72.15
85.54
75.24
86.86
77.24
88.97
80.57
79.89
67.67
82.83
71.53
84.52
74.13
MixPair
83.66
72.53
85.32
74.90
87.19
77.75
89.05
80.68
80.29
68.12
82.96
71.80
84.95
74.52
TABLE XI: Label-efficient transfer measured by mF1 and mIoU (%). P1, P3, and P6 denote 1, 3, and 6 labeled Potsdam training tiles. V3 and V6 denote 3 and 6 labeled Vaihingen training tiles. Full denotes the complete training split.
Fig. 9: Label-efficient transfer on Potsdam and Vaihingen.
Dataset
Init.
Frozen
Fine-tuning
FT( 10−2 )
mF1
mIoU
mF1
mIoU
mF1
mIoU
Potsdam
Scratch
76.83
63.74
76.83
63.74
76.83
63.74
ExaPair
86.58
76.93
88.97
80.57
89.65
81.61
SemPair
86.67
77.06
89.03
80.64
89.56
81.49
Vaihingen
Scratch
72.19
59.83
72.19
59.83
72.19
59.83
ExaPair
81.93
70.16
84.52
74.13
86.01
76.29
TABLE XII: Effect of downstream adaptation strength on mF1 and mIoU (%). The same Scratch baseline is used across all three adaptation settings, with the entire network trained from random initialization at an initial learning rate of 10−2 .
Architecture
Params.
Potsdam
Vaihingen
ExaPair
MixPair
SemPair
ExaPair
MixPair
SemPair
mF1
mIoU
mF1
mIoU
mF1
mIoU
mF1
mIoU
mF1
mIoU
mF1
mIoU
DFormer-Tiny
5.64M
88.91
80.47
89.09
80.77
89.06
80.71
83.99
73.44
83.60
72.66
83.97
73.20
DFormer-Small
19.05M
88.97
80.57
89.05
80.68
89.03
80.64
84.52
74.13
84.95
74.52
84.49
73.94
DFormer-Base
29.95M
89.25
81.02
89.16
80.86
89.30
81.08
84.72
74.26
85.42
75.34
85.37
75.26
TABLE XIII: Generalization across model scales measured by mF1 and mIoU (%). The best result for each metric within each dataset and model scale is highlighted in bold.
Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University, and Hubei Key Laboratory of Multimedia and Network Communication Engineering · Zhongguancun Academy, Beijing, China. 100094 · School of Electronic Information, Wuhan University, Wuhan, China +2
Space Applications Centre, Indian Space Research Organisation, Ahmedabad, India · Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay, Mumbai, India