As a pivotal task that bridges remote visual and linguistic understanding, Remote Sensing Image-Text Retrieval (RSITR) has attracted considerable research interest in recent years. However, almost all RSITR methods implicitly assume that image-text pairs are matched perfectly. In practice, acquiring a large set of well-aligned data pairs is often prohibitively expensive or even infeasible. In addition, we also notice that the remote sensing datasets (e.g., RSITMD) truly contain some inaccurate or mismatched image text descriptions. Based on the above observations, we reveal an important but untouched problem in RSITR, i.e., Noisy Correspondence (NC). To overcome these challenges, we propose a novel Robust Remote Sensing Image-Text Retrieval (RRSITR) paradigm that designs a self-paced learning strategy to mimic human cognitive learning patterns, thereby learning from easy to hard from multi-modal data with NC. Specifically, we first divide all training sample pairs into three categories based on the loss magnitude of each pair, i.e., clean sample pairs, ambiguous sample pairs, and noisy sample pairs. Then, we respectively estimate the reliability of each training pair by assigning a weight to each pair based on the values of the loss. Further, we respectively design a new multi-modal self-paced function to dynamically regulate the training sequence and weights of the samples, thus establishing a progressive learning process. Finally, for noisy sample pairs, we present a robust triplet loss to dynamically adjust the soft margin based on semantic similarity, thereby enhancing the robustness against noise. Extensive experiments on three popular benchmark datasets demonstrate that the proposed RRSITR significantly outperforms the state-of-the-art methods, especially in high noise rates. The code is available at: https://github.com/MSFLabX/RRSITR
Figures & tables
Figure 1: Representative examples of noise correspondence problems that really exist in the RSITMD dataset, wherein the sample pair is erroneously assumed to be semantically aligned. Since the model lacks explicit knowledge of which sample pairs are noisy during training, such incorrect supervision signals inevitably weaken retrieval performance. (a) and (b) represent clean sample pairs and noisy mismatched ones, respectively.
Figure 2: The overview of RRSITR. Specifically, we first achieve fine-grained alignment by both global and local contrastive learning. Based on contrastive loss magnitude, training pairs are categorized into clean, ambiguous, and noisy. A multimodal self-paced learning strategy then dynamically weights clean and ambiguous pairs, enabling progressive learning from easy to hard correspondences. For noisy pairs, a robust triplet loss enhances model robustness and retrieval performance.
Noise
Method
Ref.
Image-to-Text Retrieval
Text-to-Image Retrieval
mR
R@1
R@5
R@10
R@1
R@5
R@10
20%
AMFMN
TGRS’22
9.56±1.17
26.90±2.02
39.29±1.96
7.66±0.78
31.19±0.49
51.42±0.94
27.67±0.93
HVSA
TGRS’23
7.87±0.99
23.54±1.52
34.87±2.72
6.35±0.29
27.57±0.71
47.46±1.03
24.61±0.74
KAMCL
TGRS’23
10.80±0.89
27.17±1.08
39.87±1.32
9.59±0.61
34.04±0.65
51.23±1.15
28.78±0.76
SWAN
ICMR’23
9.29±2.28
24.25±1.38
37.43±2.55
7.44±0.45
30.93±0.84
52.71±0.92
27.01±0.81
PIR
ACMMM’23
10.09±1.66
29.25±1.69
44.95±1.19
8.74±0.89
33.45±1.25
54.04±1.06
30.09±0.89
Table 1: Image-text retrieval performance under different noise ratios on the RSITMD dataset.
Noise
Method
Ref.
Image-to-Text Retrieval
Text-to-Image Retrieval
mR
R@1
R@5
R@10
R@1
R@5
R@10
20%
AMFMN
TGRS’22
4.68±0.33
15.08±0.48
24.99±1.12
4.08±0.10
16.31±0.64
27.65±0.83
15.47±0.34
HVSA
TGRS’23
4.76±0.40
14.93±0.69
25.21±1.33
2.85±0.29
13.15±0.67
24.85±1.00
14.29±0.49
KAMCL
TGRS’23
7.59±0.79
20.48±0.57
31.31±0.41
5.75±0.23
21.38±0.46
34.70±0.66
20.20±0.28
SWAN
ICMR’23
4.50±0.62
13.76±0.82
22.71±1.04
3.45±0.21
15.43±0.85
28.54±0.69
14.73±0.25
PIR
ACMMM’23
7.36±0.38
20.29±0.64
31.64±1.46
5.41±0.32
20.68±0.32
34.84±0.85
20.04±0.50
Table 2: Image-text retrieval performance under different noise ratios on the RSICD dataset.
Noise
Method
Ref.
Image-to-Text Retrieval
Text-to-Image Retrieval
mR
R@1
R@5
R@10
R@1
R@5
R@10
20%
AMFMN
TGRS’22
5.47±0.24
21.99±0.79
36.89±0.73
4.47±0.24
18.66±0.23
31.97±0.32
19.91±0.38
HVSA
TGRS’23
2.80±0.34
10.97±0.34
19.92±0.50
1.68±0.07
7.96±0.20
15.50±0.17
9.81±0.18
KAMCL
TGRS’23
14.01±0.47
43.13±0.61
60.81±0.67
9.12±0.15
30.40±0.29
46.23±0.36
33.95±0.34
SWAN
ICMR’23
2.42±0.20
11.43±0.59
20.98±0.41
2.72±0.12
11.85±0.23
21.69±0.49
11.85±0.20
PIR
ACMMM’23
10.61±0.30
37.13±0.88
56.07±0.47
7.55±0.20
27.57±0.24
43.30±0.32
30.37±0.30
Table 3: Image-text retrieval performance under different noise ratios on the NWPU dataset.
Method
Image-to-Text Retrieval
Text-to-Image Retrieval
mR
R@1
R@5
R@10
R@1
R@5
R@10
#1
15.44
31.64
45.49
11.99
38.13
57.57
33.38
#2
12.08
27.79
39.42
10.29
35.47
53.73
29.80
#3
13.94
29.20
40.66
11.96
34.84
50.64
30.21
#4
13.76
30.22
43.36
11.88
37.50
56.72
32.24
#5
0.18
0.97
1.64
0.36
1.42
2.76
1.22
Table 4: Ablation studies on RSITMD with 80% noise ratio.
Figure 3: Density versus weight of all sample pairs on RSITMD during training at the 1st epoch (left) and the 50th epoch (right).
Figure 4: Parameters analysis (i.e., λ1 and λ2 ) on RSITMD.
Remote sensing (RS) image-text retrieval plays a critical role in understanding massive RS imagery. However, the dense multi-object distribution and complex backgrounds in RS imagery make it difficult to simultaneously achieve fine-grained cross-modal alignment and efficient retrieval. Existing methods either rely on complex cross-modal interactions that lead to low retrieval efficiency, or depend on large-scale vision-language adaptation, which requires massive data and computational resources. To address these issues, we develop a fast-then-fine (FTF) retrieval framework centered on interaction-free multi-granular recall and parameter-free text-guided cross-modal reranking with the Balanced Text-guided Interaction Block (BTIB). Specifically, interaction-free multi-granular recall pre-computes gallery embeddings and matches them with the corresponding query embedding without full-gallery token-level text-image interaction, while text-guided cross-modal reranking with BTIB refines local semantic correspondence within the top-K candidate set. Furthermore, an inter- and intra-modal loss is designed to jointly optimize cross-modal alignment across multi-granular representations. Extensive experiments on public benchmarks demonstrate that the FTF achieves competitive retrieval accuracy while significantly improving retrieval efficiency compared with existing methods.
Xi Chen, Xu Chen, Xiamgyang Jia +3
School of Computer Science, Wuhan University, China · Beijing Institute for General Artificial Intelligence (BIGAI), China
Remote sensing (RS) image-text retrieval plays a critical role in understanding massive RS imagery. However, the dense multi-object distribution and complex backgrounds in RS imagery make it difficult to simultaneously achieve fine-grained cross-modal alignment and efficient retrieval. Existing methods either rely on complex cross-modal interactions that lead to low retrieval efficiency, or depend on large-scale vision-language model pre-training, which requires massive data and computational resources. To address these issues, we propose a fast-then-fine (FTF) two-stage retrieval framework that decomposes retrieval into a text-agnostic recall stage for efficient candidate selection and a text-guided rerank stage for fine-grained alignment. Specifically, in the recall stage, text-agnostic coarse-grained representations are employed for efficient candidate selection; in the rerank stage, a parameter-free balanced text-guided interaction block enhances fine-grained alignment without introducing additional learnable parameters. Furthermore, an inter- and intra-modal loss is designed to jointly optimize cross-modal alignment across multi-granular representations. Extensive experiments on public benchmarks demonstrate that the FTF achieves competitive retrieval accuracy while significantly improving retrieval efficiency compared with existing methods.
Xi Chen, Xu Chen, Xiangyang Jia +3
School of Computer Science, Wuhan University, Wuhan, China · Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China
Existing language-image pre-training for remote sensing object detection is constrained by Monolithic Label Learning, which relies on exhaustively enumerating open-set categories via black-box data to acquire fine-grained representations, creating a dependency incompatible with the domain's inherent data scarcity. To transcend this bottleneck, we propose SLIP-RS, establishing a Structured-Attribute Decoupling Paradigm that maps the open-ended category space into a finite, physically meaningful attribute space, unlocking fine-grained discriminability via explicit structural logic. This paradigm is realized via two technical pillars: (1) Structured-Attribute Contrastive Learning, which enforces the learning of decoupled intrinsic visual logic via combinatorial attribute augmentation; and (2) Conformal Attribute Reliability Engine, which leverages conformal prediction theory to rigorously distill high-fidelity supervision from noisy sources, yielding RS-Attribute-15M, the largest dataset with over 15 million attribute annotations. Extensive experiments demonstrate that SLIP-RS establishes unprecedented performance in fine-grained detection and cross-domain generalization, validating structured attributes as a vital foundation for remote sensing. Code: https://github.com/facias914/SLIP-RS.
Chenxu Wang, Yuxuan Li, Yunheng Li +3
1VCIP, CS, Nankai University · 2NKIARI, Shenzhen Futian · 3National University of Defense Technology