MarsRetrieval: Benchmarking Vision-Language Models for Planetary-Scale Geospatial Retrieval on Mars
Authors: Shuoyuan Wang, Yiran Wang, Hongxin Wei
Organizations: Department of Statistics and Data Science, Southern University of Science and Technology, Shenzhen, China · Department of Earth and Space Sciences, Southern University of Science and Technology, Shenzhen, China
Data-driven approaches like deep learning are rapidly advancing planetary science, particularly in Mars exploration. Despite recent progress, most existing benchmarks remain confined to closed-set supervised visual tasks and do not support text-guided retrieval for geospatial discovery. We introduce MarsRetrieval, a retrieval benchmark for evaluating vision-language models for Martian geospatial discovery. MarsRetrieval includes three tasks: (1) paired image-text retrieval, (2) landform retrieval, and (3) global geo-localization, covering multiple spatial scales and diverse geomorphic origins. We propose a unified retrieval-centric protocol to benchmark multimodal embedding architectures, including contrastive dual-tower encoders and generative vision-language models. Our evaluation shows MarsRetrieval is challenging: even strong foundation models often fail to capture domain-specific geomorphic distinctions. We further show that domain-specific fine-tuning is critical for generalizable geospatial discovery in planetary settings. Our code is available at https://github.com/ml-stat-Sustech/MarsRetrieval
Figures & tables
Figure 1: An overview of tasks in MarsRetrieval. The benchmark consists of 3 challenging tasks for cross-modal retrieval in Martian exploration. See Table 1 for data sources and retrieval directions.
Task
Data Sources
Modalities
Paired Image-Text Retrieval
Public + Curated
T → I; I → T
Landform Retrieval
CTX, HiRISE
T → I
Global Geo-Localization
CTX
T → I; I → I
Table 1: Data sources and retrieval directions of the three tasks in MarsRetrieval. We denote the modalities with I =image, T =text.
Figure 2: Planetary scale coverage in task 1 (Paired Image-Text Retrieval). We include Martian paired vision-language samples from orbital terrain to rover instrumentation.
Figure 3: Dataset distribution across the 7 major genetic classes of Martian landforms in Task 2 (Landform Retrieval). We provided the detailed sample distribution in Figure 7 .
Figure 4: Global ground-truth distribution from scientific catalogs [ 41 , 58 , 53 , 40 , 34 ] for Task 3. Different colors denote five target landforms, showing their distributions across Mars.
Size
Model
Backbone
T → I
I → T
R@1
R@10
MRR
MedR
R@1
R@10
MRR
MedR
Encoder-based Models
0.4B
DFN2B-CLIP-ViT-L-14
ViT-L/14
17.32
42.50
25.67
17
20.94
47.57
29.87
12
ViT-L-16-SigLIP-384
ViT-L/16
21.60
42.94
29.20
18
21.73
45.34
29.78
15
aimv2-large-patch14-224
ViT-L/14
10.06
29.43
16.83
43
11.76
33.89
19.15
34
ViT-L-16-SigLIP2-512
ViT-L/16
25.89
50.33
34.06
10
29.38
55.27
37.95
7
Table 2: Retrieval results of Task 1 (Paired Image-Text Retrieval) using Seed2.0-refined captions. We report bidirectional retrieval performance on various vision-language models, grouped by model scale. Best result per column within each model group is in bold. Bootstrap 95% confidence intervals are reported in Appendix E.2 .
Model
mAP
nDCG@10
Hits@10
Encoder-based Models
DFN2B-CLIP-ViT-L-14
9.48
12.39
47.92
ViT-L-16-SigLIP-384
8.59
9.46
37.50
aimv2-large-patch14-224
6.12
7.07
43.75
ViT-L-16-SigLIP2-512
9.59
10.84
60.42
PE-Core-L-14-336
20.93
25.80
72.92
Table 3: Retrieval results of Landform Retrieval (task 2). MLLM-based models are reported at the largest available scale of each model family. Best results within each model group are in bold.
Model
All
w/o MarScope
w/ MarScope
SigLIP2
10.84
12.34
10.49
PE-Core
25.80
24.86
26.02
MarScope
74.42
76.43
73.97
WeMM-9B
17.60
10.51
19.23
Ovis-VL-9B
21.39
12.04
23.61
Table 4: Dataset bias analysis on Task 2. We report nDCG@10 for all 48 classes and subsets collected with and without MarScope. MarScope outperforms general models on both subsets.
Model
Alluvial Fans
Glacier-Like Forms
Landslides
Pitted Cones
Yardangs
Text
Image
Text
Image
Text
Image
Text
Image
Text
Image
Encoder-based VLMs
DFN2B-CLIP-ViT-L-14
1.47
2.32
1.96
17.90
29.27
4.76
7.29
16.71
0.24
28.86
ViT-L-16-SigLIP-384
6.82
1.73
2.29
11.70
23.54
1.98
5.25
14.06
1.65
35.34
aimv2-large-patch14-224
1.54
1.11
2.27
11.65
26.68
2.04
1.70
12.26
1.26
21.86
ViT-L-16-SigLIP2-512
1.55
2.94
6.33
21.97
25.83
2.48
3.68
14.87
1.51
27.79
Table 5: Retrieval results of Global Geo-Localization (Task 3) on various foundation models. We report AUPRC for both text-based and image-based retrieval across five Martian landforms. Best results within the encoder-based and MLLM-based groups are in bold.
Figure 5: Global distribution of Pitted Cones in Task 3. The distribution retrieved by text-based query shows a more similar result to the ground truth catalog.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11Figure 12Figure 13
Stage
Input
Retained
Caption deduplication
86,182
61,868
Score-based filtering
61,868
16,327
Text K-center sampling
16,327
5,000
Image K-center sampling
5,000
2,500
Two-stage expert validation
2,500
2,287
Appendix
Table 6: Number of image-text pairs at each stage of Task 1.
Condition
Mars Relevant
Image-Text Consistent
Eligible
Distinctive
Overall Accept (95% CI)
No score
45.67
82.00
42.67
69.33
25.87 [23.71, 28.14]
CLIP only
45.60
82.33
42.33
72.47
33.87 [31.78, 36.09]
LLM only
48.67
78.13
45.87
69.47
35.67 [33.52, 37.94]
MLLM only
52.27
76.07
53.67
72.13
39.53 [37.28, 41.89]
All three
61.33
79.93
59.20
74.73
46.47 [44.07, 48.93]
Appendix
Table 7: Acceptance rates (%) of 5,000-pair subsets filtered with no score, each score alone and all three scores, judged by Seed2.0-Lite. The overall acceptance is pooled over three sampling seeds, with the Wilson 95% confidence interval.
Setting
CLIP / LLM / MLLM
Filtered Pool
Audit Accept, % (95% CI)
Rank Corr.
Lenient
0.30 / 0.78 / 0.78
39,061
42.4 [38.4, 46.6]
0.8
Default
0.32 / 0.82 / 0.82
16,327
46.0 [41.9, 50.3]
1.0
Strict
0.34 / 0.86 / 0.86
7,957
61.0 [56.8, 65.4]
1.0
Appendix
Table 8: Sensitivity to the score-based filtering thresholds. Audit acceptance is judged by Seed2.0-Lite, and rank correlation is the Spearman correlation of the T → I MRR ranking of four models with the default setting.
Setting
Qwen3-VL
SigLIP2
PE-Core
Lenient
40.06 [38.57, 41.55]
45.63 [44.22, 47.05]
51.43 [50.08, 52.80]
Default
36.98 [35.50, 38.44]
41.66 [40.23, 43.09]
47.83 [46.44, 49.24]
Strict
25.53 [24.11, 26.96]
27.91 [26.49, 29.33]
31.95 [30.53, 33.37]
Appendix
Table 9: T → I MRR (%) with 95% confidence intervals on fixed 3,500-pair subsets constructed under each threshold setting (10,000 paired query-bootstrap replicates).
Figure 6: Image-based query examples for Task 3 (Global Geo-Localization)
Figure 7: Sample distribution of the 48 long-tailed geomorphic categories utilized in Landform Retrieval (Task 2).
Figure 8: Ablation on caption refinement in Paired Image–Text Retrieval (task 1). The results on MRR indicate high-quality captioning leads to stronger vision–language alignment.
Model
Backbone
Size
T → I
I → T
R@1
R@10
MRR
MedR
R@1
R@10
MRR
MedR
Encoder-based Models
DFN2B-CLIP-ViT-L-14
ViT-L/14
0.4B
8.26
22.47
12.92
116
8.05
20.73
12.58
130
ViT-L-16-SigLIP-384
ViT-L/16
0.4B
11.28
22.43
15.27
167
9.01
20.24
13.02
176
aimv2-large-patch14-224
ViT-L/14
0.4B
4.68
14.43
8.11
233
4.77
14.12
8.09
241
ViT-L-16-SigLIP2-512
ViT-L/16
0.4B
13.16
27.72
18.21
89
12.42
26.98
17.34
87
Appendix
Table 10: Performance comparison of different models on Paired Image-Text Retrieval (Task 1) with raw (unrefined) captions. Notably, performance across all models is significantly lower compared to the results with refined captions (see Table 2 ). Bold indicates the best result within each model group.
Model
1
3
5
7
10
MarScope
68.74
71.89
71.87
72.44
72.32
PE-Core-L-14-336
20.67
20.93
20.06
19.68
19.84
Qwen3-VL-Embedding
8.43
8.69
7.63
7.73
7.51
Ops-MM-embedding-v1
9.52
9.62
7.28
8.35
8.50
Appendix
Table 11: Ablation on different prompt templates in Landform Retrieval (task 2). We use 3 templates throughout the paper.
Category Group
#Classes
DFN2B-CLIP
MarScope
Gain
Earth-shared
22
15.85
77.25
+61.39
Planetary-specific
26
9.47
72.05
+62.59
Appendix
Table 12: Task 2 nDCG@10 of DFN2B-CLIP-ViT-L-14 and MarScope on Earth-shared and planetary-specific categories.
Figure 9: Ablation on the spatial tolerance radius r in Global Geo-Localization (task 3). We use r=0.5 throughout the paper.
Model
Alluvial Fans
Glacier-Like Forms
Landslides
Pitted Cones
Yardangs
Text
Image
Text
Image
Text
Image
Text
Image
Text
Image
Encoder-based Models
DFN2B-CLIP-ViT-L-14
6.19
7.94
7.16
34.98
47.90
13.69
19.38
36.52
2.92
47.11
ViT-L-16-SigLIP-384
16.26
6.71
7.69
24.89
38.73
8.95
17.12
29.13
7.97
51.75
aimv2-large-patch14-224
6.83
4.67
8.02
24.76
42.63
8.84
9.51
28.80
6.41
38.01
ViT-L-16-SigLIP2-512
7.13
9.38
15.32
39.40
42.96
9.82
15.47
32.32
8.01
46.21
Appendix
Table 13: Performance comparison of Global Geo-Localization (Task 3) measured by the optimal F1-score ( F1@K∗ ). The domain-specialized MarScope outperforms general foundation models on most landform categories. This gap highlights the necessity of explicit fine-tuning with planetary knowledge for precise large-scale scientific discovery. Best results within the encoder-based and MLLM-based groups are in bold.
Figure 10: Sensitivity of ( F1@K ) to retrieval depth K in task 3. Text-based queries exhibit better peak performance and stability.
Figure 11: The visualization of planetary-scale geomorphic distributions generated by MarScope (Task 3). MarScope’s text-based retrieval produces distributions that are more consistent with global catalogues compared to image-based retrieval.
Task 1 (MRR)
Task 2
Task 3 (AUPRC)
Backbone
Setting
T → I
I → T
mAP
Text
Image
DFN2B-CLIP-ViT-L-14
Zero-shot
25.67
29.87
9.48
8.05
14.11
Fine-tuned (MarScope)
47.70
46.57
71.89
42.17
31.91
ViT-L-16-SigLIP2-512
Zero-shot
34.06
37.95
9.59
7.78
14.01
Fine-tuned
44.37
42.83
68.13
41.85
34.15
Qwen3-VL-Embedding-2B
Zero-shot
29.85
29.03
8.68
3.64
12.17
Appendix
Table 14: Zero-shot and domain-specific fine-tuned results of three backbones. Task 1 reports MRR with Seed2.0 captions, Task 2 reports mAP, and Task 3 reports AUPRC averaged over the five landform categories. Best fine-tuned results are in bold.
T → I
I → T
Model
MRR
95% CI
MRR
95% CI
Encoder-based Models
DFN2B-CLIP-ViT-L-14
25.67
[24.19, 27.15]
29.87
[28.29, 31.46]
ViT-L-16-SigLIP-384
29.20
[27.58, 30.81]
29.78
[28.20, 31.41]
aimv2-large-patch14-224
16.83
[15.60, 18.08]
19.15
[17.86, 20.49]
ViT-L-16-SigLIP2-512
34.06
[32.40, 35.77]
37.95
[36.25, 39.67]
Appendix
Table 15: Task 1 MRR with paired bootstrap 95% confidence intervals (10,000 resamples of the 2,287 image-text pairs, Seed2.0 captions).
Task 1 (MRR)
Task 2
Task 3 (AUPRC)
Seed
T → I
I → T
mAP
nDCG@10
Text
Image
7
35.91
34.18
67.52
70.57
43.66
30.90
8
44.76
37.09
70.52
74.45
42.42
29.32
9
37.04
31.37
70.89
74.31
40.02
32.09
Mean ± std
39.24 ± 4.82
34.21 ± 2.86
69.64 ± 1.85
73.11 ± 2.20
42.04 ± 1.85
30.77 ± 1.39
Appendix
Table 16: MarScope trained with three random seeds. Task 1 reports MRR with Seed2.0 captions, Task 2 reports mAP and nDCG@10, and Task 3 reports AUPRC averaged over the five landform categories. The last row reports mean ± sample standard deviation.
Task 1 (MRR)
Task 2
Task 3 (AUPRC)
Model
Adapted On
T → I
I → T
nDCG@10
Text
Image
OpenAI CLIP-ViT-L-14
Generic
15.32
20.19
9.98
0.82
13.35
+ RemoteCLIP
Earth observation
8.55
17.51
3.42
0.57
12.37
+ SkyCLIP
Earth observation
16.52
20.92
11.90
1.95
11.47
LongCLIP-ViT-L-14
Generic
21.93
21.88
9.14
5.15
16.69
+ DGTRS-CLIP
Earth observation
15.44
19.82
10.48
3.75
15.83
Appendix
Table 17: Earth-observation CLIP models and their pre-trained backbones. Task 1 reports MRR with Seed2.0 captions, Task 2 reports nDCG@10, and Task 3 reports AUPRC averaged over the five landform categories.
Task 1 (MRR)
Task 2
Backbone
Resolution
T → I
I → T
nDCG@10
PE-Core-L-14
224
37.02
37.55
26.94
336
37.50
38.76
28.94
448
37.50
38.50
28.48
PE-Core-G-14
224
20.56
19.62
7.61
336
42.14
40.05
32.74
Appendix
Table 18: PE-Core-L-14 and PE-Core-G-14 evaluated at the same input resolutions. Task 1 reports MRR with Seed2.0 captions and Task 2 reports nDCG@10. Native resolutions are underlined.
Department of Electrical and Computer Engineering The Grainger College of Engineering University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA · Department of Computer Science University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA