MarsRetrieval: Benchmarking Vision-Language Models for Planetary-Scale Geospatial Retrieval on Mars
Authors: Shuoyuan Wang, Yiran Wang, Hongxin Wei
Organizations: Department of Statistics and Data Science, Southern University of Science and Technology, Shenzhen, China · Department of Earth and Space Sciences, Southern University of Science and Technology, Shenzhen, China
Data-driven approaches like deep learning are rapidly advancing planetary science, particularly in Mars exploration. Despite recent progress, most existing benchmarks remain confined to closed-set supervised visual tasks and do not support text-guided retrieval for geospatial discovery. We introduce MarsRetrieval, a retrieval benchmark for evaluating vision-language models for Martian geospatial discovery. MarsRetrieval includes three tasks: (1) paired image-text retrieval, (2) landform retrieval, and (3) global geo-localization, covering multiple spatial scales and diverse geomorphic origins. We propose a unified retrieval-centric protocol to benchmark multimodal embedding architectures, including contrastive dual-tower encoders and generative vision-language models. Our evaluation shows MarsRetrieval is challenging: even strong foundation models often fail to capture domain-specific geomorphic distinctions. We further show that domain-specific fine-tuning is critical for generalizable geospatial discovery in planetary settings. Our code is available at https://github.com/ml-stat-Sustech/MarsRetrieval
Figures & tables
Figure 1: An overview of tasks in MarsRetrieval. The benchmark consists of 3 challenging tasks for cross-modal retrieval in Martian exploration. See Table 1 for data sources and retrieval directions.
Task
Data Sources
Modalities
Paired Image-Text Retrieval
Public + Curated
T → I; I → T
Landform Retrieval
CTX, HiRISE
T → I
Global Geo-Localization
CTX
T → I; I → I
Table 1: Data sources and retrieval directions of the three tasks in MarsRetrieval. We denote the modalities with I =image, T =text.
Figure 2: Planetary scale coverage in task 1 (Paired Image-Text Retrieval). We include Martian paired vision-language samples from orbital terrain to rover instrumentation.
Figure 3: Dataset distribution across the 7 major genetic classes of Martian landforms in Task 2 (Landform Retrieval). We provided the detailed sample distribution in Figure 7 .
Figure 4: Global ground-truth distribution from scientific catalogs [ 41 , 58 , 53 , 40 , 34 ] for Task 3. Different colors denote five target landforms, showing their distributions across Mars.
Size
Model
Backbone
T → I
I → T
R@1
R@10
MRR
MedR
R@1
R@10
MRR
MedR
Encoder-based Models
0.4B
DFN2B-CLIP-ViT-L-14
ViT-L/14
17.32
42.50
25.67
17
20.94
47.57
29.87
12
ViT-L-16-SigLIP-384
ViT-L/16
21.60
42.94
29.20
18
21.73
45.34
29.78
15
aimv2-large-patch14-224
ViT-L/14
10.06
29.43
16.83
43
11.76
33.89
19.15
34
ViT-L-16-SigLIP2-512
ViT-L/16
25.89
50.33
34.06
10
29.38
55.27
37.95
7
Table 2: Retrieval results of Task 1 (Paired Image-Text Retrieval) using Seed2.0-refined captions. We report bidirectional retrieval performance on various vision-language models, grouped by model scale. Best result per column within each model group is in bold. Bootstrap 95% confidence intervals are reported in Appendix E.2 .
Model
mAP
nDCG@10
Hits@10
Encoder-based Models
DFN2B-CLIP-ViT-L-14
9.48
12.39
47.92
ViT-L-16-SigLIP-384
8.59
9.46
37.50
aimv2-large-patch14-224
6.12
7.07
43.75
ViT-L-16-SigLIP2-512
9.59
10.84
60.42
PE-Core-L-14-336
20.93
25.80
72.92
Table 3: Retrieval results of Landform Retrieval (task 2). MLLM-based models are reported at the largest available scale of each model family. Best results within each model group are in bold.
Model
All
w/o MarScope
w/ MarScope
SigLIP2
10.84
12.34
10.49
PE-Core
25.80
24.86
26.02
MarScope
74.42
76.43
73.97
WeMM-9B
17.60
10.51
19.23
Ovis-VL-9B
21.39
12.04
23.61
Table 4: Dataset bias analysis on Task 2. We report nDCG@10 for all 48 classes and subsets collected with and without MarScope. MarScope outperforms general models on both subsets.
Model
Alluvial Fans
Glacier-Like Forms
Landslides
Pitted Cones
Yardangs
Text
Image
Text
Image
Text
Image
Text
Image
Text
Image
Encoder-based VLMs
DFN2B-CLIP-ViT-L-14
1.47
2.32
1.96
17.90
29.27
4.76
7.29
16.71
0.24
28.86
ViT-L-16-SigLIP-384
6.82
1.73
2.29
11.70
23.54
1.98
5.25
14.06
1.65
35.34
aimv2-large-patch14-224
1.54
1.11
2.27
11.65
26.68
2.04
1.70
12.26
1.26
21.86
ViT-L-16-SigLIP2-512
1.55
2.94
6.33
21.97
25.83
2.48
3.68
14.87
1.51
27.79
Table 5: Retrieval results of Global Geo-Localization (Task 3) on various foundation models. We report AUPRC for both text-based and image-based retrieval across five Martian landforms. Best results within the encoder-based and MLLM-based groups are in bold.
Figure 5: Global distribution of Pitted Cones in Task 3. The distribution retrieved by text-based query shows a more similar result to the ground truth catalog.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11Figure 12Figure 13
Stage
Input
Retained
Caption deduplication
86,182
61,868
Score-based filtering
61,868
16,327
Text K-center sampling
16,327
5,000
Image K-center sampling
5,000
2,500
Two-stage expert validation
2,500
2,287
Appendix
Table 6: Number of image-text pairs at each stage of Task 1.
Condition
Mars Relevant
Image-Text Consistent
Eligible
Distinctive
Overall Accept (95% CI)
No score
45.67
82.00
42.67
69.33
25.87 [23.71, 28.14]
CLIP only
45.60
82.33
42.33
72.47
33.87 [31.78, 36.09]
LLM only
48.67
78.13
45.87
69.47
35.67 [33.52, 37.94]
MLLM only
52.27
76.07
53.67
72.13
39.53 [37.28, 41.89]
All three
61.33
79.93
59.20
74.73
46.47 [44.07, 48.93]
Appendix
Table 7: Acceptance rates (%) of 5,000-pair subsets filtered with no score, each score alone and all three scores, judged by Seed2.0-Lite. The overall acceptance is pooled over three sampling seeds, with the Wilson 95% confidence interval.
Setting
CLIP / LLM / MLLM
Filtered Pool
Audit Accept, % (95% CI)
Rank Corr.
Lenient
0.30 / 0.78 / 0.78
39,061
42.4 [38.4, 46.6]
0.8
Default
0.32 / 0.82 / 0.82
16,327
46.0 [41.9, 50.3]
1.0
Strict
0.34 / 0.86 / 0.86
7,957
61.0 [56.8, 65.4]
1.0
Appendix
Table 8: Sensitivity to the score-based filtering thresholds. Audit acceptance is judged by Seed2.0-Lite, and rank correlation is the Spearman correlation of the T → I MRR ranking of four models with the default setting.
Setting
Qwen3-VL
SigLIP2
PE-Core
Lenient
40.06 [38.57, 41.55]
45.63 [44.22, 47.05]
51.43 [50.08, 52.80]
Default
36.98 [35.50, 38.44]
41.66 [40.23, 43.09]
47.83 [46.44, 49.24]
Strict
25.53 [24.11, 26.96]
27.91 [26.49, 29.33]
31.95 [30.53, 33.37]
Appendix
Table 9: T → I MRR (%) with 95% confidence intervals on fixed 3,500-pair subsets constructed under each threshold setting (10,000 paired query-bootstrap replicates).
Figure 6: Image-based query examples for Task 3 (Global Geo-Localization)
Figure 7: Sample distribution of the 48 long-tailed geomorphic categories utilized in Landform Retrieval (Task 2).
Figure 8: Ablation on caption refinement in Paired Image–Text Retrieval (task 1). The results on MRR indicate high-quality captioning leads to stronger vision–language alignment.
Model
Backbone
Size
T → I
I → T
R@1
R@10
MRR
MedR
R@1
R@10
MRR
MedR
Encoder-based Models
DFN2B-CLIP-ViT-L-14
ViT-L/14
0.4B
8.26
22.47
12.92
116
8.05
20.73
12.58
130
ViT-L-16-SigLIP-384
ViT-L/16
0.4B
11.28
22.43
15.27
167
9.01
20.24
13.02
176
aimv2-large-patch14-224
ViT-L/14
0.4B
4.68
14.43
8.11
233
4.77
14.12
8.09
241
ViT-L-16-SigLIP2-512
ViT-L/16
0.4B
13.16
27.72
18.21
89
12.42
26.98
17.34
87
Appendix
Table 10: Performance comparison of different models on Paired Image-Text Retrieval (Task 1) with raw (unrefined) captions. Notably, performance across all models is significantly lower compared to the results with refined captions (see Table 2 ). Bold indicates the best result within each model group.
Model
1
3
5
7
10
MarScope
68.74
71.89
71.87
72.44
72.32
PE-Core-L-14-336
20.67
20.93
20.06
19.68
19.84
Qwen3-VL-Embedding
8.43
8.69
7.63
7.73
7.51
Ops-MM-embedding-v1
9.52
9.62
7.28
8.35
8.50
Appendix
Table 11: Ablation on different prompt templates in Landform Retrieval (task 2). We use 3 templates throughout the paper.
Category Group
#Classes
DFN2B-CLIP
MarScope
Gain
Earth-shared
22
15.85
77.25
+61.39
Planetary-specific
26
9.47
72.05
+62.59
Appendix
Table 12: Task 2 nDCG@10 of DFN2B-CLIP-ViT-L-14 and MarScope on Earth-shared and planetary-specific categories.
Figure 9: Ablation on the spatial tolerance radius r in Global Geo-Localization (task 3). We use r=0.5 throughout the paper.
Model
Alluvial Fans
Glacier-Like Forms
Landslides
Pitted Cones
Yardangs
Text
Image
Text
Image
Text
Image
Text
Image
Text
Image
Encoder-based Models
DFN2B-CLIP-ViT-L-14
6.19
7.94
7.16
34.98
47.90
13.69
19.38
36.52
2.92
47.11
ViT-L-16-SigLIP-384
16.26
6.71
7.69
24.89
38.73
8.95
17.12
29.13
7.97
51.75
aimv2-large-patch14-224
6.83
4.67
8.02
24.76
42.63
8.84
9.51
28.80
6.41
38.01
ViT-L-16-SigLIP2-512
7.13
9.38
15.32
39.40
42.96
9.82
15.47
32.32
8.01
46.21
Appendix
Table 13: Performance comparison of Global Geo-Localization (Task 3) measured by the optimal F1-score ( F1@K∗ ). The domain-specialized MarScope outperforms general foundation models on most landform categories. This gap highlights the necessity of explicit fine-tuning with planetary knowledge for precise large-scale scientific discovery. Best results within the encoder-based and MLLM-based groups are in bold.
Figure 10: Sensitivity of ( F1@K ) to retrieval depth K in task 3. Text-based queries exhibit better peak performance and stability.
Figure 11: The visualization of planetary-scale geomorphic distributions generated by MarScope (Task 3). MarScope’s text-based retrieval produces distributions that are more consistent with global catalogues compared to image-based retrieval.
Task 1 (MRR)
Task 2
Task 3 (AUPRC)
Backbone
Setting
T → I
I → T
mAP
Text
Image
DFN2B-CLIP-ViT-L-14
Zero-shot
25.67
29.87
9.48
8.05
14.11
Fine-tuned (MarScope)
47.70
46.57
71.89
42.17
31.91
ViT-L-16-SigLIP2-512
Zero-shot
34.06
37.95
9.59
7.78
14.01
Fine-tuned
44.37
42.83
68.13
41.85
34.15
Qwen3-VL-Embedding-2B
Zero-shot
29.85
29.03
8.68
3.64
12.17
Appendix
Table 14: Zero-shot and domain-specific fine-tuned results of three backbones. Task 1 reports MRR with Seed2.0 captions, Task 2 reports mAP, and Task 3 reports AUPRC averaged over the five landform categories. Best fine-tuned results are in bold.
T → I
I → T
Model
MRR
95% CI
MRR
95% CI
Encoder-based Models
DFN2B-CLIP-ViT-L-14
25.67
[24.19, 27.15]
29.87
[28.29, 31.46]
ViT-L-16-SigLIP-384
29.20
[27.58, 30.81]
29.78
[28.20, 31.41]
aimv2-large-patch14-224
16.83
[15.60, 18.08]
19.15
[17.86, 20.49]
ViT-L-16-SigLIP2-512
34.06
[32.40, 35.77]
37.95
[36.25, 39.67]
Appendix
Table 15: Task 1 MRR with paired bootstrap 95% confidence intervals (10,000 resamples of the 2,287 image-text pairs, Seed2.0 captions).
Task 1 (MRR)
Task 2
Task 3 (AUPRC)
Seed
T → I
I → T
mAP
nDCG@10
Text
Image
7
35.91
34.18
67.52
70.57
43.66
30.90
8
44.76
37.09
70.52
74.45
42.42
29.32
9
37.04
31.37
70.89
74.31
40.02
32.09
Mean ± std
39.24 ± 4.82
34.21 ± 2.86
69.64 ± 1.85
73.11 ± 2.20
42.04 ± 1.85
30.77 ± 1.39
Appendix
Table 16: MarScope trained with three random seeds. Task 1 reports MRR with Seed2.0 captions, Task 2 reports mAP and nDCG@10, and Task 3 reports AUPRC averaged over the five landform categories. The last row reports mean ± sample standard deviation.
Task 1 (MRR)
Task 2
Task 3 (AUPRC)
Model
Adapted On
T → I
I → T
nDCG@10
Text
Image
OpenAI CLIP-ViT-L-14
Generic
15.32
20.19
9.98
0.82
13.35
+ RemoteCLIP
Earth observation
8.55
17.51
3.42
0.57
12.37
+ SkyCLIP
Earth observation
16.52
20.92
11.90
1.95
11.47
LongCLIP-ViT-L-14
Generic
21.93
21.88
9.14
5.15
16.69
+ DGTRS-CLIP
Earth observation
15.44
19.82
10.48
3.75
15.83
Appendix
Table 17: Earth-observation CLIP models and their pre-trained backbones. Task 1 reports MRR with Seed2.0 captions, Task 2 reports nDCG@10, and Task 3 reports AUPRC averaged over the five landform categories.
Task 1 (MRR)
Task 2
Backbone
Resolution
T → I
I → T
nDCG@10
PE-Core-L-14
224
37.02
37.55
26.94
336
37.50
38.76
28.94
448
37.50
38.50
28.48
PE-Core-G-14
224
20.56
19.62
7.61
336
42.14
40.05
32.74
Appendix
Table 18: PE-Core-L-14 and PE-Core-G-14 evaluated at the same input resolutions. Task 1 reports MRR with Seed2.0 captions and Task 2 reports nDCG@10. Native resolutions are underlined.
High-resolution orbital imagery offers a rich record of the Martian surface, but sparse geological labels limit supervised representation learning. We present MarsRecon, a geospatially aware pipeline for learning visual and multimodal representations from HiRISE observations of Olympus Mons. The pipeline calibrates NASA Planetary Data System products, extracts valid georeferenced patches, and trains a masked autoencoder on unlabeled imagery. Increasing input resolution and filtering invalid tokens reduced held-out reconstruction loss from 0.1751 to 0.1342 in the principal Stage A model series. We then freeze the visual encoder and align its features with observation text, coordinates, and local--global image context. The strongest current local-primary model achieves image-to-text recall@10 of 0.3787, text-to-image recall@10 of 0.9161, and local-to-global recall@10 of 0.4350 on the held-out test split. These results establish a working Mars-specific pretraining and retrieval pipeline; further crop-overlap controls and downstream geological evaluations are needed to assess the broader utility of its embeddings.
Akshay Naik, Marius F. R. Juston, Jay Mahajan
Department of Electrical and Computer Engineering The Grainger College of Engineering University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA · Department of Computer Science University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA
Recent advances in multimodal vision-language models (VLMs) have enabled joint reasoning over visual and textual information, yet their application to planetary science remains largely unexplored. A key hindrance is the absence of large-scale datasets that pair real planetary imagery with detailed scientific descriptions. In this work, we introduce LLaVA-LE (Large Language-and-Vision Assistant for Lunar Exploration), a vision-language model specialized for lunar surface and subsurface characterization. To enable this capability, we curate a new large-scale multimodal lunar dataset, LUCID (LUnar Caption Image Dataset) consisting of 96k high-resolution panchromatic images paired with detailed captions describing lunar terrain characteristics, and 81k question-answer (QA) pairs derived from approximately 20k images in the LUCID dataset. Leveraging this dataset, we fine-tune LLaVA using a two-stage training curriculum: (1) concept alignment for domain-specific terrain description, and (2) instruction-tuned visual question answering. We further design evaluation benchmarks spanning multiple levels of reasoning complexity relevant to lunar terrain analysis. Evaluated against GPT and Gemini judges, LLaVA-LE achieves a 3.3x overall performance gain over Base LLaVA and 2.1x over our Stage 1 model, with a reasoning score of 1.070, exceeding the judge's own reference score, highlighting the effectiveness of domain-specific multimodal data and instruction tuning to advance VLMs in planetary exploration. Code is available at https://github.com/OSUPCVLab/LLaVA-LE.
Aerial navigation on Mars requires vision-based pipelines that are robust to the diverse illumination conditions and terrain morphology of the Martian surface. A key bottleneck for training and evaluating such methods is the scarcity of large-scale, annotated aerial datasets. We present MARTIAN, an open-source Blender-based rendering framework that leverages real HiRISE orbital map products to synthesize realistic aerial views of the Martian terrain under controllable lighting conditions and at varying altitudes. MARTIAN generates observations with accurate pose annotations, directly addressing the scarcity of training data for vision-based navigation on Mars. The framework has been validated through its deployment in concurrent work on map-based localization systems for Ingenuity and future Mars rotorcraft, where synthetically trained deep image matchers were successfully evaluated on real Mars imagery. MARTIAN is publicly available at: https://github.com/nasa-jpl/martian.
Dario Pisanti, Georgios Georgakis
Space Robotics Research Group, SnT, University of Luxembourg. · Jet Propulsion Laboratory, California Institute of Technology.