Organizations: Vermont Artificial Intelligence Lab, Department of Computer Science, University of Vermont · Intelligent Machines Lab, Department of Artificial Intelligence, Information Technology University · Institute of Artificial Intelligence, University of Central Florida
Natural-language Guided Cross-view Geo-localization (NGCG) aims to retrieve geo-tagged satellite imagery using textual descriptions of ground scenes. While recent NGCG methods commonly rely on CLIP-style dual-encoder architectures, they often suffer from weak cross-modal generalization and require complex architectural designs. In contrast, Multimodal Large Language Models (MLLMs) offer powerful semantic reasoning capabilities but are not directly optimized for retrieval tasks. In this work, we present a simple yet effective framework to adapt MLLMs for NGCG via parameter-efficient finetuning. Our approach optimizes latent representations within the MLLM while preserving its pretrained multimodal knowledge, enabling strong cross-modal alignment without redesigning model architectures. Through systematic analysis of diverse variables, from model backbone to feature aggregation, we provide practical and generalizable insights for leveraging MLLMs in NGCG. Our method achieves SOTA on GeoText-1652 with a 12.2% improvement in Text-to-Image Recall@1 and secures top performance in 5 out of 12 subtasks on CVG-Text, all while surpassing baselines with far fewer trainable parameters. These results position MLLMs as a robust foundation for semantic cross-view retrieval and pave the way for MLLM-based NGCG to be adopted as a scalable, powerful alternative to traditional dual-encoder designs. Project page and code are available at https://yuqichen888.github.io/NGCG-MLLMs-web/.
Figures & tables
Figure 1: Comparison between the CLIP-style Dual-Encoder Architecture and Our Unified MLLM Framework.
Aspect
Research Question
Arch.
(RQ1) How does the parameter scale of the MLLM backbones affect NGCG performance?
(RQ2) What is the best way to aggregate sequence embeddings to capture cross-view scene semantics?
Adap.
(RQ3) Which tuning strategy is optimal? Specifically, Full Fine-Tuning (FFT) versus LoRA-based adaptation?
(RQ4) How do the hyperparameter configurations of the parameter-efficient module (e.g., LoRA) influence adaptation performance?
Opt.
(RQ5) How do design choices within the contrastive learning objective influence the discriminative capability and stability of the learned embeddings?
Table 1: Classification of Research Questions (RQs) for adapting MLLMs to NGCG, categorized by key aspects: Architecture (Arch.), Adaptation (Adap.), and Optimization (Opt.).
Figure 2: The framework leverages a pre-trained MLLM for feature encoding, processing both text and visual data via its shared weight design. The adaptation framework relies on LoRA, combined with an InfoNCE loss function, to align the MLLM’s internal representation space for the NGCG retrieval task.
Method
#TPs
Text → Image
Image → Text
R@1
R@5
R@10
R@1
R@5
R@10
UNITER [ 9 ]
300M
10.6
20.4
26.1
21.4
43.4
59.5
METER [ 12 ]
380M
11.3
21.5
27.3
22.7
46.3
60.7
ALBEF [ 25 ]
210M
12.3
22.8
28.6
22.9
49.5
62.3
XVLM [ 52 ]
216M
13.1
23.5
29.2
23.6
50.0
63.2
GeoText [ 10 ]
216M
13.6
24.6
31.2
26.3
53.7
66.9
Table 2: Comparison of natural-language guided cross-view geo-localization performance on the GeoText-1652 dataset. We report recall rates (%) at different top-K retrieval thresholds. R@K represents the Top-K recall rate. #TPs: Number of Trainable Parameters. [Key: Best , Second Best ]
Method
Text → Image
Image → Text
R@1
R@5
R@10
R@1
R@5
R@10
CLIP-L/14 [ 35 ]
37.7
52.9
60.3
49.9
82.0
90.9
GeoText [ 10 ]
29.9
46.3
54.1
50.1
81.2
90.3
Ours InternVL
45.7
62.9
70.3
58.3
85.9
92.9
Table 3: Performance comparison on the GeoText-1652 24G test set. We report Recall@K (R@K, %) for both Text-to-Image (T → I) and Image-to-Text (I → T) retrieval tasks. Bold values indicate the best performance.
Method
# Trainable Params
Ground Text → Satellite Image
Ground Text → OSM Image
NewYork
Brisbane
Tokyo
NewYork
Brisbane
Tokyo
R@1
L@50
R@1
L@50
R@1
L@50
R@1
L@50
R@1
L@50
R@1
L@50
ViLT [ 21 ]
87.4M
11.58
15.58
11.00
14.50
10.83
15.50
5.83
9.92
8.67
11.75
4.67
9.17
X-VLM [ 52 ]
216M
15.74
16.86
15.67
17.60
12.46
14.34
16.14
17.26
20.46
21.94
9.53
10.94
SigLIP-B/16 [ 53 ]
203M
19.67
21.08
19.58
22.00
15.58
17.92
20.17
21.58
25.58
27.42
11.92
13.67
CLIP-L/14 [ 35 ]
428M
35.08
37.08
34.08
37.25
28.08
30.50
31.50
33.58
32.50
34.67
21.00
23.17
Table 4: Comparison of natural-language guided cross-view geo-localization performance on the CVG-Text dataset for Text-to-Satellite Image and Text-to-OSM Image Retrieval. We report recall rates (%) and localization hit rates (%) based on thresholded retrieval. R@1 represents Top-1 Recall (the correct candidate is the top result). L@50 represents the localization recall rate, where the correct candidate is retrieved and the predicted location error is less than 50 meters. [Key: Best , Second Best ]
Task
Backbone
#TPs
NewYork
Brisbane
Tokyo
R@1
L@50
R@1
L@50
R@1
L@50
G2S
InternVL
9M
44.75
47.50
45.25
49.00
33.00
36.50
S-500M
6.4M
36.50
39.17
38.25
42.50
26.83
30.25
S-256M
4.1M
33.25
35.83
34.83
38.17
24.17
27.17
G2O
InternVL
9M
56.92
63.33
37.17
41.83
39.17
44.33
S-500M
6.4M
53.92
60.00
40.33
45.25
29.17
36.58
Table 5: Analysis of different scales MLLM backbone on the CVG-Text Benchmark. Metrics R@1 and L@50 are reported. “G2S” is short for “ground text → satellite” and “G2O” is short for “ground text → OSM image”. “S-500M” and “S-256M” are short for “SmolVLM-500M” and “SmolVLM-256M”, respectively. #TPs: Number of Trainable Parameters. [Key: Best ]
Pooling
R@1
R@5
R@10
L@50
L@100
L@150
[EOS]
44.75
75.92
86.00
47.50
51.33
54.25
Query
36.83
72.50
83.58
39.17
43.00
45.33
Avg.
36.92
72.50
82.67
39.50
43.58
46.25
Table 6: Analysis on the feature aggregation strategy on Text-to-Satellite task of CVG-Text New York Subset. “[EOS] is short for [EOS] token pooling. “Avg.” is short for average pooling. “Query” is short for query-based pooling. [Key: Best ]
backbone
Ground Text → Satellite Image
New York
Brisbane
Tokyo
R@1
R@5
L@50
L@100
R@1
R@5
L@50
L@100
R@1
R@5
L@50
L@100
SmolVLM-500M (Full)
20.67
54.33
22.50
25.58
21.08
52.33
23.33
26.00
13.17
43.92
16.67
20.83
SmolVLM-500M (LoRA)
36.50
70.67
39.17
43.33
38.25
70.00
42.50
46.25
26.83
61.17
30.25
35.17
Gain
+15.83
+16.34
+16.67
+17.75
+17.17
+17.67
+19.17
+20.25
+13.66
+17.25
+13.58
+14.34
InternVL(Full)
41.58
74.75
44.17
48.17
38.67
73.58
42.92
46.00
28.83
63.83
32.25
37.75
Table 7: Comparison of fine-tuning strategies on SmolVLM-500M and InternVL3.5-1B for Text-to-Satellite on the CVG-Text Benchmark.
backbone
Ground Text → OSM Image
New York
Brisbane
Tokyo
R@1
R@5
L@50
L@100
R@1
R@5
L@50
L@100
R@1
R@5
L@50
L@100
SmolVLM-500M (Full)
41.58
64.83
45.42
49.17
38.83
62.25
43.67
46.50
22.67
46.33
28.42
35.17
SmolVLM-500M (LoRA)
53.92
76.42
60.00
63.83
40.33
64.83
45.25
48.92
29.17
54.92
36.58
42.50
Gain
+12.34
+11.59
+14.58
+14.66
+1.5
+2.58
+1.58
+2.42
+6.5
+8.59
+8.16
+7.33
Table 8: Comparison of fine-tuning strategies on SmolVLM-500M for Text-to-OSM on the CVG-Text Benchmark.
rα
α
R@1
R@5
L@50
L@100
1
16
29.08
66.5
31.58
35.83
2
32
34.00
69.92
36.5
40.58
4
64
39.25
75.75
41.75
45.17
8
128
44.75
75.92
47.50
51.33
Table 9: Analysis of LoRA Scaling Factor ( α ) on on Text-to-Satellite task of CVG-Text New York subset. The LoRA rank ( r ) is fixed at 16, and the scaling factor α is varied. [Key: Best ]
τ
R@1
R@5
L@50
L@100
0.02
41.33
74.33
44.66
48.00
0.03
44.75
75.92
47.50
51.33
0.05
41.83
75.75
44.58
48.50
0.07
39.25
75.75
41.75
45.17
0.1
37.42
72.08
39.67
43.25
Learnable
39.08
75.92
41.58
44.92
Table 10: Analysis of the loss temperature( τ ) on Text-to-Satellite task of CVG-Text New York subset. [Key: Best ]
Figure 3: Visualization of Text-to-Satellite Image Retrieval Results on CVG-Text. Four text-satellite pairs are shown. Left is the query text. On the right is the the retrieved satellite images. Ground Truth satellite images are outlined with a green border, while incorrect matches are outlined with a red border.
Method
GFLOPs (G)
Latency (ms)
Memory (MB)
GeoText
49.34
9.64
1268.54
CrossText2Loc
200.25
20.22
2979.86
Ours-S
130.24
33.28
682.98
Ours-I
581.35
100.44
2258.85
Table 11: Efficiency Analysis. The computational cost of Ours-S with SmolVLM-256M and Ours-I with InternVL3.5-1B against two dual-encoder baselines. Bold values indicate the most efficient performance in each category.
School of Computer Engineering and Science, Shanghai University, Shanghai 200444, China · Faculty of Science and Technology, and Institute of Collaborative Innovation, University of Macau, Macau 999078, China
School of Remote Sensing and Information Engineering, Wuhan University, Wuhan, 430079, China · First Surveying and Mapping Institute of Hunan Province, Changsha, 421001, China