What Does CLIP Learn for Regional Geolocalization? Probing Visual Cues and Scene Configuration After Adaptation
Authors: Changyu Lee, Yeonsoo Park, Abdullah Alfarrarjeh, Seon Ho Kim
Organizations: University of Southern California Los Angeles, USA · USC IMSC San Francisco, USA · German Jordanian University Amman, Jordan · Integrated Media Systems Center (IMSC) University of Southern California Los Angeles, USA
Large collections of street-view imagery provide rich visual information about urban environments, but extracting fine-grained geographic information from such data remains challenging. In particular, fine-grained regional geolocalization is challenging because nearby areas often share coarse geographic cues. We study regional geolocalization within a metropolitan area and ask whether pretrained CLIP features are sufficient for regional discrimination, and what visual information supports performance after adaptation. Using 9,085 street-view images from eight Greater Los Angeles regions, we compare zero-shot CLIP, frozen-encoder readouts, partial encoder updating, Low-Rank Adaptation (LoRA), and full fine-tuning. Frozen readouts remain near the 39.03% zero-shot accuracy, whereas encoder adaptation achieves 75.94-82.10%. Full fine-tuning also reduces the mean distance to the predicted region center from 12.30 km to 3.86 km. We probe these gains through semantic cue removal, appearance reduction using edge maps and blur, and scene-configuration disruption using patch scrambling. Adapted models achieve higher edge and blur accuracy and switch 42.92-45.56% of predictions after scrambling, compared with 10.79-14.60% for frozen methods. However, adaptation does not improve the fraction of performance retained after appearance reduction, while vegetation and sky remain influential. A Caltech101 control further shows that scrambling sensitivity is not unique to geolocalization. Overall, encoder adaptation substantially improves nearby-region discrimination and is associated with greater sensitivity to intact scene configuration, without evidence that coarse structure alone becomes sufficient for prediction. These conclusions concern viewpoint variation near known locations rather than geographically disjoint generalization.
Figures & tables
Fig. 1: Study design and motivating questions. We first evaluate regional geolocalization across increasing levels of CLIP adaptation. The performance comparison motivates an analysis of the visual information associated with adaptation. Complementary interventions measure selected cue dependence, information retained after appearance reduction, and sensitivity to layout disruption. Their joint interpretation considers both scene-configuration sensitivity and appearance-cue dependence.
Fig. 2: Dataset geography and representative scenes for the eight regions. Each group shows sampled locations, a distinctive view, and an ambiguous view. The displayed N is the sample count for that region; annotations report test images whose requested coordinate also occurs in training.
Region
Area (km 2 )
Available
Sampled
Downtown LA
5.13
4,000
998
Hollywood
6.16
4,000
1,197
Beverly Hills
8.98
4,000
1,746
Santa Monica
6.42
4,000
1,248
Venice Beach
3.08
3,921
599
Koreatown
3.08
599
599
TABLE I: Area-normalized sampling of the LA dataset.
Method
Trainable components
Learning rate
Zero-shot
None
–
LP-T
Text-feature scale and bias
10−4
LP-C
Linear classification layer
10−4
PU
Final visual block, ln_post , and head
10−5
LoRA
Adapters in attention and MLP projections
10−4
Full-FT
Full image encoder and classifier
10−5
TABLE II: Configuration of the adaptation strategies.
Fig. 3: Intervention summary: (a) regional gains, (b) cue removal and matched random masks, (c) accuracy and retention after appearance reduction, and (d) layout disruption in geolocalization and Caltech101. Panels (c) and (d) contrast structural sufficiency and sensitivity. Acc: accuracy; Ctrl: control; Geo: geolocalization; SS: structural-sufficiency retention.
Strategy
Accuracy (%)
Centroid Error (km)
Zero-shot
39.03
12.30
LP-T
41.45
12.25
LP-C
38.52
11.22
PU
75.94
5.08
LoRA
78.36
4.21
Full-FT
82.10
3.86
TABLE III: Adaptation performance on original test images. Lower centroid error is better.
Region
Zero-shot (%)
Full-FT (%)
Gain
Downtown LA
35.57
90.60
+55.03
Pasadena
35.56
90.22
+54.66
Beverly Hills
60.69
84.35
+23.66
Santa Monica
36.36
82.35
+45.99
Hollywood
16.11
76.67
+60.56
Long Beach
36.67
75.56
+38.89
TABLE IV: Per-region accuracy and percentage-point gain after full fine-tuning.
Fig. 4: Qualitative cases from Downtown Los Angeles, Hollywood, Santa Monica, Koreatown, and Beverly Hills. Columns show training/test viewing directions, four cue masks, Canny edges, and Patch-32 scrambling with zero-shot, LoRA, and Full-FT predictions. Rows 1–4 share coordinates across viewing directions; the Beverly Hills pair is 0.68 km apart.
Condition
Accuracy (%)
Relative drop (%)
Clean
89.65
0.00
Patch-16
33.90
62.19
Patch-32
61.40
31.51
Patch-64
78.60
12.33
TABLE VIII: Caltech101 zero-shot accuracy under the same patch-scrambling protocol.
Prompt condition
Accuracy (%)
Length-controlled
38.30
Original baseline
39.03
Visual-only descriptors
36.24
Swapped descriptors
11.15
TABLE IX: Zero-shot accuracy under prompt controls. Chance accuracy is 12.5%.
This paper proposes Spatially-Weighted CLIP (SW-CLIP), a novel framework for street-view geo-localization that explicitly incorporates spatial autocorrelation into vision-language contrastive learning. Unlike conventional CLIP-based methods that treat all non-matching samples as equally negative, SW-CLIP leverages Tobler's First Law of Geography to model geographic relationships through distance-aware soft supervision. Specifically, we introduce a location-as-text representation to encode geographic positions and replace one-hot InfoNCE targets with spatially weighted soft labels derived from geodesic distance. Additionally, a neighborhood-consistency regularization is employed to preserve local spatial structure in the embedding space. Experiments on a multi-city dataset demonstrate that SW-CLIP significantly improves geo-localization accuracy, reduces long-tail errors, and enhances spatial coherence compared to standard CLIP. The results highlight the importance of shifting from semantic alignment to geographic alignment for robust geo-localization and provide a general paradigm for integrating spatial principles into multimodal representation learning.
Ting Han, Fengjiao Li, Chunsong Chen +3
School of Geospatial Engineering and Science, Sun Yat-Sen University, Zhuhai, China · School of System Science and Engineering, Sun Yat-Sen University, Guangzhou, China
Worldwide image geo-localization aims to determine where on Earth a single image was captured. However, visually similar scenes may lie thousands of kilometers apart, so methods that localize primarily by appearance often mistake a distant look-alike for the true location. We attribute this failure to a structural cause: in existing methods, GPS coordinates serve only as training supervision, and the distance relationships among locations never enter the learned representation. To address this, we propose GeoMetric, a retrieval-based framework that encodes GPS coordinates relationally rather than in isolation, injecting the distance structure among locations into both representation learning and inference. GeoMetric comprises three components: (1) a Transformer-based GPS encoder with distance-aware location attention that modulates inter-sample aggregation by great-circle proximity; (2) a trimodal contrastive objective that aligns images, geo-textual descriptions, and GPS embeddings in a unified space; and (3) a retrieval-augmented inference stage that supplies large multimodal models (LMMs) with contrastive candidate context for grounded coordinate reasoning. Extensive experiments on IM2GPS, IM2GPS3k, YFCC4k, and YFCC26k demonstrate that GeoMetric consistently outperforms state-of-the-art methods, improving street-level accuracy (within 1 km) by 1.5%, 0.9%, 6.9%, and 2.5%, respectively. Controlled ablations confirm that the gains originate from the proposed geographic encoding rather than any specific LMM.
Junchao Cui, Xuanzi Ma, Wenqi Shi +3
Information Engineering University Zhengzhou, China
The emergence of Vision-Language Models (VLMs) has introduced new paradigms for global image geo-localization through retrieval-augmented generation (RAG) and reasoning-driven inference. However, RAG methods are constrained by retrieval database quality, while reasoning-driven approaches fail to internalize image locatability, relying on inefficient, fixed-depth reasoning paths that increase hallucinations and degrade accuracy. To overcome these limitations, we introduce an Optimized Locatability Score that quantifies an image's suitability for deep reasoning in geo-localization. Using this metric, we curate Geo-ADAPT-51K, a locatability-stratified reasoning dataset enriched with augmented reasoning trajectories for complex visual scenes. Building on this foundation, we propose a two-stage Group Relative Policy Optimization (GRPO) curriculum with customized reward functions that regulate adaptive reasoning depth, visual grounding, and hierarchical geographical accuracy. Our framework, Geo-ADAPT, learns an adaptive reasoning policy, achieves state-of-the-art performance across multiple geo-localization benchmarks, and substantially reduces hallucinations by reasoning both adaptively and efficiently.
Bo Yu, Fengze Yang, Yiming Liu +6
The University of Utah · The University of Oklahoma · Worcester Polytechnic Institute +2