What Does CLIP Learn for Regional Geolocalization? Probing Visual Cues and Scene Configuration After Adaptation
Authors: Changyu Lee, Yeonsoo Park, Abdullah Alfarrarjeh, Seon Ho Kim
Organizations: University of Southern California Los Angeles, USA · USC IMSC San Francisco, USA · German Jordanian University Amman, Jordan · Integrated Media Systems Center (IMSC) University of Southern California Los Angeles, USA
Large collections of street-view imagery provide rich visual information about urban environments, but extracting fine-grained geographic information from such data remains challenging. In particular, fine-grained regional geolocalization is challenging because nearby areas often share coarse geographic cues. We study regional geolocalization within a metropolitan area and ask whether pretrained CLIP features are sufficient for regional discrimination, and what visual information supports performance after adaptation. Using 9,085 street-view images from eight Greater Los Angeles regions, we compare zero-shot CLIP, frozen-encoder readouts, partial encoder updating, Low-Rank Adaptation (LoRA), and full fine-tuning. Frozen readouts remain near the 39.03% zero-shot accuracy, whereas encoder adaptation achieves 75.94-82.10%. Full fine-tuning also reduces the mean distance to the predicted region center from 12.30 km to 3.86 km. We probe these gains through semantic cue removal, appearance reduction using edge maps and blur, and scene-configuration disruption using patch scrambling. Adapted models achieve higher edge and blur accuracy and switch 42.92-45.56% of predictions after scrambling, compared with 10.79-14.60% for frozen methods. However, adaptation does not improve the fraction of performance retained after appearance reduction, while vegetation and sky remain influential. A Caltech101 control further shows that scrambling sensitivity is not unique to geolocalization. Overall, encoder adaptation substantially improves nearby-region discrimination and is associated with greater sensitivity to intact scene configuration, without evidence that coarse structure alone becomes sufficient for prediction. These conclusions concern viewpoint variation near known locations rather than geographically disjoint generalization.
Figures & tables
Fig. 1: Study design and motivating questions. We first evaluate regional geolocalization across increasing levels of CLIP adaptation. The performance comparison motivates an analysis of the visual information associated with adaptation. Complementary interventions measure selected cue dependence, information retained after appearance reduction, and sensitivity to layout disruption. Their joint interpretation considers both scene-configuration sensitivity and appearance-cue dependence.
Fig. 2: Dataset geography and representative scenes for the eight regions. Each group shows sampled locations, a distinctive view, and an ambiguous view. The displayed N is the sample count for that region; annotations report test images whose requested coordinate also occurs in training.
Region
Area (km 2 )
Available
Sampled
Downtown LA
5.13
4,000
998
Hollywood
6.16
4,000
1,197
Beverly Hills
8.98
4,000
1,746
Santa Monica
6.42
4,000
1,248
Venice Beach
3.08
3,921
599
Koreatown
3.08
599
599
TABLE I: Area-normalized sampling of the LA dataset.
Method
Trainable components
Learning rate
Zero-shot
None
–
LP-T
Text-feature scale and bias
10−4
LP-C
Linear classification layer
10−4
PU
Final visual block, ln_post , and head
10−5
LoRA
Adapters in attention and MLP projections
10−4
Full-FT
Full image encoder and classifier
10−5
TABLE II: Configuration of the adaptation strategies.
Fig. 3: Intervention summary: (a) regional gains, (b) cue removal and matched random masks, (c) accuracy and retention after appearance reduction, and (d) layout disruption in geolocalization and Caltech101. Panels (c) and (d) contrast structural sufficiency and sensitivity. Acc: accuracy; Ctrl: control; Geo: geolocalization; SS: structural-sufficiency retention.
Strategy
Accuracy (%)
Centroid Error (km)
Zero-shot
39.03
12.30
LP-T
41.45
12.25
LP-C
38.52
11.22
PU
75.94
5.08
LoRA
78.36
4.21
Full-FT
82.10
3.86
TABLE III: Adaptation performance on original test images. Lower centroid error is better.
Region
Zero-shot (%)
Full-FT (%)
Gain
Downtown LA
35.57
90.60
+55.03
Pasadena
35.56
90.22
+54.66
Beverly Hills
60.69
84.35
+23.66
Santa Monica
36.36
82.35
+45.99
Hollywood
16.11
76.67
+60.56
Long Beach
36.67
75.56
+38.89
TABLE IV: Per-region accuracy and percentage-point gain after full fine-tuning.
Fig. 4: Qualitative cases from Downtown Los Angeles, Hollywood, Santa Monica, Koreatown, and Beverly Hills. Columns show training/test viewing directions, four cue masks, Canny edges, and Patch-32 scrambling with zero-shot, LoRA, and Full-FT predictions. Rows 1–4 share coordinates across viewing directions; the Beverly Hills pair is 0.68 km apart.
Condition
Accuracy (%)
Relative drop (%)
Clean
89.65
0.00
Patch-16
33.90
62.19
Patch-32
61.40
31.51
Patch-64
78.60
12.33
TABLE VIII: Caltech101 zero-shot accuracy under the same patch-scrambling protocol.
Prompt condition
Accuracy (%)
Length-controlled
38.30
Original baseline
39.03
Visual-only descriptors
36.24
Swapped descriptors
11.15
TABLE IX: Zero-shot accuracy under prompt controls. Chance accuracy is 12.5%.
School of Geospatial Engineering and Science, Sun Yat-Sen University, Zhuhai, China · School of System Science and Engineering, Sun Yat-Sen University, Guangzhou, China