Zero-shot Vision-Language Reranking for Cross-View Geolocalization
Authors: Yunus Talha Erzurumlu, John E. Anderson, William J. Shuart, Charles Toth, Alper Yilmaz
Organizations: Dept. of Electrical and Computer Engineering, The Ohio State University, 281 W Lane Ave, Columbus, Ohio, USA · US Army Corps of Engineers Geospatial Research Lab, Corbin Field Station, Woodford, Virginia, USA · Dept. of Civil Engineering, The Ohio State University, 281 W Lane Ave, Columbus, Ohio, USA
Cross-view geolocalization (CVGL) systems, while effective at retrieving a list of relevant candidates (high Recall@k), often fail to identify the single best match (low Top-1 accuracy). This work investigates the use of zero-shot Vision-Language Models (VLMs) as rerankers to address this gap. We propose a two-stage framework: state-of-the-art (SOTA) retrieval followed by VLM reranking. We systematically compare two strategies: (1) Pointwise (scoring candidates individually) and (2) Pairwise (comparing candidates relatively). Experiments on the VIGOR dataset show a clear divergence: all pointwise methods cause a catastrophic drop in performance or no change at all. In contrast, a pairwise comparison strategy using LLaVA improves Top-1 accuracy over the strong retrieval baseline. Our analysis concludes that, these VLMs are poorly calibrated for absolute relevance scoring but are effective at fine-grained relative visual judgment, making pairwise reranking a promising direction for enhancing CVGL precision.
Figures & tables
Figure 1: Example of the cross-view geolocalization challenge. a. Ground-level query images. b. Several aerial image candidates, only one of which is the correct match, demonstrating the difficulty caused by viewpoint and appearance differences.
Figure 2: The proposed two-stage framework. Stage 1 uses a SOTA retrieval model to get top- K candidates . Stage 2 uses a VLM reranker to produce the final ranked list.
Method
R@1 (%)
R@3 (%)
R@5 (%)
Baseline (Retrieval Only)
61.20
73.80
82.40
Pointwise Methods
LLaVA Direct
61.20
73.80
82.40
Qwen Direct
25.00
49.00
65.20
LLaVA Likert
4.80
15.20
26.20
Qwen Likert
14.80
34.80
50.80
Table 1: Performance comparison of VLM reranking strategies on the VIGOR dataset (500 test queries). Baseline performance is from the initial SOTA retrieval model. Results are shown in Recall@k (%). ’-’ indicates metric not applicable or computed. Best result for each metric is in bold .
Figure 3: Analysis of pointwise reranking failures. Plots (a–b) show score distributions for direct score prediction, (c–d) for Likert-scale prediction, (e–f) for Yes/No prediction, and (g–h) for Yes/No prediction with explicit reasoning. In all cases, the distributions for correct and incorrect candidates strongly overlap, indicating that the VLM-assigned scores (whether direct, Likert, or Yes/No) do not provide a clear, separable signal to distinguish the true match from other plausible-but-incorrect candidates.
Figure 4: Main Recall Performance Comparison. This plot shows Recall@1, Recall@3, and Recall@5 for the baseline retrieval model and all VLM reranking strategies, sorted by R@1 performance. LLaVA Pairwise (64.80% R@1) is the only method to achieve a significant improvement over the Baseline (61.20% R@1). In contrast, all other pointwise methods (Direct, Likert, Yes/No) cause a catastrophic drop in performance, or no change at all.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Some qualitative results comparing the base model’s ranking and the VLM’s final selection. In each example, the satellite images are displayed in the order originally produced by the base model, not in the reranked order. The blue outlined image indicates the VLM’s top-1 selection among these candidates.