GeoContext: One Context Ladder, Two Failure Modes in Vision-Language Geolocation: Flat Reliance on User-Provided Location Context and False Confirmation of Location Claims

Yifan Zhang; Kai Wang

cs.CVSep 4, 2026

GeoContext: One Context Ladder, Two Failure Modes in Vision-Language Geolocation: Flat Reliance on User-Provided Location Context and False Confirmation of Location Claims

Authors: Yifan Zhang, Kai Wang

Abstract

Visual geolocation benchmarks typically ask a model where an image was captured without accounting for the location context that users often provide. We introduce GeoContext, a resource supporting two complementary tasks: GeoHint, open-ended localization given a true but coarse location hint, and GeoVerify, binary verification of whether an image was taken within 150 m of a claimed place. GeoContext constructs a context ladder by stratifying nearby reference points according to distance and referenceability, allowing the image to remain fixed while the supplied context varies. The benchmark covers 109 sites in 30 cities and evaluates five vision-language models using 21,933 GeoHint responses and 6,270 GeoVerify responses. Our evaluation reveals three main patterns. First, hint repetition varies by only 1.5 percentage points across referenceability tiers and by less than 3 points across distance bands, while the resulting localization error increases steadily with hint distance. Second, behavior depends strongly on no-context performance: at sites with low no-context accuracy, the median ratio between localization error and hint distance is approximately 1.00, whereas at higher-accuracy sites it ranges from 0.24 to 0.69. After correcting for bias introduced by the site grouping procedure, only one of the five models retains a negative accuracy estimate when given a nearby hint. Third, in GeoVerify, no model reaches d' = 1 for decoys immediately beyond the 150 m tolerance. Model rankings also change when sensitivity is separated from response bias, and 83.8% of false acceptances are reported with confidence of at least 0.8. We release the benchmark, construction pipeline, audit decisions, and scoring code.

Figures & tables

Figure 1: Construction pipeline. City screening, site selection, imagery retrieval and reference-point rating are all programmatic; the only human step is a spot audit of the retrieved images (Sec. 3.1 ). The first three stages are held fixed within a site, so the reference point is the sole manipulated input. Because the pipeline is a sampler rather than a fixed file, the benchmark can be regenerated on imagery captured after a given date — the route we intend to use for contamination control.

Model	$n$	Hit rate	$0.5$ – $1.5$	$1.5$ – $3$	$3$ – $6$	Echo rate
			Hit rate by band ( $\mathrm{km}$ )
claude-opus-5	1208	$52.6\text{\,}\mathrm{\%}$	$62.0\text{\,}\mathrm{\%}$	$49.5\text{\,}\mathrm{\%}$	$47.9\text{\,}\mathrm{\%}$	$19.2\text{\,}\mathrm{\%}$
gemini-flash	1788	$51.2\text{\,}\mathrm{\%}$	$63.4\text{\,}\mathrm{\%}$	$47.8\text{\,}\mathrm{\%}$	$44.1\text{\,}\mathrm{\%}$	$12.2\text{\,}\mathrm{\%}$
qwen3-vl-235b	1783	$32.6\text{\,}\mathrm{\%}$	$46.0\text{\,}\mathrm{\%}$	$29.2\text{\,}\mathrm{\%}$	$24.6\text{\,}\mathrm{\%}$	$24.6\text{\,}\mathrm{\%}$
qwen3-vl-8b	1790	$30.6\text{\,}\mathrm{\%}$	$46.0\text{\,}\mathrm{\%}$	$23.1\text{\,}\mathrm{\%}$	$24.5\text{\,}\mathrm{\%}$	$42.3\text{\,}\mathrm{\%}$
claude-haiku-4-5	1773	$25.0\text{\,}\mathrm{\%}$	$39.8\text{\,}\mathrm{\%}$	$20.6\text{\,}\mathrm{\%}$	$16.7\text{\,}\mathrm{\%}$	$22.8\text{\,}\mathrm{\%}$

Table 1: Per-model results, forced schema, context conditions only (baseline rows excluded). Bands are the distance from the true location to the reference point the model was handed. Echo rate is the fraction of responses naming that reference point. claude-opus-5 has fewer trials because it was sampled at one third the rate of the others, its per-call cost being an order of magnitude higher.

Figure 2: Hit rate by reference-point distance, forced schema, with sites split by whether their imagery is legible. Each plotted point rests on

11511668

observations; error bars are

95\text{\,}\mathrm{\%}

binomial intervals. The gap between the two curves is stable across all three distance bands and neither curve’s interval approaches the other’s at any band. Both curves are built from context conditions only, which the regime assignment never touches; the one comparison that does read the no-context column is corrected separately in Table 3 .

Illegible sites ( $65$ )
				Hit rate by band ( $\mathrm{km}$ )
Model	$n$	No context	With context	$0.5$ – $1.5$	$1.5$ – $3$	$3$ – $6$	Echo rate
claude-opus-5	649	$34.0\text{\,}\mathrm{\%}$	$39.6\text{\,}\mathrm{\%}$	$48.2\text{\,}\mathrm{\%}$	$37.9\text{\,}\mathrm{\%}$	$34.4\text{\,}\mathrm{\%}$	$23.9\text{\,}\mathrm{\%}$
gemini-flash	974	$47.2\text{\,}\mathrm{\%}$	$34.8\text{\,}\mathrm{\%}$	$49.5\text{\,}\mathrm{\%}$	$30.7\text{\,}\mathrm{\%}$	$26.4\text{\,}\mathrm{\%}$	$15.7\text{\,}\mathrm{\%}$
qwen3-vl-235b	974	$7.9\text{\,}\mathrm{\%}$	$17.7\text{\,}\mathrm{\%}$	$33.6\text{\,}\mathrm{\%}$	$12.1\text{\,}\mathrm{\%}$	$9.6\text{\,}\mathrm{\%}$	$30.1\text{\,}\mathrm{\%}$
qwen3-vl-8b	974	$7.9\text{\,}\mathrm{\%}$	$14.2\text{\,}\mathrm{\%}$	$29.8\text{\,}\mathrm{\%}$	$7.4\text{\,}\mathrm{\%}$	$7.3\text{\,}\mathrm{\%}$	$51.1\text{\,}\mathrm{\%}$

Table 2: Per-model results within each regime, forced schema. Sites are assigned to a regime by a fixed

50\text{\,}\mathrm{\%}

threshold on their no-context hit rate (

65

illegible,

44

legible). “No context” is the baseline condition for the same sites and model; the remaining columns cover the context conditions, with

n

counting those. Rows ordered by overall accuracy.

Regime	Model	Uncorrected	Corrected	Shift
Illegible	claude-opus-5	$+14.2$	$+9.3\pm 7.1$	$-4.8$
	gemini-flash	$+2.3$	$-3.0\pm 4.8$	$-5.3$
	qwen3-vl-235b	$+25.7$	$+19.6\pm 4.9$	$-6.1$
	qwen3-vl-8b	$+22.0$	$+16.9\pm 4.0$	$-5.1$
	claude-haiku-4-5	$+19.2$	$+17.8\pm 4.0$	$-1.4$
Legible	claude-opus-5	$-7.0$	$+1.4\pm 5.6$	$+8.3$

Table 3: The no-context comparison before and after correcting for the fact that the no-context column also assigns the regime. “Uncorrected” is the near-band figure minus the “No context” column of Table 2 ; “corrected” assigns the regime from a random half of each site’s baseline responses and measures the column on the other half, averaged over

200

splits. An average of

19

of the

109

sites change side per split.

Figure 3: Echo rate by reference-point distance and referenceability tier, forced schema,

109

sites. Within every distance band the three referenceability tiers agree to within about

5

points: a reference point almost nobody uses for navigation is repeated about as often as one everybody uses. Distance does have an effect, but a small and non-monotone one — pooled over tiers the rate falls from

27.4\text{\,}\mathrm{\%}

22.1\text{\,}\mathrm{\%}

between the first and second band and returns to

24.6\text{\,}\mathrm{\%}

in the third, so moving the reference point from

0.5\text{\,}\mathrm{km}

6\text{\,}\mathrm{km}

away changes it by

2.8

points.

By referenceability tier
	$0.5$ – $1.5$	$1.5$ – $3$	$3$ – $6$	All
High ( $4$ – $5$ )	27.3	21.0	24.9	24.4
Mid ( $2$ – $3$ )	27.6	23.6	26.5	25.8
Low ( $0$ – $1$ )	27.0	21.1	21.9	22.9
Pooled
All	27.4	22.1	24.6	24.6

Table 4: Echo rate (

\mathrm{\%}

). Flat along both axes.

Regime	Band ( $\mathrm{km}$ )	$n$	$d_{\mathrm{ref}}$	$\mathrm{err}_{\mathrm{km}}$	Median ratio
Illegible (65 sites)	$0.5$ – $1.5$	1163	1.01	1.27	1.00
	$1.5$ – $3$	1325	2.22	2.17	1.00
	$3$ – $6$	1452	4.20	3.04	0.90
Legible (44 sites)	$0.5$ – $1.5$	1102	0.95	0.54	0.69
	$1.5$ – $3$	1194	2.11	0.75	0.44
	$3$ – $6$	1323	4.41	0.99	0.24

Table 5: Distance to the reference point and distance to the model’s answer, by regime and band. The

d_{\mathrm{ref}}

and

\mathrm{err}_{\mathrm{km}}

columns are geometric means; the last column is the median of the per-response ratio

\mathrm{err}_{\mathrm{km}}/d_{\mathrm{ref}}

. Computed only on responses resolving finer than city level, so the constant city-centroid fallback (Sec. 4.3 ) cannot drive the ratio.

Model	$n$	$\beta$	$p$	$\beta$ (fine only)
claude-haiku-4-5	1767	$+0.1046$	$3.2\times 10^{-12}$	$+0.1026$
qwen3-vl-235b	1766	$+0.0961$	$4.8\times 10^{-12}$	$+0.0966$
qwen3-vl-8b	1777	$+0.0812$	$3.4\times 10^{-12}$	$+0.0805$
gemini-flash	1758	$+0.0713$	$3.2\times 10^{-10}$	$+0.0795$
claude-opus-5	1206	$+0.0500$	$5.6\times 10^{-4}$	$+0.0596$

Table 6:

\log_{10}\mathrm{err}_{\mathrm{km}}\sim d_{\mathrm{ref}}

per model, forced schema, context conditions only. “Fine only” excludes city-level resolutions. Every slope is positive and significant.

	Illegible		Legible
Model	$n$	$\beta$	$n$	$\beta$
claude-opus-5	647	$+0.0684$	559	$+0.0347$
gemini-flash	955	$+0.0935$	803	$+0.0497$
qwen3-vl-235b	970	$+0.1223$	796	$+0.0704$
qwen3-vl-8b	968	$+0.1116$	809	$+0.0521$
claude-haiku-4-5	972	$+0.1015$	795	$+0.1110$

Table 7:

\log_{10}\mathrm{err}_{\mathrm{km}}\sim d_{\mathrm{ref}}

fitted separately within each regime, forced schema, context conditions only. A larger slope means the answer degrades faster as the reference point is moved away.

Regime	Schema	Near	Mid	Far	Overall	Paired $\Delta$
Illegible	forced	36.1	17.7	15.8	22.6	$+0.46$ , $p=$ 0.49$$
Illegible	forced_warn	35.4	18.1	17.3	23.0	$+0.46$ , $p=$ 0.49$$
Legible	forced	67.9	51.4	48.2	55.2	$\mathbf{+3.94}$ , $\mathbf{p=$ 0.0008 $}$
Legible	forced_warn	69.4	55.5	54.2	59.2	$\mathbf{+3.94}$ , $\mathbf{p=$ 0.0008 $}$
All sites	forced	50.6	33.0	30.5	37.4	$+2.01$ , $p=$ 0.002$$
All sites	forced_warn	50.7	35.0	33.9	39.3	$+2.01$ , $p=$ 0.002$$

Table 8: Hit rate (

\mathrm{\%}

) by regime, strictly paired.

\Delta

is the paired difference with a two-sided permutation test,

10\,000

sign flips.

Model	$n$ pairs	All sites	Illegible	Legible
claude-opus-5	1200	$+3.67$	$+1.54$	$+6.17$
gemini-flash	1754	$+2.51$	$+2.05$	$+3.08$
qwen3-vl-235b	1753	$+2.97$	$+0.10$	$+6.54$
qwen3-vl-8b	1754	$+1.45$	$-0.72$	$+4.17$
claude-haiku-4-5	1754	$+0.00$	$-0.31$	$+0.38$

Table 9: Paired difference in hit rate (percentage points) from adding the warning, by model. Positive means forced_warn scored higher than forced on the same items. Rows ordered by overall accuracy.

Regime	Schema	Near	Mid	Far	Overall	Ratio (near/mid/far)
Illegible	forced	37.4	16.7	15.7	22.2	1.00 / 1.01 / 0.87
Illegible	forced_chain	39.8	14.9	15.5	22.2	1.00 / 1.02 / 0.87
Legible	forced	66.7	49.0	41.5	51.2	0.60 / 0.50 / 0.50
Legible	forced_chain	63.2	51.6	44.4	52.2	0.60 / 0.43 / 0.35

Table 10: forced_chain versus forced , strictly paired, three models. Hit rate in

\mathrm{\%}

; ratio is median error/reference distance, computed on non-city-level resolutions only.

Model	evidence	Infra.	Veg.	Arch.	Script	Terrain
gemini-flash	$99.7\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$21.2\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$2.6\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$
qwen3-vl-235b	$100.0\text{\,}\mathrm{\%}$	$3.8\text{\,}\mathrm{\%}$	$29.8\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$7.2\text{\,}\mathrm{\%}$	$4.5\text{\,}\mathrm{\%}$
claude-haiku-4-5	$99.8\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$27.6\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$7.2\text{\,}\mathrm{\%}$	$0.4\text{\,}\mathrm{\%}$

Table 11: forced_chain compliance,

2799

responses. The second column gives the share of responses carrying an evidence object; the remaining columns give the share of each category’s slots answered "not visible" , the response the prompt requires in place of a guess.

Response rates
	gemini-	claude-	qwen3-	qwen3-	claude-
	flash	opus-5	vl-8b	vl-235b	haiku-4-5	Pooled
$H$ (signal accepted)	$93.5\text{\,}\mathrm{\%}$	$47.2\text{\,}\mathrm{\%}$	$44.4\text{\,}\mathrm{\%}$	$70.4\text{\,}\mathrm{\%}$	$78.7\text{\,}\mathrm{\%}$	$69.9\text{\,}\mathrm{\%}$
$F$ (decoy accepted)	$35.1\text{\,}\mathrm{\%}$	$13.1\text{\,}\mathrm{\%}$	$18.8\text{\,}\mathrm{\%}$	$43.8\text{\,}\mathrm{\%}$	$61.1\text{\,}\mathrm{\%}$	$37.7\text{\,}\mathrm{\%}$
control: swapped image	$3.5\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$18.1\text{\,}\mathrm{\%}$	$5.0\text{\,}\mathrm{\%}$
control: gray field	$0.0\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$	$0.0\text{\,}\mathrm{\%}$

Table 12: GeoVerify results, one column per model.

H

is the hit rate on signal trials (true-positive rate) and

F

the false-acceptance rate; the control rows give

F

when the image is swapped for one from another continent and when it is replaced by a uniform gray field.

d^{\prime}

is sensitivity by decoy distance and

\delta_{1}

the discrimination radius, the nearest band reaching

d^{\prime}\geq 1

. The criterion

c

is negative when a model is biased toward accepting a claim and positive when biased toward rejecting one. The last two rows describe the confidence attached to false acceptances. Log-linear correction applied ( Hautus, 1995 ) .

Figure 4: Sensitivity against decoy distance, one line per model; the same numbers as the

d^{\prime}

block of Table 12 , drawn to show the shape. The dashed line marks

d^{\prime}=1

, conventional usable discrimination. The shaded strip is the band that begins immediately above the

150\text{\,}\mathrm{m}

tolerance — the discrimination a deployment actually has to make — and no model crosses the dashed line inside it. Every model improves as the decoy is moved away, and they separate by a factor of three by the farthest band, so the difficulty is a property of the distance rather than of the task as a whole.

Geospatial Reasoning Benchmarks

See all 31 papers

Where Do Vision-Language Models Fail? World Scale Analysis for Image Geolocalization
Apr 17, 2026Siddhant Bharadwaj, Ashish Vashist, Fahimul Aleem +1VLM Evaluation Large Vision-Language Models
University of Central Florida
ERGeoBench:A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models
May 29, 2026Kaiwen Xue, Tao Wei, Guoxin Zhang +5Multimodal Large Language Models Visual Spatial Reasoning
Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Materials Science and Engineering, Shanghai Jiao Tong University, Shanghai, China +2
Learning to Wander: Improving the Global Image Geolocation Ability of LMMs via Actionable Reasoning
Mar 11, 2026Yushuo Zheng, Huiyu Duan, Zicheng Zhang +2Multimodal Large Language Models VLM Reasoning
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory

Apr 17, 2026J/K move · Enter open · S save

Where Do Vision-Language Models Fail? World Scale Analysis for Image Geolocalization

Siddhant Bharadwaj, Ashish Vashist, Fahimul Aleem, Shruti Vyas

University of Central Florida

VLM Evaluation Large Vision-Language Models Geospatial Reasoning VLM Reasoning

Open paper AskPDF arXiv

Image geolocalization has traditionally been addressed through retrieval-based place recognition or geometry-based visual localization pipelines. Recent advances in Vision-Language Models (VLMs) have demonstrated strong zero-shot reasoning capabilities across multimodal tasks, yet their performance in geographic inference remains underexplored. In this work, we present a systematic evaluation of multiple state-of-the-art VLMs for country-level image geolocalization using ground-view imagery only. Instead of relying on image matching, GPS metadata, or task-specific training, we evaluate prompt-based country prediction in a zero-shot setting. The selected models are tested on three geographically diverse datasets to assess their robustness and generalization ability. Our results reveal substantial variation across models, highlighting the potential of semantic reasoning for coarse geolocalization and the limitations of current VLMs in capturing fine-grained geographic cues. This study provides the first focused comparison of modern VLMs for country-level geolocalization and establishes a foundation for future research at the intersection of multimodal reasoning and geographic understanding.

Abstract

Geospatial Reasoning Benchmarks

Explore similar work

Where Do Vision-Language Models Fail? World Scale Analysis for Image Geolocalization

ERGeoBench:A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models

Learning to Wander: Improving the Global Image Geolocation Ability of LMMs via Actionable Reasoning