GPSBench: Do Large Language Models Understand GPS Coordinates?

Thinh Hung Truong; Jey Han Lau; Jianzhong Qi

cs.AIFeb 18, 2026

GPSBench: Do Large Language Models Understand GPS Coordinates?

Authors: Thinh Hung Truong, Jey Han Lau, Jianzhong Qi

Organizations: University of Melbourne.

Abstract

Large Language Models (LLMs) are increasingly deployed in applications that interact with the physical world, such as navigation, robotics, or mapping, making robust geospatial reasoning a critical capability. Despite that, LLMs' ability to reason about GPS coordinates and real-world geography remains underexplored. We introduce GPSBench, a dataset of 57,800 samples across 17 tasks for evaluating geospatial reasoning in LLMs, spanning geometric coordinate operations (e.g., distance and bearing computation) and reasoning that integrates coordinates with world knowledge. Focusing on intrinsic model capabilities rather than tool use, we evaluate 14 state-of-the-art LLMs and find that GPS reasoning remains challenging, with substantial variation across tasks: models are generally more reliable at real-world geographic reasoning than at geometric computations. Geographic knowledge degrades hierarchically, with strong country-level performance but weak city-level localization, while robustness to coordinate noise suggests genuine coordinate understanding rather than memorization. We further show that GPS-coordinate augmentation can improve in downstream geospatial tasks, and that finetuning induces trade-offs between gains in geometric computation and degradation in world knowledge. Our dataset and reproducible code are available at https://github.com/joey234/gpsbench

Figures & tables

Track	Task	Sample	Type
Pure GPS	Format Conversion	Convert 32°3’9.0”S, 115°53’16.2”E to decimal $\rightarrow$ -32.0525, 115.8878	G
	Coordinate Transformation	Transform (19.37, 95.22) to Universal Transverse Mercator (UTM) $\rightarrow$ Zone 46N, 732668E, 2143080N	G
	Distance Calculation	Distance from (-2.13, -47.56) to (-22.63, -47.05)? $\rightarrow$ 2,280 km	G, S
	Bearing Computation	Bearing from (49.28, -123.13) to (5.89, 5.68)? $\rightarrow$ 55.2° (NE)	G, S
	Coordinate Interpolation	50% along (29.93, 117.95) to (-6.99, 106.55)? $\rightarrow$ (11.53, 111.86)	G, R
	Area & Perimeter	Area of polygon [(-2.85, 33.08), (21.15, 72.96), (29.35, 105.89)]? $\rightarrow$ 30,829 km²	G, S

Table 1: Summary of GPSBench tasks organized by spatial knowledge type following the landmark-route-survey framework. Types: L =Landmark, R =Route, S =Survey, G =Geometric. Samples are shorten for brevity.

Figure 1: Geographic coverage of GPSBench . 18,196 unique locations from GeoNames span six continents: Asia (33.7%), Europe (23.2%), North America (16.8%), South America (13.9%), Africa (9.4%), and Oceania (2.7%).

Figure 2: Applied vs Pure GPS Track performance. Most models cluster below the diagonal (favoring Applied), while GPT-5.1 and Gemini-2.5-Pro uniquely excel at Pure GPS computation.

Figure 3: Mean accuracy across all models per task, sorted by difficulty. Error bars show

\pm

1 standard deviation. Tasks cluster into solved (

>

95%), brittle (25–95%), and unsolved (

<

25%) tiers.

Figure 4: Regional performance by subregion across all GPSBench tasks. Applied Track shows the largest gaps: North America (75.8%) vs. East Asia (59.6%), a 16.2% difference. Pure GPS Track is more uniform across most subregions.

Figure 5: Hierarchical degradation in geographic knowledge. Models perform well at country-level but accuracy drops sharply for finer granularities, with city-level accuracy below 25% for all.

Figure 6: Effect of coordinate noise across granularity levels. All levels show flat performance regardless of noise magnitude (10m to 1km), suggesting models rely on generalized geographic knowledge rather than memorized coordinate-to-place mappings.

Figure 7: Effect of GPS coordinate augmentation on downstream tasks. MapEval shows +6.1% improvement; Hierarchical Spatial shows +22.7%, with GPS eliminating hierarchical and alignment biases.

Figure 8: Zero-shot vs. finetuned Qwen3-30B. Finetuning improves geometric computation but degrades world-knowledge tasks.

Figure 9: Scaling analysis for Mistral and Qwen3 model families. Both families show overall improvement with scale.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Endpoint / Model ID	Params	Type	Temp.	Provider	API
GPT-5.1	gpt-5.1	undisclosed	Reasoning	0	OpenAI	OpenAI
GPT-4.1	gpt-4.1	undisclosed	Non-reasoning	0	OpenAI	OpenAI
GPT-5-mini	gpt-5-mini-2025-08-07	undisclosed	Reasoning	0	OpenAI	OpenAI
GPT-4.1-mini	gpt-4.1-mini	undisclosed	Non-reasoning	0	OpenAI	OpenAI
GPT-5-nano	gpt-5-nano-2025-08-07	undisclosed	Reasoning	0	OpenAI	OpenAI
Gemini-2.5-Pro	gemini-2.5-pro	undisclosed	Reasoning	0	Google	Google AI

Appendix

Table 2: Model details for GPSBench evaluation. All models use temperature 0 for reproducibility. Parameter counts are estimates where not officially disclosed.

Model	Applied	Pure GPS	Gap
GPT-5.1	72.0	84.4	$-$ 12.4
GPT-5-mini	74.1	73.0	+1.1
GPT-5-nano	63.5	55.2	+8.2
GPT-4.1	73.3	60.9	+12.3
GPT-4.1-mini	70.5	56.6	+13.9
Gemini-2.5-Pro	71.7	76.7	$-$ 5.0

Appendix

Table 3: Overall model performance on GPSBench (% accuracy).

Model	Name	Route	Spatial	Bound.	Terrain	Prox.	Rel. Pos.	Place
GPT-5.1	97.7	81.7	93.7	98.1	72.6	73.3	58.5	2.5
GPT-5-mini	100.0	90.1	95.2	67.4	70.6	89.6	56.6	12.9
GPT-5-nano	99.9	84.8	59.4	98.5	67.4	33.1	52.6	1.3
GPT-4.1	100.0	88.6	94.2	45.2	71.3	84.4	64.8	16.8
GPT-4.1-mini	100.0	89.3	92.5	56.0	70.4	82.4	53.1	4.7
Gemini-2.5-Pro	99.9	91.7	95.4	21.5	69.2	86.4	62.2	23.0

Appendix

Table 4: Per-task accuracy (%) over Applied Track tasks. Bold indicates best per task.

Figure 10: Per-task accuracy (%) across all models. (a) Applied Track tasks requiring world knowledge integration. (b) Pure GPS Track tasks requiring geometric computation. Green indicates high accuracy, red indicates low accuracy.

Model	BBox	Format	Dist.	Bearing	Rel. Pos.	Route	Coord.	Interp.	Area
GPT-5.1	99.7	87.8	99.9	99.9	70.7	96.3	90.8	71.4	43.4
GPT-5-mini	97.5	88.0	99.9	99.8	46.6	91.8	79.4	54.1	0.0
GPT-5-nano	97.5	88.0	80.4	67.3	65.5	26.4	54.7	0.3	17.1
GPT-4.1	100.0	88.1	99.1	97.5	47.5	46.0	59.0	11.4	0.0
GPT-4.1-mini	99.9	88.1	94.8	90.9	51.0	59.0	16.3	9.1	0.0
Gemini-2.5-Pro	100.0	88.1	99.9	99.2	69.7	43.6	84.2	52.1	53.9

Appendix

Table 5: Per-task accuracy (%) over Pure GPS Track tasks. Bold indicates best per task.

Task	Dominant Error Pattern
Place Association	Nearby city substitution
Polygon Area	Formula stated but not executed
Coord. Interpolation	Cumulative numerical errors
Coord. Transformation	Nonlinear projection arithmetic
Bearing/Distance	Computational approximations
Spatial Patterns	Heuristic-based reasoning

Appendix

Table 6: Dominant error patterns by task category.

Applied Track
Task	N.Am	S.Am	W.Eur	E.Eur	E.Asia	S.Asia	SE.Asia	M.East	Africa	Oceania
Place Assoc.	14.7	6.7	11.0	8.3	3.8	2.6	5.9	5.0	4.5	8.0
Name Disamb.	99.1	99.5	99.1	98.8	98.8	97.1	99.3	–	99.4	100.0
Rel. Position	48.1	55.4	53.7	59.7	44.6	56.5	53.1	53.4	55.8	56.5
Proximity	61.4	63.6	69.6	70.6	63.7	65.6	63.2	68.5	60.8	59.2
Route Analysis	85.8	81.8	91.4	88.6	80.9	86.2	84.3	88.8	88.2	84.0

Appendix

Table 7: Accuracy (%) by subregion across all GPSBench tasks. Bold indicates highest accuracy per task; ‘–’ indicates insufficient data.

Figure 11: Regional performance across all GPSBench tasks by subregion. Applied Track tasks (top) show varying degrees of geographic bias, with Place Association exhibiting the strongest disparity. Pure GPS Track tasks (bottom) show minimal regional variation, confirming that geometric computation is largely region-agnostic.

Model	City	Province	Country	Region
GPT-5.1	12.9	61.2	81.1	0.4
GPT-5-mini	2.5	44.4	77.1	0.3
GPT-5-nano	1.3	32.2	71.9	0.2
GPT-4.1	16.8	61.7	85.4	1.4
GPT-4.1-mini	4.7	49.1	82.1	2.4
Gemini-2.5-Pro	23.0	72.9	96.5	40.0

Appendix

Table 8: Place Association accuracy (%) by geographic granularity level across all 14 models.

Noise	Country	Province	City
Clean	80.7	46.3	7.4
10m	79.1 ( $-$ 1.6)	51.8 (+5.5)	5.8 ( $-$ 1.6)
50m	80.1 ( $-$ 0.5)	52.1 (+5.8)	9.3 (+1.9)
100m	82.3 (+1.6)	45.8 ( $-$ 0.5)	7.1 ( $-$ 0.3)
500m	82.1 (+1.5)	49.7 (+3.4)	7.0 ( $-$ 0.5)
1km	81.0 (+0.4)	48.3 (+2.0)	7.0 ( $-$ 0.4)

Appendix

Table 9: Mean accuracy (%) by noise level and granularity. All levels show stable performance across noise magnitudes, suggesting generalized knowledge rather than memorization.

Model	Clean (%)	$\Delta$ 10m	$\Delta$ 50m	$\Delta$ 100m	$\Delta$ 500m	$\Delta$ 1km
GPT-5.1	13.1	$-$ 2.2	+3.7	$-$ 2.0	$-$ 0.4	$-$ 0.5
GPT-5-mini	1.7	$-$ 1.1	+0.1	+2.2	+2.5	+0.8
GPT-5-nano	1.1	$-$ 1.1	+1.9	+0.2	$-$ 1.1	+0.9
GPT-4.1	16.0	$-$ 2.7	+5.7	$-$ 1.5	+0.4	+2.3
GPT-4.1-mini	5.7	$-$ 3.9	+0.9	+0.9	$-$ 0.9	$-$ 2.7
Gemini-2.5-Pro	24.0	$-$ 1.0	+4.9	$-$ 3.6	$-$ 3.4	$-$ 2.7

Appendix

Table 10: Place association accuracy change (

\Delta

%) relative to clean coordinates across all 14 models. Values show difference from clean baseline. Note: no correlation between noise magnitude and

|\Delta|

(mean

|\Delta|

= 1.3–2.1% across all levels).

Dataset	Category	N	Base	+GPS	$\Delta$
MapEval	Trip planning	11	90.9	100.0	+9.1
	POI queries	51	72.5	76.5	+3.9
	Nearby search	3	66.7	100.0	+33.3
	Overall	66	75.8	81.8	+6.1
Hierarchical Spatial	Hierarchical bias	10	70.0	100.0	+30.0
	Alignment bias	4	50.0	100.0	+50.0

Appendix

Table 11: Effect of GPS augmentation on downstream tasks (%). Results using GPT-4.1.

Track	Zero-shot	Finetuned	$\Delta$
Applied	52.3%	50.7%	$-$ 1.6%
Pure GPS	53.1%	57.4%	+4.3%
Combined	52.7%	54.1%	+1.5%

Appendix

Table 12: Overall finetuning results on Qwen3-30B.

Applied Track
Task	Zero-shot	Finetuned	$\Delta$
Place Association	0.8	1.7	+0.9
Name Disambiguation	99.7	82.6	$-$ 17.1
Relative Position	42.5	22.9	$-$ 19.6
Proximity	24.2	15.4	$-$ 8.8
Route Analysis	85.7	89.9	+4.2

Appendix

Table 13: Per-task finetuning comparison on Qwen3-30B (%). Green: improvement (

>

5%), red: degradation (

<

-

5%).

Geospatial Reasoning

Geospatial reasoning studies how AI interprets locations, maps, spatial relationships, and Earth-observation data to answer questions and make decisions.

See all 76 papers

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning
Aug 4, 2026Martin Böckling, Elizaveta Nosova, Heiko Paulheim +1Spatial Reasoning Multilingual QA
Data and Web Science Group, University of Mannheim, Germany
GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks
Aug 7, 2026Rodrigo Ferreira Rodrigues, Karim Radouane, Jose G Moreno +1Spatial Reasoning Benchmarks LLM Evaluation
University of Toulouse Toulouse, France
GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks
Oct 8, 2026Amirhossein Vahdat, Thierry Badard, Luc LamontagneLLM Evaluation Geospatial Reasoning Benchmarks
Département des Sciences Géomatiques, Université Laval, QC, Canada · Département d’informatique et de génie logiciel, Université Laval, QC, Canada

Aug 4, 2026J/K move · Enter open · S save

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

Martin Böckling, Elizaveta Nosova, Heiko Paulheim, Andreea Iana

Data and Web Science Group, University of Mannheim, Germany

Spatial Reasoning Multilingual QA Multilingual Language Model Evaluation Spatial Reasoning Benchmarks Geospatial Reasoning Benchmarks Jurisdictional Bias in Language Models+3

Open paper AskPDF arXiv

Geospatial reasoning, i.e., computing distances, containment, and other spatial relations over real-world entities, is central to navigation and logistics, yet large language models (LLMs) struggle with the required geometric and topological computation despite storing considerable geographic knowledge. Existing benchmarks localize these failures only partially: they are synthetic or smallscale, largely monolingual, and offer limited control over geographic coverage. We introduce MultiGlobeQA, a multilingual benchmark of 46,060 question-answer pairs spanning 14 spatial-function families and 15 answer formats, with execution-based ground truth over three knowledge graphs. It covers 201 countries and territories via income- and density-stratified sampling, with parallel questions in English and 16 additional high- and low-resource languages. Across parametric, reasoning, and agentic settings, LLMs collapse on tasks requiring grid indexing and shape computation, while topological relations and directions fare best. Retrieval and tool use yield considerable gains, yet performance plateaus below two thirds even when gold facts are supplied, indicating that computation, not access to knowledge, is the bottleneck. Models also underperform on low-income regions, a gap that gold facts widen rather than close.

Abstract

Geospatial Reasoning

Explore similar work

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks

GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks