EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration

Jianfei Wu; Zhichun Wang; Zhensheng Wang; Zhiyu He

cs.AIApr 8, 2026

EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration

Authors: Jianfei Wu, Zhichun Wang, Zhensheng Wang, Zhiyu He

Organizations: School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China · Beijing Key Laboratory of Artificial Intelligence for Education, Beijing 100875, China · Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education, Beijing 100875, China · College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China

Abstract

While Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, their potential for purpose-driven exploration in dynamic geo-spatial environments remains under-investigated. Existing Geo-Spatial Question Answering (GSQA) benchmarks predominantly focus on static retrieval, failing to capture the complexity of real-world planning that involves dynamic user locations and compound constraints. To bridge this gap, we introduce EVGeoQA, a novel benchmark built upon Electric Vehicle (EV) charging scenarios that features a distinct location-anchored and dual-objective design. Specifically, each query in EVGeoQA is explicitly bound to a user's real-time coordinate and integrates the dual objectives of a charging necessity and a co-located activity preference. To systematically assess models in such complex settings, we further propose GeoRover, a general evaluation framework based on a tool-augmented agent architecture to evaluate the LLMs' capacity for dynamic, multi-objective exploration. Our experiments reveal that while LLMs successfully utilize tools to address sub-tasks, they struggle with long-range spatial exploration. Notably, we observe an emergent capability: LLMs can summarize historical exploration trajectories to enhance exploration efficiency. These findings establish EVGeoQA as a challenging testbed for future geo-spatial intelligence. The dataset and prompts are available at https://github.com/kg-bnu/EVGeoQA.

Figures & tables

Figure 1: An illustrative EVGeoQA query. Identifying the optimal target requires combining semantic understanding with real-time location and POI information.

City	Hangzhou	Qingdao	Linyi
Stations	258	165	157
Locations	997	995	997
POIs	25	23	21
QA Pairs	19940	14162	14416

Table 1: Statistics of the EVGeoQA Dataset.

Figure 2: (a) Distribution of query-anchored locations in Qingdao, reflecting the concentration within densely populated regions and along road networks. (b) Distribution of contextual POIs surrounding charging stations in Qingdao, showcasing categorical diversity and non-uniform spatial coverage.

Figure 3: GeoRover Framework Overview. The agent leverages interactive tools to explore the geo-spatial environment, synthesizing the tool invocation trajectory to locate a station that satisfies both charging and activity demands.

Distance	<10km			<20km			No Limit
Model	Hits@1	Hits@2	Hits@3	Hits@1	Hits@2	Hits@3	Hits@1	Hits@2	Hits@3
Qwen3-8B	0.3663	0.4916	0.5381	0.2469	0.3421	0.3867	0.2061	0.2889	0.3295
Qwen3-8B*	0.3965	0.4909	0.5011	0.3076	0.3972	0.4513	0.2637	0.3452	0.4375
Qwen3-30B-a3b	0.3445	0.5251	0.5810	0.2718	0.3717	0.4012	0.2339	0.2942	0.3133
Qwen3-30B-a3b*	0.3906	0.5396	0.5667	0.2804	0.3979	0.4285	0.2283	0.3215	0.3418
Qwen25-72b	0.4833	0.6379	0.7073	0.3827	0.5479	0.6037	0.3333	0.4878	0.5614

Table 2: Experimental results on the EVGeoQA dataset, with the best results in bold . LLMs employing the Thinking mechanism are marked with *.

Distance	<10km	<20km	No Limit
Qwen3-8b	0.35	1.36	2.12
Qwen3-8b*	0.79	2.11	4.03
Qwen3-30b-a3b	0.37	1.13	3.26
Qwen3-30b-a3b*	0.42	1.72	3.74
Gemini-2.5-Flash	0.40	2.77	3.91
Gemini-2.5-Pro*	1.12	3.31	5.62

Table 3: Average tool invocation frequency of the ChangeLocation Tool across different distance tiers in Linyi.

Figure 4: Distribution of Error Causes in Linyi

Figure 5: Case Study: Visualization of a multi-step exploration trajectory by Gemini-2.5-Pro* in Qingdao.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Illustration of the Multi-Source Fusion strategy for user location synthesis.

Figure 7: Spatial distribution of charging stations across the three representative cities. Visualizations are zoomed into city centers for clarity. The full dataset is available in our code repository.

Geospatial QA

See all 17 papers

Aug 7, 2026

GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks

In the context of geodata, existing Large Language Models have often been studied in a homogeneous setting, which has considerably limited insights into their generalization capabilities. In this paper, we present \benchName, a comprehensive benchmark for probing LLMs on geo-related tasks. We leverage a careful selection of twelve publicly available datasets from diverse geo-related tasks and domains, and evaluate a set of LLMs on geo-spatial and temporal understanding using our benchmark. Our results show that reasoning and size have a strong impact on overall performance. GeoBenchLLM is publicly available at https://github.com/Rfr2003/GeoBenchLLM.

Rodrigo Ferreira Rodrigues, Karim Radouane, Jose G Moreno +1

University of Toulouse Toulouse, France

Spatial Reasoning Benchmarks LLM Evaluation Temporal Reasoning in Language Models Geospatial Reasoning Language Modeling Geospatial Reasoning Benchmarks

Ask PDF

Oct 8, 2026

GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks

Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements. This omission is critical as retrieval oriented geo-analytic tasks cannot be supported reliably without first identifying the necessary datasets and constraints. GeoFitBench-LLM is introduced as a benchmark for assessing whether LLMs can infer the datasets, the supporting resources and the constraints required by geo-analytic requests in natural language. The benchmark is structured around two reasoning dimensions, Spatio-Temporal Relational and Semantic Grounding, reflecting long-standing challenges in geospatial information retrieval and related natural language tasks. Ground truth requirements are encoded in a fixed, provider-independent schema for reproducible evaluation of resource identification and attribute-level specification. Experiments on seven models under minimal prompting indicate that identifying the necessary resources is often achievable, with GPT-5.4 Thinking achieving the best result by recovering the complete set of resources in 66.67% of benchmark requests. Results further indicate that accurate fitness-for-use specification remains more difficult as inferential demands increase.

Amirhossein Vahdat, Thierry Badard, Luc Lamontagne

Département des Sciences Géomatiques, Université Laval, QC, Canada · Département d’informatique et de génie logiciel, Université Laval, QC, Canada

LLM Evaluation Geospatial Reasoning Benchmarks Geospatial Reasoning LLM Grounding Requirements Engineering Spatial Reasoning Benchmarks

Ask PDF

Aug 4, 2026

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

Geospatial reasoning, i.e., computing distances, containment, and other spatial relations over real-world entities, is central to navigation and logistics, yet large language models (LLMs) struggle with the required geometric and topological computation despite storing considerable geographic knowledge. Existing benchmarks localize these failures only partially: they are synthetic or smallscale, largely monolingual, and offer limited control over geographic coverage. We introduce MultiGlobeQA, a multilingual benchmark of 46,060 question-answer pairs spanning 14 spatial-function families and 15 answer formats, with execution-based ground truth over three knowledge graphs. It covers 201 countries and territories via income- and density-stratified sampling, with parallel questions in English and 16 additional high- and low-resource languages. Across parametric, reasoning, and agentic settings, LLMs collapse on tasks requiring grid indexing and shape computation, while topological relations and directions fare best. Retrieval and tool use yield considerable gains, yet performance plateaus below two thirds even when gold facts are supplied, indicating that computation, not access to knowledge, is the bottleneck. Models also underperform on low-income regions, a gap that gold facts widen rather than close.

Martin Böckling, Elizaveta Nosova, Heiko Paulheim +1

Data and Web Science Group, University of Mannheim, Germany

Spatial Reasoning Multilingual QA Multilingual Language Model Evaluation Spatial Reasoning Benchmarks Geospatial Reasoning Benchmarks Jurisdictional Bias in Language Models+3

Ask PDF

Abstract

Geospatial QA

Explore similar work

GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks

GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning