GeoAgentBench: A Dynamic Execution Benchmark for Tool-Augmented Agents in Spatial Analysis

Bo Yu; Cheng Yang; Dongyang Hou; Chengfu Liu; Jiayao Liu; Chi Wang; Zhiming Zhang; Haifeng Li

cs.AIApr 15, 2026

GeoAgentBench: A Dynamic Execution Benchmark for Tool-Augmented Agents in Spatial Analysis

Authors: Bo Yu, Cheng Yang, Dongyang Hou, Chengfu Liu, Jiayao Liu, Chi Wang, Zhiming Zhang, Haifeng Li,

Organizations: School of Geosciences and Info-Physics, Central South University, Changsha, China. · School of Earth Sciences and Spatial Information Engineering, Hunan University of Science and Technology, Xiangtan, China.

Abstract

The integration of Large Language Models (LLMs) into Geographic Information Systems (GIS) marks a paradigm shift toward autonomous spatial analysis. However, evaluating these LLM-based agents remains challenging due to the complex, multi-step nature of geospatial workflows. Existing benchmarks primarily rely on static text or code matching, neglecting dynamic runtime feedback and the multimodal nature of spatial outputs. To address this gap, we introduce GeoAgentBench (GABench), a dynamic and interactive evaluation benchmark tailored for tool-augmented GIS agents. GABench provides a realistic execution sandbox integrating 117 atomic GIS tools, encompassing 53 typical spatial analysis tasks across 6 core GIS domains. Recognizing that precise parameter configuration is the primary determinant of execution success in dynamic GIS environments, we designed the Parameter Execution Accuracy (PEA) metric, which utilizes a "Last-Attempt Alignment" strategy to quantify the fidelity of implicit parameter inference. Complementing this, a Vision-Language Model (VLM) based verification is proposed to assess data-spatial accuracy and cartographic style adherence. Furthermore, to address the frequent task failures caused by parameter misalignments and runtime anomalies, we developed a novel agent architecture, Plan-and-React, that mimics expert cognitive workflows by decoupling global orchestration from step-wise reactive execution. Extensive experiments with seven representative LLMs demonstrate that the Plan-and-React paradigm significantly outperforms traditional frameworks, achieving the optimal balance between logical rigor and execution robustness, particularly in multi-step reasoning and error recovery. Our findings highlight current capability boundaries and establish a robust standard for assessing and advancing the next generation of autonomous GeoAI.

Figures & tables

	General Benchmarks 1	GIS Code Gen 2	GIS Planning Gen 3	GeoPlan-Bench ( Li et al., 2025b )	GABench
Domain Focus	General API	Geospatial	Geospatial	Geospatial	Geospatial
Interaction	Tool-Aug.	Code Gen	Text Plan	Tool-Aug.	Tool-Aug.
Environment	API Calling	$\times$	$\times$	Mock Tools	Interactive Sandbox
Spatial Logic	Missing	Moderate	Low	Low	High
Core Metrics	API Acc.	Similarity	Trajectory	Trajectory	Trajectory & VLM
Error Recovery	Partial	Limited	None	None	Self-correction

Table 1: Comparison between GABench and existing general and geospatial-specific agent benchmarks.

Figure 1: Overview of the GeoAgentBench (GABench) framework compared with existing paradigms. The upper panel illustrates the limitations of traditional non-interactive benchmarks: LLMs generate code or plans that are evaluated via lexical similarity (Textual Ground Truth) but frequently encounter Execution Errors (e.g., projection mismatches, topology anomalies) in real GIS environments, leaving users with unverified and often unusable scripts. The lower panel showcases GABench’s dynamic, closed-loop execution paradigm. Our framework integrates a library of 117 atomic GIS tools within an interactive sandbox, enabling agents to perceive real-time Execution Feedback (Error Messages) and iteratively refine their trajectories. Evaluation is performed across two complementary tiers: (1) Step-by-Step metrics (TAO, TIO, TEM, and the innovative PEA) that verify the logical consistency of tool-invocation actions; and (2) End-to-End multimodal verification using a VLM Evaluator, which compares the final Result Map against the GT Map to ensure both data-spatial accuracy and cartographic quality, delivering a truly verified geospatial product.

Figure 2: Overview of the GABench dataset construction and verification workflow. The left panel outlines task sourcing (53 total tasks) and the hierarchical taxonomy across six core GIS domains. The right panel depicts the iterative structural refactoring and verification process: researchers first develop 117 atomic GIS tools and re-engineer the original analytical logic into standardized, modular tool-flows. A closed-loop consistency check is then performed by matching tool-flow outputs against original code and experimental results; any discrepancies trigger iterative refinement by the researchers (indicated by the red arrow). Finally, an expert review ensures logical integrity and functional redundancy screening, certifying the verified tool-flows as the physical ground truth for the GABench dataset.

Field	Description
ID	Unique identifier for each spatial analysis task.
Domain	The specific GIS domain the task belongs to (e.g., Vector Spatial Analysis, Hydrological Analysis).
Task Description	Natural language instruction defining the analysis objective and requirements.
Data Description	Standardized file paths and metadata for the multi-source input datasets.
Drawing Style	Specific cartographic requirements, including color ramps, layer order etc.
Toolchain Length	The total number of atomic GIS tool invocations in the gold-standard workflow.

Table 2: Description of the metadata schema for GABench tasks.

Figure 3: The Plan-and-React baseline agent framework adopts a design that decouples global workflow orchestration from step-wise reactive execution. The Global Task Planner is responsible for decomposing abstract geospatial problems into logically self-consistent sequences of steps. Meanwhile, the Step-wise Reactive Executor implements tool invocation, dynamic parameter inference, and runtime error self-recovery through localized "Thought-Action-Observation" loops. This architecture mimics the cognitive paradigm of GIS experts, ensuring structural integrity while maintaining tactical flexibility in long-chain spatial analysis workflows.

Model	TAO			TIO	TEM	PEA	VLM 1	$\overline{Eff}_{ma}$	$\overline{Eff}_{mi}$
Model	R	P	F1
Qwen2.5-7B	17.94	34.91	22.01	13.56	13.56	12.42	8.1 $\pm$ 0.6	95.6	92.8
Llama-3.1-8B	49.91	66.75	52.12	38.35	29.58	24.83	16.8 $\pm$ 1.3	72.2	56.9
GPT-4o-mini	68.51	71.57	68.27	51.63	29.74	35.39	45.8 $\pm$ 1.4	85.5	81.1
GPT-4o	67.04	72.98	68.95	51.51	45.80	32.54	51.0 $\pm$ 0.4	98.1	97.4
DeepSeek-V3	82.20	81.79	79.78	62.55	50.80	40.41	63.2 $\pm$ 1.2	92.0	86.6

Table 3: Performance evaluation of LLMs under the base agent paradigm.

Model	TAO			TIO	TEM	PEA	VLM 1	$\overline{Eff}_{ma}$	$\overline{Eff}_{mi}$
Model	R	P	F1
Qwen2.5-7B	26.65	38.05	28.52	21.03	16.31	15.28	13.6 $\pm$ 0.9	85.8	66.2
Llama-3.1-8B	42.53	64.16	47.64	35.41	30.86	25.85	7.7 $\pm$ 0.2	72.2	65.2
GPT-4o-mini	67.15	67.54	65.71	46.02	31.08	32.70	36.9 $\pm$ 1.0	84.0	72.4
GPT-4o	80.37	85.81	80.65	61.65	47.33	43.71	55.6 $\pm$ 0.8	85.4	82.6
DeepSeek-V3	78.58	81.74	78.16	59.63	51.36	46.74	65.1 $\pm$ 0.9	90.0	80.8

Table 4: Performance evaluation of LLMs under the ReAct paradigm.

Model	TAO			TIO	TEM	PEA	VLM 1	$\overline{Eff}_{ma}$	$\overline{Eff}_{mi}$
Model	R	P	F1
Qwen2.5-7B	29.53	35.79	31.27	25.01	21.64	8.24	1.7 $\pm$ 0.2	100	100
Llama-3.1-8B	49.03	47.67	47.20	37.23	28.67	24.99	1.6 $\pm$ 0.2	100	100
GPT-4o-mini	68.28	75.31	70.75	49.50	46.23	30.74	3.9 $\pm$ 0.9	100	100
GPT-4o	81.58	87.18	83.63	61.56	58.18	36.95	3.7 $\pm$ 0.2	100	100
DeepSeek-V3	76.22	79.43	76.72	60.01	51.99	37.82	3.8 $\pm$ 0	99.0	97.9

Table 5: Performance evaluation of LLMs under the Plan-and-Solve paradigm.

Model	TAO			TIO	TEM	PEA	VLM 1	$\overline{Eff}_{ma}$	$\overline{Eff}_{mi}$
Model	R	P	F1
Qwen2.5-7B	41.86	54.03	44.79	26.49	16.14	16.90	20.1 $\pm$ 0.9	86.9	62.8
Llama-3.1-8B	44.26	59.34	47.11	34.07	18.93	21.59	10.1 $\pm$ 0.5	68.2	42.4
GPT-4o-mini	74.08	71.00	70.95	57.80	41.10	36.15	43.4 $\pm$ 1.7	82.6	72.2
GPT-4o	79.40	83.70	79.55	61.38	38.58	40.69	59.1 $\pm$ 1.1	81.4	79.3
DeepSeek-V3	83.79	82.20	80.64	64.48	51.16	47.34	68.5 $\pm$ 1.5	89.5	86.2

Table 6: Performance evaluation of LLMs under the Plan-and-React framework.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

Aug 3, 2026

GISAgentBench: A Practitioner-Sourced Benchmark for Evaluating LLM Agents on GIS Tasks

Geographic Information System (GIS) professionals rely on multi-step spatial analysis workflows to support decision-making in urban planning, disaster response, and environmental monitoring. The process is tedious, time-consuming, and error-prone. While recent large language model (LLM) agents equipped with external tools have the potential to automate geospatial analysis, their ability to perform realistic GIS workflows remains largely unexplored. Existing GIS agent benchmarking datasets are mostly drawn from textbooks, tutorials, or LLM-generated seeds and remain limited in size and trajectory depth. More importantly, none provides ground truth outputs. They therefore rely on surrogate signals such as code similarity, trajectory matching, or LLM and VLM judges, which can conflate workflow resemblance with task correctness. To address this gap, we introduce GISAgentBench, a benchmark of 349 multi-step GIS tasks curated from GIS Stack Exchange and instantiated on real public data across six selected geographic areas of interest. Each task ships with an executable reference trajectory and an exact ground truth output file, enabling strict, deterministic, tolerance-aware output matching beyond LLM judging. Evaluations of six LLM models reveal that realistic GIS workflows remain challenging: the best agent completes only 32.7% of tasks under strict tolerance-aware scoring, although most models produce outputs that are close to the ground truth.

Abhinav Pothuri, Zhe Jiang, Zelin Xu +1

Department of Computer & Information Science & Engineering, University of Florida · Department of Geography, University of Florida

LLM Agent Evaluation AI Agent Benchmarks Spatial Reasoning Geospatial Reasoning Benchmarks

Ask PDF

Sep 13, 2026

ANASSA: An Agentic AI Orchestration Framework for Spatial Intelligence

The emergence of large language models (LLMs) and large multimodal models (LMMs) has enabled a new class of agentic systems capable of integrating natural language understanding with tool-based execution. In geographic information systems (GIS), this shift is transforming traditional, expert-driven workflows into semiautonomous systems that can interpret user intent, construct spatial workflows, and execute geospatial analysis tasks. However, existing approaches remain limited by fragmented integration of reasoning, execution, and evaluation, particularly in complex, real-world environments. This study synthesizes recent advances in agentic GIS frameworks, benchmarks, and surveys to identify limitations in spatial reasoning, execution robustness, validation, governance, and evaluation. Building on these insights, it introduces ANASSA (Autonomous Neural Agents for Spatial Systems Architecture), an agentic AI orchestration framework that integrates structured spatial reasoning, multi-agent workflow orchestration, execution feedback, authoritative spatial validation, provenance, uncertainty handling, and human decision authority within a unified system design. The contribution is an architecture-level specification: eleven components across four layers, a six-step Geospatial AI Cognitive Loop, cross-component contracts, and governance mechanisms intended to make agentic geospatial workflows traceable, reproducible, and accountable. Empirical performance evaluation is reserved for implementation and deployment studies.

Constantinos Papantoniou, Brian Hilton

Anassa.ai USA · Research Scientist USA

LLM Agent Orchestration Geospatial Reasoning AI Agent Governance

Ask PDF

Oct 6, 2026

GeoNatureAgent (GNA): A Framework and Benchmark for Pre-Production Evaluation of Tool-Using Agents on Geospatial and Environmental Tasks

Before tool-using LLM agents are deployed in environmental and geospatial workflows, teams need evidence that an agent reliably selects the right operations against real APIs. We introduce GeoNatureAgent (GNA), a framework for pre-production evaluation of tool-using agents: a fixed sixteen-tool geospatial interface published as a Model Context Protocol (MCP) server, so the agent under test is the only variable, scored against an identical tool layer, task suite, and deterministic scorer. Its flagship instance is a 103-task benchmark (a 93-task main suite across 18 categories plus a ten-task comparison expansion) evaluated against an open, self-hostable geospatial API serving three environmental indicators across Spain and Portugal. We evaluate nine LLMs under three temperature-1.0 seeds, reporting capability and per-case cost as orthogonal axes. (1) Claude Sonnet 4 achieves the highest capability (61.7% +/- 0.7% on all 103 tasks; 60.8% on the main suite), followed closely by DeepSeek V3.2 (57.9%), while no other model exceeds 53%; (2) the cost-accuracy Pareto frontier is mostly open-weight, with DeepSeek V3.2 offering 93% of Claude's capability at 11.3x lower list-price cost; (3) under strict all-checks scoring the best model sits 24-36 points below the 85-97% reported on general-purpose GIS benchmarks, whereas per-check partial credit for the top four models (86-90%) is comparable, so much of that gap reflects scoring strictness rather than task difficulty alone. The MCP server, evaluation harness, benchmark, and API are publicly available; swapping the tool executors and task suite instantiates an equivalent benchmark for any geospatial domain.

Gabriel Diaz-Ireland, Diego Prieto-Herráez, Mario García Peces +3

Universidad Católica de Ávila (UCAV) Ávila, Spain · Independent Researcher Madrid, Spain · Department of Earth and Planetary Sciences, Johns Hopkins University Baltimore, MD, USA +1

Tool-Use Evaluation LLM Agent Evaluation AI Agent Benchmarks Geospatial Reasoning Benchmarks Geospatial Reasoning

Ask PDF

Abstract

LLM Agent Evaluation

Explore similar work

GISAgentBench: A Practitioner-Sourced Benchmark for Evaluating LLM Agents on GIS Tasks

ANASSA: An Agentic AI Orchestration Framework for Spatial Intelligence

GeoNatureAgent (GNA): A Framework and Benchmark for Pre-Production Evaluation of Tool-Using Agents on Geospatial and Environmental Tasks