GISclaw: A Comprehensive Open-Source LLM Agent System for Realistic Multi-Step Geospatial Analysis

Jinzhen Han; JinByeong Lee; Yuri Shim; Jisung Kim; Jae-Joon Lee

cs.SEMar 27, 2026

GISclaw: A Comprehensive Open-Source LLM Agent System for Realistic Multi-Step Geospatial Analysis

Authors: Jinzhen Han, JinByeong Lee, Yuri Shim, Jisung Kim, Jae-Joon Lee

Organizations: Department of Civil, Architectural & Environment Engineering, Sungkyunkwan University, Suwon, South Korea · Ministry of the Interior and Safety, South Korea · School of Geography, University of Leeds, United Kingdom · Department of Fire Safety Engineering, Jeonju University, Jeonju-si, Republic of Korea

Abstract

Most LLM-driven GIS assistants solve narrow single-step tasks tightly coupled to proprietary platforms such as ArcGIS or QGIS, limiting their use for the multi-step, cross-format pipelines that define professional geospatial analysis. We present GISclaw, a comprehensive open-source agent system that performs realistic GIS analysis end to end - spatial joins, raster algebra, kriging interpolation, machine-learning classification, network analysis, choropleth cartography - directly through Python with no commercial GIS dependency. GISclaw couples an LLM reasoning core with a persistent Python sandbox pre-loaded with the open-source geospatial stack, three engineered prompt rules (Schema Analysis, Package Constraint, Domain Knowledge Injection), and an Error-Memory module for self-correction. A single backend-agnostic architecture supports both cloud-API and locally deployed open-weight LLM backends, enabling air-gapped deployment without loss of capability. On GeoAnalystBench - 50 expert-curated multi-step tasks averaging 5.8 analytical steps across vector, raster, and tabular data - GISclaw reaches up to 100% task success and 97% mean success over three independent runs. We further conduct 1,800 controlled experiments (50 tasks x 6 backends x 2 architectures x 3 repeats) with bootstrap 95% CIs, paired Wilcoxon tests, and a composite-score sensitivity analysis (Kendall's tau median = 0.94), and introduce a three-layer evaluation protocol combining code structure, reasoning process, and type-specific output verification. The Single-Agent ReAct loop reliably outperforms the Dual-Agent Plan-Execute-Replan pipeline on every cloud backend (Cliff's delta = 0.15-0.41); only the locally deployed 14B model gains from multi-agent orchestration, suggesting architectural complexity should match model capability rather than be added by default.

Figures & tables

System	Data Types	Models	Architectures	Open-Source	Tasks	Best Success
LLM-Geo ( Li and Ning, 2023 )	Vector	1 (GPT-4)	Single	✓	Case study	$\sim$ 80%
GIS Copilot ( Akinboyewa et al., 2025 )	Vector+Raster	2 (GPT-4/3.5)	Single	✓	110	86.4%
GeoGPT ( Zhang et al., 2023 )	Vector	1 (GPT-3.5)	Single	No	Case study	—
GeoJSON Agents ( Luo et al., 2026 )	GeoJSON	Multiple	Multi (2-role)	No	70	97.1%
GeoColab ( Wu et al., 2025 )	Vector	7	Multi (3-role)	✓	50	+26.1%
GTChain ( Zhang et al., 2025b )	Vector+Raster	2 (LLaMA/GPT-4)	Single (Fine-tuned)	✓	Custom	+32.5% ‡

Table 1: Comparison of GISclaw with existing LLM-powered GIS agent systems.

Figure 1: Overview of the GISclaw system architecture. The system accepts natural-language tasks with GIS data (vector, raster, tabular) as input, routes them through a pluggable LLM backend, and executes analysis in a persistent Python sandbox. Two agent architectures—Single Agent (ReAct) and Dual Agent (Plan-Execute-Replan)—are supported, with outputs evaluated via a three-layer protocol.

Category	Libraries	Capabilities
Vector	GeoPandas, shapely, fiona, pyproj	Spatial joins, buffer, overlay, CRS transforms
Raster	rasterio, xarray, rasterstats	Band algebra, interpolation, zonal statistics
Analysis	scipy, scikit-learn, libpysal	Kriging, clustering, regression, spatial autocorrelation
Visualization	matplotlib, seaborn, geoplot, cartopy	Choropleth, heatmaps, multi-panel cartography

Table 2: Pre-loaded open-source library stack organized by analytical capability. This ecosystem replaces the need for proprietary GIS platforms.

Feature	Design Rationale
State persistence	Variables and imports survive across rounds, enabling incremental exploratory analysis
Asymmetric truncation	stdout: front-truncated (latest results); stderr: tail-truncated (root-cause error)
Variable tracking	New variables logged per round, providing the agent with execution context
10-min timeout	Prevents runaway computations ( e.g. , large-scale Kriging on consumer GPUs)

Table 3: Key design features of the persistent Python sandbox.

Figure 2: Comparison of the two agent architectures. (a) Single Agent follows a ReAct loop with Error Memory for self-correction. (b) Dual Agent decomposes tasks via a Planner, executes steps through a Worker, and adaptively replans upon failure.

Category	$N$	Representative Task
Understanding spatial distributions	19	T1: Urban heat island & elderly risk mapping via Kriging interpolation
Making predictions	8	T20: Mineral deposit prediction using Random Forest with raster features
Detecting patterns	7	T42: Airbnb price clustering via local Moran’s I spatial autocorrelation
Measuring shape & distribution	7	T36: Vegetation change detection using SAVI spectral index
Determining spatial relationships	6	T38: Travel-time isochrone computation on road networks
Optimal locations & paths	3	T32: Wildlife corridor optimization via weighted cost-surface overlay

Table 4: GeoAnalystBench task categories with representative examples. Each task includes natural-language instructions, heterogeneous input data, optional domain knowledge, and expert-authored gold-standard code.

Cloud API (pay-per-use)
Model	Backend	Cost ($/M)	Context
GPT-5.4	OpenAI	2.50 / 15.00	1M
GPT-4.1	OpenAI	2.00 / 8.00	1M
DeepSeek-V3.2	DeepSeek	0.28 / 0.42	128K
Gemini-3-Flash	Google AI	0.50 / 3.00	1M
Local deployment (zero marginal cost)

Table 5: LLM backends evaluated. Cost is per million tokens (input/output). Open-weight models are deployed on local servers at zero marginal cost.

Panel of Figure 3 — Figure 3: Multi-layer evaluation pipeline. Three complementary layers assess code-level fidelity (L1), reasoning process quality (L2), and output correctness (L3), combined into a weighted composite score $S_{\text{comp}}$ (Eq. 1 ).

	Single Agent						Dual Agent
Model	$R_{s}$	$Q_{\text{out}}$	$F_{\text{api}}$	$C_{\text{emb}}$	$\boldsymbol{S_{\textbf{comp}}}\ (\pm\textbf{CI})$	Cost	$R_{s}$	$Q_{\text{out}}$	$F_{\text{api}}$	$C_{\text{emb}}$	$\boldsymbol{S_{\textbf{comp}}}\ (\pm\textbf{CI})$	Cost
DeepSeek-V3.2	0.973	0.635	0.632	0.641	0.761 ( $\pm$ 0.023)	$0.50	0.893	0.532	0.585	0.636	0.681 ( $\pm$ 0.033)	$0.80
Gemini-3-Flash	0.967	0.635	0.662	0.647	0.765 ( $\pm$ 0.024)	$0.80	0.867	0.540	0.532	0.562	0.650 ( $\pm$ 0.036)	$1.50
GPT-5.4	1.000	0.562	0.602	0.633	0.750 ( $\pm$ 0.016)	$8.50	0.967	0.507	0.538	0.628	0.705 ( $\pm$ 0.022)	$12.30
GPT-4.1	0.967	0.601	0.663	0.604	0.747 ( $\pm$ 0.026)	$1.20	0.933	0.503	0.573	0.648	0.695 ( $\pm$ 0.027)	$3.40
Llama-3.3-70B	0.913	0.398	0.610	0.614	0.656 ( $\pm$ 0.029)	$0 †	0.893	0.344	0.577	0.620	0.627 ( $\pm$ 0.030)	$0 †

Table 6: Comprehensive evaluation across all six LLM backends and two architectures, averaged over three repeated runs (50 tasks per cell,

n=150

task-runs per model–architecture).

R_{s}

: task success rate;

Q_{\text{out}}

: output accuracy (L3, successful tasks only);

F_{\text{api}}

: API operation F1 (L1);

C_{\text{emb}}

: embedding cosine similarity (L2);

S_{\text{comp}}

: composite score (Eq. 1 ) with bootstrap 95% confidence interval. Cost is the total API expenditure averaged per benchmark sweep (50 tasks, single run). Bold = best per architecture.

Figure 4: Task success rate comparison (mean over three repeated runs): Single Agent (solid) vs. Dual Agent (hatched). All cloud and large open-weight models show the Single Agent reliably ahead; only Qwen2.5-Coder-14B improves under the Dual Agent.

Model	$\boldsymbol{S^{\text{SA}}_{\text{comp}}}$	$\boldsymbol{S^{\text{DA}}_{\text{comp}}}$	Median $\boldsymbol{\Delta}$	Cliff’s $\boldsymbol{\delta}$	$\boldsymbol{p}$ -value
Gemini-3-Flash	0.7646	0.6501	$+0.067$	$+0.41$	$<10^{-3}$
DeepSeek-V3.2	0.7612	0.6809	$+0.028$	$+0.29$	$<10^{-3}$
GPT-4.1	0.7473	0.6954	$+0.034$	$+0.26$	$<10^{-3}$
GPT-5.4	0.7504	0.7052	$+0.029$	$+0.21$	$<10^{-3}$
Llama-3.3-70B	0.6563	0.6269	$+0.010$	$+0.15$	$0.017$
Qwen2.5-Coder-14B	0.4537	0.5882	$-0.032$	$-0.20$	$<10^{-3}$

Table 7: Paired Wilcoxon signed-rank tests on task-level composite scores: Single Agent vs. Dual Agent.

n=150

paired task-runs per row (50 tasks

\times

3 runs); positive median difference and Cliff’s

\delta

favour SA. All but Llama-3.3-70B reach

p<10^{-3}

Figure 5: Mean composite score $S_{\text{comp}}$ per (model, architecture) cell with 95% bootstrap confidence intervals ( $n=150$ paired task-runs per cell). All four cloud backends and Llama-3.3-70B sit higher under SA than DA; Qwen2.5-Coder-14B is the only backend whose CI bracket clears the corresponding SA bracket on the right.

Single Agent
Model	CodeBLEU	API F1	ROUGE-L
GPT-5.4	0.172	0.602	0.136
GPT-4.1	0.171	0.659	0.142
Gemini Flash	0.168	0.662	0.141
Llama-70B	0.166	0.606	0.137
Qwen-14B	0.162	0.504	0.132

Table 8: Code quality metrics across architectures, averaged over three repeated runs (50 tasks per cell). Bold = best per architecture.

Figure 6: Per-task API F1 scores (DA architecture). Warm colors indicate higher code-level agreement with gold standards. The task-dependent variance highlights the limitation of aggregate metrics.

Figure 7: L2: Execution log embedding cosine similarity per task (mean across three repeated runs). Each dot represents one (model, task) pair. SA and DA produce comparable mean similarity, with DA showing greater task-level variance for the same model.

Single Agent
Model	Task	Data	Method	Self-Corr.	Result	Avg.
DeepSeek	4.14	4.48	3.98	3.44	4.18	4.04
Gemini Flash	3.78	4.10	3.42	3.16	3.46	3.58
GPT-5.4	3.28	3.98	3.14	3.12	3.12	3.33
GPT-4.1	3.26	3.62	3.02	2.48	2.76	3.03
Llama-70B	3.04	3.42	2.52	2.64	1.86	2.70

Table 9: LLM-as-Judge reasoning assessment (1–5 scale). Bold = best per architecture. The five dimensions capture task understanding, data handling, methodology, self-correction, and result completeness.

Figure 8: L3: Per-task output accuracy scores (SA, averaged over three runs). Green = high accuracy; red = failure. Tasks T18 and T27 remain red across most models, indicating intrinsic difficulty.

Figure 9: Multi-dimensional performance profiles for SA (left) and DA (right) architectures. In SA mode, Gemini Flash and DeepSeek lead the four-dimensional composite; in DA mode, GPT-class models hold the highest composite but with compressed dimension ranges across all six backends.

Mean per-task $\Delta$ (with $-$ without) $=+0.114$ ; median $\Delta=+0.153$ .
Condition	$\boldsymbol{S_{\textbf{comp}}}$ mean	std	successes
With workflow	$0.733$	$0.196$	$47/50$
Without workflow	$0.619$	$0.180$	$48/50$
Paired Wilcoxon $p<10^{-3}$ ; Cohen’s $d=+0.51$ (medium effect).

Table 10: Workflow ablation: DeepSeek-V3.2 SA across all 50 benchmark tasks, paired by task (

n=50

). Composite scores follow Eq. 1 . The with-workflow condition delivers a statistically significant medium-sized improvement.

Figure 10: Per-task workflow ablation across all 50 tasks (DeepSeek-V3.2 SA). Top: paired composite scores with vs. without workflow. Bottom: signed per-task difference $\Delta=S_{\mathrm{comp}}^{\mathrm{with}}-S_{\mathrm{comp}}^{\mathrm{without}}$ (slate = with-workflow wins, warm = without wins); the dashed line marks the mean $\Delta=+0.114$ . The majority of tasks favour the with-workflow condition, with a small minority of reversals—most prominently T41, where the supplied workflow encodes a constraint that mismatches the model’s preferred decomposition—outweighed by the consistent positive effect on multi-step pipelines.

Figure 11: T20: Mineral deposit prospectivity prediction outputs (SA mode). Four strong models accurately reproduce the spatial distribution of tin-tungsten prospectivity, while Llama-70B generates a uniform raster and Qwen-14B fails to produce any visualization. This task requires a complete ML pipeline spanning raster feature extraction, Random Forest training, and georeferenced probability mapping.

Figure 12: T34: Road accessibility visualization outputs. Top models accurately identify and render prefecture-level accessibility patterns across Japan.

Figure 13: T07: Flood analysis outputs illustrating domain knowledge gaps. The Gold Standard (left) uses a domain-calibrated $-200$ cm threshold, while all models default to $0$ cm. Some models further suffer CRS misalignment, producing blank maps. This demonstrates that parametric domain knowledge cannot be acquired from code patterns alone.

Figure 14: T08: Fire station coverage gap analysis. The Gold Standard (top-left) uses a 2,500 m buffer; all models select smaller radii (500–1,000 m), dramatically overestimating the uncovered area. This illustrates the parametric domain knowledge gap .

Category	Failure Symptom	Fix Applied
Data Pathing	Nested SHP files cause FileNotFoundError	Flatten dataset structure
API State	rasterio.open() returns reader object; agent calls .astype()	Return raw ndarray + metadata dict
Memory	City-wide grids exceed $10^{4}$ cells; overlay OOM	Enforce grid-size templates
Type Coercion	String-encoded numerics fail during interpolation	Auto-cast with pd.to_numeric

Table 11: Infrastructure-level fixes and their impact on task success.

Geospatial Reasoning Benchmarks

See all 31 papers

GISAgentBench: A Practitioner-Sourced Benchmark for Evaluating LLM Agents on GIS Tasks
Aug 3, 2026Abhinav Pothuri, Zhe Jiang, Zelin Xu +1LLM Agent Evaluation AI Agent Benchmarks
Department of Computer & Information Science & Engineering, University of Florida · Department of Geography, University of Florida
GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models
Jun 11, 2026Gabriel Diaz-Ireland, Diego Prieto-Herráez, Mario García Peces +2LLM Agent Evaluation Geospatial Reasoning
Universidad Católica de Ávila (UCAV) Ávila, Spain · Johns Hopkins University Baltimore, MD, USA · Independent Researcher Madrid, Spain +1
ANASSA: An Agentic AI Orchestration Framework for Spatial Intelligence
Sep 13, 2026Constantinos Papantoniou, Brian HiltonLLM Agent Orchestration Geospatial Reasoning
Anassa.ai USA · Research Scientist USA

Aug 3, 2026J/K move · Enter open · S save

GISAgentBench: A Practitioner-Sourced Benchmark for Evaluating LLM Agents on GIS Tasks

Abhinav Pothuri, Zhe Jiang, Zelin Xu, Di Yang

Department of Computer & Information Science & Engineering, University of Florida · Department of Geography, University of Florida

LLM Agent Evaluation AI Agent Benchmarks Spatial Reasoning Geospatial Reasoning Benchmarks

Open paper AskPDF arXiv

Geographic Information System (GIS) professionals rely on multi-step spatial analysis workflows to support decision-making in urban planning, disaster response, and environmental monitoring. The process is tedious, time-consuming, and error-prone. While recent large language model (LLM) agents equipped with external tools have the potential to automate geospatial analysis, their ability to perform realistic GIS workflows remains largely unexplored. Existing GIS agent benchmarking datasets are mostly drawn from textbooks, tutorials, or LLM-generated seeds and remain limited in size and trajectory depth. More importantly, none provides ground truth outputs. They therefore rely on surrogate signals such as code similarity, trajectory matching, or LLM and VLM judges, which can conflate workflow resemblance with task correctness. To address this gap, we introduce GISAgentBench, a benchmark of 349 multi-step GIS tasks curated from GIS Stack Exchange and instantiated on real public data across six selected geographic areas of interest. Each task ships with an executable reference trajectory and an exact ground truth output file, enabling strict, deterministic, tolerance-aware output matching beyond LLM judging. Evaluations of six LLM models reveal that realistic GIS workflows remain challenging: the best agent completes only 32.7% of tasks under strict tolerance-aware scoring, although most models produce outputs that are close to the ground truth.

Abstract

Geospatial Reasoning Benchmarks

Explore similar work

GISAgentBench: A Practitioner-Sourced Benchmark for Evaluating LLM Agents on GIS Tasks

GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models

ANASSA: An Agentic AI Orchestration Framework for Spatial Intelligence